diff --git a/.claude/skills/exec-local-compile/SKILL.md b/.claude/skills/exec-local-compile/SKILL.md index 9a45283174fe..6a2f3184895d 100644 --- a/.claude/skills/exec-local-compile/SKILL.md +++ b/.claude/skills/exec-local-compile/SKILL.md @@ -45,7 +45,7 @@ git checkout main && git pull Run the build command (**incremental by default** — omit `-c`/`--clean` unless explicitly requested or the incremental build fails): ```bash -./scripts/build_wheel.py --use_ccache -a "" -f --nvtx +./scripts/build_wheel.py --trt_root /usr/local/tensorrt --benchmarks --use_ccache -a "" -f --nvtx ``` Replace `` with the target GPU architecture (see Architecture Reference below). If not specified by the user, auto-detect from `nvidia-smi`. @@ -66,6 +66,8 @@ python3 -c "import tensorrt_llm; print(tensorrt_llm.__version__)" | Flag | Description | |------|-------------| +| `--trt_root /usr/local/tensorrt` | TensorRT installation path (standard in NVIDIA containers) | +| `--benchmarks` | Build the C++ benchmarks | | `-a ""` | Target GPU architecture(s) | | `--nvtx` | Enable NVTX markers for profiling | | `--use_ccache` | Use ccache for faster recompilation | diff --git a/.claude/skills/exec-slurm-compile/SKILL.md b/.claude/skills/exec-slurm-compile/SKILL.md index 8a8005d06d7a..44c60ed374bd 100644 --- a/.claude/skills/exec-slurm-compile/SKILL.md +++ b/.claude/skills/exec-slurm-compile/SKILL.md @@ -204,6 +204,8 @@ A successful build ends with a message like `Successfully built tensorrt_llm` or | Flag | Description | |------|-------------| +| `--trt_root /usr/local/tensorrt` | TensorRT installation path (standard in NVIDIA containers) | +| `--benchmarks` | Build the C++ benchmarks | | `-a "100-real"` | Target architecture — `100` for Blackwell, `90` for Hopper, etc. | | `--nvtx` | Enable NVTX markers for profiling | | `--no-venv` | Skip virtual environment creation | @@ -226,6 +228,7 @@ Common architecture values: | `sbatch: error: invalid partition` | Verify partition name with `sinfo -s` | | `sbatch: error: invalid account` | Check available accounts with `sacctmgr show assoc user=$USER` | | Container image not found | Verify the `.sqsh` path exists and is readable | +| Build fails with missing TensorRT | Ensure `--trt_root` points to the correct path inside the container | | Build OOM (out of memory) | Reduce parallelism with `-j ` flag to `build_wheel.py` | | `srun: error: Unable to create step` | The node may lack enroot/pyxis — check with cluster admin | | Job stuck in `PD` state | Check `squeue -j -o %R` for the reason (e.g., resource limits, priority) | diff --git a/.claude/skills/exec-slurm-compile/scripts/compile.sh b/.claude/skills/exec-slurm-compile/scripts/compile.sh index 4822dbceb677..22b7882d55ff 100755 --- a/.claude/skills/exec-slurm-compile/scripts/compile.sh +++ b/.claude/skills/exec-slurm-compile/scripts/compile.sh @@ -19,7 +19,7 @@ # Usage: compile.sh [build_wheel_args...] # # Default build_wheel.py flags: -# -a "100-real" --nvtx --no-venv +# --trt_root /usr/local/tensorrt --benchmarks -a "100-real" --nvtx --no-venv # Any extra arguments after repo_dir are forwarded to build_wheel.py, # overriding the defaults above. @@ -36,6 +36,8 @@ if [[ $# -gt 0 ]]; then else echo "[compile.sh] Running default build command" python3 ./scripts/build_wheel.py \ + --trt_root /usr/local/tensorrt \ + --benchmarks \ -a "100-real" \ --nvtx fi diff --git a/.coderabbit.yaml b/.coderabbit.yaml index 2caf9db60128..dc89ff60b2b8 100644 --- a/.coderabbit.yaml +++ b/.coderabbit.yaml @@ -22,26 +22,7 @@ reviews: auto_title_placeholder: '@coderabbitai title' auto_title_instructions: 'Format: "[] ". Category must be one of: fix, feat, doc, infra, style, refactor, perf, test, chore, revert. Enclose the category in square brackets. Title should be concise (<= 60 chars). Example: "[feat] Add logit_bias support".' commit_status: false - high_level_summary_instructions: | - Always produce two review sections in the summary: - - **Dev Engineer Review** - Review all changes for correctness and consistency, including: - - Code changes: correctness, performance, API consistency (CODING_GUIDELINES.md), error handling, regressions. - - Config files: valid values, no typos, consistency with related configs, no unintended scope changes. - - Test list files (test-db/, qa/, waives.txt): correct format, valid test paths, appropriate bug references, no duplicates. - - **QA Engineer Review** - Always include this section when any files under tests/ are touched. - For test-list-only changes (only tests/integration/test_lists/ files): - - List which test-db/ or qa/ files were modified and what entries were added or removed. - - Verdict: "needs follow-up" if CBTS coverage data is unavailable, otherwise "sufficient" or "insufficient". - For test-code changes (files outside tests/integration/test_lists/): - - List test functions added, modified, or removed. - - State whether each is covered in tests/integration/test_lists/ (test-db/ for CI, qa/ for manual QA). - - Verdict: sufficient, insufficient, or needs follow-up. - If no test files are touched, write "No test changes." - collapse_walkthrough: false + collapse_walkthrough: true assess_linked_issues: true related_issues: true related_prs: true @@ -50,27 +31,15 @@ reviews: poem: false review_status: false auto_review: - auto_incremental_review: true + auto_incremental_review: false drafts: false base_branches: ["main", "release/.+"] path_instructions: - path: "tests/**" instructions: | Act as a QA engineer reviewing test changes and coverage for TensorRT-LLM. - Always produce a test coverage summary, even if no issues are found. - - If the change touches ONLY files under tests/integration/test_lists/ (no test-code changes): - - Report which test-db/ or qa/ list files were modified and what entries were added or removed. - - Do NOT require changed test functions for this path. - - Use verdict "needs follow-up" when cbts_touchmap.sqlite or a CBTS coverage report is unavailable - to confirm the impacted test scope; otherwise use "sufficient" or "insufficient". - - If the change includes test-code files (outside tests/integration/test_lists/), the summary must include: - 1. Which test functions were added, modified, or removed. - 2. Whether each changed test is listed in the appropriate test list files under - tests/integration/test_lists/ (test-db/ for CI, qa/ for manual QA). - 3. A coverage verdict: sufficient, insufficient, or needs follow-up. - Keep feedback actionable: reference concrete list file names when suggesting additions. + Keep feedback actionable: suggest concrete list file names and whether + coverage is sufficient, insufficient, or needs follow-up outside the PR. - path: "tests/integration/test_lists/qa/**" instructions: | Files here are manually-triggered QA perf/regression lists, maintained diff --git a/.gitattributes b/.gitattributes index a8ca34bc5151..5f797908576a 100644 --- a/.gitattributes +++ b/.gitattributes @@ -16,5 +16,4 @@ docs/source/blogs/media/tech_blog10_full_strategy_performance.png filter=lfs dif docs/source/blogs/media/tech_blog10_context_wait_performance.png filter=lfs diff=lfs merge=lfs -text cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/kernelMetaInfo_cubin.cpp filter=lfs diff=lfs merge=lfs -text cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/cubin/xqa_kernel_cubin.cpp filter=lfs diff=lfs merge=lfs -text -docs/source/blogs/media/tech_blog26_deepseek_v4_hybrid_attention.png filter=lfs diff=lfs merge=lfs -text -docs/source/blogs/media/tech_blog26_deepseek_v4_mhc_moe.png filter=lfs diff=lfs merge=lfs -text +tensorrt_llm/_torch/visual_gen/cute_dsl_kernels/blackwell/attention/cubins/*/*/*.so filter=lfs diff=lfs merge=lfs -text diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 4c72f3495a99..caca5ca47aaa 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,414 +1,219 @@ -# ============================================================================ -# PRECEDENCE: last matching pattern wins — fallback first, module rules next, governance last. -# -# Domain (in file order) Handle -# Engineering baseline/ fallback / untriaged @NVIDIA/trt-llm-devs (global `*`, declared FIRST; incl. deprecating TRT backend) -# Multi-GPU CI gate @NVIDIA/trt-llm-multi-gpu-ci-review -# Infra @NVIDIA/trt-llm-infra-devs -# Agent config @NVIDIA/trt-llm-agent-devs -# Docs / Examples @NVIDIA/trt-llm-doc-owners -# QA @NVIDIA/trt-llm-qa -# Runtime @NVIDIA/trt-llm-runtime-devs -# Kernels - Misc @NVIDIA/trt-llm-kernels-devs -# Models @NVIDIA/trt-llm-models-devs -# General Perf @NVIDIA/trt-llm-perf-devs -# KV Cache Manager @NVIDIA/trt-llm-kv-cache-manager-devs -# Disaggregated Serving @NVIDIA/trt-llm-disagg-devs -# Attention @NVIDIA/trt-llm-torch-attention-devs -# MoE @NVIDIA/trt-llm-moe-devs -# VisualGen (AIGV) @NVIDIA/trt-llm-torch-visual-gen-devs -# Scaffolding @WeiHaocheng -# Self-managed: AutoDeploy, Triton backend, Dynamo. Governance/legal: unchanged. -# -# New handles (trt-llm-devs / doc / agent / runtime / models / perf / moe / kernels / multi-gpu-ci-review) -# must be created & populated before activation (unknown owner blocks merge). -# ============================================================================ - -# ===== FALLBACK (trt-llm-devs) ===== -# Global net; every rule below overrides it. The explicit entries are visible parking (redundant with `*`). -* @NVIDIA/trt-llm-devs - -/tensorrt_llm/commands/eval.py @NVIDIA/trt-llm-devs -/tensorrt_llm/evaluate @NVIDIA/trt-llm-devs -/tensorrt_llm/tools @NVIDIA/trt-llm-devs -/tests/integration/test_lists/test-db @NVIDIA/trt-llm-devs @NVIDIA/trt-llm-qa @NVIDIA/trt-llm-infra-devs -/tests/integration/test_lists/waives.txt @NVIDIA/trt-llm-devs @NVIDIA/trt-llm-qa @NVIDIA/trt-llm-infra-devs -/tests/test_common @NVIDIA/trt-llm-devs -/tests/unittest @NVIDIA/trt-llm-devs - -# ===== TensorRT backend (will be deprecated soon) — also on the trt-llm-devs fallback ===== -/tensorrt_llm/builder.py @NVIDIA/trt-llm-devs -/tensorrt_llm/commands/build.py @NVIDIA/trt-llm-devs -/tensorrt_llm/commands/prune.py @NVIDIA/trt-llm-devs -/tensorrt_llm/commands/refit.py @NVIDIA/trt-llm-devs -/tensorrt_llm/functional.py @NVIDIA/trt-llm-devs -/tensorrt_llm/graph_rewriting.py @NVIDIA/trt-llm-devs -/tensorrt_llm/layers @NVIDIA/trt-llm-devs -/tensorrt_llm/models @NVIDIA/trt-llm-devs -/tensorrt_llm/module.py @NVIDIA/trt-llm-devs -/tensorrt_llm/network.py @NVIDIA/trt-llm-devs -/tensorrt_llm/parameter.py @NVIDIA/trt-llm-devs -/tensorrt_llm/plugin @NVIDIA/trt-llm-devs -/tensorrt_llm/python_plugin.py @NVIDIA/trt-llm-devs -/tensorrt_llm/runtime @NVIDIA/trt-llm-devs -/tensorrt_llm/tools/plugin_gen @NVIDIA/trt-llm-devs -/tensorrt_llm/top_model_mixin.py @NVIDIA/trt-llm-devs - -# ===== MULTI-GPU / MULTI-NODE CI TEST-LIST GATE (review group, NEW) ===== -# A PR adding/expanding these expensive cases must justify WHAT is tested, WHY existing coverage is -# insufficient, and WHY a unit / single-GPU test can't do it. (Handle is a placeholder.) -/tests/integration/test_lists/test-db/*multi_gpu* @NVIDIA/trt-llm-multi-gpu-ci-review -/tests/integration/test_lists/test-db/*multi_node* @NVIDIA/trt-llm-multi-gpu-ci-review -/tests/integration/test_lists/test-db/l0_dgx_* @NVIDIA/trt-llm-multi-gpu-ci-review - -# ===== INFRA ===== -/.coderabbit.yaml @NVIDIA/trt-llm-infra-devs -/.github @NVIDIA/trt-llm-infra-devs -/.pre-commit-config.yaml @NVIDIA/trt-llm-infra-devs -/docker @NVIDIA/trt-llm-infra-devs -/enroot @NVIDIA/trt-llm-infra-devs -/jenkins @NVIDIA/trt-llm-infra-devs -/ruff-legacy-baseline.json @NVIDIA/trt-llm-infra-devs -/ruff-legacy.toml @NVIDIA/trt-llm-infra-devs -/scripts @NVIDIA/trt-llm-infra-devs -/security_scanning @NVIDIA/trt-llm-infra-devs - -# ===== AGENT ===== -/.claude @NVIDIA/trt-llm-agent-devs -/.codex @NVIDIA/trt-llm-agent-devs -/AGENTS.md @NVIDIA/trt-llm-agent-devs -/CLAUDE.md @NVIDIA/trt-llm-agent-devs -/scripts/check_skill_naming_convention.py @NVIDIA/trt-llm-agent-devs - -# ===== DOCS / EXAMPLES ===== -/CODE_OF_CONDUCT.md @NVIDIA/trt-llm-doc-owners +# This file defines code ownership rules for the repository. + +## TensorRT-LLM QA +### Integration Tests +/tests/integration/test_lists/qa @NVIDIA/trt-llm-qa +/tests/integration/defs/examples/test_ray.py @NVIDIA/trt-llm-qa-function +/tests/integration/defs/examples/test_redrafter.py @NVIDIA/trt-llm-qa-function +/tests/integration/defs/accuracy @NVIDIA/trt-llm-qa-function +/tests/integration/defs/stress_test @NVIDIA/trt-llm-qa-function +/tests/integration/defs/triton_server @NVIDIA/trt-llm-qa-function +/tests/integration/defs/test_e2e.py @NVIDIA/trt-llm-qa-function +/tests/integration/defs/disaggregated @NVIDIA/trt-llm-qa-serving +/tests/integration/defs/sysinfo @NVIDIA/trt-llm-qa-perf +/tests/integration/defs/perf @NVIDIA/trt-llm-qa-perf +/tests/integration/defs/perf/disagg @NVIDIA/trt-llm-qa-serving + +## TensorRT-LLM Infra +### CI +/jenkins @NVIDIA/trt-llm-ci-infra-devs @NVIDIA/trt-llm-infra-devs +### Setup +/docker @NVIDIA/trt-llm-setup-infra-devs @NVIDIA/trt-llm-infra-devs +/.pre-commit-config.yaml @NVIDIA/trt-llm-setup-infra-devs @NVIDIA/trt-llm-infra-devs +### Github workflows +/.github @NVIDIA/trt-llm-gh-workflows-infra-devs @NVIDIA/trt-llm-infra-devs +/.coderabbit.yaml @NVIDIA/trt-llm-gh-workflows-infra-devs @NVIDIA/trt-llm-infra-devs + +## TensorRT-LLM - Docs +/docs @NVIDIA/trt-llm-doc-owners /CODING_GUIDELINES.md @NVIDIA/trt-llm-doc-owners +/CODE_OF_CONDUCT.md @NVIDIA/trt-llm-doc-owners /CONTAINER_SOURCE.md @NVIDIA/trt-llm-doc-owners /CONTRIBUTING.md @NVIDIA/trt-llm-doc-owners /README.md @NVIDIA/trt-llm-doc-owners -/SECURITY.md @NVIDIA/trt-llm-doc-owners -/docs @NVIDIA/trt-llm-doc-owners +/CLAUDE.md @NVIDIA/trt-llm-doc-owners +/AGENTS.md @NVIDIA/trt-llm-doc-owners + +## Examples /examples @NVIDIA/trt-llm-doc-owners -# ===== QA ===== -/tests/integration/defs @NVIDIA/trt-llm-devs @NVIDIA/trt-llm-qa @NVIDIA/trt-llm-infra-devs -/tests/integration/test_lists/qa @NVIDIA/trt-llm-qa +## TensorRT-LLM - Triton backend +/triton_backend @NVIDIA/trt-llm-triton-backend-devs -# ===== RUNTIME ===== -/cpp/include/tensorrt_llm/batch_manager @NVIDIA/trt-llm-runtime-devs -/cpp/include/tensorrt_llm/common @NVIDIA/trt-llm-runtime-devs -/cpp/include/tensorrt_llm/executor @NVIDIA/trt-llm-runtime-devs -/cpp/include/tensorrt_llm/layers @NVIDIA/trt-llm-runtime-devs -/cpp/include/tensorrt_llm/runtime @NVIDIA/trt-llm-runtime-devs -/cpp/tensorrt_llm/batch_manager @NVIDIA/trt-llm-runtime-devs -/cpp/tensorrt_llm/common @NVIDIA/trt-llm-runtime-devs -/cpp/tensorrt_llm/executor @NVIDIA/trt-llm-runtime-devs -/cpp/tensorrt_llm/layers @NVIDIA/trt-llm-runtime-devs -/cpp/tensorrt_llm/nanobind @NVIDIA/trt-llm-runtime-devs -/cpp/tensorrt_llm/runtime @NVIDIA/trt-llm-runtime-devs -/cpp/tensorrt_llm/testing @NVIDIA/trt-llm-runtime-devs -/cpp/tests @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/_tensorrt_engine @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/_torch @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/commands/__init__.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/commands/serve.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/commands/utils.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/executor @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/grpc @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/inputs @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/llmapi @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/llmapi/mm_encoder.py @NVIDIA/trt-llm-runtime-devs @NVIDIA/trt-llm-models-devs -/tensorrt_llm/lora_helper.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/lora_manager.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/mapping.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/metrics @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/prompt_adapter_manager.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/sampling_params.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/scheduling_params.py @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/serve @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/tokenizer @NVIDIA/trt-llm-runtime-devs -/tensorrt_llm/usage @NVIDIA/trt-llm-runtime-devs -/tests/torch @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/compilation @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/custom_ops @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/distributed @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/executor @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/lora @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/misc/test_autotuner.py @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/modules @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/multi_gpu @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/pyexecutor @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/ray_orchestrator @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/sampler @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/speculative @NVIDIA/trt-llm-runtime-devs -/tests/unittest/_torch/speculative_hw_agnostic @NVIDIA/trt-llm-runtime-devs -/tests/unittest/executor @NVIDIA/trt-llm-runtime-devs -/tests/unittest/inputs @NVIDIA/trt-llm-runtime-devs -/tests/unittest/llmapi @NVIDIA/trt-llm-runtime-devs -/tests/unittest/llmapi/apps/*multimodal* @NVIDIA/trt-llm-runtime-devs @NVIDIA/trt-llm-models-devs -/tests/unittest/pyexecutor @NVIDIA/trt-llm-runtime-devs - -# ===== KERNELS - MISC ===== -/cpp/include/tensorrt_llm/deep_gemm @NVIDIA/trt-llm-kernels-devs -/cpp/include/tensorrt_llm/kernels @NVIDIA/trt-llm-kernels-devs -/cpp/tensorrt_llm/cutlass_extensions @NVIDIA/trt-llm-kernels-devs -/cpp/tensorrt_llm/deep_gemm @NVIDIA/trt-llm-kernels-devs -/cpp/tensorrt_llm/kernels @NVIDIA/trt-llm-kernels-devs -/cpp/tensorrt_llm/thop @NVIDIA/trt-llm-kernels-devs -/cpp/tests/unit_tests/kernels @NVIDIA/trt-llm-kernels-devs -/tensorrt_llm/_torch/cuda_tile_kernels @NVIDIA/trt-llm-kernels-devs -/tensorrt_llm/_torch/cute_dsl_kernels @NVIDIA/trt-llm-kernels-devs -/tests/scripts/cute_dsl_kernels @NVIDIA/trt-llm-kernels-devs -/tests/unittest/_torch/thop @NVIDIA/trt-llm-kernels-devs - -# ===== MODELS ===== -/docs/source/features/multi-modality.md @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-doc-owners -/examples/llm-api/quickstart_multimodal.py @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-doc-owners -/examples/models @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-doc-owners -/examples/serve/*multimodal* @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-doc-owners -/scripts/generate_config_database_tests.py @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-doc-owners -/scripts/generate_config_table.py @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-doc-owners -/tensorrt_llm/_torch/models @NVIDIA/trt-llm-models-devs -/tensorrt_llm/_torch/modules/mamba @NVIDIA/trt-llm-models-devs -/tensorrt_llm/quantization @NVIDIA/trt-llm-models-devs -/tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-qa -/tests/unittest/_torch/modeling @NVIDIA/trt-llm-models-devs -/tests/unittest/_torch/models @NVIDIA/trt-llm-models-devs -/tests/unittest/_torch/modules/mamba @NVIDIA/trt-llm-models-devs -/tests/unittest/_torch/multi_gpu_modeling @NVIDIA/trt-llm-models-devs -/tests/unittest/_torch/multimodal @NVIDIA/trt-llm-models-devs -/tests/unittest/models @NVIDIA/trt-llm-models-devs -/tests/unittest/others/test_multimodal_registry.py @NVIDIA/trt-llm-models-devs - -# ===== GENERAL PERF ===== -/benchmarks @NVIDIA/trt-llm-perf-devs -/cpp/micro_benchmarks @NVIDIA/trt-llm-perf-devs -/docs/source/performance/perf-benchmarking.md @NVIDIA/trt-llm-perf-devs -/scripts/check_pinned_memory_usage.py @NVIDIA/trt-llm-perf-devs -/tensorrt_llm/bench @NVIDIA/trt-llm-perf-devs -/tensorrt_llm/commands/bench.py @NVIDIA/trt-llm-perf-devs -/tensorrt_llm/tools/layer_wise_benchmarks @NVIDIA/trt-llm-perf-devs -/tensorrt_llm/tools/profiler @NVIDIA/trt-llm-perf-devs -/tests/microbenchmarks @NVIDIA/trt-llm-perf-devs -/tests/scripts/allreduce_perf @NVIDIA/trt-llm-perf-devs -/tests/scripts/iteration_log_parser.py @NVIDIA/trt-llm-perf-devs -/tests/scripts/perf @NVIDIA/trt-llm-perf-devs -/tests/scripts/perf-sanity @NVIDIA/trt-llm-perf-devs - -# ===== KV CACHE MANAGER ===== -/cpp/include/tensorrt_llm/batch_manager/allocateKvCache* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/include/tensorrt_llm/batch_manager/blockKey* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/include/tensorrt_llm/batch_manager/evictionPolicy* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/include/tensorrt_llm/batch_manager/kvCache* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/include/tensorrt_llm/batch_manager/radixBlockTree* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/include/tensorrt_llm/batch_manager/reorderPolicy* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/include/tensorrt_llm/batch_manager/stringSetTrie* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/include/tensorrt_llm/batch_manager/templatedTrie* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tensorrt_llm/batch_manager/allocateKvCache* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tensorrt_llm/batch_manager/blockKey* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tensorrt_llm/batch_manager/evictionPolicy* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tensorrt_llm/batch_manager/kvCache* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tests/unit_tests/batch_manager/blockKey* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tests/unit_tests/batch_manager/evictionPolicy* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tests/unit_tests/batch_manager/kvCache* @NVIDIA/trt-llm-kv-cache-manager-devs -/cpp/tests/unit_tests/batch_manager/radixBlockTree* @NVIDIA/trt-llm-kv-cache-manager-devs -/tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py @NVIDIA/trt-llm-kv-cache-manager-devs -/tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py @NVIDIA/trt-llm-kv-cache-manager-devs -/tensorrt_llm/_torch/pyexecutor/resource_manager.py @NVIDIA/trt-llm-kv-cache-manager-devs -/tensorrt_llm/runtime/kv_cache_manager_v2 @NVIDIA/trt-llm-kv-cache-manager-devs -/tests/unittest/_torch/executor/test_dual_pool_kv_cache.py @NVIDIA/trt-llm-kv-cache-manager-devs -/tests/unittest/_torch/executor/test_kv_cache* @NVIDIA/trt-llm-kv-cache-manager-devs -/tests/unittest/_torch/executor/test_kv_pool_rebalance.py @NVIDIA/trt-llm-kv-cache-manager-devs -/tests/unittest/_torch/executor/test_kvcache_aware_router.py @NVIDIA/trt-llm-kv-cache-manager-devs -/tests/unittest/_torch/executor/test_mamba_cache_manager.py @NVIDIA/trt-llm-kv-cache-manager-devs -/tests/unittest/kv_cache_manager_v2_tests @NVIDIA/trt-llm-kv-cache-manager-devs - -# ===== DISAGGREGATED SERVING ===== -/cpp/include/tensorrt_llm/batch_manager/*Formatter* @NVIDIA/trt-llm-disagg-devs -/cpp/include/tensorrt_llm/batch_manager/*TransBuffer* @NVIDIA/trt-llm-disagg-devs -/cpp/include/tensorrt_llm/batch_manager/*Transceiver* @NVIDIA/trt-llm-disagg-devs -/cpp/include/tensorrt_llm/batch_manager/cacheTransferLayer* @NVIDIA/trt-llm-disagg-devs -/cpp/include/tensorrt_llm/batch_manager/disagg* @NVIDIA/trt-llm-disagg-devs -/cpp/tensorrt_llm/batch_manager/*Formatter* @NVIDIA/trt-llm-disagg-devs -/cpp/tensorrt_llm/batch_manager/*TransBuffer* @NVIDIA/trt-llm-disagg-devs -/cpp/tensorrt_llm/batch_manager/*Transceiver* @NVIDIA/trt-llm-disagg-devs -/cpp/tensorrt_llm/batch_manager/cacheTransferLayer* @NVIDIA/trt-llm-disagg-devs -/cpp/tensorrt_llm/batch_manager/disagg* @NVIDIA/trt-llm-disagg-devs -/examples/disaggregated @NVIDIA/trt-llm-disagg-devs @NVIDIA/trt-llm-doc-owners -/examples/disaggregated/slurm/benchmark @NVIDIA/trt-llm-disagg-devs @NVIDIA/trt-llm-perf-devs -/tensorrt_llm/_torch/disaggregation @NVIDIA/trt-llm-disagg-devs -/tensorrt_llm/_torch/pyexecutor/kv_cache_transceiver.py @NVIDIA/trt-llm-disagg-devs -/tensorrt_llm/disaggregated_params.py @NVIDIA/trt-llm-disagg-devs -/tensorrt_llm/serve/openai_disagg_server.py @NVIDIA/trt-llm-disagg-devs -# Disagg tests: co-own with the owning team so disagg-devs review disagg-test changes. -/tests/integration/defs/accuracy/*disagg* @NVIDIA/trt-llm-disagg-devs @NVIDIA/trt-llm-qa -/tests/integration/defs/disaggregated @NVIDIA/trt-llm-disagg-devs @NVIDIA/trt-llm-qa -/tests/integration/defs/stress_test/disagg_cancel @NVIDIA/trt-llm-disagg-devs @NVIDIA/trt-llm-qa -/tests/scripts/perf-sanity/disaggregated @NVIDIA/trt-llm-perf-devs @NVIDIA/trt-llm-disagg-devs -/tests/scripts/perf/disaggregated @NVIDIA/trt-llm-perf-devs @NVIDIA/trt-llm-disagg-devs -/tests/unittest/_torch/executor/*disagg* @NVIDIA/trt-llm-runtime-devs @NVIDIA/trt-llm-disagg-devs -/tests/unittest/_torch/multimodal/*disagg* @NVIDIA/trt-llm-models-devs @NVIDIA/trt-llm-disagg-devs -/tests/unittest/disaggregated @NVIDIA/trt-llm-disagg-devs -/tests/unittest/llmapi/*disagg* @NVIDIA/trt-llm-runtime-devs @NVIDIA/trt-llm-disagg-devs -/tests/unittest/llmapi/apps/*disagg* @NVIDIA/trt-llm-runtime-devs @NVIDIA/trt-llm-disagg-devs - -# ===== ATTENTION ===== -# Where a kernel is both a dir and sibling .cu/.h, keep the bare dir (subtree) AND a trailing-* (siblings). -/cpp/kernels @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/common/attention* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/flash_mla @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/IndexerKCache* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/IndexerTopK* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/attentionMask* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/buildRelativeAttentionBiasKernel* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/compressorKernels @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/convertReqIndexToGlobal* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/deepseekV4QNormKernel* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/flashMLA @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/fmhaDispatcher* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/fusedCatFp4* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/fusedCatFp8* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/gptKernels* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/helix* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/indexerKCache* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/indexerTopK* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/inverseRopeFp8QuantKernel* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/mla* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/multiHeadAttentionCommon.h @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/sparseAttentionKernels* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/unfusedAttentionKernels @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/unfusedAttentionKernels* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/kernels/xqaDispatcher* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/nanobind/thop @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/IndexerKCache* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/IndexerTopK* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/attention* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/compressorOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/convertReqIndexToGlobalOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/deepseekV4QNormOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/dsv3RopeOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/fmhaPackMaskOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/fusedCatFp4Op.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/fusedCatFp8Op.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/fusedQKNormRopeOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/helixPostProcessOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/inverseRopeFp8QuantOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/mla* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/relativeAttentionBiasOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tensorrt_llm/thop/trtllmGenQKVProcessOp.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tests/unit_tests/common/attentionWorkspaceTest.cpp @NVIDIA/trt-llm-torch-attention-devs -/cpp/tests/unit_tests/kernels/cascadeAttention* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tests/unit_tests/kernels/mla* @NVIDIA/trt-llm-torch-attention-devs -/cpp/tests/unit_tests/kernels/ropeTest.cu @NVIDIA/trt-llm-torch-attention-devs -/cpp/tests/unit_tests/kernels/sparseAttentionKernelsTest.cpp @NVIDIA/trt-llm-torch-attention-devs -/tensorrt_llm/_torch/attention_backend @NVIDIA/trt-llm-torch-attention-devs -/tensorrt_llm/_torch/cute_dsl_kernels/blackwell/paged_mqa_logits @NVIDIA/trt-llm-torch-attention-devs -/tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md @NVIDIA/trt-llm-torch-attention-devs -/tensorrt_llm/_torch/modules/attention.py @NVIDIA/trt-llm-torch-attention-devs -/tensorrt_llm/_torch/modules/cross_attention.py @NVIDIA/trt-llm-torch-attention-devs -/tensorrt_llm/_torch/modules/mla.py @NVIDIA/trt-llm-torch-attention-devs -/tensorrt_llm/_torch/modules/qk_norm_attention.py @NVIDIA/trt-llm-torch-attention-devs -/tensorrt_llm/_torch/modules/rotary_embedding.py @NVIDIA/trt-llm-torch-attention-devs -/tests/scripts/cute_dsl_kernels/paged_mqa_logits @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/attention @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/custom_ops/test_deepseek_v4_q_norm.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/custom_ops/test_fused_inv_rope_fp8_quant.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/modules/helix_test_utils.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/modules/test_mha_helix.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/modules/test_mla_helix.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/modules/test_rotary_embedding.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/thop/parallel/test_indexer_topk.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/thop/parallel_hw_agnostic/test_fused_qk_norm_rope.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/thop/parallel_hw_agnostic/test_helix_postprocess.py @NVIDIA/trt-llm-torch-attention-devs -/tests/unittest/_torch/thop/serial/test_fused_cat_fp8.py @NVIDIA/trt-llm-torch-attention-devs - -# ===== MoE ===== -/cpp/tensorrt_llm/deep_ep @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/*Moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/moeLoadBalance @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/thop/*Moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/thop/moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/kernel/*moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/communicationKernels/moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/cuteDslKernels/moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_gemm_kernels.h @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_util_kernels.h @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/internal_cutlass_kernels/include/moe_gemm_kernels.h @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/internal_cutlass_kernels/include/moe_kernels.h @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/llama4MinLatencyKernels/*Moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/marlin/*moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm @NVIDIA/trt-llm-kernels-devs @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/nanobind/runtime/moe* @NVIDIA/trt-llm-moe-devs -/cpp/tensorrt_llm/runtime/moeLoadBalancer @NVIDIA/trt-llm-moe-devs -/tensorrt_llm/_torch/cute_dsl_kernels/blackwell/moe_as_dense_gemm @NVIDIA/trt-llm-moe-devs -/tensorrt_llm/_torch/cute_dsl_kernels/mega_moe_nvfp4 @NVIDIA/trt-llm-moe-devs -/tensorrt_llm/_torch/custom_ops/cute_dsl_megamoe_custom_op.py @NVIDIA/trt-llm-moe-devs -/tensorrt_llm/_torch/distributed/moe_alltoall.py @NVIDIA/trt-llm-moe-devs -/tensorrt_llm/_torch/expert_statistic.py @NVIDIA/trt-llm-moe-devs -/tensorrt_llm/_torch/modules/fused_moe @NVIDIA/trt-llm-moe-devs -/tensorrt_llm/_torch/modules/fused_shared_expert.py @NVIDIA/trt-llm-moe-devs -/tensorrt_llm/deep_ep @NVIDIA/trt-llm-moe-devs -/tests/microbenchmarks/bench_moe @NVIDIA/trt-llm-moe-devs -/tests/microbenchmarks/bench_moe_comm.py @NVIDIA/trt-llm-moe-devs -/tests/microbenchmarks/compare_moe_comm.py @NVIDIA/trt-llm-moe-devs -/tests/scripts/cute_dsl_kernels/moe_as_dense_gemm @NVIDIA/trt-llm-moe-devs -/tests/scripts/cute_dsl_kernels/moe_workload_generator.py @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/modules/dwdp/test_dwdp_fixup_moe_backends.py @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/modules/fused_moe @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/modules/moe @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/modules/test_fused_moe.py @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/modules/test_fused_shared_expert.py @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/modules/test_moe_*.py @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/multi_gpu/test_moe_a2a.py @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/thop/parallel/*moe* @NVIDIA/trt-llm-moe-devs -/tests/unittest/_torch/thop/serial/*moe* @NVIDIA/trt-llm-moe-devs -/tests/unittest/bindings/test_bindings_moe.py @NVIDIA/trt-llm-moe-devs - -# ===== VisualGen / AIGV ===== -/cpp/tensorrt_llm/thop/fusedDiT* @NVIDIA/trt-llm-torch-visual-gen-devs -/examples/visual_gen @NVIDIA/trt-llm-torch-visual-gen-devs -/scripts/visualgen_eval @NVIDIA/trt-llm-torch-visual-gen-devs +# TensorRT-LLM Pytorch backend +/tensorrt_llm/_torch @NVIDIA/trt-llm-torch-devs + +## TensorRT-LLM Pytorch - VisualGen /tensorrt_llm/_torch/visual_gen @NVIDIA/trt-llm-torch-visual-gen-devs /tensorrt_llm/_torch/visual_gen/attention_backend @NVIDIA/trt-llm-torch-attention-devs @NVIDIA/trt-llm-torch-visual-gen-devs /tensorrt_llm/_torch/visual_gen/modules/attention.py @NVIDIA/trt-llm-torch-attention-devs @NVIDIA/trt-llm-torch-visual-gen-devs -/tensorrt_llm/media @NVIDIA/trt-llm-torch-visual-gen-devs -/tensorrt_llm/visual_gen @NVIDIA/trt-llm-torch-visual-gen-devs @NVIDIA/trt-llm-runtime-devs +/tensorrt_llm/visual_gen @NVIDIA/trt-llm-llmapi-devs /tests/integration/defs/examples/test_visual_gen.py @NVIDIA/trt-llm-torch-visual-gen-devs /tests/integration/defs/visual_gen @NVIDIA/trt-llm-torch-visual-gen-devs /tests/scripts/perf-sanity/visual_gen @NVIDIA/trt-llm-torch-visual-gen-devs -/tests/unittest/_torch/thop/parallel_hw_agnostic/test_fused_dit* @NVIDIA/trt-llm-torch-visual-gen-devs /tests/unittest/_torch/visual_gen @NVIDIA/trt-llm-torch-visual-gen-devs -/tests/unittest/visual_gen @NVIDIA/trt-llm-torch-visual-gen-devs -# ===== SCAFFOLDING ===== -/tensorrt_llm/scaffolding @WeiHaocheng -/tests/unittest/scaffolding @WeiHaocheng +## TensorRT-LLM Pytorch - Modules +/tensorrt_llm/_torch/modules @NVIDIA/trt-llm-torch-modules -# ===== SELF-MANAGED ===== -/docs/source/features/auto_deploy @NVIDIA/trt-llm-torch-autodeploy-devs @NVIDIA/trt-llm-doc-owners -/examples/auto_deploy @NVIDIA/trt-llm-torch-autodeploy-devs -/scripts/check_auto_deploy_imports.py @NVIDIA/trt-llm-torch-autodeploy-devs -/scripts/check_model_registry.py @NVIDIA/trt-llm-torch-autodeploy-devs +## TensorRT-LLM Pytorch Models +/tensorrt_llm/_torch/models @NVIDIA/trt-llm-torch-models-devs +/examples/models @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-doc-owners + +## TensorRT-LLM Pytorch backend - runtime +/tensorrt_llm/_torch/pyexecutor @NVIDIA/trt-llm-torch-runtime-devs +## TensorRT-LLM Pytorch backend - AutoDeploy flow /tensorrt_llm/_torch/auto_deploy @NVIDIA/trt-llm-torch-autodeploy-devs -/tests/integration/defs/accuracy/test_llm_api_autodeploy.py @NVIDIA/trt-llm-torch-autodeploy-devs @NVIDIA/trt-llm-qa -/tests/unittest/_torch/auto_deploy @NVIDIA/trt-llm-torch-autodeploy-devs +/examples/auto_deploy @NVIDIA/trt-llm-torch-autodeploy-devs +/docs/source/features/auto_deploy @NVIDIA/trt-llm-torch-autodeploy-devs @NVIDIA/trt-llm-doc-owners /tests/unittest/auto_deploy @NVIDIA/trt-llm-torch-autodeploy-devs /tests/integration/defs/accuracy/test_llm_api_autodeploy.py @NVIDIA/trt-llm-torch-autodeploy-devs @NVIDIA/trt-llm-qa-function +## TensorRT-LLM Pytorch - Speculative Decoding +/tensorrt_llm/_torch/speculative @NVIDIA/trt-llm-torch-spec-decoding + +## TensorRT-LLM Pytorch - Graph Compiler +/tensorrt_llm/_torch/compilation @NVIDIA/trt-llm-torch-graph-compiler +/tensorrt_llm/_torch/custom_ops @NVIDIA/trt-llm-torch-graph-compiler +/tensorrt_llm/_torch/autotuner.py @NVIDIA/trt-llm-torch-graph-compiler +/tests/unittest/_torch/compilation @NVIDIA/trt-llm-torch-graph-compiler +/tests/unittest/_torch/multi_gpu/test_ar_residual_norm.py @NVIDIA/trt-llm-torch-graph-compiler +/tests/unittest/_torch/multi_gpu/test_user_buffers.py @NVIDIA/trt-llm-torch-graph-compiler +/tests/unittest/_torch/thop/test_custom_ops.py @NVIDIA/trt-llm-torch-graph-compiler +/tests/unittest/_torch/misc/test_autotuner.py @NVIDIA/trt-llm-torch-graph-compiler + ## TensorRT-LLM Pytorch - Attention /tensorrt_llm/_torch/attention_backend @NVIDIA/trt-llm-torch-attention-devs /tensorrt_llm/_torch/modules/attention.py @NVIDIA/trt-llm-torch-attention-devs + +### TensorRT-LLM Pytorch - Models - Gemma +/tensorrt_llm/_torch/models/modeling_gemma3.py @NVIDIA/trt-llm-torch-models-gemma-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_gemma3vl.py @NVIDIA/trt-llm-torch-models-gemma-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tests/unittest/_torch/modeling/test_modeling_gemma3.py @NVIDIA/trt-llm-torch-models-gemma-devs @NVIDIA/trt-llm-torch-models-devs + +### TensorRT-LLM Pytorch - Models - Mistral & Mixtral +/tensorrt_llm/_torch/models/modeling_mistral.py @NVIDIA/trt-llm-torch-models-mistral-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_mistral.py @NVIDIA/trt-llm-torch-models-mistral-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_mixtral.py @NVIDIA/trt-llm-torch-models-mistral-devs @NVIDIA/trt-llm-torch-models-devs + +### TensorRT-LLM Pytorch - Models - CLIP +/tensorrt_llm/_torch/models/modeling_clip.py @NVIDIA/trt-llm-torch-models-clip-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tests/unittest/_torch/modeling/test_modeling_clip.py @NVIDIA/trt-llm-torch-models-clip-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs + +### TensorRT-LLM Pytorch - Models - Phi +/tensorrt_llm/_torch/models/modeling_phi3.py @NVIDIA/trt-llm-torch-models-phi-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_phi4mm.py @NVIDIA/trt-llm-torch-models-phi-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tests/unittest/_torch/modeling/test_modeling_phi3.py @NVIDIA/trt-llm-torch-models-phi-devs @NVIDIA/trt-llm-torch-models-devs + +### TensorRT-LLM Pytorch - Models - Deepseek +/tensorrt_llm/_torch/models/modeling_deepseekv3.py @NVIDIA/trt-llm-torch-models-deepseek-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_deepseek.py @NVIDIA/trt-llm-torch-models-deepseek-devs @NVIDIA/trt-llm-torch-models-devs + +### TensorRT-LLM Pytorch - Models - Llama +/tensorrt_llm/_torch/models/modeling_mllama.py @NVIDIA/trt-llm-torch-models-llama-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tensorrt_llm/_torch/models/modeling_llama.py @NVIDIA/trt-llm-torch-models-llama-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_llama_min_latency.py @NVIDIA/trt-llm-torch-models-llama-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_llama.py @NVIDIA/trt-llm-torch-models-llama-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_llama_min_latency.py @NVIDIA/trt-llm-torch-models-llama-devs @NVIDIA/trt-llm-torch-models-devs + +### TensorRT-LLM Pytorch - Models - Qwen +/tensorrt_llm/_torch/models/modeling_qwen3_moe.py @NVIDIA/trt-llm-torch-models-qwen-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_qwen3.py @NVIDIA/trt-llm-torch-models-qwen-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_qwen.py @NVIDIA/trt-llm-torch-models-qwen-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_qwen_moe.py @NVIDIA/trt-llm-torch-models-qwen-devs @NVIDIA/trt-llm-torch-models-devs + +### TensorRT-LLM Pytorch - Models - VLMs +/tensorrt_llm/_torch/models/modeling_qwen2vl.py @NVIDIA/trt-llm-torch-models-vlm-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tensorrt_llm/_torch/models/modeling_vila.py @NVIDIA/trt-llm-torch-models-vlm-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tests/unittest/_torch/modeling/test_modeling_vila.py @NVIDIA/trt-llm-torch-models-vlm-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tensorrt_llm/_torch/models/modeling_pixtral.py @NVIDIA/trt-llm-torch-models-vlm-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tests/unittest/_torch/modeling/test_modeling_pixtral.py @NVIDIA/trt-llm-torch-models-vlm-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs + +### TensorRT-LLM Pytorch - Models - Nemotron +/tensorrt_llm/_torch/models/modeling_nanov2vlm.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-vlm-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tensorrt_llm/_torch/models/modeling_radio.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-vlm-devs @NVIDIA/trt-llm-torch-models-devs @NVIDIA/trt-llm-multimodal-devs +/tensorrt_llm/_torch/models/modeling_nemotron.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_nemotron_nas.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_nemotron_h.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/pyexecutor/mamba_cache_manager.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/modules/mamba @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/checkpoints/hf/nemotron_h_weight_mapper.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_nemotron.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_nemotron_h.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_nemotron_nas.py @NVIDIA/trt-llm-torch-models-nemotron-devs @NVIDIA/trt-llm-torch-models-devs + +## TensorRT-LLM Multimodal - Shared Infrastructure +/tensorrt_llm/inputs/multimodal.py @NVIDIA/trt-llm-multimodal-devs +/tensorrt_llm/inputs/registry.py @NVIDIA/trt-llm-multimodal-devs +/tensorrt_llm/inputs/utils.py @NVIDIA/trt-llm-multimodal-devs +/tensorrt_llm/_torch/models/modeling_multimodal_encoder.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_multimodal_utils.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/quantization/image_processing.py @NVIDIA/trt-llm-multimodal-devs +/cpp/tensorrt_llm/executor/multimodalInput.cpp @NVIDIA/trt-llm-multimodal-devs + +### TensorRT-LLM Multimodal - VLM Models (multimodal-primary ownership) +/tensorrt_llm/_torch/models/modeling_siglip.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_llava_next.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_hyperclovax.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_parakeet.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_nemotron_nano.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_qwen3vl.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_qwen3vl_moe.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_cosmos3.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_qwen3_5.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/modeling_hunyuan_dense.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/checkpoints/hf/qwen3vl_weight_mapper.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/checkpoints/hf/cosmos3_weight_mapper.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/checkpoints/hf/qwen3vl_moe_weight_mapper.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/checkpoints/hf/qwen3_5_weight_mapper.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tensorrt_llm/_torch/models/checkpoints/hf/llava_next_weight_mapper.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs + +### TensorRT-LLM Multimodal - Tests +/tests/unittest/_torch/multimodal/ @NVIDIA/trt-llm-multimodal-devs +/tests/unittest/_torch/modeling/test_modeling_multimodal.py @NVIDIA/trt-llm-multimodal-devs +/tests/unittest/_torch/modeling/test_modeling_siglip.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/_torch/modeling/test_modeling_parakeet.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-torch-models-devs +/tests/unittest/others/test_multimodal_registry.py @NVIDIA/trt-llm-multimodal-devs +/tests/unittest/llmapi/apps/*multimodal* @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-llmapi-devs + +## TensorRT-LLM - PEFT +/tensorrt_llm/_torch/peft @NVIDIA/trt-llm-torch-peft +/tensorrt_llm/lora_manager.py @NVIDIA/trt-llm-torch-peft +/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp @NVIDIA/trt-llm-torch-peft +/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h @NVIDIA/trt-llm-torch-peft +/cpp/tensorrt_llm/runtime/loraCache.cpp @NVIDIA/trt-llm-torch-peft +/cpp/include/tensorrt_llm/runtime/loraCache.h @NVIDIA/trt-llm-torch-peft +/cpp/tensorrt_llm/runtime/loraModule.cpp @NVIDIA/trt-llm-torch-peft +/cpp/include/tensorrt_llm/runtime/loraModule.h @NVIDIA/trt-llm-torch-peft +/cpp/tensorrt_llm/runtime/loraManager.cpp @NVIDIA/trt-llm-torch-peft +/cpp/tensorrt_llm/runtime/loraManager.h @NVIDIA/trt-llm-torch-peft +/cpp/tensorrt_llm/runtime/loraUtils.cpp @NVIDIA/trt-llm-torch-peft +/cpp/tensorrt_llm/runtime/loraUtils.h @NVIDIA/trt-llm-torch-peft + + +## TensorRT-LLM trtllm-bench Reviewers +/tensorrt_llm/bench @NVIDIA/trtllm-bench-reviewers +/tensorrt_llm/commands/bench.py @NVIDIA/trtllm-bench-reviewers +docs/source/performance/perf-benchmarking.md @NVIDIA/trtllm-bench-reviewers + +## TensorRT-LLM LLM API +/tensorrt_llm/llmapi @NVIDIA/trt-llm-llmapi-devs +/tensorrt_llm/executor @NVIDIA/trt-llm-llmapi-devs +/tensorrt_llm/serve @NVIDIA/trt-llm-llmapi-devs +/tensorrt_llm/commands @NVIDIA/trt-llm-llmapi-devs + +## TensorRT-LLM Multimodal - LLM API & Serving +/tensorrt_llm/llmapi/mm_encoder.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-llmapi-devs + +## TensorRT-LLM Multimodal - Integration Tests +/tests/integration/defs/accuracy/test_llm_api_pytorch_multimodal.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-qa-function + +## TensorRT-LLM Multimodal - Examples & Docs +/examples/llm-api/quickstart_multimodal.py @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-doc-owners +/examples/serve/*multimodal* @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-doc-owners +/docs/source/features/multi-modality.md @NVIDIA/trt-llm-multimodal-devs @NVIDIA/trt-llm-doc-owners + ## TensorRT-LLM LLM Disaggregated /examples/disaggregated @NVIDIA/trt-llm-disagg-devs @NVIDIA/trt-llm-doc-owners /examples/disaggregated/slurm/benchmark @NVIDIA/trt-llm-disagg-devs @NVIDIA/trtllm-bench-reviewers @@ -437,7 +242,6 @@ /cpp/tensorrt_llm/batch_manager/allocateKvCache.cpp @NVIDIA/trt-llm-kv-cache-manager-devs /cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp @NVIDIA/trt-llm-kv-cache-manager-devs /cpp/tests/unit_tests/batch_manager/kvCacheUtilsTest.cpp @NVIDIA/trt-llm-kv-cache-manager-devs -/tensorrt_llm/_torch/attention_backend/sparse/*/cache_manager.py @NVIDIA/trt-llm-kv-cache-manager-devs /tensorrt_llm/_torch/pyexecutor/kv_cache_manager_v2.py @NVIDIA/trt-llm-kv-cache-manager-devs /tensorrt_llm/_torch/pyexecutor/resource_manager.py @NVIDIA/trt-llm-kv-cache-manager-devs /cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.h @NVIDIA/trt-llm-kv-cache-manager-devs @@ -450,41 +254,54 @@ /tests/unittest/api_stability/ @NVIDIA/trt-llm-noncommitted-api-review-committee /tests/unittest/api_stability/references_committed/ @NVIDIA/trt-llm-committed-api-review-committee /tests/unittest/dynamo/ @NVIDIA/trt-llm-dynamo-devs -/triton_backend @NVIDIA/trt-llm-triton-backend-devs - -# ===== DISAGG BLAST-RADIUS CO-OWNS (Tier-1) ===== -# Last-match cross-cutting: adds disagg-devs to shared files that can silently break disagg e2e. -/tensorrt_llm/_torch/pyexecutor/resource_manager.py @NVIDIA/trt-llm-kv-cache-manager-devs @NVIDIA/trt-llm-disagg-devs -/tensorrt_llm/_torch/pyexecutor/py_executor.py @NVIDIA/trt-llm-runtime-devs @NVIDIA/trt-llm-disagg-devs -/tensorrt_llm/_torch/pyexecutor/model_engine.py @NVIDIA/trt-llm-runtime-devs @NVIDIA/trt-llm-disagg-devs -# ===== GOVERNANCE / LEGAL GATES ===== -/.github/CODEOWNERS @NVIDIA/trt-llm-infra-devs @NVIDIA/trt-llm-oss-compliance -/.github/tava_architecture_diagram.md @NVIDIA/trt-llm-TAVA-design-change -/3rdparty/** @NVIDIA/trt-llm-oss-compliance -/ATTRIBUTIONS-*.md @NVIDIA/trt-llm-oss-compliance +## OSS Compliance & Legal - License/Attribution Protection +## IMPORTANT: Changes to any files below may impact legal compliance, attributions, and third-party licenses. +## These files require review from the TRTLLM OSS compliance team before merging to ensure proper attribution +## and license compliance when adding, removing, or changing versions of dependencies. +### License Files /LICENSE @NVIDIA/trt-llm-oss-compliance -/constraints.txt @NVIDIA/trt-llm-oss-compliance -/cpp/CMakeLists.txt @NVIDIA/trt-llm-oss-compliance -/cpp/cmake/** @NVIDIA/trt-llm-oss-compliance -/cpp/conan.lock @NVIDIA/trt-llm-oss-compliance -/cpp/conandata.yml @NVIDIA/trt-llm-oss-compliance -/cpp/conanfile.py @NVIDIA/trt-llm-oss-compliance -/cpp/libnuma_conan.py @NVIDIA/trt-llm-oss-compliance -/docker/common/** @NVIDIA/trt-llm-infra-devs @NVIDIA/trt-llm-oss-compliance -/jenkins/license_cpp.json @NVIDIA/trt-llm-infra-devs @NVIDIA/trt-llm-oss-compliance +/ATTRIBUTIONS-*.md @NVIDIA/trt-llm-oss-compliance +/jenkins/license_cpp.json @NVIDIA/trt-llm-ci-infra-devs @NVIDIA/trt-llm-infra-devs @NVIDIA/trt-llm-oss-compliance + +### Python Dependency Management +/setup.py @NVIDIA/trt-llm-oss-compliance /pyproject.toml @NVIDIA/trt-llm-oss-compliance -/requirements-dev.txt @NVIDIA/trt-llm-oss-compliance /requirements.txt @NVIDIA/trt-llm-oss-compliance -/setup.py @NVIDIA/trt-llm-oss-compliance -/tests/unittest/api_stability/ @NVIDIA/trt-llm-noncommitted-api-review-committee -/tests/unittest/api_stability/references_committed/ @NVIDIA/trt-llm-committed-api-review-committee +/requirements-dev.txt @NVIDIA/trt-llm-oss-compliance + +### C++ Build & Dependency Management +/cpp/CMakeLists.txt @NVIDIA/trt-llm-oss-compliance +/cpp/conanfile.py @NVIDIA/trt-llm-oss-compliance +/cpp/cmake/** @NVIDIA/trt-llm-oss-compliance + +### Third-Party Dependencies +## Any changes to versions, additions, or removals of third-party libraries +/3rdparty/** @NVIDIA/trt-llm-oss-compliance + +### Vendored Third-Party Code (triton-kernels) +## This is a temporary vendored copy of triton-kernels from the Triton project (MIT License). +## Do not accept contributions to this directory - it should only be updated via scripts/vendor_triton_kernels.py +## This can be removed if and when triton-kernels is published as a separate wheel. /triton_kernels/** @NVIDIA/trt-llm-oss-compliance -### Usage telemetry / privacy review -/tensorrt_llm/usage/ @NVIDIA/trt-llm-usage-telemetry-devs -/tests/unittest/usage/ @NVIDIA/trt-llm-usage-telemetry-devs -/tensorrt_llm/usage/llm_args_golden_manifest.json @NVIDIA/trt-llm-usage-telemetry-devs @NVIDIA/trt-llm-oss-compliance @NVIDIA/trt-llm-noncommitted-api-review-committee +### Docker & Installation Scripts +## These scripts install and pin dependency versions +/docker/common/** @NVIDIA/trt-llm-setup-infra-devs @NVIDIA/trt-llm-infra-devs @NVIDIA/trt-llm-oss-compliance -# Uncomment only on release branches (e.g. release/0.19): +### TAVA Architecture Diagram +/.github/tava_architecture_diagram.md @NVIDIA/trt-llm-TAVA-design-change + +### CODEOWNERS file itself +/.github/CODEOWNERS @NVIDIA/trt-llm-gh-workflows-infra-devs @NVIDIA/trt-llm-infra-devs @NVIDIA/trt-llm-oss-compliance + +# The following rule should only be uncommented on release branches (e.g., release/0.19). +# The rule below requires that any PR to release/**/* branches must be approved by at least one member +# of the NVIDIA/trt-llm-release-branch-approval team, regardless of who else approves the PR. +# Without approval from a member of this team, PRs cannot be merged to release branches. # * @NVIDIA/trt-llm-release-branch-approval + +### Telemetry / privacy review +# Golden manifest is the privacy-review artifact; route it and the usage package to the privacy owner. +/tensorrt_llm/usage/llm_args_golden_manifest.json @NVIDIA/trt-llm-oss-compliance +/tensorrt_llm/usage/ @NVIDIA/trt-llm-oss-compliance diff --git a/.github/scripts/label_component.py b/.github/scripts/label_component.py deleted file mode 100644 index 1fa64cf7351b..000000000000 --- a/.github/scripts/label_component.py +++ /dev/null @@ -1,251 +0,0 @@ -#!/usr/bin/env python3 -"""Label pull requests by component, driven by .github/CODEOWNERS. - -For each configured (CODEOWNERS team handle -> label) mapping, this resolves the -effective owners of every file a PR changes using CODEOWNERS last-match-wins -semantics, and applies the mapped label when any changed file is owned by that -team. Labels are only added, never removed. - -Usage: - # single PR (what the GitHub Action runs) - label_component.py --pr 16143 --codeowners .github/CODEOWNERS - # a few PRs - label_component.py --pr 16143 16142 - # sweep every open PR (preview first with --dry-run) - label_component.py --all-open --dry-run - -The repo defaults to the upstream NVIDIA/TensorRT-LLM. CODEOWNERS is read from ---codeowners when given, otherwise fetched from the repo's default branch. The -token comes from GITHUB_TOKEN / GH_TOKEN, falling back to `gh auth token`. -""" - -import argparse -import os -import re -import subprocess -import sys - -import requests - -GITHUB_API_URL = "https://api.github.com" -DEFAULT_REPO = "NVIDIA/TensorRT-LLM" - -# CODEOWNERS team handle (lower-cased) -> label to apply. -# Extend this dict to cover more components. -COMPONENT_LABELS = { - "@nvidia/trt-llm-torch-visual-gen-devs": "VisualGen", -} - - -# --- CODEOWNERS parsing / matching --------------------------------------- - - -def parse_codeowners(text): - """Parse CODEOWNERS into an ordered list of (compiled_regex, owners).""" - rules = [] - for raw in text.splitlines(): - line = raw.split("#", 1)[0].strip() - if not line: - continue - parts = line.split() - pattern, owners = parts[0], [o.lower() for o in parts[1:]] - rules.append((_pattern_to_regex(pattern), owners)) - return rules - - -def _pattern_to_regex(pattern): - """Translate a CODEOWNERS (gitignore-style) pattern to a regex. - - '*' matches within a path segment, '**' crosses segments, and a directory - pattern matches everything beneath it. All CODEOWNERS patterns here are - root-anchored. - """ - body = re.escape(pattern.strip("/")) - body = body.replace(r"\*\*", ".*").replace(r"\*", "[^/]*") - return re.compile(rf"(?:{body})(?:/.*)?$") - - -def owners_for_path(path, rules): - """Effective CODEOWNERS owners for a path (last matching rule wins).""" - owners = [] - for regex, rule_owners in rules: - if regex.match(path): - owners = rule_owners - return owners - - -def labels_for_files(files, rules, component_labels=COMPONENT_LABELS): - labels = set() - for path in files: - owners = owners_for_path(path, rules) - for team, label in component_labels.items(): - if team in owners: - labels.add(label) - return labels - - -# --- GitHub access ------------------------------------------------------- - - -def resolve_token(): - for var in ("GITHUB_TOKEN", "GH_TOKEN"): - if os.environ.get(var): - return os.environ[var] - try: - return subprocess.check_output(["gh", "auth", "token"], text=True).strip() - except (OSError, subprocess.CalledProcessError): - raise SystemExit("No token found: set GITHUB_TOKEN / GH_TOKEN, or run `gh auth login`.") - - -def make_session(token): - session = requests.Session() - session.headers.update( - { - "Accept": "application/vnd.github+json", - "X-GitHub-Api-Version": "2022-11-28", - "User-Agent": "trtllm-label-component/1.0", - "Authorization": f"token {token}", - } - ) - return session - - -def load_codeowners(session, repo, path): - if path: - with open(path, encoding="utf-8") as fh: - return parse_codeowners(fh.read()) - r = session.get( - f"{GITHUB_API_URL}/repos/{repo}/contents/.github/CODEOWNERS", - headers={"Accept": "application/vnd.github.raw"}, - timeout=30, - ) - r.raise_for_status() - return parse_codeowners(r.text) - - -def iter_open_prs(session, repo, limit=None): - """Yield (number, existing_labels) for open PRs; labels come free here.""" - page, seen = 1, 0 - while True: - r = session.get( - f"{GITHUB_API_URL}/repos/{repo}/pulls", - params={ - "state": "open", - "per_page": 100, - "page": page, - "sort": "created", - "direction": "desc", - }, - timeout=30, - ) - r.raise_for_status() - batch = r.json() - if not batch: - return - for pr in batch: - yield pr["number"], {lbl["name"] for lbl in pr.get("labels", [])} - seen += 1 - if limit and seen >= limit: - return - page += 1 - - -def get_changed_files(session, repo, pr_number): - files, page = [], 1 - while True: - r = session.get( - f"{GITHUB_API_URL}/repos/{repo}/pulls/{pr_number}/files", - params={"per_page": 100, "page": page}, - timeout=30, - ) - r.raise_for_status() - batch = r.json() - if not batch: - break - files.extend(f["filename"] for f in batch) - page += 1 - return files - - -def get_pr_labels(session, repo, pr_number): - r = session.get(f"{GITHUB_API_URL}/repos/{repo}/issues/{pr_number}", timeout=30) - r.raise_for_status() - return {lbl["name"] for lbl in r.json().get("labels", [])} - - -def add_labels(session, repo, pr_number, labels): - r = session.post( - f"{GITHUB_API_URL}/repos/{repo}/issues/{pr_number}/labels", - json={"labels": labels}, - timeout=30, - ) - r.raise_for_status() - - -def process_pr(session, repo, pr_number, rules, existing_labels, dry_run): - """Return the labels added (or that would be added). Empty if none.""" - files = get_changed_files(session, repo, pr_number) - wanted = labels_for_files(files, rules) - to_add = sorted(wanted - existing_labels) - if not to_add: - return [] - if dry_run: - print(f"PR #{pr_number}: would add {to_add}") - else: - add_labels(session, repo, pr_number, to_add) - print(f"PR #{pr_number}: added {to_add}") - return to_add - - -# --- CLI ----------------------------------------------------------------- - - -def parse_args(argv): - ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - ap.add_argument("--repo", default=DEFAULT_REPO, help=f"owner/name (default: {DEFAULT_REPO})") - target = ap.add_mutually_exclusive_group(required=True) - target.add_argument("--pr", type=int, nargs="+", metavar="N", help="label these PR number(s)") - target.add_argument("--all-open", action="store_true", help="label every open PR in the repo") - ap.add_argument( - "--codeowners", - metavar="PATH", - help="local CODEOWNERS file; if omitted, fetched from the repo's default branch", - ) - ap.add_argument( - "--limit", type=int, metavar="N", help="with --all-open, cap the number of PRs scanned" - ) - ap.add_argument( - "--dry-run", action="store_true", help="report what would change without adding labels" - ) - return ap.parse_args(argv) - - -def main(argv=None): - args = parse_args(argv) - session = make_session(resolve_token()) - rules = load_codeowners(session, args.repo, args.codeowners) - - if args.pr: - targets = [(n, get_pr_labels(session, args.repo, n)) for n in args.pr] - else: - targets = iter_open_prs(session, args.repo, args.limit) - - scanned, labeled, failed = 0, 0, 0 - for number, existing in targets: - scanned += 1 - # Isolate per-PR failures so one bad PR (transient 5xx, missing label) - # does not abort an --all-open sweep. - try: - if process_pr(session, args.repo, number, rules, existing, args.dry_run): - labeled += 1 - except requests.HTTPError as exc: - failed += 1 - print(f"PR #{number}: failed ({exc})", file=sys.stderr) - - verb = "would be labeled" if args.dry_run else "labeled" - print(f"Scanned {scanned} PR(s); {labeled} {verb}; {failed} failed.") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/.github/workflows/blossom-ci.yml b/.github/workflows/blossom-ci.yml index f5a60658d8dd..3682a308b805 100644 --- a/.github/workflows/blossom-ci.yml +++ b/.github/workflows/blossom-ci.yml @@ -57,7 +57,6 @@ jobs: "anish-shanbhag", "arekay", "arysef", - "asfiyab-nvidia", "aswinvisva", "athena-nv", "atrifex", @@ -76,8 +75,6 @@ jobs: "bo-nv", "bobboli", "Boreas618", - "BowenFu", - "BoyueZ", "brb-nv", "brnguyen2", "byshiue", @@ -137,11 +134,9 @@ jobs: "govind-ramnarayan", "greg-kwasniewski1", "guangyunh-nv", - "GuanhuaWang2001", "guqiqi", "h-guo18", "HandongLi-01", - "haow-nv", "hchings", "hello-11", "heyuhhh", @@ -155,7 +150,6 @@ jobs: "indrajit96", "inocsin", "ISEEKYAN", - "ishovkun", "ixlmar", "IzzyPutterman", "Jackch-NV", @@ -174,7 +168,6 @@ jobs: "Jie-Fang", "jiefangz-nv", "jieli-matrix", - "jingyu-ml", "JintaoPengCS", "jinyangyuan-nvidia", "jinzh-nvidia", @@ -187,7 +180,6 @@ jobs: "JunyiXu-nv", "JyChang012", "kaiyux", - "Kambili", "kanghui0204", "karljang", "karthikvetrivel", @@ -204,7 +196,6 @@ jobs: "lancelly", "LarryXFly", "latency1024", - "leo0519", "leslie-fang25", "lfr-0531", "liji-nv", @@ -219,7 +210,6 @@ jobs: "lkomali", "longcheng-nv", "longlee0622", - "lori-ren", "lowsfer", "lucaslie", "lucifer1004", @@ -229,7 +219,6 @@ jobs: "MatthiasKohl", "mayani-nv", "meenchen", - "mgluhovskoi", "mikeiovine", "milesial", "MinaHuai", @@ -276,7 +265,6 @@ jobs: "pengbowang-nv", "PerkzZheng", "poweiw", - "pranav-nvidia", "qiangxu1996", "qiaoxj07", "QiJune", @@ -310,7 +298,6 @@ jobs: "shuyixiong", "shyeh25", "SimengLiu-nv", - "siyidNV", "sklevtsov-nvidia", "StanleySun639", "stnie", @@ -357,7 +344,6 @@ jobs: "viraatc", "Wanli-Jiang", "WeiHaocheng", - "weiminwang-nv", "weireweire", "wenmingw", "wili-65535", @@ -367,7 +353,6 @@ jobs: "wyw1267", "xavier-nvidia", "xd-nv", - "xguannv", "xiaoweiw-nv", "xinhe-nv", "xmchen1987", diff --git a/.github/workflows/bot-command.yml b/.github/workflows/bot-command.yml index 30b5de652f55..0a112dbf5435 100644 --- a/.github/workflows/bot-command.yml +++ b/.github/workflows/bot-command.yml @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); @@ -52,14 +52,14 @@ jobs: "`--disable-reuse-test ` *(OPTIONAL)* : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.\n\n" + "`--disable-fail-fast ` *(OPTIONAL)* : Disable fail fast on build/tests/infra failures.\n\n" + "`--skip-test ` *(OPTIONAL)* : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does **NOT** update GitHub check status.\n\n" + - "`--stage-list \"A10-PyTorch-1, xxx\"` *(OPTIONAL)* : Only run the specified test stages. Supports wildcard `*` for pattern matching (e.g., `\"*PerfSanity*\"` matches all stages containing PerfSanity). Examples: \"A10-PyTorch-1, xxx\", \"*PerfSanity*\". The patterns `\"*\"`, `\"*Post-Merge*\"`, and `\"*PerfSanity*\"`, including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the `ci: post-merge approved` PR label. Note: Does **NOT** update GitHub check status.\n\n" + + "`--stage-list \"A10-PyTorch-1, xxx\"` *(OPTIONAL)* : Only run the specified test stages. Supports wildcard `*` for pattern matching (e.g., `\"*PerfSanity*\"` matches all stages containing PerfSanity). Examples: \"A10-PyTorch-1, xxx\", \"*PerfSanity*\". Note: Does **NOT** update GitHub check status.\n\n" + "`--gpu-type \"A30, H100_PCIe\"` *(OPTIONAL)* : Only run the test stages on the specified GPU types. Examples: \"A30, H100_PCIe\". Note: Does **NOT** update GitHub check status.\n\n" + "`--test-backend \"pytorch, cpp\"` *(OPTIONAL)* : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: \"pytorch, cpp\" (does not run test stages with tensorrt or triton backend). Note: Does **NOT** update GitHub pipeline status.\n\n" + "`--only-multi-gpu-test ` *(OPTIONAL)* : Only run the multi-GPU tests. Note: Does **NOT** update GitHub check status.\n\n" + "`--disable-multi-gpu-test ` *(OPTIONAL)* : Disable the multi-GPU tests. Note: Does **NOT** update GitHub check status.\n\n" + "`--add-multi-gpu-test ` *(OPTIONAL)* : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.\n\n" + - "`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline. Requires the `ci: post-merge approved` PR label applied by an active member of `NVIDIA/trt-llm-ci-approvers`. The approval label remains in place when new commits are pushed.\n\n" + - "`--extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard `*` for pattern matching. Examples: --extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\", --extra-stage \"*Post-Merge*\". The patterns `\"*\"`, `\"*Post-Merge*\"`, and `\"*PerfSanity*\"`, including equivalent escaped or repeated-star forms and their use in comma-separated lists, require the `ci: post-merge approved` PR label.\n\n" + + "`--post-merge ` *(OPTIONAL)* : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.\n\n" + + "`--extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\"` *(OPTIONAL)* : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard `*` for pattern matching. Examples: --extra-stage \"H100_PCIe-TensorRT-Post-Merge-1, xxx\", --extra-stage \"*Post-Merge*\".\n\n" + "`--detailed-log ` *(OPTIONAL)* : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.\n\n" + "`--debug ` *(OPTIONAL)* : **Experimental feature**. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the `stage-list` parameter to access the appropriate container environment. Note: Does **NOT** update GitHub check status.\n\n" + "`--high-priority ` *(OPTIONAL)* : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.\n\n" + diff --git a/.github/workflows/label_component_pr.yml b/.github/workflows/label_component_pr.yml deleted file mode 100644 index 5d584eb237cd..000000000000 --- a/.github/workflows/label_component_pr.yml +++ /dev/null @@ -1,42 +0,0 @@ -name: Label Component for PR - -on: - pull_request_target: - types: [opened, reopened] - -permissions: - contents: read - pull-requests: write - -jobs: - label-component: - runs-on: ubuntu-latest - if: github.repository == 'NVIDIA/TensorRT-LLM' - # This workflow is advisory: it must never turn a PR check red. Every step - # is continue-on-error, so the check is always green even if setup or - # labeling fails. - steps: - - name: Checkout base repository - continue-on-error: true - uses: actions/checkout@v6 - with: - persist-credentials: false - - - name: Set up Python - continue-on-error: true - uses: actions/setup-python@v6 - with: - python-version: '3.x' - - - name: Install dependencies - continue-on-error: true - run: pip install requests - - - name: Label PR by component - continue-on-error: true - env: - GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - run: >- - python .github/scripts/label_component.py - --pr ${{ github.event.pull_request.number }} - --codeowners .github/CODEOWNERS diff --git a/.github/workflows/lfs-sync.yml b/.github/workflows/lfs-sync.yml index 3143afedbd90..bccdedc858e1 100644 --- a/.github/workflows/lfs-sync.yml +++ b/.github/workflows/lfs-sync.yml @@ -44,12 +44,9 @@ jobs: steps: - name: Checkout merge commit without LFS smudge - uses: actions/checkout@v6 + uses: actions/checkout@v4 with: ref: ${{ github.event.pull_request.merge_commit_sha }} - # This job only handles merged PRs and does not execute code from the - # checked-out tree, so explicitly allow the required merge checkout. - allow-unsafe-pr-checkout: true lfs: false fetch-depth: 2 token: ${{ secrets.GITHUB_TOKEN }} @@ -203,7 +200,7 @@ jobs: - name: Comment on PR — already in storage if: steps.detect-lfs.outputs.lfs_status == 'present' - uses: actions/github-script@v8 + uses: actions/github-script@v7 with: script: | const files = require('fs').readFileSync('/tmp/lfs_files.txt', 'utf8').trim().split('\n'); @@ -224,7 +221,7 @@ jobs: - name: Comment on PR — sync succeeded if: steps.detect-lfs.outputs.lfs_status == 'pending' && steps.verify.outcome == 'success' - uses: actions/github-script@v8 + uses: actions/github-script@v7 with: script: | const files = require('fs').readFileSync('/tmp/lfs_files.txt', 'utf8').trim().split('\n'); @@ -245,7 +242,7 @@ jobs: - name: Comment on PR — sync failed if: always() && steps.detect-lfs.outputs.lfs_status == 'pending' && (steps.fetch-lfs.outcome != 'success' || steps.push-lfs.outcome != 'success' || steps.verify.outcome != 'success') - uses: actions/github-script@v8 + uses: actions/github-script@v7 with: script: | const forkRepo = context.payload.pull_request.head.repo.full_name; diff --git a/.github/workflows/llm-api-compatibility.yml b/.github/workflows/llm-api-compatibility.yml index e93fea709ef9..5c727b659e7c 100644 --- a/.github/workflows/llm-api-compatibility.yml +++ b/.github/workflows/llm-api-compatibility.yml @@ -33,8 +33,6 @@ jobs: const referencePrefixes = [ 'tests/unittest/api_stability/references/', 'tests/unittest/api_stability/references_committed/', - 'tensorrt_llm/llmapi/llm_args.py', - 'tensorrt_llm/usage/llm_args_golden_manifest.json', ]; // Keep these names in sync with the PR template and API-change guide. const compatibleApiLabel = 'api-compatible'; diff --git a/.github/workflows/post-merge-approval.yml b/.github/workflows/post-merge-approval.yml deleted file mode 100644 index 66b3490843dd..000000000000 --- a/.github/workflows/post-merge-approval.yml +++ /dev/null @@ -1,222 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -name: Guard Post-Merge Approval Label - -on: - # Intentional: this runs static default-branch API logic only. It never checks - # out or executes PR code; it uses GitHub APIs to validate membership and - # manage this PR's approval label/comment. - pull_request_target: - types: [labeled] - -permissions: - contents: read - pull-requests: write - -jobs: - guard-post-merge-approval: - concurrency: - group: post-merge-approval-${{ github.event.pull_request.number }} - cancel-in-progress: false - # Keep a started job eligible to reach fail-closed cleanup after cancellation. - if: >- - always() && - github.repository == 'NVIDIA/TensorRT-LLM' && - github.event.action == 'labeled' && - github.event.label.name == 'ci: post-merge approved' - runs-on: ubuntu-latest - timeout-minutes: 5 - steps: - - name: Validate post-merge approver - id: validate - if: github.event.action == 'labeled' - uses: actions/github-script@v8 - with: - github-token: ${{ secrets.TRTLLM_AGENT_SHARED_TOKEN }} - result-encoding: string - script: | - const approvalLabel = 'ci: post-merge approved'; - const owner = context.repo.owner; - const repo = context.repo.repo; - const issueNumber = context.payload.pull_request.number; - let actor = context.payload.sender?.login || context.actor; - let labelEventId = ''; - let timelineVerified = false; - - try { - const events = await github.paginate( - github.rest.issues.listEventsForTimeline, - { owner, repo, issue_number: issueNumber, per_page: 100 } - ); - const latestApprovalEvent = events - .filter( - (event) => - event.event === 'labeled' && - event.label?.name?.toLowerCase() === approvalLabel.toLowerCase() - ) - .at(-1); - if (latestApprovalEvent) { - actor = latestApprovalEvent.actor?.login || actor; - labelEventId = String(latestApprovalEvent.id); - timelineVerified = true; - } else { - core.warning('Could not identify the latest post-merge approval event.'); - } - } catch (error) { - core.warning( - 'Could not read the latest post-merge approval event: ' + error.message - ); - } - - core.setOutput('validated_actor', actor); - core.setOutput('label_event_id', labelEventId); - if (!timelineVerified) { - return 'false'; - } - try { - const response = await github.request( - 'GET /orgs/{org}/teams/{team_slug}/memberships/{username}', - { - org: 'NVIDIA', - team_slug: 'trt-llm-ci-approvers', - username: actor, - } - ); - const authorized = response.data.state === 'active'; - console.log( - actor + ' active membership in NVIDIA/trt-llm-ci-approvers: ' + authorized - ); - return authorized ? 'true' : 'false'; - } catch (error) { - if (error.status === 404) { - console.log( - actor + ' is not an active member of NVIDIA/trt-llm-ci-approvers.' - ); - } else { - core.warning( - 'Could not verify post-merge approver ' + actor + ': ' + error.message - ); - } - return 'false'; - } - - - name: Clear unauthorized post-merge approval - # Let a started job attempt fail-closed cleanup after normal cancellation. - if: always() && steps.validate.outputs.result != 'true' - uses: actions/github-script@v8 - env: - VALIDATED_ACTOR: ${{ steps.validate.outputs.validated_actor }} - VALIDATED_LABEL_EVENT_ID: ${{ steps.validate.outputs.label_event_id }} - with: - github-token: ${{ secrets.GITHUB_TOKEN }} - script: | - const approvalLabel = 'ci: post-merge approved'; - const actor = - process.env.VALIDATED_ACTOR || - context.payload.sender?.login || - context.actor; - const validatedEventId = process.env.VALIDATED_LABEL_EVENT_ID || ''; - const owner = context.repo.owner; - const repo = context.repo.repo; - const issueNumber = context.payload.pull_request.number; - - try { - const pullRequest = await github.rest.pulls.get({ - owner, - repo, - pull_number: issueNumber, - }); - const labelIsPresent = pullRequest.data.labels.some( - (label) => - label.name?.toLowerCase() === approvalLabel.toLowerCase() - ); - if (!labelIsPresent) { - console.log('Post-merge approval label is already absent; no cleanup needed.'); - return; - } - } catch (error) { - core.warning( - 'Could not read the current post-merge approval label state; continuing validation: ' + - error.message - ); - } - - try { - const events = await github.paginate( - github.rest.issues.listEventsForTimeline, - { owner, repo, issue_number: issueNumber, per_page: 100 } - ); - const latestApprovalEvent = events - .filter( - (event) => - event.event === 'labeled' && - event.label?.name?.toLowerCase() === approvalLabel.toLowerCase() - ) - .at(-1); - const latestEventId = latestApprovalEvent - ? String(latestApprovalEvent.id) - : ''; - const latestActor = latestApprovalEvent?.actor?.login || ''; - if (!latestApprovalEvent) { - core.warning( - 'Could not identify the latest post-merge approval event; removing the label to fail closed.' - ); - } else if (!latestActor) { - core.warning( - 'Could not identify the latest post-merge approval actor; removing the label to fail closed.' - ); - } else if (!validatedEventId) { - core.warning( - 'The validation run did not bind an approval event; removing the label to fail closed.' - ); - } else if ( - latestEventId !== validatedEventId || - latestActor !== actor - ) { - console.log( - 'A newer post-merge approval event was found; leaving it for its own validation run.' - ); - return; - } - } catch (error) { - core.warning( - 'Could not re-check the latest post-merge approval event; removing the label to fail closed: ' + - error.message - ); - } - - try { - await github.rest.issues.removeLabel({ - owner, - repo, - issue_number: issueNumber, - name: approvalLabel, - }); - } catch (error) { - if (error.status !== 404) { - throw error; - } - } - - await github.rest.issues.createComment({ - owner, - repo, - issue_number: issueNumber, - body: - 'Removed the "' + approvalLabel + '" label because @' + actor + - ' could not be verified as an active member of ' + - 'NVIDIA/trt-llm-ci-approvers. Ask a member of that team to apply it.', - }); diff --git a/.gitignore b/.gitignore index 9a9332335ca5..47e39a1b2d71 100644 --- a/.gitignore +++ b/.gitignore @@ -57,6 +57,7 @@ tensorrt_llm/flash_mla/ tensorrt_llm/flash_mla_cpp_tllm.*.so tensorrt_llm/flash_mla_cpp_tllm.pyi tensorrt_llm/runtime/kv_cache_manager_v2/**/*.so +!tensorrt_llm/_torch/visual_gen/cute_dsl_kernels/blackwell/attention/cubins/**/*.so **/*__mypyc*.so tensorrt_llm/scripts *docs/cpp_docs* diff --git a/.gitmodules b/.gitmodules index 627760b34da2..e69de29bb2d1 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +0,0 @@ -[submodule "3rdparty/MSA"] - path = 3rdparty/MSA - url = https://gitlab.com/nvidia/tensorrt-llm/oss-components/msa.git diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 37c51fbbb377..fcd86c8884da 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -9,6 +9,14 @@ common-files: &common_files | .devcontainer/make_env.py | .github/scripts/label_community_user.py | .github/scripts/pr_checklist_check.py | + benchmarks/cpp/__init__.py | + benchmarks/cpp/prepare_dataset.py | + benchmarks/cpp/utils/__init__.py | + benchmarks/cpp/utils/convert_nemo_dataset.py | + benchmarks/cpp/utils/generate_rand_loras.py | + benchmarks/cpp/utils/prepare_real_data.py | + benchmarks/cpp/utils/prepare_synthetic_data.py | + benchmarks/cpp/utils/utils.py | cpp/conanfile.py | cpp/kernels/fmha_v2/conftest.py | cpp/kernels/fmha_v2/fmha_test.py | @@ -33,17 +41,58 @@ common-files: &common_files | cpp/tensorrt_llm/deep_ep/strip_nvshmem_helper.py | cpp/tensorrt_llm/kernels/cutlass_kernels/python/generate_kernels.py | cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/copy_cu.py | + cpp/tests/resources/scripts/build_chatglm_engines.py | + cpp/tests/resources/scripts/build_eagle_engines.py | + cpp/tests/resources/scripts/build_enc_dec_engines.py | + cpp/tests/resources/scripts/build_engines_utils.py | + cpp/tests/resources/scripts/build_gpt_engines.py | + cpp/tests/resources/scripts/build_gptj_engines.py | + cpp/tests/resources/scripts/build_llama_engines.py | + cpp/tests/resources/scripts/build_mamba_engines.py | + cpp/tests/resources/scripts/build_medusa_engines.py | + cpp/tests/resources/scripts/build_recurrentgemma_engines.py | + cpp/tests/resources/scripts/build_redrafter_engines.py | + cpp/tests/resources/scripts/generate_expected_chatglm_output.py | + cpp/tests/resources/scripts/generate_expected_eagle_output.py | + cpp/tests/resources/scripts/generate_expected_enc_dec_output.py | + cpp/tests/resources/scripts/generate_expected_gpt_output.py | + cpp/tests/resources/scripts/generate_expected_gptj_output.py | + cpp/tests/resources/scripts/generate_expected_llama_output.py | + cpp/tests/resources/scripts/generate_expected_mamba_output.py | + cpp/tests/resources/scripts/generate_expected_medusa_output.py | + cpp/tests/resources/scripts/generate_expected_recurrentgemma_output.py | + cpp/tests/resources/scripts/generate_expected_redrafter_output.py | + cpp/tests/resources/scripts/generate_hf_gpt_output.py | cpp/tests/resources/scripts/generate_test_lora_weights.py | + cpp/tests/resources/scripts/io_converter.py | docs/source/conf.py | docs/source/helper.py | examples/apps/chat.py | examples/apps/fastapi_server.py | + examples/bindings/executor/example_advanced.py | + examples/bindings/executor/example_basic.py | + examples/bindings/executor/example_debug.py | + examples/bindings/executor/example_logits_processor.py | examples/disaggregated/clients/disagg_client.py | examples/disaggregated/slurm/benchmark/submit.py | + examples/dora/normalize_weights.py | + examples/eagle/convert_checkpoint.py | + examples/eval_long_context.py | + examples/generate_checkpoint_config.py | + examples/generate_xgrammar_tokenizer_info.py | + examples/hf_lora_convert.py | examples/infinitebench/args.py | examples/infinitebench/compute_scores.py | examples/infinitebench/construct_synthetic_dataset.py | examples/infinitebench/eval_utils.py | + examples/llm-api/_tensorrt_engine/llm_eagle2_decoding.py | + examples/llm-api/_tensorrt_engine/llm_eagle_decoding.py | + examples/llm-api/_tensorrt_engine/llm_inference_customize.py | + examples/llm-api/_tensorrt_engine/llm_inference_kv_events.py | + examples/llm-api/_tensorrt_engine/llm_lookahead_decoding.py | + examples/llm-api/_tensorrt_engine/llm_medusa_decoding.py | + examples/llm-api/_tensorrt_engine/llm_quantization.py | + examples/llm-api/_tensorrt_engine/quickstart_example.py | examples/llm-api/llm_guided_decoding.py | examples/llm-api/llm_inference.py | examples/llm-api/llm_inference_async.py | @@ -63,17 +112,122 @@ common-files: &common_files | examples/llm-api/quickstart_example.py | examples/llm-api/quickstart_multimodal.py | examples/llm-api/star_attention.py | + examples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.py | examples/longbench/eval_longbench_v1.py | + examples/medusa/convert_checkpoint.py | + examples/mmlu.py | + examples/models/contrib/baichuan/convert_checkpoint.py | + examples/models/contrib/bloom/convert_checkpoint.py | + examples/models/contrib/chatglm-6b/tokenization_chatglm.py | + examples/models/contrib/chatglm2-6b/tokenization_chatglm.py | + examples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.py | + examples/models/contrib/cogvlm/convert_checkpoint.py | + examples/models/contrib/dbrx/convert_checkpoint.py | + examples/models/contrib/deepseek_v1/__init__.py | + examples/models/contrib/deepseek_v1/convert_checkpoint.py | + examples/models/contrib/deepseek_v2/convert_checkpoint.py | + examples/models/contrib/dit/convert_checkpoint.py | + examples/models/contrib/dit/diffusion.py | + examples/models/contrib/dit/sample.py | + examples/models/contrib/dit/utils_modelopt.py | + examples/models/contrib/dit/vae_decoder_trt.py | + examples/models/contrib/falcon/convert_checkpoint.py | + examples/models/contrib/gptj/convert_checkpoint.py | + examples/models/contrib/gptneox/convert_checkpoint.py | + examples/models/contrib/grok/convert_checkpoint.py | + examples/models/contrib/mmdit/convert_checkpoint.py | + examples/models/contrib/mmdit/sample.py | + examples/models/contrib/mpt/convert_checkpoint.py | + examples/models/contrib/opt/convert_checkpoint.py | + examples/models/contrib/sdxl/build_sdxl_unet.py | + examples/models/contrib/sdxl/pipeline_stable_diffusion_xl.py | + examples/models/contrib/sdxl/run_sdxl.py | + examples/models/contrib/stdit/aspect.py | + examples/models/contrib/stdit/convert_checkpoint.py | + examples/models/contrib/stdit/pipeline_tllm.py | + examples/models/contrib/stdit/sample.py | + examples/models/contrib/stdit/scheduler.py | + examples/models/contrib/stdit/text_encoder.py | + examples/models/contrib/stdit/utils.py | + examples/models/contrib/stdit/vae.py | + examples/models/contrib/stdit/video_transforms.py | + examples/models/core/bert/__init__.py | + examples/models/core/bert/convert_checkpoint.py | + examples/models/core/bert/run.py | + examples/models/core/bert/utils.py | + examples/models/core/commandr/convert_checkpoint.py | + examples/models/core/enc_dec/__init__.py | + examples/models/core/enc_dec/convert_checkpoint.py | + examples/models/core/enc_dec/helper.py | + examples/models/core/enc_dec/run.py | + examples/models/core/gemma/convert_checkpoint.py | + examples/models/core/glm-4-9b/convert_checkpoint.py | + examples/models/core/glm-4-9b/tokenization_chatglm.py | + examples/models/core/gpt/convert_checkpoint.py | + examples/models/core/gpt/merge_ptuning_tables.py | + examples/models/core/gpt/nemo_lora_convert.py | + examples/models/core/gpt/nemo_prompt_convert.py | + examples/models/core/gpt/run_hf.py | examples/models/core/gpt_oss/openai_chat_client_function_calling.py | + examples/models/core/internlm2/convert_checkpoint.py | examples/models/core/kimi_k2/kimi_k2_tool_calling_example.py | + examples/models/core/llama/convert_checkpoint.py | + examples/models/core/llama/summarize_long.py | + examples/models/core/mamba/convert_checkpoint.py | + examples/models/core/mllama/convert_checkpoint.py | + examples/models/core/multimodal/__init__.py | + examples/models/core/multimodal/build_multimodal_engine.py | + examples/models/core/multimodal/eval.py | + examples/models/core/multimodal/run.py | + examples/models/core/multimodal/utils.py | + examples/models/core/nemotron_nas/calibration_utils.py | + examples/models/core/nemotron_nas/convert_checkpoint.py | + examples/models/core/phi/convert_checkpoint.py | + examples/models/core/qwen/convert_checkpoint.py | + examples/models/core/qwen2audio/run.py | + examples/models/core/qwen2audio/run_chat.py | + examples/models/core/qwen2audio/utils.py | + examples/models/core/qwenvl/run.py | + examples/models/core/qwenvl/run_chat.py | + examples/models/core/qwenvl/show_pic.py | + examples/models/core/qwenvl/vit_onnx_trt.py | + examples/models/core/recurrentgemma/convert_checkpoint.py | + examples/models/core/vit/convert_checkpoint.py | + examples/models/core/whisper/convert_checkpoint.py | + examples/models/core/whisper/distil_whisper/convert_from_distil_whisper.py | + examples/models/core/whisper/run.py | + examples/models/core/whisper/tokenizer.py | + examples/models/core/whisper/whisper_utils.py | + examples/ngram/run_dtm_ngram.py | + examples/openai_triton/manual_plugin/build.py | + examples/openai_triton/manual_plugin/fmha_triton.py | + examples/openai_triton/manual_plugin/plugin.py | + examples/openai_triton/manual_plugin/run.py | + examples/openai_triton/plugin_autogen/build_engine.py | + examples/openai_triton/plugin_autogen/kernel_config.py | + examples/openai_triton/plugin_autogen/run_engine.py | + examples/python_plugin/build_lookup.py | + examples/python_plugin/plugin_lib/__init__.py | + examples/python_plugin/plugin_lib/lookup_kernel.py | + examples/python_plugin/plugin_lib/lookup_plugin.py | + examples/python_plugin/run_lookup.py | + examples/quantization/quantize.py | examples/quantization/quantize_mixed_precision_moe.py | examples/ray_orchestrator/llm_inference_async_ray.py | examples/ray_orchestrator/llm_inference_distributed_ray.py | + examples/redrafter/convert_checkpoint.py | + examples/run.py | examples/scaffolding/contrib/AsyncGeneration/stream_generation_controller.py | examples/scaffolding/contrib/DeepConf/run_generation.py | examples/scaffolding/contrib/Dynasor/scaffolding_dynasor_run.py | examples/scaffolding/contrib/TreeInference/run_mcts_example.py | examples/scaffolding/contrib/TreeInference/run_tot_example.py | + examples/scaffolding/contrib/mcp/e2b/e2bserver.py | + examples/scaffolding/contrib/mcp/e2b/main.py | + examples/scaffolding/contrib/mcp/mcptest.py | + examples/scaffolding/contrib/mcp/weather/weather.py | + examples/scaffolding/contrib/mcp/websearch/main.py | + examples/scaffolding/contrib/mcp/websearch/websearch.py | examples/scaffolding/run_basic_generation.py | examples/scaffolding/run_best_of_n_with_reward.py | examples/scaffolding/run_majority_vote_aime24.py | @@ -83,6 +237,8 @@ common-files: &common_files | examples/serve/openai_completion_client.py | examples/serve/openai_completion_client_for_lora.py | examples/serve/openai_completion_client_json_schema.py | + examples/summarize.py | + examples/utils.py | examples/wide_ep/ep_load_balancer/generate_eplb_config.py | examples/wide_ep/ep_load_balancer/report_load_statistics.py | examples/wide_ep/ep_load_balancer/utils.py | @@ -90,10 +246,12 @@ common-files: &common_files | jenkins/scripts/mergeWaiveList.py | jenkins/scripts/open_search_db.py | jenkins/scripts/test_rerun.py | + scripts/build_cpp_examples.py | scripts/build_wheel.py | scripts/check_test_list.py | scripts/dco_check.py | scripts/format_test_list.py | + scripts/generate_duration.py | scripts/generate_lock_file.py | scripts/get_wheel_from_package.py | scripts/git_replace.py | @@ -104,6 +262,7 @@ common-files: &common_files | setup.py | tensorrt_llm/__init__.py | tensorrt_llm/_ray_utils.py | + tensorrt_llm/_tensorrt_engine/__init__.py | tensorrt_llm/_torch/__init__.py | tensorrt_llm/_torch/attention_backend/__init__.py | tensorrt_llm/_torch/attention_backend/flashinfer.py | @@ -307,6 +466,7 @@ common-files: &common_files | tensorrt_llm/_torch/pyexecutor/guided_decoder.py | tensorrt_llm/_torch/pyexecutor/handle_additional_outputs.py | tensorrt_llm/_torch/pyexecutor/handle_logits.py | + tensorrt_llm/_torch/pyexecutor/kv_cache_connector.py | tensorrt_llm/_torch/pyexecutor/kv_cache_transceiver.py | tensorrt_llm/_torch/pyexecutor/layerwise_nvtx_marker.py | tensorrt_llm/_torch/pyexecutor/llm_request.py | @@ -344,6 +504,11 @@ common-files: &common_files | tensorrt_llm/bench/benchmark/utils/asynchronous.py | tensorrt_llm/bench/benchmark/utils/general.py | tensorrt_llm/bench/benchmark/utils/processes.py | + tensorrt_llm/bench/build/__init__.py | + tensorrt_llm/bench/build/build.py | + tensorrt_llm/bench/build/dataclasses.py | + tensorrt_llm/bench/build/tuning.py | + tensorrt_llm/bench/build/utils.py | tensorrt_llm/bench/dataclasses/__init__.py | tensorrt_llm/bench/dataclasses/configuration.py | tensorrt_llm/bench/dataclasses/engine.py | @@ -353,9 +518,13 @@ common-files: &common_files | tensorrt_llm/bench/dataclasses/statistics.py | tensorrt_llm/bench/utils/__init__.py | tensorrt_llm/bench/utils/data.py | + tensorrt_llm/builder.py | tensorrt_llm/commands/__init__.py | tensorrt_llm/commands/bench.py | + tensorrt_llm/commands/build.py | tensorrt_llm/commands/eval.py | + tensorrt_llm/commands/prune.py | + tensorrt_llm/commands/refit.py | tensorrt_llm/commands/serve.py | tensorrt_llm/evaluate/__init__.py | tensorrt_llm/evaluate/cnn_dailymail.py | @@ -391,7 +560,23 @@ common-files: &common_files | tensorrt_llm/inputs/multimodal.py | tensorrt_llm/inputs/registry.py | tensorrt_llm/inputs/utils.py | + tensorrt_llm/layers/__init__.py | + tensorrt_llm/layers/activation.py | + tensorrt_llm/layers/attention.py | + tensorrt_llm/layers/cast.py | + tensorrt_llm/layers/conv.py | + tensorrt_llm/layers/embedding.py | + tensorrt_llm/layers/language_adapter.py | + tensorrt_llm/layers/linear.py | + tensorrt_llm/layers/lora.py | + tensorrt_llm/layers/mlp.py | + tensorrt_llm/layers/moe.py | + tensorrt_llm/layers/normalization.py | + tensorrt_llm/layers/pooling.py | + tensorrt_llm/layers/recurrent.py | + tensorrt_llm/layers/ssm.py | tensorrt_llm/llmapi/__init__.py | + tensorrt_llm/llmapi/build_cache.py | tensorrt_llm/llmapi/disagg_utils.py | tensorrt_llm/llmapi/kv_cache_type.py | tensorrt_llm/llmapi/llm.py | @@ -414,17 +599,179 @@ common-files: &common_files | tensorrt_llm/metrics/enums.py | tensorrt_llm/models/__init__.py | tensorrt_llm/models/automodel.py | + tensorrt_llm/models/baichuan/__init__.py | + tensorrt_llm/models/baichuan/config.py | + tensorrt_llm/models/baichuan/convert.py | + tensorrt_llm/models/baichuan/model.py | + tensorrt_llm/models/bert/__init__.py | + tensorrt_llm/models/bert/config.py | + tensorrt_llm/models/bert/convert.py | + tensorrt_llm/models/bert/model.py | + tensorrt_llm/models/bloom/__init__.py | + tensorrt_llm/models/bloom/model.py | + tensorrt_llm/models/chatglm/__init__.py | + tensorrt_llm/models/chatglm/config.py | + tensorrt_llm/models/chatglm/convert.py | + tensorrt_llm/models/chatglm/model.py | + tensorrt_llm/models/clip/__init__.py | + tensorrt_llm/models/clip/model.py | + tensorrt_llm/models/cogvlm/__init__.py | + tensorrt_llm/models/cogvlm/config.py | + tensorrt_llm/models/cogvlm/convert.py | + tensorrt_llm/models/cogvlm/model.py | + tensorrt_llm/models/commandr/__init__.py | + tensorrt_llm/models/commandr/config.py | + tensorrt_llm/models/commandr/model.py | tensorrt_llm/models/convert_utils.py | + tensorrt_llm/models/dbrx/__init__.py | + tensorrt_llm/models/dbrx/config.py | + tensorrt_llm/models/dbrx/model.py | + tensorrt_llm/models/deepseek_v1/__init__.py | + tensorrt_llm/models/deepseek_v1/config.py | + tensorrt_llm/models/deepseek_v1/convert.py | + tensorrt_llm/models/deepseek_v1/model.py | + tensorrt_llm/models/deepseek_v2/__init__.py | + tensorrt_llm/models/deepseek_v2/config.py | + tensorrt_llm/models/deepseek_v2/convert.py | + tensorrt_llm/models/deepseek_v2/model.py | + tensorrt_llm/models/dit/__init__.py | + tensorrt_llm/models/dit/model.py | + tensorrt_llm/models/eagle/__init__.py | + tensorrt_llm/models/eagle/config.py | + tensorrt_llm/models/eagle/model.py | + tensorrt_llm/models/enc_dec/__init__.py | + tensorrt_llm/models/enc_dec/model.py | + tensorrt_llm/models/falcon/__init__.py | + tensorrt_llm/models/falcon/config.py | + tensorrt_llm/models/falcon/convert.py | + tensorrt_llm/models/falcon/model.py | + tensorrt_llm/models/gemma/__init__.py | + tensorrt_llm/models/gemma/config.py | + tensorrt_llm/models/gemma/convert.py | + tensorrt_llm/models/gemma/model.py | + tensorrt_llm/models/gemma/smoothquant.py | + tensorrt_llm/models/gemma/utils/__init__.py | + tensorrt_llm/models/gemma/utils/layers.py | + tensorrt_llm/models/gemma/utils/modules.py | + tensorrt_llm/models/gemma/utils/params.py | + tensorrt_llm/models/gemma/utils/positional_embeddings.py | + tensorrt_llm/models/gemma/utils/sampler.py | + tensorrt_llm/models/gemma/utils/transformer.py | + tensorrt_llm/models/gemma/weight.py | + tensorrt_llm/models/generation_mixin.py | + tensorrt_llm/models/gpt/__init__.py | + tensorrt_llm/models/gpt/config.py | + tensorrt_llm/models/gpt/convert.py | + tensorrt_llm/models/gpt/model.py | + tensorrt_llm/models/gptj/__init__.py | + tensorrt_llm/models/gptj/config.py | + tensorrt_llm/models/gptj/convert.py | + tensorrt_llm/models/gptj/model.py | + tensorrt_llm/models/gptneox/__init__.py | + tensorrt_llm/models/gptneox/model.py | + tensorrt_llm/models/grok/__init__.py | + tensorrt_llm/models/grok/convert.py | + tensorrt_llm/models/grok/model.py | + tensorrt_llm/models/grok/weight.py | + tensorrt_llm/models/llama/__init__.py | + tensorrt_llm/models/llama/config.py | + tensorrt_llm/models/llama/convert.py | + tensorrt_llm/models/llama/model.py | + tensorrt_llm/models/mamba/__init__.py | + tensorrt_llm/models/mamba/config.py | + tensorrt_llm/models/mamba/convert.py | + tensorrt_llm/models/mamba/model.py | + tensorrt_llm/models/medusa/__init__.py | + tensorrt_llm/models/medusa/config.py | + tensorrt_llm/models/medusa/model.py | + tensorrt_llm/models/medusa/weight.py | + tensorrt_llm/models/mllama/__init__.py | + tensorrt_llm/models/mllama/config.py | + tensorrt_llm/models/mllama/model.py | + tensorrt_llm/models/mmdit_sd3/__init__.py | + tensorrt_llm/models/mmdit_sd3/config.py | + tensorrt_llm/models/mmdit_sd3/model.py | + tensorrt_llm/models/model_weights_loader.py | tensorrt_llm/models/modeling_utils.py | + tensorrt_llm/models/mpt/__init__.py | + tensorrt_llm/models/mpt/model.py | + tensorrt_llm/models/multimodal_encoders/__init__.py | + tensorrt_llm/models/multimodal_encoders/config.py | + tensorrt_llm/models/multimodal_encoders/model.py | + tensorrt_llm/models/nemotron_nas/__init__.py | + tensorrt_llm/models/nemotron_nas/config.py | + tensorrt_llm/models/nemotron_nas/convert.py | + tensorrt_llm/models/nemotron_nas/layer_config.py | + tensorrt_llm/models/nemotron_nas/model.py | + tensorrt_llm/models/opt/__init__.py | + tensorrt_llm/models/opt/model.py | + tensorrt_llm/models/phi/__init__.py | + tensorrt_llm/models/phi/config.py | + tensorrt_llm/models/phi/convert.py | + tensorrt_llm/models/phi/model.py | + tensorrt_llm/models/phi3/__init__.py | + tensorrt_llm/models/phi3/config.py | + tensorrt_llm/models/phi3/convert.py | + tensorrt_llm/models/phi3/model.py | + tensorrt_llm/models/phi3/split_weights.py | + tensorrt_llm/models/qwen/__init__.py | + tensorrt_llm/models/qwen/config.py | + tensorrt_llm/models/qwen/convert.py | + tensorrt_llm/models/qwen/model.py | + tensorrt_llm/models/qwen/utils.py | + tensorrt_llm/models/recurrentgemma/__init__.py | + tensorrt_llm/models/recurrentgemma/model.py | + tensorrt_llm/models/redrafter/__init__.py | + tensorrt_llm/models/redrafter/drafter.py | + tensorrt_llm/models/redrafter/model.py | + tensorrt_llm/models/redrafter/redrafter_helper.py | + tensorrt_llm/models/stdit/__init__.py | + tensorrt_llm/models/stdit/config.py | + tensorrt_llm/models/stdit/model.py | + tensorrt_llm/models/unet/__init__.py | + tensorrt_llm/models/unet/attention.py | + tensorrt_llm/models/unet/embeddings.py | + tensorrt_llm/models/unet/pp/__init__.py | + tensorrt_llm/models/unet/pp/attention.py | + tensorrt_llm/models/unet/pp/conv2d.py | + tensorrt_llm/models/unet/pp/groupnorm.py | + tensorrt_llm/models/unet/pp/unet_pp.py | + tensorrt_llm/models/unet/resnet.py | + tensorrt_llm/models/unet/unet_2d_blocks.py | + tensorrt_llm/models/unet/unet_2d_condition.py | + tensorrt_llm/models/unet/weights.py | + tensorrt_llm/network.py | + tensorrt_llm/parameter.py | + tensorrt_llm/plugin/__init__.py | + tensorrt_llm/plugin/plugin.py | tensorrt_llm/quantization/__init__.py | tensorrt_llm/quantization/functional.py | + tensorrt_llm/quantization/image_processing.py | + tensorrt_llm/quantization/layers.py | tensorrt_llm/quantization/mode.py | + tensorrt_llm/quantization/quantize.py | + tensorrt_llm/quantization/quantize_by_modelopt.py | tensorrt_llm/quantization/utils/__init__.py | tensorrt_llm/quantization/utils/fp4_utils.py | tensorrt_llm/quantization/utils/fp8_utils.py | tensorrt_llm/ray_stub.py | tensorrt_llm/runtime/__init__.py | + tensorrt_llm/runtime/enc_dec_model_runner.py | + tensorrt_llm/runtime/generation.py | + tensorrt_llm/runtime/kv_cache_manager.py | + tensorrt_llm/runtime/medusa_utils.py | tensorrt_llm/runtime/memory_pools/__init__.py | + tensorrt_llm/runtime/memory_pools/memory_pools_allocator.py | + tensorrt_llm/runtime/memory_pools/pool.py | + tensorrt_llm/runtime/memory_pools/pools_kv_cache_manager.py | + tensorrt_llm/runtime/model_runner.py | + tensorrt_llm/runtime/model_runner_cpp.py | + tensorrt_llm/runtime/multimodal_model_runner.py | + tensorrt_llm/runtime/processor_wrapper/__init__.py | + tensorrt_llm/runtime/processor_wrapper/mllama_processor_wrapper.py | + tensorrt_llm/runtime/processor_wrapper/processor_wrapper.py | + tensorrt_llm/runtime/redrafter_utils.py | + tensorrt_llm/runtime/session.py | tensorrt_llm/scaffolding/__init__.py | tensorrt_llm/scaffolding/benchmark.py | tensorrt_llm/scaffolding/contrib/AsyncGeneration/stream_generation.py | @@ -479,7 +826,12 @@ common-files: &common_files | tensorrt_llm/tokenizer/tokenizer.py | tensorrt_llm/tools/__init__.py | tensorrt_llm/tools/importlib_utils.py | + tensorrt_llm/tools/multimodal_builder.py | + tensorrt_llm/tools/onnx_utils.py | tensorrt_llm/tools/plugin_gen/__init__.py | + tensorrt_llm/tools/plugin_gen/core.py | + tensorrt_llm/tools/plugin_gen/plugin_gen.py | + tensorrt_llm/tools/plugin_gen/shape_infer.py | tensorrt_llm/tools/ppl.py | tensorrt_llm/tools/profiler/nsys_profile_tools/gputrc2graph.py | tensorrt_llm/version.py | @@ -488,7 +840,9 @@ common-files: &common_files | tests/integration/defs/accuracy/accuracy_core.py | tests/integration/defs/accuracy/scripts/collect_evaluated_accuracies.py | tests/integration/defs/accuracy/scripts/compute_theta_and_thresholds.py | + tests/integration/defs/accuracy/test_cli_flow.py | tests/integration/defs/accuracy/test_disaggregated_serving.py | + tests/integration/defs/accuracy/test_llm_api.py | tests/integration/defs/accuracy/test_llm_api_autodeploy.py | tests/integration/defs/accuracy/test_llm_api_pytorch.py | tests/integration/defs/accuracy/test_llm_api_pytorch_ray.py | @@ -497,29 +851,63 @@ common-files: &common_files | tests/integration/defs/conftest.py | tests/integration/defs/cpp/conftest.py | tests/integration/defs/cpp/cpp_common.py | + tests/integration/defs/cpp/test_e2e.py | tests/integration/defs/cpp/test_multi_gpu.py | tests/integration/defs/cpp/test_unit_tests.py | + tests/integration/defs/deterministic/mixtral_deterministic.py | + tests/integration/defs/deterministic/test_mixtral_deterministic.py | tests/integration/defs/disaggregated/test_auto_scaling.py | tests/integration/defs/disaggregated/test_disaggregated.py | tests/integration/defs/disaggregated/test_disaggregated_etcd.py | tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py | tests/integration/defs/disaggregated/test_workers.py | + tests/integration/defs/examples/run_llm_fp8_quant_llama_70b.py | tests/integration/defs/examples/run_llm_quickstart_atexit.py | tests/integration/defs/examples/serve/test_serve.py | tests/integration/defs/examples/serve/test_serve_negative.py | tests/integration/defs/examples/test_ad_guided_decoding.py | + tests/integration/defs/examples/test_bert.py | + tests/integration/defs/examples/test_bindings.py | + tests/integration/defs/examples/test_chatglm.py | + tests/integration/defs/examples/test_commandr.py | + tests/integration/defs/examples/test_draft_target_model.py | + tests/integration/defs/examples/test_eagle.py | + tests/integration/defs/examples/test_enc_dec.py | + tests/integration/defs/examples/test_exaone.py | + tests/integration/defs/examples/test_gemma.py | tests/integration/defs/examples/test_gpt.py | + tests/integration/defs/examples/test_gptj.py | + tests/integration/defs/examples/test_granite.py | + tests/integration/defs/examples/test_internlm.py | + tests/integration/defs/examples/test_llama.py | tests/integration/defs/examples/test_llm_api_with_mpi.py | + tests/integration/defs/examples/test_mamba.py | + tests/integration/defs/examples/test_medusa.py | + tests/integration/defs/examples/test_mistral.py | + tests/integration/defs/examples/test_mixtral.py | + tests/integration/defs/examples/test_multimodal.py | + tests/integration/defs/examples/test_nemotron.py | + tests/integration/defs/examples/test_nemotron_nas.py | + tests/integration/defs/examples/test_ngram.py | + tests/integration/defs/examples/test_openai.py | tests/integration/defs/examples/test_phi.py | + tests/integration/defs/examples/test_qwen.py | + tests/integration/defs/examples/test_qwen2audio.py | + tests/integration/defs/examples/test_qwenvl.py | tests/integration/defs/examples/test_ray.py | + tests/integration/defs/examples/test_recurrentgemma.py | + tests/integration/defs/examples/test_redrafter.py | + tests/integration/defs/examples/test_whisper.py | tests/integration/defs/llmapi/__init__.py | tests/integration/defs/llmapi/_run_llmapi_llm.py | tests/integration/defs/llmapi/test_llm_api_connector.py | tests/integration/defs/llmapi/test_llm_api_qa.py | + tests/integration/defs/llmapi/test_llm_e2e.py | tests/integration/defs/llmapi/test_llm_examples.py | tests/integration/defs/local_venv.py | tests/integration/defs/perf/__init__.py | tests/integration/defs/perf/allowed_configs.py | + tests/integration/defs/perf/build.py | tests/integration/defs/perf/create_perf_comparison_report.py | tests/integration/defs/perf/data.py | tests/integration/defs/perf/data_export.py | @@ -540,21 +928,38 @@ common-files: &common_files | tests/integration/defs/test_fmha.py | tests/integration/defs/test_list_parser.py | tests/integration/defs/test_list_validation.py | + tests/integration/defs/test_mlpf_results.py | tests/integration/defs/test_sanity.py | tests/integration/defs/test_unittests.py | tests/integration/defs/triton_server/__init__.py | + tests/integration/defs/triton_server/build_engines.py | tests/integration/defs/triton_server/common.py | tests/integration/defs/triton_server/conftest.py | + tests/integration/defs/triton_server/local_venv.py | + tests/integration/defs/triton_server/rcca/bug_4323566/inflight_batcher_llm_client_with_end_id.py | + tests/integration/defs/triton_server/runner_interface.py | tests/integration/defs/triton_server/test_list_parser.py | + tests/integration/defs/triton_server/test_triton.py | + tests/integration/defs/triton_server/test_triton_llm.py | + tests/integration/defs/triton_server/test_triton_memleak.py | + tests/integration/defs/triton_server/test_triton_multi_node.py | + tests/integration/defs/triton_server/test_triton_rcca.py | tests/integration/defs/triton_server/trt_test_alternative.py | tests/integration/defs/trt_test_alternative.py | tests/integration/defs/utils/__init__.py | tests/integration/defs/utils/periodic_junit.py | tests/integration/defs/utils/timeout_manager.py | tests/microbenchmarks/all_reduce.py | + tests/microbenchmarks/build_time_benchmark.py | + tests/microbenchmarks/build_time_dashboard.py | tests/scripts/allreduce_perf/allreduce_heuristic_code_gen.py | tests/scripts/allreduce_perf/allreduce_perf_viz.py | tests/scripts/iteration_log_parser.py | + tests/scripts/perf-sanity/parse_benchmark_results.py | + tests/scripts/perf-sanity/run_benchmark_serve.py | + tests/unittest/_torch/attention/sparse/test_dsa_indexer.py | + tests/unittest/_torch/attention/sparse/test_flash_mla.py | + tests/unittest/_torch/attention/sparse/test_rocketkv.py | tests/unittest/_torch/attention/sparse/test_sparse_mla_forward.py | tests/unittest/_torch/attention/test_attention.py | tests/unittest/_torch/attention/test_attention_mla.py | @@ -575,6 +980,7 @@ common-files: &common_files | tests/unittest/_torch/misc/test_virtual_memory.py | tests/unittest/_torch/modeling/test_modeling_bert.py | tests/unittest/_torch/modeling/test_modeling_clip.py | + tests/unittest/_torch/modeling/test_modeling_exaone4.py | tests/unittest/_torch/modeling/test_modeling_gemma3.py | tests/unittest/_torch/modeling/test_modeling_gpt_oss.py | tests/unittest/_torch/modeling/test_modeling_llama.py | @@ -598,6 +1004,8 @@ common-files: &common_files | tests/unittest/_torch/modules/test_moe_routing.py | tests/unittest/_torch/modules/test_rotary_embedding.py | tests/unittest/_torch/modules/test_triton_linear.py | + tests/unittest/_torch/modules/tests_lora_modules/test_lora_attention_pytorch_flow_vs_trt.py | + tests/unittest/_torch/modules/tests_lora_modules/test_lora_plugin_vs_lora_op.py | tests/unittest/_torch/multi_gpu/test_allreduce.py | tests/unittest/_torch/multi_gpu/test_alltoall.py | tests/unittest/_torch/multi_gpu/test_ar_residual_norm.py | @@ -624,10 +1032,24 @@ common-files: &common_files | tests/unittest/_torch/sampler/test_beam_search.py | tests/unittest/_torch/sampler/test_best_of_n.py | tests/unittest/_torch/sampler/test_trtllm_sampler.py | + tests/unittest/_torch/speculative/test_draft_target.py | + tests/unittest/_torch/speculative/test_draft_token_tree_sampling.py | + tests/unittest/_torch/speculative/test_draft_token_tree_verification.py | + tests/unittest/_torch/speculative/test_dynamic_spec_decode.py | tests/unittest/_torch/speculative/test_eagle3.py | + tests/unittest/_torch/speculative/test_kv_cache_reuse.py | + tests/unittest/_torch/speculative/test_mtp.py | + tests/unittest/_torch/speculative/test_ngram.py | + tests/unittest/_torch/speculative/test_save_state.py | + tests/unittest/_torch/speculative/test_spec_gate.py | + tests/unittest/_torch/speculative/test_torch_rejection_sampling.py | + tests/unittest/_torch/speculative/test_user_provided.py | tests/unittest/_torch/test_connector.py | tests/unittest/_torch/test_torch_multi_arange.py | tests/unittest/_torch/thop/parallel/deep_gemm_tests.py | + tests/unittest/_torch/thop/parallel/test_causal_conv1d_op.py | + tests/unittest/_torch/thop/parallel/test_cublas_mm.py | + tests/unittest/_torch/thop/parallel/test_custom_ops.py | tests/unittest/_torch/thop/parallel/test_dsv3_fused_a_gemm.py | tests/unittest/_torch/thop/parallel/test_dsv3_router_gemm.py | tests/unittest/_torch/thop/parallel/test_finegrained_mixed_dtype_gemm.py | @@ -641,6 +1063,11 @@ common-files: &common_files | tests/unittest/_torch/thop/parallel/test_fp8_per_tensor_scale_tllmg_gemm.py | tests/unittest/_torch/thop/parallel/test_fp8_quantize.py | tests/unittest/_torch/thop/parallel/test_fp8_rowwise_linear.py | + tests/unittest/_torch/thop/parallel/test_fused_qk_norm_rope.py | + tests/unittest/_torch/thop/parallel/test_logits_bitmask_op.py | + tests/unittest/_torch/thop/parallel/test_mamba2_chunk_ss_update.py | + tests/unittest/_torch/thop/parallel/test_mamba_conv1d_op.py | + tests/unittest/_torch/thop/parallel/test_noaux_tc.py | tests/unittest/_torch/thop/parallel/test_scaled_mm.py | tests/unittest/_torch/thop/parallel/test_selective_scan_op.py | tests/unittest/_torch/thop/parallel/test_tinygemm2.py | @@ -654,6 +1081,7 @@ common-files: &common_files | tests/unittest/_torch/thop/serial/test_moe_alltoall.py | tests/unittest/api_stability/api_stability_core.py | tests/unittest/api_stability/test_llm_api.py | + tests/unittest/bindings/binding_test_utils.py | tests/unittest/bindings/test_bindings_moe.py | tests/unittest/bindings/test_bindings_ut.py | tests/unittest/bindings/test_executor_bindings.py | @@ -684,10 +1112,12 @@ common-files: &common_files | tests/unittest/llmapi/apps/_test_openai_chat_harmony.py | tests/unittest/llmapi/apps/_test_openai_chat_multimodal.py | tests/unittest/llmapi/apps/_test_openai_completions.py | + tests/unittest/llmapi/apps/_test_openai_consistent_chat.py | tests/unittest/llmapi/apps/_test_openai_lora.py | tests/unittest/llmapi/apps/_test_openai_metrics.py | tests/unittest/llmapi/apps/_test_openai_misc.py | tests/unittest/llmapi/apps/_test_openai_mmencoder.py | + tests/unittest/llmapi/apps/_test_openai_multi_chat.py | tests/unittest/llmapi/apps/_test_openai_multi_gpu.py | tests/unittest/llmapi/apps/_test_openai_multi_nodes.py | tests/unittest/llmapi/apps/_test_openai_perf_metrics.py | @@ -710,12 +1140,15 @@ common-files: &common_files | tests/unittest/llmapi/run_llm_exit.py | tests/unittest/llmapi/run_llm_with_postproc.py | tests/unittest/llmapi/test_additional_model_outputs.py | + tests/unittest/llmapi/test_build_cache.py | tests/unittest/llmapi/test_executor.py | tests/unittest/llmapi/test_gc_utils.py | tests/unittest/llmapi/test_llm.py | tests/unittest/llmapi/test_llm_args.py | tests/unittest/llmapi/test_llm_download.py | tests/unittest/llmapi/test_llm_kv_cache_events.py | + tests/unittest/llmapi/test_llm_models.py | + tests/unittest/llmapi/test_llm_multi_gpu.py | tests/unittest/llmapi/test_llm_multi_gpu_pytorch.py | tests/unittest/llmapi/test_llm_pytorch.py | tests/unittest/llmapi/test_llm_quant.py | @@ -726,14 +1159,25 @@ common-files: &common_files | tests/unittest/llmapi/test_serialization.py | tests/unittest/llmapi/test_utils.py | tests/unittest/others/__init__.py | + tests/unittest/others/test_builder.py | tests/unittest/others/test_convert_spec_decoding_mask_to_packed_mask.py | + tests/unittest/others/test_debugging_api.py | tests/unittest/others/test_exception.py | tests/unittest/others/test_export.py | + tests/unittest/others/test_graph_rewriter.py | + tests/unittest/others/test_kv_cache_manager.py | tests/unittest/others/test_kv_cache_transceiver.py | tests/unittest/others/test_kv_cache_update.py | + tests/unittest/others/test_layer.py | + tests/unittest/others/test_leak.py | tests/unittest/others/test_mapping.py | + tests/unittest/others/test_model_dtype.py | + tests/unittest/others/test_module.py | tests/unittest/others/test_multimodal_registry.py | + tests/unittest/others/test_plugins.py | + tests/unittest/others/test_precision_control.py | tests/unittest/others/test_pretrained_config.py | + tests/unittest/others/test_session.py | tests/unittest/others/test_time_breakdown.py | tests/unittest/profile_utils.py | tests/unittest/scaffolding/__init__.py | @@ -742,34 +1186,194 @@ common-files: &common_files | tests/unittest/scaffolding/test_scaffolding.py | tests/unittest/scaffolding/test_task_collection.py | tests/unittest/scaffolding/test_worker.py | + tests/unittest/test_model_runner_cpp.py | tests/unittest/test_pip_install.py | tests/unittest/tools/__init__.py | + tests/unittest/tools/plugin_gen/__init__.py | + tests/unittest/tools/plugin_gen/kernel_config.py | + tests/unittest/tools/plugin_gen/test_core.py | + tests/unittest/tools/plugin_gen/test_plugin_gen.py | + tests/unittest/tools/plugin_gen/test_shape_infer.py | tests/unittest/tools/test_prepare_dataset.py | tests/unittest/tools/test_test_to_stage_mapping.py | + tests/unittest/trt/__init__.py | + tests/unittest/trt/attention/test_bert_attention.py | + tests/unittest/trt/attention/test_gpt_attention.py | + tests/unittest/trt/attention/test_gpt_attention_IFB.py | + tests/unittest/trt/attention/test_gpt_attention_no_cache.py | + tests/unittest/trt/attention/test_sage_attention.py | + tests/unittest/trt/functional/__init__.py | + tests/unittest/trt/functional/test_alibi.py | + tests/unittest/trt/functional/test_allreduce_norm.py | + tests/unittest/trt/functional/test_allreduce_prepost_residual_norm.py | + tests/unittest/trt/functional/test_arange.py | + tests/unittest/trt/functional/test_argmax.py | + tests/unittest/trt/functional/test_assertion.py | + tests/unittest/trt/functional/test_avg_pool2d.py | + tests/unittest/trt/functional/test_cast.py | + tests/unittest/trt/functional/test_conv2d.py | + tests/unittest/trt/functional/test_conv3d.py | + tests/unittest/trt/functional/test_cos.py | + tests/unittest/trt/functional/test_cumsum.py | + tests/unittest/trt/functional/test_dora.py | + tests/unittest/trt/functional/test_einsum.py | + tests/unittest/trt/functional/test_embedding_single_gpu.py | + tests/unittest/trt/functional/test_exp.py | + tests/unittest/trt/functional/test_expand.py | + tests/unittest/trt/functional/test_flatten.py | + tests/unittest/trt/functional/test_flip.py | + tests/unittest/trt/functional/test_fp4_gemm.py | + tests/unittest/trt/functional/test_fp4_gemm_ootb.py | + tests/unittest/trt/functional/test_gather.py | + tests/unittest/trt/functional/test_gather_nd.py | + tests/unittest/trt/functional/test_geglu.py | + tests/unittest/trt/functional/test_gelu.py | + tests/unittest/trt/functional/test_gemm_swiglu.py | + tests/unittest/trt/functional/test_group_norm.py | + tests/unittest/trt/functional/test_identity.py | + tests/unittest/trt/functional/test_index_select.py | + tests/unittest/trt/functional/test_interpolate.py | + tests/unittest/trt/functional/test_logsoftmax.py | + tests/unittest/trt/functional/test_lora.py | + tests/unittest/trt/functional/test_low_latency_gemm.py | + tests/unittest/trt/functional/test_mamba_conv1d.py | + tests/unittest/trt/functional/test_masked_scatter.py | + tests/unittest/trt/functional/test_masked_select.py | + tests/unittest/trt/functional/test_matmul.py | + tests/unittest/trt/functional/test_meshgrid2d.py | + tests/unittest/trt/functional/test_moe.py | + tests/unittest/trt/functional/test_nccl.py | + tests/unittest/trt/functional/test_nonzero.py | + tests/unittest/trt/functional/test_outer.py | + tests/unittest/trt/functional/test_pad.py | + tests/unittest/trt/functional/test_permute.py | + tests/unittest/trt/functional/test_pp_reduce_scatter.py | + tests/unittest/trt/functional/test_quant.py | + tests/unittest/trt/functional/test_rearrange.py | + tests/unittest/trt/functional/test_repeat.py | + tests/unittest/trt/functional/test_repeat_interleave.py | + tests/unittest/trt/functional/test_rg_lru.py | + tests/unittest/trt/functional/test_sample.py | + tests/unittest/trt/functional/test_scatter.py | + tests/unittest/trt/functional/test_scatter_nd.py | + tests/unittest/trt/functional/test_select.py | + tests/unittest/trt/functional/test_selective_scan.py | + tests/unittest/trt/functional/test_sigmoid.py | + tests/unittest/trt/functional/test_silu.py | + tests/unittest/trt/functional/test_sin.py | + tests/unittest/trt/functional/test_slice.py | + tests/unittest/trt/functional/test_softplus.py | + tests/unittest/trt/functional/test_split.py | + tests/unittest/trt/functional/test_squeeze.py | + tests/unittest/trt/functional/test_swiglu.py | + tests/unittest/trt/functional/test_topk.py | + tests/unittest/trt/functional/test_transpose.py | + tests/unittest/trt/functional/test_unbind.py | + tests/unittest/trt/functional/test_unsqueeze.py | + tests/unittest/trt/functional/test_view.py | + tests/unittest/trt/functional/test_where.py | + tests/unittest/trt/model/__init__.py | + tests/unittest/trt/model/eagle/test_decode_draft_tokens_plugin.py | + tests/unittest/trt/model/eagle/test_prepare_drafter_inputs_plugin.py | + tests/unittest/trt/model/eagle/test_sample_accept_draft_tokens_plugin.py | + tests/unittest/trt/model/redrafter/test_beams2tree.py | + tests/unittest/trt/model/redrafter/test_draft_token.py | + tests/unittest/trt/model/redrafter/test_draft_token_indices.py | + tests/unittest/trt/model/redrafter/test_gather_beams.py | + tests/unittest/trt/model/redrafter/test_mask.py | + tests/unittest/trt/model/redrafter/test_packed_position_ids.py | + tests/unittest/trt/model/redrafter/test_prefix_match_indices.py | + tests/unittest/trt/model/redrafter/test_prepare_input.py | + tests/unittest/trt/model/redrafter/test_process_logits.py | + tests/unittest/trt/model/redrafter/test_top1.py | + tests/unittest/trt/model/redrafter/test_unpack_gen_data.py | + tests/unittest/trt/model/redrafter/test_validate.py | + tests/unittest/trt/model/test_gpt.py | + tests/unittest/trt/model/test_gpt_e2e.py | + tests/unittest/trt/model/test_llama.py | + tests/unittest/trt/model/test_mamba.py | + tests/unittest/trt/model/test_mistral.py | + tests/unittest/trt/model/test_nemotron_nas.py | + tests/unittest/trt/model/test_phi.py | + tests/unittest/trt/model/test_unet.py | + tests/unittest/trt/model_api/test_model_api_multi_gpu.py | + tests/unittest/trt/model_api/test_model_level_api.py | + tests/unittest/trt/model_api/test_model_quantization.py | + tests/unittest/trt/python_plugin/plugin_wrapper_utils.py | + tests/unittest/trt/python_plugin/test_plugin_wrapper.py | + tests/unittest/trt/quantization/__init__.py | + tests/unittest/trt/quantization/_utils.py | + tests/unittest/trt/quantization/test_fp8_quantization.py | + tests/unittest/trt/quantization/test_fp8_rowwise_gemm.py | + tests/unittest/trt/quantization/test_functional.py | + tests/unittest/trt/quantization/test_mode.py | + tests/unittest/trt/quantization/test_moe_weight_only_quant_matmul.py | + tests/unittest/trt/quantization/test_qserve_gemm.py | + tests/unittest/trt/quantization/test_quant.py | + tests/unittest/trt/quantization/test_quant_layer.py | + tests/unittest/trt/quantization/test_smooth_quant_gemm.py | + tests/unittest/trt/quantization/test_smooth_quant_layer_norm.py | + tests/unittest/trt/quantization/test_smooth_quant_rms_norm.py | + tests/unittest/trt/quantization/test_weight_only_groupwise_quant_matmul.py | + tests/unittest/trt/quantization/test_weight_only_quant_matmul.py | tests/unittest/utils/__init__.py | tests/unittest/utils/cpp_paths.py | tests/unittest/utils/llm_data.py | tests/unittest/utils/runtime_defaults.py | + tests/unittest/utils/test_medusa_utils.py | tests/unittest/utils/test_prebuilt_whl_cpp_extensions.py | tests/unittest/utils/test_util.py | tests/unittest/utils/torch_ref.py | tests/unittest/utils/util.py | + triton_backend/all_models/disaggregated_serving/disaggregated_serving_bls/1/model.py | + triton_backend/all_models/gpt/postprocessing/1/model.py | + triton_backend/all_models/gpt/preprocessing/1/model.py | + triton_backend/all_models/gpt/tensorrt_llm/1/model.py | + triton_backend/all_models/inflight_batcher_llm/postprocessing/1/model.py | + triton_backend/all_models/inflight_batcher_llm/preprocessing/1/model.py | + triton_backend/all_models/inflight_batcher_llm/tensorrt_llm/1/model.py | + triton_backend/all_models/inflight_batcher_llm/tensorrt_llm_bls/1/lib/decode.py | + triton_backend/all_models/inflight_batcher_llm/tensorrt_llm_bls/1/lib/triton_decoder.py | + triton_backend/all_models/inflight_batcher_llm/tensorrt_llm_bls/1/model.py | triton_backend/all_models/llmapi/tensorrt_llm/1/helpers.py | triton_backend/all_models/llmapi/tensorrt_llm/1/model.py | + triton_backend/all_models/multimodal/multimodal_encoders/1/model.py | + triton_backend/all_models/multimodal/multimodal_encoders/1/multimodal_utils.py | + triton_backend/all_models/tests/test_decode.py | triton_backend/all_models/tests/test_llmapi_python_backend.py | + triton_backend/all_models/tests/test_multi_image_preprocess.py | + triton_backend/all_models/tests/test_multimodal_encoders.py | + triton_backend/all_models/tests/test_python_backend.py | + triton_backend/all_models/tests/test_triton_decoder.py | + triton_backend/all_models/whisper/whisper_bls/1/fbank.py | + triton_backend/all_models/whisper/whisper_bls/1/model.py | + triton_backend/all_models/whisper/whisper_bls/1/tokenizer.py | + triton_backend/ci/L0_backend_trtllm/base_metrics_verification_tests.py | + triton_backend/ci/L0_backend_trtllm/custom_metrics_verification_tests.py | + triton_backend/inflight_batcher_llm/client/__init__.py | + triton_backend/inflight_batcher_llm/client/e2e_grpc_speculative_decoding_client.py | + triton_backend/inflight_batcher_llm/client/end_to_end_grpc_client.py | + triton_backend/inflight_batcher_llm/client/inflight_batcher_llm_client.py | triton_backend/scripts/launch_triton_server.py | triton_backend/tools/__init__.py | triton_backend/tools/fill_template.py | + triton_backend/tools/gpt/benchmark_core_model.py | + triton_backend/tools/gpt/client.py | + triton_backend/tools/gpt/client_async.py | + triton_backend/tools/gpt/end_to_end_test.py | + triton_backend/tools/gpt/gen_input_data.py | triton_backend/tools/inflight_batcher_llm/benchmark_core_model.py | triton_backend/tools/inflight_batcher_llm/end_to_end_test.py | triton_backend/tools/inflight_batcher_llm/speculative_decoding_test.py | triton_backend/tools/inflight_batcher_llm/test_max_queue_size.py | triton_backend/tools/llmapi_client.py | + triton_backend/tools/multimodal/client.py | triton_backend/tools/tests/__init__.py | triton_backend/tools/tests/test_fill_template.py | triton_backend/tools/tests/test_llmapi_cancel.py | triton_backend/tools/utils/__init__.py | - triton_backend/tools/utils/utils.py + triton_backend/tools/utils/utils.py | + triton_backend/tools/whisper/client.py )$ # Used by ruff hooks: main ruff (exclude: *legacy_files) and @@ -779,6 +1383,14 @@ legacy-files: &legacy_files | .devcontainer/make_env.py | .github/scripts/label_community_user.py | .github/scripts/pr_checklist_check.py | + benchmarks/cpp/__init__.py | + benchmarks/cpp/prepare_dataset.py | + benchmarks/cpp/utils/__init__.py | + benchmarks/cpp/utils/convert_nemo_dataset.py | + benchmarks/cpp/utils/generate_rand_loras.py | + benchmarks/cpp/utils/prepare_real_data.py | + benchmarks/cpp/utils/prepare_synthetic_data.py | + benchmarks/cpp/utils/utils.py | cpp/conanfile.py | cpp/kernels/fmha_v2/conftest.py | cpp/kernels/fmha_v2/fmha_test.py | @@ -803,17 +1415,58 @@ legacy-files: &legacy_files | cpp/tensorrt_llm/deep_ep/strip_nvshmem_helper.py | cpp/tensorrt_llm/kernels/cutlass_kernels/python/generate_kernels.py | cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/copy_cu.py | + cpp/tests/resources/scripts/build_chatglm_engines.py | + cpp/tests/resources/scripts/build_eagle_engines.py | + cpp/tests/resources/scripts/build_enc_dec_engines.py | + cpp/tests/resources/scripts/build_engines_utils.py | + cpp/tests/resources/scripts/build_gpt_engines.py | + cpp/tests/resources/scripts/build_gptj_engines.py | + cpp/tests/resources/scripts/build_llama_engines.py | + cpp/tests/resources/scripts/build_mamba_engines.py | + cpp/tests/resources/scripts/build_medusa_engines.py | + cpp/tests/resources/scripts/build_recurrentgemma_engines.py | + cpp/tests/resources/scripts/build_redrafter_engines.py | + cpp/tests/resources/scripts/generate_expected_chatglm_output.py | + cpp/tests/resources/scripts/generate_expected_eagle_output.py | + cpp/tests/resources/scripts/generate_expected_enc_dec_output.py | + cpp/tests/resources/scripts/generate_expected_gpt_output.py | + cpp/tests/resources/scripts/generate_expected_gptj_output.py | + cpp/tests/resources/scripts/generate_expected_llama_output.py | + cpp/tests/resources/scripts/generate_expected_mamba_output.py | + cpp/tests/resources/scripts/generate_expected_medusa_output.py | + cpp/tests/resources/scripts/generate_expected_recurrentgemma_output.py | + cpp/tests/resources/scripts/generate_expected_redrafter_output.py | + cpp/tests/resources/scripts/generate_hf_gpt_output.py | cpp/tests/resources/scripts/generate_test_lora_weights.py | + cpp/tests/resources/scripts/io_converter.py | docs/source/conf.py | docs/source/helper.py | examples/apps/chat.py | examples/apps/fastapi_server.py | + examples/bindings/executor/example_advanced.py | + examples/bindings/executor/example_basic.py | + examples/bindings/executor/example_debug.py | + examples/bindings/executor/example_logits_processor.py | examples/disaggregated/clients/disagg_client.py | examples/disaggregated/slurm/benchmark/submit.py | + examples/dora/normalize_weights.py | + examples/eagle/convert_checkpoint.py | + examples/eval_long_context.py | + examples/generate_checkpoint_config.py | + examples/generate_xgrammar_tokenizer_info.py | + examples/hf_lora_convert.py | examples/infinitebench/args.py | examples/infinitebench/compute_scores.py | examples/infinitebench/construct_synthetic_dataset.py | examples/infinitebench/eval_utils.py | + examples/llm-api/_tensorrt_engine/llm_eagle2_decoding.py | + examples/llm-api/_tensorrt_engine/llm_eagle_decoding.py | + examples/llm-api/_tensorrt_engine/llm_inference_customize.py | + examples/llm-api/_tensorrt_engine/llm_inference_kv_events.py | + examples/llm-api/_tensorrt_engine/llm_lookahead_decoding.py | + examples/llm-api/_tensorrt_engine/llm_medusa_decoding.py | + examples/llm-api/_tensorrt_engine/llm_quantization.py | + examples/llm-api/_tensorrt_engine/quickstart_example.py | examples/llm-api/llm_guided_decoding.py | examples/llm-api/llm_inference.py | examples/llm-api/llm_inference_async.py | @@ -833,17 +1486,122 @@ legacy-files: &legacy_files | examples/llm-api/quickstart_example.py | examples/llm-api/quickstart_multimodal.py | examples/llm-api/star_attention.py | + examples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.py | examples/longbench/eval_longbench_v1.py | + examples/medusa/convert_checkpoint.py | + examples/mmlu.py | + examples/models/contrib/baichuan/convert_checkpoint.py | + examples/models/contrib/bloom/convert_checkpoint.py | + examples/models/contrib/chatglm-6b/tokenization_chatglm.py | + examples/models/contrib/chatglm2-6b/tokenization_chatglm.py | + examples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.py | + examples/models/contrib/cogvlm/convert_checkpoint.py | + examples/models/contrib/dbrx/convert_checkpoint.py | + examples/models/contrib/deepseek_v1/__init__.py | + examples/models/contrib/deepseek_v1/convert_checkpoint.py | + examples/models/contrib/deepseek_v2/convert_checkpoint.py | + examples/models/contrib/dit/convert_checkpoint.py | + examples/models/contrib/dit/diffusion.py | + examples/models/contrib/dit/sample.py | + examples/models/contrib/dit/utils_modelopt.py | + examples/models/contrib/dit/vae_decoder_trt.py | + examples/models/contrib/falcon/convert_checkpoint.py | + examples/models/contrib/gptj/convert_checkpoint.py | + examples/models/contrib/gptneox/convert_checkpoint.py | + examples/models/contrib/grok/convert_checkpoint.py | + examples/models/contrib/mmdit/convert_checkpoint.py | + examples/models/contrib/mmdit/sample.py | + examples/models/contrib/mpt/convert_checkpoint.py | + examples/models/contrib/opt/convert_checkpoint.py | + examples/models/contrib/sdxl/build_sdxl_unet.py | + examples/models/contrib/sdxl/pipeline_stable_diffusion_xl.py | + examples/models/contrib/sdxl/run_sdxl.py | + examples/models/contrib/stdit/aspect.py | + examples/models/contrib/stdit/convert_checkpoint.py | + examples/models/contrib/stdit/pipeline_tllm.py | + examples/models/contrib/stdit/sample.py | + examples/models/contrib/stdit/scheduler.py | + examples/models/contrib/stdit/text_encoder.py | + examples/models/contrib/stdit/utils.py | + examples/models/contrib/stdit/vae.py | + examples/models/contrib/stdit/video_transforms.py | + examples/models/core/bert/__init__.py | + examples/models/core/bert/convert_checkpoint.py | + examples/models/core/bert/run.py | + examples/models/core/bert/utils.py | + examples/models/core/commandr/convert_checkpoint.py | + examples/models/core/enc_dec/__init__.py | + examples/models/core/enc_dec/convert_checkpoint.py | + examples/models/core/enc_dec/helper.py | + examples/models/core/enc_dec/run.py | + examples/models/core/gemma/convert_checkpoint.py | + examples/models/core/glm-4-9b/convert_checkpoint.py | + examples/models/core/glm-4-9b/tokenization_chatglm.py | + examples/models/core/gpt/convert_checkpoint.py | + examples/models/core/gpt/merge_ptuning_tables.py | + examples/models/core/gpt/nemo_lora_convert.py | + examples/models/core/gpt/nemo_prompt_convert.py | + examples/models/core/gpt/run_hf.py | examples/models/core/gpt_oss/openai_chat_client_function_calling.py | + examples/models/core/internlm2/convert_checkpoint.py | examples/models/core/kimi_k2/kimi_k2_tool_calling_example.py | + examples/models/core/llama/convert_checkpoint.py | + examples/models/core/llama/summarize_long.py | + examples/models/core/mamba/convert_checkpoint.py | + examples/models/core/mllama/convert_checkpoint.py | + examples/models/core/multimodal/__init__.py | + examples/models/core/multimodal/build_multimodal_engine.py | + examples/models/core/multimodal/eval.py | + examples/models/core/multimodal/run.py | + examples/models/core/multimodal/utils.py | + examples/models/core/nemotron_nas/calibration_utils.py | + examples/models/core/nemotron_nas/convert_checkpoint.py | + examples/models/core/phi/convert_checkpoint.py | + examples/models/core/qwen/convert_checkpoint.py | + examples/models/core/qwen2audio/run.py | + examples/models/core/qwen2audio/run_chat.py | + examples/models/core/qwen2audio/utils.py | + examples/models/core/qwenvl/run.py | + examples/models/core/qwenvl/run_chat.py | + examples/models/core/qwenvl/show_pic.py | + examples/models/core/qwenvl/vit_onnx_trt.py | + examples/models/core/recurrentgemma/convert_checkpoint.py | + examples/models/core/vit/convert_checkpoint.py | + examples/models/core/whisper/convert_checkpoint.py | + examples/models/core/whisper/distil_whisper/convert_from_distil_whisper.py | + examples/models/core/whisper/run.py | + examples/models/core/whisper/tokenizer.py | + examples/models/core/whisper/whisper_utils.py | + examples/ngram/run_dtm_ngram.py | + examples/openai_triton/manual_plugin/build.py | + examples/openai_triton/manual_plugin/fmha_triton.py | + examples/openai_triton/manual_plugin/plugin.py | + examples/openai_triton/manual_plugin/run.py | + examples/openai_triton/plugin_autogen/build_engine.py | + examples/openai_triton/plugin_autogen/kernel_config.py | + examples/openai_triton/plugin_autogen/run_engine.py | + examples/python_plugin/build_lookup.py | + examples/python_plugin/plugin_lib/__init__.py | + examples/python_plugin/plugin_lib/lookup_kernel.py | + examples/python_plugin/plugin_lib/lookup_plugin.py | + examples/python_plugin/run_lookup.py | + examples/quantization/quantize.py | examples/quantization/quantize_mixed_precision_moe.py | examples/ray_orchestrator/llm_inference_async_ray.py | examples/ray_orchestrator/llm_inference_distributed_ray.py | + examples/redrafter/convert_checkpoint.py | + examples/run.py | examples/scaffolding/contrib/AsyncGeneration/stream_generation_controller.py | examples/scaffolding/contrib/DeepConf/run_generation.py | examples/scaffolding/contrib/Dynasor/scaffolding_dynasor_run.py | examples/scaffolding/contrib/TreeInference/run_mcts_example.py | examples/scaffolding/contrib/TreeInference/run_tot_example.py | + examples/scaffolding/contrib/mcp/e2b/e2bserver.py | + examples/scaffolding/contrib/mcp/e2b/main.py | + examples/scaffolding/contrib/mcp/mcptest.py | + examples/scaffolding/contrib/mcp/weather/weather.py | + examples/scaffolding/contrib/mcp/websearch/main.py | + examples/scaffolding/contrib/mcp/websearch/websearch.py | examples/scaffolding/run_basic_generation.py | examples/scaffolding/run_best_of_n_with_reward.py | examples/scaffolding/run_majority_vote_aime24.py | @@ -853,6 +1611,8 @@ legacy-files: &legacy_files | examples/serve/openai_completion_client.py | examples/serve/openai_completion_client_for_lora.py | examples/serve/openai_completion_client_json_schema.py | + examples/summarize.py | + examples/utils.py | examples/wide_ep/ep_load_balancer/generate_eplb_config.py | examples/wide_ep/ep_load_balancer/report_load_statistics.py | examples/wide_ep/ep_load_balancer/utils.py | @@ -860,10 +1620,12 @@ legacy-files: &legacy_files | jenkins/scripts/mergeWaiveList.py | jenkins/scripts/open_search_db.py | jenkins/scripts/test_rerun.py | + scripts/build_cpp_examples.py | scripts/build_wheel.py | scripts/check_test_list.py | scripts/dco_check.py | scripts/format_test_list.py | + scripts/generate_duration.py | scripts/generate_lock_file.py | scripts/get_wheel_from_package.py | scripts/git_replace.py | @@ -874,6 +1636,7 @@ legacy-files: &legacy_files | setup.py | tensorrt_llm/__init__.py | tensorrt_llm/_ray_utils.py | + tensorrt_llm/_tensorrt_engine/__init__.py | tensorrt_llm/_torch/__init__.py | tensorrt_llm/_torch/attention_backend/__init__.py | tensorrt_llm/_torch/attention_backend/flashinfer.py | @@ -1077,6 +1840,7 @@ legacy-files: &legacy_files | tensorrt_llm/_torch/pyexecutor/guided_decoder.py | tensorrt_llm/_torch/pyexecutor/handle_additional_outputs.py | tensorrt_llm/_torch/pyexecutor/handle_logits.py | + tensorrt_llm/_torch/pyexecutor/kv_cache_connector.py | tensorrt_llm/_torch/pyexecutor/kv_cache_transceiver.py | tensorrt_llm/_torch/pyexecutor/layerwise_nvtx_marker.py | tensorrt_llm/_torch/pyexecutor/llm_request.py | @@ -1114,6 +1878,11 @@ legacy-files: &legacy_files | tensorrt_llm/bench/benchmark/utils/asynchronous.py | tensorrt_llm/bench/benchmark/utils/general.py | tensorrt_llm/bench/benchmark/utils/processes.py | + tensorrt_llm/bench/build/__init__.py | + tensorrt_llm/bench/build/build.py | + tensorrt_llm/bench/build/dataclasses.py | + tensorrt_llm/bench/build/tuning.py | + tensorrt_llm/bench/build/utils.py | tensorrt_llm/bench/dataclasses/__init__.py | tensorrt_llm/bench/dataclasses/configuration.py | tensorrt_llm/bench/dataclasses/engine.py | @@ -1123,9 +1892,13 @@ legacy-files: &legacy_files | tensorrt_llm/bench/dataclasses/statistics.py | tensorrt_llm/bench/utils/__init__.py | tensorrt_llm/bench/utils/data.py | + tensorrt_llm/builder.py | tensorrt_llm/commands/__init__.py | tensorrt_llm/commands/bench.py | + tensorrt_llm/commands/build.py | tensorrt_llm/commands/eval.py | + tensorrt_llm/commands/prune.py | + tensorrt_llm/commands/refit.py | tensorrt_llm/commands/serve.py | tensorrt_llm/evaluate/__init__.py | tensorrt_llm/evaluate/cnn_dailymail.py | @@ -1161,7 +1934,23 @@ legacy-files: &legacy_files | tensorrt_llm/inputs/multimodal.py | tensorrt_llm/inputs/registry.py | tensorrt_llm/inputs/utils.py | + tensorrt_llm/layers/__init__.py | + tensorrt_llm/layers/activation.py | + tensorrt_llm/layers/attention.py | + tensorrt_llm/layers/cast.py | + tensorrt_llm/layers/conv.py | + tensorrt_llm/layers/embedding.py | + tensorrt_llm/layers/language_adapter.py | + tensorrt_llm/layers/linear.py | + tensorrt_llm/layers/lora.py | + tensorrt_llm/layers/mlp.py | + tensorrt_llm/layers/moe.py | + tensorrt_llm/layers/normalization.py | + tensorrt_llm/layers/pooling.py | + tensorrt_llm/layers/recurrent.py | + tensorrt_llm/layers/ssm.py | tensorrt_llm/llmapi/__init__.py | + tensorrt_llm/llmapi/build_cache.py | tensorrt_llm/llmapi/disagg_utils.py | tensorrt_llm/llmapi/kv_cache_type.py | tensorrt_llm/llmapi/llm.py | @@ -1184,17 +1973,179 @@ legacy-files: &legacy_files | tensorrt_llm/metrics/enums.py | tensorrt_llm/models/__init__.py | tensorrt_llm/models/automodel.py | + tensorrt_llm/models/baichuan/__init__.py | + tensorrt_llm/models/baichuan/config.py | + tensorrt_llm/models/baichuan/convert.py | + tensorrt_llm/models/baichuan/model.py | + tensorrt_llm/models/bert/__init__.py | + tensorrt_llm/models/bert/config.py | + tensorrt_llm/models/bert/convert.py | + tensorrt_llm/models/bert/model.py | + tensorrt_llm/models/bloom/__init__.py | + tensorrt_llm/models/bloom/model.py | + tensorrt_llm/models/chatglm/__init__.py | + tensorrt_llm/models/chatglm/config.py | + tensorrt_llm/models/chatglm/convert.py | + tensorrt_llm/models/chatglm/model.py | + tensorrt_llm/models/clip/__init__.py | + tensorrt_llm/models/clip/model.py | + tensorrt_llm/models/cogvlm/__init__.py | + tensorrt_llm/models/cogvlm/config.py | + tensorrt_llm/models/cogvlm/convert.py | + tensorrt_llm/models/cogvlm/model.py | + tensorrt_llm/models/commandr/__init__.py | + tensorrt_llm/models/commandr/config.py | + tensorrt_llm/models/commandr/model.py | tensorrt_llm/models/convert_utils.py | + tensorrt_llm/models/dbrx/__init__.py | + tensorrt_llm/models/dbrx/config.py | + tensorrt_llm/models/dbrx/model.py | + tensorrt_llm/models/deepseek_v1/__init__.py | + tensorrt_llm/models/deepseek_v1/config.py | + tensorrt_llm/models/deepseek_v1/convert.py | + tensorrt_llm/models/deepseek_v1/model.py | + tensorrt_llm/models/deepseek_v2/__init__.py | + tensorrt_llm/models/deepseek_v2/config.py | + tensorrt_llm/models/deepseek_v2/convert.py | + tensorrt_llm/models/deepseek_v2/model.py | + tensorrt_llm/models/dit/__init__.py | + tensorrt_llm/models/dit/model.py | + tensorrt_llm/models/eagle/__init__.py | + tensorrt_llm/models/eagle/config.py | + tensorrt_llm/models/eagle/model.py | + tensorrt_llm/models/enc_dec/__init__.py | + tensorrt_llm/models/enc_dec/model.py | + tensorrt_llm/models/falcon/__init__.py | + tensorrt_llm/models/falcon/config.py | + tensorrt_llm/models/falcon/convert.py | + tensorrt_llm/models/falcon/model.py | + tensorrt_llm/models/gemma/__init__.py | + tensorrt_llm/models/gemma/config.py | + tensorrt_llm/models/gemma/convert.py | + tensorrt_llm/models/gemma/model.py | + tensorrt_llm/models/gemma/smoothquant.py | + tensorrt_llm/models/gemma/utils/__init__.py | + tensorrt_llm/models/gemma/utils/layers.py | + tensorrt_llm/models/gemma/utils/modules.py | + tensorrt_llm/models/gemma/utils/params.py | + tensorrt_llm/models/gemma/utils/positional_embeddings.py | + tensorrt_llm/models/gemma/utils/sampler.py | + tensorrt_llm/models/gemma/utils/transformer.py | + tensorrt_llm/models/gemma/weight.py | + tensorrt_llm/models/generation_mixin.py | + tensorrt_llm/models/gpt/__init__.py | + tensorrt_llm/models/gpt/config.py | + tensorrt_llm/models/gpt/convert.py | + tensorrt_llm/models/gpt/model.py | + tensorrt_llm/models/gptj/__init__.py | + tensorrt_llm/models/gptj/config.py | + tensorrt_llm/models/gptj/convert.py | + tensorrt_llm/models/gptj/model.py | + tensorrt_llm/models/gptneox/__init__.py | + tensorrt_llm/models/gptneox/model.py | + tensorrt_llm/models/grok/__init__.py | + tensorrt_llm/models/grok/convert.py | + tensorrt_llm/models/grok/model.py | + tensorrt_llm/models/grok/weight.py | + tensorrt_llm/models/llama/__init__.py | + tensorrt_llm/models/llama/config.py | + tensorrt_llm/models/llama/convert.py | + tensorrt_llm/models/llama/model.py | + tensorrt_llm/models/mamba/__init__.py | + tensorrt_llm/models/mamba/config.py | + tensorrt_llm/models/mamba/convert.py | + tensorrt_llm/models/mamba/model.py | + tensorrt_llm/models/medusa/__init__.py | + tensorrt_llm/models/medusa/config.py | + tensorrt_llm/models/medusa/model.py | + tensorrt_llm/models/medusa/weight.py | + tensorrt_llm/models/mllama/__init__.py | + tensorrt_llm/models/mllama/config.py | + tensorrt_llm/models/mllama/model.py | + tensorrt_llm/models/mmdit_sd3/__init__.py | + tensorrt_llm/models/mmdit_sd3/config.py | + tensorrt_llm/models/mmdit_sd3/model.py | + tensorrt_llm/models/model_weights_loader.py | tensorrt_llm/models/modeling_utils.py | + tensorrt_llm/models/mpt/__init__.py | + tensorrt_llm/models/mpt/model.py | + tensorrt_llm/models/multimodal_encoders/__init__.py | + tensorrt_llm/models/multimodal_encoders/config.py | + tensorrt_llm/models/multimodal_encoders/model.py | + tensorrt_llm/models/nemotron_nas/__init__.py | + tensorrt_llm/models/nemotron_nas/config.py | + tensorrt_llm/models/nemotron_nas/convert.py | + tensorrt_llm/models/nemotron_nas/layer_config.py | + tensorrt_llm/models/nemotron_nas/model.py | + tensorrt_llm/models/opt/__init__.py | + tensorrt_llm/models/opt/model.py | + tensorrt_llm/models/phi/__init__.py | + tensorrt_llm/models/phi/config.py | + tensorrt_llm/models/phi/convert.py | + tensorrt_llm/models/phi/model.py | + tensorrt_llm/models/phi3/__init__.py | + tensorrt_llm/models/phi3/config.py | + tensorrt_llm/models/phi3/convert.py | + tensorrt_llm/models/phi3/model.py | + tensorrt_llm/models/phi3/split_weights.py | + tensorrt_llm/models/qwen/__init__.py | + tensorrt_llm/models/qwen/config.py | + tensorrt_llm/models/qwen/convert.py | + tensorrt_llm/models/qwen/model.py | + tensorrt_llm/models/qwen/utils.py | + tensorrt_llm/models/recurrentgemma/__init__.py | + tensorrt_llm/models/recurrentgemma/model.py | + tensorrt_llm/models/redrafter/__init__.py | + tensorrt_llm/models/redrafter/drafter.py | + tensorrt_llm/models/redrafter/model.py | + tensorrt_llm/models/redrafter/redrafter_helper.py | + tensorrt_llm/models/stdit/__init__.py | + tensorrt_llm/models/stdit/config.py | + tensorrt_llm/models/stdit/model.py | + tensorrt_llm/models/unet/__init__.py | + tensorrt_llm/models/unet/attention.py | + tensorrt_llm/models/unet/embeddings.py | + tensorrt_llm/models/unet/pp/__init__.py | + tensorrt_llm/models/unet/pp/attention.py | + tensorrt_llm/models/unet/pp/conv2d.py | + tensorrt_llm/models/unet/pp/groupnorm.py | + tensorrt_llm/models/unet/pp/unet_pp.py | + tensorrt_llm/models/unet/resnet.py | + tensorrt_llm/models/unet/unet_2d_blocks.py | + tensorrt_llm/models/unet/unet_2d_condition.py | + tensorrt_llm/models/unet/weights.py | + tensorrt_llm/network.py | + tensorrt_llm/parameter.py | + tensorrt_llm/plugin/__init__.py | + tensorrt_llm/plugin/plugin.py | tensorrt_llm/quantization/__init__.py | tensorrt_llm/quantization/functional.py | + tensorrt_llm/quantization/image_processing.py | + tensorrt_llm/quantization/layers.py | tensorrt_llm/quantization/mode.py | + tensorrt_llm/quantization/quantize.py | + tensorrt_llm/quantization/quantize_by_modelopt.py | tensorrt_llm/quantization/utils/__init__.py | tensorrt_llm/quantization/utils/fp4_utils.py | tensorrt_llm/quantization/utils/fp8_utils.py | tensorrt_llm/ray_stub.py | tensorrt_llm/runtime/__init__.py | + tensorrt_llm/runtime/enc_dec_model_runner.py | + tensorrt_llm/runtime/generation.py | + tensorrt_llm/runtime/kv_cache_manager.py | + tensorrt_llm/runtime/medusa_utils.py | tensorrt_llm/runtime/memory_pools/__init__.py | + tensorrt_llm/runtime/memory_pools/memory_pools_allocator.py | + tensorrt_llm/runtime/memory_pools/pool.py | + tensorrt_llm/runtime/memory_pools/pools_kv_cache_manager.py | + tensorrt_llm/runtime/model_runner.py | + tensorrt_llm/runtime/model_runner_cpp.py | + tensorrt_llm/runtime/multimodal_model_runner.py | + tensorrt_llm/runtime/processor_wrapper/__init__.py | + tensorrt_llm/runtime/processor_wrapper/mllama_processor_wrapper.py | + tensorrt_llm/runtime/processor_wrapper/processor_wrapper.py | + tensorrt_llm/runtime/redrafter_utils.py | + tensorrt_llm/runtime/session.py | tensorrt_llm/scaffolding/__init__.py | tensorrt_llm/scaffolding/benchmark.py | tensorrt_llm/scaffolding/contrib/AsyncGeneration/stream_generation.py | @@ -1249,7 +2200,12 @@ legacy-files: &legacy_files | tensorrt_llm/tokenizer/tokenizer.py | tensorrt_llm/tools/__init__.py | tensorrt_llm/tools/importlib_utils.py | + tensorrt_llm/tools/multimodal_builder.py | + tensorrt_llm/tools/onnx_utils.py | tensorrt_llm/tools/plugin_gen/__init__.py | + tensorrt_llm/tools/plugin_gen/core.py | + tensorrt_llm/tools/plugin_gen/plugin_gen.py | + tensorrt_llm/tools/plugin_gen/shape_infer.py | tensorrt_llm/tools/ppl.py | tensorrt_llm/tools/profiler/nsys_profile_tools/gputrc2graph.py | tensorrt_llm/version.py | @@ -1258,7 +2214,9 @@ legacy-files: &legacy_files | tests/integration/defs/accuracy/accuracy_core.py | tests/integration/defs/accuracy/scripts/collect_evaluated_accuracies.py | tests/integration/defs/accuracy/scripts/compute_theta_and_thresholds.py | + tests/integration/defs/accuracy/test_cli_flow.py | tests/integration/defs/accuracy/test_disaggregated_serving.py | + tests/integration/defs/accuracy/test_llm_api.py | tests/integration/defs/accuracy/test_llm_api_autodeploy.py | tests/integration/defs/accuracy/test_llm_api_pytorch.py | tests/integration/defs/accuracy/test_llm_api_pytorch_ray.py | @@ -1267,29 +2225,63 @@ legacy-files: &legacy_files | tests/integration/defs/conftest.py | tests/integration/defs/cpp/conftest.py | tests/integration/defs/cpp/cpp_common.py | + tests/integration/defs/cpp/test_e2e.py | tests/integration/defs/cpp/test_multi_gpu.py | tests/integration/defs/cpp/test_unit_tests.py | + tests/integration/defs/deterministic/mixtral_deterministic.py | + tests/integration/defs/deterministic/test_mixtral_deterministic.py | tests/integration/defs/disaggregated/test_auto_scaling.py | tests/integration/defs/disaggregated/test_disaggregated.py | tests/integration/defs/disaggregated/test_disaggregated_etcd.py | tests/integration/defs/disaggregated/test_disaggregated_single_gpu.py | tests/integration/defs/disaggregated/test_workers.py | + tests/integration/defs/examples/run_llm_fp8_quant_llama_70b.py | tests/integration/defs/examples/run_llm_quickstart_atexit.py | tests/integration/defs/examples/serve/test_serve.py | tests/integration/defs/examples/serve/test_serve_negative.py | tests/integration/defs/examples/test_ad_guided_decoding.py | + tests/integration/defs/examples/test_bert.py | + tests/integration/defs/examples/test_bindings.py | + tests/integration/defs/examples/test_chatglm.py | + tests/integration/defs/examples/test_commandr.py | + tests/integration/defs/examples/test_draft_target_model.py | + tests/integration/defs/examples/test_eagle.py | + tests/integration/defs/examples/test_enc_dec.py | + tests/integration/defs/examples/test_exaone.py | + tests/integration/defs/examples/test_gemma.py | tests/integration/defs/examples/test_gpt.py | + tests/integration/defs/examples/test_gptj.py | + tests/integration/defs/examples/test_granite.py | + tests/integration/defs/examples/test_internlm.py | + tests/integration/defs/examples/test_llama.py | tests/integration/defs/examples/test_llm_api_with_mpi.py | + tests/integration/defs/examples/test_mamba.py | + tests/integration/defs/examples/test_medusa.py | + tests/integration/defs/examples/test_mistral.py | + tests/integration/defs/examples/test_mixtral.py | + tests/integration/defs/examples/test_multimodal.py | + tests/integration/defs/examples/test_nemotron.py | + tests/integration/defs/examples/test_nemotron_nas.py | + tests/integration/defs/examples/test_ngram.py | + tests/integration/defs/examples/test_openai.py | tests/integration/defs/examples/test_phi.py | + tests/integration/defs/examples/test_qwen.py | + tests/integration/defs/examples/test_qwen2audio.py | + tests/integration/defs/examples/test_qwenvl.py | tests/integration/defs/examples/test_ray.py | + tests/integration/defs/examples/test_recurrentgemma.py | + tests/integration/defs/examples/test_redrafter.py | + tests/integration/defs/examples/test_whisper.py | tests/integration/defs/llmapi/__init__.py | tests/integration/defs/llmapi/_run_llmapi_llm.py | tests/integration/defs/llmapi/test_llm_api_connector.py | tests/integration/defs/llmapi/test_llm_api_qa.py | + tests/integration/defs/llmapi/test_llm_e2e.py | tests/integration/defs/llmapi/test_llm_examples.py | tests/integration/defs/local_venv.py | tests/integration/defs/perf/__init__.py | tests/integration/defs/perf/allowed_configs.py | + tests/integration/defs/perf/build.py | tests/integration/defs/perf/create_perf_comparison_report.py | tests/integration/defs/perf/data.py | tests/integration/defs/perf/data_export.py | @@ -1310,21 +2302,38 @@ legacy-files: &legacy_files | tests/integration/defs/test_fmha.py | tests/integration/defs/test_list_parser.py | tests/integration/defs/test_list_validation.py | + tests/integration/defs/test_mlpf_results.py | tests/integration/defs/test_sanity.py | tests/integration/defs/test_unittests.py | tests/integration/defs/triton_server/__init__.py | + tests/integration/defs/triton_server/build_engines.py | tests/integration/defs/triton_server/common.py | tests/integration/defs/triton_server/conftest.py | + tests/integration/defs/triton_server/local_venv.py | + tests/integration/defs/triton_server/rcca/bug_4323566/inflight_batcher_llm_client_with_end_id.py | + tests/integration/defs/triton_server/runner_interface.py | tests/integration/defs/triton_server/test_list_parser.py | + tests/integration/defs/triton_server/test_triton.py | + tests/integration/defs/triton_server/test_triton_llm.py | + tests/integration/defs/triton_server/test_triton_memleak.py | + tests/integration/defs/triton_server/test_triton_multi_node.py | + tests/integration/defs/triton_server/test_triton_rcca.py | tests/integration/defs/triton_server/trt_test_alternative.py | tests/integration/defs/trt_test_alternative.py | tests/integration/defs/utils/__init__.py | tests/integration/defs/utils/periodic_junit.py | tests/integration/defs/utils/timeout_manager.py | tests/microbenchmarks/all_reduce.py | + tests/microbenchmarks/build_time_benchmark.py | + tests/microbenchmarks/build_time_dashboard.py | tests/scripts/allreduce_perf/allreduce_heuristic_code_gen.py | tests/scripts/allreduce_perf/allreduce_perf_viz.py | tests/scripts/iteration_log_parser.py | + tests/scripts/perf-sanity/parse_benchmark_results.py | + tests/scripts/perf-sanity/run_benchmark_serve.py | + tests/unittest/_torch/attention/sparse/test_dsa_indexer.py | + tests/unittest/_torch/attention/sparse/test_flash_mla.py | + tests/unittest/_torch/attention/sparse/test_rocketkv.py | tests/unittest/_torch/attention/sparse/test_sparse_mla_forward.py | tests/unittest/_torch/attention/test_attention.py | tests/unittest/_torch/attention/test_attention_mla.py | @@ -1345,6 +2354,7 @@ legacy-files: &legacy_files | tests/unittest/_torch/misc/test_virtual_memory.py | tests/unittest/_torch/modeling/test_modeling_bert.py | tests/unittest/_torch/modeling/test_modeling_clip.py | + tests/unittest/_torch/modeling/test_modeling_exaone4.py | tests/unittest/_torch/modeling/test_modeling_gemma3.py | tests/unittest/_torch/modeling/test_modeling_gpt_oss.py | tests/unittest/_torch/modeling/test_modeling_llama.py | @@ -1368,6 +2378,8 @@ legacy-files: &legacy_files | tests/unittest/_torch/modules/test_moe_routing.py | tests/unittest/_torch/modules/test_rotary_embedding.py | tests/unittest/_torch/modules/test_triton_linear.py | + tests/unittest/_torch/modules/tests_lora_modules/test_lora_attention_pytorch_flow_vs_trt.py | + tests/unittest/_torch/modules/tests_lora_modules/test_lora_plugin_vs_lora_op.py | tests/unittest/_torch/multi_gpu/test_allreduce.py | tests/unittest/_torch/multi_gpu/test_alltoall.py | tests/unittest/_torch/multi_gpu/test_ar_residual_norm.py | @@ -1394,10 +2406,24 @@ legacy-files: &legacy_files | tests/unittest/_torch/sampler/test_beam_search.py | tests/unittest/_torch/sampler/test_best_of_n.py | tests/unittest/_torch/sampler/test_trtllm_sampler.py | + tests/unittest/_torch/speculative/test_draft_target.py | + tests/unittest/_torch/speculative/test_draft_token_tree_sampling.py | + tests/unittest/_torch/speculative/test_draft_token_tree_verification.py | + tests/unittest/_torch/speculative/test_dynamic_spec_decode.py | tests/unittest/_torch/speculative/test_eagle3.py | + tests/unittest/_torch/speculative/test_kv_cache_reuse.py | + tests/unittest/_torch/speculative/test_mtp.py | + tests/unittest/_torch/speculative/test_ngram.py | + tests/unittest/_torch/speculative/test_save_state.py | + tests/unittest/_torch/speculative/test_spec_gate.py | + tests/unittest/_torch/speculative/test_torch_rejection_sampling.py | + tests/unittest/_torch/speculative/test_user_provided.py | tests/unittest/_torch/test_connector.py | tests/unittest/_torch/test_torch_multi_arange.py | tests/unittest/_torch/thop/parallel/deep_gemm_tests.py | + tests/unittest/_torch/thop/parallel/test_causal_conv1d_op.py | + tests/unittest/_torch/thop/parallel/test_cublas_mm.py | + tests/unittest/_torch/thop/parallel/test_custom_ops.py | tests/unittest/_torch/thop/parallel/test_dsv3_fused_a_gemm.py | tests/unittest/_torch/thop/parallel/test_dsv3_router_gemm.py | tests/unittest/_torch/thop/parallel/test_finegrained_mixed_dtype_gemm.py | @@ -1411,6 +2437,11 @@ legacy-files: &legacy_files | tests/unittest/_torch/thop/parallel/test_fp8_per_tensor_scale_tllmg_gemm.py | tests/unittest/_torch/thop/parallel/test_fp8_quantize.py | tests/unittest/_torch/thop/parallel/test_fp8_rowwise_linear.py | + tests/unittest/_torch/thop/parallel/test_fused_qk_norm_rope.py | + tests/unittest/_torch/thop/parallel/test_logits_bitmask_op.py | + tests/unittest/_torch/thop/parallel/test_mamba2_chunk_ss_update.py | + tests/unittest/_torch/thop/parallel/test_mamba_conv1d_op.py | + tests/unittest/_torch/thop/parallel/test_noaux_tc.py | tests/unittest/_torch/thop/parallel/test_scaled_mm.py | tests/unittest/_torch/thop/parallel/test_selective_scan_op.py | tests/unittest/_torch/thop/parallel/test_tinygemm2.py | @@ -1424,6 +2455,7 @@ legacy-files: &legacy_files | tests/unittest/_torch/thop/serial/test_moe_alltoall.py | tests/unittest/api_stability/api_stability_core.py | tests/unittest/api_stability/test_llm_api.py | + tests/unittest/bindings/binding_test_utils.py | tests/unittest/bindings/test_bindings_moe.py | tests/unittest/bindings/test_bindings_ut.py | tests/unittest/bindings/test_executor_bindings.py | @@ -1454,10 +2486,12 @@ legacy-files: &legacy_files | tests/unittest/llmapi/apps/_test_openai_chat_harmony.py | tests/unittest/llmapi/apps/_test_openai_chat_multimodal.py | tests/unittest/llmapi/apps/_test_openai_completions.py | + tests/unittest/llmapi/apps/_test_openai_consistent_chat.py | tests/unittest/llmapi/apps/_test_openai_lora.py | tests/unittest/llmapi/apps/_test_openai_metrics.py | tests/unittest/llmapi/apps/_test_openai_misc.py | tests/unittest/llmapi/apps/_test_openai_mmencoder.py | + tests/unittest/llmapi/apps/_test_openai_multi_chat.py | tests/unittest/llmapi/apps/_test_openai_multi_gpu.py | tests/unittest/llmapi/apps/_test_openai_multi_nodes.py | tests/unittest/llmapi/apps/_test_openai_perf_metrics.py | @@ -1480,12 +2514,15 @@ legacy-files: &legacy_files | tests/unittest/llmapi/run_llm_exit.py | tests/unittest/llmapi/run_llm_with_postproc.py | tests/unittest/llmapi/test_additional_model_outputs.py | + tests/unittest/llmapi/test_build_cache.py | tests/unittest/llmapi/test_executor.py | tests/unittest/llmapi/test_gc_utils.py | tests/unittest/llmapi/test_llm.py | tests/unittest/llmapi/test_llm_args.py | tests/unittest/llmapi/test_llm_download.py | tests/unittest/llmapi/test_llm_kv_cache_events.py | + tests/unittest/llmapi/test_llm_models.py | + tests/unittest/llmapi/test_llm_multi_gpu.py | tests/unittest/llmapi/test_llm_multi_gpu_pytorch.py | tests/unittest/llmapi/test_llm_pytorch.py | tests/unittest/llmapi/test_llm_quant.py | @@ -1496,14 +2533,25 @@ legacy-files: &legacy_files | tests/unittest/llmapi/test_serialization.py | tests/unittest/llmapi/test_utils.py | tests/unittest/others/__init__.py | + tests/unittest/others/test_builder.py | tests/unittest/others/test_convert_spec_decoding_mask_to_packed_mask.py | + tests/unittest/others/test_debugging_api.py | tests/unittest/others/test_exception.py | tests/unittest/others/test_export.py | + tests/unittest/others/test_graph_rewriter.py | + tests/unittest/others/test_kv_cache_manager.py | tests/unittest/others/test_kv_cache_transceiver.py | tests/unittest/others/test_kv_cache_update.py | + tests/unittest/others/test_layer.py | + tests/unittest/others/test_leak.py | tests/unittest/others/test_mapping.py | + tests/unittest/others/test_model_dtype.py | + tests/unittest/others/test_module.py | tests/unittest/others/test_multimodal_registry.py | + tests/unittest/others/test_plugins.py | + tests/unittest/others/test_precision_control.py | tests/unittest/others/test_pretrained_config.py | + tests/unittest/others/test_session.py | tests/unittest/others/test_time_breakdown.py | tests/unittest/profile_utils.py | tests/unittest/scaffolding/__init__.py | @@ -1512,45 +2560,203 @@ legacy-files: &legacy_files | tests/unittest/scaffolding/test_scaffolding.py | tests/unittest/scaffolding/test_task_collection.py | tests/unittest/scaffolding/test_worker.py | + tests/unittest/test_model_runner_cpp.py | tests/unittest/test_pip_install.py | tests/unittest/tools/__init__.py | + tests/unittest/tools/plugin_gen/__init__.py | + tests/unittest/tools/plugin_gen/kernel_config.py | + tests/unittest/tools/plugin_gen/test_core.py | + tests/unittest/tools/plugin_gen/test_plugin_gen.py | + tests/unittest/tools/plugin_gen/test_shape_infer.py | tests/unittest/tools/test_prepare_dataset.py | tests/unittest/tools/test_test_to_stage_mapping.py | + tests/unittest/trt/__init__.py | + tests/unittest/trt/attention/test_bert_attention.py | + tests/unittest/trt/attention/test_gpt_attention.py | + tests/unittest/trt/attention/test_gpt_attention_IFB.py | + tests/unittest/trt/attention/test_gpt_attention_no_cache.py | + tests/unittest/trt/attention/test_sage_attention.py | + tests/unittest/trt/functional/__init__.py | + tests/unittest/trt/functional/test_alibi.py | + tests/unittest/trt/functional/test_allreduce_norm.py | + tests/unittest/trt/functional/test_allreduce_prepost_residual_norm.py | + tests/unittest/trt/functional/test_arange.py | + tests/unittest/trt/functional/test_argmax.py | + tests/unittest/trt/functional/test_assertion.py | + tests/unittest/trt/functional/test_avg_pool2d.py | + tests/unittest/trt/functional/test_cast.py | + tests/unittest/trt/functional/test_conv2d.py | + tests/unittest/trt/functional/test_conv3d.py | + tests/unittest/trt/functional/test_cos.py | + tests/unittest/trt/functional/test_cumsum.py | + tests/unittest/trt/functional/test_dora.py | + tests/unittest/trt/functional/test_einsum.py | + tests/unittest/trt/functional/test_embedding_single_gpu.py | + tests/unittest/trt/functional/test_exp.py | + tests/unittest/trt/functional/test_expand.py | + tests/unittest/trt/functional/test_flatten.py | + tests/unittest/trt/functional/test_flip.py | + tests/unittest/trt/functional/test_fp4_gemm.py | + tests/unittest/trt/functional/test_fp4_gemm_ootb.py | + tests/unittest/trt/functional/test_gather.py | + tests/unittest/trt/functional/test_gather_nd.py | + tests/unittest/trt/functional/test_geglu.py | + tests/unittest/trt/functional/test_gelu.py | + tests/unittest/trt/functional/test_gemm_swiglu.py | + tests/unittest/trt/functional/test_group_norm.py | + tests/unittest/trt/functional/test_identity.py | + tests/unittest/trt/functional/test_index_select.py | + tests/unittest/trt/functional/test_interpolate.py | + tests/unittest/trt/functional/test_logsoftmax.py | + tests/unittest/trt/functional/test_lora.py | + tests/unittest/trt/functional/test_low_latency_gemm.py | + tests/unittest/trt/functional/test_mamba_conv1d.py | + tests/unittest/trt/functional/test_masked_scatter.py | + tests/unittest/trt/functional/test_masked_select.py | + tests/unittest/trt/functional/test_matmul.py | + tests/unittest/trt/functional/test_meshgrid2d.py | + tests/unittest/trt/functional/test_moe.py | + tests/unittest/trt/functional/test_nccl.py | + tests/unittest/trt/functional/test_nonzero.py | + tests/unittest/trt/functional/test_outer.py | + tests/unittest/trt/functional/test_pad.py | + tests/unittest/trt/functional/test_permute.py | + tests/unittest/trt/functional/test_pp_reduce_scatter.py | + tests/unittest/trt/functional/test_quant.py | + tests/unittest/trt/functional/test_rearrange.py | + tests/unittest/trt/functional/test_repeat.py | + tests/unittest/trt/functional/test_repeat_interleave.py | + tests/unittest/trt/functional/test_rg_lru.py | + tests/unittest/trt/functional/test_sample.py | + tests/unittest/trt/functional/test_scatter.py | + tests/unittest/trt/functional/test_scatter_nd.py | + tests/unittest/trt/functional/test_select.py | + tests/unittest/trt/functional/test_selective_scan.py | + tests/unittest/trt/functional/test_sigmoid.py | + tests/unittest/trt/functional/test_silu.py | + tests/unittest/trt/functional/test_sin.py | + tests/unittest/trt/functional/test_slice.py | + tests/unittest/trt/functional/test_softplus.py | + tests/unittest/trt/functional/test_split.py | + tests/unittest/trt/functional/test_squeeze.py | + tests/unittest/trt/functional/test_swiglu.py | + tests/unittest/trt/functional/test_topk.py | + tests/unittest/trt/functional/test_transpose.py | + tests/unittest/trt/functional/test_unbind.py | + tests/unittest/trt/functional/test_unsqueeze.py | + tests/unittest/trt/functional/test_view.py | + tests/unittest/trt/functional/test_where.py | + tests/unittest/trt/model/__init__.py | + tests/unittest/trt/model/eagle/test_decode_draft_tokens_plugin.py | + tests/unittest/trt/model/eagle/test_prepare_drafter_inputs_plugin.py | + tests/unittest/trt/model/eagle/test_sample_accept_draft_tokens_plugin.py | + tests/unittest/trt/model/redrafter/test_beams2tree.py | + tests/unittest/trt/model/redrafter/test_draft_token.py | + tests/unittest/trt/model/redrafter/test_draft_token_indices.py | + tests/unittest/trt/model/redrafter/test_gather_beams.py | + tests/unittest/trt/model/redrafter/test_mask.py | + tests/unittest/trt/model/redrafter/test_packed_position_ids.py | + tests/unittest/trt/model/redrafter/test_prefix_match_indices.py | + tests/unittest/trt/model/redrafter/test_prepare_input.py | + tests/unittest/trt/model/redrafter/test_process_logits.py | + tests/unittest/trt/model/redrafter/test_top1.py | + tests/unittest/trt/model/redrafter/test_unpack_gen_data.py | + tests/unittest/trt/model/redrafter/test_validate.py | + tests/unittest/trt/model/test_gpt.py | + tests/unittest/trt/model/test_gpt_e2e.py | + tests/unittest/trt/model/test_llama.py | + tests/unittest/trt/model/test_mamba.py | + tests/unittest/trt/model/test_mistral.py | + tests/unittest/trt/model/test_nemotron_nas.py | + tests/unittest/trt/model/test_phi.py | + tests/unittest/trt/model/test_unet.py | + tests/unittest/trt/model_api/test_model_api_multi_gpu.py | + tests/unittest/trt/model_api/test_model_level_api.py | + tests/unittest/trt/model_api/test_model_quantization.py | + tests/unittest/trt/python_plugin/plugin_wrapper_utils.py | + tests/unittest/trt/python_plugin/test_plugin_wrapper.py | + tests/unittest/trt/quantization/__init__.py | + tests/unittest/trt/quantization/_utils.py | + tests/unittest/trt/quantization/test_fp8_quantization.py | + tests/unittest/trt/quantization/test_fp8_rowwise_gemm.py | + tests/unittest/trt/quantization/test_functional.py | + tests/unittest/trt/quantization/test_mode.py | + tests/unittest/trt/quantization/test_moe_weight_only_quant_matmul.py | + tests/unittest/trt/quantization/test_qserve_gemm.py | + tests/unittest/trt/quantization/test_quant.py | + tests/unittest/trt/quantization/test_quant_layer.py | + tests/unittest/trt/quantization/test_smooth_quant_gemm.py | + tests/unittest/trt/quantization/test_smooth_quant_layer_norm.py | + tests/unittest/trt/quantization/test_smooth_quant_rms_norm.py | + tests/unittest/trt/quantization/test_weight_only_groupwise_quant_matmul.py | + tests/unittest/trt/quantization/test_weight_only_quant_matmul.py | tests/unittest/utils/__init__.py | tests/unittest/utils/cpp_paths.py | tests/unittest/utils/llm_data.py | tests/unittest/utils/runtime_defaults.py | + tests/unittest/utils/test_medusa_utils.py | tests/unittest/utils/test_prebuilt_whl_cpp_extensions.py | tests/unittest/utils/test_util.py | tests/unittest/utils/torch_ref.py | tests/unittest/utils/util.py | + triton_backend/all_models/disaggregated_serving/disaggregated_serving_bls/1/model.py | + triton_backend/all_models/gpt/postprocessing/1/model.py | + triton_backend/all_models/gpt/preprocessing/1/model.py | + triton_backend/all_models/gpt/tensorrt_llm/1/model.py | + triton_backend/all_models/inflight_batcher_llm/postprocessing/1/model.py | + triton_backend/all_models/inflight_batcher_llm/preprocessing/1/model.py | + triton_backend/all_models/inflight_batcher_llm/tensorrt_llm/1/model.py | + triton_backend/all_models/inflight_batcher_llm/tensorrt_llm_bls/1/lib/decode.py | + triton_backend/all_models/inflight_batcher_llm/tensorrt_llm_bls/1/lib/triton_decoder.py | + triton_backend/all_models/inflight_batcher_llm/tensorrt_llm_bls/1/model.py | triton_backend/all_models/llmapi/tensorrt_llm/1/helpers.py | triton_backend/all_models/llmapi/tensorrt_llm/1/model.py | + triton_backend/all_models/multimodal/multimodal_encoders/1/model.py | + triton_backend/all_models/multimodal/multimodal_encoders/1/multimodal_utils.py | + triton_backend/all_models/tests/test_decode.py | triton_backend/all_models/tests/test_llmapi_python_backend.py | + triton_backend/all_models/tests/test_multi_image_preprocess.py | + triton_backend/all_models/tests/test_multimodal_encoders.py | + triton_backend/all_models/tests/test_python_backend.py | + triton_backend/all_models/tests/test_triton_decoder.py | + triton_backend/all_models/whisper/whisper_bls/1/fbank.py | + triton_backend/all_models/whisper/whisper_bls/1/model.py | + triton_backend/all_models/whisper/whisper_bls/1/tokenizer.py | + triton_backend/ci/L0_backend_trtllm/base_metrics_verification_tests.py | + triton_backend/ci/L0_backend_trtllm/custom_metrics_verification_tests.py | + triton_backend/inflight_batcher_llm/client/__init__.py | + triton_backend/inflight_batcher_llm/client/e2e_grpc_speculative_decoding_client.py | + triton_backend/inflight_batcher_llm/client/end_to_end_grpc_client.py | + triton_backend/inflight_batcher_llm/client/inflight_batcher_llm_client.py | triton_backend/scripts/launch_triton_server.py | triton_backend/tools/__init__.py | triton_backend/tools/fill_template.py | + triton_backend/tools/gpt/benchmark_core_model.py | + triton_backend/tools/gpt/client.py | + triton_backend/tools/gpt/client_async.py | + triton_backend/tools/gpt/end_to_end_test.py | + triton_backend/tools/gpt/gen_input_data.py | triton_backend/tools/inflight_batcher_llm/benchmark_core_model.py | triton_backend/tools/inflight_batcher_llm/end_to_end_test.py | triton_backend/tools/inflight_batcher_llm/speculative_decoding_test.py | triton_backend/tools/inflight_batcher_llm/test_max_queue_size.py | triton_backend/tools/llmapi_client.py | + triton_backend/tools/multimodal/client.py | triton_backend/tools/tests/__init__.py | triton_backend/tools/tests/test_fill_template.py | triton_backend/tools/tests/test_llmapi_cancel.py | triton_backend/tools/utils/__init__.py | - triton_backend/tools/utils/utils.py + triton_backend/tools/utils/utils.py | + triton_backend/tools/whisper/client.py )$ # <<<< END AUTO-GENERATED >>>> # Files to be subjected to static analysis static-analysis-files: &static_analysis_files | (?x)^( - tensorrt_llm/_torch/pyexecutor/sampler/sampler.py | - tensorrt_llm/_torch/pyexecutor/sampler/sampling_utils.py | - tensorrt_llm/_torch/pyexecutor/sampler/ops/interface.py | - tensorrt_llm/_torch/pyexecutor/sampler/ops/flashinfer.py | - tensorrt_llm/_torch/pyexecutor/sampler/ops/vanilla.py | + tensorrt_llm/_torch/pyexecutor/sampler.py | + tensorrt_llm/_torch/pyexecutor/sampling_utils.py | + tensorrt_llm/_torch/pyexecutor/sampling_utils_flashinfer.py | tests/unittest/_torch/sampler/test_torch_sampler.py | tests/unittest/_torch/sampler/test_beam_search.py | tests/unittest/_torch/sampler/test_beam_search_util.py | diff --git a/3rdparty/MSA b/3rdparty/MSA deleted file mode 160000 index e2ebe7656649..000000000000 --- a/3rdparty/MSA +++ /dev/null @@ -1 +0,0 @@ -Subproject commit e2ebe7656649f619af0ad1d457b534283034655e diff --git a/AGENTS.md b/AGENTS.md index cf6c717d0928..3d2241e5f0ad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,7 @@ # AGENTS.md TensorRT-LLM: open-source library for optimized LLM inference on NVIDIA GPUs. -Python and C++ codebase with PyTorch and AutoDeploy execution paths. +Python and C++ codebase supporting TensorRT engine-based and PyTorch-based execution paths. > If a `CLAUDE.local.md` file exists alongside this file, read and respect it — it contains developer-specific overrides that supplement this shared guidance. @@ -13,8 +13,6 @@ Python and C++ codebase with PyTorch and AutoDeploy execution paths. - `git commit -s` (DCO sign-off required). Never attribute AI tools in sign-off line. Always rely on `git` to do the sign off instead of directly adding sign off in commit message. - Do not add co-authors to the git commit message unless explicitly instructed to do so by the user. - `pre-commit` hooks run on commit — if files are modified by hooks, re-stage and commit again -- LLM args or nested-config changes must run `python3 scripts/generate_llm_args_golden_manifest.py` and commit - `tensorrt_llm/usage/llm_args_golden_manifest.json`; new fields require telemetry/privacy CODEOWNER approval - PR title format: `[JIRA/NVBUG/None][type] description` (e.g., `[TRTLLM-5516][perf] optimize cuda graph padding`) - Set `LLM_MODELS_ROOT` env var when running tests that need model weights @@ -55,16 +53,17 @@ See [architecture diagram](.github/tava_architecture_diagram.md) for the full Me |---------|--------|-------------|----------| | **PyTorch** | Default | `TorchLlmArgs` | `_torch/pyexecutor/` → `PyExecutor` → PyTorch Engine | | **AutoDeploy** | Beta | `_torch/auto_deploy/` shim | `_torch/auto_deploy/shim/ad_executor.py` → adapts `PyExecutor` → graph transforms + torch.export | +| **TensorRT** | Legacy | `TrtLlmArgs` | `builder.py` → `trtllm.Executor` → TensorRT Engine | ### Shared C++ Core (via Nanobind) -Both backends share these C++ components: +Both PyTorch and TensorRT backends share these C++ components: - **Scheduling pipeline**: Scheduler → BatchManager (in-flight batching) → KV Cache Manager - **Decoding pipeline**: Decoder (token generation orchestration) → Sampling ### Request Flow ```text -HuggingFace Model → LLM API → Executor (PyTorch/AutoDeploy) +HuggingFace Model → LLM API → Executor (PyTorch/AutoDeploy/TensorRT) → Scheduler → Model Forward → Decoder → Sampling → Generated Tokens ``` @@ -83,7 +82,7 @@ HuggingFace Model → LLM API → Executor (PyTorch/AutoDeploy) | `tensorrt_llm/models/modeling_utils.py` | Base classes for all models (`PretrainedConfig`, `PretrainedModel`) | | `tensorrt_llm/executor/executor.py` | Execution abstraction (`GenerationExecutor`) | | `tensorrt_llm/models/automodel.py` | Auto-discovery and model registry | -| `tensorrt_llm/_torch/models/` | PyTorch backend model implementations (distinct from the top-level `models/` package) | +| `tensorrt_llm/_torch/models/` | PyTorch backend model implementations (distinct from `models/` used by TensorRT backend) | | `tensorrt_llm/_torch/modules/ATTENTION_DEVELOPER_GUIDE.md` | Attention, MLA, backend families, sparse backends, metadata contracts, and KV-cache behavior - **read before modifying `tensorrt_llm/_torch/modules/attention.py`, `tensorrt_llm/_torch/modules/mla.py`, or `tensorrt_llm/_torch/attention_backend/`** | | `tensorrt_llm/_torch/modules/fused_moe/MOE_DEVELOPER_GUIDE.md` | MoE architecture, backends, communication, development patterns — **read before modifying MoE code** | | `CODING_GUIDELINES.md` | C++ and Python coding standards (referenced throughout, must read before contributing) | @@ -92,7 +91,7 @@ HuggingFace Model → LLM API → Executor (PyTorch/AutoDeploy) | Pattern | Key Points | |---------|------------| -| **Config hierarchy** | `BaseLlmArgs` → `TorchLlmArgs`, model-specific defaults override generics, Pydantic validation | +| **Config hierarchy** | `BaseLlmArgs` → `TrtLlmArgs` / `TorchLlmArgs`, model-specific defaults override generics, Pydantic validation | | **Model architecture** | Each model: `Config` (inherits `PretrainedConfig`) + `ForCausalLM` (inherits `PretrainedModel`) | | **Model defaults** | Architecture-specific overrides in `llm_utils.py` (attention kernels, quant, spec decoding, cache) | | **Attention backends** | `TorchLlmArgs.attn_backend` selects kernel: `TRTLLM` (default), `FlashInfer`, `FlashAttention` | @@ -124,6 +123,7 @@ Key files: - **Avoid broad exception handling** — catch specific exceptions, not bare `except:` (see `CODING_GUIDELINES.md`). - **One concern per PR** — avoid scope creep. If a PR touches unrelated areas, split it. - **User-facing configuration classes** - when editing or defining any user-facing configuration classes (particularly `BaseLlmArgs` or any class used in its fields), you **MUST** follow the Pydantic guidelines in `CODING_GUIDELINES.md`. +- **TensorRT backend is legacy** — `TrtLlmArgs` / `backend="tensorrt"` and all exclusive tooling (`trtllm-build`, `trtllm-refit`, `convert_checkpoint.py`, `ModelRunner*`) are legacy. Bug fixes OK; new features target PyTorch or AutoDeploy. ## Development Workflow @@ -145,9 +145,8 @@ Key files: The `gh` CLI uses `~/.config/gh` by default for authentication. Different GitHub hosts or forks may require a different config directory. **Before running any `gh` command** (e.g., `gh pr create`, `gh api`, `gh pr comment`): 1. Check if the user has specified a custom `GH_CONFIG_DIR` (e.g., in `CLAUDE.local.md` or environment). If so, use it. -2. If not explicitly set, default to `~/.config/gh`; do not ask for confirmation. +2. If not explicitly set, **ask the user** whether the default `~/.config/gh` is correct or if a different directory should be used. This is especially relevant when the PR target is a fork (e.g., `nv-auto-deploy/TensorRT-LLM`) rather than `NVIDIA/TensorRT-LLM`. 3. Prefix all `gh` commands with the resolved config dir: `GH_CONFIG_DIR=<path> gh ...` -4. If the command fails due to missing authentication or the wrong GitHub host/account, report the failure and ask for the correct `GH_CONFIG_DIR`. ## CI / Testing diff --git a/ATTRIBUTIONS-Python.md b/ATTRIBUTIONS-Python.md index 7e74b846a85f..e43a78cafda0 100644 --- a/ATTRIBUTIONS-Python.md +++ b/ATTRIBUTIONS-Python.md @@ -5261,7 +5261,7 @@ For more information, please refer to <http://unlicense.org> - `Tracker`: https://github.com/tox-dev/py-filelock/issues -## flashinfer-python (0.6.15) +## flashinfer-python (0.6.14) ### Licenses License: `Apache-2.0` diff --git a/CODING_GUIDELINES.md b/CODING_GUIDELINES.md index 8e4d0b5be7bf..33031ece5247 100644 --- a/CODING_GUIDELINES.md +++ b/CODING_GUIDELINES.md @@ -551,9 +551,6 @@ When defining any user-facing configuration classes (particularly `LlmArgs` or a - Prefer `PositiveInt`, `NonNegativeInt`, `NonNegativeFloat`, `PositiveFloat`, `Field(gt=0)`, `Field(ge=0)`, etc. for numeric constraints instead of defining custom validators - Use `Field(min_length=1)` to enforce minimum length of a list -- After changing LLM args or nested configs, run `python3 scripts/generate_llm_args_golden_manifest.py` and commit - `tensorrt_llm/usage/llm_args_golden_manifest.json`; new fields require telemetry/privacy CODEOWNER approval. - **Validation:** - Use `@field_validator` and `@model_validator` instead of manual `validate()` or `is_valid()` methods - Raise `ValueError` instead of using assertions diff --git a/LICENSE b/LICENSE index ed5ca3b261ba..8ba867f30567 100644 --- a/LICENSE +++ b/LICENSE @@ -19,13 +19,6 @@ Original Source: https://github.com/Dao-AILab/causal-conv1d Copyright (c) 2024, Tri Dao. Licensed under the BSD 3-Clause License --------------------------------------------------------------------------------- -CUTLASS --------------------------------------------------------------------------------- -Original Source: https://github.com/NVIDIA/cutlass -Copyright (c) 2017 - 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -Licensed under the BSD 3-Clause License - -------------------------------------------------------------------------------- flash-attention -------------------------------------------------------------------------------- @@ -41,14 +34,6 @@ Original Source: https://github.com/fla-org/flash-linear-attention Copyright (c) 2023-2025 Songlin Yang Licensed under the MIT License --------------------------------------------------------------------------------- -FlashInfer --------------------------------------------------------------------------------- -Original Source: https://github.com/flashinfer-ai/flashinfer -Copyright 2025-2026 NVIDIA -Copyright 2023-2026 FlashInfer community (https://flashinfer.ai/) -Licensed under the Apache License 2.0 - -------------------------------------------------------------------------------- InstructEval -------------------------------------------------------------------------------- @@ -74,14 +59,6 @@ Original Source: https://github.com/state-spaces/mamba Copyright 2023 Tri Dao, Albert Gu Licensed under the Apache License 2.0 --------------------------------------------------------------------------------- -MSA (MiniMax Sparse Attention) --------------------------------------------------------------------------------- -Original Source: https://github.com/MiniMax-AI/MSA -Copyright (c) 2026 MiniMax -Licensed under the MIT License - - -------------------------------------------------------------------------------- Quack -------------------------------------------------------------------------------- diff --git a/README.md b/README.md index e1324129fd92..943083d6c4c6 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ TensorRT LLM [![python](https://img.shields.io/badge/python-3.10-green)](https://www.python.org/downloads/release/python-31012/) [![cuda](https://img.shields.io/badge/cuda-13.2.1-green)](https://developer.nvidia.com/cuda-downloads) [![torch](https://img.shields.io/badge/torch-2.11.0-green)](https://pytorch.org) -[![version](https://img.shields.io/badge/release-1.3.0rc23-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py) +[![version](https://img.shields.io/badge/release-1.3.0rc21-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py) [![license](https://img.shields.io/badge/license-Apache%202-blue)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/LICENSE) [Architecture](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)   |   [Performance](https://nvidia.github.io/TensorRT-LLM/developer-guide/perf-overview.html)   |   [Examples](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html)   |   [Documentation](https://nvidia.github.io/TensorRT-LLM/)   |   [Roadmap](https://github.com/NVIDIA/TensorRT-LLM/issues?q=is%3Aissue%20state%3Aopen%20label%3Aroadmap) @@ -22,9 +22,6 @@ TensorRT LLM <!-- Use github markdown link to link for the latest blog since the doc build has not happened yet. When the doc build is updated, it should be updated to the webpage link. --> -* [07/17] DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM -✨ [➡️ link](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog26_DeepSeek_V4_on_NVIDIA_Blackwell_Model_Specific_and_Agentic_Workload_Optimizations_in_TensorRT-LLM.md) - * [07/01] Scaling Video Generation Across NVL72 Rack with TensorRT-LLM ✨ [➡️ link](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md) diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 000000000000..5d89f412ac5c --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,10 @@ +# TensorRT-LLM Benchmarks + +## Overview + +There are currently two workflows to benchmark TensorRT-LLM: +* [`trtllm-bench`](../docs/source/developer-guide/perf-benchmarking.md) + - `trtllm-bench` is native to TensorRT-LLM and is a Python benchmarker for reproducing and testing the performance of TensorRT-LLM. + - _NOTE_: This benchmarking suite is a current work in progress and is prone to large changes. +* [C++ benchmarks](./cpp) + - The recommended workflow that uses TensorRT-LLM C++ API and can take advantage of the latest features of TensorRT-LLM. diff --git a/benchmarks/cpp/CMakeLists.txt b/benchmarks/cpp/CMakeLists.txt new file mode 100644 index 000000000000..cb5ef1ee928b --- /dev/null +++ b/benchmarks/cpp/CMakeLists.txt @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +include_directories(${PROJECT_SOURCE_DIR}/include) + +set(TOP_LEVEL_DIR "${PROJECT_SOURCE_DIR}/..") + +add_custom_target(benchmarks) + +if(NOT TARGET cxxopts::cxxopts) + add_subdirectory(${CMAKE_BINARY_DIR}/_deps/cxxopts-src + ${CMAKE_CURRENT_BINARY_DIR}/cxxopts) +endif() + +function(add_benchmark test_name test_src) + add_executable(${test_name} ${test_src} utils/utils.cpp) + + target_link_libraries( + ${test_name} PUBLIC ${SHARED_TARGET} nvinfer_plugin_tensorrt_llm + cxxopts::cxxopts) + + target_compile_features(${test_name} PRIVATE cxx_std_17) + target_compile_definitions(${test_name} + PUBLIC TOP_LEVEL_DIR="${TOP_LEVEL_DIR}") + add_dependencies(benchmarks ${test_name}) +endfunction() + +add_benchmark(bertBenchmark bertBenchmark.cpp) +add_benchmark(gptManagerBenchmark gptManagerBenchmark.cpp) +add_benchmark(disaggServerBenchmark disaggServerBenchmark.cpp) diff --git a/benchmarks/cpp/README.md b/benchmarks/cpp/README.md new file mode 100644 index 000000000000..ae3287faf06c --- /dev/null +++ b/benchmarks/cpp/README.md @@ -0,0 +1,367 @@ +# Benchmark C++ Runtime + +This document explains how to benchmark the models supported by TensorRT-LLM on a single GPU, a single node with +multiple GPUs or multiple nodes with multiple GPUs using the C++ runtime. + +## Usage + +### 1. Build TensorRT-LLM and benchmarking source code + +Please follow the [`installation document`](../../README.md#installation) to build TensorRT-LLM. + +Note that the benchmarking source code for C++ runtime is not built by default, you can use the argument `--benchmarks` in [`build_wheel.py`](source:scripts/build_wheel.py) to build the corresponding executable. + +### 2. Launch C++ benchmarking (Inflight/V1 batching) + +#### Prepare dataset + +Run a preprocessing script to prepare/generate dataset into a json that `gptManagerBenchmark` can consume later. The processed output json has *input tokens length, input token ids and output tokens length*. + +For `tokenizer`, specifying the path to the local tokenizer that have already been downloaded, or simply the name of the tokenizer from HuggingFace like `meta-llama/Llama-2-7b` will both work. The tokenizer will be downloaded automatically for the latter case. + +This tool can be used in 3 different modes of traffic generation: `dataset`, `token-norm-dist` and `token-unif-dist`. + +##### 1 – Dataset + +The tool will tokenize the words and instruct the model to generate a specified number of output tokens for a request. + +``` +python3 prepare_dataset.py \ + --tokenizer <path/to/tokenizer> \ + --output preprocessed_dataset.json + dataset + --dataset-name <name of the dataset> \ + --dataset-split <split of the dataset to use> \ + --dataset-input-key <dataset dictionary key for input> \ + --dataset-prompt-key <dataset dictionary key for prompt> \ + --dataset-output-key <dataset dictionary key for output> \ + [--num-requests 100] \ + [--max-input-len 1000] \ + [--output-len-dist 100,10] +``` + +For datasets that don't have prompt key, set --dataset-prompt instead. +Take [cnn_dailymail dataset](https://huggingface.co/datasets/abisee/cnn_dailymail) for example: +``` +python3 prepare_dataset.py \ + --tokenizer <path/to/tokenizer> \ + --output cnn_dailymail.json + dataset + --dataset-name cnn_dailymail \ + --dataset-split validation \ + --dataset-config-name 3.0.0 \ + --dataset-input-key article \ + --dataset-prompt "Summarize the following article:" \ + --dataset-output-key "highlights" \ + [--num-requests 100] \ + [--max-input-len 1000] \ + [--output-len-dist 100,10] +``` + +##### 2 – Normal token length distribution + +This mode allows the user to generate normally distributed token lengths with a mean and std deviation specified. +For example, setting `mean=100` and `stdev=10` would generate requests where 95.4% of values are in <80,120> range following the normal probability distribution. Setting `stdev=0` will generate all requests with the same mean number of tokens. + +``` +python prepare_dataset.py \ + --output token-norm-dist.json \ + --tokenizer <path/to/tokenizer> \ + token-norm-dist \ + --num-requests 100 \ + --input-mean 100 --input-stdev 10 \ + --output-mean 15 --output-stdev 0 +``` + +##### 2 – Uniform token length distribution + +This mode allows the user to generate uniformly distributed token lengths with min and max lengths specified. +For example, setting `min=50` and `max=100` would generate requests where lengths are in the range `[50, 100]` following the uniform probability distribution. Setting `min=x` and `max=x` will generate all requests with the same mean number of tokens `x`. + +``` +python prepare_dataset.py \ + --output token-norm-dist.json \ + --tokenizer <path/to/tokenizer> \ + token-unif-dist \ + --num-requests 100 \ + --input-min 50 --input-max 100 \ + --output-min 10 --output-max 15 +``` + + +#### Prepare TensorRT-LLM engines + +Before you launch C++ benchmarking, please make sure that you have already built engine(s) using `trtllm-build` command. For more details on building engine(s), please refer to the [Quick Start Guide](../../docs/source/quick-start-guide.md). + +#### Launch benchmarking + +For detailed usage, you can do the following +``` +cd cpp/build + +# You can directly execute the binary for help information +./benchmarks/gptManagerBenchmark --help +``` + +`gptManagerBenchmark` now supports decoder-only models and encoder-decoder models. + +1. Decoder-only Models + + To benchmark decoder-only models, pass in the engine path with `--engine_dir` as executable input argument. + + Take GPT-350M as an example for 2-GPU inflight batching + ``` + mpirun -n 2 ./benchmarks/gptManagerBenchmark \ + --engine_dir ../../examples/models/core/gpt/trt_engine/gpt2-ib/fp16/2-gpu/ \ + --request_rate 10 \ + --dataset ../../benchmarks/cpp/preprocessed_dataset.json \ + --max_num_samples 500 + ``` + + `gptManagerBenchmark` by default uses the high-level C++ API defined by the `executor::Executor` class (see `cpp/include/tensorrt_llm/executor/executor.h`). + +2. Encoder-Decoder Models + To benchmark encoder-decoder models, pass in the encoder engine path with `--encoder_engine_dir` and the decoder engine path with `--decoder_engine_dir` as executable input arguments. `--decoder_engine_dir` is an alias of `--engine_dir`. + + Currently encoder-decoder engines only support `--api executor`, `--type IFB`, `--enable_kv_cache_reuse false`, which are all default values so no specific settings required. + + Prepare t5-small engine from [examples/models/core/enc_dec](/examples/models/core/enc_dec/README.md#convert-and-split-weights) for the encoder-decoder 4-GPU inflight batching example. + + Prepare the dataset suitable for engine input lengths. + ``` + python prepare_dataset.py \ + --tokenizer <path/to/tokenizer> \ + --output cnn_dailymail.json \ + dataset \ + --dataset-name cnn_dailymail \ + --dataset-split validation \ + --dataset-config-name 3.0.0 \ + --dataset-input-key article \ + --dataset-prompt "Summarize the following article:" \ + --dataset-output-key "highlights" \ + --num-requests 100 \ + --max-input-len 512 \ + --output-len-dist 128,20 + ``` + + Run the benchmark + ``` + mpirun --allow-run-as-root -np 4 ./benchmarks/gptManagerBenchmark \ + --encoder_engine_dir ../../examples/models/core/enc_dec/tmp/trt_engines/t5-small-4gpu/bfloat16/encoder \ + --decoder_engine_dir ../../examples/models/core/enc_dec/tmp/trt_engines/t5-small-4gpu/bfloat16/decoder \ + --dataset cnn_dailymail.json + ``` + + +#### Emulated static batching + +To emulate the deprecated `gptSessionBenchmark` static batching, you can use `gptManagerBenchmark` with the `--static_emulated_batch_size` and `--static_emulated-timeout` arguments. + +Given a `static_emulated_batch_size` of `n` the server will wait for `n` requests to arrive before submitting them to the batch manager at once. If the `static_emulated_timeout` (in ms) is reached before `n` requests are collected, the batch will be submitted prematurely with the current request count. New batches will only be submitted once the previous batch has been processed comepletely. + +Datasets with fixed input/output lengths for benchmarking can be generated with the preprocessing script, e.g. +``` + python prepare_dataset.py \ + --output tokens-fixed-lengths.json \ + --tokenizer <path/to/tokenizer> \ + token-norm-dist \ + --num-requests 128 \ + --input-mean 60 --input-stdev 0 \ + --output-mean 20 --output-stdev 0 +``` + +Take GPT-350M as an example for single GPU with static batching +``` +./benchmarks/gptManagerBenchmark \ + --engine_dir ../../examples/models/core/gpt/trt_engine/gpt2/fp16/1-gpu/ \ + --request_rate -1 \ + --static_emulated_batch_size 32 \ + --static_emulated_timeout 100 \ + --dataset ../../benchmarks/cpp/tokens-fixed-lengths.json +``` + +#### Benchmarking LoRA + +Using either of the `prepare_dataset.py` methods above, add `--rand-task-id <start-id> <end-id>` to the command. This will add a random `task_id` from `<start-id>` to `<end-id>` inclusive. +You can then use `utils/generate_rand_loras.py` to generate random LoRA weights for benchmarking purposes. `utils/generate_rand_loras.py` takes an example LoRA for the model you are benchmarking. +Then you can run `gptManagerBenchmark` with `--type IFB` and `--lora_dir /path/to/utils/generate_rand_loras/output` + +End-to-end LoRA benchmarking script + +``` +git-lfs clone https://huggingface.co/meta-llama/Llama-2-13b-hf +git-lfs clone https://huggingface.co/hfl/chinese-llama-2-lora-13b + +MODEL_CHECKPOINT=Llama-2-13b-hf +CONVERTED_CHECKPOINT=Llama-2-13b-hf-ckpt +TOKENIZER=Llama-2-13b-hf +LORA_ENGINE=Llama-2-13b-hf-engine + +DTYPE=float16 +TP=2 +PP=1 +MAX_LEN=1024 +MAX_BATCH=32 +NUM_LAYERS=40 +MAX_LORA_RANK=64 +NUM_LORA_MODS=7 +EOS_ID=2 + +SOURCE_LORA=chinese-llama-2-lora-13b +CPP_LORA=chinese-llama-2-lora-13b-cpp + +EG_DIR=/tmp/lora-eg + +# Build lora enabled engine +python examples/models/core/llama/convert_checkpoint.py --model_dir ${MODEL_CHECKPOINT} \ + --output_dir ${CONVERTED_CHECKPOINT} \ + --dtype ${DTYPE} \ + --tp_size ${TP} \ + --pp_size 1 + +${HOME}/.local/bin/trtllm-build \ + --checkpoint_dir ${CONVERTED_CHECKPOINT} \ + --output_dir ${LORA_ENGINE} \ + --max_batch_size ${MAX_BATCH} \ + --max_input_len $MAX_LEN \ + --max_seq_len $((2*${MAX_LEN})) \ + --gemm_plugin float16 \ + --lora_plugin float16 \ + --use_paged_context_fmha enable \ + --lora_target_modules attn_q attn_k attn_v attn_dense mlp_h_to_4h mlp_4h_to_h mlp_gate \ + --max_lora_rank ${MAX_LORA_RANK} + +NUM_LORAS=(8 16) +NUM_REQUESTS=1024 + +# Convert LoRA to cpp format +python examples/hf_lora_convert.py \ + -i $SOURCE_LORA \ + --storage-type $DTYPE \ + -o $CPP_LORA + +# Prepare datasets +mkdir -p $EG_DIR/data + +# Prepare dataset without lora_task_id +python benchmarks/cpp/prepare_dataset.py \ + --output "${EG_DIR}/data/token-norm-dist.json" \ + --tokenizer $TOKENIZER \ + token-norm-dist \ + --num-requests $NUM_REQUESTS \ + --input-mean 256 --input-stdev 16 --output-mean 128 --output-stdev 24 + +# Prepare dataset with lora_task_ids from 0 - $nloras +for nloras in ${NUM_LORAS[@]}; do + python benchmarks/cpp/prepare_dataset.py \ + --output "${EG_DIR}/data/token-norm-dist-lora-${nloras}.json" \ + --rand-task-id 0 $(( $nloras - 1 )) \ + --tokenizer $TOKENIZER \ + token-norm-dist \ + --num-requests $NUM_REQUESTS \ + --input-mean 256 --input-stdev 16 --output-mean 128 --output-stdev 24 +done + +# Generate random lora weights for 16 adapters +python benchmarks/cpp/utils/generate_rand_loras.py ${CPP_LORA} ${EG_DIR}/loras 16 + +# Perform benchmarking + +# First run inference without LoRAs +mkdir -p ${EG_DIR}/log-base-lora +mpirun -n ${TP} --output-filename ${EG_DIR}/log-base-lora \ + cpp/build/benchmarks/gptManagerBenchmark \ + --engine_dir $LORA_ENGINE \ + --type IFB \ + --dataset "${EG_DIR}/data/token-norm-dist.json" \ + --lora_host_cache_bytes 8589934592 \ + --lora_num_device_mod_layers $(( 32 * $NUM_LAYERS * $NUM_LORA_MODS * $MAX_LORA_RANK )) \ + --kv_cache_free_gpu_mem_fraction 0.70 \ + --log_level info \ + --eos_id ${EOS_ID} + +# Now run inference with various numbers or loras +# The host cache is set large enough to hold all the LoRAs in lora_dir +# GPU cache is set to hold 16 LoRAs +# This benchmark will preload all the LoRAs into the host cache +# We run inference on a range of active LoRAs exercising different cache miss rates. +for nloras in ${NUM_LORAS[@]}; do + mkdir -p ${EG_DIR}/log-lora-${nloras} + mpirun -n ${TP} --output-filename "${EG_DIR}/log-lora-${nloras}" \ + cpp/build/benchmarks/gptManagerBenchmark \ + --engine_dir $LORA_ENGINE \ + --type IFB \ + --dataset "${EG_DIR}/data/token-norm-dist-lora-${nloras}.json" \ + --lora_host_cache_bytes 8589934592 \ + --lora_num_device_mod_layers $(( 16 * $NUM_LAYERS * $NUM_LORA_MODS * $MAX_LORA_RANK )) \ + --kv_cache_free_gpu_mem_fraction 0.70 \ + --log_level info \ + --eos_id ${EOS_ID} \ + --lora_dir ${EG_DIR}/loras +done +``` + +### 3. [DEPRECATED] Launch C++ static batching benchmarking (Fixed BatchSize/InputLen/OutputLen) + +#### Prepare TensorRT-LLM engine(s) + +Before you launch C++ benchmarking, please make sure that you have already built engine(s) using TensorRT-LLM API, C++ benchmarking code cannot generate engine(s) for you. + +Use `trtllm-build` to build the TRT-LLM engine. Alternatively, if you have already benchmarked Python Runtime, you can reuse the engine(s) built previously, please see that [`document`](../python/README.md). + +#### Launch benchmarking + +For detailed usage, you can do the following +``` +cd cpp/build + +# You can directly execute the binary for help information +./benchmarks/bertBenchmark --help +``` + +*Please note that the expected outputs in that document are only for reference, specific performance numbers depend on the GPU you're using.* + + +### 4.launch C++ disaggServerBenchmark +Currently ,TensorRT-LLM has limited support for disaggregated inference, where context and generation phases of a request can run on different executors. `disaggServerBenchmark` is a tool to benchmark disaggregated inference. + +#### Usage +For detailed usage, you can do the following +``` +cd cpp/build + +# You can directly execute the binary for help information +./benchmarks/disaggServerBenchmark --help +``` +`disaggServerBenchmark` only supports `decoder-only` models. +Here is the basic usage: +``` +export TRTLLM_USE_UCX_KVCACHE=1 +mpirun -n ${proc} benchmarks/disaggServerBenchmark --context_engine_dirs ${context_engine_0},${context_engine_1}...,${context_engine_{m-1}} \ +--generation_engine_dirs ${generation_engine_0},${generation_engine_1}...,${generation_engine_{n-1}} --dataset ${dataset_path} +``` +This command will launch m context engines and n generation engines. You need to ensure `proc` is equal to the sum of the number of processes required for each engine plus 1. Since we use orchestrator mode for `disaggServerBenchmark` we need an additional process as the orchestrator. For example, if there are two context engines (one is TP2_PP1,another is TP1_PP1) and two generation engines(one is TP2_PP1,another is TP1_PP1), then the `proc` value should be set to 7. + +for example: +``` +export TRTLLM_USE_UCX_KVCACHE=1 +mpirun -n 7 benchmarks/disaggServerBenchmark --context_engine_dirs ${llama_7b_tp2_pp1_dir},${llama_7b_tp1_pp1_dir} --generation_engine_dirs ${llama_7b_tp1_pp1_dir},${llama_7b_tp2_pp1_dir} --dataset ${dataset_path} + +# need 6 gpus and 7 processes to launch the benchmark. +``` + +#### Known Issues + +##### 1. error `All available sequence slots are used` + +If generation_engine's pp_size >1, the error "All available sequence slots are used" may occur, setting and adjusting the parameter `--request_rate` may help alleviate the problem. + +##### 2.KVCache transfers are by default via PCIE on single node. +Currently, because of the dependency libraries,KVCache transfers are by default via PCIE on single node. + +If you want to use NVLink, please check the UCX version in the container by running: +``` +ucx_info -v +``` +If the UCX version is less than or equal to 1.17, set `UCX_RNDV_FRAG_MEM_TYPE=cuda` to enable KvCache transfers using NVLink. +If the UCX version is 1.18, please set `UCX_CUDA_COPY_ASYNC_MEM_TYPE=cuda` to enable KvCache transfers using NVLink. diff --git a/tensorrt_llm/_torch/kv_cache_compression/__init__.py b/benchmarks/cpp/__init__.py similarity index 100% rename from tensorrt_llm/_torch/kv_cache_compression/__init__.py rename to benchmarks/cpp/__init__.py diff --git a/benchmarks/cpp/bertBenchmark.cpp b/benchmarks/cpp/bertBenchmark.cpp new file mode 100644 index 000000000000..cc10a5b49eee --- /dev/null +++ b/benchmarks/cpp/bertBenchmark.cpp @@ -0,0 +1,260 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +#include <NvInfer.h> +#include <cxxopts.hpp> +#include <nlohmann/json.hpp> + +#include <chrono> +#include <filesystem> +#include <fstream> +#include <iostream> +#include <sstream> +#include <string> + +using namespace tensorrt_llm::runtime; + +namespace trt = nvinfer1; + +namespace +{ + +std::string engineFilename( + std::filesystem::path const& dataPath, WorldConfig const& worldConfig, std::string const& model) +{ + auto constexpr allowExceptions = true; + auto constexpr ignoreComments = true; + auto const jsonFilePath = dataPath / "config.json"; + TLLM_CHECK_WITH_INFO( + std::filesystem::exists(jsonFilePath), std::string("File does not exist: ") + jsonFilePath.string()); + std::ifstream jsonStream(jsonFilePath); + auto const json = nlohmann::json::parse(jsonStream, nullptr, allowExceptions, ignoreComments); + auto const& builderConfig = json.at("builder_config"); + auto const precision = builderConfig.at("precision").template get<std::string>(); + auto const worldSize = builderConfig.at("tensor_parallel").template get<SizeType32>(); + + TLLM_CHECK_WITH_INFO(worldSize == worldConfig.getSize(), "world size mismatch"); + return model + "_" + precision + "_tp" + std::to_string(worldConfig.getSize()) + "_rank" + + std::to_string(worldConfig.getRank()) + ".engine"; +} + +void benchmarkBert(std::string const& modelName, std::filesystem::path const& dataPath, + std::vector<int> const& batchSizes, std::vector<int> const& inLens, bool useGpuDirectStorage, + std::vector<float> const& gpuWeightsPercents, std::shared_ptr<nvinfer1::ILogger> const& logger, int warmUp, + int numRuns, int duration) +{ + auto const worldConfig = WorldConfig::mpi(); + auto const enginePath = dataPath / engineFilename(dataPath, worldConfig, modelName); + + for (float gpuWeightsPercent : gpuWeightsPercents) + { + auto rt = std::make_shared<TllmRuntime>( + RawEngine(enginePath), logger.get(), useGpuDirectStorage, gpuWeightsPercent); + rt->addContext(0); + for (auto inLen : inLens) + { + for (auto const batchSize : batchSizes) + { + auto& allocator = rt->getBufferManager(); + TllmRuntime::TensorMap tensorMap{}; + + // input_ids + std::vector<SizeType32> inputIdsHost(batchSize * inLen, inLen); + auto inputIdsBuffer = std::shared_ptr<ITensor>{ + allocator.copyFrom(inputIdsHost, ITensor::makeShape({batchSize, inLen}), MemoryType::kGPU)}; + allocator.setZero(*inputIdsBuffer); + tensorMap.insert(std::make_pair("input_ids", inputIdsBuffer)); + // input_lengths + std::vector<SizeType32> inputLengthsHost(batchSize); + auto inLensBuffer = std::shared_ptr<ITensor>{ + allocator.copyFrom(inputLengthsHost, ITensor::makeShape({batchSize}), MemoryType::kGPU)}; + allocator.setZero(*inLensBuffer); + tensorMap.insert(std::make_pair("input_lengths", inLensBuffer)); + + rt->setInputTensors(0, tensorMap); + rt->setOutputTensors(0, tensorMap); + cudaDeviceSynchronize(); + + for (auto r = 0; r < warmUp; ++r) + { + rt->executeContext(0); + rt->getStream().synchronize(); + } + cudaDeviceSynchronize(); + + int iterIdx = 0; + float curDuration = 0; + while (iterIdx < numRuns || curDuration / 1000 < duration) + { + auto const start = std::chrono::steady_clock::now(); + rt->executeContext(0); + rt->getStream().synchronize(); + auto const end = std::chrono::steady_clock::now(); + + iterIdx += 1; + curDuration += (static_cast<float>( + std::chrono::duration_cast<std::chrono::microseconds>(end - start).count()) + / 1000); + } + printf("Benchmarking done. Iteration: %d, duration: %.2f sec.\n", iterIdx, curDuration / 1000); + + auto averageLatency = curDuration / iterIdx; + + if (worldConfig.getRank() == 0) + { + printf("[BENCHMARK] batch_size %d input_length %d latency(ms) %.2f\n", batchSize, inLen, + averageLatency); + } + } + } + } +} + +} // namespace + +int main(int argc, char* argv[]) +{ + cxxopts::Options options("TensorRT LLM C++ Runtime Benchmark", "TensorRT LLM C++ Runtime Benchmark for BERT."); + options.add_options()("h,help", "Print usage"); + options.add_options()( + "m,model", "Model name specified for engines.", cxxopts::value<std::string>()->default_value("bert_base")); + options.add_options()("engine_dir", "Directory that store the engines.", cxxopts::value<std::string>()); + options.add_options()("batch_size", + "Specify batch size(s) you want to benchmark. Multiple batch sizes can be separated by \";\", example: " + "\"1;8;64\".", + cxxopts::value<std::string>()->default_value("8")); + options.add_options()("input_len", + "Specify input length(s) you want to benchmark. Multiple input lengths can be " + "separated by \";\", example: \"60;128\".", + cxxopts::value<std::string>()->default_value("128")); + + options.add_options()("log_level", "Choose log level between verbose/info/warning/error/internal_error.", + cxxopts::value<std::string>()->default_value("error")); + options.add_options()( + "warm_up", "Specify warm up iterations before benchmark starts.", cxxopts::value<int>()->default_value("2")); + options.add_options()("num_runs", "Minimal number of iterations to run during benchmarking.", + cxxopts::value<int>()->default_value("10")); + options.add_options()("duration", "Minimal duration of iterations to measure in seconds.", + cxxopts::value<int>()->default_value("60")); + options.add_options()("gpu_weights_percent", + "Specify the percentage of weights that reside on GPU (from 0.0 to 1.0). Multiple percentages can be separated " + "by \";\", " + "example: \"0.0;0.5;1.0\".", + cxxopts::value<std::string>()->default_value("1.0")); + options.add_options()("use_gpu_direct_storage", "Enable GPUDirect Storage (GDS) for loading engine.", + cxxopts::value<bool>()->default_value("false")); + + auto result = options.parse(argc, argv); + + if (result.count("help")) + { + std::cout << options.help() << std::endl; + exit(0); + } + + // Argument: Engine directory + if (!result.count("engine_dir")) + { + std::cout << options.help() << std::endl; + TLLM_LOG_ERROR("Please specify engine directory."); + return 1; + } + + // Argument: Batch sizes + std::istringstream ssBatchSizesArg; + ssBatchSizesArg.str(result["batch_size"].as<std::string>()); + std::vector<int> batchSizes; + for (std::string token; std::getline(ssBatchSizesArg, token, ';');) + { + batchSizes.push_back(std::stoi(token)); + } + + // Argument : Input lengths + std::istringstream ssInLenArg; + ssInLenArg.str(result["input_len"].as<std::string>()); + std::vector<int> inLens; + for (std::string token; std::getline(ssInLenArg, token, ';');) + { + inLens.push_back(std::stoi(token)); + } + + // Argument: GPU weights percentage + std::istringstream ssGpuPercentArg; + ssGpuPercentArg.str(result["gpu_weights_percent"].as<std::string>()); + std::vector<float> gpuWeightsPercents; + for (std::string token; std::getline(ssGpuPercentArg, token, ';');) + { + auto gpuWeightsPercent = std::stof(token); + if (gpuWeightsPercent < 0 || gpuWeightsPercent > 1) + { + TLLM_LOG_ERROR( + "--gpu_weights_percent must have percents between 0.0 and 1.0 but got: %f", gpuWeightsPercent); + return 1; + } + gpuWeightsPercents.push_back(gpuWeightsPercent); + } + + // Argument: Log level + auto logger = std::make_shared<TllmLogger>(); + auto const logLevel = result["log_level"].as<std::string>(); + if (logLevel == "verbose") + { + logger->setLevel(trt::ILogger::Severity::kVERBOSE); + } + else if (logLevel == "info") + { + logger->setLevel(trt::ILogger::Severity::kINFO); + } + else if (logLevel == "warning") + { + logger->setLevel(trt::ILogger::Severity::kWARNING); + } + else if (logLevel == "error") + { + logger->setLevel(trt::ILogger::Severity::kERROR); + } + else if (logLevel == "internal_error") + { + logger->setLevel(trt::ILogger::Severity::kINTERNAL_ERROR); + } + else + { + TLLM_LOG_ERROR("Unexpected log level: " + logLevel); + return 1; + } + initTrtLlmPlugins(logger.get()); + + try + { + benchmarkBert(result["model"].as<std::string>(), result["engine_dir"].as<std::string>(), batchSizes, inLens, + result["use_gpu_direct_storage"].as<bool>(), gpuWeightsPercents, logger, result["warm_up"].as<int>(), + result["num_runs"].as<int>(), result["duration"].as<int>()); + } + catch (std::exception const& e) + { + TLLM_LOG_ERROR(e.what()); + return 1; + } + return 0; +} diff --git a/benchmarks/cpp/disaggServerBenchmark.cpp b/benchmarks/cpp/disaggServerBenchmark.cpp new file mode 100644 index 000000000000..bc3a7a2659fd --- /dev/null +++ b/benchmarks/cpp/disaggServerBenchmark.cpp @@ -0,0 +1,1582 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/disaggServerUtil.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/gptJsonConfig.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "utils/utils.h" + +#include "cxxopts.hpp" +#include <nlohmann/json.hpp> + +#include <chrono> +#include <cstdint> +#include <cstdio> +#include <filesystem> +#include <memory> +#include <mutex> +#include <numeric> +#include <optional> +#include <string> +#include <thread> +#include <unordered_map> +#include <vector> + +using namespace tensorrt_llm::batch_manager; +using namespace tensorrt_llm::runtime; +using namespace tensorrt_llm::benchmark; +using namespace tensorrt_llm::executor::disagg_executor; +namespace texec = tensorrt_llm::executor; +namespace trt = nvinfer1; + +namespace +{ + +class Recorder +{ + +public: + explicit Recorder(std::string opCsvFile, bool streaming = false, int beamWidth = 1, + bool calculateKvCacheTransferTime = true, bool calculateQueueTime = true, std::string responsesJsonFile = "", + bool excludeInputInOutput = false) + : mOpCsvFile(std::move(opCsvFile)) + , mStreaming(streaming) + , mBeamWidth(beamWidth) + , mRespJsonFile(std::move(responsesJsonFile)) + , mOutputHasInput(!excludeInputInOutput) + , mCalculateKVCacheTransferTime(calculateKvCacheTransferTime) + , mCalculateQueueTime(calculateQueueTime) + { + } + + void initialize() + { + mStart = std::chrono::steady_clock::now(); + mSeqLatency.mDataTimes.clear(); + mFtLatency.mDataTimes.clear(); + mGenLatency.mDataTimes.clear(); + mGenFirstTokenLatency.mDataTimes.clear(); + mGenT2TLatency.mDataTimes.clear(); + mGenExcludeFirstIterT2TLatency.mDataTimes.clear(); + mContextReqQueuingLatency.mDataTimes.clear(); + mGenReqQueuingLatency.mDataTimes.clear(); + mGenReqKvCacheTransferLatency.mDataTimes.clear(); + mKvCacheThroughput.mDataTps.clear(); + } + + void finalize() + { + mEnd = std::chrono::steady_clock::now(); + } + + void recordContextQueueLatency(std::vector<float> const& latencies) + { + mContextReqQueuingLatency.mDataTimes.insert( + mContextReqQueuingLatency.mDataTimes.end(), latencies.begin(), latencies.end()); + } + + void recordGenQueueLatency(std::vector<float> const& latencies) + { + mGenReqQueuingLatency.mDataTimes.insert( + mGenReqQueuingLatency.mDataTimes.end(), latencies.begin(), latencies.end()); + } + + void recordKvCacheTransferLatency(std::vector<float> const& latencies) + { + mGenReqKvCacheTransferLatency.mDataTimes.insert( + mGenReqKvCacheTransferLatency.mDataTimes.end(), latencies.begin(), latencies.end()); + } + + void recordKvCacheThroughput(std::vector<float> const& throughputs) + { + mKvCacheThroughput.mDataTps.insert(mKvCacheThroughput.mDataTps.end(), throughputs.begin(), throughputs.end()); + } + + void recordContextStart(SizeType32 inputLength, SizeType32 maxNewTokens, uint64_t requestId, + std::chrono::time_point<std::chrono::steady_clock> const& start) + { + mRequestBenchInfos[requestId] = BenchInfo(inputLength, start); + } + + void recordContextEnd(tensorrt_llm::executor::IdType requestId, bool hasError) + { + TLLM_CHECK(mRequestBenchInfos.find(requestId) != mRequestBenchInfos.end()); + mRequestBenchInfos.at(requestId).contextEnd = std::chrono::steady_clock::now(); + mRequestBenchInfos.at(requestId).contextHasError = hasError; + mRequestBenchInfos.at(requestId).decodingIter += 1; + } + + void recordToken(tensorrt_llm::executor::IdType requestId) + { + TLLM_CHECK(mStreaming); + TLLM_CHECK_WITH_INFO(mBeamWidth == 1, "gptManagerBenchmark streaming mode does not support beam > 1"); + TLLM_CHECK(mRequestBenchInfos.find(requestId) != mRequestBenchInfos.end()); + + if (!mRequestBenchInfos.at(requestId).genFirstTokenSeen) + { + mRequestBenchInfos.at(requestId).genFirstTokenTs = std::chrono::steady_clock::now(); + mRequestBenchInfos.at(requestId).genFirstTokenSeen = true; + } + mRequestBenchInfos.at(requestId).decodingIter += 1; + } + + void recordToken(tensorrt_llm::executor::IdType requestId, texec::Response const& response) + { + + TLLM_CHECK(mRequestBenchInfos.find(requestId) != mRequestBenchInfos.end()); + + auto outputTokenIds = response.getResult().outputTokenIds; + + int32_t outputLength = 1; + for (auto const& beam : outputTokenIds) + { + outputLength = std::max(static_cast<int32_t>(beam.size()), outputLength); + } + + mRequestBenchInfos[requestId].outputLength += outputLength; + this->recordToken(requestId); + } + + void recordGenStart( + tensorrt_llm::executor::IdType requestId, std::chrono::time_point<std::chrono::steady_clock> const& start) + { + + TLLM_CHECK(mRequestBenchInfos.find(requestId) != mRequestBenchInfos.end()); + mRequestBenchInfos.at(requestId).genStart = start; + } + + void recordGenEnd(tensorrt_llm::executor::IdType requestId, bool hasError) + { + TLLM_CHECK(mRequestBenchInfos.find(requestId) != mRequestBenchInfos.end()); + mRequestBenchInfos.at(requestId).genEnd = std::chrono::steady_clock::now(); + mRequestBenchInfos.at(requestId).genHasError = hasError; + } + + void recordGenEnd(tensorrt_llm::executor::IdType requestId, texec::Response const& response) + { + recordGenEnd(requestId, response.hasError()); + if (!response.hasError()) + { + if (!mStreaming) + { + TLLM_LOG_DEBUG("response.getResult().outputTokenIds"); + auto outputTokenIds = response.getResult().outputTokenIds; + + int32_t outSeqLen = 0; + for (auto const& beam : outputTokenIds) + { + outSeqLen = std::max(static_cast<int32_t>(beam.size()), outSeqLen); + } + if (mOutputHasInput) + { + int inputSeqLen = mRequestBenchInfos[requestId].inputLength; + outSeqLen -= inputSeqLen; + } + mRequestBenchInfos[requestId].outputLength = outSeqLen; + mRequestBenchInfos[requestId].decodingIter = response.getResult().decodingIter; + } + else + { + recordToken(requestId, response); + } + } + } + + void reserve(size_t size) + { + mRequestBenchInfos.reserve(size); + } + + void calculateLatencies() + { + for (auto& reqInfo : mRequestBenchInfos) + { + + reqInfo.second.latency + = std::chrono::duration<float, std::milli>(reqInfo.second.genEnd - reqInfo.second.contextStart).count(); + reqInfo.second.firstTokenLatency + = std::chrono::duration<float, std::milli>(reqInfo.second.contextEnd - reqInfo.second.contextStart) + .count(); + reqInfo.second.genLatency + = std::chrono::duration<float, std::milli>(reqInfo.second.genEnd - reqInfo.second.genStart).count(); + if (mStreaming) + { + reqInfo.second.genFirstTokenLatency + = std::chrono::duration<float, std::milli>(reqInfo.second.genFirstTokenTs - reqInfo.second.genStart) + .count(); + // include the latency of the second token+ kv Cache transfer latency + + if (reqInfo.second.outputLength > 1) + { + reqInfo.second.avgGenT2TLatency + = std::chrono::duration<float, std::milli>(reqInfo.second.genEnd - reqInfo.second.genStart) + .count() + / static_cast<float>(reqInfo.second.outputLength - 1); + } + if (reqInfo.second.outputLength > 2) + { + reqInfo.second.avgGenExcludeFirstIterT2TLatency + = std::chrono::duration<float, std::milli>( + reqInfo.second.genEnd - reqInfo.second.genFirstTokenTs) + .count() + / static_cast<float>(reqInfo.second.outputLength - 2); + } + } + } + } + + void calculateMetrics() + { + + calculateLatencies(); + + int totalOutputTokens{0}; + int totalDecodingIter{0}; + mNumContextErrorSamples = 0; + mNumGenErrorSamples = 0; + mNumSamples = 0; + for (auto const& reqInfo : mRequestBenchInfos) + { + + if (!reqInfo.second.contextHasError && !reqInfo.second.genHasError) + { + mSeqLatency.mDataTimes.push_back(reqInfo.second.latency); + mNumSamples++; + } + if (!reqInfo.second.contextHasError) + { + mFtLatency.mDataTimes.push_back(reqInfo.second.firstTokenLatency); + } + else + { + mNumContextErrorSamples++; + } + if (!reqInfo.second.genHasError) + { + mGenLatency.mDataTimes.push_back(reqInfo.second.genLatency); + totalOutputTokens += reqInfo.second.outputLength; + totalDecodingIter += reqInfo.second.decodingIter; + if (mStreaming) + { + mGenFirstTokenLatency.mDataTimes.push_back(reqInfo.second.genFirstTokenLatency); + + if (reqInfo.second.avgGenT2TLatency.has_value()) + { + mGenT2TLatency.mDataTimes.push_back(reqInfo.second.avgGenT2TLatency.value()); + } + if (reqInfo.second.avgGenExcludeFirstIterT2TLatency.has_value()) + { + mGenExcludeFirstIterT2TLatency.mDataTimes.push_back( + reqInfo.second.avgGenExcludeFirstIterT2TLatency.value()); + } + } + } + else + { + mNumGenErrorSamples++; + } + } + mTotalLatency = std::chrono::duration<float, std::milli>(mEnd - mStart).count(); + mSeqThroughput = mNumSamples / (mTotalLatency / 1000); + mTokenThroughput = totalOutputTokens / (mTotalLatency / 1000); + mAcceptanceRate = totalDecodingIter + ? (static_cast<float>(totalOutputTokens) / static_cast<float>(totalDecodingIter)) + : 0.0F; + + mSeqLatency.calculate(); + mFtLatency.calculate(); + mGenLatency.calculate(); + if (mStreaming) + { + + mGenFirstTokenLatency.calculate(); + + if (!mGenT2TLatency.mDataTimes.empty()) + { + mGenT2TLatency.calculate(); + std::vector<float> userTokensPerSecond; + userTokensPerSecond.reserve(mGenT2TLatency.mDataTimes.size()); + for (auto const& latency : mGenT2TLatency.mDataTimes) + { + userTokensPerSecond.push_back(1000.F / latency); + } + mAvgUserTokensPerSecond = std::accumulate(userTokensPerSecond.begin(), userTokensPerSecond.end(), 0.F) + / userTokensPerSecond.size(); + } + if (!mGenExcludeFirstIterT2TLatency.mDataTimes.empty()) + { + + mGenExcludeFirstIterT2TLatency.calculate(); + } + } + if (mCalculateQueueTime) + { + + mContextReqQueuingLatency.calculate(); + mGenReqQueuingLatency.calculate(); + } + if (mCalculateKVCacheTransferTime) + { + mGenReqKvCacheTransferLatency.calculate(); + mKvCacheThroughput.calculate(); + } + } + + void report() + { + printf("[BENCHMARK] num_samples %d\n", mNumSamples); + printf("[BENCHMARK] num_context_error_samples %d\n", mNumContextErrorSamples); + printf("[BENCHMARK] num_gen_error_samples %d\n", mNumGenErrorSamples); + printf("\n[BENCHMARK] num_samples %d\n", mNumSamples); + printf("[BENCHMARK] total_latency(ms) %.2f\n", mTotalLatency); + printf("[BENCHMARK] seq_throughput(seq/sec) %.2f\n", mSeqThroughput); + printf("[BENCHMARK] token_throughput(token/sec) %.2f\n", mTokenThroughput); + if (mStreaming) + { + printf("[BENCHMARK] user_tokens_per_second(tokens/sec/user) %.2f\n", mAvgUserTokensPerSecond); + } + printf("[BENCHMARK] avg_acceptance_rate(tokens/decoding steps) %.2f\n\n", mAcceptanceRate); + + mSeqLatency.report(); + mFtLatency.report(); + mGenLatency.report(); + if (mStreaming) + { + mGenFirstTokenLatency.report(); + mGenT2TLatency.report(); + mGenExcludeFirstIterT2TLatency.report(); + } + if (mCalculateQueueTime) + { + mContextReqQueuingLatency.report(); + mGenReqQueuingLatency.report(); + } + if (mCalculateKVCacheTransferTime) + { + mGenReqKvCacheTransferLatency.report(); + mKvCacheThroughput.report(); + } + } + + void writeOpMetricsToCsv() + { + if (!mOpCsvFile.empty()) + { + std::vector<std::string> headers{"num_samples", "num_context_error_samples", "num_gen_error_samples", + "total_latency(ms)", "seq_throughput(seq/sec)", "token_throughput(token/sec)"}; + auto seqLatencyHeader = mSeqLatency.genHeaders(); + headers.insert(headers.end(), std::make_move_iterator(seqLatencyHeader.begin()), + std::make_move_iterator(seqLatencyHeader.end())); + auto contextLatencyHeader = mFtLatency.genHeaders(); + headers.insert(headers.end(), std::make_move_iterator(contextLatencyHeader.begin()), + std::make_move_iterator(contextLatencyHeader.end())); + auto genLatencyHeader = mGenLatency.genHeaders(); + headers.insert(headers.end(), std::make_move_iterator(genLatencyHeader.begin()), + std::make_move_iterator(genLatencyHeader.end())); + if (mStreaming) + { + auto genFirstTokenHeader = mGenFirstTokenLatency.genHeaders(); + headers.insert(headers.end(), std::make_move_iterator(genFirstTokenHeader.begin()), + std::make_move_iterator(genFirstTokenHeader.end())); + auto genIngterHeader = mGenT2TLatency.genHeaders(); + headers.insert(headers.end(), std::make_move_iterator(genIngterHeader.begin()), + std::make_move_iterator(genIngterHeader.end())); + auto excludeFirstIterIngterHeader = mGenExcludeFirstIterT2TLatency.genHeaders(); + headers.insert(headers.end(), std::make_move_iterator(excludeFirstIterIngterHeader.begin()), + std::make_move_iterator(excludeFirstIterIngterHeader.end())); + headers.push_back("avg_user_tokens_per_second(tokens/sec/user)"); + } + if (mCalculateKVCacheTransferTime) + { + auto genReqKVCacheTransferHeader = mGenReqKvCacheTransferLatency.genHeaders(); + headers.insert(headers.end(), std::make_move_iterator(genReqKVCacheTransferHeader.begin()), + std::make_move_iterator(genReqKVCacheTransferHeader.end())); + auto kvCacheTpHeader = mKvCacheThroughput.genHeaders(); + headers.insert(headers.end(), std::make_move_iterator(kvCacheTpHeader.begin()), + std::make_move_iterator(kvCacheTpHeader.end())); + } + + std::ofstream outputFile(mOpCsvFile); + + if (outputFile.is_open()) + { + for (auto const& header : headers) + { + outputFile << header << ","; + } + outputFile << "\n"; + + outputFile << mNumSamples << "," << mNumContextErrorSamples << "," << mNumGenErrorSamples << "," + << mTotalLatency << "," << mSeqThroughput << "," << mTokenThroughput << "," << mSeqLatency + << "," << mFtLatency << "," << mGenLatency; + if (mStreaming) + { + + outputFile << "," << mGenFirstTokenLatency << "," << mGenT2TLatency << "," + << mGenExcludeFirstIterT2TLatency << "," << mAvgUserTokensPerSecond; + } + if (mCalculateKVCacheTransferTime) + { + outputFile << "," << mGenReqKvCacheTransferLatency << "," << mKvCacheThroughput; + } + + outputFile << "\n"; + } + else + { + std::cerr << "Error opening file '" << mOpCsvFile << "' for writing.\n"; + } + } + } + +private: + struct BenchInfo + { + BenchInfo() = default; + + BenchInfo(int inputLength, std::chrono::time_point<std::chrono::steady_clock> start) + : inputLength(inputLength) + , contextStart(start) + { + } + + int inputLength{}; + int outputLength{}; + std::chrono::time_point<std::chrono::steady_clock> contextStart; + std::chrono::time_point<std::chrono::steady_clock> contextEnd; + std::chrono::time_point<std::chrono::steady_clock> genFirstTokenTs; + std::chrono::time_point<std::chrono::steady_clock> genStart; + std::chrono::time_point<std::chrono::steady_clock> genEnd; + float latency{}; // millisecond + float genLatency{}; + bool contextHasError{false}; + bool genHasError{false}; + float firstTokenLatency{}; + float genFirstTokenLatency{}; + std::optional<float> avgGenT2TLatency; + std::optional<float> avgGenExcludeFirstIterT2TLatency; + bool genFirstTokenSeen{false}; + SizeType32 decodingIter{0}; + }; + + std::unordered_map<uint64_t, BenchInfo> mRequestBenchInfos; + + std::chrono::time_point<std::chrono::steady_clock> mStart; + std::chrono::time_point<std::chrono::steady_clock> mEnd; + int mNumSamples{}; + int mNumContextErrorSamples{}; + int mNumGenErrorSamples{}; + float mTotalLatency{}; + float mSeqThroughput{}; + RecordTimeMetric mSeqLatency{"sequence_latency"}; + RecordTimeMetric mFtLatency{"context_latency"}; + RecordTimeMetric mGenLatency{"gen_latency"}; + + RecordTimeMetric mGenFirstTokenLatency{"time_to_gen_first_token"}; + RecordTimeMetric mGenT2TLatency{"inter_token_latency"}; + RecordTimeMetric mGenExcludeFirstIterT2TLatency{"exclude_first_iter_inter_token_latency"}; + RecordTimeMetric mContextReqQueuingLatency{"context_req_queueing_latency"}; + + RecordTimeMetric mGenReqQueuingLatency{"gen_req_queueing_latency"}; + RecordTimeMetric mGenReqKvCacheTransferLatency{"gen_req_kv_cache_transfer_latency"}; + + RecordBwMetric mKvCacheThroughput{"gen_req_kv_cache_transfer_throughput"}; + + float mTokenThroughput{}; + float mAcceptanceRate{}; + + std::string mOpCsvFile; + bool mStreaming; + int mBeamWidth; + std::string mRespJsonFile; + std::unordered_map<uint64_t, tensorrt_llm::executor::TensorPtr> mResponseTensors; + bool mOutputHasInput; + bool mCalculateKVCacheTransferTime; + bool mCalculateQueueTime; + float mAvgUserTokensPerSecond{}; +}; + +texec::Request makeExecutorContextRequest(Sample const& sample, SizeType32 const& beamWidth, + std::optional<SizeType32> const& eosId, std::optional<SizeType32> const& padId, bool streaming = false, + bool const& returnContextLogits = false, bool const& returnGenerationLogits = false, + std::optional<texec::LoraConfig> const& loraConfig = std::nullopt, + std::optional<texec::LookaheadDecodingConfig> const& lookaheadConfig = std::nullopt, + std::optional<texec::VecTokens> const& encoderInputTokenIds = std::nullopt) +{ + auto samplingConfig = texec::SamplingConfig{beamWidth}; + auto outputConfig = texec::OutputConfig{false, returnContextLogits, returnGenerationLogits, false}; + auto request + = texec::Request(sample.inputIds, sample.outputLen, streaming, samplingConfig, outputConfig, eosId, padId, + std::nullopt, // positionIds + std::nullopt, // badWords + std::nullopt, // stopWords + std::nullopt, // embeddingBias + std::nullopt, // speculativeDecoding + std::nullopt, // pTuning + std::nullopt, // multimodalInput + std::nullopt, // multimodalEmbedding + std::nullopt, // mRopeConfig + loraConfig, // loraConfig + lookaheadConfig, // lookaheadConfig + std::nullopt, // kvCacheRetentionConfig + std::nullopt, // logitsPostProcessorName + std::nullopt, // logitsPostProcessor + encoderInputTokenIds.has_value() ? encoderInputTokenIds : std::nullopt, + std::nullopt); // cacheSalt + request.setRequestType(tensorrt_llm::executor::RequestType::REQUEST_TYPE_CONTEXT_ONLY); + return request; +} + +class DisaggExecutorServer +{ + +public: + DisaggExecutorServer(std::vector<std::filesystem::path> const& contextEnginePaths, + std::vector<std::filesystem::path> const& genEnginePaths, + std::optional<std::vector<std::vector<SizeType32>>> const& deviceIdsForInstance, int32_t maxBeamWidth, + texec::CapacitySchedulerPolicy capacitySchedulerPolicy, BenchmarkParams const& benchmarkParams, + std::shared_ptr<Recorder> recorder, std::chrono::milliseconds waitSleep, bool logIterationData, + bool hasContextAwaitThreads, bool hasGenAwaitThreads) + : mRecorder(std::move(recorder)) + , mWaitSleep(waitSleep) + , mConcurrency(benchmarkParams.concurrency) + , mShutdown(false) + , mLogIterationData(logIterationData) + , mEnableCollectKvCacheTransferTime(benchmarkParams.enableCollectkvCacheTransferTime) + , mEnableCollectIterStats(benchmarkParams.enableCollectIterStats) + { + + int worldRank = tensorrt_llm::mpi::MpiComm::world().getRank(); + int worldSize = tensorrt_llm::mpi::MpiComm::world().getSize(); + mIsOrchestrator = (worldRank == 0); + auto contextNum = contextEnginePaths.size(); + auto genNum = genEnginePaths.size(); + int deviceCount = -1; + TLLM_CUDA_CHECK(cudaGetDeviceCount(&deviceCount)); + + std::vector<std::unique_ptr<tensorrt_llm::executor::Executor>> instances; + auto instanceNum = genNum + contextNum; + if (worldRank == 0) + { + TLLM_LOG_INFO("context enigne num :%d gen enigne num:%d", contextNum, genNum); + } + + int startRank = 0; + std::vector<texec::ExecutorConfig> ctxExecutorConfigs; + std::vector<texec::ExecutorConfig> genExecutorConfigs; + for (auto in = 0; in < instanceNum; in++) + { + auto&& enginePath = in < contextNum ? contextEnginePaths.at(in) : genEnginePaths.at(in - contextNum); + auto decoderJsonConfig = tensorrt_llm::runtime::GptJsonConfig::parse(enginePath / "config.json"); + size_t instanceRanks = decoderJsonConfig.getWorldSize(); + std::vector<SizeType32> participateRank(instanceRanks); + std::vector<SizeType32> deviceIds; + if (deviceIdsForInstance.has_value()) + { + deviceIds = deviceIdsForInstance.value().at(in); + } + for (int i = 0; i < instanceRanks; i++) + { + startRank++; + participateRank.at(i) = startRank; + if (!deviceIdsForInstance.has_value()) + { + deviceIds.push_back((startRank - 1) % deviceCount); + } + } + texec::DynamicBatchConfig dynamicBatchConfig(benchmarkParams.enableBatchSizeTuning); + texec::SchedulerConfig schedulerConfig(capacitySchedulerPolicy, std::nullopt, dynamicBatchConfig); + texec::KvCacheConfig kvCacheConfig(benchmarkParams.enableBlockReuse, + benchmarkParams.maxTokensInPagedKvCache, benchmarkParams.maxAttentionWindowVec, + benchmarkParams.sinkTokenLength, benchmarkParams.freeGpuMemoryFractions.at(in), + benchmarkParams.kvHostCacheSize); + texec::ExtendedRuntimePerfKnobConfig extendedRuntimePerfKnobConfig(benchmarkParams.multiBlockMode, + benchmarkParams.enableContextFMHAFP32Acc, benchmarkParams.cudaGraphMode, + benchmarkParams.cudaGraphCacheSize); + texec::ExecutorConfig executorConfig(maxBeamWidth, schedulerConfig, kvCacheConfig, + benchmarkParams.enableChunekedContextVec.at(in).value_or(false)); + executorConfig.setGpuWeightsPercent(benchmarkParams.gpuWeightsPercent); + texec::OrchestratorConfig orchestratorConfig{mIsOrchestrator, "", nullptr, false}; + texec::ParallelConfig parallelConfig{tensorrt_llm::executor::CommunicationType::kMPI, + tensorrt_llm::executor::CommunicationMode::kORCHESTRATOR, deviceIds, participateRank, + orchestratorConfig}; + executorConfig.setParallelConfig(parallelConfig); + if (benchmarkParams.maxBatchSizes.at(in)) + { + executorConfig.setMaxBatchSize(benchmarkParams.maxBatchSizes.at(in).value()); + } + if (benchmarkParams.maxNumTokensVec.at(in)) + { + executorConfig.setMaxNumTokens(benchmarkParams.maxNumTokensVec.at(in).value()); + } + + executorConfig.setDecodingConfig( + texec::DecodingConfig(benchmarkParams.medusaChoices.has_value() ? texec::DecodingMode::Medusa() + : benchmarkParams.executorLookaheadConfig.has_value() ? texec::DecodingMode::Lookahead() + : texec::DecodingMode::Auto(), + benchmarkParams.executorLookaheadConfig, benchmarkParams.medusaChoices)); + executorConfig.setExtendedRuntimePerfKnobConfig(extendedRuntimePerfKnobConfig); + executorConfig.setCacheTransceiverConfig( + texec::CacheTransceiverConfig(texec::CacheTransceiverConfig::BackendType::DEFAULT)); + constexpr int maxIterationsForRequestStats = 1000; + if (mEnableCollectKvCacheTransferTime) + { + executorConfig.setRequestStatsMaxIterations(maxIterationsForRequestStats); + } + if (!benchmarkParams.enableCollectIterStats) + { + executorConfig.setIterStatsMaxIterations(0); + } + + if (in < contextNum) + { + ctxExecutorConfigs.push_back(executorConfig); + } + else + { + genExecutorConfigs.push_back(executorConfig); + } + } + + mDisaggExecutor = std::make_unique<DisaggExecutorOrchestrator>(contextEnginePaths, genEnginePaths, + ctxExecutorConfigs, genExecutorConfigs, hasContextAwaitThreads, hasGenAwaitThreads); + + if (mIsOrchestrator) + { + + if (mEnableCollectIterStats || mEnableCollectKvCacheTransferTime) + { + mCollectStatsThread = std::thread(&DisaggExecutorServer::collectStats, this); + } + } + tensorrt_llm::mpi::MpiComm::world().barrier(); + } + + std::vector<tensorrt_llm::executor::IdType> enqueueContext(std::vector<texec::Request> const& requests, + std::optional<int> selectContextId = std::nullopt, bool warmup = false, bool batch = false) + { + std::vector<SizeType32> inputLengths; + std::vector<SizeType32> maxNewTokens; + if (!warmup) + { + for (auto const& request : requests) + { + inputLengths.push_back(static_cast<SizeType32>(request.getInputTokenIds().size())); + maxNewTokens.push_back(request.getMaxTokens()); + } + } + auto const start = std::chrono::steady_clock::now(); + std::vector<tensorrt_llm::executor::IdType> globalReqIds + = mDisaggExecutor->enqueueContext(requests, selectContextId, batch); + if (!warmup) + { + for (size_t i = 0; i < requests.size(); ++i) + { + mRecorder->recordContextStart(inputLengths.at(i), maxNewTokens.at(i), globalReqIds.at(i), start); + } + } + mNumContextActive += requests.size(); + return globalReqIds; + } + + void enqueueGeneration(std::vector<texec::Request> const& requests, + std::vector<tensorrt_llm::executor::IdType> const& globalRequestIds, + std::optional<int> selectGenIdx = std::nullopt, bool warmup = false, bool batch = false) + { + TLLM_CHECK(globalRequestIds.size() == requests.size()); + auto const start = std::chrono::steady_clock::now(); + mDisaggExecutor->enqueueGeneration(requests, globalRequestIds, selectGenIdx, batch); + if (!warmup) + { + for (int i = 0; i < requests.size(); i++) + { + + mRecorder->recordGenStart(globalRequestIds.at(i), start); + } + } + mNumGenActive += requests.size(); + } + + std::vector<ResponseWithId> waitForContextResponse(SizeType32 numRequests, bool warmup = false) + { + std::vector<ResponseWithId> ret; + ret.reserve(numRequests); + while ((mNumContextActive != 0) || (mNumContextFinished < numRequests)) + { + auto responses = mDisaggExecutor->awaitContextResponses(mWaitSleep); + for (auto&& response : responses) + { + TLLM_CHECK(response.response.getResult().isFinal); + if (response.response.getResult().isFinal) + { + mNumContextActive--; + mNumContextFinished++; + } + if (!warmup) + { + mRecorder->recordContextEnd(response.gid, response.response.hasError()); + } + ret.emplace_back(std::move(response)); + } + } + return ret; + } + + void waitForGenResponse(SizeType32 numRequests, bool warmup = false) + { + while (mNumGenActive > 0 || (mNumGenFinished < numRequests)) + { + auto responses = mDisaggExecutor->awaitGenerationResponses(mWaitSleep); + for (auto&& response : responses) + { + if (response.response.getResult().isFinal) + { + mNumGenActive--; + mNumGenFinished++; + + if (!warmup) + { + mRecorder->recordGenEnd(response.gid, response.response); + } + } + else + { + // streaming + if (!warmup && !response.response.hasError()) + { + mRecorder->recordToken(response.gid, response.response); + } + } + } + } + } + + bool canEnqueue(int numSentRequests) const + { + return mIsOrchestrator && (!mConcurrency || (numSentRequests - mNumGenFinished < mConcurrency)); + } + + ~DisaggExecutorServer() + { + mShutdown = true; + if (mCollectStatsThread.joinable()) + { + mCollectStatsThread.join(); + } + } + + void resetNumFinished() + { + mNumContextFinished = 0; + mNumGenFinished = 0; + } + + void resetNumActive() + { + mNumContextActive = 0; + mNumGenActive = 0; + } + + void collectStats() const + { + while (!mShutdown) + { + std::vector<std::deque<tensorrt_llm::executor::IterationStats>> contextStats; + std::vector<std::deque<tensorrt_llm::executor::IterationStats>> generationStats; + std::vector<std::deque<tensorrt_llm::executor::RequestStatsPerIteration>> + generationRequestStatsPerIteration; + contextStats.reserve(mDisaggExecutor->getContextExecutors().size()); + for (auto&& executor : mDisaggExecutor->getContextExecutors()) + { + if (executor->canEnqueueRequests()) + { + contextStats.emplace_back(executor->getLatestIterationStats()); + } + } + generationStats.reserve(mDisaggExecutor->getGenExecutors().size()); + for (auto&& executor : mDisaggExecutor->getGenExecutors()) + { + if (executor->canEnqueueRequests()) + { + if (mEnableCollectIterStats) + { + generationStats.emplace_back(executor->getLatestIterationStats()); + } + if (mEnableCollectKvCacheTransferTime) + { + + generationRequestStatsPerIteration.emplace_back(executor->getLatestRequestStats()); + } + } + } + if (mEnableCollectIterStats) + { + for (std::size_t i = 0; i < contextStats.size(); i++) + { + auto const& iterStats = contextStats.at(i); + for (auto const& stat : iterStats) + { + SizeType32 numNewActiveRequests = stat.numNewActiveRequests; + if (numNewActiveRequests > 0) + { + auto avgQueueingTime + = static_cast<float>(stat.newActiveRequestsQueueLatencyMS / numNewActiveRequests); + std::vector<float> requestsQueueLatencyMS(numNewActiveRequests, avgQueueingTime); + mRecorder->recordContextQueueLatency(requestsQueueLatencyMS); + } + if (mLogIterationData) + { + TLLM_LOG_INFO( + "ctx_id %d, ctx_stat: %s", i, texec::JsonSerialization::toJsonStr(stat).c_str()); + } + } + } + + for (std::size_t i = 0; i < generationStats.size(); i++) + { + auto const& iterStats = generationStats.at(i); + for (auto const& stat : iterStats) + { + SizeType32 numNewActiveRequests = stat.numNewActiveRequests; + if (numNewActiveRequests > 0) + { + float avgQueueingTime + = static_cast<float>(stat.newActiveRequestsQueueLatencyMS / numNewActiveRequests); + std::vector<float> requestsQueueLatencyMS(numNewActiveRequests, avgQueueingTime); + mRecorder->recordGenQueueLatency(requestsQueueLatencyMS); + } + if (mLogIterationData) + { + TLLM_LOG_INFO( + "gen_id %d, gen_stat: %s", i, texec::JsonSerialization::toJsonStr(stat).c_str()); + } + } + } + } + + if (mEnableCollectKvCacheTransferTime) + { + for (std::size_t i = 0; i < generationRequestStatsPerIteration.size(); i++) + { + auto const& stats = generationRequestStatsPerIteration.at(i); + for (auto const& stat : stats) + { + std::vector<float> kvCacheTransferMs; + std::vector<float> kvCacheThroughput; + for (auto const& requestStat : stat.requestStats) + { + if (requestStat.stage == tensorrt_llm::executor::RequestStage::kGENERATION_COMPLETE) + { + kvCacheTransferMs.push_back( + static_cast<float>(requestStat.disServingStats->kvCacheTransferMS)); + kvCacheThroughput.push_back(static_cast<float>(requestStat.disServingStats->kvCacheSize) + * 8 / (static_cast<float>(requestStat.disServingStats->kvCacheTransferMS) / 1000) + / 1e9f); + } + } + if (kvCacheTransferMs.size() > 0) + { + mRecorder->recordKvCacheTransferLatency(kvCacheTransferMs); + } + if (kvCacheThroughput.size() > 0) + { + mRecorder->recordKvCacheThroughput(kvCacheThroughput); + } + if (mLogIterationData) + { + TLLM_LOG_INFO( + "gen_id %d, gen_req_stat: %s", i, texec::JsonSerialization::toJsonStr(stat).c_str()); + } + } + } + } + auto const waitSleep = std::chrono::milliseconds(50); + std::this_thread::sleep_for(waitSleep); + } + } + + std::unique_ptr<DisaggExecutorOrchestrator> const& getDisaggExecutor() const noexcept + { + return mDisaggExecutor; + } + +private: + std::unique_ptr<DisaggExecutorOrchestrator> mDisaggExecutor; + + std::atomic<bool> mShutdown{false}; + bool mIsOrchestrator{false}; + + std::shared_ptr<Recorder> mRecorder; + std::chrono::milliseconds mWaitSleep; + std::optional<int> mConcurrency; + bool mLogIterationData{false}; + bool const mEnableCollectKvCacheTransferTime; + bool const mEnableCollectIterStats; + std::thread mCollectStatsThread; + std::atomic<uint64_t> mNumGenFinished{0}; + std::atomic<uint64_t> mNumContextFinished{0}; + std::atomic<uint64_t> mNumGenActive{0}; + std::atomic<uint64_t> mNumContextActive{0}; +}; + +} // namespace + +void benchmark(std::vector<std::filesystem::path> const& contextEngineDirs, + std::vector<std::filesystem::path> const& generationEngineDirs, + std::optional<std::vector<std::vector<int>>> const& deviceIdsForInstances, std::string const& datasetPath, + std::string const& opCsvFile, int maxNumSamples, int beamWidth, int warmUp, std::optional<int32_t> const& eosId, + std::optional<int32_t> const& padId, BenchmarkParams const& benchmarkParams, + texec::CapacitySchedulerPolicy capacitySchedulerPolicy, std::chrono::milliseconds waitSleep, + bool returnContextLogits, bool returnGenerationLogits, std::optional<int> const staticEmulatedBatchSize, + bool logIterationData, std::optional<SizeType32> const maxPromptLen, bool hasContextAwait, bool hasGenAwait) +{ + + auto const& world = tensorrt_llm::mpi::MpiComm::world(); + auto worldRank = world.getRank(); + + // Load dataset + auto const samples = parseWorkloadJson(datasetPath, maxNumSamples, maxPromptLen); + auto const numSamples = samples.size(); + auto recorder = std::make_shared<Recorder>(opCsvFile, benchmarkParams.streaming, beamWidth, + benchmarkParams.enableCollectkvCacheTransferTime, benchmarkParams.enableCollectIterStats); + auto disaggExecutor = std::make_shared<DisaggExecutorServer>(contextEngineDirs, generationEngineDirs, + deviceIdsForInstances, beamWidth, capacitySchedulerPolicy, benchmarkParams, recorder, waitSleep, + logIterationData, hasContextAwait, hasGenAwait); + constexpr size_t numMap = 8; + std::vector<std::unordered_map<tensorrt_llm::executor::IdType, tensorrt_llm::executor::Request>> gidToRequestMaps( + numMap); + std::vector<std::mutex> mtxForMaps(numMap); + + auto fillRequestMap = [&](std::vector<tensorrt_llm::executor::IdType> const& reqIds, + std::vector<tensorrt_llm::executor::Request>&& requests) + { + TLLM_CHECK(reqIds.size() == requests.size()); + for (size_t i = 0; i < reqIds.size(); i++) + { + + size_t mapIdx = reqIds[i] % numMap; + std::scoped_lock<std::mutex> lock(mtxForMaps[mapIdx]); + gidToRequestMaps.at(mapIdx).emplace(reqIds[i], std::move(requests[i])); + } + }; + + auto makeGenRequest = [&](std::vector<ResponseWithId>&& contextResponse) + { + std::vector<tensorrt_llm::executor::IdType> gids; + gids.reserve(contextResponse.size()); + std::vector<tensorrt_llm::executor::Request> genRequest; + genRequest.reserve(contextResponse.size()); + for (auto&& ctxResponse : contextResponse) + { + gids.emplace_back(ctxResponse.gid); + size_t mapIdx = ctxResponse.gid % numMap; + + std::unique_lock<std::mutex> lock(mtxForMaps[mapIdx]); + TLLM_CHECK(gidToRequestMaps.at(mapIdx).find(ctxResponse.gid) != gidToRequestMaps.at(mapIdx).end()); + auto ctxRequest = std::move(gidToRequestMaps.at(mapIdx).at(ctxResponse.gid)); + gidToRequestMaps.at(mapIdx).erase(ctxResponse.gid); + lock.unlock(); + ctxRequest.setRequestType(tensorrt_llm::executor::RequestType::REQUEST_TYPE_GENERATION_ONLY); + ctxRequest.setContextPhaseParams(ctxResponse.response.getResult().contextPhaseParams.value()); + genRequest.emplace_back(std::move(ctxRequest)); + } + return std::make_pair(genRequest, gids); + }; + if (worldRank == 0) + { + { // warmup + TLLM_LOG_INFO("Warmup start"); + + size_t contextNum = contextEngineDirs.size(); + size_t generationNum = generationEngineDirs.size(); + for (auto con = 0; con < contextNum; con++) + { + for (auto gen = 0; gen < generationNum; gen++) + { + std::vector<tensorrt_llm::executor::Request> contextRequests; + contextRequests.reserve(warmUp); + for (int i = 0; i < warmUp; ++i) + { + contextRequests.emplace_back(makeExecutorContextRequest(samples[0], beamWidth, eosId, padId, + benchmarkParams.streaming, returnContextLogits, returnGenerationLogits, std::nullopt, + benchmarkParams.requestLookaheadConfig)); + } + auto reqIds = disaggExecutor->enqueueContext(contextRequests, con, true); + fillRequestMap(reqIds, std::move(contextRequests)); + auto contextResponse = disaggExecutor->waitForContextResponse(warmUp, true); + auto&& [genRequests, gids] = makeGenRequest(std::move(contextResponse)); + disaggExecutor->enqueueGeneration(genRequests, gids, gen, true); + disaggExecutor->waitForGenResponse(warmUp, true); + disaggExecutor->resetNumFinished(); + disaggExecutor->resetNumActive(); + } + } + + auto const warmUpWaitSleep = std::chrono::milliseconds(50); + std::this_thread::sleep_for(warmUpWaitSleep); + TLLM_LOG_INFO("Warmup done"); + } + + { + + auto timeDelays = computeTimeDelays(benchmarkParams, numSamples - 1); + + std::vector<texec::Request> contextRequests; + + for (std::size_t i = 0; i < numSamples; ++i) + { + std::optional<texec::LoraConfig> loraConfig = std::nullopt; + contextRequests.emplace_back(makeExecutorContextRequest(samples[i], beamWidth, eosId, padId, + benchmarkParams.streaming, returnContextLogits, returnGenerationLogits, loraConfig, + benchmarkParams.requestLookaheadConfig)); + } + + bool const hasDelay + = std::any_of(timeDelays.begin(), timeDelays.end(), [](auto const& delay) { return delay > 0.0; }); + disaggExecutor->resetNumFinished(); + disaggExecutor->resetNumActive(); + + recorder->reserve(numSamples); + recorder->initialize(); + if (!staticEmulatedBatchSize) + { + + std::thread waitContextResponseAndEnqueGenThread{[&]() + { + auto numRequest = numSamples; + while (numRequest > 0) + { + auto contextResponseWithIds + = disaggExecutor->getDisaggExecutor()->awaitContextResponses(waitSleep); + if (contextResponseWithIds.empty()) + { + continue; + } + for (auto&& contextResponseWithId : contextResponseWithIds) + { + recorder->recordContextEnd( + contextResponseWithId.gid, contextResponseWithId.response.hasError()); + } + numRequest -= contextResponseWithIds.size(); + auto&& [genReqeust, genGids] = makeGenRequest(std::move(contextResponseWithIds)); + disaggExecutor->enqueueGeneration(genReqeust, genGids); + } + }}; + + std::thread waitGenResponseThread{[&]() { disaggExecutor->waitForGenResponse(numSamples); }}; + int numSentRequests = 0; + while (numSentRequests < numSamples) + { + + if (disaggExecutor->canEnqueue(numSentRequests)) + { + auto gids = disaggExecutor->enqueueContext({contextRequests.at(numSentRequests)}); + fillRequestMap(gids, {contextRequests.at(numSentRequests)}); + + if (hasDelay && numSentRequests < numSamples - 1) + { + std::this_thread::sleep_for( + std::chrono::milliseconds(static_cast<int>(timeDelays.at(numSentRequests) * 1000))); + } + numSentRequests += 1; + } + } + waitContextResponseAndEnqueGenThread.join(); + waitGenResponseThread.join(); + } + else + { + TLLM_CHECK_WITH_INFO( + !hasDelay, "Executor benchmark doesn't support delays with emulated static batch sizes"); + auto numRequests = contextRequests.size(); + int maxBatchSize = staticEmulatedBatchSize.value(); + for (int req = 0; req < numRequests; req += maxBatchSize) + { + auto batchSize = std::min(static_cast<size_t>(maxBatchSize), numRequests - req); + + std::vector<texec::Request> requestsBatch(std::make_move_iterator(contextRequests.begin() + req), + std::make_move_iterator(contextRequests.begin() + req + static_cast<int64_t>(batchSize))); + // Enqueue in batches + + auto reqIds = disaggExecutor->enqueueContext(requestsBatch); + fillRequestMap(reqIds, std::move(requestsBatch)); + auto contextResponse = disaggExecutor->waitForContextResponse(static_cast<SizeType32>(batchSize)); + auto&& [genRequests, genReqIds] = makeGenRequest(std::move(contextResponse)); + disaggExecutor->enqueueGeneration(genRequests, genReqIds); + disaggExecutor->waitForGenResponse(static_cast<SizeType32>(batchSize)); + + // Wait for current batch to be done + } + } + } + recorder->finalize(); + // sleep for collect stats + if (benchmarkParams.enableCollectIterStats || benchmarkParams.enableCollectkvCacheTransferTime) + { + auto const collectWaitSleep = std::chrono::milliseconds(50); + std::this_thread::sleep_for(collectWaitSleep); + } + recorder->calculateMetrics(); + recorder->report(); + recorder->writeOpMetricsToCsv(); + } +} + +int main(int argc, char* argv[]) + +{ + cxxopts::Options options("TensorRT LLM DisaggServer Benchmark"); + options.add_options()("h,help", "Print usage"); + options.add_options()("context_engine_dirs", "Directories that store context engines,separator is a ,", + cxxopts::value<std::vector<std::string>>()); + options.add_options()("generation_engine_dirs", "Directories that store generation engines,separator is a , ", + cxxopts::value<std::vector<std::string>>()); + options.add_options()("device_ids_for_instances", + "device ids for each instances , example: \"[[0,1],[2,3],[4,5,6,7]]\" ", cxxopts::value<std::string>()); + options.add_options()("dataset", "Dataset that is used for benchmarking BatchManager.", + cxxopts::value<std::string>()->default_value("")); + options.add_options()( + "output_csv", "Write output metrics to CSV", cxxopts::value<std::string>()->default_value("")); + options.add_options()("max_num_samples", "maximum number of samples to use from dataset/generate", + cxxopts::value<int>()->default_value("100000")); + options.add_options()( + "beam_width", "Specify beam width you want to benchmark.", cxxopts::value<int>()->default_value("1")); + options.add_options()( + "warm_up", "Specify warm up iterations before benchmark starts.", cxxopts::value<int>()->default_value("2")); + options.add_options()( + "eos_id", "Specify the end-of-sequence token id.", cxxopts::value<TokenIdType>()->default_value("-1")); + options.add_options()("pad_id", "Specify the padding token id.", cxxopts::value<TokenIdType>()); + options.add_options()("max_tokens_in_paged_kvcache", "Max tokens in paged K-V Cache.", cxxopts::value<int>()); + options.add_options()( + "max_attention_window", "Max KV cache length per sequence", cxxopts::value<std::vector<int>>()); + options.add_options()("sink_token_len", "Sink token length in kv cache per sequence.", cxxopts::value<int>()); + options.add_options()( + "random_seed", "integer random seed for exponential time delays.", cxxopts::value<int>()->default_value("420")); + options.add_options()("kv_cache_free_gpu_mem_fractions", "K-V Cache Free Gpu Mem Fraction,each for per instance", + cxxopts::value<std::vector<float>>()); + options.add_options()("request_rate", + "request rate in reqs/sec. Skipping this arg or negative value will trigger offline/0-delay.", + cxxopts::value<float>()); + options.add_options()("concurrency", "Concurrent number of connections with the server.", cxxopts::value<int>()); + options.add_options()("max_batch_sizes", "The max runtime batch size when benchmarking, each for per instance", + cxxopts::value<std::vector<int>>()); + options.add_options()("max_num_tokens_per_instance", + "The max runtime number of tokens per batch when benchmarking, each for per instance", + cxxopts::value<std::vector<int>>()); + options.add_options()( + "enable_batch_size_tuning", "Dynamic tuning of batch size", cxxopts::value<bool>()->default_value("false")); + options.add_options()("enable_exp_delays", "Enables exponential delay distr to mimic real world request arrival", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("streaming", "Operate in streaming mode", cxxopts::value<bool>()->default_value("false")); + options.add_options()( + "enable_kv_cache_reuse", "Enables the KV cache reuse.", cxxopts::value<bool>()->default_value("false")); + options.add_options()("enable_chunked_context_per_instance", "Whether to enable context chunking for per instance", + cxxopts::value<std::vector<bool>>()->default_value("false")); + options.add_options()( + "return_context_logits", "Whether to return context logits.", cxxopts::value<bool>()->default_value("false")); + options.add_options()("return_generation_logits", "Whether to return generation logits.", + cxxopts::value<bool>()->default_value("false")); + + options.add_options()("scheduler_policy", + "Choose scheduler policy between max_utilization/guaranteed_no_evict/static_batch.", + cxxopts::value<std::string>()->default_value("guaranteed_no_evict")); + + options.add_options()("static_emulated_batch_size", + "Emulate static batching performance with the provided batch size.", cxxopts::value<SizeType32>()); + options.add_options()("log_level", "Choose log level between verbose/info/warning/error/internal_error.", + cxxopts::value<std::string>()->default_value("error")); + options.add_options()("log_iteration_data", "On each decoder iteration, print batch state metadata.", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("wait_sleep", "Specify how many milliseconds to sleep each iteration of waitForEmpty loop.", + cxxopts::value<int>()->default_value("25")); + options.add_options()("kv_host_cache_bytes", + "Size of secondary memory pool used for offloading kv cache blocks (in bytes).", + cxxopts::value<size_t>()->default_value("0")); + options.add_options()( + "max_prompt_len", "Truncate all prompts from dataset to the length specified.", cxxopts::value<SizeType32>()); + options.add_options()("gpu_weights_percent", + "Specify the percentage of weights that reside on GPU (from 0.0 to 1.0).", + cxxopts::value<float>()->default_value("1.0")); + options.add_options()( + "medusa_choices", "Medusa choices in the format of [[0], [0, 1], [0, 0, 1]]", cxxopts::value<std::string>()); + options.add_options()("multi_block_mode", + "Distribute the work across multiple CUDA thread-blocks on the GPU for masked MHA kernel", + cxxopts::value<bool>()->default_value("true")); + options.add_options()("cuda_graph_mode", "When enabled, inference is executed with cuda graph.", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("cuda_graph_cache_size", + "Specify how many cuda graphs are cached in the runtime. Larger cache gives better perf, but consumes more GPU " + "memory.", + cxxopts::value<SizeType32>()->default_value("0")); + options.add_options()("enable_context_fmha_fp32_acc", "Enable FMHA runner FP32 accumulation", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("executor_lookahead_config", + "lookahead config in the format of [max_window_size, max_ngram_size, max_verification_set_size]", + cxxopts::value<std::string>()); + options.add_options()("request_lookahead_config", + "lookahead config in the format of [max_window_size, max_ngram_size, max_verification_set_size], and each <= " + "executor lookahead config", + cxxopts::value<std::string>()); + options.add_options()("context_await", "When enabled, will has a thread to await context response.", + cxxopts::value<bool>()->default_value("true")); + options.add_options()("gen_await", "When enabled,will has a thread to await gen response.", + cxxopts::value<bool>()->default_value("true")); + options.add_options()("enable_collect_kvcache_transfer_time", "When enabled, will collect kvcache transfer time.", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("enable_collect_iter_stats", "When enabled, will collect iteration stats.", + cxxopts::value<bool>()->default_value("false")); + + auto result = options.parse(argc, argv); + + if ((result.count("context_engine_dirs") == 0) || (result.count("generation_engine_dirs") == 0)) + { + std::cout << options.help() << std::endl; + TLLM_LOG_ERROR("Please specify context engine and generation engine directory."); + return 1; + } + // Argument: Log level + auto logger = std::make_shared<TllmLogger>(); + auto const logLevel = result["log_level"].as<std::string>(); + if (logLevel == "verbose") + { + logger->setLevel(trt::ILogger::Severity::kVERBOSE); + } + else if (logLevel == "info") + { + logger->setLevel(trt::ILogger::Severity::kINFO); + } + else if (logLevel == "warning") + { + logger->setLevel(trt::ILogger::Severity::kWARNING); + } + else if (logLevel == "error") + { + logger->setLevel(trt::ILogger::Severity::kERROR); + } + else if (logLevel == "internal_error") + { + logger->setLevel(trt::ILogger::Severity::kINTERNAL_ERROR); + } + else + { + TLLM_LOG_ERROR("Unexpected log level: " + logLevel); + return 1; + } + + initTrtLlmPlugins(logger.get()); + + // Argument: Dataset + auto const datasetPath = result["dataset"].as<std::string>(); + auto const maxNumSamples = result["max_num_samples"].as<int>(); + + // Argument: Output metrics CSV + auto const opCsvFile = result["output_csv"].as<std::string>(); + + // Argument: beam width + auto const beamWidth = result["beam_width"].as<int>(); + TLLM_CHECK_WITH_INFO(beamWidth == 1, "Currently only support beamWidth=1"); + // Argument: wait_sleep + auto const waitSleep = std::chrono::milliseconds(result["wait_sleep"].as<int>()); + auto const hasContextAwait = result["context_await"].as<bool>(); + auto const hasGenAwait = result["gen_await"].as<bool>(); + BenchmarkParams benchmarkParams; + benchmarkParams.enableCollectkvCacheTransferTime = result["enable_collect_kvcache_transfer_time"].as<bool>(); + benchmarkParams.enableCollectIterStats = result["enable_collect_iter_stats"].as<bool>(); + + std::vector<std::string> contextEngineDirs = result["context_engine_dirs"].as<std::vector<std::string>>(); + std::vector<std::string> generationEngineDirs = result["generation_engine_dirs"].as<std::vector<std::string>>(); + if (tensorrt_llm::mpi::MpiComm::world().getRank() == 0) + { + std::string contextEngineStrings; + for (auto&& contextEngineDir : contextEngineDirs) + { + contextEngineStrings += contextEngineDir + ","; + } + std::string generationEnginesStrings; + for (auto&& genEngineDir : generationEngineDirs) + { + generationEnginesStrings += genEngineDir + ","; + } + TLLM_LOG_INFO( + "Will Launch benchmark with %d context engines and %d generation engines. Context Engines:%s ; Generation " + "Engines:%s ;", + contextEngineDirs.size(), generationEngineDirs.size(), contextEngineStrings.c_str(), + generationEnginesStrings.c_str()); + } + std::vector<std::filesystem::path> contextEnigePaths; + std::vector<std::filesystem::path> generationEnginePaths; + + contextEnigePaths.reserve(contextEngineDirs.size()); + + for (auto& contextEngineDir : contextEngineDirs) + { + + contextEnigePaths.emplace_back(contextEngineDir); + } + generationEnginePaths.reserve(generationEngineDirs.size()); + for (auto& genEngineDir : generationEngineDirs) + { + + generationEnginePaths.emplace_back(genEngineDir); + } + + int const instanceNum = contextEngineDirs.size() + generationEngineDirs.size(); + // Argument: Max tokens in paged K-V Cache + if (result.count("max_tokens_in_paged_kvcache")) + { + benchmarkParams.maxTokensInPagedKvCache = result["max_tokens_in_paged_kvcache"].as<int>(); + } + + // Argument: Max KV cache length + if (result.count("max_attention_window")) + { + benchmarkParams.maxAttentionWindowVec = result["max_attention_window"].as<std::vector<int>>(); + } + + // Argument: Sink token length + if (result.count("sink_token_len")) + { + benchmarkParams.sinkTokenLength = result["sink_token_len"].as<int>(); + } + + if (result.count("random_seed")) + { + benchmarkParams.randomSeed = result["random_seed"].as<int>(); + } + + // Argument: K-V Cache Free Gpu Mem Fraction + benchmarkParams.freeGpuMemoryFractions.resize(instanceNum); + if (result.count("kv_cache_free_gpu_mem_fractions")) + { + auto fractions = result["kv_cache_free_gpu_mem_fractions"].as<std::vector<float>>(); + TLLM_CHECK_WITH_INFO(fractions.size() == instanceNum || fractions.size() == 1, + "the number of fraction should be equal to the number of instances or equal to 1"); + for (int i = 0; i < instanceNum; i++) + { + benchmarkParams.freeGpuMemoryFractions.at(i) = fractions.size() == 1 ? fractions[0] : fractions[i]; + } + } + + // Argument: Enable dynamic tuning of batch size + benchmarkParams.enableBatchSizeTuning = result["enable_batch_size_tuning"].as<bool>(); + + // Argument: Enable KV cache reuse + benchmarkParams.enableBlockReuse = result["enable_kv_cache_reuse"].as<bool>(); + + // Argument: streaming + benchmarkParams.streaming = result["streaming"].as<bool>(); + + TLLM_CHECK_WITH_INFO(!(result.count("request_rate") && result.count("concurrency")), + "request_rate and concurrency cannot be specified at the same time."); + + // Argument: request rate + if (result.count("request_rate")) + { + benchmarkParams.requestRate = result["request_rate"].as<float>(); + } + + // Argument: concurrency + if (result.count("concurrency")) + { + benchmarkParams.concurrency = result["concurrency"].as<int>(); + } + + // Argument: max_batch_sizes + benchmarkParams.maxBatchSizes.resize(instanceNum); + if (result.count("max_batch_sizes")) + { + auto batchSizes = result["max_batch_sizes"].as<std::vector<int>>(); + TLLM_CHECK_WITH_INFO(batchSizes.size() == instanceNum || batchSizes.size() == 1, + "the number of batch size should be equal to the number of instances or equal to 1"); + for (int i = 0; i < instanceNum; i++) + { + benchmarkParams.maxBatchSizes.at(i) = batchSizes.size() == 1 ? batchSizes[0] : batchSizes[i]; + } + } + + // Argument: max_num_tokens_per_instance + benchmarkParams.maxNumTokensVec.resize(instanceNum); + if (result.count("max_num_tokens_per_instance")) + { + auto maxNumTokensVec = result["max_num_tokens_per_instance"].as<std::vector<int>>(); + TLLM_CHECK_WITH_INFO(maxNumTokensVec.size() == instanceNum || maxNumTokensVec.size() == 1, + "the number of max_num_tokens should be equal to the number of instances or equal to 1"); + for (int i = 0; i < instanceNum; i++) + { + benchmarkParams.maxNumTokensVec.at(i) + = maxNumTokensVec.size() == 1 ? maxNumTokensVec[0] : maxNumTokensVec[i]; + } + } + + benchmarkParams.enableExpDelays = result["enable_exp_delays"].as<bool>(); + + // Argument: Enable batch stats output + bool logIterationData = result["log_iteration_data"].as<bool>(); + + // Argument: Enable chunked context + benchmarkParams.enableChunekedContextVec.resize(instanceNum); + if (result.count("enable_chunked_context_per_instance")) + { + auto enableChunkedContextVec = result["enable_chunked_context_per_instance"].as<std::vector<bool>>(); + + TLLM_CHECK_WITH_INFO(enableChunkedContextVec.size() == instanceNum || enableChunkedContextVec.size() == 1, + "the number of enable_chunked_context_per_instance should be equal to the number of instances or equal to " + "1"); + for (int i = 0; i < instanceNum; i++) + { + benchmarkParams.enableChunekedContextVec.at(i) + = enableChunkedContextVec.size() == 1 ? enableChunkedContextVec[0] : enableChunkedContextVec[i]; + } + } + // Argument: Enable return context logits + bool returnContextLogits = result["return_context_logits"].as<bool>(); + TLLM_CHECK_WITH_INFO(returnContextLogits == false, "Currently disaggServer don't support returnContextLogits!"); + // Argument: Enable return context logits + bool returnGenerationLogits = result["return_generation_logits"].as<bool>(); + TLLM_CHECK_WITH_INFO( + returnGenerationLogits == false, "Currently disaggServer don't support returnGenerationLogits!"); + + if (result.count("lora_dir")) + { + TLLM_CHECK_WITH_INFO(false, "Currently disaggServer don't support lora!"); + benchmarkParams.loraDir = result["lora_dir"].as<std::string>(); + } + if (result.count("lora_host_cache_bytes")) + { + TLLM_CHECK_WITH_INFO(false, "Currently disaggServer don't support lora!"); + + benchmarkParams.loraHostCacheSize = result["lora_host_cache_bytes"].as<size_t>(); + } + if (result.count("lora_num_device_mod_layers")) + { + TLLM_CHECK_WITH_INFO(false, "Currently disaggServer don't support lora!"); + + benchmarkParams.loraDeviceNumModLayers = result["lora_num_device_mod_layers"].as<SizeType32>(); + } + + // Argument: How many KV cache blocks (as fraction of number of GPU kv cache blocks). + benchmarkParams.kvHostCacheSize = result["kv_host_cache_bytes"].as<size_t>(); + TLLM_CHECK_WITH_INFO( + benchmarkParams.kvHostCacheSize == false, "Currently disaggServer don't support kv_host_cache!"); + + // Argument: Medusa choices for the Medusa speculative decoding. + if (result.count("medusa_choices")) + { + TLLM_CHECK_WITH_INFO(false, "Currently disaggServer don't support medusa!"); + + benchmarkParams.medusaChoices = parseVectorOfVectors(result["medusa_choices"].as<std::string>()); + } + if (result.count("executor_lookahead_config")) + { + TLLM_CHECK_WITH_INFO(false, "Currently disaggServer don't support lookhead!"); + + benchmarkParams.executorLookaheadConfig + = parseLookaheadConfig(result["executor_lookahead_config"].as<std::string>()); + } + if (result.count("request_lookahead_config")) + { + TLLM_CHECK_WITH_INFO(false, "Currently disaggServer don't support lookhead!"); + + benchmarkParams.requestLookaheadConfig + = parseLookaheadConfig(result["request_lookahead_config"].as<std::string>()); + } + + // Argument: multi_block_mode + benchmarkParams.multiBlockMode = result["multi_block_mode"].as<bool>(); + + // Argument: enable_context_fmha_fp32_acc + benchmarkParams.enableContextFMHAFP32Acc = result["enable_context_fmha_fp32_acc"].as<bool>(); + + // Argument: cuda_graph_mode + benchmarkParams.cudaGraphMode = result["cuda_graph_mode"].as<bool>(); + + // Argument: cuda_graph_cache_size + benchmarkParams.cudaGraphCacheSize = result["cuda_graph_cache_size"].as<SizeType32>(); + + std::optional<TokenIdType> padId; + // Argument: Padding token id + if (result.count("pad_id")) + { + padId = result["pad_id"].as<TokenIdType>(); + } + + // Argument: End-of-sentence token id + std::optional<TokenIdType> eosId = result["eos_id"].as<TokenIdType>(); + + std::optional<std::chrono::milliseconds> batchTimeout; + + std::optional<SizeType32> staticEmulatedBatchSize; + // Argument: Static emulated batch size + if (result.count("static_emulated_batch_size")) + { + staticEmulatedBatchSize = result["static_emulated_batch_size"].as<SizeType32>(); + } + + // Argument: Scheduler policy + texec::CapacitySchedulerPolicy capacitySchedulerPolicy; + auto const capacitySchedulerPolicyArg = result["scheduler_policy"].as<std::string>(); + if (capacitySchedulerPolicyArg == "max_utilization") + { + capacitySchedulerPolicy = texec::CapacitySchedulerPolicy::kMAX_UTILIZATION; + } + else if (capacitySchedulerPolicyArg == "guaranteed_no_evict") + { + capacitySchedulerPolicy = texec::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT; + } + else if (capacitySchedulerPolicyArg == "static_batch") + { + capacitySchedulerPolicy = texec::CapacitySchedulerPolicy::kSTATIC_BATCH; + } + else + { + TLLM_LOG_ERROR("Unexpected scheduler policy: " + capacitySchedulerPolicyArg); + return 1; + } + + // Argument: max_prompt_len + std::optional<SizeType32> maxPromptLen; + if (result.count("max_prompt_len")) + { + maxPromptLen = result["max_prompt_len"].as<SizeType32>(); + } + + // Argument: GPU weights percentage + auto gpuWeightsPercent = result["gpu_weights_percent"].as<float>(); + if (gpuWeightsPercent < 0 || gpuWeightsPercent > 1) + { + TLLM_LOG_ERROR("--gpu_weights_percent must be between 0.0 and 1.0 but got: %f", gpuWeightsPercent); + return 1; + } + benchmarkParams.gpuWeightsPercent = gpuWeightsPercent; + + std::optional<std::vector<std::vector<int>>> deviceIdsForInstance = std::nullopt; + if (result.count("device_ids_for_instances")) + { + deviceIdsForInstance = parseVectorOfVectors(result["device_ids_for_instances"].as<std::string>()); + } + benchmark(contextEnigePaths, generationEnginePaths, deviceIdsForInstance, datasetPath, opCsvFile, maxNumSamples, + beamWidth, result["warm_up"].as<int>(), eosId, padId, benchmarkParams, capacitySchedulerPolicy, waitSleep, + returnContextLogits, returnContextLogits, staticEmulatedBatchSize, logIterationData, maxPromptLen, + hasContextAwait, hasGenAwait); +} diff --git a/benchmarks/cpp/gptManagerBenchmark.cpp b/benchmarks/cpp/gptManagerBenchmark.cpp new file mode 100644 index 000000000000..287cbba343ce --- /dev/null +++ b/benchmarks/cpp/gptManagerBenchmark.cpp @@ -0,0 +1,1557 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/tensor.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/gptJsonConfig.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tensorrt_llm/runtime/utils/numpyUtils.h" +#include "tensorrt_llm/runtime/worldConfig.h" +#include "utils/utils.h" + +#include <chrono> +#include <cstdint> +#include <cxxopts.hpp> +#include <iostream> +#include <memory> +#include <nlohmann/json.hpp> +#include <numeric> +#include <optional> +#include <string> +#include <thread> +#include <utility> + +using namespace tensorrt_llm::batch_manager; +using namespace tensorrt_llm::runtime; +using namespace tensorrt_llm::benchmark; +namespace texec = tensorrt_llm::executor; +namespace trt = nvinfer1; +namespace fs = std::filesystem; + +namespace +{ + +using TensorPtr = ITensor::SharedPtr; + +class LoraLib +{ +public: + LoraLib(std::string const& loraDir) + : mLoraDir(loraDir) + , mBufferManager(std::make_shared<CudaStream>()) + , mTaskPaths(parseDirPaths(mLoraDir)) + , mLoras(readLoras(mTaskPaths)) + { + } + + TensorPtr getLoraWeights(uint64_t taskId) const + { + return mLoras.at(taskId).first; + } + + TensorPtr getLoraConfig(uint64_t taskId) const + { + return mLoras.at(taskId).second; + } + + void clear() + { + mLoras.clear(); + } + + std::map<uint64_t, std::pair<TensorPtr, TensorPtr>> const& getLoras() + { + return mLoras; + } + +private: + std::string const mLoraDir; + BufferManager mBufferManager; + std::map<uint64_t, fs::path> mTaskPaths; + std::map<uint64_t, std::pair<TensorPtr, TensorPtr>> mLoras; + + std::map<uint64_t, std::pair<TensorPtr, TensorPtr>> readLoras(std::map<uint64_t, fs::path> taskPaths) + { + std::map<uint64_t, std::pair<TensorPtr, TensorPtr>> loras; + for (auto const& [id, p] : taskPaths) + { + TensorPtr loraWeights + = utils::loadNpy(mBufferManager, (p / "model.lora_weights.npy").string(), MemoryType::kCPU); + TensorPtr loraConfig + = utils::loadNpy(mBufferManager, (p / "model.lora_config.npy").string(), MemoryType::kCPU); + loras.insert_or_assign(id, std::make_pair(loraWeights, loraConfig)); + } + return loras; + } + + std::map<uint64_t, fs::path> parseDirPaths(std::string const& loraDir) + { + std::map<uint64_t, fs::path> taskPaths; + if (loraDir == "") + { + return taskPaths; + } + for (auto const& entry : fs::recursive_directory_iterator(loraDir)) + { + if (entry.is_directory()) + { + auto taskId = parseId(entry.path()); + taskPaths.insert_or_assign(taskId, entry.path()); + } + } + return taskPaths; + } + + uint64_t parseId(fs::path p) + { + auto fn = p.filename().string(); + auto dashPos = fn.find_first_of("-"); + std::string idStr = fn; + if (dashPos != std::string::npos) + { + auto idStr = fn.substr(0, dashPos); + } + uint64_t id = static_cast<uint64_t>(std::stoi(idStr)); + return id; + } +}; + +} // namespace + +struct BenchInfo +{ + BenchInfo() = default; + + BenchInfo(int inputLength, std::chrono::time_point<std::chrono::steady_clock> start) + : inputLength(inputLength) + , start(start) + { + } + + int inputLength; + int outputLength{0}; + std::chrono::time_point<std::chrono::steady_clock> start; + std::chrono::time_point<std::chrono::steady_clock> end; + std::chrono::time_point<std::chrono::steady_clock> firstTokenTs; + float latency{}; // millisecond + bool hasError{false}; + float firstTokenLatency{}; + std::optional<float> avgGenT2TLatency{}; + bool firstTokenSeen{false}; + SizeType32 decodingIter{0}; +}; + +class Recorder +{ + using TensorPtr = ITensor::SharedPtr; + +public: + explicit Recorder(std::string opCsvFile, bool streaming = false, int beamWidth = 1, + std::string responsesJsonFile = "", bool excludeInputInOutput = false) + : mOpCsvFile(std::move(opCsvFile)) + , mStreaming(streaming) + , mBeamWidth(beamWidth) + , mRespJsonFile(std::move(responsesJsonFile)) + , mOutputHasInput(!excludeInputInOutput) + { + } + + void initialize() + { + mStart = std::chrono::steady_clock::now(); + mRequestsQueueingLatencies.clear(); + } + + void finalize() + { + mEnd = std::chrono::steady_clock::now(); + } + + void recordQueueLatency(std::vector<float> const& latencies) + { + mRequestsQueueingLatencies.insert(mRequestsQueueingLatencies.end(), latencies.begin(), latencies.end()); + } + + // number of output tokens not calculated from output sequence here, instead set to max_output_len + // - if eos_id == -1 (default behavior), this is correct since output seq will have max permissible length. + // - However, if eos_id != -1, the token size of output sequence may be less than max_output_len, and token + // throughput may be inaccurate + void recordStart( + SizeType32 inputLength, uint64_t requestId, std::chrono::time_point<std::chrono::steady_clock> const& start) + { + TLLM_CHECK_WITH_INFO(mRequestBenchInfos.find(requestId) == mRequestBenchInfos.end(), + "Request %lu already exists in record before start, please report a bug to developers.", requestId); + std::lock_guard<std::mutex> const lock(mRequestBenchInfosMutex); + mRequestBenchInfos[requestId] = BenchInfo(inputLength, start); + } + + void recordToken( + texec::Response const& response, std::chrono::time_point<std::chrono::steady_clock> const& tokenTime) + { + auto const requestId = response.getRequestId(); + auto outputTokenIds = response.getResult().outputTokenIds; + + int32_t outputLength = 1; + for (auto const& beam : outputTokenIds) + { + outputLength = std::max(static_cast<int32_t>(beam.size()), outputLength); + } + + std::lock_guard<std::mutex> const lock(mRequestBenchInfosMutex); + mRequestBenchInfos[requestId].outputLength += outputLength; + + if (!mRequestBenchInfos[requestId].firstTokenSeen) + { + mRequestBenchInfos[requestId].firstTokenTs = tokenTime; + mRequestBenchInfos[requestId].firstTokenSeen = true; + } + + mRequestBenchInfos[requestId].decodingIter += 1; + } + + void recordEnd(texec::Response const& response, std::chrono::time_point<std::chrono::steady_clock> const& end) + { + auto const requestId = response.getRequestId(); + // Get the actual output length + if (!response.hasError()) + { + if (!mStreaming) + { + TLLM_LOG_DEBUG("response.getResult().outputTokenIds"); + auto outputTokenIds = response.getResult().outputTokenIds; + + int32_t outSeqLen = 0; + for (auto const& beam : outputTokenIds) + { + outSeqLen = std::max(static_cast<int32_t>(beam.size()), outSeqLen); + } + if (mOutputHasInput) + { + int inputSeqLen = mRequestBenchInfos[requestId].inputLength; + outSeqLen -= inputSeqLen; + } + std::lock_guard<std::mutex> const lock(mRequestBenchInfosMutex); + mRequestBenchInfos[requestId].outputLength = outSeqLen; + mRequestBenchInfos[requestId].decodingIter = response.getResult().decodingIter; + + // We record the first beam for the response file + mResponseTensors[requestId] = outputTokenIds[0]; + } + else + { + TLLM_CHECK_WITH_INFO(mBeamWidth == 1, "gptManagerBenchmark streaming mode does not support beam > 1"); + this->recordToken(response, end); + } + } + + std::lock_guard<std::mutex> const lock(mRequestBenchInfosMutex); + mRequestBenchInfos[requestId].end = end; + mRequestBenchInfos[requestId].hasError = response.hasError(); + } + + float calcPercentile(std::vector<float> const& latencies, int percentile) + { + int const index = static_cast<int>(std::ceil((percentile / 100.0) * latencies.size())) - 1; + return latencies[index]; + } + + void calculateLatencies() + { + for (auto& reqInfo : mRequestBenchInfos) + { + reqInfo.second.latency + = std::chrono::duration<float, std::milli>(reqInfo.second.end - reqInfo.second.start).count(); + if (mStreaming) + { + reqInfo.second.firstTokenLatency + = std::chrono::duration<float, std::milli>(reqInfo.second.firstTokenTs - reqInfo.second.start) + .count(); + if (reqInfo.second.outputLength > 1) + { + reqInfo.second.avgGenT2TLatency + = std::chrono::duration<float, std::milli>(reqInfo.second.end - reqInfo.second.firstTokenTs) + .count() + / static_cast<float>(reqInfo.second.outputLength - 1); + } + } + } + } + + void calculateMetrics() + { + calculateLatencies(); + + std::vector<float> reqLatencies; + std::vector<float> ftLatencies; + std::vector<float> genT2TLatencies; + std::vector<float> userTokensPerSecond; + + int totalOutputTokens{0}; + int totalDecodingIter{0}; + mNumErrorSamples = 0; + mNumSamples = 0; + for (auto reqInfo : mRequestBenchInfos) + { + if (!reqInfo.second.hasError) + { + reqLatencies.push_back(reqInfo.second.latency); + totalOutputTokens += reqInfo.second.outputLength; + totalDecodingIter += reqInfo.second.decodingIter; + + if (mStreaming) + { + ftLatencies.push_back(reqInfo.second.firstTokenLatency); + + if (reqInfo.second.avgGenT2TLatency) + { + genT2TLatencies.push_back(reqInfo.second.avgGenT2TLatency.value()); + } + if (reqInfo.second.avgGenT2TLatency.value() > 0) + { + userTokensPerSecond.push_back(1000.F / reqInfo.second.avgGenT2TLatency.value()); + } + } + ++mNumSamples; + } + else + { + ++mNumErrorSamples; + } + } + + mTotalLatency = std::chrono::duration<float, std::milli>(mEnd - mStart).count(); + mSeqThroughput = mNumSamples / (mTotalLatency / 1000); + mTokenThroughput = totalOutputTokens / (mTotalLatency / 1000); + mAcceptanceRate = totalDecodingIter + ? (static_cast<float>(totalOutputTokens) / static_cast<float>(totalDecodingIter)) + : 0.0f; + + mAvgSeqLatency = std::accumulate(reqLatencies.begin(), reqLatencies.end(), 0.F) / reqLatencies.size(); + + std::sort(reqLatencies.begin(), reqLatencies.end()); + + mP99SeqLatency = calcPercentile(reqLatencies, 99); + mP90SeqLatency = calcPercentile(reqLatencies, 90); + mP50SeqLatency = calcPercentile(reqLatencies, 50); + mMaxSeqLatency = reqLatencies.back(); + mMinSeqLatency = reqLatencies.front(); + + if (mStreaming) + { + mAvgFtLatency = std::accumulate(ftLatencies.begin(), ftLatencies.end(), 0.F) / ftLatencies.size(); + + std::sort(ftLatencies.begin(), ftLatencies.end()); + + mP99FtLatency = calcPercentile(ftLatencies, 99); + mP90FtLatency = calcPercentile(ftLatencies, 90); + mP50FtLatency = calcPercentile(ftLatencies, 50); + mMaxFtLatency = ftLatencies.back(); + mMinFtLatency = ftLatencies.front(); + + if (!genT2TLatencies.empty()) + { + mAvgGenT2TLatency + = std::accumulate(genT2TLatencies.begin(), genT2TLatencies.end(), 0.F) / genT2TLatencies.size(); + + std::sort(genT2TLatencies.begin(), genT2TLatencies.end()); + + mP99GenT2TLatency = calcPercentile(genT2TLatencies, 99); + mP90GenT2TLatency = calcPercentile(genT2TLatencies, 90); + mP50GenT2TLatency = calcPercentile(genT2TLatencies, 50); + mMaxGenT2TLatency = genT2TLatencies.back(); + mMinGenT2TLatency = genT2TLatencies.front(); + } + + if (!userTokensPerSecond.empty()) + { + mAvgUserTokensPerSecond = std::accumulate(userTokensPerSecond.begin(), userTokensPerSecond.end(), 0.F) + / userTokensPerSecond.size(); + std::sort(userTokensPerSecond.begin(), userTokensPerSecond.end()); + mP99UserTokensPerSecond = calcPercentile(userTokensPerSecond, 99); + mP90UserTokensPerSecond = calcPercentile(userTokensPerSecond, 90); + mP50UserTokensPerSecond = calcPercentile(userTokensPerSecond, 50); + mMaxUserTokensPerSecond = userTokensPerSecond.back(); + mMinUserTokensPerSecond = userTokensPerSecond.front(); + } + + mAvgReqQueueingLatency + = std::accumulate(mRequestsQueueingLatencies.begin(), mRequestsQueueingLatencies.end(), 0.F) + / mRequestsQueueingLatencies.size(); + std::sort(mRequestsQueueingLatencies.begin(), mRequestsQueueingLatencies.end()); + mP99ReqQueueingLatency = calcPercentile(mRequestsQueueingLatencies, 99); + mP90ReqQueueingLatency = calcPercentile(mRequestsQueueingLatencies, 90); + mP50ReqQueueingLatency = calcPercentile(mRequestsQueueingLatencies, 50); + mMaxReqQueueingLatency = mRequestsQueueingLatencies.back(); + mMinReqQueueingLatency = mRequestsQueueingLatencies.front(); + } + } + + void report() + { + + printf("[BENCHMARK] num_samples %d\n", mNumSamples); + printf("[BENCHMARK] num_error_samples %d\n", mNumErrorSamples); + printf("\n[BENCHMARK] num_samples %d\n", mNumSamples); + printf("[BENCHMARK] total_latency(ms) %.2f\n", mTotalLatency); + printf("[BENCHMARK] seq_throughput(seq/sec) %.2f\n", mSeqThroughput); + printf("[BENCHMARK] token_throughput(token/sec) %.2f\n", mTokenThroughput); + printf("[BENCHMARK] avg_acceptance_rate(tokens/decoding steps) %.2f\n\n", mAcceptanceRate); + + printf("[BENCHMARK] avg_sequence_latency(ms) %.2f\n", mAvgSeqLatency); + printf("[BENCHMARK] max_sequence_latency(ms) %.2f\n", mMaxSeqLatency); + printf("[BENCHMARK] min_sequence_latency(ms) %.2f\n", mMinSeqLatency); + printf("[BENCHMARK] p99_sequence_latency(ms) %.2f\n", mP99SeqLatency); + printf("[BENCHMARK] p90_sequence_latency(ms) %.2f\n", mP90SeqLatency); + printf("[BENCHMARK] p50_sequence_latency(ms) %.2f\n\n", mP50SeqLatency); + + if (mStreaming) + { + printf("[BENCHMARK] avg_time_to_first_token(ms) %.2f\n", mAvgFtLatency); + printf("[BENCHMARK] max_time_to_first_token(ms) %.2f\n", mMaxFtLatency); + printf("[BENCHMARK] min_time_to_first_token(ms) %.2f\n", mMinFtLatency); + printf("[BENCHMARK] p99_time_to_first_token(ms) %.2f\n", mP99FtLatency); + printf("[BENCHMARK] p90_time_to_first_token(ms) %.2f\n", mP90FtLatency); + printf("[BENCHMARK] p50_time_to_first_token(ms) %.2f\n\n", mP50FtLatency); + + printf("[BENCHMARK] avg_inter_token_latency(ms) %.2f\n", mAvgGenT2TLatency); + printf("[BENCHMARK] max_inter_token_latency(ms) %.2f\n", mMaxGenT2TLatency); + printf("[BENCHMARK] min_inter_token_latency(ms) %.2f\n", mMinGenT2TLatency); + printf("[BENCHMARK] p99_inter_token_latency(ms) %.2f\n", mP99GenT2TLatency); + printf("[BENCHMARK] p90_inter_token_latency(ms) %.2f\n", mP90GenT2TLatency); + printf("[BENCHMARK] p50_inter_token_latency(ms) %.2f\n\n", mP50GenT2TLatency); + + printf("[BENCHMARK] avg_user_tokens_per_second(tokens/sec/user) %.2f\n", mAvgUserTokensPerSecond); + printf("[BENCHMARK] max_user_tokens_per_second(tokens/sec/user) %.2f\n", mMaxUserTokensPerSecond); + printf("[BENCHMARK] min_user_tokens_per_second(tokens/sec/user) %.2f\n", mMinUserTokensPerSecond); + printf("[BENCHMARK] p99_user_tokens_per_second(tokens/sec/user) %.2f\n", mP99UserTokensPerSecond); + printf("[BENCHMARK] p90_user_tokens_per_second(tokens/sec/user) %.2f\n", mP90UserTokensPerSecond); + printf("[BENCHMARK] p50_user_tokens_per_second(tokens/sec/user) %.2f\n\n", mP50UserTokensPerSecond); + + printf("[BENCHMARK] avg_request_queueing_latency(ms) %.2f\n", mAvgReqQueueingLatency); + printf("[BENCHMARK] max_request_queueing_latency(ms) %.2f\n", mMaxReqQueueingLatency); + printf("[BENCHMARK] min_request_queueing_latency(ms) %.2f\n", mMinReqQueueingLatency); + printf("[BENCHMARK] p99_request_queueing_latency(ms) %.2f\n", mP99ReqQueueingLatency); + printf("[BENCHMARK] p90_request_queueing_latency(ms) %.2f\n", mP90ReqQueueingLatency); + printf("[BENCHMARK] p50_request_queueing_latency(ms) %.2f\n\n", mP50ReqQueueingLatency); + } + } + + void writeOpMetricsToCsv() + { + if (!mOpCsvFile.empty()) + { + std::vector<std::string> headers = {"num_samples", "num_error_samples", "total_latency(ms)", + "seq_throughput(seq/sec)", "token_throughput(token/sec)", "avg_sequence_latency(ms)", + "max_sequence_latency(ms)", "min_sequence_latency(ms)", "p99_sequence_latency(ms)", + "p90_sequence_latency(ms)", "p50_sequence_latency(ms)", "avg_acceptance_rate(tokens/decoding steps)"}; + + if (mStreaming) + { + std::vector<std::string> streamingHeaders = { + "avg_time_to_first_token(ms)", + "max_time_to_first_token(ms)", + "min_time_to_first_token(ms)", + "p99_time_to_first_token(ms)", + "p90_time_to_first_token(ms)", + "p50_time_to_first_token(ms)", + "avg_inter_token_latency(ms)", + "max_inter_token_latency(ms)", + "min_inter_token_latency(ms)", + "p99_inter_token_latency(ms)", + "p90_inter_token_latency(ms)", + "p50_inter_token_latency(ms)", + "avg_user_tokens_per_second(tokens/sec/user)", + "max_user_tokens_per_second(tokens/sec/user)", + "min_user_tokens_per_second(tokens/sec/user)", + "p99_user_tokens_per_second(tokens/sec/user)", + "p90_user_tokens_per_second(tokens/sec/user)", + "p50_user_tokens_per_second(tokens/sec/user)", + }; + + headers.insert(headers.end(), streamingHeaders.begin(), streamingHeaders.end()); + } + + std::ofstream outputFile(mOpCsvFile); + + if (outputFile.is_open()) + { + for (auto const& header : headers) + { + outputFile << header << ","; + } + outputFile << "\n"; + outputFile << mNumSamples << "," << mNumErrorSamples << "," << mTotalLatency << "," << mSeqThroughput + << "," << mTokenThroughput << "," << mAvgSeqLatency << "," << mMaxSeqLatency << "," + << mMinSeqLatency << "," << mP99SeqLatency << "," << mP90SeqLatency << "," << mP50SeqLatency + << "," << mAcceptanceRate; + if (mStreaming) + { + outputFile << "," << mAvgFtLatency << "," << mMaxFtLatency << "," << mMinFtLatency << "," + << mP99FtLatency << "," << mP90FtLatency << "," << mP50FtLatency << "," + << mAvgGenT2TLatency << "," << mMaxGenT2TLatency << "," << mMinGenT2TLatency << "," + << mP99GenT2TLatency << "," << mP90GenT2TLatency << "," << mP50GenT2TLatency << "," + << mAvgUserTokensPerSecond << "," << mMaxUserTokensPerSecond << "," + << mMinUserTokensPerSecond << "," << mP99UserTokensPerSecond << "," + << mP90UserTokensPerSecond << "," << mP50UserTokensPerSecond << ","; + } + + outputFile << "\n"; + } + else + { + std::cerr << "Error opening file '" << mOpCsvFile << "' for writing.\n"; + } + } + } + + void dumpResponseSeqs() + { + if (mRespJsonFile.empty()) + return; + nlohmann::json jsonResponses = nlohmann::json::array(); + for (auto const& [respId, respTokensTensor] : mResponseTensors) + { + auto respTokens = mResponseTensors[respId]; + int respLength = respTokens.size(); + int* respBufferPtr = respTokens.data(); + + if (mOutputHasInput) + { + int inputSeqLen = mRequestBenchInfos[respId].inputLength; + respBufferPtr += inputSeqLen; + respLength -= inputSeqLen; + } + + std::vector<int32_t> outputTokens(respLength); + std::copy(respBufferPtr, respBufferPtr + respLength, outputTokens.begin()); + + nlohmann::json currResp; + currResp["response_id"] = respId; + currResp["response_tokens"] = outputTokens; + jsonResponses.push_back(currResp); + } + std::ofstream outFile(mRespJsonFile); + outFile << jsonResponses; + outFile.close(); + } + +private: + std::unordered_map<uint64_t, BenchInfo> mRequestBenchInfos; + + std::chrono::time_point<std::chrono::steady_clock> mStart; + std::chrono::time_point<std::chrono::steady_clock> mEnd; + int mNumSamples{}; + int mNumErrorSamples{}; + float mTotalLatency{}; + float mSeqThroughput{}; + float mAvgSeqLatency{}; + float mAvgGenT2TLatency{}; + float mAvgUserTokensPerSecond{}; + float mAvgFtLatency{}; + float mTokenThroughput{}; + float mAcceptanceRate{}; + float mP99SeqLatency{}; + float mP90SeqLatency{}; + float mP50SeqLatency{}; + float mMaxSeqLatency{}; + float mMinSeqLatency{}; + float mP99FtLatency{}; + float mP90FtLatency{}; + float mP50FtLatency{}; + float mMaxFtLatency{}; + float mMinFtLatency{}; + float mP99GenT2TLatency{}; + float mP90GenT2TLatency{}; + float mP50GenT2TLatency{}; + float mMaxGenT2TLatency{}; + float mMinGenT2TLatency{}; + float mP99UserTokensPerSecond{}; + float mP90UserTokensPerSecond{}; + float mP50UserTokensPerSecond{}; + float mMaxUserTokensPerSecond{}; + float mMinUserTokensPerSecond{}; + float mAvgReqQueueingLatency{}; + float mP99ReqQueueingLatency{}; + float mP90ReqQueueingLatency{}; + float mP50ReqQueueingLatency{}; + float mMaxReqQueueingLatency{}; + float mMinReqQueueingLatency{}; + std::vector<float> mRequestsQueueingLatencies{}; + + std::string mOpCsvFile; + bool mStreaming; + int mBeamWidth; + std::string mRespJsonFile; + std::unordered_map<uint64_t, texec::VecTokens> mResponseTensors; + bool mOutputHasInput; + std::mutex mRequestBenchInfosMutex; + +}; // class Recorder + +class ExecutorServer +{ +public: + ExecutorServer(std::optional<std::filesystem::path> const& decoderTrtEnginePath, + std::optional<std::filesystem::path> const& encoderTrtEnginePath, texec::BatchingType batchingType, + int32_t maxBeamWidth, texec::CapacitySchedulerPolicy capacitySchedulerPolicy, + BenchmarkParams const& benchmarkParams, std::shared_ptr<Recorder> recorder, std::chrono::milliseconds waitSleep, + bool logIterationData, texec::ModelType executorModelType) + : mRecorder(std::move(recorder)) + , mWaitSleep(waitSleep) + , mConcurrency(benchmarkParams.concurrency) + , mActiveCount(0) + , mNumFinished(0) + , mShutdown(false) + , mLogIterationData(logIterationData) + { + texec::DynamicBatchConfig dynamicBatchConfig( + benchmarkParams.enableBatchSizeTuning, benchmarkParams.enableMaxNumTokensTuning); + texec::SchedulerConfig schedulerConfig(capacitySchedulerPolicy, std::nullopt, dynamicBatchConfig); + + texec::KvCacheConfig kvCacheConfig(benchmarkParams.enableBlockReuse, benchmarkParams.maxTokensInPagedKvCache, + benchmarkParams.maxAttentionWindowVec, benchmarkParams.sinkTokenLength, + benchmarkParams.freeGpuMemoryFraction, benchmarkParams.kvHostCacheSize, + benchmarkParams.crossKvCacheFraction); + texec::PeftCacheConfig peftCacheConfig(0, benchmarkParams.loraDeviceNumModLayers, 8, 64, 4, 4, 4, 24, 8, + std::nullopt, benchmarkParams.loraHostCacheSize); + texec::ExtendedRuntimePerfKnobConfig extendedRuntimePerfKnobConfig(benchmarkParams.multiBlockMode, + benchmarkParams.enableContextFMHAFP32Acc, benchmarkParams.cudaGraphMode, + benchmarkParams.cudaGraphCacheSize); + texec::ExecutorConfig executorConfig( + maxBeamWidth, schedulerConfig, kvCacheConfig, benchmarkParams.enableChunkedContext, true); + executorConfig.setEnableTrtOverlap(benchmarkParams.enableTrtOverlap); + executorConfig.setGpuWeightsPercent(benchmarkParams.gpuWeightsPercent); + executorConfig.setPeftCacheConfig(peftCacheConfig); + executorConfig.setBatchingType(batchingType); + if (benchmarkParams.maxBatchSize) + { + executorConfig.setMaxBatchSize(benchmarkParams.maxBatchSize.value()); + } + if (benchmarkParams.maxNumTokens) + { + executorConfig.setMaxNumTokens(benchmarkParams.maxNumTokens.value()); + } + + auto decodingMode = texec::DecodingMode::Auto(); + if (benchmarkParams.medusaChoices.has_value()) + { + decodingMode = texec::DecodingMode::Medusa(); + } + else if (benchmarkParams.executorLookaheadConfig.has_value()) + { + decodingMode = texec::DecodingMode::Lookahead(); + } + else if (benchmarkParams.eagleConfig.has_value()) + { + decodingMode = texec::DecodingMode::Eagle(); + } + + executorConfig.setDecodingConfig(texec::DecodingConfig(decodingMode, benchmarkParams.executorLookaheadConfig, + benchmarkParams.medusaChoices, benchmarkParams.eagleConfig)); + executorConfig.setExtendedRuntimePerfKnobConfig(extendedRuntimePerfKnobConfig); + + if (executorModelType == texec::ModelType::kDECODER_ONLY) + { + mExecutor + = std::make_unique<texec::Executor>(decoderTrtEnginePath.value(), executorModelType, executorConfig); + } + else if (executorModelType == texec::ModelType::kENCODER_DECODER) + { + mExecutor = std::make_unique<texec::Executor>( + encoderTrtEnginePath.value(), decoderTrtEnginePath.value(), executorModelType, executorConfig); + } + else if (executorModelType == texec::ModelType::kENCODER_ONLY) + { + mExecutor + = std::make_unique<texec::Executor>(encoderTrtEnginePath.value(), executorModelType, executorConfig); + } + else + { + TLLM_LOG_ERROR("not a supported executor model type in executor server."); + } + + auto const& world = tensorrt_llm::mpi::MpiComm::world(); + auto worldRank = world.getRank(); + if (worldRank == 0) + { + mCollectStatsThread = std::thread(&ExecutorServer::collectStats, this); + } + } + + ~ExecutorServer() + { + mShutdown = true; + if (mCollectStatsThread.joinable()) + { + mCollectStatsThread.join(); + } + } + + void enqueue(std::vector<texec::Request> requests, bool warmup = false) + { + try + { + std::vector<SizeType32> inputLengths; + for (auto const& request : requests) + { + inputLengths.push_back(request.getInputTokenIds().size()); + } + auto const start = std::chrono::steady_clock::now(); + auto reqIds = mExecutor->enqueueRequests(std::move(requests)); + for (int req = 0; req < reqIds.size(); ++req) + { + if (!warmup) + { + mRecorder->recordStart(inputLengths.at(req), reqIds.at(req), start); + } + mActiveCount++; + } + } + catch (std::exception const& e) + { + TLLM_THROW("%s", e.what()); + } + } + + void resetNumFinished() + { + mNumFinished = 0; + } + + bool canEnqueue(int numSentRequests) const + { + return !mConcurrency || (numSentRequests - mNumFinished < mConcurrency); + } + + void waitForResponses(SizeType32 numRequests, bool warmup = false) + { + while (mActiveCount || (mNumFinished < numRequests)) + { + auto responses = mExecutor->awaitResponses(mWaitSleep); + auto const tokenTime = std::chrono::steady_clock::now(); + for (auto const& response : responses) + { + if (response.getResult().isFinal) + { + mActiveCount--; + mNumFinished++; + if (!warmup) + { + mRecorder->recordEnd(response, tokenTime); + } + } + else + { + if (!warmup && !response.hasError()) + { + mRecorder->recordToken(response, tokenTime); + } + } + } + } + } + + void collectStats() const + { + while (!mShutdown) + { + auto iterStats = mExecutor->getLatestIterationStats(); + for (auto const& iterStat : iterStats) + { + SizeType32 numNewActiveRequests = iterStat.numNewActiveRequests; + if (numNewActiveRequests > 0) + { + float avgQueueingTime + = static_cast<float>(iterStat.newActiveRequestsQueueLatencyMS / numNewActiveRequests); + std::vector<float> requestsQueueLatencyMS(numNewActiveRequests, avgQueueingTime); + mRecorder->recordQueueLatency(requestsQueueLatencyMS); + } + if (mLogIterationData) + { + TLLM_LOG_INFO(texec::JsonSerialization::toJsonStr(iterStat)); + } + } + auto const waitSleep = std::chrono::milliseconds(50); + std::this_thread::sleep_for(waitSleep); + } + } + +private: + std::unique_ptr<texec::Executor> mExecutor; + std::thread mCollectStatsThread; + std::shared_ptr<Recorder> mRecorder; + std::chrono::milliseconds mWaitSleep; + std::optional<int> mConcurrency; + std::atomic<uint64_t> mActiveCount; + std::atomic<uint64_t> mNumFinished; + std::atomic<bool> mShutdown; + bool mLogIterationData; +}; // class ExecutorServer + +namespace +{ + +texec::Request makeExecutorRequest(Sample const& sample, SizeType32 const& beamWidth, + std::optional<SizeType32> const& eosId, std::optional<SizeType32> const& padId, bool streaming = false, + bool const& returnContextLogits = false, bool const& returnGenerationLogits = false, + std::optional<texec::LoraConfig> const& loraConfig = std::nullopt, + std::optional<texec::LookaheadDecodingConfig> const& lookaheadConfig = std::nullopt, + std::optional<texec::VecTokens> encoderInputTokenIds = std::nullopt, + std::optional<float> temperature = std::nullopt) +{ + auto samplingConfig = texec::SamplingConfig{beamWidth}; + samplingConfig.setTemperature(temperature); + auto outputConfig = texec::OutputConfig{false, returnContextLogits, returnGenerationLogits, false}; + return texec::Request(sample.inputIds, sample.outputLen, streaming, samplingConfig, outputConfig, eosId, padId, + std::nullopt, // positionIds + std::nullopt, // badWords + std::nullopt, // stopWords + std::nullopt, // embeddingBias + std::nullopt, // speculativeDecoding + std::nullopt, // pTuning + std::nullopt, // multimodalInput + std::nullopt, // multimodalEmbedding + std::nullopt, // mRopeConfig + loraConfig, // loraConfig + lookaheadConfig, // lookaheadConfig + std::nullopt, // kvCacheRetentionConfig + std::nullopt, // logitsPostProcessorName + std::nullopt, // logitsPostProcessor + encoderInputTokenIds.has_value() ? encoderInputTokenIds : std::nullopt, + std::nullopt); // cacheSalt +} + +void benchmarkExecutor(std::optional<std::filesystem::path> const& decoderEngineDir, + std::optional<std::filesystem::path> const& encoderEngineDir, texec::BatchingType batchingType, + std::string const& datasetPath, std::string const& opCsvFile, int maxNumSamples, int beamWidth, int warmUp, + std::optional<int32_t> const& eosId, std::optional<int32_t> const& padId, BenchmarkParams const& benchmarkParams, + texec::CapacitySchedulerPolicy capacitySchedulerPolicy, std::chrono::milliseconds waitSleep, + bool returnContextLogits, bool returnGenerationLogits, std::optional<int> const staticEmulatedBatchSize, + bool logIterationData, std::optional<SizeType32> const maxPromptLen, texec::ModelType executorModelType, + std::string const& responsesJsonFile) +{ + auto const& world = tensorrt_llm::mpi::MpiComm::world(); + auto worldRank = world.getRank(); + + // Load dataset + auto const samples = parseWorkloadJson(datasetPath, maxNumSamples, maxPromptLen); + auto const numSamples = samples.size(); + + auto recorder = std::make_shared<Recorder>(opCsvFile, benchmarkParams.streaming, beamWidth, responsesJsonFile); + int32_t decoderStartTokenId = 0; + std::shared_ptr<ExecutorServer> executorServer; + + if (executorModelType == texec::ModelType::kDECODER_ONLY) + { + TLLM_CHECK_WITH_INFO( + decoderEngineDir.has_value(), "decoder models require a path to decoder engine in executor benchmark."); + executorServer + = std::make_shared<ExecutorServer>(decoderEngineDir.value(), std::nullopt, batchingType, beamWidth, + capacitySchedulerPolicy, benchmarkParams, recorder, waitSleep, logIterationData, executorModelType); + } + else if (executorModelType == texec::ModelType::kENCODER_DECODER) + { + TLLM_CHECK_WITH_INFO(encoderEngineDir.has_value(), + "encoder-decoder models require a path to encoder engine in executor benchmark."); + executorServer = std::make_shared<ExecutorServer>(decoderEngineDir.value(), encoderEngineDir.value(), + batchingType, beamWidth, capacitySchedulerPolicy, benchmarkParams, recorder, waitSleep, logIterationData, + executorModelType); + try + { + std::ifstream decoderJsonConfigPath(decoderEngineDir.value() / "config.json"); + auto const decoderPretrainedConfig + = nlohmann::json::parse(decoderJsonConfigPath, nullptr, true, true).at("pretrained_config"); + decoderStartTokenId = decoderPretrainedConfig.at("decoder_start_token_id").template get<int32_t>(); + } + catch (nlohmann::json::out_of_range& e) + { + TLLM_LOG_ERROR( + "Parameter %s cannot be read from decoder config.json in pretrained_config. Using default id %d.", + std::string("decoder_start_token_id").c_str(), decoderStartTokenId); + } + catch (nlohmann::json::type_error const& e) + { + TLLM_LOG_ERROR( + "Parameter %s has error type in decoder config.json in pretrained_config. Using default id %d.", + std::string("decoder_start_token_id").c_str(), decoderStartTokenId); + } + } + else if (executorModelType == texec::ModelType::kENCODER_ONLY) + { + TLLM_CHECK_WITH_INFO( + encoderEngineDir.has_value(), "encoder models require a path to encoder engine in executor benchmark."); + executorServer + = std::make_shared<ExecutorServer>(std::nullopt, encoderEngineDir.value(), batchingType, beamWidth, + capacitySchedulerPolicy, benchmarkParams, recorder, waitSleep, logIterationData, executorModelType); + } + else + { + TLLM_LOG_ERROR("not a supported executor model type in executor benchmark."); + return; + } + + if (worldRank == 0) + { + if (benchmarkParams.loraDir) + { + auto startLoraLoad = std::chrono::steady_clock::now(); + LoraLib loras(benchmarkParams.loraDir.value()); + std::vector<texec::Request> requests; + for (auto& [taskId, p] : loras.getLoras()) + { + // squeeze lora configs and weights since LoraConfig requires them to be 2D tensors + p.first->squeeze(0); + p.second->squeeze(0); + texec::LoraConfig loraConfig( + taskId, texec::detail::ofITensor(p.first), texec::detail::ofITensor(p.second)); + if (executorModelType == texec::ModelType::kENCODER_DECODER) + { + Sample s{std::vector<int32_t>{decoderStartTokenId}, 1, static_cast<int32_t>(taskId)}; + requests.emplace_back(makeExecutorRequest(s, beamWidth, eosId, padId, false, false, false, + loraConfig, std::nullopt, std::vector<int32_t>{1, 2, 3, 4, 5})); + } + else + { + Sample s{std::vector<int32_t>{1, 2, 3, 4, 5}, 1, static_cast<int32_t>(taskId)}; + requests.emplace_back( + makeExecutorRequest(s, beamWidth, eosId, padId, false, false, false, loraConfig, std::nullopt)); + } + } + executorServer->enqueue(std::move(requests), true); + executorServer->waitForResponses(loras.getLoras().size(), true); + auto endLoraLoad = std::chrono::steady_clock::now(); + printf("[BENCHMARK] time to preload LoRAs(ms) %.2f\n", + std::chrono::duration<float, std::milli>(endLoraLoad - startLoraLoad).count()); + } + // Warm up + { + std::vector<texec::Request> requests; + for (auto i = 0; i < warmUp; ++i) + { + if (executorModelType == texec::ModelType::kENCODER_DECODER) + { + Sample s{std::vector<int32_t>{decoderStartTokenId}, samples[0].outputLen, samples[0].taskId}; + requests.emplace_back(makeExecutorRequest(s, beamWidth, eosId, padId, benchmarkParams.streaming, + returnContextLogits, returnGenerationLogits, std::nullopt, + benchmarkParams.requestLookaheadConfig, samples[0].inputIds)); + } + else + { + requests.emplace_back(makeExecutorRequest(samples[0], beamWidth, eosId, padId, + benchmarkParams.streaming, returnContextLogits, returnGenerationLogits, std::nullopt, + benchmarkParams.requestLookaheadConfig, std::nullopt, benchmarkParams.temperature)); + } + } + executorServer->enqueue(std::move(requests), true); + executorServer->waitForResponses(warmUp, true); + } + + // Benchmark + { + auto timeDelays = computeTimeDelays(benchmarkParams, numSamples - 1); + + // Create requests + recorder->initialize(); + std::vector<texec::Request> requests; + + for (std::size_t i = 0; i < numSamples; ++i) + { + std::optional<texec::LoraConfig> loraConfig; + if (samples[i].taskId >= 0) + { + loraConfig = texec::LoraConfig(samples[i].taskId); + } + if (executorModelType == texec::ModelType::kENCODER_DECODER) + { + Sample s{std::vector<int32_t>{decoderStartTokenId}, samples[i].outputLen, samples[i].taskId}; + requests.emplace_back(makeExecutorRequest(s, beamWidth, eosId, padId, benchmarkParams.streaming, + returnContextLogits, returnGenerationLogits, loraConfig, benchmarkParams.requestLookaheadConfig, + samples[i].inputIds)); + } + else + { + requests.emplace_back(makeExecutorRequest(samples[i], beamWidth, eosId, padId, + benchmarkParams.streaming, returnContextLogits, returnGenerationLogits, loraConfig, + benchmarkParams.requestLookaheadConfig, std::nullopt, benchmarkParams.temperature)); + } + } + + bool const hasDelay + = std::any_of(timeDelays.begin(), timeDelays.end(), [](auto const& delay) { return delay > 0.0; }); + executorServer->resetNumFinished(); + if (!staticEmulatedBatchSize) + { + // Launch a thread that will wait for responses + std::thread waitThread( + [numSamples, executorServer]() { executorServer->waitForResponses(numSamples); }); + + // Enqueue requests one by one + int numSentRequests = 0; + while (numSentRequests < numSamples) + { + if (executorServer->canEnqueue(numSentRequests)) + { + executorServer->enqueue({requests.at(numSentRequests)}); + if (hasDelay && numSentRequests < numSamples - 1) + { + std::this_thread::sleep_for( + std::chrono::milliseconds(static_cast<int>(timeDelays.at(numSentRequests) * 1000))); + } + numSentRequests += 1; + } + } + waitThread.join(); + } + else + { + TLLM_CHECK_WITH_INFO( + !hasDelay, "Executor benchmark doesn't support delays with emulated static batch sizes"); + SizeType32 numRequests = requests.size(); + SizeType32 maxBatchSize = staticEmulatedBatchSize.value(); + for (SizeType32 req = 0; req < numRequests; req += maxBatchSize) + { + auto batchSize = std::min(maxBatchSize, numRequests - req); + + std::vector<texec::Request> requestsBatch(std::make_move_iterator(requests.begin() + req), + std::make_move_iterator(requests.begin() + req + batchSize)); + // Enqueue in batches + executorServer->enqueue(std::move(requestsBatch)); + // Wait for current batch to be done + executorServer->waitForResponses(batchSize); + } + } + } + recorder->finalize(); + recorder->calculateMetrics(); + recorder->report(); + recorder->writeOpMetricsToCsv(); + recorder->dumpResponseSeqs(); + // Send terminateReqId to terminate servers on all ranks + // Sever on rank 0 will broadcast the terminate signal to other servers on multi-GPU cases + } +} + +} // namespace + +int main(int argc, char* argv[]) +{ + cxxopts::Options options( + "TensorRT LLM BatchManager Benchmark", "TensorRT LLM BatchManager Benchmark for GPT and GPT-like models."); + options.add_options()("h,help", "Print usage"); + options.add_options()("engine_dir, decoder_engine_dir", "Directory that store the engines of decoder models.", + cxxopts::value<std::string>()); + options.add_options()( + "encoder_engine_dir", "Directory that store the engines of the encoder models.", cxxopts::value<std::string>()); + options.add_options()( + "api", "API type: gptManager or executor.", cxxopts::value<std::string>()->default_value("executor")); + options.add_options()("type", + "Batching type: choose between inflight/static. (IFB/V1 options are going to be deprecated)", + cxxopts::value<std::string>()->default_value("inflight")); + options.add_options()("dataset", "Dataset that is used for benchmarking BatchManager.", + cxxopts::value<std::string>()->default_value("")); + options.add_options()( + "output_csv", "Write output metrics to CSV", cxxopts::value<std::string>()->default_value("")); + options.add_options()("max_num_samples", "maximum number of samples to use from dataset/generate", + cxxopts::value<int>()->default_value("100000")); + options.add_options()( + "beam_width", "Specify beam width you want to benchmark.", cxxopts::value<int>()->default_value("1")); + options.add_options()( + "warm_up", "Specify warm up iterations before benchmark starts.", cxxopts::value<int>()->default_value("2")); + options.add_options()( + "eos_id", "Specify the end-of-sequence token id.", cxxopts::value<TokenIdType>()->default_value("-1")); + options.add_options()("pad_id", "Specify the padding token id.", cxxopts::value<TokenIdType>()); + options.add_options()("max_tokens_in_paged_kvcache", "Max tokens in paged K-V Cache.", cxxopts::value<int>()); + options.add_options()( + "max_attention_window", "Max KV cache length per sequence", cxxopts::value<std::vector<int>>()); + options.add_options()("sink_token_len", "Sink token length in kv cache per sequence.", cxxopts::value<int>()); + options.add_options()( + "random_seed", "integer random seed for exponential time delays.", cxxopts::value<int>()->default_value("420")); + options.add_options()( + "kv_cache_free_gpu_mem_fraction", "K-V Cache Free Gpu Mem Fraction.", cxxopts::value<float>()); + options.add_options()( + "cross_kv_cache_fraction", "Cross K-V Cache Fraction (from 0.0 to 1.0).", cxxopts::value<float>()); + options.add_options()("request_rate", + "request rate in reqs/sec. Skipping this arg or negative value will trigger offline/0-delay.", + cxxopts::value<float>()); + options.add_options()("concurrency", "Concurrent number of connections with the server.", cxxopts::value<int>()); + options.add_options()("max_batch_size", "The max runtime batch size when benchmarking", cxxopts::value<int>()); + options.add_options()( + "max_num_tokens", "The max runtime number of tokens per batch when benchmarking", cxxopts::value<int>()); + options.add_options()( + "enable_batch_size_tuning", "Dynamic tuning of batch size", cxxopts::value<bool>()->default_value("false")); + options.add_options()("enable_max_num_tokens_tuning", "Dynamic tuning of max num tokens", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("enable_exp_delays", "Enables exponential delay distr to mimic real world request arrival", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("streaming", + "Operate in streaming mode. Note: it reflects time-to-first-token and inter-token-latency", + cxxopts::value<bool>()->default_value("false")); + options.add_options()( + "enable_kv_cache_reuse", "Enables the KV cache reuse.", cxxopts::value<bool>()->default_value("true")); + options.add_options()( + "enable_chunked_context", "Whether to enable context chunking.", cxxopts::value<bool>()->default_value("true")); + options.add_options()( + "return_context_logits", "Whether to return context logits.", cxxopts::value<bool>()->default_value("false")); + options.add_options()("return_generation_logits", "Whether to return generation logits.", + cxxopts::value<bool>()->default_value("false")); + + options.add_options()("scheduler_policy", + "Choose scheduler policy between max_utilization/guaranteed_no_evict/static_batch.", + cxxopts::value<std::string>()->default_value("guaranteed_no_evict")); + + options.add_options()("static_emulated_batch_size", + "Emulate static batching performance with the provided batch size.", cxxopts::value<SizeType32>()); + options.add_options()("log_level", "Choose log level between verbose/info/warning/error/internal_error.", + cxxopts::value<std::string>()->default_value("warning")); + options.add_options()("log_iteration_data", "On each decoder iteration, print batch state metadata.", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("wait_sleep", "Specify how many milliseconds to sleep each iteration of waitForEmpty loop.", + cxxopts::value<int>()->default_value("25")); + options.add_options()("lora_dir", "Directory containing LoRAs", cxxopts::value<std::string>()->default_value("")); + options.add_options()("lora_host_cache_bytes", "LoRA host cache memory in bytes", cxxopts::value<size_t>()); + options.add_options()("lora_num_device_mod_layers", "LoRA number 1d cache rows", cxxopts::value<int>()); + options.add_options()("kv_host_cache_bytes", + "Size of secondary memory pool used for offloading kv cache blocks (in bytes).", + cxxopts::value<size_t>()->default_value("0")); + options.add_options()( + "max_prompt_len", "Truncate all prompts from dataset to the length specified.", cxxopts::value<SizeType32>()); + + options.add_options()("gpu_weights_percent", + "Specify the percentage of weights that reside on GPU (from 0.0 to 1.0).", + cxxopts::value<float>()->default_value("1.0")); + options.add_options()( + "medusa_choices", "Medusa choices in the format of [[0], [0, 1], [0, 0, 1]]", cxxopts::value<std::string>()); + options.add_options()( + "eagle_choices", "Eagle choices in the format of [[0], [0, 1], [0, 0, 1]]", cxxopts::value<std::string>()); + options.add_options()("eagle_posterior_threshold", + "Minimum token probability threshold for typical acceptance. Enables typical acceptance in Eagle", + cxxopts::value<float>()); + options.add_options()("temperature", "Sampling temperature for each request", cxxopts::value<float>()); + options.add_options()( + "eagle_use_dynamic_tree", "Whether to use Eagle-2", cxxopts::value<bool>()->default_value("false")); + options.add_options()("eagle_dynamic_tree_max_top_k", + "The max topK for dynamic tree, also the number of draft tokens that will expand for each node", + cxxopts::value<SizeType32>()); + + options.add_options()("multi_block_mode", + "Distribute the work across multiple CUDA thread-blocks on the GPU for masked MHA kernel", + cxxopts::value<bool>()->default_value("true")); + options.add_options()("cuda_graph_mode", "When enabled, inference is executed with cuda graph.", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("cuda_graph_cache_size", + "Specify how many cuda graphs are cached in the runtime. Larger cache gives better perf, but consumes more GPU " + "memory.", + cxxopts::value<SizeType32>()->default_value("0")); + options.add_options()("enable_trt_overlap", "Enable TRT Overlap", cxxopts::value<bool>()->default_value("false")); + + options.add_options()("enable_context_fmha_fp32_acc", "Enable FMHA runner FP32 accumulation", + cxxopts::value<bool>()->default_value("false")); + options.add_options()("executor_lookahead_config", + "lookahead config in the format of [max_window_size, max_ngram_size, max_verification_set_size]", + cxxopts::value<std::string>()); + options.add_options()("request_lookahead_config", + "lookahead config in the format of [max_window_size, max_ngram_size, max_verification_set_size], and each <= " + "executor lookahead config", + cxxopts::value<std::string>()); + options.add_options()("responses_json", "Write output response sequences to a json file", + cxxopts::value<std::string>()->default_value("")); + + auto result = options.parse(argc, argv); + + if (result.count("help")) + { + std::cout << options.help() << std::endl; + return 0; + } + + // Argument: Engine directory + if (!result.count("engine_dir") && !result.count("encoder_engine_dir")) + { + std::cout << options.help() << std::endl; + TLLM_LOG_ERROR("Please specify engine directory."); + return 1; + } + + // Argument: Batching Type + auto const type = result["type"].as<std::string>(); + texec::BatchingType batchingType{texec::BatchingType::kINFLIGHT}; + if (type == "V1" || type == "static") + { + if (type == "V1") + { + TLLM_LOG_WARNING("type option \"V1\" is going to be renamed to \"static\"."); + } + bool streaming = result["streaming"].as<bool>(); + if (streaming) + { + TLLM_LOG_ERROR("Streaming is not supported in static batching.\n"); + return 1; + } + batchingType = texec::BatchingType::kSTATIC; + } + else if (type == "IFB" || type == "inflight") + { + if (type == "IFB") + { + TLLM_LOG_WARNING("type option \"IFB\" is going to be renamed to \"inflight\"."); + } + batchingType = texec::BatchingType::kINFLIGHT; + } + else + { + TLLM_LOG_ERROR("Unexpected batching type: %s", type.c_str()); + return 1; + } + + // Argument: Dataset + auto const datasetPath = result["dataset"].as<std::string>(); + auto const maxNumSamples = result["max_num_samples"].as<int>(); + + // Argument: Output metrics CSV + auto const opCsvFile = result["output_csv"].as<std::string>(); + + // Argument: beam width + auto const beamWidth = result["beam_width"].as<int>(); + + // Argument: wait_sleep + auto const waitSleep = std::chrono::milliseconds(result["wait_sleep"].as<int>()); + BenchmarkParams benchmarkParams; + + // Argument: Max tokens in paged K-V Cache + if (result.count("max_tokens_in_paged_kvcache")) + { + benchmarkParams.maxTokensInPagedKvCache = result["max_tokens_in_paged_kvcache"].as<int>(); + } + + // Argument: Max KV cache length + if (result.count("max_attention_window")) + { + benchmarkParams.maxAttentionWindowVec = result["max_attention_window"].as<std::vector<int>>(); + } + + // Argument: Sink token length + if (result.count("sink_token_len")) + { + benchmarkParams.sinkTokenLength = result["sink_token_len"].as<int>(); + } + + if (result.count("random_seed")) + { + benchmarkParams.randomSeed = result["random_seed"].as<int>(); + } + + // Argument: K-V Cache Free Gpu Mem Fraction + if (result.count("kv_cache_free_gpu_mem_fraction")) + { + benchmarkParams.freeGpuMemoryFraction = result["kv_cache_free_gpu_mem_fraction"].as<float>(); + } + // Argument: K-V Cache Cross Attention Fraction. Only applicable to enc-dec models. + if (result.count("encoder_engine_dir") && result.count("decoder_engine_dir")) + { + if (result.count("cross_kv_cache_fraction")) + { + benchmarkParams.crossKvCacheFraction = result["cross_kv_cache_fraction"].as<float>(); + } + else + { + benchmarkParams.crossKvCacheFraction + = 0.5f; // default value if not set. but non enc-dec should not even have this param set + } + } + + // Argument: Enable dynamic tuning of batch size + benchmarkParams.enableBatchSizeTuning = result["enable_batch_size_tuning"].as<bool>(); + + // Argument: Enable dynamic tuning of max num tokens + benchmarkParams.enableMaxNumTokensTuning = result["enable_max_num_tokens_tuning"].as<bool>(); + + // Argument: Enable KV cache reuse + benchmarkParams.enableBlockReuse = result["enable_kv_cache_reuse"].as<bool>(); + + // Argument: streaming + benchmarkParams.streaming = result["streaming"].as<bool>(); + + TLLM_CHECK_WITH_INFO(!(result.count("request_rate") && result.count("concurrency")), + "request_rate and concurrency cannot be specified at the same time."); + + // Argument: request rate + if (result.count("request_rate")) + { + benchmarkParams.requestRate = result["request_rate"].as<float>(); + } + + // Argument: concurrency + if (result.count("concurrency")) + { + benchmarkParams.concurrency = result["concurrency"].as<int>(); + } + + // Argument: request rate + if (result.count("max_batch_size")) + { + benchmarkParams.maxBatchSize = result["max_batch_size"].as<int>(); + } + + // Argument: request rate + if (result.count("max_num_tokens")) + { + benchmarkParams.maxNumTokens = result["max_num_tokens"].as<int>(); + } + + benchmarkParams.enableExpDelays = result["enable_exp_delays"].as<bool>(); + + // Argument: Enable batch stats output + bool logIterationData = result["log_iteration_data"].as<bool>(); + + if (logIterationData) + { + TLLM_LOG_WARNING("Setting log_iteration_data to true adds overheads and may result in lower perf"); + } + + // Argument: Enable chunked context + benchmarkParams.enableChunkedContext = result["enable_chunked_context"].as<bool>(); + + // Argument: Enable return context logits + bool returnContextLogits = result["return_context_logits"].as<bool>(); + + // Argument: Enable return context logits + bool returnGenerationLogits = result["return_generation_logits"].as<bool>(); + + if (result.count("lora_dir")) + { + benchmarkParams.loraDir = result["lora_dir"].as<std::string>(); + } + if (result.count("lora_host_cache_bytes")) + { + benchmarkParams.loraHostCacheSize = result["lora_host_cache_bytes"].as<size_t>(); + } + if (result.count("lora_num_device_mod_layers")) + { + benchmarkParams.loraDeviceNumModLayers = result["lora_num_device_mod_layers"].as<SizeType32>(); + } + + // Argument: How many KV cache blocks (as fraction of number of GPU kv cache blocks). + benchmarkParams.kvHostCacheSize = result["kv_host_cache_bytes"].as<size_t>(); + + // Argument: Medusa choices for the Medusa speculative decoding. + if (result.count("medusa_choices")) + { + benchmarkParams.medusaChoices = parseVectorOfVectors(result["medusa_choices"].as<std::string>()); + } + // Argument: Eagle choices for the Eagle speculative decoding. + if (result.count("eagle_choices") || result.count("eagle_posterior_threshold") + || result.count("eagle_use_dynamic_tree") || result.count("eagle_dynamic_tree_max_top_k")) + { + std::optional<float> posteriorThreshold; + if (result.count("eagle_posterior_threshold")) + { + posteriorThreshold = result["eagle_posterior_threshold"].as<float>(); + } + std::optional<texec::EagleChoices> choices; + if (result.count("eagle_choices")) + { + choices = parseVectorOfVectors(result["eagle_choices"].as<std::string>()); + } + bool eagleUseDynamicTree = false; + if (result.count("eagle_use_dynamic_tree")) + { + eagleUseDynamicTree = result["eagle_use_dynamic_tree"].as<bool>(); + } + std::optional<SizeType32> eagleDynamicTreeMaxTopK; + if (result.count("eagle_dynamic_tree_max_top_k")) + { + eagleDynamicTreeMaxTopK = result["eagle_dynamic_tree_max_top_k"].as<SizeType32>(); + } + benchmarkParams.eagleConfig = texec::EagleConfig( + choices, !posteriorThreshold.has_value(), posteriorThreshold, eagleUseDynamicTree, eagleDynamicTreeMaxTopK); + } + if (result.count("temperature")) + { + benchmarkParams.temperature = result["temperature"].as<float>(); + } + + if (result.count("executor_lookahead_config")) + { + benchmarkParams.executorLookaheadConfig + = parseLookaheadConfig(result["executor_lookahead_config"].as<std::string>()); + } + if (result.count("request_lookahead_config")) + { + benchmarkParams.requestLookaheadConfig + = parseLookaheadConfig(result["request_lookahead_config"].as<std::string>()); + } + + // Argument: multi_block_mode + benchmarkParams.multiBlockMode = result["multi_block_mode"].as<bool>(); + + // Argument: enable_context_fmha_fp32_acc + benchmarkParams.enableContextFMHAFP32Acc = result["enable_context_fmha_fp32_acc"].as<bool>(); + + // Argument: cuda_graph_mode + benchmarkParams.cudaGraphMode = result["cuda_graph_mode"].as<bool>(); + + // Argument: cuda_graph_cache_size + benchmarkParams.cudaGraphCacheSize = result["cuda_graph_cache_size"].as<SizeType32>(); + + // Argument: enable_trt_overlap + benchmarkParams.enableTrtOverlap = result["enable_trt_overlap"].as<bool>(); + + std::optional<TokenIdType> padId; + // Argument: Padding token id + if (result.count("pad_id")) + { + padId = result["pad_id"].as<TokenIdType>(); + } + + // Argument: End-of-sentence token id + std::optional<TokenIdType> eosId = result["eos_id"].as<TokenIdType>(); + + std::optional<SizeType32> staticEmulatedBatchSize; + // Argument: Static emulated batch size + if (result.count("static_emulated_batch_size")) + { + staticEmulatedBatchSize = result["static_emulated_batch_size"].as<SizeType32>(); + } + + // Argument: Scheduler policy + texec::CapacitySchedulerPolicy capacitySchedulerPolicy; + auto const capacitySchedulerPolicyArg = result["scheduler_policy"].as<std::string>(); + if (capacitySchedulerPolicyArg == "max_utilization") + { + capacitySchedulerPolicy = texec::CapacitySchedulerPolicy::kMAX_UTILIZATION; + } + else if (capacitySchedulerPolicyArg == "guaranteed_no_evict") + { + capacitySchedulerPolicy = texec::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT; + } + else if (capacitySchedulerPolicyArg == "static_batch") + { + capacitySchedulerPolicy = texec::CapacitySchedulerPolicy::kSTATIC_BATCH; + } + else + { + TLLM_LOG_ERROR("Unexpected scheduler policy: " + capacitySchedulerPolicyArg); + return 1; + } + + // Argument: max_prompt_len + std::optional<SizeType32> maxPromptLen; + if (result.count("max_prompt_len")) + { + maxPromptLen = result["max_prompt_len"].as<SizeType32>(); + } + + // Argument: GPU weights percentage + auto gpuWeightsPercent = result["gpu_weights_percent"].as<float>(); + if (gpuWeightsPercent < 0 || gpuWeightsPercent > 1) + { + TLLM_LOG_ERROR("--gpu_weights_percent must be between 0.0 and 1.0 but got: %f", gpuWeightsPercent); + return 1; + } + benchmarkParams.gpuWeightsPercent = gpuWeightsPercent; + + // Argument: Log level + auto logger = std::make_shared<TllmLogger>(); + auto const logLevel = result["log_level"].as<std::string>(); + if (logLevel == "verbose") + { + logger->setLevel(trt::ILogger::Severity::kVERBOSE); + } + else if (logLevel == "info") + { + logger->setLevel(trt::ILogger::Severity::kINFO); + } + else if (logLevel == "warning") + { + logger->setLevel(trt::ILogger::Severity::kWARNING); + } + else if (logLevel == "error") + { + logger->setLevel(trt::ILogger::Severity::kERROR); + } + else if (logLevel == "internal_error") + { + logger->setLevel(trt::ILogger::Severity::kINTERNAL_ERROR); + } + else + { + TLLM_LOG_ERROR("Unexpected log level: " + logLevel); + return 1; + } + + initTrtLlmPlugins(logger.get()); + + // Argument: output sequences JSON + auto const responsesJsonFile = result["responses_json"].as<std::string>(); + + // Argument: API + auto const api = result["api"].as<std::string>(); + if (api == "executor") + { + texec::ModelType executorModelType; + std::optional<std::string> decoderEngineDir = std::nullopt, encoderEngineDir = std::nullopt; + if (result.count("encoder_engine_dir") && result.count("decoder_engine_dir")) + { + TLLM_CHECK_WITH_INFO(api == "executor", "encoder-decoder only support executor api."); + TLLM_CHECK_WITH_INFO( + batchingType == texec::BatchingType::kINFLIGHT, "encoder-decoder only support inflight batching."); + executorModelType = texec::ModelType::kENCODER_DECODER; + encoderEngineDir = result["encoder_engine_dir"].as<std::string>(); + decoderEngineDir = result["decoder_engine_dir"].as<std::string>(); + } + else if (result.count("engine_dir")) + { + executorModelType = texec::ModelType::kDECODER_ONLY; + decoderEngineDir = result["engine_dir"].as<std::string>(); + } + else + { + executorModelType = texec::ModelType::kENCODER_ONLY; + encoderEngineDir = result["encoder_engine_dir"].as<std::string>(); + } + try + { + benchmarkExecutor(decoderEngineDir, encoderEngineDir, batchingType, datasetPath, opCsvFile, maxNumSamples, + beamWidth, result["warm_up"].as<int>(), eosId, padId, benchmarkParams, capacitySchedulerPolicy, + waitSleep, returnContextLogits, returnGenerationLogits, staticEmulatedBatchSize, logIterationData, + maxPromptLen, executorModelType, responsesJsonFile); + } + catch (std::exception const& e) + { + TLLM_LOG_ERROR(e.what()); + return 1; + } + } + else if (api == "gptManager") + { + TLLM_LOG_ERROR("gptManager is deprecated, please use the executor API."); + return 1; + } + else + { + TLLM_LOG_ERROR("api parameter must be gptManager or executor"); + return 1; + } + + return 0; +} diff --git a/benchmarks/cpp/prepare_dataset.py b/benchmarks/cpp/prepare_dataset.py new file mode 100644 index 000000000000..3b9665fd2902 --- /dev/null +++ b/benchmarks/cpp/prepare_dataset.py @@ -0,0 +1,118 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import logging +from typing import Optional, Tuple + +import click +from pydantic import BaseModel, model_validator +from transformers import AutoTokenizer +from utils.prepare_real_data import dataset +from utils.prepare_synthetic_data import token_norm_dist, token_unif_dist + + +class RootArgs(BaseModel): + tokenizer: str + output: str + random_seed: int + task_id: int + std_out: bool + trust_remote_code: bool = False + rand_task_id: Optional[Tuple[int, int]] + lora_dir: Optional[str] = None + + @model_validator(mode='after') + def validate_tokenizer(self): + try: + tokenizer = AutoTokenizer.from_pretrained( + self.tokenizer, + padding_side='left', + trust_remote_code=self.trust_remote_code) + except EnvironmentError as e: + raise ValueError( + f"Cannot find a tokenizer from the given string because of {e}\nPlease set tokenizer to the directory that contains the tokenizer, or set to a model name in HuggingFace." + ) + tokenizer.pad_token = tokenizer.eos_token + self.tokenizer = tokenizer + + return self + + +@click.group(deprecated=True) +@click.option( + "--tokenizer", + required=True, + type=str, + help= + "Tokenizer dir for the model run by gptManagerBenchmark, or the model name from HuggingFace." +) +@click.option("--output", + type=str, + help="Output json filename.", + default="preprocessed_dataset.json") +@click.option( + "--stdout", + is_flag=True, + help="Print output to stdout with a JSON dataset entry on each line.", + default=False) +@click.option("--random-seed", + required=False, + type=int, + help="random seed for token_ids", + default=420) +@click.option("--task-id", type=int, default=-1, help="LoRA task id") +@click.option("--rand-task-id", + type=int, + default=None, + nargs=2, + help="Random LoRA Tasks") +@click.option("--lora-dir", + type=str, + default=None, + help="Directory containing LoRA adapters") +@click.option("--log-level", + default="info", + type=click.Choice(['info', 'debug']), + help="Logging level.") +@click.option("--trust-remote-code", + is_flag=True, + default=False, + envvar="TRUST_REMOTE_CODE", + help="Trust remote code.") +@click.pass_context +def cli(ctx, **kwargs): + """This script generates dataset input for gptManagerBenchmark.""" + if kwargs['log_level'] == 'info': + logging.basicConfig(level=logging.INFO) + elif kwargs['log_level'] == 'debug': + logging.basicConfig(level=logging.DEBUG) + else: + raise ValueError(f"Unsupported logging level {kwargs['log_level']}") + + ctx.obj = RootArgs(tokenizer=kwargs['tokenizer'], + output=kwargs['output'], + std_out=kwargs['stdout'], + random_seed=kwargs['random_seed'], + task_id=kwargs['task_id'], + rand_task_id=kwargs['rand_task_id'], + lora_dir=kwargs['lora_dir'], + trust_remote_code=kwargs['trust_remote_code']) + + +cli.add_command(dataset) +cli.add_command(token_norm_dist) +cli.add_command(token_unif_dist) + +if __name__ == "__main__": + cli() diff --git a/tensorrt_llm/bench/tuning/__init__.py b/benchmarks/cpp/utils/__init__.py similarity index 100% rename from tensorrt_llm/bench/tuning/__init__.py rename to benchmarks/cpp/utils/__init__.py diff --git a/benchmarks/cpp/utils/convert_nemo_dataset.py b/benchmarks/cpp/utils/convert_nemo_dataset.py new file mode 100644 index 000000000000..6f4884347677 --- /dev/null +++ b/benchmarks/cpp/utils/convert_nemo_dataset.py @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +#!/usr/bin/env python3 + +import argparse +import json + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("input") + parser.add_argument("output") + + args = parser.parse_args() + + output_o = [] + + with open(args.input, 'r') as infile: + for _l in infile: + l = _l.strip() + if len(l) == 0: + continue + o = json.loads(l) + output_o.append({ + "input": o["prompt"], + "instruction": "", + "output": o["completion"] + }) + + with open(args.output, 'w') as outfile: + json.dump(output_o, outfile) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/cpp/utils/generate_rand_loras.py b/benchmarks/cpp/utils/generate_rand_loras.py new file mode 100644 index 000000000000..12eb1fdc3648 --- /dev/null +++ b/benchmarks/cpp/utils/generate_rand_loras.py @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +#!/usr/bin/env python3 + +import argparse +import os +from pathlib import Path + +import numpy as np + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("input_lora") + parser.add_argument("output") + parser.add_argument("num_loras", type=int) + + args = parser.parse_args() + + lora_path = Path(args.input_lora) + weights_path = lora_path / "model.lora_weights.npy" + config_path = lora_path / "model.lora_config.npy" + + weights = np.load(weights_path) + config = np.load(config_path) + + for i in range(args.num_loras): + out_path = Path(args.output) / str(i) + os.makedirs(out_path, exist_ok=True) + w = np.random.normal(0, 2, weights.shape).astype(weights.dtype) + np.save(out_path / "model.lora_weights.npy", w) + np.save(out_path / "model.lora_config.npy", config) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/cpp/utils/prepare_real_data.py b/benchmarks/cpp/utils/prepare_real_data.py new file mode 100644 index 000000000000..4441c57ee4bc --- /dev/null +++ b/benchmarks/cpp/utils/prepare_real_data.py @@ -0,0 +1,434 @@ +import logging +import random +import re +import tempfile +from pathlib import Path +from typing import Optional + +import click +import datasets +from PIL import Image +from pydantic import BaseModel, model_validator +from utils.utils import (get_norm_dist_lengths, multimodal_dataset_dump, + print_multimodal_dataset, print_text_dataset, + text_dataset_dump) + + +def validate_output_len_dist(ctx, param, value): + """Validate the --output-len-dist option.""" + if value is None: + return value + m = re.match(r"(\d+),(\d+)", value) + if m: + return int(m.group(1)), int(m.group(2)) + else: + raise AssertionError( + "Incorrect specification for --output-len-dist. Correct format: --output-len-dist <output_len_mean>,<output_len_stdev>" + ) + + +class DatasetConfig(BaseModel): + """Dataset configurations.""" + """Name of the dataset on HuggingFace.""" + name: Optional[str] = None + """Config name of the dataset if existing.""" + config_name: Optional[str] = None + """Split of the dataset. Typical values: train, validation, test. Setting to None will include all splits.""" + split: Optional[str] + """The dataset dictionary used for the input sentence.""" + input_key: Optional[str] = None + """The dataset dictionary key used for the prompt of the input sentence. Must not be set when prompt is set.""" + image_key: Optional[str] = None + """The dataset dictionary key used for the images.""" + prompt_key: Optional[str] = None + """The prompt sentence to be added to the input sentence. Must not be set when prompt_key is set.""" + prompt: Optional[str] = None + """The dataset dictionary key used to derive the output sequence length. Set to None if the dataset does not have a key for output.""" + output_key: Optional[str] + """The local path to the dataset to be loaded when using a local cache.""" + local_path: Optional[str] = None + + @model_validator(mode='after') + def check_prompt(self) -> 'DatasetConfig': + if self.prompt_key and self.prompt: + raise AssertionError( + "--prompt-key and --prompt cannot be set at the same time.") + if (not self.prompt_key) and (not self.prompt): + raise AssertionError("Either --prompt-key or --prompt must be set.") + return self + + @model_validator(mode='after') + def check_name_and_local_path(self) -> 'DatasetConfig': + if self.name and self.local_path: + raise AssertionError( + "--dataset-name and --dataset-local-path cannot be set at the same time." + ) + if (not self.name) and (not self.local_path): + raise AssertionError( + "Either --dataset-name or --dataset-local-path must be set.") + return self + + @property + def query(self): + """Generate the query for HuggingFace `datasets.load_dataset()`""" + first_arg = self.local_path if self.local_path else self.name + + if self.config_name: + return [first_arg, self.config_name] + else: + return [first_arg] + + @property + def display_name(self) -> str: + """Returns a human-readable identifier for error messages.""" + # model_validator ensures exactly one of name or local_path is set + if self.name is not None: + return self.name + return self.local_path + + def get_prompt(self, req): + """Get the prompt sentence from the given request.""" + if self.prompt_key: + assert self.prompt_key in req, ( + f"Dataset {self.display_name} does not have key '{self.prompt_key}'. " + "Please set --prompt-key to one of the available keys: " + f"{req.keys()}") + return req[self.prompt_key] + else: + return self.prompt + + def get_input(self, req): + """Get the input sentence from the given request.""" + assert self.input_key in req, ( + f"Dataset {self.display_name} does not have key '{self.input_key}'. " + "Please set --input-key to one of the available keys: " + f"{req.keys()}") + return req[self.input_key] + + def get_images(self, req): + """Get the images from the given request.""" + image_keys = [self.image_key + ] + [f"{self.image_key}_{i}" for i in range(1, 8)] + assert any(key in req for key in image_keys), ( + f"Dataset {self.display_name} does not have key '{self.image_key}'. " + "Please set --dataset-image-key to one of the available keys: " + f"{req.keys()}") + images = [] + for key in image_keys: + if key in req and req[key] is not None: + images.append(req[key]) + return images + + def get_output(self, req): + """Get the output sentence from the given request.""" + if self.output_key is None: + raise RuntimeError( + "--output-key is not set. Please either:\n" + "1. Define output length through --output-len-dist.\n" + f"2. If the dataset {self.display_name} has key for golden output and " + "you wish to set output length to the length of the golden " + "output, set --output-key.") + assert self.output_key in req, ( + f"Dataset {self.display_name} does not have key '{self.output_key}'. " + "Please set --output-key to one of the available keys: " + f"{req.keys()}") + return req[self.output_key] + + +def _create_dataset_load_error(e: ValueError) -> ValueError: + """Create a more informative ValueError from a dataset loading error. + + Args: + e: The original ValueError from datasets.load_dataset(). + Returns: + A new ValueError with additional context. + """ + error_msg = str(e) + if "Config" in error_msg: + error_msg += "\n Please add the config name to the dataset config yaml." + elif "split" in error_msg: + error_msg += "\n Please specify supported split in the dataset config yaml." + return ValueError(error_msg) + + +def load_dataset(dataset_config: DatasetConfig): + """Load dataset from local path or HuggingFace. + Args: + dataset_config: A `DatasetConfig` object that defines the dataset to load. + Returns: + Dataset iterator. + Raises: + ValueError: When dataset loading fails due to incorrect dataset config setting. + """ + if dataset_config.local_path: + return load_dataset_from_local(dataset_config) + else: + return load_dataset_from_hf(dataset_config) + + +def load_dataset_from_hf(dataset_config: DatasetConfig): + """Load dataset from HuggingFace. + + Args: + dataset_config: A `DatasetConfig` object that defines the dataset to load. + Returns: + Dataset iterator. + Raises: + ValueError: When dataset loading fails due to incorrect dataset config setting. + """ + logging.debug( + f"Loading dataset from HF: query={dataset_config.query}, split={dataset_config.split}" + ) + + try: + dataset = iter( + datasets.load_dataset(*dataset_config.query, + split=dataset_config.split, + streaming=True, + trust_remote_code=True)) + except ValueError as e: + raise _create_dataset_load_error(e) + + logging.debug("Finished loading HF dataset") + + return dataset + + +def load_dataset_from_local(dataset_config: DatasetConfig): + """Load dataset from local path. + + Args: + dataset_config: A `DatasetConfig` object that defines the dataset to load. + Returns: + Dataset iterator. + Raises: + FileNotFoundError: When local dataset path does not exist. + ValueError: When dataset loading fails due to incorrect dataset config setting. + """ + + local_path = Path(dataset_config.local_path) + + if not local_path.exists(): + raise FileNotFoundError( + f"Local dataset path {local_path} does not exist.") + + logging.debug( + f"Loading dataset from local path: path={local_path}, query={dataset_config.query}, split={dataset_config.split}" + ) + + # If it's a directory we can use the normal loader, otherwise custom loader + # depends on the file extension + if local_path.is_dir(): + try: + dataset = datasets.load_dataset(*dataset_config.query, + split=dataset_config.split, + trust_remote_code=True) + except ValueError as e: + raise _create_dataset_load_error(e) + else: + format_map = { + ".json": "json", + ".jsonl": "json", + ".csv": "csv", + ".parquet": "parquet", + } + + file_extension = local_path.suffix + dataset_type = format_map.get(file_extension) + + if dataset_type is None: + raise ValueError(f"Unsupported file extension: {file_extension}") + + try: + dataset = datasets.load_dataset(dataset_type, + data_files=str(local_path), + split=dataset_config.split) + except ValueError as e: + raise _create_dataset_load_error(e) + + logging.debug("Finished loading local dataset") + + return iter(dataset) + + +@click.command() +@click.option("--dataset-name", type=str, help="Dataset name in HuggingFace.") +@click.option("--dataset-config-name", + type=str, + default=None, + help="Dataset config name in HuggingFace (if exists).") +@click.option("--dataset-split", + type=str, + required=True, + help="Split of the dataset to use.") +@click.option("--dataset-input-key", + type=str, + help="The dataset dictionary key for input.") +@click.option("--dataset-image-key", + type=str, + default="image", + help="The dataset dictionary key for images.") +@click.option("--dataset-prompt-key", + type=str, + default=None, + help="The dataset dictionary key for prompt (if exists).") +@click.option( + "--dataset-local-path", + type=str, + default=None, + help= + "The local path to the dataset to be loaded when using an offline cache.") +@click.option( + "--dataset-prompt", + type=str, + default=None, + help="The prompt string when there is no prompt key for the dataset.") +@click.option("--dataset-output-key", + type=str, + default=None, + help="The dataset dictionary key for output (if exists).") +@click.option( + "--num-requests", + type=int, + default=None, + help= + "Number of requests to be generated. Will be capped to min(dataset.num_rows, num_requests)." +) +@click.option( + "--max-input-len", + type=int, + default=None, + help= + "Maximum input sequence length for a given request. This will be used to filter out the requests with long input sequence length. Default will include all the requests." +) +@click.option( + "--output-len-dist", + type=str, + default=None, + callback=validate_output_len_dist, + help= + "Output length distribution. Default will be the length of the golden output from the dataset. Format: <output_len_mean>,<output_len_stdev>. E.g. 100,10 will randomize the output length with mean=100 and variance=10." +) +@click.pass_obj +def dataset(root_args, **kwargs): + """Prepare dataset from real dataset.""" + dataset_config = DatasetConfig(**{ + k[8:]: v + for k, v in kwargs.items() if k.startswith('dataset_') + }) + + input_ids = [] + input_lens = [] + output_lens = [] + task_ids = [] + req_cnt = 0 + modality = None + multimodal_texts = [] + multimodal_image_paths = [] + for req in load_dataset(dataset_config): + if any(key in req for key in ['image', 'image_1', 'video']): + # multimodal input + if 'video' in req and req['video'] is not None: + assert "Not supported yet" + assert kwargs['output_len_dist'] is not None, ( + "Output length distribution must be set for multimodal requests." + ) + modality = 'image' + text = dataset_config.get_prompt(req) + images = dataset_config.get_images(req) + image_paths = [] + for image in images: + if image is not None: + if isinstance(image, str): + image_paths.append(image) + elif isinstance(image, Image.Image): + with tempfile.NamedTemporaryFile( + suffix=".jpg", delete=False) as tmp_file: + logging.debug(f"Saving image to {tmp_file.name}") + image = image.convert("RGB") + image.save(tmp_file, "JPEG") + filepath = tmp_file.name + image_paths.append(filepath) + else: + raise ValueError(f"Invalid image path: {image}") + multimodal_texts.append(text) + multimodal_image_paths.append(image_paths) + else: + # text input + prompt = dataset_config.get_prompt( + req) + ' ' + dataset_config.get_input(req) + logging.debug(f"Input sequence: {prompt}") + line = root_args.tokenizer.encode(prompt) + if kwargs['max_input_len'] and len(line) > kwargs['max_input_len']: + continue + input_ids.append(line) + input_lens.append(len(line)) + + # output if fetch from golden + if kwargs['output_len_dist'] is None: + output_lens.append( + len( + root_args.tokenizer.encode( + dataset_config.get_output(req)))) + + # lora task id + task_id = root_args.task_id + if root_args.rand_task_id is not None: + min_id, max_id = root_args.rand_task_id + task_id = random.randint(min_id, max_id) + task_ids.append(task_id) + + req_cnt += 1 + if kwargs['num_requests'] and req_cnt >= kwargs['num_requests']: + break + + if kwargs['num_requests'] and (len(input_ids) if modality is None else len( + multimodal_texts)) < kwargs['num_requests']: + logging.warning( + f"Number of requests={len(input_ids) if modality is None else len(multimodal_texts)} is" + f" smaller than the num-requests user set={kwargs['num_requests']}." + ) + + # output if randomized + if kwargs['output_len_dist'] is not None: + osl_mean, osl_stdev = kwargs['output_len_dist'] + output_lens = get_norm_dist_lengths( + osl_mean, osl_stdev, + len(input_ids) if modality is None else len(multimodal_texts), + root_args.random_seed) + logging.debug(f"Input lengths: {[len(i) for i in input_ids]}") + logging.debug(f"Output lengths: {output_lens}") + if modality is not None: + logging.debug(f"Modality: {modality}") + + if modality is not None: + if not root_args.std_out: + multimodal_dataset_dump( + multimodal_texts, multimodal_image_paths, output_lens, task_ids, + { + "workload_type": "dataset", + "tokenizer": root_args.tokenizer.__class__.__name__, + "num_requests": len(task_ids), + "max_output_len": max(output_lens) + }, root_args.output) + else: + print_multimodal_dataset( + multimodal_texts, + multimodal_image_paths, + output_lens, + ) + else: + if not root_args.std_out: + text_dataset_dump( + input_lens, input_ids, output_lens, task_ids, { + "workload_type": "dataset", + "tokenizer": root_args.tokenizer.__class__.__name__, + "num_requests": len(input_ids), + "max_input_len": max(input_lens), + "max_output_len": max(output_lens) + }, root_args.output) + else: + print_text_dataset( + input_ids, + output_lens, + ) diff --git a/benchmarks/cpp/utils/prepare_synthetic_data.py b/benchmarks/cpp/utils/prepare_synthetic_data.py new file mode 100644 index 000000000000..b072b712d085 --- /dev/null +++ b/benchmarks/cpp/utils/prepare_synthetic_data.py @@ -0,0 +1,164 @@ +import random +import warnings + +import click +from utils.utils import (gen_random_tokens, get_norm_dist_lengths, + get_unif_dist_lengths, print_text_dataset, + text_dataset_dump) + + +def _generate_task_ids_and_lora_config(root_args, num_reqs): + """Generate task IDs and determine LoRA configuration based on root_args.""" + if root_args.rand_task_id is None: + task_ids = [root_args.task_id for _ in range(num_reqs)] + else: + min_id, max_id = root_args.rand_task_id + task_ids = [random.randint(min_id, max_id) for _ in range(num_reqs)] + + use_task_ids = root_args.task_id != -1 or root_args.rand_task_id is not None + + # Determine if LoRA should be used (requires both task IDs and lora_dir) + use_lora = use_task_ids and root_args.lora_dir is not None + + # Warn if task IDs are specified but no LoRA directory is provided + if use_task_ids and not use_lora: + warnings.warn( + "Task IDs require LoRA directory. Use --lora-dir or omit task IDs.", + UserWarning) + + return (task_ids, task_ids if use_task_ids else None, { + "lora_dir": root_args.lora_dir + } if use_lora else None) + + +@click.command() +@click.option("--num-requests", + required=True, + type=int, + help='Number of requests to be generated') +@click.option('--input-mean', + required=True, + type=int, + help='normal dist mean for input tokens') +@click.option('--input-stdev', + required=True, + type=int, + help='normal dist stdev for input tokens') +@click.option('--output-mean', + required=True, + type=int, + help='normal dist mean for output tokens') +@click.option('--output-stdev', + required=True, + type=int, + help='normal dist stdev for output tokens') +@click.pass_obj +def token_norm_dist(root_args, **kwargs): + """Prepare synthetic dataset by generating random tokens with normal dist lengths.""" + input_ids = [] + input_lens = [] + output_lens = [] + + input_lens = get_norm_dist_lengths(kwargs['input_mean'], + kwargs['input_stdev'], + kwargs['num_requests'], + root_args.random_seed) + + num_reqs = len(input_lens) + output_lens = get_norm_dist_lengths(kwargs['output_mean'], + kwargs['output_stdev'], num_reqs, + root_args.random_seed) + + max_input_len = max(input_lens) + max_output_len = max(output_lens) + + input_ids = gen_random_tokens(input_lens, root_args.tokenizer, + root_args.random_seed) + + task_ids, print_task_ids, lora_config = _generate_task_ids_and_lora_config( + root_args, num_reqs) + + if not root_args.std_out: + text_dataset_dump( + input_lens, input_ids, output_lens, task_ids, { + "workload_type": "token-norm-dist", + "input_mean": kwargs['input_mean'], + "input_stdev": kwargs['input_stdev'], + "output_mean": kwargs['output_mean'], + "output_stdev": kwargs['output_stdev'], + "num_requests": kwargs['num_requests'], + "tokenize_vocabsize": root_args.tokenizer.vocab_size, + "max_input_len": max_input_len, + "max_output_len": max_output_len + }, root_args.output) + else: + print_text_dataset(input_ids, + output_lens, + task_ids=print_task_ids, + lora_config=lora_config) + + +@click.command() +@click.option("--num-requests", + required=True, + type=int, + help='Number of requests to be generated') +@click.option('--input-min', + required=True, + type=int, + help='uniform dist (inclusive) min for input tokens') +@click.option('--input-max', + required=True, + type=int, + help='normal dist (inclusive) max for input tokens') +@click.option('--output-min', + required=True, + type=int, + help='normal dist (inclusive) min for output tokens') +@click.option('--output-max', + required=True, + type=int, + help='normal dist (inclusive) max for output tokens') +@click.pass_obj +def token_unif_dist(root_args, **kwargs): + """Prepare synthetic dataset by generating random tokens with normal uniformly lengths.""" + input_ids = [] + input_lens = [] + output_lens = [] + + input_lens = get_unif_dist_lengths(kwargs['input_min'], kwargs['input_max'], + kwargs['num_requests'], + root_args.random_seed) + + num_reqs = len(input_lens) + output_lens = get_unif_dist_lengths(kwargs['output_min'], + kwargs['output_max'], num_reqs, + root_args.random_seed) + + max_input_len = max(input_lens) + max_output_len = max(output_lens) + + input_ids = gen_random_tokens(input_lens, root_args.tokenizer, + root_args.random_seed) + + task_ids, print_task_ids, lora_config = _generate_task_ids_and_lora_config( + root_args, num_reqs) + + if not root_args.std_out: + text_dataset_dump( + input_lens, input_ids, output_lens, task_ids, { + "workload_type": "token-unif-dist", + "input_min": kwargs['input_min'], + "input_max": kwargs['input_max'], + "output_min": kwargs['output_min'], + "output_max": kwargs['output_max'], + "num_requests": kwargs['num_requests'], + "tokenize_vocabsize": root_args.tokenizer.vocab_size, + "max_input_len": max_input_len, + "max_output_len": max_output_len + }, root_args.output) + else: + print_text_dataset(input_ids, + output_lens, + task_ids=print_task_ids, + lora_config=lora_config) diff --git a/benchmarks/cpp/utils/utils.cpp b/benchmarks/cpp/utils/utils.cpp new file mode 100644 index 000000000000..0cbcf1c0468d --- /dev/null +++ b/benchmarks/cpp/utils/utils.cpp @@ -0,0 +1,170 @@ + +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & + *AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "utils.h" +#include "tensorrt_llm/common/config.h" +#include "tensorrt_llm/common/logger.h" +#include <random> + +#include <filesystem> +#include <fstream> + +TRTLLM_NAMESPACE_BEGIN + +namespace benchmark +{ + +std::vector<std::vector<SizeType32>> parseVectorOfVectors(std::string const& input) +{ + std::vector<std::vector<SizeType32>> result; + std::regex outer_regex(R"(\[(.*?)\])"); + std::regex inner_regex(R"(\d+)"); + auto outer_begin = std::sregex_iterator(input.begin(), input.end(), outer_regex); + auto outer_end = std::sregex_iterator(); + + for (std::sregex_iterator i = outer_begin; i != outer_end; ++i) + { + std::smatch match = *i; + std::string inner_str = match.str(1); + std::vector<int> inner_vec; + auto inner_begin = std::sregex_iterator(inner_str.begin(), inner_str.end(), inner_regex); + auto inner_end = std::sregex_iterator(); + + for (std::sregex_iterator j = inner_begin; j != inner_end; ++j) + { + std::smatch inner_match = *j; + inner_vec.push_back(std::stoi(inner_match.str())); + } + result.push_back(inner_vec); + } + return result; +} + +texec::LookaheadDecodingConfig parseLookaheadConfig(std::string const& input) +{ + std::regex regex("\\[ *(\\d+) *, *(\\d+) *, *(\\d+) *\\]"); + std::smatch match; + if (std::regex_match(input, match, regex)) + { + TLLM_CHECK(match.size() == 4); + auto w = std::stoi(match[1]); + auto n = std::stoi(match[2]); + auto g = std::stoi(match[3]); + return texec::LookaheadDecodingConfig(w, n, g); + } + else + { + TLLM_LOG_WARNING("cannot parse lookahead config from '%s'", input.c_str()); + return texec::LookaheadDecodingConfig(); + } +} + +Samples parseWorkloadJson( + std::filesystem::path const& datasetPath, int maxNumSamples, std::optional<SizeType32> const maxPromptLen) +{ + auto constexpr allowExceptions = true; + auto constexpr ignoreComments = true; + TLLM_CHECK_WITH_INFO(std::filesystem::exists(datasetPath), "File does not exist: %s", datasetPath.c_str()); + std::ifstream jsonStream(datasetPath); + auto json = nlohmann::json::parse(jsonStream, nullptr, allowExceptions, ignoreComments); + + Samples samples; + + for (auto const& sample : json["samples"]) + { + if (samples.size() >= maxNumSamples) + break; + int32_t taskId = sample.count("task_id") ? sample["task_id"].template get<int32_t>() : -1; + auto input_ids(sample["input_ids"].template get<std::vector<int32_t>>()); + if (maxPromptLen && (input_ids.size() > maxPromptLen.value())) + { + input_ids.resize(maxPromptLen.value()); + } + samples.emplace_back(Sample{std::move(input_ids), sample["output_len"], taskId}); + } + + if (samples.size() < maxNumSamples) + { + TLLM_LOG_WARNING( + "Dataset size %zu is smaller than given max_num_samples " + "%d, max_num_samples will be ignored.\n", + samples.size(), maxNumSamples); + } + return samples; +} + +std::vector<double> generateRandomExponentialValues(int count, float lambda, int seed) +{ + // Set a constant seed for reproducibility + std::mt19937 gen(seed); + + // Create an exponential distribution object + std::exponential_distribution<double> distribution(lambda); + + // Generate random numbers from the exponential distribution + std::vector<double> randomValues; + for (int i = 0; i < count; ++i) + { + double randomValue = distribution(gen); + randomValues.push_back(randomValue); + } + + return randomValues; +} + +std::vector<double> computeTimeDelays(BenchmarkParams const& benchmarkParams, int numDelays) +{ + std::vector<double> timeDelays; + if (benchmarkParams.requestRate.has_value() && benchmarkParams.requestRate.value() > 0.0) + { + if (benchmarkParams.enableExpDelays) + { + timeDelays = generateRandomExponentialValues( + numDelays, benchmarkParams.requestRate.value(), benchmarkParams.randomSeed); + } + else + { + timeDelays.assign(numDelays, 1.0 / benchmarkParams.requestRate.value()); + } + } + else + { + timeDelays.assign(numDelays, 0.0); + } + + return timeDelays; +} + +std::ostream& operator<<(std::ostream& os, RecordTimeMetric const& metric) +{ + os << metric.mAvg << "," << metric.mMax << "," << metric.mMin << "," << metric.mP99 << "," << metric.mP90 << "," + << metric.mP50; + return os; +} + +std::ostream& operator<<(std::ostream& os, RecordBwMetric const& metric) +{ + os << metric.mAvg << "," << metric.mMax << "," << metric.mMin << "," << metric.mP99 << "," << metric.mP90 << "," + << metric.mP50; + return os; +} + +} // namespace benchmark + +TRTLLM_NAMESPACE_END diff --git a/benchmarks/cpp/utils/utils.h b/benchmarks/cpp/utils/utils.h new file mode 100644 index 000000000000..fba30fee69ae --- /dev/null +++ b/benchmarks/cpp/utils/utils.h @@ -0,0 +1,244 @@ + +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/config.h" +#include "tensorrt_llm/executor/executor.h" + +#include <cstdint> +#include <cxxopts.hpp> +#include <iostream> +#include <nlohmann/json.hpp> +#include <numeric> +#include <optional> +#include <string> +#include <utility> + +#pragma once + +TRTLLM_NAMESPACE_BEGIN + +namespace benchmark +{ + +// using namespace tensorrt_llm::batch_manager; +using namespace tensorrt_llm::runtime; + +namespace texec = tensorrt_llm::executor; + +std::vector<std::vector<SizeType32>> parseVectorOfVectors(std::string const& input); + +texec::LookaheadDecodingConfig parseLookaheadConfig(std::string const& input); + +struct BenchmarkParams +{ + std::optional<SizeType32> maxTokensInPagedKvCache{std::nullopt}; + std::optional<float> freeGpuMemoryFraction{std::nullopt}; + std::vector<std::optional<float>> freeGpuMemoryFractions{std::nullopt}; + + std::optional<float> crossKvCacheFraction{std::nullopt}; + bool enableTrtOverlap{false}; + bool enableBatchSizeTuning{false}; + bool enableMaxNumTokensTuning{false}; + bool enableBlockReuse{false}; + bool enableChunkedContext{true}; + bool streaming{false}; + bool enableExpDelays{false}; + std::vector<std::optional<bool>> enableChunekedContextVec{std::nullopt}; + std::optional<float> requestRate{std::nullopt}; + std::optional<int> concurrency{std::nullopt}; + std::optional<SizeType32> maxBatchSize{std::nullopt}; + std::vector<std::optional<SizeType32>> maxBatchSizes{std::nullopt}; + std::optional<SizeType32> maxNumTokens{std::nullopt}; + std::vector<std::optional<SizeType32>> maxNumTokensVec{std::nullopt}; + int randomSeed = 430; + std::optional<std::vector<int>> maxAttentionWindowVec{std::nullopt}; + std::optional<int> sinkTokenLength{std::nullopt}; + bool multiBlockMode{true}; + bool enableContextFMHAFP32Acc{false}; + bool cudaGraphMode{false}; + SizeType32 cudaGraphCacheSize{0}; + + // lora / peft params + std::optional<std::string> loraDir{std::nullopt}; + SizeType32 loraDeviceNumModLayers{0}; + size_t loraHostCacheSize{1024 * 2024 * 1024}; + + // KV cache block offloading + size_t kvHostCacheSize{0}; + + // Weights offloading + float gpuWeightsPercent{1.0}; + + // Decoding params + std::optional<std::vector<std::vector<SizeType32>>> medusaChoices; + + std::optional<texec::EagleConfig> eagleConfig; + std::optional<float> temperature; + + std::optional<texec::LookaheadDecodingConfig> executorLookaheadConfig; + std::optional<texec::LookaheadDecodingConfig> requestLookaheadConfig; + + bool enableCollectkvCacheTransferTime = false; + bool enableCollectIterStats = false; +}; + +struct RecordTimeMetric +{ + + RecordTimeMetric(std::string tag) + : mTag(std::move(tag)) + { + } + + std::string mTag; + + std::vector<float> mDataTimes; + + float mAvg; + float mP99; + float mP95; + float mP90; + float mP50; + float mMax; + float mMin; + + static float calcPercentile(std::vector<float> const& latencies, int percentile) + { + int const index = static_cast<int>(std::ceil((percentile / 100.0) * latencies.size())) - 1; + return latencies[index]; + } + + void calculate() + { + TLLM_CHECK_WITH_INFO(mDataTimes.size() > 0, "No data to calculate for tag:%s", mTag.c_str()); + mAvg = std::accumulate(mDataTimes.begin(), mDataTimes.end(), 0.F) / mDataTimes.size(); + + std::sort(mDataTimes.begin(), mDataTimes.end()); + + mP99 = calcPercentile(mDataTimes, 99); + mP90 = calcPercentile(mDataTimes, 90); + mP50 = calcPercentile(mDataTimes, 50); + mMax = mDataTimes.back(); + mMin = mDataTimes.front(); + } + + void report() const + { + + printf("[BENCHMARK] avg_%s(ms) %.2f\n", mTag.c_str(), mAvg); + printf("[BENCHMARK] max_%s(ms) %.2f\n", mTag.c_str(), mMax); + printf("[BENCHMARK] min_%s(ms) %.2f\n", mTag.c_str(), mMin); + + printf("[BENCHMARK] p99_%s(ms) %.2f\n", mTag.c_str(), mP99); + + printf("[BENCHMARK] p90_%s(ms) %.2f\n", mTag.c_str(), mP90); + + printf("[BENCHMARK] p50_%s(ms) %.2f\n\n", mTag.c_str(), mP50); + } + + std::vector<std::string> genHeaders() const + { + std::string timeTag = mTag + "(ms)"; + return { + "avg_" + timeTag, "max_" + timeTag, "min_" + timeTag, "p99" + timeTag, "p90" + timeTag, "p50" + timeTag}; + } +}; + +struct RecordBwMetric +{ + + RecordBwMetric(std::string tag) + : mTag(std::move(tag)) + { + } + + std::string mTag; + + std::vector<float> mDataTps; + + float mAvg; + float mP99; + float mP95; + float mP90; + float mP50; + float mMax; + float mMin; + + static float calcPercentile(std::vector<float> const& throughputs, int percentile) + { + int const index = static_cast<int>(std::ceil((percentile / 100.0) * throughputs.size())) - 1; + return throughputs[index]; + } + + void calculate() + { + TLLM_CHECK_WITH_INFO(mDataTps.size() > 0, "No data to calculate for tag:%s", mTag.c_str()); + mAvg = std::accumulate(mDataTps.begin(), mDataTps.end(), 0.F) / mDataTps.size(); + + std::sort(mDataTps.begin(), mDataTps.end(), std::greater<float>()); + + mP99 = calcPercentile(mDataTps, 99); + mP90 = calcPercentile(mDataTps, 90); + mP50 = calcPercentile(mDataTps, 50); + mMax = mDataTps.front(); + mMin = mDataTps.back(); + } + + void report() const + { + + printf("[BENCHMARK] avg_%s(Gb/sec) %.8f\n", mTag.c_str(), mAvg); + printf("[BENCHMARK] max_%s(Gb/sec) %.8f\n", mTag.c_str(), mMax); + printf("[BENCHMARK] min_%s(Gb/sec) %.8f\n", mTag.c_str(), mMin); + + printf("[BENCHMARK] p99_%s(Gb/sec) %.8f\n", mTag.c_str(), mP99); + + printf("[BENCHMARK] p90_%s(Gb/sec) %.8f\n", mTag.c_str(), mP90); + + printf("[BENCHMARK] p50_%s(Gb/sec) %.8f\n\n", mTag.c_str(), mP50); + } + + std::vector<std::string> genHeaders() const + { + std::string tpTag = mTag + "(Gb/sec)"; + return {"avg_" + tpTag, "max_" + tpTag, "min_" + tpTag, "p99" + tpTag, "p90" + tpTag, "p50" + tpTag}; + } +}; + +std::ostream& operator<<(std::ostream& os, RecordTimeMetric const& metric); +std::ostream& operator<<(std::ostream& os, RecordBwMetric const& metric); + +struct Sample +{ + std::vector<int32_t> inputIds; + int32_t outputLen; + int32_t taskId; +}; + +using Samples = std::vector<Sample>; + +Samples parseWorkloadJson( + std::filesystem::path const& datasetPath, int maxNumSamples, std::optional<SizeType32> const maxPromptLen); + +std::vector<double> generateRandomExponentialValues(int count, float lambda, int seed); + +std::vector<double> computeTimeDelays(BenchmarkParams const& benchmarkParams, int numDelays); + +} // namespace benchmark + +TRTLLM_NAMESPACE_END diff --git a/benchmarks/cpp/utils/utils.py b/benchmarks/cpp/utils/utils.py new file mode 100644 index 000000000000..c395cf6c9449 --- /dev/null +++ b/benchmarks/cpp/utils/utils.py @@ -0,0 +1,168 @@ +import json +import math +import os +import random +from typing import List, Union + +import numpy as np +from pydantic import BaseModel + + +class TextSample(BaseModel): + input_len: int + input_ids: List[int] + output_len: int + task_id: int + + +class MultimodalSample(BaseModel): + task_id: int + prompt: str + media_paths: List[str] + output_len: int + + +class Workload(BaseModel): + metadata: dict + samples: List[Union[TextSample, MultimodalSample]] = [] + + def __init__(self, **kwargs) -> None: + super().__init__(**kwargs) + self.setup_workload_name() + + def setup_workload_name(self): + # Keys to ignore + ignore_keys = ['tokenizer'] + # Create a string by concatenating keys and values with "__" + workload_name = '__'.join(f'{key}:{value}' + for key, value in self.metadata.items() + if key not in ignore_keys) + self.metadata.setdefault('workload_name', workload_name) + + +def text_dataset_dump(input_lens, input_ids, output_lens, task_ids, metadata, + output_file): + samples = [] + for i in range(len(input_ids)): + samples.append( + TextSample(input_len=input_lens[i], + input_ids=input_ids[i], + output_len=output_lens[i], + task_id=task_ids[i])) + workload = Workload(metadata=metadata, samples=samples) + os.makedirs(os.path.dirname(output_file), exist_ok=True) + with open(output_file, 'w') as f: + json.dump(workload.model_dump(), f) + + +def multimodal_dataset_dump(multimodal_texts, multimodal_image_paths, + output_lens, task_ids, metadata, output_file): + samples = [] + for i in range(len(multimodal_texts)): + samples.append( + MultimodalSample(task_id=task_ids[i], + prompt=multimodal_texts[i], + media_paths=multimodal_image_paths[i], + output_len=output_lens[i])) + workload = Workload(metadata=metadata, samples=samples) + os.makedirs(os.path.dirname(output_file), exist_ok=True) + with open(output_file, 'w') as f: + json.dump(workload.model_dump(), f) + + +def print_text_dataset(input_ids, output_lens, task_ids=None, lora_config=None): + for i, input_tokens in enumerate(input_ids): + d = { + "task_id": i, + "input_ids": input_tokens, + "output_tokens": output_lens[i] + } + + # Add LoRA request if task_ids indicate LoRA usage + if task_ids is not None and lora_config is not None: + task_id = task_ids[i] + if task_id != -1: # -1 means no LoRA + d["lora_request"] = { + "lora_name": + f"lora_{task_id}", + "lora_int_id": + task_id, + "lora_path": + os.path.join(lora_config.get("lora_dir", "loras"), + str(task_id)) + } + + print(json.dumps(d, separators=(',', ':'), ensure_ascii=False)) + + +def print_multimodal_dataset(multimodal_texts, multimodal_image_paths, + output_lens): + for i, (text, image_paths) in enumerate( + zip(multimodal_texts, multimodal_image_paths)): + d = { + "task_id": i, + "prompt": text, + "media_paths": image_paths, + "output_tokens": output_lens[i] + } + print(json.dumps(d, separators=(',', ':'), ensure_ascii=False)) + + +def get_list_of_delays(delay_dist, mean_time_bet_reqs, num_reqs, random_seed): + if delay_dist == "constant": + delays = [mean_time_bet_reqs] * num_reqs + elif delay_dist == "exponential_dist": + delays = get_exponential_dist_delays(mean_time_bet_reqs, num_reqs, + random_seed) + + return delays + + +def get_exponential_dist_delays(mean_time_bet_reqs, num_reqs, random_seed): + # set seed for determinism + np.random.seed(random_seed) + return np.random.exponential(mean_time_bet_reqs, num_reqs).tolist() + + +def get_norm_dist_lengths(mean, stdev, num_reqs, random_seed): + # set seed for determinism + np.random.seed(random_seed) + numbers_list = np.random.normal(loc=mean, scale=stdev, + size=num_reqs).tolist() + return [max(1, math.ceil(x)) for x in numbers_list] + + +def get_unif_dist_lengths(min_len, max_len, num_reqs, random_seed): + # set seed for determinism + rng = np.random.default_rng(random_seed) + numbers = rng.integers(low=min_len, high=max_len + 1, size=num_reqs) + return numbers.tolist() + + +def gen_random_tokens(ip_lens, tokenizer, random_seed): + + def get_sample_from_population(population_range, sample_size): + # random.sample can not sample a value more than once. hence the check + if sample_size < len(population_range): + sample = random.sample(population_range, sample_size) + else: + sample = random.choices(population_range, k=sample_size) + + return sample + + input_ids = [] + random.seed(random_seed) + for ip_len in ip_lens: + start_ids = get_sample_from_population(range(0, tokenizer.vocab_size), + ip_len) + # Make sure it does not contain EOS token + eos_id = tokenizer.encode(tokenizer.eos_token, add_special_tokens=False) + while set(eos_id).issubset(start_ids): + tmp_id = (eos_id[0] + 1) % tokenizer.vocab_size + start_ids = [ + tmp_id if element == eos_id[0] else element + for element in start_ids + ] + input_ids.append(start_ids) + + return input_ids diff --git a/cpp/CMakeLists.txt b/cpp/CMakeLists.txt index 5dec92599ae6..7a46725c80c7 100644 --- a/cpp/CMakeLists.txt +++ b/cpp/CMakeLists.txt @@ -45,6 +45,7 @@ add_compile_definitions(TRTLLM_ABI_NAMESPACE=${TRTLLM_ABI_NAMESPACE}) # Build options option(BUILD_PYT "Build in PyTorch TorchScript class mode" ON) option(BUILD_TESTS "Build Google tests" ON) +option(BUILD_BENCHMARKS "Build benchmarks" ON) option(BUILD_DEEP_EP "Build the Deep EP module" ON) option(BUILD_DEEP_GEMM "Build the DeepGEMM module" ON) option(BUILD_FLASH_MLA "Build the FlashMLA module" ON) @@ -125,6 +126,12 @@ else() message(STATUS "Not building Google tests") endif() +if(BUILD_BENCHMARKS) + message(STATUS "Building benchmarks") +else() + message(STATUS "Not building benchmarks") +endif() + if(BUILD_MICRO_BENCHMARKS) message(STATUS "Building C++ micro benchmarks") else() @@ -242,6 +249,8 @@ if(ENABLE_MULTI_DEVICE) endif() # TRT dependencies +find_package(TensorRT 10 REQUIRED COMPONENTS OnnxParser) +set(TRT_LIB TensorRT::NvInfer) get_filename_component(TRT_LLM_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR} PATH) @@ -282,6 +291,7 @@ include_directories( ${CUDAToolkit_INCLUDE_DIRS} ${CUDAToolkit_INCLUDE_DIRS}/cccl ${CUDNN_ROOT_DIR}/include + $<TARGET_PROPERTY:TensorRT::NvInfer,INTERFACE_INCLUDE_DIRECTORIES> ${maybe_nvtx_includedir} ${CMAKE_BINARY_DIR}/_deps/cutlass-src/include ${CMAKE_BINARY_DIR}/_deps/cutlass-src/tools/util/include @@ -396,13 +406,8 @@ endif() option(ENABLE_BOLT_COMPATIBLE "Enable BOLT-compatible build flags" OFF) if(ENABLE_BOLT_COMPATIBLE AND NOT WIN32) message(STATUS "BOLT compatible flags enabled") - # Compiler flags for C/C++ -fno-reorder-blocks-and-partition is GCC-only - # (required for GCC 8+ per BOLT docs). Clang does not support this flag and - # does not enable the corresponding optimization by default. - if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU") - add_compile_options(-fno-reorder-blocks-and-partition) - endif() - add_compile_options(-fno-plt) + # Compiler flags for C/C++ + add_compile_options(-fno-reorder-blocks-and-partition -fno-plt) # Linker flags - applies to shared, module, and executable targets add_link_options(-Wl,--emit-relocs) # Disable stripping - required for BOLT (affects pybind11 POST_BUILD strip) @@ -659,6 +664,11 @@ if(BUILD_TESTS) add_subdirectory(tests) endif() +if(BUILD_BENCHMARKS) + add_subdirectory(${TRT_LLM_ROOT_DIR}/benchmarks/cpp + ${CMAKE_BINARY_DIR}/benchmarks) +endif() + if(BUILD_MICRO_BENCHMARKS) add_subdirectory(${TRT_LLM_ROOT_DIR}/cpp/micro_benchmarks ${CMAKE_BINARY_DIR}/micro_benchmarks) @@ -673,6 +683,6 @@ if(MEASURE_BUILD_TIME) endif() set(BUILD_WHEEL_TARGETS - tensorrt_llm + tensorrt_llm;nvinfer_plugin_tensorrt_llm CACHE STRING "Targets used to build wheel") add_custom_target(build_wheel_targets DEPENDS ${BUILD_WHEEL_TARGETS}) diff --git a/cpp/cmake/modules/FindTensorRT.cmake b/cpp/cmake/modules/FindTensorRT.cmake new file mode 100644 index 000000000000..9e7e35b51bae --- /dev/null +++ b/cpp/cmake/modules/FindTensorRT.cmake @@ -0,0 +1,190 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# + +# TensorRT install path in docker image +set(TensorRT_WELL_KNOWN_ROOT /usr/local/tensorrt) + +find_path( + TensorRT_INCLUDE_DIR + NAMES NvInfer.h + PATHS ${TensorRT_WELL_KNOWN_ROOT}/include) + +function(_tensorrt_get_version) + unset(TensorRT_VERSION_STRING PARENT_SCOPE) + set(_hdr_file "${TensorRT_INCLUDE_DIR}/NvInferVersion.h") + + if(NOT EXISTS "${_hdr_file}") + return() + endif() + + file(STRINGS "${_hdr_file}" IS_10_11_NEW_MACRO REGEX "TRT_MAJOR_ENTERPRISE") + if(IS_10_11_NEW_MACRO) + file(STRINGS "${_hdr_file}" VERSION_STRINGS + REGEX "#define TRT_.+_ENTERPRISE.*") + foreach(TYPE MAJOR MINOR PATCH BUILD) + string(REGEX MATCH "TRT_${TYPE}_ENTERPRISE [0-9]+" TRT_TYPE_STRING + ${VERSION_STRINGS}) + string(REGEX MATCH "[0-9]+" TensorRT_VERSION_${TYPE} ${TRT_TYPE_STRING}) + endforeach(TYPE) + else() + file(STRINGS "${_hdr_file}" VERSION_STRINGS REGEX "#define NV_TENSORRT_.*") + foreach(TYPE MAJOR MINOR PATCH BUILD) + string(REGEX MATCH "NV_TENSORRT_${TYPE} [0-9]+" TRT_TYPE_STRING + ${VERSION_STRINGS}) + string(REGEX MATCH "[0-9]+" TensorRT_VERSION_${TYPE} ${TRT_TYPE_STRING}) + endforeach(TYPE) + endif() + + set(TensorRT_VERSION_MAJOR + ${TensorRT_VERSION_MAJOR} + PARENT_SCOPE) + set(TensorRT_VERSION_STRING + "${TensorRT_VERSION_MAJOR}.${TensorRT_VERSION_MINOR}.${TensorRT_VERSION_PATCH}.${TensorRT_VERSION_BUILD}" + PARENT_SCOPE) +endfunction(_tensorrt_get_version) + +_tensorrt_get_version() + +macro(_tensorrt_find_dll VAR) + find_file( + ${VAR} + NAMES ${ARGN} + HINTS ${TensorRT_ROOT} + PATH_SUFFIXES bin) +endmacro(_tensorrt_find_dll) + +find_library( + TensorRT_LIBRARY + NAMES "nvinfer_${TensorRT_VERSION_MAJOR}" nvinfer + PATHS ${TensorRT_WELL_KNOWN_ROOT}/lib) + +if(WIN32) + _tensorrt_find_dll(TensorRT_DLL "nvinfer_${TensorRT_VERSION_MAJOR}.dll" + nvinfer.dll) +endif() + +if(TensorRT_LIBRARY) + set(TensorRT_LIBRARIES ${TensorRT_LIBRARIES} ${TensorRT_LIBRARY}) +endif(TensorRT_LIBRARY) + +if(TensorRT_FIND_COMPONENTS) + list(REMOVE_ITEM TensorRT_FIND_COMPONENTS "nvinfer") + + if("OnnxParser" IN_LIST TensorRT_FIND_COMPONENTS) + find_path( + TensorRT_OnnxParser_INCLUDE_DIR + NAMES NvOnnxParser.h + PATHS ${TensorRT_WELL_KNOWN_ROOT}/include) + + find_library( + TensorRT_OnnxParser_LIBRARY + NAMES "nvonnxparser_${TensorRT_VERSION_MAJOR}" nvonnxparser + PATHS ${TensorRT_WELL_KNOWN_ROOT}/lib) + if(TensorRT_OnnxParser_LIBRARY AND TensorRT_LIBRARIES) + set(TensorRT_LIBRARIES ${TensorRT_LIBRARIES} + ${TensorRT_OnnxParser_LIBRARY}) + set(TensorRT_OnnxParser_FOUND TRUE) + endif() + + if(WIN32) + _tensorrt_find_dll( + TensorRT_OnnxParser_DLL "nvonnxparser_${TensorRT_VERSION_MAJOR}.dll" + nvonnxparser.dll) + endif() + endif() + + if("Plugin" IN_LIST TensorRT_FIND_COMPONENTS) + find_path( + TensorRT_Plugin_INCLUDE_DIR + NAMES NvInferPlugin.h + PATHS ${TensorRT_WELL_KNOWN_ROOT}/include) + + find_library( + TensorRT_Plugin_LIBRARY + NAMES "nvinfer_plugin_${TensorRT_VERSION_MAJOR}" nvinfer_plugin + PATHS ${TensorRT_WELL_KNOWN_ROOT}/lib) + + if(TensorRT_Plugin_LIBRARY AND TensorRT_LIBRARIES) + set(TensorRT_LIBRARIES ${TensorRT_LIBRARIES} ${TensorRT_Plugin_LIBRARY}) + set(TensorRT_Plugin_FOUND TRUE) + endif() + + if(WIN32) + _tensorrt_find_dll( + TensorRT_Plugin_DLL "nvinfer_plugin_${TensorRT_VERSION_MAJOR}.dll" + nvinfer_plugin.dll) + endif() + endif() +endif() + +include(FindPackageHandleStandardArgs) +find_package_handle_standard_args( + TensorRT + FOUND_VAR TensorRT_FOUND + REQUIRED_VARS TensorRT_LIBRARY TensorRT_LIBRARIES TensorRT_INCLUDE_DIR + VERSION_VAR TensorRT_VERSION_STRING + HANDLE_COMPONENTS) + +if(NOT TARGET TensorRT::NvInfer) + add_library(TensorRT::NvInfer SHARED IMPORTED) + target_include_directories(TensorRT::NvInfer SYSTEM + INTERFACE "${TensorRT_INCLUDE_DIR}") + if(WIN32) + set_property(TARGET TensorRT::NvInfer PROPERTY IMPORTED_LOCATION + "${TensorRT_DLL}") + set_property(TARGET TensorRT::NvInfer PROPERTY IMPORTED_IMPLIB + "${TensorRT_LIBRARY}") + else() + set_property(TARGET TensorRT::NvInfer PROPERTY IMPORTED_LOCATION + "${TensorRT_LIBRARY}") + endif() +endif() + +if(NOT TARGET TensorRT::OnnxParser AND "OnnxParser" IN_LIST + TensorRT_FIND_COMPONENTS) + add_library(TensorRT::OnnxParser SHARED IMPORTED) + target_include_directories(TensorRT::OnnxParser SYSTEM + INTERFACE "${TensorRT_OnnxParser_INCLUDE_DIR}") + target_link_libraries(TensorRT::OnnxParser INTERFACE TensorRT::NvInfer) + if(WIN32) + set_property(TARGET TensorRT::OnnxParser + PROPERTY IMPORTED_LOCATION "${TensorRT_OnnxParser_DLL}") + set_property(TARGET TensorRT::OnnxParser + PROPERTY IMPORTED_IMPLIB "${TensorRT_OnnxParser_LIBRARY}") + else() + set_property(TARGET TensorRT::OnnxParser + PROPERTY IMPORTED_LOCATION "${TensorRT_OnnxParser_LIBRARY}") + endif() +endif() + +if(NOT TARGET TensorRT::Plugin AND "Plugin" IN_LIST TensorRT_FIND_COMPONENTS) + add_library(TensorRT::Plugin SHARED IMPORTED) + target_include_directories(TensorRT::Plugin SYSTEM + INTERFACE "${TensorRT_Plugin_INCLUDE_DIR}") + target_link_libraries(TensorRT::Plugin INTERFACE TensorRT::NvInfer) + if(WIN32) + set_property(TARGET TensorRT::Plugin PROPERTY IMPORTED_LOCATION + "${TensorRT_Plugin_DLL}") + set_property(TARGET TensorRT::Plugin PROPERTY IMPORTED_IMPLIB + "${TensorRT_Plugin_LIBRARY}") + else() + set_property(TARGET TensorRT::Plugin PROPERTY IMPORTED_LOCATION + "${TensorRT_Plugin_LIBRARY}") + endif() +endif() + +mark_as_advanced(TensorRT_INCLUDE_DIR TensorRT_LIBRARY TensorRT_LIBRARIES) diff --git a/cpp/include/tensorrt_llm/batch_manager/cacheTransceiver.h b/cpp/include/tensorrt_llm/batch_manager/cacheTransceiver.h index 0c7724c657fc..30fe338baa52 100644 --- a/cpp/include/tensorrt_llm/batch_manager/cacheTransceiver.h +++ b/cpp/include/tensorrt_llm/batch_manager/cacheTransceiver.h @@ -22,19 +22,16 @@ #include "tensorrt_llm/batch_manager/llmRequest.h" #include "tensorrt_llm/batch_manager/rnnCacheTransBuffer.h" #include "tensorrt_llm/batch_manager/rnnStateManager.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/cacheCommunicator.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" #include "tensorrt_llm/runtime/utils/pgUtils.h" #include <cstddef> -#include <fstream> #include <future> #include <memory> #include <mutex> #include <optional> #include <pybind11/pybind11.h> -#include <string> #include <torch/csrc/jit/python/pybind_utils.h> #include <torch/custom_class.h> #include <torch/python.h> @@ -250,17 +247,6 @@ class BaseCacheTransceiver [[nodiscard]] virtual bool checkGenTransferComplete() const = 0; virtual bool cancelRequest(std::shared_ptr<LlmRequest> llmRequest) = 0; - - /// Get the serialized DataTransceiverState (CacheState + CommState) for this transceiver. - [[nodiscard]] virtual std::vector<char> getSerializedDataTransceiverState() const - { - return {}; - } - - [[nodiscard]] virtual bool hasPoisonedTransferBuffer() const - { - return false; - } }; class CacheTransceiver : public BaseCacheTransceiver @@ -268,22 +254,24 @@ class CacheTransceiver : public BaseCacheTransceiver public: CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheManager, executor::kv_cache::CacheState::ModelConfig const& cacheStateModelCfg, runtime::WorldConfig const& worldConfig, - std::vector<SizeType32> const& attentionLayerNumPerPP, tensorrt_llm::DataType dataType, + std::vector<SizeType32> const& attentionLayerNumPerPP, nvinfer1::DataType dataType, executor::kv_cache::CacheState::AttentionType attentionType = executor::kv_cache::CacheState::AttentionType::kDEFAULT, std::optional<executor::CacheTransceiverConfig> cacheTransceiverConfig = std::nullopt, + rnn_state_manager::RnnStateManager* rnnStateManager = nullptr, std::vector<SizeType32> const& rnnLayerNumPerPP = {}); CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheManager, std::vector<SizeType32> numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, runtime::WorldConfig const& worldConfig, - std::vector<SizeType32> const& attentionLayerNumPerPP, tensorrt_llm::DataType dataType, + std::vector<SizeType32> const& attentionLayerNumPerPP, nvinfer1::DataType dataType, executor::kv_cache::CacheState::AttentionType attentionType = executor::kv_cache::CacheState::AttentionType::kDEFAULT, std::optional<executor::CacheTransceiverConfig> cacheTransceiverConfig = std::nullopt, + rnn_state_manager::RnnStateManager* rnnStateManager = nullptr, std::vector<SizeType32> const& rnnLayerNumPerPP = {}) : CacheTransceiver(cacheManager, executor::kv_cache::CacheState::ModelConfig{numKvHeadsPerLayer, sizePerHead, tokensPerBlock}, worldConfig, - attentionLayerNumPerPP, dataType, attentionType, cacheTransceiverConfig, rnnLayerNumPerPP) + attentionLayerNumPerPP, dataType, attentionType, cacheTransceiverConfig, rnnStateManager, rnnLayerNumPerPP) { } @@ -306,19 +294,11 @@ class CacheTransceiver : public BaseCacheTransceiver virtual bool cancelRequest(std::shared_ptr<LlmRequest> llmRequest) override; - [[nodiscard]] std::vector<char> getSerializedDataTransceiverState() const override; - - [[nodiscard]] bool hasPoisonedTransferBuffer() const override; - private: void initializeCommState(); void setContextState(LlmRequest* llmRequest); - // Append one row per completed request to the gen-side transfer summary CSV. Opens the file - // lazily on first use; expects timing to already be synced across ranks by the caller. - void writeGenTransferSummary(std::vector<LlmRequest*> const& completedRequests); - std::unique_ptr<CacheSender> mCacheSender; std::unique_ptr<CacheReceiver> mCacheReceiver; // shared_ptr (not raw LlmRequest*) so the futures hold a strong reference for @@ -326,12 +306,9 @@ class CacheTransceiver : public BaseCacheTransceiver // request while a C++ status check still dereferences it. std::vector<std::pair<std::shared_ptr<LlmRequest>, std::future<void>>> mSenderFutures; std::vector<std::pair<std::shared_ptr<LlmRequest>, std::future<void>>> mRequesterFutures; - // Dedup timeout logs separately from accepted cancellation requests so a - // backend that initially declines cancellation is retried on later polls. + // Dedup sets so observe-only timeout WARN logs fire at most once per stuck request. std::unordered_set<LlmRequest::RequestIdType> mTimedOutSenderIds; std::unordered_set<LlmRequest::RequestIdType> mTimedOutRequesterIds; - std::unordered_set<LlmRequest::RequestIdType> mCancelRequestedSenderIds; - std::unordered_set<LlmRequest::RequestIdType> mCancelRequestedRequesterIds; std::unordered_set<LlmRequest::RequestIdType> mCompletedSenderRequestIds; std::unordered_set<LlmRequest::RequestIdType> mFailedSenderRequestIds; std::unordered_map<LlmRequest::RequestIdType, std::shared_ptr<LlmRequest>> mSenderRequestsAwaitingConsensus; @@ -351,16 +328,10 @@ class CacheTransceiver : public BaseCacheTransceiver std::vector<std::unique_ptr<kv_cache_manager::CacheTransBufferManager>> mCacheTransBufferManagers; std::vector<BaseTransBufferManager*> mCacheTransBufferManagerPtrs; + rnn_state_manager::RnnStateManager* mRnnStateManager{nullptr}; // TODO(shreyasm): update this to use same container as kv by using base trans buffers instead std::unique_ptr<rnn_state_manager::RnnCacheTransBufferManager> mRnnCacheTransBufferManager{nullptr}; - // Unique instance identifier for CSV file naming (avoids collisions across gen instances) - std::string mInstanceId; - - // Gen-side transfer summary CSV (written after timing sync) - std::ofstream mGenTransferSummaryFile; - std::mutex mGenTransferSummaryMutex; - // library handle to the communicator related features, // this is used to defer dependency resolution until needed. static std::mutex mDllMutex; diff --git a/cpp/include/tensorrt_llm/batch_manager/createNewDecoderRequests.h b/cpp/include/tensorrt_llm/batch_manager/createNewDecoderRequests.h index 600927af9645..bc619a34bc03 100644 --- a/cpp/include/tensorrt_llm/batch_manager/createNewDecoderRequests.h +++ b/cpp/include/tensorrt_llm/batch_manager/createNewDecoderRequests.h @@ -20,7 +20,6 @@ #include "tensorrt_llm/batch_manager/common.h" #include "tensorrt_llm/common/algorithm.h" #include "tensorrt_llm/common/optionalRef.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -67,17 +66,16 @@ class CreateNewDecoderRequests : Algorithm std::vector<executor::LookaheadDecodingConfig>> operator()(runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, executor::DecodingConfig const& decodingConfig, RequestVector const& contextRequests, - tensorrt_llm::DataType logitsType, DecoderInputBuffers& inputBuffers, - runtime::decoder::DecoderState& decoderState, CudaStream const& runtimeStream, CudaStream const& decoderStream, - SizeType32 maxSequenceLength, SizeType32 beamWidth, OptionalRef<MedusaBuffers const> medusaBuffers) const; + nvinfer1::DataType logitsType, DecoderInputBuffers& inputBuffers, runtime::decoder::DecoderState& decoderState, + CudaStream const& runtimeStream, CudaStream const& decoderStream, SizeType32 maxSequenceLength, + SizeType32 beamWidth, OptionalRef<MedusaBuffers const> medusaBuffers) const; [[nodiscard]] std::tuple<std::vector<SharedConstPtr>, std::vector<executor::LookaheadDecodingConfig>> createDecoderRequests(RequestVector const& finishedContextRequests, TensorPtr const& inputIds, executor::DecodingConfig const& decodingConfig, runtime::decoder::DecoderState& decoderState, - tensorrt_llm::DataType logitsType, runtime::ModelConfig const& modelConfig, - runtime::WorldConfig const& worldConfig, runtime::CudaStream const& runtimeStream, - runtime::CudaStream const& decoderStream, SizeType32 maxSequenceLength, - OptionalRef<MedusaBuffers const> medusaBuffers) const; + nvinfer1::DataType logitsType, runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + runtime::CudaStream const& runtimeStream, runtime::CudaStream const& decoderStream, + SizeType32 maxSequenceLength, OptionalRef<MedusaBuffers const> medusaBuffers) const; private: bool mSpeculativeDecodingFastLogits; diff --git a/cpp/include/tensorrt_llm/batch_manager/guidedDecoder.h b/cpp/include/tensorrt_llm/batch_manager/guidedDecoder.h new file mode 100644 index 000000000000..9a577b61ad51 --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/guidedDecoder.h @@ -0,0 +1,67 @@ +/* + * Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/iTensor.h" + +namespace xgrammar +{ +class GrammarMatcher; +class GrammarCompiler; +} // namespace xgrammar + +namespace tensorrt_llm::batch_manager +{ +class DecoderInputBuffers; + +class GuidedDecoder +{ +public: + using TensorPtr = runtime::ITensor::SharedPtr; + using SizeType32 = tensorrt_llm::runtime::SizeType32; + using BitmaskT = uint32_t; + + GuidedDecoder(executor::GuidedDecodingConfig const& guidedDecodingConfig, SizeType32 maxNumSequences, + SizeType32 vocabSizePadded, nvinfer1::DataType logitsDtype, runtime::BufferManager const& runtimeBufferManager); + void build(ScheduledRequests const& scheduledRequests); + void execute(DecoderInputBuffers const& decoderInputBuffers, runtime::BufferManager const& runtimeBufferManager); + +private: + executor::GuidedDecodingConfig::GuidedDecodingBackend mGuidedDecodingBackend; + std::vector<std::shared_ptr<xgrammar::GrammarMatcher>> mXGrammarMatchers; + std::shared_ptr<xgrammar::GrammarCompiler> mXGrammarCompiler; + + SizeType32 mMaxNumSequences; + SizeType32 mVocabSizePadded; + SizeType32 mBitmaskSize; // CeilDiv(vocabSizePadded, 32) + nvinfer1::DataType mLogitsDtype; + + TensorPtr mLogitsBitmask; // [mMaxNumRequests, mBitmaskSize] + TensorPtr mLogitsBitmaskHost; // [mMaxNumRequests, mBitmaskSize] + TensorPtr mLogitsBitmaskPtrVec; // [mMaxNumRequests], pointers to the logitsBitmask in a batch + TensorPtr mLogitsBitmaskPtrVecHost; // [mMaxNumRequests] + TensorPtr mLogitsPtrVec; // [mMaxNumRequests], pointers to the logits in a batch + TensorPtr mLogitsPtrVecHost; // [mMaxNumRequests] + + // BufferManager with a dedicated stream for async copy of buffers for guided decoding. + runtime::BufferManager mCopyBufferManager; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/batch_manager/handleContextLogits.h b/cpp/include/tensorrt_llm/batch_manager/handleContextLogits.h new file mode 100644 index 000000000000..cb77545578c8 --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/handleContextLogits.h @@ -0,0 +1,53 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/common/algorithm.h" +#include "tensorrt_llm/common/optionalRef.h" +#include "tensorrt_llm/runtime/modelConfig.h" + +namespace tensorrt_llm::runtime +{ +class BufferManager; +class CudaStream; +} // namespace tensorrt_llm::runtime + +namespace tensorrt_llm::batch_manager +{ + +class DecoderInputBuffers; +class MedusaBuffers; + +class HandleContextLogits : Algorithm +{ +public: + template <typename T> + using OptionalRef = tensorrt_llm::common::OptionalRef<T>; + + constexpr static auto name{"HandleContextLogits"}; + + HandleContextLogits() = default; + + runtime::SizeType32 operator()(DecoderInputBuffers& inputBuffers, RequestVector const& contextRequests, + runtime::ITensor::SharedPtr const& logits, std::vector<runtime::SizeType32> const& numContextLogitsVec, + runtime::ModelConfig const& modelConfig, runtime::BufferManager const& manager, + OptionalRef<MedusaBuffers> medusaBuffers) const; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/batch_manager/handleGenerationLogits.h b/cpp/include/tensorrt_llm/batch_manager/handleGenerationLogits.h new file mode 100644 index 000000000000..f9fd58800a6f --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/handleGenerationLogits.h @@ -0,0 +1,53 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "common.h" +#include "tensorrt_llm/common/algorithm.h" +#include "tensorrt_llm/common/optionalRef.h" +#include "tensorrt_llm/runtime/modelConfig.h" + +namespace tensorrt_llm::runtime +{ +class BufferManager; +} // namespace tensorrt_llm::runtime + +namespace tensorrt_llm::batch_manager +{ + +class DecoderInputBuffers; +class RuntimeBuffers; +class MedusaBuffers; + +class HandleGenerationLogits : Algorithm +{ +public: + template <typename T> + using OptionalRef = tensorrt_llm::common::OptionalRef<T>; + + constexpr static auto name{"HandleGenerationLogits"}; + + HandleGenerationLogits() = default; + + void operator()(DecoderInputBuffers& inputBuffers, RequestVector const& generationRequests, + runtime::ITensor::SharedPtr const& logits, runtime::SizeType32 logitsIndex, + runtime::ModelConfig const& modelConfig, runtime::BufferManager const& manager, + OptionalRef<RuntimeBuffers> genRuntimeBuffers, OptionalRef<MedusaBuffers> medusaBuffers) const; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/batch_manager/kvCacheManager.h b/cpp/include/tensorrt_llm/batch_manager/kvCacheManager.h index 04b6c230e75a..299bf1b570f4 100644 --- a/cpp/include/tensorrt_llm/batch_manager/kvCacheManager.h +++ b/cpp/include/tensorrt_llm/batch_manager/kvCacheManager.h @@ -23,7 +23,6 @@ #include "tensorrt_llm/batch_manager/llmRequest.h" // TODO forward declare #include "tensorrt_llm/batch_manager/radixBlockTree.h" #include "tensorrt_llm/common/optionalRef.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/executor/transferAgent.h" #include "tensorrt_llm/kernels/kvCacheIndex.h" @@ -33,6 +32,7 @@ #include "tensorrt_llm/runtime/iBuffer.h" #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/worldConfig.h" +#include <NvInferRuntime.h> #include <algorithm> #include <array> @@ -140,7 +140,7 @@ struct PoolConfiguration { SizeType32 windowSize; SizeType32 sizePerHead; - tensorrt_llm::DataType dtype; + nvinfer1::DataType dtype; }; struct LinearAttentionMetadata @@ -872,7 +872,7 @@ class WindowBlockManager using BlockMap = std::unordered_multimap<size_t, BlockPtr>; using BlockMapIterRange = std::pair<BlockMap::const_iterator, BlockMap::const_iterator>; - explicit WindowBlockManager(tensorrt_llm::DataType dtype, SizeType32 windowSize, + explicit WindowBlockManager(nvinfer1::DataType dtype, SizeType32 windowSize, std::vector<SizeType32> const& managedLayers, std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, bool isSWA, SizeType32 blocksInPrimaryPool, SizeType32 blocksInSecondaryPool, SizeType32 maxNumSequences, std::shared_ptr<runtime::CudaStream> stream, @@ -1037,7 +1037,7 @@ class WindowBlockManager //! host pools with mixed precisions when constructed with a per-window //! dtype map. Empty pools or NVFP4-scale pools are routed through the //! per-pool tensor metadata instead. - [[nodiscard]] tensorrt_llm::DataType getDataType() const noexcept + [[nodiscard]] nvinfer1::DataType getDataType() const noexcept { return mDataType; } @@ -1127,7 +1127,7 @@ class WindowBlockManager [[nodiscard]] SizeType32 getNumEltsPerContainer() const { #ifdef ENABLE_FP4 - return mDataType == tensorrt_llm::DataType::kFP4 ? 2 : 1; + return mDataType == nvinfer1::DataType::kFP4 ? 2 : 1; #else return 1; #endif @@ -1192,7 +1192,7 @@ class WindowBlockManager return mLayerToIndexWithinPool.at(layerIdx); } - void setOffsets(kernels::KVCacheIndex* offsetsPtr, tensorrt_llm::Dims const& offsetsShape, SizeType32 beamIdx, + void setOffsets(kernels::KVCacheIndex* offsetsPtr, nvinfer1::Dims const& offsetsShape, SizeType32 beamIdx, SizeType32 blockIdx, KVCacheBlock::IdType blockId) const; //! \brief Bring offloaded block from secondary to primary memory. @@ -1251,29 +1251,14 @@ class WindowBlockManager return mEnablePartialReuse; } - //! \brief Look up the block chain matching blockKey in the reuse tree. [[nodiscard]] std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKey(BlockKey const& blockKey); - //! \brief Same lookup, additionally pinning matched blocks; on a miss all pins are - //! rolled back and pinnedBlockIds is cleared. - [[nodiscard]] std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKey( - BlockKey const& blockKey, std::vector<KVCacheBlock::IdType>& pinnedBlockIds); - [[nodiscard]] std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKeys( std::vector<BlockKey> const& blockKeys); //! \brief Unpin blocks by block ids directly void unpinBlocksById(std::vector<KVCacheBlock::IdType> const& blockIds); - //! \brief Pin a block: claim it from the eviction policy if free, then take a reference. - //! Safe to call from cache-transceiver threads: block bookkeeping is serialized by the - //! lookup-tree mutex, which every mutating entry point acquires. - void pinBlock(BlockPtr const& block); - - //! \brief Inverse of pinBlock: drop one reference and release the block back to the - //! eviction policy once no references remain. - void unpinBlock(BlockPtr const& block); - void truncateBlocks(LlmRequest::VecTokens const& targetTokens, SizeType32 numTokensToKeep); void resetReuseState() @@ -1289,10 +1274,6 @@ class WindowBlockManager } private: - //! \brief Shared implementation of the findBlocksInReuseTreeByBlockKey overloads. - [[nodiscard]] std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKeyImpl( - BlockKey const& blockKey, bool pinBlocks, std::vector<KVCacheBlock::IdType>& pinnedBlockIds); - //! \brief Walk the reuse tree with precomputed per-block keys (no lock; callers must hold mLookupTree->getMutex()). [[nodiscard]] std::shared_ptr<KVCacheBlock> searchReuseTree(std::vector<BlockKey> const& blockKeys); @@ -1372,7 +1353,7 @@ class WindowBlockManager } private: - tensorrt_llm::DataType mDataType; + nvinfer1::DataType mDataType; SizeType32 mWindowSize; // Number of blocks in pools @@ -1500,7 +1481,7 @@ class BlockManager explicit BlockManager(std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, CudaStreamPtr stream, SizeType32 maxSequenceLength, SizeType32 maxBeamWidth, - std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, SizeType32 sinkBubbleLength, + std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkBubbleLength, SizeType32 chunkSize, CacheType cacheType = CacheType::kSELF, std::optional<executor::RetentionPriority> secondaryOffloadMinPriority = std::nullopt, std::shared_ptr<KVCacheEventManager> eventManager = nullptr, bool enablePartialReuse = true, @@ -1582,7 +1563,7 @@ class BlockManager void releaseLastBlock(GenerationRequest& sequence, SizeType32 windowSize); - void setOffsets(kernels::KVCacheIndex* offsetsPtr, tensorrt_llm::Dims const& offsetsShape, SizeType32 beamIdx, + void setOffsets(kernels::KVCacheIndex* offsetsPtr, nvinfer1::Dims const& offsetsShape, SizeType32 beamIdx, SizeType32 blockIdx, KVCacheBlock::IdType blockId, SizeType32 windowSize) const; //! \brief Combined prefix reuse analysis — single radix tree walk. @@ -1640,9 +1621,9 @@ class BlockManager //! \brief Convenience: window_size -> dataType, derived from getPoolConfigurations(). //! For one-pool-per-window managers only; multi-pool-per-window will collide. - [[nodiscard]] std::map<SizeType32, tensorrt_llm::DataType> getDataTypePerWindow() const + [[nodiscard]] std::map<SizeType32, nvinfer1::DataType> getDataTypePerWindow() const { - std::map<SizeType32, tensorrt_llm::DataType> result; + std::map<SizeType32, nvinfer1::DataType> result; for (auto const& [windowSize, manager] : mWindowBlockManagers) { result[windowSize] = manager.getDataType(); @@ -1655,7 +1636,7 @@ class BlockManager return mWindowBlockManagers.at(windowSize).getSizePerHead(); } - [[nodiscard]] tensorrt_llm::DataType getDataTypeForWindow(SizeType32 windowSize) const + [[nodiscard]] nvinfer1::DataType getDataTypeForWindow(SizeType32 windowSize) const { return mWindowBlockManagers.at(windowSize).getDataType(); } @@ -1855,12 +1836,6 @@ class BlockManager return mWindowBlockManagers.at(windowSize).findBlocksInReuseTreeByBlockKey(blockKey); } - [[nodiscard]] std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKey( - BlockKey const& blockKey, SizeType32 windowSize, std::vector<KVCacheBlock::IdType>& pinnedBlockIds) - { - return mWindowBlockManagers.at(windowSize).findBlocksInReuseTreeByBlockKey(blockKey, pinnedBlockIds); - } - [[nodiscard]] std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKeys( std::vector<BlockKey> const& blockKeys, SizeType32 windowSize) { @@ -2214,7 +2189,7 @@ class BaseKVCacheManager /// head_dim=512). Empty vector = uniform @p sizePerHead / @p dtype across all windows. /// @return Map from window size to tuple of (primary blocks, secondary blocks) [[nodiscard]] static BlocksPerWindow calculateMaxNumBlocks(executor::KvCacheConfig const& config, - tensorrt_llm::DataType dtype, std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, + nvinfer1::DataType dtype, std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, tensorrt_llm::runtime::WorldConfig const& worldConfig, std::map<SizeType32, std::vector<SizeType32>> const& windowSizeToLayers, uint64_t allottedPrimaryMemBytes, uint64_t allottedSecondaryMemBytes, size_t extraCostMemory, SizeType32 kvFactor, SizeType32 maxBatchSize, @@ -2235,11 +2210,6 @@ class BaseKVCacheManager BlockKey const& blockKey, SizeType32 windowSize) = 0; - //! \brief Pinning lookup: pins matched blocks and records their ids for unpinBlocksById. - [[nodiscard]] virtual std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKey( - BlockKey const& blockKey, SizeType32 windowSize, std::vector<KVCacheBlock::IdType>& pinnedBlockIds) - = 0; - [[nodiscard]] virtual std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKeys( std::vector<BlockKey> const& blockKeys, SizeType32 windowSize) = 0; @@ -2306,7 +2276,7 @@ class KVCacheManager : public BaseKVCacheManager //! and disagg transfer machinery applies natively. Empty vector = uniform. KVCacheManager(std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, SizeType32 maxBeamWidth, - std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, SizeType32 sinkTokenLength, + std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkTokenLength, CudaStreamPtr stream, SizeType32 maxSequenceLength, SizeType32 chunkSize, bool enableBlockReuse = false, CacheType cacheType = CacheType::kSELF, std::optional<executor::RetentionPriority> secondaryOffloadMinPriority = std::nullopt, @@ -2320,7 +2290,7 @@ class KVCacheManager : public BaseKVCacheManager KVCacheManager(std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, SizeType32 maxBeamWidth, - std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, SizeType32 sinkTokenLength, + std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkTokenLength, int64_t stream, SizeType32 maxSequenceLength, SizeType32 chunkSize, bool enableBlockReuse = false, CacheType cacheType = CacheType::kSELF, std::optional<executor::RetentionPriority> secondaryOffloadMinPriority = std::nullopt, @@ -2334,7 +2304,7 @@ class KVCacheManager : public BaseKVCacheManager KVCacheManager(SizeType32 numLayers, SizeType32 numKvHeads, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, SizeType32 maxBeamWidth, - std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, SizeType32 sinkTokenLength, + std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkTokenLength, CudaStreamPtr stream, SizeType32 maxSequenceLength, SizeType32 chunkSize, bool enableBlockReuse = true, CacheType cacheType = CacheType::kSELF, std::optional<executor::RetentionPriority> secondaryOffloadMinPriority = std::nullopt, @@ -2348,7 +2318,7 @@ class KVCacheManager : public BaseKVCacheManager KVCacheManager(SizeType32 numLayers, SizeType32 numKvHeads, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, SizeType32 maxBeamWidth, - std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, SizeType32 sinkTokenLength, + std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkTokenLength, int64_t stream, SizeType32 maxSequenceLength, SizeType32 chunkSize, bool enableBlockReuse = false, CacheType cacheType = CacheType::kSELF, bool enablePartialReuse = true, bool copyOnpartialReuse = true, bool enableIndexerKCache = false, SizeType32 indexerKCacheQuantBlockSize = 128, @@ -2608,24 +2578,6 @@ class KVCacheManager : public BaseKVCacheManager [[nodiscard]] std::vector<executor::IdType> commitAndGetBlockHashesForRequest( LlmRequest const& llmRequest, SizeType32 windowSize) override; - //! @brief Translate logical block IDs into primary-pool block indices. - //! @details A block ID is stable for the lifetime of a block, but its position inside the - //! memory pool can change after offload/onboard cycles. This function performs - //! that translation. The returned index is the value of - //! `KVCacheBlock::getMemoryPoolBlockIndex()` for each input, with the pool flag - //! stripped (see `kernels::KVCacheIndex::get()`), so it is only meaningful for - //! blocks resident in the primary pool. Every referenced block must therefore be - //! primary; this is asserted. Callers (e.g. the disaggregation cache transceiver - //! on the Python side) cannot check residency themselves, and the invariant holds - //! because allocation onboards offloaded blocks and offload only ever selects free - //! blocks — a violation indicates a block-lifetime bug. - //! @param blockIds IDs to translate. - //! @param windowSize Attention window the IDs belong to (selects the WindowBlockManager). - //! @throws Aborts via TLLM_CHECK_WITH_INFO if any referenced block is not found or is not - //! currently in the primary pool. - [[nodiscard]] std::vector<kernels::KVCacheIndex::UnderlyingType> getMemoryPoolBlockIndicesByBlockIds( - std::vector<KVCacheBlock::IdType> const& blockIds, SizeType32 windowSize) const; - std::optional<KVCacheBlock::IdType> getLastBlockId(LlmRequest::RequestIdType requestId) const override; /// @brief Calculates the number of kv-cache blocks that a sequence will require, for a single beam. @@ -2680,12 +2632,6 @@ class KVCacheManager : public BaseKVCacheManager return mBlockManager.findBlocksInReuseTreeByBlockKey(blockKey, windowSize); } - std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKey( - BlockKey const& blockKey, SizeType32 windowSize, std::vector<KVCacheBlock::IdType>& pinnedBlockIds) override - { - return mBlockManager.findBlocksInReuseTreeByBlockKey(blockKey, windowSize, pinnedBlockIds); - } - std::shared_ptr<KVCacheBlock> findBlocksInReuseTreeByBlockKeys( std::vector<BlockKey> const& blockKeys, SizeType32 windowSize) override { @@ -2718,7 +2664,7 @@ class KVCacheManager : public BaseKVCacheManager SizeType32 mMaxNumSequences; // Maximum beam width SizeType32 mMaxBeamWidth; - tensorrt_llm::DataType mDataType; + nvinfer1::DataType mDataType; // Maximum kv cache length per sequence SizeType32 mMaxAttentionWindow; // Number of tokens per block diff --git a/cpp/include/tensorrt_llm/batch_manager/llmRequest.h b/cpp/include/tensorrt_llm/batch_manager/llmRequest.h index dea7bda60274..bc1ca3e6d012 100644 --- a/cpp/include/tensorrt_llm/batch_manager/llmRequest.h +++ b/cpp/include/tensorrt_llm/batch_manager/llmRequest.h @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iBuffer.h" @@ -84,15 +83,6 @@ enum LlmRequestType class ContextProgress; -// Process-global offset between the local steady clock and the global steady -// clock (rank 0's steady clock). The storage lives in a single translation unit -// (llmRequest.cpp) and is reached through this accessor so that -// libtensorrt_llm.so and the nanobind extension module share one copy across -// .so boundaries. An inline-static member would instead give each shared object -// its own copy, so an offset calibrated on one side would be invisible to the -// other. -std::optional<std::chrono::steady_clock::duration>& globalSteadyClockOffset(); - template <typename TTensor, typename TStream = runtime::BufferManager::CudaStreamPtr> class GenericLlmRequest { @@ -1243,18 +1233,6 @@ class GenericLlmRequest mEstimatedReusableTokens = estimatedReusableTokens; } - //! Get the absolute context positions at which recurrent-state snapshots are expected. - [[nodiscard]] std::vector<SizeType32> const& getExpectedSnapshotPoints() const noexcept - { - return mExpectedSnapshotPoints; - } - - //! Set the absolute context positions at which recurrent-state snapshots are expected. - void setExpectedSnapshotPoints(std::vector<SizeType32> expectedSnapshotPoints) - { - mExpectedSnapshotPoints = std::move(expectedSnapshotPoints); - } - void setDraftTokens(std::shared_ptr<VecTokens> const& draftTokens) { mDraftTokens = draftTokens; @@ -1334,7 +1312,7 @@ class GenericLlmRequest mEncoderOutput = std::move(encoderOutput); } - void allocEncoderOutputHost(SizeType32 encoderHiddenSize, tensorrt_llm::DataType dataType) + void allocEncoderOutputHost(SizeType32 encoderHiddenSize, nvinfer1::DataType dataType) { mEncoderOutputHost = runtime::BufferManager::pinned( runtime::ITensor::makeShape({getEncoderOutputLen(), encoderHiddenSize}), dataType); @@ -1350,13 +1328,13 @@ class GenericLlmRequest return mEncoderHiddenStates; } - void allocEncoderOutput(runtime::BufferManager const& manager, tensorrt_llm::DataType dataType) + void allocEncoderOutput(runtime::BufferManager const& manager, nvinfer1::DataType dataType) { // unique_ptr --> shared_ptr ownership move mEncoderOutput = std::move(manager.emptyTensor(runtime::MemoryType::kGPU, dataType)); } - void allocEncoderHiddenStates(runtime::BufferManager const& manager, tensorrt_llm::DataType dataType) + void allocEncoderHiddenStates(runtime::BufferManager const& manager, nvinfer1::DataType dataType) { // unique_ptr --> shared_ptr ownership move mEncoderHiddenStates = std::move(manager.emptyTensor(runtime::MemoryType::kGPU, dataType)); @@ -1474,7 +1452,7 @@ class GenericLlmRequest mContextLogitsHost = std::move(contextLogitsHost); } - void allocContextLogitsHost(SizeType32 vocabSizePadded, tensorrt_llm::DataType logitsDataType) + void allocContextLogitsHost(SizeType32 vocabSizePadded, nvinfer1::DataType logitsDataType) { mContextLogitsHost = runtime::BufferManager::pinnedPool( runtime::ITensor::makeShape({mPromptLen, vocabSizePadded}), logitsDataType); @@ -1493,7 +1471,7 @@ class GenericLlmRequest mGenerationLogitsHost = std::move(generationLogitsHost); } - void allocGenerationLogitsHost(SizeType32 vocabSizePadded, tensorrt_llm::DataType logitsDataType) + void allocGenerationLogitsHost(SizeType32 vocabSizePadded, nvinfer1::DataType logitsDataType) { if (mIsStreaming) { @@ -1512,7 +1490,7 @@ class GenericLlmRequest } } - void allocTargetModelAcceptedTokenLogitsHost(SizeType32 vocabSizePadded, tensorrt_llm::DataType logitsDataType) + void allocTargetModelAcceptedTokenLogitsHost(SizeType32 vocabSizePadded, nvinfer1::DataType logitsDataType) { mGenerationLogitsHost = runtime::BufferManager::pinnedPool( runtime::ITensor::makeShape({1, getNumDraftTokens() + 1, vocabSizePadded}), logitsDataType); @@ -1866,18 +1844,14 @@ class GenericLlmRequest mDecodingIter = iter; } - // Callers must pass a global-steady-clock time point (getSteadyClockNow(), - // or a value merged from such time points). Normalizing again here would - // apply the global steady clock offset twice, which corrupts cross-node - // min/max merging whenever the offset is non-zero. void setKvCacheTransferStart(TimePoint time) const { - mPerfMetrics.timingMetrics.kvCacheTransferStart = time; + mPerfMetrics.timingMetrics.kvCacheTransferStart = maybeToGlobalSteadyClock(time); } void setKvCacheTransferEnd(TimePoint time) const { - mPerfMetrics.timingMetrics.kvCacheTransferEnd = time; + mPerfMetrics.timingMetrics.kvCacheTransferEnd = maybeToGlobalSteadyClock(time); } TimePoint getKvCacheTransferStart() const @@ -2053,8 +2027,8 @@ class GenericLlmRequest return mUseDraftModel; } - // If the global steady clock offset is set, return a global steady clock time point, otherwise return local steady - // clock time point + // If sGlobalSteadyClockOffset is set, return a global steady clock time point, otherwise return local steady clock + // time point [[nodiscard]] static TimePoint getSteadyClockNow() { return maybeToGlobalSteadyClock(std::chrono::steady_clock::now()); @@ -2084,6 +2058,9 @@ class GenericLlmRequest // current position of the prompt tuning table (only used in chunked prefill mode) SizeType32 mPtableCurrentPosition{0}; + // The offset between local steady clock and global steady clock (at rank 0) + inline static std::optional<Duration> sGlobalSteadyClockOffset{std::nullopt}; + protected: bool mIsStreaming; @@ -2119,9 +2096,6 @@ class GenericLlmRequest // the authoritative mPrepopulatedPromptLen and advances context position. mutable SizeType32 mEstimatedReusableTokens{0}; - // Absolute context positions at which recurrent-state snapshots are expected. - std::vector<SizeType32> mExpectedSnapshotPoints; - SizeType32 mMaxSentTokenLen; std::optional<TensorPtr> mEmbeddingBias{std::nullopt}; @@ -2382,7 +2356,7 @@ class GenericLlmRequest auto const numWords = static_cast<SizeType32>(words.size()); auto const shape = runtime::ITensor::makeShape({2, numWords}); - auto tensor = runtime::BufferManager::pinnedPool(shape, tensorrt_llm::DataType::kINT32); + auto tensor = runtime::BufferManager::pinnedPool(shape, nvinfer1::DataType::kINT32); auto* data = runtime::bufferCast<int32_t>(*tensor); std::memcpy(data, words.data(), numWords * sizeof(int32_t)); std::memcpy(data + numWords, offsets.data(), numWords * sizeof(int32_t)); @@ -2395,10 +2369,9 @@ class GenericLlmRequest static TimePoint maybeToGlobalSteadyClock(TimePoint const& time_point) { - auto const& offset = globalSteadyClockOffset(); - if (offset.has_value()) + if (sGlobalSteadyClockOffset.has_value()) { - return time_point + *offset; + return time_point + *sGlobalSteadyClockOffset; } return time_point; } diff --git a/cpp/include/tensorrt_llm/batch_manager/logitsPostProcessor.h b/cpp/include/tensorrt_llm/batch_manager/logitsPostProcessor.h new file mode 100644 index 000000000000..1916a915e337 --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/logitsPostProcessor.h @@ -0,0 +1,53 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "common.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/common/algorithm.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +namespace tensorrt_llm::runtime +{ +class CudaStream; +} + +namespace tensorrt_llm::batch_manager +{ +class DecoderInputBuffers; + +class LogitsPostProcessor : Algorithm +{ +public: + using CudaStreamPtr = std::shared_ptr<runtime::CudaStream>; + + using LogitsPostProcessorBatched = std::function<void(std::vector<batch_manager::LlmRequest::RequestIdType> const&, + std::vector<batch_manager::LlmRequest::TensorPtr>&, + std::vector<std::reference_wrapper<batch_manager::LlmRequest::BeamTokens const>> const&, CudaStreamPtr const&, + std::vector<std::optional<batch_manager::LlmRequest::RequestIdType>> const&)>; + + constexpr static auto name{"LogitsPostProcessor"}; + + LogitsPostProcessor() = default; + + bool operator()(DecoderInputBuffers& inputBuffers, bool replicateLogitsPostProcessor, + runtime::WorldConfig const& worldConfig, CudaStreamPtr const& stream, + std::optional<LogitsPostProcessorBatched> const& logitsPostProcessorBatched = std::nullopt) const; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.h b/cpp/include/tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.h new file mode 100644 index 000000000000..245f4b4b5286 --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.h @@ -0,0 +1,56 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "common.h" +#include "tensorrt_llm/common/algorithm.h" +#include "tensorrt_llm/common/optionalRef.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iGptDecoderBatched.h" +#include "tensorrt_llm/runtime/modelConfig.h" + +namespace tensorrt_llm::runtime::decoder +{ +class DecoderState; +} // namespace tensorrt_llm::runtime::decoder + +namespace tensorrt_llm::batch_manager +{ +class DecoderInputBuffers; +class RuntimeBuffers; + +class MakeDecodingBatchInputOutput : Algorithm +{ +public: + constexpr static auto name{"MakeDecodingBatchInputOutput"}; + + using SizeType32 = tensorrt_llm::runtime::SizeType32; + using TensorPtr = runtime::ITensor::SharedPtr; + template <typename T> + using OptionalRef = tensorrt_llm::common::OptionalRef<T>; + + MakeDecodingBatchInputOutput() = default; + + void operator()(DecoderInputBuffers& inputBuffers, runtime::decoder::DecoderState& decoderState, + runtime::ModelConfig const& modelConfig, OptionalRef<RuntimeBuffers> fusedRuntimeBuffers) const; + + static void createDecoderBatchInputs(DecoderInputBuffers& inputBuffers, std::vector<SizeType32> const& activeSlots, + runtime::decoder::DecoderState const& decoderState); +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/batch_manager/medusaBuffers.h b/cpp/include/tensorrt_llm/batch_manager/medusaBuffers.h index 5342591840a8..ba29be6ede81 100644 --- a/cpp/include/tensorrt_llm/batch_manager/medusaBuffers.h +++ b/cpp/include/tensorrt_llm/batch_manager/medusaBuffers.h @@ -22,6 +22,7 @@ #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/modelConfig.h" #include "tensorrt_llm/runtime/promptTuningParams.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" #include "tensorrt_llm/runtime/worldConfig.h" namespace tensorrt_llm::batch_manager @@ -35,6 +36,10 @@ class MedusaBuffers using TensorPtr = runtime::ITensor::SharedPtr; using TensorMap = runtime::StringPtrMap<runtime::ITensor>; + MedusaBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, runtime::BufferManager const& manager, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + executor::DecodingConfig const& decodingConfig, runtime::TllmRuntime const& runtime); + void reshape(SizeType32 numCtxSequences, SizeType32 numGenSequences, SizeType32 tokensPerStep); void insertInputTensors( diff --git a/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h b/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h index ed928e96d811..cf65753783e8 100644 --- a/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h +++ b/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h @@ -25,7 +25,7 @@ #include "tensorrt_llm/runtime/workerPool.h" #include "tensorrt_llm/runtime/worldConfig.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <future> #include <memory> @@ -147,7 +147,7 @@ class PeftCacheManager : public BasePeftCacheManager void updateTaskState(uint64_t taskId, uint64_t reqId, bool terminate = false, bool pause = false); static std::pair<uint64_t, uint64_t> getMaxNumSlots(PeftCacheManagerConfig const& config, - tensorrt_llm::DataType dataType, uint64_t pageWidth, uint64_t max1dModSize, + nvinfer1::DataType dataType, uint64_t pageWidth, uint64_t max1dModSize, runtime::BufferManager const& bufferManager); static std::pair<runtime::LoraCachePageManagerConfig, runtime::LoraCachePageManagerConfig> getPageManagerConfig( diff --git a/cpp/include/tensorrt_llm/batch_manager/promptTuningBuffers.h b/cpp/include/tensorrt_llm/batch_manager/promptTuningBuffers.h new file mode 100644 index 000000000000..a1d8849a8811 --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/promptTuningBuffers.h @@ -0,0 +1,106 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/promptTuningParams.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +namespace tensorrt_llm::batch_manager +{ + +class PromptTuningBuffers +{ + +public: + using SizeType32 = tensorrt_llm::runtime::SizeType32; + using ITensor = tensorrt_llm::runtime::ITensor; + using TensorPtr = runtime::ITensor::SharedPtr; + + runtime::PromptTuningParams mPromptTuningParams; + SizeType32 mMaxPromptVocabSize; + + PromptTuningBuffers(SizeType32 maxBatchSize, runtime::BufferManager const& manager, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig); + + PromptTuningBuffers(SizeType32 maxBatchSize, runtime::BufferManager const& manager, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, bool promptTableOffloading); + + void validate(std::optional<TensorPtr> const& optReqPromptEmbeddingTable, + std::optional<SizeType32> const& optReqPromptVocabSize); + + void fill(RequestVector const& contextRequests, RequestVector const& genRequests, + runtime::BufferManager const& manager, bool packed); + + /* + * The below functions are specific for Chunked Prefill mode + * Chunk Ptable with Ping-Pong Buffer Implementation + * ----------------------------------------------- + * + * Overview: + * The chunk ptable (prompt tuning table) system uses a ping-pong buffer mechanism to efficiently + * manage large embedding tables when operating in context Prefill mode. This allows + * for processing of large embedding tables by loading them in chunks from CPU to GPU memory, + * enabling support for tables that exceed available GPU memory. + * + * Key Components: + * 1. Ping-Pong Buffers (mChunkPtableBuffers): + * - Two alternating GPU buffers that store chunks of the embedding table + * - While the current buffer is being processed by the model, + * the next chunk can be asynchronously loaded into the other buffer + * - Managed through mChunkPtableCurrentIndex (toggles between 0 and 1) + * 2. Start Positions Tracking (mChunkPtableBufferStartPositions): + * - Mainly used for multi-batch processing + * - Maintains the starting position of each batch's data within each buffer + * - Maintained separately for each ping-pong buffer + * + * Memory Optimization: + * - Only two GPU buffers are maintained regardless of total embedding table size + * - Each buffer size is limited to contextChunkSize * hiddenSize + * - Efficient memory usage through chunk-based processing + */ + + bool mPromptTableOffloading; + + bool mChunkPtableInitialized{false}; + std::optional<std::array<TensorPtr, 2>> mChunkPtableBuffers; + std::optional<std::vector<std::vector<SizeType32>>> mChunkPtableBufferStartPositions; + size_t mChunkPtableCurrentIndex{0}; + + void initializeChunkPtableBuffers(runtime::BufferManager const& manager, runtime::ModelConfig const& modelConfig, + SizeType32 contextChunkSize, std::shared_ptr<LlmRequest> const& llmReq); + + void switchChunkPtableBuffer(); + + size_t getChunkPtableCurrentIndex(); + + [[nodiscard]] TensorPtr& getChunkPtableBuffer(size_t index); + + [[nodiscard]] SizeType32 getChunkPtableBufferSliceSize(size_t index, size_t batchIdx); + + [[nodiscard]] SizeType32 getChunkPtableBufferStartPosition(size_t index, size_t batchIdx); + + void updateBufferStartPosition(size_t index, SizeType32 numRows); + + void clearBufferStartPositions(size_t index); +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/batch_manager/rnnCacheFormatter.h b/cpp/include/tensorrt_llm/batch_manager/rnnCacheFormatter.h index b57d08f2c12b..1bc3fadf62f4 100644 --- a/cpp/include/tensorrt_llm/batch_manager/rnnCacheFormatter.h +++ b/cpp/include/tensorrt_llm/batch_manager/rnnCacheFormatter.h @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -34,11 +34,14 @@ class TransferSession; namespace rnn_state_manager { +class RnnStateManager; class RnnCacheTransBufferManager; } // namespace rnn_state_manager /// @brief RNN Cache Formatter for formatting/unformatting RNN states during transfer. -/// Uses unified pool mode via BaseKVCacheManager (for CppMambaHybridCacheManager, block-indexed pool). +/// Supports two operating modes: +/// - Slot mode: uses RnnStateManager (for CppMambaCacheManager, separate tensor storage) +/// - Unified pool mode: uses BaseKVCacheManager (for CppMambaHybridCacheManager, block-indexed pool) class RnnCacheFormatter : public kv_cache_manager::BaseCacheFormatter { public: @@ -46,6 +49,12 @@ class RnnCacheFormatter : public kv_cache_manager::BaseCacheFormatter using CacheState = executor::kv_cache::CacheState; using RequestIdType = tensorrt_llm::batch_manager::RequestIdType; + /// @brief Constructor for slot-based mode (CppMambaCacheManager with RnnStateManager). + /// @param rnnStateManager Pointer to the RNN state manager. + /// @param rnnCacheTransBufferManager Pointer to the RNN cache transfer buffer manager. + RnnCacheFormatter(rnn_state_manager::RnnStateManager* rnnStateManager, + rnn_state_manager::RnnCacheTransBufferManager* rnnCacheTransBufferManager); + /// @brief Constructor for unified pool mode (CppMambaHybridCacheManager). /// @param kvCacheManager Pointer to the KV cache manager with unified pool. /// @param rnnCacheTransBufferManager Pointer to the RNN cache transfer buffer manager. @@ -72,13 +81,39 @@ class RnnCacheFormatter : public kv_cache_manager::BaseCacheFormatter CacheState const& selfConfig, SizeType32 selfIdx, CacheState const& destConfig, std::vector<SizeType32> const& counterPartRanks) const override; - /// @brief Returns the KV cache manager. + /// @brief Returns the KV cache manager (non-null in unified pool mode). [[nodiscard]] kv_cache_manager::BaseKVCacheManager* getCacheManager() const noexcept override { return mKvCacheManager; } + /// @brief Get the RNN state manager (non-null in slot mode). + /// @return Pointer to the RNN state manager. + [[nodiscard]] rnn_state_manager::RnnStateManager* getRnnStateManager() const noexcept + { + return mRnnStateManager; + } + + /// @brief Check if operating in unified pool mode. + [[nodiscard]] bool isUnifiedPoolMode() const noexcept + { + return mKvCacheManager != nullptr; + } + private: + /// @brief Format logic for slot-based path (RnnStateManager). + void formatSlotMode(TransferSession& session); + + /// @brief Unformat logic for slot-based path (RnnStateManager). + void unformatSlotMode(TransferSession& session); + + /// @brief Format logic for unified pool path (BaseKVCacheManager). + void formatUnifiedPoolMode(TransferSession& session); + + /// @brief Unformat logic for unified pool path (BaseKVCacheManager). + void unformatUnifiedPoolMode(TransferSession& session); + + rnn_state_manager::RnnStateManager* mRnnStateManager{nullptr}; rnn_state_manager::RnnCacheTransBufferManager* mRnnCacheTransBufferManager; kv_cache_manager::BaseKVCacheManager* mKvCacheManager{nullptr}; }; diff --git a/cpp/include/tensorrt_llm/batch_manager/rnnStateManager.h b/cpp/include/tensorrt_llm/batch_manager/rnnStateManager.h index c4f97950a6b9..5c0bfe136de2 100644 --- a/cpp/include/tensorrt_llm/batch_manager/rnnStateManager.h +++ b/cpp/include/tensorrt_llm/batch_manager/rnnStateManager.h @@ -17,7 +17,6 @@ #pragma once #include "tensorrt_llm/batch_manager/common.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -43,8 +42,8 @@ class RnnStateManager runtime::WorldConfig const& worldConfig, tensorrt_llm::runtime::BufferManager const& bufferManager); RnnStateManager(SizeType32 dState, SizeType32 dConv, SizeType32 numHeads, SizeType32 nGroups, SizeType32 headDim, - SizeType32 maxBatchSize, runtime::WorldConfig const& worldConfig, int64_t stream, tensorrt_llm::DataType dtype, - tensorrt_llm::DataType ssmCacheDtype, std::vector<SizeType32> const& ppLayers, SizeType32 numLayers); + SizeType32 maxBatchSize, runtime::WorldConfig const& worldConfig, int64_t stream, nvinfer1::DataType dtype, + nvinfer1::DataType ssmCacheDtype, std::vector<SizeType32> const& ppLayers, SizeType32 numLayers); void getPtrBuffers(TensorMap& inputBuffers, runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig) const; @@ -69,9 +68,9 @@ class RnnStateManager [[nodiscard]] TensorPtr getSsmStates() const; - [[nodiscard]] tensorrt_llm::DataType getConvStateDataType() const noexcept; + [[nodiscard]] nvinfer1::DataType getConvStateDataType() const noexcept; - [[nodiscard]] tensorrt_llm::DataType getSsmStateDataType() const noexcept; + [[nodiscard]] nvinfer1::DataType getSsmStateDataType() const noexcept; [[nodiscard]] executor::kv_cache::CacheState::RnnModelConfig getRnnCacheStateModelConfig() const noexcept; @@ -112,8 +111,8 @@ class RnnStateManager std::vector<SizeType32> mFreeBlocks; std::unordered_map<RequestIdType, SizeType32> mCacheIndex; std::optional<runtime::BufferManager> mBufferManager; - tensorrt_llm::DataType mDtype{tensorrt_llm::DataType::kFLOAT}; - tensorrt_llm::DataType mSsmCacheDtype{tensorrt_llm::DataType::kFLOAT}; + nvinfer1::DataType mDtype{nvinfer1::DataType::kFLOAT}; + nvinfer1::DataType mSsmCacheDtype{nvinfer1::DataType::kFLOAT}; // RNN model config (global values before TP/PP split) SizeType32 mDState{0}; diff --git a/cpp/include/tensorrt_llm/batch_manager/runtimeBuffers.h b/cpp/include/tensorrt_llm/batch_manager/runtimeBuffers.h new file mode 100644 index 000000000000..97a4ae67acdd --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/runtimeBuffers.h @@ -0,0 +1,326 @@ +/* + * Copyright (c) 2023-2026, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/batch_manager/rnnStateManager.h" +#include "tensorrt_llm/common/optionalRef.h" +#include "tensorrt_llm/runtime/eagleBuffers.h" +#include "tensorrt_llm/runtime/explicitDraftTokensBuffers.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/lookaheadBuffers.h" +#include "tensorrt_llm/runtime/loraManager.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +#include <cstddef> +#include <memory> +#include <optional> +#include <vector> + +namespace tensorrt_llm::runtime +{ +class TllmRuntime; + +namespace decoder +{ +class DecoderState; +} // namespace decoder +} // namespace tensorrt_llm::runtime + +namespace tensorrt_llm::batch_manager +{ + +namespace kv_cache_manager +{ +class BaseKVCacheManager; +} // namespace kv_cache_manager + +class LlmRequest; + +class EncoderBuffers; +class LoraBuffers; +class MedusaBuffers; +class PromptTuningBuffers; +class RnnStateBuffers; +class TransformerBuffers; + +class RuntimeBuffers +{ +public: + static constexpr auto kLogitsTensorName = "logits"; + static constexpr auto kHiddenStatesOutputTensorName = "hidden_states_output"; + static constexpr auto kHiddenStatesInputTensorName = "hidden_states_input"; + static constexpr auto kInputIdsTensorName = "input_ids"; + static constexpr auto kLastTokenIdsTensorName = "last_token_ids"; + static constexpr auto kHostRequestTypesTensorName = "host_request_types"; + static constexpr auto kContextLengthsTensorName = "context_lengths"; + static constexpr auto kHostContextLengthsTensorName = "host_context_lengths"; + static constexpr auto kSequenceLengthsTensorName = "sequence_length"; + static constexpr auto kPromptEmbeddingTableTensorName = "prompt_embedding_table"; + static constexpr auto kTasksTensorName = "tasks"; + static constexpr auto kPromptVocabSizeTensorName = "prompt_vocab_size"; + static constexpr auto kMRopeRotaryCosSinTensorName = "mrope_rotary_cos_sin"; + static constexpr auto kMRopePositionDeltasTensorName = "mrope_position_deltas"; + + using SizeType32 = runtime::SizeType32; + using TensorPtr = runtime::ITensor::SharedPtr; + using TensorMap = runtime::ITensor::TensorMap; + using PeftTable = runtime::LoraManager::PeftTable; + template <typename T> + using OptionalRef = tensorrt_llm::common::OptionalRef<T>; + + [[nodiscard]] SizeType32 constexpr getContextIndex() const noexcept + { + return contextIndex; + }; + + void constexpr setContextIndex(SizeType32 index) noexcept + { + contextIndex = index; + }; + + [[nodiscard]] SizeType32 constexpr getNumContextTokens() const noexcept + { + return numContextTokens; + }; + + [[nodiscard]] BatchState getBatchState() const noexcept + { + return {numContextRequests, numGenRequests, getNumTokens(), maxKvCacheLengthRounded}; + }; + +private: + [[nodiscard]] SizeType32 constexpr getNumRequests() const noexcept + { + return numContextRequests + numGenRequests; + }; + + [[nodiscard]] SizeType32 constexpr getNumSequences() const noexcept + { + return numContextRequests + numGenSequences; + }; + + [[nodiscard]] SizeType32 constexpr getNumTokens() const noexcept + { + return numContextTokens + numGenTokens; + }; + + //! Sizes + SizeType32 numContextRequests{}; + SizeType32 numGenRequests{}; + SizeType32 numGenSequences{}; + SizeType32 numContextTokens{}; + SizeType32 numGenTokens{}; + SizeType32 numLogits{}; + SizeType32 maxKvCacheLengthRounded{}; + + //! General + TensorPtr inputsIds; + + TensorPtr contextLengthsHost; + TensorPtr contextLengthsDevice; + TensorPtr sequenceLengthsHost; + + //! Index of selected runtime context. + SizeType32 contextIndex{}; + SizeType32 maxContextLength{}; + +public: + TensorPtr sequenceLengthsDevice; + bool promptTableOffloading; + + //! Prompt-Tuning + std::unique_ptr<PromptTuningBuffers> promptTuningBuffers; + +private: + //! Runtime + //! Type of host tensor: 0 for context, 1 for generation + TensorPtr requestTypes; + + TensorPtr lastTokenIdsHost; + TensorPtr lastTokenIdsDevice; + TensorPtr logitsIdsHost; + + //! Pipeline-Parallelism + TensorPtr hiddenStates; + + //! Mrope + TensorPtr mropeRotaryCosSin; + TensorPtr mropePositionDeltas; + + //! LoRA + std::unique_ptr<LoraBuffers> loraBuffers; + +public: + //! Additional buffers depending on model type + std::unique_ptr<TransformerBuffers> transformerBuffers; + std::unique_ptr<RnnStateBuffers> rnnStateBuffers; + + //! Encoder-Decoder + std::unique_ptr<EncoderBuffers> encoderBuffers; + + //! Medusa + std::unique_ptr<MedusaBuffers> mMedusaBuffers; + //! Lookahead decoding + std::unique_ptr<runtime::LookaheadRuntimeBuffers> mLookaheadBuffers; + //! Explicit draft tokens decoding + std::unique_ptr<runtime::ExplicitDraftTokensBuffers> mExplicitDraftTokensBuffers; + //! Eagle decoding + std::unique_ptr<runtime::EagleBuffers> mEagleBuffers; + + //! Language adapter routing information if language adapter is presented, [numTokens, numLanguages] + TensorPtr languageAdapterRoutings; + + TensorPtr cacheIndirDecoderIOBatchedCopySrcOffsets; + TensorPtr cacheIndirDecoderIOBatchedCopyDstOffsets; + TensorPtr cacheIndirDecoderIOBatchedCopySizes; + + //! Logits + std::vector<SizeType32> numContextLogits; + TensorPtr logits; + + //! Helper cache for store generation logits + struct GenerationLogitsCache + { + static constexpr auto kCACHE_LENGTH = 8; + + //! Buffer for logits between steps to prevent from being overwritten + //! [kCACHE_LENGTH, maxBatchSize * maxBeamWidth, vocabSizePadded] + TensorPtr logits; + //! Record the usage offset of the cacheGenerationLogits buffer + SizeType32 offset{0}; + + //! Temporarily store the transposed results of multiple fragment logits, [maxBeamWidth, kCACHE_LENGTH] + TensorPtr transposedLogits; + + //! Temporarily store logits buffer address during the transposing, [maxBatchSize, kCACHE_LENGTH] + //! One row per batch slot (same layout as fragmentPointerHost) so concurrent flushes for + //! different requests in the same batch never clobber each other's pointer arrays. + TensorPtr fragmentPointerDevice; + + //! Temporarily store logits buffer address during the transposing, [maxBatchSize, kCACHE_LENGTH] + TensorPtr fragmentPointerHost; + + //! Cycling index for workspace + size_t workIdx{0}; + + void cycleWorkIdx() + { + workIdx = (workIdx + 1) % (fragmentPointerHost->getShape().d[0]); + } + + //! Returns matching host and device pointer rows for the current workIdx, then advances + //! workIdx. Always call this instead of the individual getters to avoid ordering bugs. + [[nodiscard]] std::pair<TensorPtr, TensorPtr> getFragmentPointerSlot() + { + TensorPtr host = runtime::ITensor::slice(fragmentPointerHost, workIdx, 1); + TensorPtr device = runtime::ITensor::slice(fragmentPointerDevice, workIdx, 1); + cycleWorkIdx(); + return {std::move(host), std::move(device)}; + }; + }; + + GenerationLogitsCache generationLogitsCache; + + //! Mapping from batch idx to slot id + TensorPtr seqSlots; + TensorPtr seqSlotsDevice; + + //! Explicitly device-copy src offsets to reduce warp stalls in copy batch kernel invocation + //! [mMaxNumRequests], on gpu + TensorPtr mCacheIndirDecoderIOBatchedCopySrcOffsetsSliceDevice; + //! Explicitly device-copy dst offsets to reduce warp stalls in copy batch kernel invocation + //! [mMaxNumRequests], on gpu + TensorPtr mCacheIndirDecoderIOBatchedCopyDstOffsetsSliceDevice; + //! Explicitly device-copy size to reduce warp stalls in copy batch kernel invocation + //! [mMaxNumRequests], on gpu + TensorPtr mCacheIndirDecoderIOBatchedCopyCopySizesDevice; + +private: + //! Re-capture cuda graph when max kv cache len of the batch has changed on kKV_CACHE_LEN_CUDA_GRAPH_ROUND_SIZE. + static SizeType32 constexpr kKV_CACHE_LEN_CUDA_GRAPH_ROUND_SIZE{256}; + + TensorMap mAdditionalOutputTensors; // Tensors storing additional output tensors. + + //! Engine I/O + TensorMap inputMap; + TensorMap outputMap; + +public: + RuntimeBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, + std::vector<SizeType32> const& maxAttentionWindowVec, SizeType32 maxAttentionWindow, SizeType32 sinkTokenLen, + runtime::TllmRuntime const& runtime, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig, executor::DecodingConfig const& decodingConfig, + bool gatherGenerationLogits, std::optional<SizeType32> maxNumTokens = std::nullopt, + std::optional<std::vector<executor::AdditionalModelOutput>> const& additionalModelOutputs = std::nullopt, + bool promptTableOffloading = false); + + RuntimeBuffers(RuntimeBuffers const& other) = delete; + RuntimeBuffers& operator=(RuntimeBuffers const& other) = delete; + RuntimeBuffers(RuntimeBuffers&& other) = delete; + RuntimeBuffers& operator=(RuntimeBuffers&& other) = delete; + + ~RuntimeBuffers(); + + std::tuple<SizeType32, TensorMap const&, TensorMap&> prepareStep(RequestVector const& contextRequests, + RequestVector const& genRequests, SizeType32 maxBeamWidth, SizeType32 maxAttentionWindow, + runtime::decoder::DecoderState const& decoderState, kv_cache_manager::BaseKVCacheManager* kvCacheManager, + kv_cache_manager::BaseKVCacheManager* crossKvCacheManager, rnn_state_manager::RnnStateManager* rnnStateManager, + PeftTable const& peftTable, runtime::TllmRuntime const& runtime, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig, bool gatherGenerationLogits, bool trtOverlap, + OptionalRef<runtime::ITensor const> newOutputTokens = std::nullopt); + + void prepareBuffersForCudaGraph(SizeType32 maxSequenceLength); + + void prepareExplicitDraftTokenBuffers(runtime::ExplicitDraftTokensBuffers::Inputs const& explicitDraftTokensBuffers, + runtime::TllmRuntime const& runtime, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig); + + void prepareEagleBuffers(RequestVector const& contextRequests, RequestVector const& genRequests, + runtime::EagleBuffers::Inputs const& eagleBuffers, runtime::TllmRuntime const& runtime, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig); + +private: + void create(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, + SizeType32 maxAttentionWindow, SizeType32 sinkTokenLen, runtime::TllmRuntime const& runtime, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + executor::DecodingConfig const& decodingConfig, bool gatherGenerationLogits, + std::optional<std::vector<executor::AdditionalModelOutput>> const& additionalModelOutputs = std::nullopt); + + //! @brief set max sizes for pre-allocation + void setMaxBufferSizes(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, runtime::ModelConfig const& modelConfig, + std::optional<SizeType32> maxNumRuntimeTokens); + + //! @brief set sizes depending on scheduled requests + void setBufferSizes(RequestVector const& contextRequests, RequestVector const& genRequests); + + void reshape(runtime::TllmRuntime const& runtime, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig, bool gatherGenerationLogits); + + void setFromInputs(RequestVector const& contextRequests, RequestVector const& genRequests, SizeType32 maxBeamWidth, + SizeType32 maxAttentionWindow, runtime::decoder::DecoderState const& decoderState, + kv_cache_manager::BaseKVCacheManager* kvCacheManagerPtr, + kv_cache_manager::BaseKVCacheManager* crossKvCacheManagerPtr, + rnn_state_manager::RnnStateManager* rnnStateManagerPtr, PeftTable const& peftTable, + runtime::TllmRuntime const& runtime, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig, bool trtOverlap, OptionalRef<runtime::ITensor const> newOutputTokens); + + void fillIOMaps(runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig); +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/batch_manager/transformerBuffers.h b/cpp/include/tensorrt_llm/batch_manager/transformerBuffers.h new file mode 100644 index 000000000000..b5254c6357b4 --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/transformerBuffers.h @@ -0,0 +1,145 @@ +/* + * Copyright (c) 2023-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/batch_manager/kvCacheType.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +namespace tensorrt_llm::runtime +{ +class TllmRuntime; +class MulticastTensor; +} // namespace tensorrt_llm::runtime + +namespace tensorrt_llm::batch_manager +{ + +namespace kv_cache_manager +{ +class BaseKVCacheManager; +} + +class TransformerBuffers +{ +public: + using SizeType32 = runtime::SizeType32; + using TensorPtr = runtime::ITensor::SharedPtr; + using TensorMap = runtime::StringPtrMap<runtime::ITensor>; + + static constexpr auto kCrossAttentionMaskTensorName = "cross_attention_mask"; + static constexpr auto kCrossAttentionPackedMaskTensorName = "cross_attention_packed_mask"; + static constexpr auto kPositionIdsTensorName = "position_ids"; + static constexpr auto kCacheIndirectionsTensorName = "cache_indirection"; + static constexpr auto kHostPastKeyValueLengthsTensorName = "host_past_key_value_lengths"; + static constexpr auto kHostSinkTokenLengthTensorName = "host_sink_token_length"; + static constexpr auto kHostMaxAttentionWindowSizesTensorName = "host_max_attention_window_sizes"; + static constexpr auto kHostContextProgressTensorName = "host_context_progress"; + static constexpr auto kKvCacheBlockOffsetsTensorName = "kv_cache_block_offsets"; + static constexpr auto kHostKvCacheBlockOffsetsTensorName = "host_kv_cache_block_offsets"; + static constexpr auto kCrossKvCacheBlockOffsetsTensorName = "cross_kv_cache_block_offsets"; + static constexpr auto kHostCrossKvCacheBlockOffsetsTensorName = "host_cross_kv_cache_block_offsets"; + static constexpr auto kHostCrossKvCachePoolPointersTensorName = "host_cross_kv_cache_pool_pointers"; + static constexpr auto kHostCrossKvCachePoolMappingTensorName = "host_cross_kv_cache_pool_mapping"; + static constexpr auto kSkipCrossAttentionBlocksTensorName = "skip_cross_attn_blocks"; + + TensorPtr pastKeyValueLengths; // Host tensor + TensorPtr positionIds; + + // max kv cache lengths. + TensorPtr maxAttentionWindows; + // sink token lengths. + TensorPtr sinkTokenLengths; + TensorPtr cacheIndirection; + TensorPtr kvCacheBlockOffsetsHost; // [numPools, maxBatch * maxBeamWidth, 2, maxBlocksPerSeq] + TensorPtr kvCacheBlockOffsetsDevice; // [numPools, maxBatch * maxBeamWidth, 2, maxBlocksPerSeq] + TensorPtr contextProgressHost; + + // Cross attention buffers + TensorPtr crossKvCacheBlockPoolPointers = nullptr; + TensorPtr crossKvCacheBlockPoolMapping = nullptr; + TensorPtr crossKvCacheBlockOffsetsHost = nullptr; + TensorPtr crossKvCacheBlockOffsetsDevice = nullptr; + TensorPtr crossAttentionMaskCopySrcOffsets = nullptr; // [maxNumRequest] pinned memory. + TensorPtr crossAttentionMaskCopyDstOffsets = nullptr; // [maxNumRequest] pinned memory. + TensorPtr crossAttentionMaskCopySizes = nullptr; // [maxNumRequest] pinned memory. + TensorPtr crossAttentionMaskDevice = nullptr; // [maxNumTokens, maxEncoderOutputLen] + // This is created to allow mixed memory types of crossAttentionMask (i.e. CPU and GPU). + TensorPtr crossAttentionMaskPinnedHost = nullptr; // [maxNumTokens, maxEncoderOutputLen] + // See more details in tensorrt_llm/kernels/contextFusedMultiHeadAttention/fmhaPackedMask.cu. + // The attention packed mask for FMHA where each bit represents one mask. + TensorPtr crossAttentionPackedMaskDevice + = nullptr; // [maxBatchSize, maxInputLengthInBatch, roundUp(maxEncoderOutputLen, 32)] + // The number of cumulative Q sequence lengths in the mask input, which is used to get mask offsets for different + // requests. + TensorPtr crossAttentionCuQSeqLensDevice = nullptr; // [maxBatchSize + 1] + // The number of cumulative Q sequence lengths in the packed mask, which is used to get mask offsets for different + // requests. + TensorPtr crossAttentionPackedMaskCuMaskRowsDevice = nullptr; // [maxBatchSize + 1] + + TensorPtr cacheIndirBatchedCopySrcOffsets; + TensorPtr cacheIndirBatchedCopyDstOffsets; + TensorPtr cacheIndirBatchedCopySizes; + + TensorPtr fillValuesAlt; + TensorPtr fillValuesAltDevice; + TensorPtr seqSlotsAlt; + TensorPtr seqSlotsAltDevice; + TensorPtr skipCrossAttnBlocks; + + std::shared_ptr<tensorrt_llm::runtime::MulticastTensor> gemmAllReduceOutput; + + TransformerBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, + std::vector<SizeType32> const& maxAttentionWindowVec, SizeType32 maxAttentionWindow, SizeType32 sinkTokenLen, + runtime::TllmRuntime const& runtime, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig); + + void reshape(SizeType32 numSequences, SizeType32 numInputTokens); + + void reshapeKvTensors(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, SizeType32 maxBlocksPerSeq, + kv_cache_manager::CacheType kvCacheType, SizeType32 numPools, runtime::BufferManager const& manager); + + void getBuffers(TensorMap& inputBuffers, TensorMap& outputBuffers, runtime::ModelConfig const& modelConfig) const; + + void copyPositionIds(runtime::TllmRuntime const& runtime, std::vector<SizeType32> const& positionIdsHost, + bool isChatGlm, TensorPtr const& decoderPositionIds); + + void copyKvBlockOffsets(RequestVector const& contextRequests, RequestVector const& genRequests, + kv_cache_manager::BaseKVCacheManager const* kvCacheManager, + kv_cache_manager::BaseKVCacheManager const* crossKvCacheManager, runtime::BufferManager const& manager); + + // Copy CacheIndirection from `decoderCacheIndirectionOutput` to `this->cacheIndirection` + void copyCacheIndirection(RequestVector const& genRequests, TensorPtr const& decoderCacheIndirectionOutput, + runtime::CudaStream const& stream); + + void copyCrossAttentionMasks(RequestVector const& contextRequests, RequestVector const& genRequests, + TensorPtr const& decoderContextLengthsDevice, TensorPtr const& encoderInputLengths, + SizeType32 maxDecoderContextLength, SizeType32 maxEncoderInputLengthInBatch, + runtime::TllmRuntime const& runtime); + + void copySkipCrossAttnBlocks(bool const& _skipCrossAttnBlocks, runtime::TllmRuntime const& runtime); + +private: + SizeType32 maxInputLen; + SizeType32 maxEncoderOutputLen; + SizeType32 maxNumTokens; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/batch_manager/updateDecoderBuffers.h b/cpp/include/tensorrt_llm/batch_manager/updateDecoderBuffers.h new file mode 100644 index 000000000000..526a756e5546 --- /dev/null +++ b/cpp/include/tensorrt_llm/batch_manager/updateDecoderBuffers.h @@ -0,0 +1,51 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/algorithm.h" +#include "tensorrt_llm/runtime/modelConfig.h" + +namespace tensorrt_llm::runtime +{ +class BufferManager; +class CudaEvent; + +namespace decoder +{ +class DecoderState; +} // namespace decoder +} // namespace tensorrt_llm::runtime + +namespace tensorrt_llm::batch_manager +{ + +class DecoderOutputBuffers; + +class UpdateDecoderBuffers : Algorithm +{ +public: + constexpr static auto name{"UpdateDecoderBuffers"}; + + UpdateDecoderBuffers() = default; + + runtime::CudaEvent operator()(runtime::ModelConfig const& modelConfig, DecoderOutputBuffers& decoderOutputBuffers, + runtime::BufferManager const& copyBufferManager, runtime::decoder::DecoderState const& decoderState, + bool returnLogProbs, runtime::CudaEvent const& decoderFinishEvent) const; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/include/tensorrt_llm/common/dataType.h b/cpp/include/tensorrt_llm/common/dataType.h index 9b3bb5fdf0f0..2f19404f9c94 100644 --- a/cpp/include/tensorrt_llm/common/dataType.h +++ b/cpp/include/tensorrt_llm/common/dataType.h @@ -19,7 +19,7 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/tllmException.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <map> TRTLLM_NAMESPACE_BEGIN @@ -27,61 +27,61 @@ TRTLLM_NAMESPACE_BEGIN namespace common { -constexpr static size_t getDTypeSize(tensorrt_llm::DataType type) +constexpr static size_t getDTypeSize(nvinfer1::DataType type) { switch (type) { - case tensorrt_llm::DataType::kINT64: return 8; - case tensorrt_llm::DataType::kINT32: [[fallthrough]]; - case tensorrt_llm::DataType::kFLOAT: return 4; - case tensorrt_llm::DataType::kBF16: [[fallthrough]]; - case tensorrt_llm::DataType::kHALF: return 2; - case tensorrt_llm::DataType::kBOOL: [[fallthrough]]; - case tensorrt_llm::DataType::kUINT8: [[fallthrough]]; - case tensorrt_llm::DataType::kINT8: [[fallthrough]]; - case tensorrt_llm::DataType::kFP8: return 1; - case tensorrt_llm::DataType::kINT4: TLLM_THROW("Cannot determine size of INT4 data type"); - case tensorrt_llm::DataType::kFP4: TLLM_THROW("Cannot determine size of FP4 data type"); + case nvinfer1::DataType::kINT64: return 8; + case nvinfer1::DataType::kINT32: [[fallthrough]]; + case nvinfer1::DataType::kFLOAT: return 4; + case nvinfer1::DataType::kBF16: [[fallthrough]]; + case nvinfer1::DataType::kHALF: return 2; + case nvinfer1::DataType::kBOOL: [[fallthrough]]; + case nvinfer1::DataType::kUINT8: [[fallthrough]]; + case nvinfer1::DataType::kINT8: [[fallthrough]]; + case nvinfer1::DataType::kFP8: return 1; + case nvinfer1::DataType::kINT4: TLLM_THROW("Cannot determine size of INT4 data type"); + case nvinfer1::DataType::kFP4: TLLM_THROW("Cannot determine size of FP4 data type"); default: TLLM_THROW("Unknown dtype %d", static_cast<int>(type)); } return 0; } -constexpr static size_t getDTypeSizeInBits(tensorrt_llm::DataType type) +constexpr static size_t getDTypeSizeInBits(nvinfer1::DataType type) { switch (type) { - case tensorrt_llm::DataType::kINT64: return 64; - case tensorrt_llm::DataType::kINT32: [[fallthrough]]; - case tensorrt_llm::DataType::kFLOAT: return 32; - case tensorrt_llm::DataType::kBF16: [[fallthrough]]; - case tensorrt_llm::DataType::kHALF: return 16; - case tensorrt_llm::DataType::kBOOL: [[fallthrough]]; - case tensorrt_llm::DataType::kUINT8: [[fallthrough]]; - case tensorrt_llm::DataType::kINT8: [[fallthrough]]; - case tensorrt_llm::DataType::kFP8: return 8; - case tensorrt_llm::DataType::kINT4: [[fallthrough]]; - case tensorrt_llm::DataType::kFP4: return 4; + case nvinfer1::DataType::kINT64: return 64; + case nvinfer1::DataType::kINT32: [[fallthrough]]; + case nvinfer1::DataType::kFLOAT: return 32; + case nvinfer1::DataType::kBF16: [[fallthrough]]; + case nvinfer1::DataType::kHALF: return 16; + case nvinfer1::DataType::kBOOL: [[fallthrough]]; + case nvinfer1::DataType::kUINT8: [[fallthrough]]; + case nvinfer1::DataType::kINT8: [[fallthrough]]; + case nvinfer1::DataType::kFP8: return 8; + case nvinfer1::DataType::kINT4: [[fallthrough]]; + case nvinfer1::DataType::kFP4: return 4; default: TLLM_THROW("Unknown dtype %d", static_cast<int>(type)); } return 0; } -[[maybe_unused]] static std::string getDtypeString(tensorrt_llm::DataType type) +[[maybe_unused]] static std::string getDtypeString(nvinfer1::DataType type) { switch (type) { - case tensorrt_llm::DataType::kFLOAT: return "fp32"; break; - case tensorrt_llm::DataType::kHALF: return "fp16"; break; - case tensorrt_llm::DataType::kINT8: return "int8"; break; - case tensorrt_llm::DataType::kINT32: return "int32"; break; - case tensorrt_llm::DataType::kBOOL: return "bool"; break; - case tensorrt_llm::DataType::kUINT8: return "uint8"; break; - case tensorrt_llm::DataType::kFP8: return "fp8"; break; - case tensorrt_llm::DataType::kBF16: return "bf16"; break; - case tensorrt_llm::DataType::kINT64: return "int64"; break; - case tensorrt_llm::DataType::kINT4: return "int4"; break; - case tensorrt_llm::DataType::kFP4: return "fp4"; break; + case nvinfer1::DataType::kFLOAT: return "fp32"; break; + case nvinfer1::DataType::kHALF: return "fp16"; break; + case nvinfer1::DataType::kINT8: return "int8"; break; + case nvinfer1::DataType::kINT32: return "int32"; break; + case nvinfer1::DataType::kBOOL: return "bool"; break; + case nvinfer1::DataType::kUINT8: return "uint8"; break; + case nvinfer1::DataType::kFP8: return "fp8"; break; + case nvinfer1::DataType::kBF16: return "bf16"; break; + case nvinfer1::DataType::kINT64: return "int64"; break; + case nvinfer1::DataType::kINT4: return "int4"; break; + case nvinfer1::DataType::kFP4: return "fp4"; break; default: throw std::runtime_error("Unsupported data type"); break; } diff --git a/cpp/include/tensorrt_llm/common/logger.h b/cpp/include/tensorrt_llm/common/logger.h index 9073d21f0088..d14b4c02e992 100644 --- a/cpp/include/tensorrt_llm/common/logger.h +++ b/cpp/include/tensorrt_llm/common/logger.h @@ -50,6 +50,8 @@ constexpr std::string_view formatModule(std::string_view module) return "deepgemm"; else if (module == "executor") return "executor"; + else if (module == "executor_worker") + return "exec_wkr"; else if (module == "flash_mla") return "flashmla"; else if (module == "kernels") @@ -58,6 +60,8 @@ constexpr std::string_view formatModule(std::string_view module) return "layers"; else if (module == "nanobind") return "nanobind"; + else if (module == "plugins") + return "plugins"; else if (module == "runtime") return "runtime"; else if (module == "testing") diff --git a/cpp/include/tensorrt_llm/common/tllmDataType.h b/cpp/include/tensorrt_llm/common/tllmDataType.h deleted file mode 100644 index 9e5567ce280e..000000000000 --- a/cpp/include/tensorrt_llm/common/tllmDataType.h +++ /dev/null @@ -1,85 +0,0 @@ -/* - * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#pragma once - -#include "tensorrt_llm/common/config.h" - -#include <cstdint> - -//! \file tllmDataType.h -//! -//! Standalone, TensorRT-free runtime types that replace the \c nvinfer1 -//! types the shared C++ core historically used as common currency: -//! \c tensorrt_llm::common::DataType and \c tensorrt_llm::common::Dims. -//! These are defined here so the retained tree (runtime, batch manager, executor, -//! kernels and the nanobind bridge) compiles and links without the TensorRT -//! library. They are hoisted into the \c tensorrt_llm namespace with -//! using-declarations because they are common currency across the whole tree -//! (\c tensorrt_llm::DataType / \c tensorrt_llm::Dims). -//! -//! The \c DataType enumerator values intentionally mirror the legacy -//! \c nvinfer1::DataType integer values so that previously-serialized executor -//! configs and KV-cache metadata remain byte-compatible. The \c Dims layout -//! mirrors the legacy \c nvinfer1::Dims (\c int32_t \c nbDims followed by -//! \c int64_t \c d[8]) for the same reason. - -TRTLLM_NAMESPACE_BEGIN - -namespace common -{ - -//! \brief Standalone data-type enum. Values mirror the legacy -//! \c nvinfer1::DataType for serialization/format compatibility. -enum class DataType : int32_t -{ - kFLOAT = 0, - kHALF = 1, - kINT8 = 2, - kINT32 = 3, - kBOOL = 4, - kUINT8 = 5, - kFP8 = 6, - kBF16 = 7, - kINT64 = 8, - kINT4 = 9, - kFP4 = 10, - kE8M0 = 11, -}; - -//! \brief Standalone dimensions type. Layout mirrors the legacy -//! \c nvinfer1::Dims (rank plus up to \c MAX_DIMS 64-bit extents) so serialized -//! shapes remain compatible. -class Dims -{ -public: - //! The maximum rank (number of dimensions) supported for a tensor. - static constexpr int32_t MAX_DIMS{8}; - - //! The rank (number of dimensions). - int32_t nbDims; - - //! The extent of each dimension. - int64_t d[MAX_DIMS]; -}; - -} // namespace common - -using common::DataType; -using common::Dims; - -TRTLLM_NAMESPACE_END diff --git a/cpp/include/tensorrt_llm/executor/dataTransceiverState.h b/cpp/include/tensorrt_llm/executor/dataTransceiverState.h index 578e53b81dbf..5067ae61dc83 100644 --- a/cpp/include/tensorrt_llm/executor/dataTransceiverState.h +++ b/cpp/include/tensorrt_llm/executor/dataTransceiverState.h @@ -18,7 +18,6 @@ #include "tensorrt_llm/batch_manager/llmRequest.h" #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/runtime/modelConfig.h" #include "tensorrt_llm/runtime/worldConfig.h" @@ -51,7 +50,7 @@ class CacheState final }; CacheState(ModelConfig modelConfig, runtime::WorldConfig const& worldConfig, - std::vector<SizeType32> const& attentionLayerNumPerPP, tensorrt_llm::DataType dataType, + std::vector<SizeType32> const& attentionLayerNumPerPP, nvinfer1::DataType dataType, AttentionType attentionType = AttentionType::kDEFAULT, int kvFactor = 2, bool enableBlockReuse = false, bool enablePartialReuse = false, bool hasIndexerKCache = false, SizeType32 indexerDimPerHead = 0, SizeType32 indexerKCacheQuantBlockSize = 128, bool indexerKCacheUseFp4 = false) @@ -72,7 +71,7 @@ class CacheState final CacheState(std::vector<SizeType32> nbKvHeadPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, SizeType32 tensorParallelism, SizeType32 pipelineParallelism, SizeType32 contextParallelism, - std::vector<SizeType32> const& attentionLayerNumPerPP, tensorrt_llm::DataType dataType, + std::vector<SizeType32> const& attentionLayerNumPerPP, nvinfer1::DataType dataType, AttentionType attentionType = AttentionType::kDEFAULT, int kvFactor = 2, bool enableAttentionDP = false, int DPrank = 0, int DPsize = 0, bool enableBlockReuse = false, bool enablePartialReuse = false, bool hasIndexerKCache = false, SizeType32 indexerDimPerHead = 0, SizeType32 indexerKCacheQuantBlockSize = 128, @@ -93,7 +92,7 @@ class CacheState final CacheState(SizeType32 nbAttentionLayers, SizeType32 nbKvHeads, SizeType32 sizePerHead, SizeType32 tokensPerBlock, SizeType32 tensorParallelism, SizeType32 pipelineParallelism, SizeType32 contextParallelism, - std::vector<SizeType32> const& attentionLayerNumPerPP, tensorrt_llm::DataType dataType, + std::vector<SizeType32> const& attentionLayerNumPerPP, nvinfer1::DataType dataType, AttentionType attentionType = AttentionType::kDEFAULT, int kvFactor = 2, bool enableAttentionDP = false, int DPrank = 0, int DPsize = 0, bool enableBlockReuse = false, bool enablePartialReuse = false, bool hasIndexerKCache = false, SizeType32 indexerDimPerHead = 0, SizeType32 indexerKCacheQuantBlockSize = 128, @@ -239,8 +238,8 @@ class CacheState final RnnModelConfig mModelConfig; /// Number of RNN layers per pipeline parallelism rank. std::vector<SizeType32> mLayerNumPerPP; - tensorrt_llm::DataType mConvStateDataType; - tensorrt_llm::DataType mSsmStateDataType; + nvinfer1::DataType mConvStateDataType; + nvinfer1::DataType mSsmStateDataType; [[nodiscard]] bool operator==(RnnCacheState const& other) const noexcept { @@ -264,7 +263,7 @@ class CacheState final return mAttentionConfig; } - [[nodiscard]] tensorrt_llm::DataType const& getDataType() const + [[nodiscard]] nvinfer1::DataType const& getDataType() const { return mDataType; } @@ -309,7 +308,7 @@ class CacheState final } void setRnnConfig(RnnModelConfig rnnModelConfig, std::vector<SizeType32> rnnLayerNumPerPP, - tensorrt_llm::DataType convStateDataType, tensorrt_llm::DataType ssmStateDataType) + nvinfer1::DataType convStateDataType, nvinfer1::DataType ssmStateDataType) { mRnnCacheState = RnnCacheState{ std::move(rnnModelConfig), std::move(rnnLayerNumPerPP), convStateDataType, ssmStateDataType}; @@ -326,12 +325,12 @@ class CacheState final return getRnnCacheState().mModelConfig; } - [[nodiscard]] tensorrt_llm::DataType getConvStateDataType() const + [[nodiscard]] nvinfer1::DataType getConvStateDataType() const { return getRnnCacheState().mConvStateDataType; } - [[nodiscard]] tensorrt_llm::DataType getSsmStateDataType() const + [[nodiscard]] nvinfer1::DataType getSsmStateDataType() const { return getRnnCacheState().mSsmStateDataType; } @@ -396,7 +395,7 @@ class CacheState final friend class tensorrt_llm::executor::Serialization; ModelConfig mModelConfig; ParallelConfig mParallelConfig; - tensorrt_llm::DataType mDataType; + nvinfer1::DataType mDataType; AttentionConfig mAttentionConfig; bool mEnableBlockReuse{false}; bool mEnablePartialReuse{false}; @@ -619,22 +618,9 @@ class DataTransceiverState final return mCacheState.has_value() && mCacheState->hasRnnConfig(); } - /// @brief Set only when exported via CacheTransceiver::getSerializedDataTransceiverState: - /// transfers driven by such a state have no LlmRequest on the sender. - [[nodiscard]] bool isArbitraryTransferState() const noexcept - { - return mIsArbitraryTransferState; - } - - void setIsArbitraryTransferState(bool isArbitraryTransferState) noexcept - { - mIsArbitraryTransferState = isArbitraryTransferState; - } - [[nodiscard]] bool operator==(DataTransceiverState const& other) const noexcept { - return mCacheState == other.mCacheState && mCommState == other.mCommState - && mIsArbitraryTransferState == other.mIsArbitraryTransferState; + return mCacheState == other.mCacheState && mCommState == other.mCommState; } [[nodiscard]] std::string toString() const @@ -655,7 +641,6 @@ class DataTransceiverState final friend class Serialization; std::optional<kv_cache::CacheState> mCacheState; std::optional<kv_cache::CommState> mCommState; - bool mIsArbitraryTransferState{false}; }; } // namespace tensorrt_llm::executor diff --git a/cpp/include/tensorrt_llm/executor/disaggServerUtil.h b/cpp/include/tensorrt_llm/executor/disaggServerUtil.h new file mode 100644 index 000000000000..b68dce78738a --- /dev/null +++ b/cpp/include/tensorrt_llm/executor/disaggServerUtil.h @@ -0,0 +1,158 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/executor/executor.h" + +#include <cstdio> +#include <filesystem> +#include <memory> +#include <optional> +#include <vector> + +namespace tensorrt_llm::executor::disagg_executor +{ + +namespace texec = tensorrt_llm::executor; + +struct ResponseWithId +{ + + tensorrt_llm::executor::Response response; + IdType gid; + + ResponseWithId(tensorrt_llm::executor::Response&& response, IdType gid) + : response(std::move(response)) + , gid(gid) + { + } + + ResponseWithId(tensorrt_llm::executor::Response const& response, IdType gid) + : response(response) + , gid(gid) + { + } + + ResponseWithId(ResponseWithId&& other) noexcept + : response(std::move(other.response)) + , gid(other.gid) + { + other.gid = {}; + } + + ResponseWithId(ResponseWithId const& other) = default; + + ResponseWithId& operator=(ResponseWithId&& other) noexcept + { + if (this != &other) + { + response = std::move(other.response); + gid = other.gid; + other.gid = {}; + } + return *this; + } + + ResponseWithId& operator=(ResponseWithId const& other) + { + + if (this != &other) + { + response = other.response; + gid = other.gid; + } + return *this; + } + + ~ResponseWithId() = default; +}; + +class DisaggExecutorOrchestrator +{ +public: + /// @brief Constructs a DisaggExecutorOrchestrator object. + /// + /// @param ctxEnginePaths A vector of file paths to context engine files. + /// @param genEnginePaths A vector of file paths to generation engine files. + /// @param ctxExecutorConfigs A vector of ExecutorConfig for context executors. + /// @param genExecutorConfigs A vector of ExecutorConfig for generation executors. + /// @param hasContextAwaitThreads Whether or not there are threads that receive response for each generation + /// executor. + /// @param hasGenAwaitThreads Whether or not there are threads that receive response for each generation executor. + + DisaggExecutorOrchestrator(std::vector<std::filesystem::path> const& ctxEnginePaths, + std::vector<std::filesystem::path> const& genEnginePaths, + std::vector<executor::ExecutorConfig> const& ctxExecutorConfigs, + std::vector<executor::ExecutorConfig> const& genExecutorConfigs, bool hasContextAwaitThreads, + bool hasGenAwaitThreads); + + /// @brief Enqueue context-only requests to context executors. + /// @param requests A vector of context-only requests. + /// @param selectContextId The index of the context executor to use. If `std::nullopt`, the executor that has the + /// smallest number of inflight requests will be used. + /// @param batch If true,enqueue requests in same context executor.If false, will try to use a different executor + /// for each request. + /// @return A vector of global request ids, corresponding to the order of the requests in `requests`, the id + /// returned may be different from the request id in each executor. + [[nodiscard]] std::vector<IdType> enqueueContext(std::vector<texec::Request> const& requests, + std::optional<int> selectContextId = std::nullopt, bool batch = false); + + /// @brief Enqueue generation-only requests to generation executors. + /// @param requests A vector of generation-only requests. + /// @param globalRequestIds A vector of global request ids, corresponding to the order of the requests,and must be + /// the ids returned by the enqueueContext function. + /// @param selectGenIdx The index of the generation executor to use. If `std::nullopt`, the executor that has the + /// smallest number of inflight requests will be used. + /// @param batch If true,enqueue requests in same generation executor.If false, will try to use a different executor + /// for each request. + + void enqueueGeneration(std::vector<texec::Request> const& requests, std::vector<IdType> const& globalRequestIds, + std::optional<int> selectGenIdx = std::nullopt, bool batch = false); + + /// @brief Await for context responses + /// @param timeout The maximum time to wait for new responses + /// @param contextIdx The index of the context executor to use. If `std::nullopt`, return ready responses in all + /// context executors,if `hasContextAwaitThreads` is true, then this parameter must be std::nullopt. + /// @return A vector of responses with corresponding global request ids + + [[nodiscard]] std::vector<ResponseWithId> awaitContextResponses( + std::optional<std::chrono::milliseconds> const& timeout, std::optional<int> contextIdx = std::nullopt); + + /// @brief Await for generation responses + /// @param timeout The maximum time to wait for new responses. + /// @param genIdx The index of the generation executor to use. If `std::nullopt`, return ready responses in all + /// generation executors,if `hasGenAwaitThreads` is true, then this parameter must be std::nullopt. + /// @return A vector of responses with corresponding global request ids. + [[nodiscard]] std::vector<ResponseWithId> awaitGenerationResponses( + std::optional<std::chrono::milliseconds> const& timeout, std::optional<int> genIdx = std::nullopt); + + /// @brief Indicates if the current process is allowed to enqueueRequests + [[nodiscard]] bool canEnqueue() const; + + /// @brief Get context executors + [[nodiscard]] std::vector<std::unique_ptr<texec::Executor>> const& getContextExecutors() const; + + /// @brief Get generation executors + [[nodiscard]] std::vector<std::unique_ptr<texec::Executor>> const& getGenExecutors() const; + + ~DisaggExecutorOrchestrator(); + +private: + class Impl; + std::unique_ptr<Impl> mImpl; +}; +} // namespace tensorrt_llm::executor::disagg_executor diff --git a/cpp/include/tensorrt_llm/executor/executor.h b/cpp/include/tensorrt_llm/executor/executor.h index acc0efe18966..825b8ad75959 100644 --- a/cpp/include/tensorrt_llm/executor/executor.h +++ b/cpp/include/tensorrt_llm/executor/executor.h @@ -18,7 +18,6 @@ #include "tensorrt_llm/executor/tensor.h" #include "tensorrt_llm/executor/types.h" -#include "tensorrt_llm/executor/version.h" #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/runtimeDefaults.h" @@ -51,11 +50,9 @@ namespace tensorrt_llm::executor using SizeType32 = tensorrt_llm::runtime::SizeType32; /// @brief Version of TRT-LLM -inline char const* version() noexcept -{ - return kTensorRtLlmVersion; -} +char const* version() noexcept; +class Model; class Serialization; class DataTransceiverState; @@ -1236,11 +1233,6 @@ class DebugConfig SizeType32 mDebugTensorsMaxIterations; }; -/// @brief Configuration for the orchestrator communication mode. -/// @deprecated Orchestrator mode is non-functional: the worker binary it spawned -/// (executorWorker) was removed together with the TensorRT backend. This class is -/// retained only for serialization and Python-binding compatibility and is a -/// candidate for removal in a follow-up (needs API-stability review). class OrchestratorConfig { public: @@ -1843,13 +1835,7 @@ using KVCacheEventData = std::variant<KVCacheCreatedData, KVCacheStoredData, KVC struct KVCacheEvent { KVCacheEvent(IdType eventId, KVCacheEventData data, SizeType32 windowSize, - std::optional<SizeType32> attentionDpRank = std::nullopt) - : eventId{eventId} - , data{std::move(data)} - , windowSize{windowSize} - , attentionDpRank{attentionDpRank} - { - } + std::optional<SizeType32> attentionDpRank = std::nullopt); /// @brief The unique id of this event IdType eventId; @@ -1877,6 +1863,119 @@ class KVCacheEventManager std::shared_ptr<tensorrt_llm::batch_manager::kv_cache_manager::BaseKVCacheManager> kvCacheManager; }; +/// @brief The executor is responsible for receiving new requests and sending responses, and running the inference +class Executor +{ + +public: + /// @brief + /// @param modelPath Path to the folder that defines the model to run + /// @param modelType The type of model + /// @param executorConfig The configuration for the executor + Executor(std::filesystem::path const& modelPath, ModelType modelType, ExecutorConfig const& executorConfig); + + Executor(std::filesystem::path const& encoderModelPath, std::filesystem::path const& decoderModelPath, + ModelType modelType, ExecutorConfig const& executorConfig); + + Executor(BufferView const& engineBuffer, std::string const& jsonConfigStr, ModelType modelType, + ExecutorConfig const& executorConfig, + std::optional<std::map<std::string, Tensor>> const& managedWeights = std::nullopt); + + Executor(BufferView const& encoderEngineBuffer, std::string const& encoderJsonConfigStr, + BufferView const& decoderEngineBuffer, std::string const& decoderJsonConfigStr, ModelType modelType, + ExecutorConfig const& executorConfig); + + Executor(std::shared_ptr<Model> model, ExecutorConfig const& executorConfig); + + Executor( + std::shared_ptr<Model> encoderModel, std::shared_ptr<Model> decoderModel, ExecutorConfig const& executorConfig); + + ~Executor(); + Executor(Executor const& executor) = delete; + Executor& operator=(Executor const& executor) = delete; + Executor(Executor&&) = default; + Executor& operator=(Executor&&) = default; + + /// @brief Enqueue a new request + /// @param request The LLM request which contains input tokens and request parameters + /// @return A unique id that identifies the request + [[nodiscard]] IdType enqueueRequest(Request const& request); + + /// @brief Enqueue a batch of request + [[nodiscard]] std::vector<IdType> enqueueRequests(std::vector<Request> const& requests); + + /// @brief Await for ready responses + /// + /// This overload awaits for any ready responses. In particular, if several requests + /// have been enqueued, this method will provide any ready responses without order guarantees. + /// @param timeout The maximum time to wait for new responses + /// @return A vector of responses + [[nodiscard]] std::vector<Response> awaitResponses( + std::optional<std::chrono::milliseconds> const& timeout = std::nullopt); + + /// @brief Await for ready responses + /// @param id A request id + /// @param timeout The maximum time to wait for new responses + /// @return A vector of responses + [[nodiscard]] std::vector<Response> awaitResponses( + IdType const& requestId, std::optional<std::chrono::milliseconds> const& timeout = std::nullopt); + + /// @brief Await for multiple ready responses + /// + /// A multiple ID request behaves as if awaitResponses(IdType, timeout) + /// were invoked on all IDs. The returned vector contains + /// a vector of responses per ID in the same order specified by the requestIds. + /// The same behaviour as awaitResponses(IdType, timeout) applies: + /// * Responses may be empty. + /// * If all responses have already been given for one of the requestIds, + /// then this method will hang unless a timeout is specified. + /// @param requestIds Ids requested + /// @param timeout The maximum time to wait for new responses + /// @return A vector of vector of responses + [[nodiscard]] std::vector<std::vector<Response>> awaitResponses( + std::vector<IdType> const& requestIds, std::optional<std::chrono::milliseconds> const& timeout = std::nullopt); + + /// @brief Get the number of ready responses + /// @param requestId An optional request id + /// @return The number of ready responses + [[nodiscard]] SizeType32 getNumResponsesReady(std::optional<IdType> const& requestId = std::nullopt) const; + + /// @brief Cancel the request with provided request id + /// @param id The request id for which to cancel the response + void cancelRequest(IdType requestId); + + /// @brief Signals the server to shutdown. + /// @details This call is blocking. Only returns when all requests have terminated or timeout has been reached + void shutdown(); + + /// @brief Returns the per-iterations statistics computed since last call to getLatestIterationStats. + /// Contains at most iterStatsMaxIterations iterations, or all iterations when set to -1. + /// @return Iteration stats + std::deque<IterationStats> getLatestIterationStats(); + + /// @brief Returns the request stats of each iteration computed since last call to getLatestRequestStats. + /// Contains at most requestStatsMaxIterations iterations, or all iterations when set to -1. + /// @return Request stats grouped by iterations + std::deque<RequestStatsPerIteration> getLatestRequestStats(); + + /// @brief Returns the debug tensors of each iteration computed since last call to getLatestDebugTensors. + /// Contains at most debugTensorsMaxIterations iterations. + /// @return Request debug tensors grouped by iterations + std::deque<DebugTensorsPerIteration> getLatestDebugTensors(); + + /// @brief Indicates if the current process is allowed to enqueueRequests + [[nodiscard]] bool canEnqueueRequests() const; + + /// @brief Indicates if the current process participates in this executor instance + [[nodiscard]] bool isParticipant() const; + + std::optional<std::shared_ptr<KVCacheEventManager>> getKVCacheEventManager() const; + +private: + class Impl; + std::unique_ptr<Impl> mImpl; +}; + /// @brief Class with utility functions to serialize statistics to json string class JsonSerialization { diff --git a/cpp/include/tensorrt_llm/executor/transferAgent.h b/cpp/include/tensorrt_llm/executor/transferAgent.h index 5f175a33723c..532f0ae70c44 100644 --- a/cpp/include/tensorrt_llm/executor/transferAgent.h +++ b/cpp/include/tensorrt_llm/executor/transferAgent.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -216,20 +216,12 @@ struct VmmDescSplitter /// For non-VRAM or addresses not in the map, descs pass through unchanged. [[nodiscard]] static MemoryDescs splitDescsWithRegionMap(MemoryDescs const& descs, VramRegionMap const& regionMap); - /// @brief Split paired src/dst descs at chunk boundaries, then coalesce contiguous pieces. - /// src is split by localRegionMap, dst is split by remoteRegionMap; each piece size is - /// min(srcPiece, dstPiece, remaining). Pairs are sorted by src address, and adjacent pieces - /// whose src AND dst are both contiguous (same deviceId) are merged — but a merged desc never - /// crosses a chunk boundary on either side, and never spans two distinct regions, so every - /// output desc stays within a single registered memory region. Merging requires region - /// metadata: a piece whose address misses the region map on either side is never merged, - /// because two unknown regions are indistinguishable and a merge could cross a chunk or - /// registration boundary. With no region metadata the result is split-only. Non-kVRAM descs - /// pass through unchanged (no region info is available to bound the merge). - /// @param enableCoalesce When false, only split at chunk boundaries without merging pieces. - [[nodiscard]] static std::pair<MemoryDescs, MemoryDescs> splitAndCoalesceTransferDescs(MemoryDescs const& srcDescs, - MemoryDescs const& dstDescs, VramRegionMap const& localRegionMap, VramRegionMap const& remoteRegionMap, - bool enableCoalesce = true); + /// @brief Split paired src/dst descs using local and remote region maps. + /// src is split by localRegionMap, dst is split by remoteRegionMap. + /// The final piece size is min(srcPiece, dstPiece, remaining). + [[nodiscard]] static std::pair<MemoryDescs, MemoryDescs> splitTransferDescsWithRegionMaps( + MemoryDescs const& srcDescs, MemoryDescs const& dstDescs, VramRegionMap const& localRegionMap, + VramRegionMap const& remoteRegionMap); /// @brief Split VRAM descs at VMM chunk boundaries detected via cuMemGetAddressRange. /// For cudaMalloc memory (single allocation), descs pass through unchanged. @@ -364,13 +356,6 @@ class TransferStatus virtual ~TransferStatus() = default; [[nodiscard]] virtual bool isCompleted() const = 0; virtual TransferState wait(int64_t timeout_ms = -1) const = 0; - - /// Release the backend transfer request handle. A true return means the backend accepted the handle release; it - /// does not prove remote memory quiescence. - [[nodiscard]] virtual bool release() - { - return false; - } }; struct BaseAgentConfig diff --git a/cpp/include/tensorrt_llm/plugins/api/tllmPlugin.h b/cpp/include/tensorrt_llm/plugins/api/tllmPlugin.h new file mode 100644 index 000000000000..e3d4613e3d00 --- /dev/null +++ b/cpp/include/tensorrt_llm/plugins/api/tllmPlugin.h @@ -0,0 +1,74 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include <cstdint> +#include <mutex> + +// Forward declarations +namespace nvinfer1 +{ +class ILoggerFinder; +class ILogger; + +namespace v_1_0 +{ +class IPluginCreator; +class IPluginCreatorV3One; +class IPluginCreatorInterface; +} // namespace v_1_0 + +} // namespace nvinfer1 + +namespace tensorrt_llm::plugins::api +{ + +auto constexpr kDefaultNamespace = "tensorrt_llm"; + +class LoggerManager +{ +public: + //! Set the logger finder. + void setLoggerFinder(nvinfer1::ILoggerFinder* finder); + + //! Get the logger. + [[maybe_unused]] nvinfer1::ILogger* logger(); + + static LoggerManager& getInstance() noexcept; + + static nvinfer1::ILogger* defaultLogger() noexcept; + +private: + LoggerManager() = default; + + nvinfer1::ILoggerFinder* mLoggerFinder{nullptr}; + std::mutex mMutex; +}; +} // namespace tensorrt_llm::plugins::api + +extern "C" +{ + // This function is used for explicitly registering the TRT-LLM plugins and the default logger. + bool initTrtLlmPlugins(void* logger = tensorrt_llm::plugins::api::LoggerManager::defaultLogger(), + char const* libNamespace = tensorrt_llm::plugins::api::kDefaultNamespace); + + // The functions below are used by TensorRT to when loading a shared plugin library with automatic registering. + // see https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#generating-plugin-library + [[maybe_unused]] void setLoggerFinder([[maybe_unused]] nvinfer1::ILoggerFinder* finder); + [[maybe_unused]] nvinfer1::v_1_0::IPluginCreator* const* getPluginCreators(std::int32_t& nbCreators); + [[maybe_unused]] nvinfer1::v_1_0::IPluginCreatorInterface* const* getCreators(std::int32_t& nbCreators); +} diff --git a/cpp/include/tensorrt_llm/runtime/bufferManager.h b/cpp/include/tensorrt_llm/runtime/bufferManager.h index 321a96ba321e..8357443dc5ea 100644 --- a/cpp/include/tensorrt_llm/runtime/bufferManager.h +++ b/cpp/include/tensorrt_llm/runtime/bufferManager.h @@ -17,10 +17,10 @@ #pragma once #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/cudaStream.h" #include "tensorrt_llm/runtime/iBuffer.h" #include "tensorrt_llm/runtime/iTensor.h" +#include <NvInferRuntime.h> #include <cstring> #include <memory> @@ -62,63 +62,63 @@ class BufferManager } } - static auto constexpr kBYTE_TYPE = tensorrt_llm::DataType::kUINT8; + static auto constexpr kBYTE_TYPE = nvinfer1::DataType::kUINT8; //! \brief Allocates an `IBuffer` of the given size on the GPU, using cudaMallocAsync. - [[nodiscard]] IBufferPtr gpu(std::size_t size, tensorrt_llm::DataType type = kBYTE_TYPE) const; + [[nodiscard]] IBufferPtr gpu(std::size_t size, nvinfer1::DataType type = kBYTE_TYPE) const; //! \brief Allocates an `ITensor` of the given dimensions on the GPU, using cudaMallocAsync. - [[nodiscard]] ITensorPtr gpu(tensorrt_llm::Dims dims, tensorrt_llm::DataType type = kBYTE_TYPE) const; + [[nodiscard]] ITensorPtr gpu(nvinfer1::Dims dims, nvinfer1::DataType type = kBYTE_TYPE) const; //! \brief Allocates an `IBuffer` of the given size on the GPU, using cudaMalloc. - [[nodiscard]] static IBufferPtr gpuSync(std::size_t size, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static IBufferPtr gpuSync(std::size_t size, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates an `ITensor` of the given dimensions on the GPU, using cudaMalloc. - [[nodiscard]] static ITensorPtr gpuSync(tensorrt_llm::Dims dims, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static ITensorPtr gpuSync(nvinfer1::Dims dims, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates an `IBuffer` of the given size on the CPU. - [[nodiscard]] static IBufferPtr cpu(std::size_t size, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static IBufferPtr cpu(std::size_t size, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates an `ITensor` of the given dimensions on the CPU. - [[nodiscard]] static ITensorPtr cpu(tensorrt_llm::Dims dims, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static ITensorPtr cpu(nvinfer1::Dims dims, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates a pinned `IBuffer` of the given size on the CPU. - [[nodiscard]] static IBufferPtr pinned(std::size_t size, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static IBufferPtr pinned(std::size_t size, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates a pinned `ITensor` of the given dimensions on the CPU. - [[nodiscard]] static ITensorPtr pinned(tensorrt_llm::Dims dims, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static ITensorPtr pinned(nvinfer1::Dims dims, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates a pinned `IBuffer` of the given size on the CPU in the default memory pool. - [[nodiscard]] static IBufferPtr pinnedPool(std::size_t size, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static IBufferPtr pinnedPool(std::size_t size, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates a pinned `ITensor` of the given dimensions on the CPU in the default memory pool. - [[nodiscard]] static ITensorPtr pinnedPool(tensorrt_llm::Dims dims, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static ITensorPtr pinnedPool(nvinfer1::Dims dims, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates an `IBuffer` of the given size in UVM. - [[nodiscard]] static IBufferPtr managed(std::size_t size, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static IBufferPtr managed(std::size_t size, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates an `ITensor` of the given dimensions in UVM. - [[nodiscard]] static ITensorPtr managed(tensorrt_llm::Dims dims, tensorrt_llm::DataType type = kBYTE_TYPE); + [[nodiscard]] static ITensorPtr managed(nvinfer1::Dims dims, nvinfer1::DataType type = kBYTE_TYPE); //! \brief Allocates an `ITensor` of the given dimensions for NVLS - [[nodiscard]] static ITensorPtr ipcNvls(std::set<int> ranks, tensorrt_llm::Dims dims, tensorrt_llm::DataType type); + [[nodiscard]] static ITensorPtr ipcNvls(std::set<int> ranks, nvinfer1::Dims dims, nvinfer1::DataType type); //! \brief Allocates an `IBuffer` of the given size and memory type. [[nodiscard]] IBufferPtr allocate( - MemoryType memoryType, std::size_t size, tensorrt_llm::DataType type = kBYTE_TYPE) const; + MemoryType memoryType, std::size_t size, nvinfer1::DataType type = kBYTE_TYPE) const; //! \brief Allocates an `ITensor` of the given dimensions and memory type. [[nodiscard]] ITensorPtr allocate( - MemoryType memoryType, tensorrt_llm::Dims dims, tensorrt_llm::DataType type = kBYTE_TYPE) const; + MemoryType memoryType, nvinfer1::Dims dims, nvinfer1::DataType type = kBYTE_TYPE) const; //! \brief Create an empty `IBuffer` of the given memory type. It may be resized later. - [[nodiscard]] IBufferPtr emptyBuffer(MemoryType memoryType, tensorrt_llm::DataType type = kBYTE_TYPE) const + [[nodiscard]] IBufferPtr emptyBuffer(MemoryType memoryType, nvinfer1::DataType type = kBYTE_TYPE) const { return allocate(memoryType, 0, type); } //! \brief Create an empty `ITensor` of the given memory type. It may be reshaped later. - [[nodiscard]] ITensorPtr emptyTensor(MemoryType memoryType, tensorrt_llm::DataType type = kBYTE_TYPE) const + [[nodiscard]] ITensorPtr emptyTensor(MemoryType memoryType, nvinfer1::DataType type = kBYTE_TYPE) const { return allocate(memoryType, ITensor::makeShape({}), type); } @@ -167,7 +167,7 @@ class BufferManager //! \brief Copy `src` into a new `ITensor` with a potentially different memory type. template <typename T> - [[nodiscard]] ITensorPtr copyFrom(T* src, tensorrt_llm::Dims dims, MemoryType memoryType) const + [[nodiscard]] ITensorPtr copyFrom(T* src, nvinfer1::Dims dims, MemoryType memoryType) const { auto buffer = allocate(memoryType, dims, TRTDataType<std::remove_cv_t<T>>::value); copy(src, *buffer); @@ -176,7 +176,7 @@ class BufferManager //! \brief Copy `src` into a new `ITensor` with a potentially different memory type. template <typename T> - [[nodiscard]] ITensorPtr copyFrom(std::vector<T> const& src, tensorrt_llm::Dims dims, MemoryType memoryType) const + [[nodiscard]] ITensorPtr copyFrom(std::vector<T> const& src, nvinfer1::Dims dims, MemoryType memoryType) const { TLLM_CHECK_WITH_INFO(src.size() == ITensor::volumeNonNegative(dims), common::fmtstr("[TensorRT-LLM][ERROR] Incompatible size %lu and dims %s", src.size(), diff --git a/cpp/include/tensorrt_llm/runtime/decoderState.h b/cpp/include/tensorrt_llm/runtime/decoderState.h index ea2c767c0478..95d7ff0ffac9 100644 --- a/cpp/include/tensorrt_llm/runtime/decoderState.h +++ b/cpp/include/tensorrt_llm/runtime/decoderState.h @@ -18,7 +18,6 @@ #include "decodingInput.h" #include "decodingOutput.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/speculativeDecodingMode.h" @@ -53,7 +52,7 @@ class DecoderState //! @brief Setup buffers for the decoder excluding speculative decoding. void setup(SizeType32 maxNumSequences, SizeType32 maxBeamWidth, SizeType32 maxAttentionWindow, - SizeType32 sinkTokenLength, SizeType32 maxSequenceLength, tensorrt_llm::DataType dtype, + SizeType32 sinkTokenLength, SizeType32 maxSequenceLength, nvinfer1::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig, BufferManager const& bufferManager); //! @brief Setup buffers for the cache indirection. @@ -63,7 +62,7 @@ class DecoderState //! @brief Setup buffers for speculative decoding. void setupSpeculativeDecoding(SpeculativeDecodingMode const& speculativeDecodingMode, - SizeType32 maxTokensPerEngineStep, tensorrt_llm::DataType dtype, ModelConfig const& modelConfig, + SizeType32 maxTokensPerEngineStep, nvinfer1::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig, BufferManager const& bufferManager); //! @brief Disable lookahead decoding. @@ -200,7 +199,7 @@ class DecoderState [[nodiscard]] DecodingOutput& getJointDecodingOutput() const; private: - void setupBuffers(tensorrt_llm::DataType dtype, BufferManager const& bufferManager); + void setupBuffers(nvinfer1::DataType dtype, BufferManager const& bufferManager); void reshapeBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, SizeType32 maxAttentionWindow, SizeType32 sinkTokenLength, SizeType32 maxSequenceLength, ModelConfig const& modelConfig, WorldConfig const& worldConfig, BufferManager const& bufferManager); @@ -209,8 +208,8 @@ class DecoderState void reshapeCacheIndirectionBuffers( SizeType32 maxBatchSize, SizeType32 maxBeamWidth, SizeType32 maxAttentionWindow); - void setupSpeculativeDecodingBuffers(SpeculativeDecodingMode speculativeDecodingMode, tensorrt_llm::DataType dtype, - BufferManager const& bufferManager); + void setupSpeculativeDecodingBuffers( + SpeculativeDecodingMode speculativeDecodingMode, nvinfer1::DataType dtype, BufferManager const& bufferManager); void reshapeSpeculativeDecodingBuffers(SpeculativeDecodingMode const& speculativeDecodingMode, SizeType32 maxTokensPerEngineStep, ModelConfig const& modelConfig, WorldConfig const& worldConfig, BufferManager const& bufferManager); diff --git a/cpp/include/tensorrt_llm/runtime/gptDecoder.h b/cpp/include/tensorrt_llm/runtime/gptDecoder.h index 5a785e84fe75..7e0cc1bb56d2 100644 --- a/cpp/include/tensorrt_llm/runtime/gptDecoder.h +++ b/cpp/include/tensorrt_llm/runtime/gptDecoder.h @@ -22,7 +22,7 @@ #include "tensorrt_llm/runtime/decodingOutput.h" #include "tensorrt_llm/runtime/samplingConfig.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <curand_kernel.h> #include <memory> @@ -55,7 +55,7 @@ class IGptDecoder /// @param explicitDraftTokensDType is only used by ExplicitDraftTokens model to WAR the lack of bf16 decoder. virtual void setup(SamplingConfig const& samplingConfig, size_t batchSize, TensorConstPtr const& batchSlots, std::optional<DecodingOutput> const& output = std::nullopt, - std::optional<tensorrt_llm::DataType> explicitDraftTokensDType = std::nullopt, + std::optional<nvinfer1::DataType> explicitDraftTokensDType = std::nullopt, std::optional<std::vector<TensorConstPtr>> const& lookaheadPrompt = std::nullopt, std::optional<std::vector<executor::LookaheadDecodingConfig>> const& lookaheadAlgoConfigs = std::nullopt) = 0; @@ -70,7 +70,7 @@ class IGptDecoder std::optional<SamplingConfig> const& samplingConfig, SizeType32 batchSize, TensorConstPtr batchSlots) = 0; - static std::unique_ptr<IGptDecoder> create(executor::DecodingMode const& mode, tensorrt_llm::DataType dtype, + static std::unique_ptr<IGptDecoder> create(executor::DecodingMode const& mode, nvinfer1::DataType dtype, size_t maxNumSequences, size_t maxBeamWidth, size_t vocabSize, size_t vocabSizePadded, BufferManager::CudaStreamPtr const& stream, std::shared_ptr<SpeculativeDecodingModule const> const& speculativeDecodingModule = nullptr); @@ -90,7 +90,7 @@ class GptDecoder : public virtual IGptDecoder void setup(SamplingConfig const& samplingConfig, size_t batchSize, TensorConstPtr const& batchSlots, std::optional<DecodingOutput> const& output = std::nullopt, - std::optional<tensorrt_llm::DataType> explicitDraftTokensDType = std::nullopt, + std::optional<nvinfer1::DataType> explicitDraftTokensDType = std::nullopt, std::optional<std::vector<TensorConstPtr>> const& lookaheadPrompt = std::nullopt, std::optional<std::vector<executor::LookaheadDecodingConfig>> const& lookaheadAlgoConfigs = std::nullopt) override; @@ -121,17 +121,17 @@ class GptDecoder : public virtual IGptDecoder executor::DecodingMode mDecodingMode; }; -inline std::unique_ptr<IGptDecoder> IGptDecoder::create(executor::DecodingMode const& mode, - tensorrt_llm::DataType dtype, size_t maxNumSequences, size_t maxBeamWidth, size_t vocabSize, size_t vocabSizePadded, +inline std::unique_ptr<IGptDecoder> IGptDecoder::create(executor::DecodingMode const& mode, nvinfer1::DataType dtype, + size_t maxNumSequences, size_t maxBeamWidth, size_t vocabSize, size_t vocabSizePadded, BufferManager::CudaStreamPtr const& stream, std::shared_ptr<SpeculativeDecodingModule const> const& speculativeDecodingModule) { switch (dtype) { - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: return std::make_unique<GptDecoder<float>>( mode, maxNumSequences, maxBeamWidth, vocabSize, vocabSizePadded, stream, speculativeDecodingModule); - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: return std::make_unique<GptDecoder<half>>( mode, maxNumSequences, maxBeamWidth, vocabSize, vocabSizePadded, stream, speculativeDecodingModule); default: diff --git a/cpp/include/tensorrt_llm/runtime/gptDecoderBatched.h b/cpp/include/tensorrt_llm/runtime/gptDecoderBatched.h index d5447f441163..9fcd3262c8ca 100644 --- a/cpp/include/tensorrt_llm/runtime/gptDecoderBatched.h +++ b/cpp/include/tensorrt_llm/runtime/gptDecoderBatched.h @@ -16,7 +16,6 @@ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaEvent.h" #include "tensorrt_llm/runtime/cudaStream.h" @@ -49,7 +48,7 @@ class GptDecoderBatched : public IGptDecoderBatched explicit GptDecoderBatched(CudaStreamPtr stream); void setup(executor::DecodingMode const& mode, SizeType32 maxNumSequences, SizeType32 maxBeamWidth, - tensorrt_llm::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig) override; + nvinfer1::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig) override; void disableLookahead(RequestVector const& genRequests, TensorPtr const& batchSlots) override; diff --git a/cpp/include/tensorrt_llm/runtime/iBuffer.h b/cpp/include/tensorrt_llm/runtime/iBuffer.h index bf63d3a0da7b..91d5cd739f32 100644 --- a/cpp/include/tensorrt_llm/runtime/iBuffer.h +++ b/cpp/include/tensorrt_llm/runtime/iBuffer.h @@ -22,7 +22,7 @@ #include "tensorrt_llm/kernels/kvCacheIndex.h" #include "tensorrt_llm/runtime/common.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cstdint> #ifdef ENABLE_FP8 @@ -88,13 +88,13 @@ struct MemoryTypeString<MemoryType::kPINNEDPOOL> }; //! \brief For converting a TensorRT data type to a C++ data type. -template <tensorrt_llm::DataType kDataType, bool kIsUnsigned = false, bool kIsPointer = false> +template <nvinfer1::DataType kDataType, bool kIsUnsigned = false, bool kIsPointer = false> struct DataTypeTraits { }; template <> -struct DataTypeTraits<tensorrt_llm::DataType::kFLOAT> +struct DataTypeTraits<nvinfer1::DataType::kFLOAT> { using type = float; static char constexpr name[] = "float"; @@ -102,7 +102,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kFLOAT> }; template <> -struct DataTypeTraits<tensorrt_llm::DataType::kHALF> +struct DataTypeTraits<nvinfer1::DataType::kHALF> { using type = half; static char constexpr name[] = "half"; @@ -110,7 +110,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kHALF> }; template <> -struct DataTypeTraits<tensorrt_llm::DataType::kINT8> +struct DataTypeTraits<nvinfer1::DataType::kINT8> { using type = std::int8_t; static char constexpr name[] = "int8"; @@ -118,7 +118,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kINT8> }; template <> -struct DataTypeTraits<tensorrt_llm::DataType::kINT32> +struct DataTypeTraits<nvinfer1::DataType::kINT32> { using type = std::int32_t; static char constexpr name[] = "int32"; @@ -126,7 +126,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kINT32> }; template <> -struct DataTypeTraits<tensorrt_llm::DataType::kINT64> +struct DataTypeTraits<nvinfer1::DataType::kINT64> { using type = std::int64_t; static char constexpr name[] = "int64"; @@ -134,7 +134,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kINT64> }; template <> -struct DataTypeTraits<tensorrt_llm::DataType::kINT32, true> +struct DataTypeTraits<nvinfer1::DataType::kINT32, true> { using type = std::uint32_t; static char constexpr name[] = "uint32"; @@ -142,7 +142,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kINT32, true> }; template <> -struct DataTypeTraits<tensorrt_llm::DataType::kINT64, true> +struct DataTypeTraits<nvinfer1::DataType::kINT64, true> { using type = std::uint64_t; static char constexpr name[] = "uint64"; @@ -150,7 +150,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kINT64, true> }; template <bool kUnsigned> -struct DataTypeTraits<tensorrt_llm::DataType::kBOOL, kUnsigned> +struct DataTypeTraits<nvinfer1::DataType::kBOOL, kUnsigned> { using type = bool; static char constexpr name[] = "bool"; @@ -158,7 +158,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kBOOL, kUnsigned> }; template <bool kUnsigned> -struct DataTypeTraits<tensorrt_llm::DataType::kUINT8, kUnsigned> +struct DataTypeTraits<nvinfer1::DataType::kUINT8, kUnsigned> { using type = std::uint8_t; static char constexpr name[] = "uint8"; @@ -167,7 +167,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kUINT8, kUnsigned> #ifdef ENABLE_BF16 template <> -struct DataTypeTraits<tensorrt_llm::DataType::kBF16> +struct DataTypeTraits<nvinfer1::DataType::kBF16> { using type = __nv_bfloat16; static char constexpr name[] = "bfloat16"; @@ -177,7 +177,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kBF16> #ifdef ENABLE_FP8 template <> -struct DataTypeTraits<tensorrt_llm::DataType::kFP8> +struct DataTypeTraits<nvinfer1::DataType::kFP8> { using type = __nv_fp8_e4m3; static char constexpr name[] = "fp8"; @@ -185,7 +185,7 @@ struct DataTypeTraits<tensorrt_llm::DataType::kFP8> }; #endif -template <tensorrt_llm::DataType kDataType, bool kUnsigned> +template <nvinfer1::DataType kDataType, bool kUnsigned> struct DataTypeTraits<kDataType, kUnsigned, true> { using type = typename DataTypeTraits<kDataType, kUnsigned, false>::type*; @@ -193,26 +193,26 @@ struct DataTypeTraits<kDataType, kUnsigned, true> static auto constexpr size = sizeof(type); }; -//! \brief A wrapper around `tensorrt_llm::DataType` that provides a support for pointer types. +//! \brief A wrapper around `nvinfer1::DataType` that provides a support for pointer types. class BufferDataType { public: constexpr BufferDataType( // NOLINT(*-explicit-constructor) - tensorrt_llm::DataType dataType, bool _unsigned = false, bool pointer = false) + nvinfer1::DataType dataType, bool _unsigned = false, bool pointer = false) : mDataType{dataType} , mUnsigned{_unsigned} , mPointer{pointer} { } - static auto constexpr kTrtPointerType = tensorrt_llm::DataType::kINT64; + static auto constexpr kTrtPointerType = nvinfer1::DataType::kINT64; - constexpr operator tensorrt_llm::DataType() const noexcept // NOLINT(*-explicit-constructor) + constexpr operator nvinfer1::DataType() const noexcept // NOLINT(*-explicit-constructor) { return mPointer ? kTrtPointerType : mDataType; } - [[nodiscard]] constexpr tensorrt_llm::DataType getDataType() const noexcept + [[nodiscard]] constexpr nvinfer1::DataType getDataType() const noexcept { return mDataType; } @@ -226,24 +226,24 @@ class BufferDataType { switch (mDataType) { - case tensorrt_llm::DataType::kBOOL: [[fallthrough]]; - case tensorrt_llm::DataType::kUINT8: return true; + case nvinfer1::DataType::kBOOL: [[fallthrough]]; + case nvinfer1::DataType::kUINT8: return true; default: return mUnsigned; } } [[nodiscard]] constexpr std::size_t getSize() const noexcept { - return tensorrt_llm::common::getDTypeSize(static_cast<tensorrt_llm::DataType>(*this)); + return tensorrt_llm::common::getDTypeSize(static_cast<nvinfer1::DataType>(*this)); } [[nodiscard]] constexpr std::size_t getSizeInBits() const noexcept { - return tensorrt_llm::common::getDTypeSizeInBits(static_cast<tensorrt_llm::DataType>(*this)); + return tensorrt_llm::common::getDTypeSizeInBits(static_cast<nvinfer1::DataType>(*this)); } private: - tensorrt_llm::DataType mDataType; + nvinfer1::DataType mDataType; bool mUnsigned; bool mPointer; }; @@ -257,62 +257,62 @@ struct TRTDataType template <> struct TRTDataType<float> { - static constexpr auto value = tensorrt_llm::DataType::kFLOAT; + static constexpr auto value = nvinfer1::DataType::kFLOAT; }; template <> struct TRTDataType<half> { - static constexpr auto value = tensorrt_llm::DataType::kHALF; + static constexpr auto value = nvinfer1::DataType::kHALF; }; template <> struct TRTDataType<std::int8_t> { - static constexpr auto value = tensorrt_llm::DataType::kINT8; + static constexpr auto value = nvinfer1::DataType::kINT8; }; template <> struct TRTDataType<std::int32_t> { - static constexpr auto value = tensorrt_llm::DataType::kINT32; + static constexpr auto value = nvinfer1::DataType::kINT32; }; template <> struct TRTDataType<std::uint32_t> { - static constexpr auto value = BufferDataType{tensorrt_llm::DataType::kINT32, true}; + static constexpr auto value = BufferDataType{nvinfer1::DataType::kINT32, true}; }; template <> struct TRTDataType<std::int64_t> { - static constexpr auto value = tensorrt_llm::DataType::kINT64; + static constexpr auto value = nvinfer1::DataType::kINT64; }; template <> struct TRTDataType<std::uint64_t> { - static constexpr auto value = BufferDataType{tensorrt_llm::DataType::kINT64, true}; + static constexpr auto value = BufferDataType{nvinfer1::DataType::kINT64, true}; }; template <> struct TRTDataType<bool> { - static constexpr auto value = tensorrt_llm::DataType::kBOOL; + static constexpr auto value = nvinfer1::DataType::kBOOL; }; template <> struct TRTDataType<std::uint8_t> { - static constexpr auto value = tensorrt_llm::DataType::kUINT8; + static constexpr auto value = nvinfer1::DataType::kUINT8; }; #ifdef ENABLE_BF16 template <> struct TRTDataType<__nv_bfloat16> { - static constexpr auto value = tensorrt_llm::DataType::kBF16; + static constexpr auto value = nvinfer1::DataType::kBF16; }; #endif @@ -320,7 +320,7 @@ struct TRTDataType<__nv_bfloat16> template <> struct TRTDataType<__nv_fp8_e4m3> { - static constexpr auto value = tensorrt_llm::DataType::kFP8; + static constexpr auto value = nvinfer1::DataType::kFP8; }; #endif @@ -380,7 +380,7 @@ class IBuffer using SharedPtr = std::shared_ptr<IBuffer>; using UniqueConstPtr = std::unique_ptr<IBuffer const>; using SharedConstPtr = std::shared_ptr<IBuffer const>; - using DataType = tensorrt_llm::DataType; + using DataType = nvinfer1::DataType; //! //! \brief Returns a pointer to underlying array. diff --git a/cpp/include/tensorrt_llm/runtime/iGptDecoderBatched.h b/cpp/include/tensorrt_llm/runtime/iGptDecoderBatched.h index b664bc007f0e..ab55b754f9be 100644 --- a/cpp/include/tensorrt_llm/runtime/iGptDecoderBatched.h +++ b/cpp/include/tensorrt_llm/runtime/iGptDecoderBatched.h @@ -16,7 +16,6 @@ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/cudaEvent.h" #include "tensorrt_llm/runtime/cudaStream.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -52,7 +51,7 @@ class IGptDecoderBatched //! @brief Setup the decoder before calling `forward()` virtual void setup(executor::DecodingMode const& mode, SizeType32 maxNumSequences, SizeType32 maxBeamWidth, - tensorrt_llm::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig) + nvinfer1::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig) = 0; //! @brief Disable Lookahead decoding. diff --git a/cpp/include/tensorrt_llm/runtime/iTensor.h b/cpp/include/tensorrt_llm/runtime/iTensor.h index a85291dd8263..eb5c10eeb691 100644 --- a/cpp/include/tensorrt_llm/runtime/iTensor.h +++ b/cpp/include/tensorrt_llm/runtime/iTensor.h @@ -20,7 +20,7 @@ #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/iBuffer.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <algorithm> #include <cstdint> @@ -33,6 +33,11 @@ #include <string> #include <type_traits> +namespace nvinfer1 +{ +class IExecutionContext; +} + namespace tensorrt_llm::runtime { @@ -45,7 +50,7 @@ class ITensor : virtual public IBuffer using SharedPtr = std::shared_ptr<ITensor>; using UniqueConstPtr = std::unique_ptr<ITensor const>; using SharedConstPtr = std::shared_ptr<ITensor const>; - using Shape = tensorrt_llm::Dims; + using Shape = nvinfer1::Dims; using DimType64 = std::remove_reference_t<decltype(Shape::d[0])>; using TensorMap = runtime::StringPtrMap<runtime::ITensor>; @@ -347,9 +352,9 @@ class ITensor : virtual public IBuffer //! \param shape The shape of the tensor. //! \param capacity The capacity of the buffer. //! \return An `ITensor`. - static UniquePtr wrap(void* data, tensorrt_llm::DataType type, Shape const& shape, std::size_t capacity); + static UniquePtr wrap(void* data, nvinfer1::DataType type, Shape const& shape, std::size_t capacity); - static UniquePtr wrap(void* data, tensorrt_llm::DataType type, Shape const& shape) + static UniquePtr wrap(void* data, nvinfer1::DataType type, Shape const& shape) { return wrap(data, type, shape, volumeNonNegative(shape)); } diff --git a/cpp/include/tensorrt_llm/runtime/lookaheadBuffers.h b/cpp/include/tensorrt_llm/runtime/lookaheadBuffers.h index 26c6e3886be4..ecaa439f2d52 100644 --- a/cpp/include/tensorrt_llm/runtime/lookaheadBuffers.h +++ b/cpp/include/tensorrt_llm/runtime/lookaheadBuffers.h @@ -17,9 +17,9 @@ #pragma once #include "tensorrt_llm/executor/executor.h" -#include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" #include "tensorrt_llm/runtime/worldConfig.h" namespace tensorrt_llm::runtime @@ -37,4 +37,47 @@ class LookaheadDecodingBuffers TensorPtr positionIds; }; +class LookaheadRuntimeBuffers +{ +public: + using TensorPtr = ITensor::SharedPtr; + using TensorMap = StringPtrMap<ITensor>; + + LookaheadRuntimeBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, BufferManager const& manager, + ModelConfig const& modelConfig, WorldConfig const& worldConfig, executor::DecodingConfig const& decodingConfig, + TllmRuntime const& runtime); + + void setFromInputs(SizeType32 numCtxSequences, SizeType32 numGenSequences, ITensor const& requestTypes, + ITensor const& seqSlots, LookaheadDecodingBuffers const& decoderLookaheadBuffers, TllmRuntime const& runtime, + ModelConfig const& modelConfig, WorldConfig const& worldConfig) const; + + void reshape(SizeType32 numCtxSequences, SizeType32 numGenSequences, SizeType32 tokensPerStep); + + void insertInputTensors(TensorMap& inputBuffers, TensorMap& outputBuffers, WorldConfig const& worldConfig) const; + + void enableLookaheadDecoding(SizeType32 maxBatchSize, SizeType32 tokensPerStep); + + void disableLookaheadDecoding(); + +public: + TensorPtr cumSumLength; // [1] the cumulative sum of generation length, on pinned + TensorPtr packedMasksDevice; // [forwardBatchSize, tokensPerStep, numPackedMasks], on gpu + TensorPtr generationLengthsDevice; // [forwardBatchSize], on gpu + TensorPtr positionOffsetsDevice; // [forwardBatchSize, tokensPerStep], on gpu + TensorPtr positionIdsDevice; // [forwardBatchSize, tokensPerStep], on gpu + + TensorPtr packedMaskHost; + TensorPtr generationLengthsHost; + TensorPtr positionOffsetsHost; + TensorPtr positionIdsHost; + + TensorPtr packedMaskHostCopy; + TensorPtr generationLengthsHostCopy; + TensorPtr positionOffsetsHostCopy; + TensorPtr positionIdsHostCopy; + TensorPtr useSpecDecoding; + + TensorPtr batchSlotsHostCopy; +}; + } // namespace tensorrt_llm::runtime diff --git a/cpp/include/tensorrt_llm/runtime/loraCache.h b/cpp/include/tensorrt_llm/runtime/loraCache.h index eb4ef57494ea..1d242cdc80c5 100644 --- a/cpp/include/tensorrt_llm/runtime/loraCache.h +++ b/cpp/include/tensorrt_llm/runtime/loraCache.h @@ -25,6 +25,8 @@ #include "tensorrt_llm/runtime/modelConfig.h" #include "tensorrt_llm/runtime/worldConfig.h" +#include <NvInferRuntime.h> + #include <deque> #include <list> #include <map> diff --git a/cpp/include/tensorrt_llm/runtime/loraCachePageManagerConfig.h b/cpp/include/tensorrt_llm/runtime/loraCachePageManagerConfig.h index cf1b1a6aac18..a995304e94ec 100644 --- a/cpp/include/tensorrt_llm/runtime/loraCachePageManagerConfig.h +++ b/cpp/include/tensorrt_llm/runtime/loraCachePageManagerConfig.h @@ -20,7 +20,7 @@ #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/iBuffer.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <ostream> #include <sstream> @@ -36,7 +36,7 @@ namespace tensorrt_llm::runtime class LoraCachePageManagerConfig { public: - explicit constexpr LoraCachePageManagerConfig(runtime::MemoryType memType, tensorrt_llm::DataType dType, + explicit constexpr LoraCachePageManagerConfig(runtime::MemoryType memType, nvinfer1::DataType dType, SizeType32 totalNumPages, SizeType32 maxPagesPerBlock, SizeType32 slotsPerPage, SizeType32 pageWidth, SizeType32 numCopyStreams) : mMemoryType(memType) @@ -59,12 +59,12 @@ class LoraCachePageManagerConfig mMemoryType = memoryType; } - [[nodiscard]] tensorrt_llm::DataType constexpr getDataType() const noexcept + [[nodiscard]] nvinfer1::DataType constexpr getDataType() const noexcept { return mDataType; } - void constexpr setDataType(tensorrt_llm::DataType const& dtype) noexcept + void constexpr setDataType(nvinfer1::DataType const& dtype) noexcept { mDataType = dtype; } @@ -131,7 +131,7 @@ class LoraCachePageManagerConfig private: runtime::MemoryType mMemoryType; - tensorrt_llm::DataType mDataType; + nvinfer1::DataType mDataType; /* * Number cache pages in the cache. @@ -154,7 +154,7 @@ inline std::ostream& operator<<(std::ostream& os, LoraCachePageManagerConfig con { os << "{" << "memoryType=" << static_cast<typename std::underlying_type<runtime::MemoryType>::type>(c.getMemoryType()) - << " dataType=" << static_cast<typename std::underlying_type<tensorrt_llm::DataType>::type>(c.getDataType()) + << " dataType=" << static_cast<typename std::underlying_type<nvinfer1::DataType>::type>(c.getDataType()) << " totalNumPages=" << c.getTotalNumPages() << " maxPagesPerBlock=" << c.getMaxPagesPerBlock() << " slotsPerPage=" << c.getSlotsPerPage() << " pageWidth=" << c.getPageWidth() << " initToZero=" << c.getInitToZero() << "}"; diff --git a/cpp/include/tensorrt_llm/runtime/modelConfig.h b/cpp/include/tensorrt_llm/runtime/modelConfig.h index b5f18da07f3b..5bfe7bce9d58 100644 --- a/cpp/include/tensorrt_llm/runtime/modelConfig.h +++ b/cpp/include/tensorrt_llm/runtime/modelConfig.h @@ -23,7 +23,7 @@ #include "tensorrt_llm/runtime/speculativeDecodingMode.h" #include "tensorrt_llm/runtime/speculativeDecodingModule.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <array> namespace tensorrt_llm::runtime @@ -101,7 +101,7 @@ class ModelConfig }; explicit ModelConfig(SizeType32 vocabSize, SizeType32 nbLayers, SizeType32 nbAttentionLayers, - SizeType32 nbRnnLayers, SizeType32 nbHeads, SizeType32 hiddenSize, tensorrt_llm::DataType dtype) + SizeType32 nbRnnLayers, SizeType32 nbHeads, SizeType32 hiddenSize, nvinfer1::DataType dtype) : mVocabSize(vocabSize) , mNbLayers(nbLayers) , mNbAttentionLayers(nbAttentionLayers) @@ -137,7 +137,7 @@ class ModelConfig , mUsePositionEmbedding(false) , mUseTokenTypeEmbedding(false) , mSpeculativeDecodingMode(SpeculativeDecodingMode::None()) - , mLogitsDtype(tensorrt_llm::DataType::kFLOAT) + , mLogitsDtype(nvinfer1::DataType::kFLOAT) , mUseShapeInference(true) , mManageWeightsType(ManageWeightsType::kDisabled) , mSkipCrossAttnBlocks(false) @@ -331,7 +331,7 @@ class ModelConfig mSizePerHead = sizePerHead; } - [[nodiscard]] tensorrt_llm::DataType constexpr getDataType() const noexcept + [[nodiscard]] nvinfer1::DataType constexpr getDataType() const noexcept { return mDataType; } @@ -735,20 +735,20 @@ class ModelConfig resetSpeculativeDecodingModule(); } - [[nodiscard]] tensorrt_llm::DataType getKvDataType() const + [[nodiscard]] nvinfer1::DataType getKvDataType() const { if (getQuantMode().hasFp8KvCache()) { - return tensorrt_llm::DataType::kFP8; + return nvinfer1::DataType::kFP8; } if (getQuantMode().hasInt8KvCache()) { - return tensorrt_llm::DataType::kINT8; + return nvinfer1::DataType::kINT8; } else if (getQuantMode().hasFp4KvCache()) { #ifdef ENABLE_FP4 - return tensorrt_llm::DataType::kFP4; + return nvinfer1::DataType::kFP4; #else throw std::runtime_error("Model has FP4 KV cache, but TRT-LLM was not compiled with FP4 enabled."); #endif @@ -800,22 +800,22 @@ class ModelConfig return mSpeculativeDecodingMode; } - void setLogitsDtype(tensorrt_llm::DataType inputDtype) noexcept + void setLogitsDtype(nvinfer1::DataType inputDtype) noexcept { mLogitsDtype = inputDtype; } - [[nodiscard]] tensorrt_llm::DataType constexpr getLogitsDtype() const noexcept + [[nodiscard]] nvinfer1::DataType constexpr getLogitsDtype() const noexcept { return mLogitsDtype; } - void setGemmAllReduceDtype(tensorrt_llm::DataType inputDtype) noexcept + void setGemmAllReduceDtype(nvinfer1::DataType inputDtype) noexcept { mGemmAllReduceDtype = inputDtype; } - [[nodiscard]] tensorrt_llm::DataType constexpr getGemmAllReduceDtype() const noexcept + [[nodiscard]] nvinfer1::DataType constexpr getGemmAllReduceDtype() const noexcept { return mGemmAllReduceDtype; } @@ -945,10 +945,10 @@ class ModelConfig SizeType32 mNbHeads; SizeType32 mHiddenSize; SizeType32 mSizePerHead; - tensorrt_llm::DataType mDataType; + nvinfer1::DataType mDataType; bool mUseGptAttentionPlugin; bool mUseGemmAllReducePlugin; - tensorrt_llm::DataType mGemmAllReduceDtype; + nvinfer1::DataType mGemmAllReduceDtype; bool mUseMambaConv1dPlugin; bool mInputPacked; bool mPagedState; @@ -998,7 +998,7 @@ class ModelConfig SpeculativeDecodingMode mSpeculativeDecodingMode; // Logits datatype - tensorrt_llm::DataType mLogitsDtype; + nvinfer1::DataType mLogitsDtype; bool mUseShapeInference; ManageWeightsType mManageWeightsType; std::string mModelName; diff --git a/cpp/include/tensorrt_llm/runtime/rawEngine.h b/cpp/include/tensorrt_llm/runtime/rawEngine.h new file mode 100644 index 000000000000..b219cbe03382 --- /dev/null +++ b/cpp/include/tensorrt_llm/runtime/rawEngine.h @@ -0,0 +1,123 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/executor/tensor.h" + +#include <NvInferRuntime.h> +#include <filesystem> +#include <map> +#include <optional> + +namespace tensorrt_llm::runtime +{ + +class RawEngine +{ +public: + enum Type + { + FilePath, + AddressWithSize, + HostMemory + }; + + explicit RawEngine(std::filesystem::path enginePath) noexcept + : mType(FilePath) + , mEnginePath(std::move(enginePath)) + { + } + + explicit RawEngine(void const* engineAddr, std::size_t engineSize) noexcept + : mType(AddressWithSize) + , mEngineAddr(engineAddr) + , mEngineSize(engineSize) + { + } + + explicit RawEngine(nvinfer1::IHostMemory const* engineBuffer) noexcept + : mType(HostMemory) + , mEngineBuffer(engineBuffer) + { + } + + [[nodiscard]] Type getType() const + { + return mType; + } + + [[nodiscard]] std::filesystem::path getPath() const + { + TLLM_CHECK(mEnginePath.has_value()); + return mEnginePath.value(); + } + + [[nodiscard]] std::optional<std::filesystem::path> getPathOpt() const + { + return mEnginePath; + } + + void setPath(std::filesystem::path enginePath) + { + mEnginePath = std::move(enginePath); + } + + [[nodiscard]] std::optional<std::map<std::string, tensorrt_llm::executor::Tensor>> const& + getManagedWeightsMapOpt() const + { + return mManagedWeightsMap; + } + + void setManagedWeightsMap(std::map<std::string, tensorrt_llm::executor::Tensor> managedWeightsMap) + { + mManagedWeightsMap = std::move(managedWeightsMap); + } + + [[nodiscard]] void const* getAddress() const + { + TLLM_CHECK(mType == AddressWithSize); + return mEngineAddr; + } + + [[nodiscard]] std::size_t getSize() const + { + TLLM_CHECK(mType == AddressWithSize); + return mEngineSize; + } + + [[nodiscard]] nvinfer1::IHostMemory const* getHostMemory() const + { + TLLM_CHECK(mType == HostMemory); + return mEngineBuffer; + } + +private: + Type mType; + std::optional<std::filesystem::path> mEnginePath; + + struct + { + void const* mEngineAddr{}; + std::size_t mEngineSize{}; + }; + + nvinfer1::IHostMemory const* mEngineBuffer{}; + std::optional<std::map<std::string, tensorrt_llm::executor::Tensor>> mManagedWeightsMap; +}; + +} // namespace tensorrt_llm::runtime diff --git a/cpp/include/tensorrt_llm/runtime/tllmLogger.h b/cpp/include/tensorrt_llm/runtime/tllmLogger.h new file mode 100644 index 000000000000..dd3806ec5242 --- /dev/null +++ b/cpp/include/tensorrt_llm/runtime/tllmLogger.h @@ -0,0 +1,34 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include <NvInferRuntime.h> + +namespace tensorrt_llm::runtime +{ + +class TllmLogger : public nvinfer1::ILogger +{ +public: + void log(Severity severity, nvinfer1::AsciiChar const* msg) noexcept override; + + Severity getLevel(); + + void setLevel(Severity level); +}; + +} // namespace tensorrt_llm::runtime diff --git a/cpp/include/tensorrt_llm/runtime/utils/debugUtils.h b/cpp/include/tensorrt_llm/runtime/utils/debugUtils.h index 3f6c307a3cde..68064d74c7e2 100644 --- a/cpp/include/tensorrt_llm/runtime/utils/debugUtils.h +++ b/cpp/include/tensorrt_llm/runtime/utils/debugUtils.h @@ -15,7 +15,6 @@ */ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" namespace tensorrt_llm::runtime::utils @@ -25,7 +24,7 @@ template <typename T> bool tensorHasInvalid(ITensor const& tensor, BufferManager const& manager, std::string const& infoStr); bool tensorHasInvalid( - size_t M, size_t K, tensorrt_llm::DataType type, void const* data, cudaStream_t stream, std::string const& infoStr); + size_t M, size_t K, nvinfer1::DataType type, void const* data, cudaStream_t stream, std::string const& infoStr); int stallStream( char const* name, std::optional<cudaStream_t> stream = std::nullopt, std::optional<int> delay = std::nullopt); diff --git a/cpp/include/tensorrt_llm/runtime/utils/pgUtils.h b/cpp/include/tensorrt_llm/runtime/utils/pgUtils.h index 049de46cb26f..32a3f1c86c9d 100644 --- a/cpp/include/tensorrt_llm/runtime/utils/pgUtils.h +++ b/cpp/include/tensorrt_llm/runtime/utils/pgUtils.h @@ -86,8 +86,6 @@ c10::intrusive_ptr<c10d::ProcessGroup> get_local_pg(); void init_pg(c10::intrusive_ptr<c10d::ProcessGroup> const& process_group_world, c10::intrusive_ptr<c10d::ProcessGroup> const& process_group_local); -void shutdown_pg(); - // Tensor wrapping utilities for ProcessGroup operations inline torch::Tensor wrap_tensor(torch::Tensor data) { diff --git a/cpp/include/tensorrt_llm/runtime/worldConfig.h b/cpp/include/tensorrt_llm/runtime/worldConfig.h index 272b0fec5ada..9ff2d0970df7 100644 --- a/cpp/include/tensorrt_llm/runtime/worldConfig.h +++ b/cpp/include/tensorrt_llm/runtime/worldConfig.h @@ -18,6 +18,7 @@ #include "tensorrt_llm/runtime/common.h" +#include <NvInferRuntime.h> #include <optional> #include <vector> diff --git a/cpp/micro_benchmarks/mixtureOfExpertsBackendBenchmarkFixture.h b/cpp/micro_benchmarks/mixtureOfExpertsBackendBenchmarkFixture.h index 354e11184f8d..f466a65e871f 100644 --- a/cpp/micro_benchmarks/mixtureOfExpertsBackendBenchmarkFixture.h +++ b/cpp/micro_benchmarks/mixtureOfExpertsBackendBenchmarkFixture.h @@ -31,7 +31,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/memoryUtils.h" #include "tensorrt_llm/common/nvtxUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/cutlass_kernels/cutlass_preprocessors.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaStream.h" @@ -381,44 +380,44 @@ class MixtureOfExpertsBenchmark : public ::benchmark::Fixture int64_t mNumExpertsPerNode{}; int64_t mK{}; - constexpr static tensorrt_llm::DataType toDTypeID() + constexpr static nvinfer1::DataType toDTypeID() { if (FP8 || WFP4AFP8) - return tensorrt_llm::DataType::kFP8; + return nvinfer1::DataType::kFP8; if (NVFP4) - return tensorrt_llm::DataType::kFP4; + return nvinfer1::DataType::kFP4; if (INT_QUANT && INT4) - return tensorrt_llm::DataType::kINT4; + return nvinfer1::DataType::kINT4; if (INT_QUANT) - return tensorrt_llm::DataType::kINT8; + return nvinfer1::DataType::kINT8; if (std::is_same_v<DataType, float>) - return tensorrt_llm::DataType::kFLOAT; + return nvinfer1::DataType::kFLOAT; if (std::is_same_v<DataType, half>) - return tensorrt_llm::DataType::kHALF; + return nvinfer1::DataType::kHALF; #ifdef ENABLE_BF16 if (std::is_same_v<DataType, nv_bfloat16>) - return tensorrt_llm::DataType::kBF16; + return nvinfer1::DataType::kBF16; #endif TLLM_THROW("Unrecognised format"); }; - constexpr static tensorrt_llm::DataType toWTypeID() + constexpr static nvinfer1::DataType toWTypeID() { if (FP8) - return tensorrt_llm::DataType::kFP8; + return nvinfer1::DataType::kFP8; if (NVFP4 || WFP4AFP8) - return tensorrt_llm::DataType::kFP4; + return nvinfer1::DataType::kFP4; if (INT_QUANT && INT4) - return tensorrt_llm::DataType::kINT4; + return nvinfer1::DataType::kINT4; if (INT_QUANT) - return tensorrt_llm::DataType::kINT8; + return nvinfer1::DataType::kINT8; if (std::is_same_v<DataType, float>) - return tensorrt_llm::DataType::kFLOAT; + return nvinfer1::DataType::kFLOAT; if (std::is_same_v<DataType, half>) - return tensorrt_llm::DataType::kHALF; + return nvinfer1::DataType::kHALF; #ifdef ENABLE_BF16 if (std::is_same_v<DataType, nv_bfloat16>) - return tensorrt_llm::DataType::kBF16; + return nvinfer1::DataType::kBF16; #endif TLLM_THROW("Unrecognised format"); }; @@ -428,31 +427,31 @@ class MixtureOfExpertsBenchmark : public ::benchmark::Fixture { if constexpr (std::is_same_v<T, SafeFP8>) { - return tensorrt_llm::DataType::kFP8; + return nvinfer1::DataType::kFP8; } else if constexpr (std::is_same_v<T, SafeFP4>) { - return tensorrt_llm::DataType::kFP4; + return nvinfer1::DataType::kFP4; } else if constexpr (std::is_same_v<T, uint8_t>) { - return tensorrt_llm::DataType::kINT8; + return nvinfer1::DataType::kINT8; } else if constexpr (std::is_same_v<T, cutlass::uint4b_t>) { - return tensorrt_llm::DataType::kINT4; + return nvinfer1::DataType::kINT4; } else if constexpr (std::is_same_v<T, nv_bfloat16>) { - return tensorrt_llm::DataType::kBF16; + return nvinfer1::DataType::kBF16; } else if constexpr (std::is_same_v<T, half>) { - return tensorrt_llm::DataType::kHALF; + return nvinfer1::DataType::kHALF; } else if constexpr (std::is_same_v<T, float>) { - return tensorrt_llm::DataType::kFLOAT; + return nvinfer1::DataType::kFLOAT; } else { diff --git a/cpp/tensorrt_llm/CMakeLists.txt b/cpp/tensorrt_llm/CMakeLists.txt index 5f5e37836a05..afd2d3a1f415 100644 --- a/cpp/tensorrt_llm/CMakeLists.txt +++ b/cpp/tensorrt_llm/CMakeLists.txt @@ -144,6 +144,8 @@ add_subdirectory(common) add_subdirectory(kernels) add_subdirectory(layers) add_subdirectory(runtime) +add_subdirectory(testing) +add_subdirectory(executor_worker) set(BATCH_MANAGER_TARGET tensorrt_llm_batch_manager_static) set(BATCH_MANAGER_TARGET_ARCH ${TARGET_ARCH}) @@ -174,6 +176,7 @@ set(TRTLLM_LINK_LIBS ${CUBLASLT_LIB} ${CURAND_LIB} ${CMAKE_DL_LIBS} + ${TRT_LIB} common_src kernels_src flash_mla_src @@ -196,6 +199,7 @@ set(TRTLLM_LINK_LIBS cute_dsl_src layers_src runtime_src + testing_src compressorKernels_src mhcKernels_src userbuffers_src @@ -306,3 +310,5 @@ endif() if(BUILD_FLASH_MLA) add_subdirectory(flash_mla) endif() + +add_subdirectory(plugins) diff --git a/cpp/tensorrt_llm/batch_manager/CMakeLists.txt b/cpp/tensorrt_llm/batch_manager/CMakeLists.txt index c7fea62c023a..1389d0de068f 100644 --- a/cpp/tensorrt_llm/batch_manager/CMakeLists.txt +++ b/cpp/tensorrt_llm/batch_manager/CMakeLists.txt @@ -33,20 +33,34 @@ set(SRCS contextTransferCoordinator.cpp dataTransceiver.cpp decoderBuffers.cpp + encoderBuffers.cpp + guidedDecoder.cpp + handleContextLogits.cpp + handleGenerationLogits.cpp kvCacheManager.cpp kvCacheEventManager.cpp kvCacheTransferManager.cpp kvCacheManagerV2Utils.cpp kvCacheManagerV2Utils.cu llmRequest.cpp + logitsPostProcessor.cpp + loraBuffers.cpp + makeDecodingBatchInputOutput.cpp medusaBuffers.cpp microBatchScheduler.cpp pauseRequests.cpp peftCacheManager.cpp + promptTuningBuffers.cpp + rnnStateBuffers.cpp rnnStateManager.cpp rnnCacheFormatter.cpp rnnCacheTransBuffer.cpp + runtimeBuffers.cpp sequenceSlotManager.cpp + transformerBuffers.cpp + trtEncoderModel.cpp + trtGptModelInflightBatching.cpp + updateDecoderBuffers.cpp utils/debugUtils.cpp utils/inflightBatchingUtils.cpp utils/logitsThread.cpp diff --git a/cpp/tensorrt_llm/batch_manager/baseTransBuffer.cpp b/cpp/tensorrt_llm/batch_manager/baseTransBuffer.cpp index 1fb522afd7be..fe47f6c531ad 100644 --- a/cpp/tensorrt_llm/batch_manager/baseTransBuffer.cpp +++ b/cpp/tensorrt_llm/batch_manager/baseTransBuffer.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -20,62 +20,15 @@ #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/opUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" -#include <chrono> -#include <exception> #include <mutex> namespace tensorrt_llm::batch_manager { -namespace -{ - -char const* bufferKindName(BufferKind kind) -{ - switch (kind) - { - case BufferKind::kKV: return "kv"; - case BufferKind::kKV_INDEXER: return "kv_indexer"; - case BufferKind::kRNN: return "rnn"; - } - return "unknown"; -} - -} // namespace - void BufferIndexHolder::release() noexcept { - if (mMgr == nullptr) - { - return; - } - if (mHeld) - { - try - { - if (mIsRecv) - { - mMgr->freeBufferIndexForRecv(mIndex); - } - else - { - mMgr->freeBufferIndexForSend(mIndex); - } - } - catch (...) - { - // noexcept: swallow so the destructor can never throw. - } - } - mHeld = false; - mMgr = nullptr; -} - -void BufferIndexHolder::poison() noexcept -{ - if (mMgr == nullptr) + if (!mHeld || mMgr == nullptr) { return; } @@ -83,23 +36,22 @@ void BufferIndexHolder::poison() noexcept { if (mIsRecv) { - mMgr->poisonBufferIndexForRecv(mIndex); + mMgr->freeBufferIndexForRecv(mIndex); } else { - mMgr->poisonBufferIndexForSend(mIndex); + mMgr->freeBufferIndexForSend(mIndex); } } catch (...) { - // noexcept: poison is a fail-closed best effort from exception paths. + // noexcept: swallow so the destructor can never throw. } mHeld = false; - mMgr = nullptr; } BaseTransBufferManager::BaseTransBufferManager( - size_t transferBufferSize, tensorrt_llm::DataType dataType, std::optional<size_t> maxNumTokens) + size_t transferBufferSize, nvinfer1::DataType dataType, std::optional<size_t> maxNumTokens) : mDataType{dataType} , mBufferManager{std::make_shared<runtime::CudaStream>()} , mMaxNumTokens{maxNumTokens} @@ -126,11 +78,9 @@ BaseTransBufferManager::BaseTransBufferManager( allocateBuffer(); } -std::optional<int> BaseTransBufferManager::assignBufferIndexForSend( - std::atomic<bool> const* perRequestCancel, int64_t waitSliceMs) +std::optional<int> BaseTransBufferManager::assignBufferIndexForSend() { - return assignBufferIndex( - mConcurrenceSendResource, mSendBufferCount, mOnlyUseDynamicBuffer, perRequestCancel, waitSliceMs); + return assignBufferIndex(mConcurrenceSendResource, mSendBufferCount, mOnlyUseDynamicBuffer); } void BaseTransBufferManager::freeBufferIndexForSend(std::optional<int> bufferId) @@ -138,16 +88,9 @@ void BaseTransBufferManager::freeBufferIndexForSend(std::optional<int> bufferId) freeBufferIndex(mConcurrenceSendResource, bufferId, mSendBufferCount, mOnlyUseDynamicBuffer); } -void BaseTransBufferManager::poisonBufferIndexForSend(std::optional<int> bufferId) noexcept +std::optional<int> BaseTransBufferManager::assignBufferIndexForRecv() { - poisonBufferIndex(mConcurrenceSendResource, bufferId, mSendBufferCount, mOnlyUseDynamicBuffer, "send"); -} - -std::optional<int> BaseTransBufferManager::assignBufferIndexForRecv( - std::atomic<bool> const* perRequestCancel, int64_t waitSliceMs) -{ - return assignBufferIndex( - mConcurrenceRecvResource, mRecvBufferCount, mOnlyUseDynamicBuffer, perRequestCancel, waitSliceMs); + return assignBufferIndex(mConcurrenceRecvResource, mRecvBufferCount, mOnlyUseDynamicBuffer); } void BaseTransBufferManager::freeBufferIndexForRecv(std::optional<int> bufferId) @@ -155,11 +98,6 @@ void BaseTransBufferManager::freeBufferIndexForRecv(std::optional<int> bufferId) freeBufferIndex(mConcurrenceRecvResource, bufferId, mRecvBufferCount, mOnlyUseDynamicBuffer); } -void BaseTransBufferManager::poisonBufferIndexForRecv(std::optional<int> bufferId) noexcept -{ - poisonBufferIndex(mConcurrenceRecvResource, bufferId, mRecvBufferCount, mOnlyUseDynamicBuffer, "recv"); -} - std::tuple<std::vector<runtime::ITensor::SharedPtr>, size_t, bool> BaseTransBufferManager::getOrAllocateSendBuffers( std::optional<int> bufferId, int targetNum, std::vector<size_t> const& requestedNumberOfElements, runtime::BufferManager const& bufferManagerToUse) @@ -311,51 +249,16 @@ void BaseTransBufferManager::allocateBuffer() } } -std::optional<int> BaseTransBufferManager::assignBufferIndex(ConcurrenceResource& resource, size_t bufferCount, - bool onlyUseDynamicBuffer, std::atomic<bool> const* perRequestCancel, int64_t waitSliceMs) +std::optional<int> BaseTransBufferManager::assignBufferIndex( + ConcurrenceResource& resource, size_t bufferCount, bool onlyUseDynamicBuffer) { - auto const isCancelled = [perRequestCancel]() - { return perRequestCancel != nullptr && perRequestCancel->load(std::memory_order_relaxed); }; - if (isCancelled()) - { - TLLM_THROW("Cache transfer buffer acquisition cancelled"); - } if (onlyUseDynamicBuffer) { - TLLM_CHECK_WITH_INFO(!resource.mPoisoned.load(std::memory_order_relaxed), - "Cannot assign dynamic cache transfer buffer kind=%s because the transfer buffer pool is poisoned", - bufferKindName(getBufferKind())); return std::nullopt; } std::unique_lock lk(resource.mBuffersMutex); - auto const predicate = [&resource, bufferCount]() - { - return resource.mPoisoned.load(std::memory_order_relaxed) - || static_cast<size_t>(resource.mConcurrence) < bufferCount; - }; - if (perRequestCancel == nullptr) - { - resource.mBuffersCV.wait(lk, predicate); - } - else - { - auto const slice = std::chrono::milliseconds{waitSliceMs}; - while (!predicate()) - { - resource.mBuffersCV.wait_for(lk, slice); - if (isCancelled()) - { - TLLM_THROW("Cache transfer buffer acquisition cancelled"); - } - } - } - if (isCancelled()) - { - TLLM_THROW("Cache transfer buffer acquisition cancelled"); - } - TLLM_CHECK_WITH_INFO(!resource.mPoisoned.load(std::memory_order_relaxed), - "Cannot assign cache transfer buffer kind=%s because the transfer buffer pool is poisoned", - bufferKindName(getBufferKind())); + resource.mBuffersCV.wait( + lk, [&resource, bufferCount]() { return static_cast<size_t>(resource.mConcurrence) < bufferCount; }); int bufferId = -1; for (size_t i = 0; i < bufferCount; i++) { @@ -385,12 +288,6 @@ void BaseTransBufferManager::freeBufferIndex( TLLM_CHECK(static_cast<size_t>(bufferId.value()) < bufferCount); { std::scoped_lock lk(resource.mBuffersMutex); - if (resource.mBufferIndexFlag[bufferId.value()] == 2) - { - TLLM_LOG_ERROR("Refusing to free poisoned cache transfer buffer kind=%s index=%d", - bufferKindName(getBufferKind()), bufferId.value()); - return; - } resource.mBufferIndexFlag[bufferId.value()] = 0; } resource.mConcurrence--; @@ -398,48 +295,6 @@ void BaseTransBufferManager::freeBufferIndex( } } -void BaseTransBufferManager::poisonBufferIndex(ConcurrenceResource& resource, std::optional<int> bufferId, - size_t bufferCount, bool onlyUseDynamicBuffer, char const* direction) noexcept -{ - resource.mPoisoned.store(true, std::memory_order_relaxed); - - if (onlyUseDynamicBuffer) - { - TLLM_LOG_ERROR("Poisoned dynamic %s cache transfer buffer kind=%s; process restart is required", direction, - bufferKindName(getBufferKind())); - resource.mBuffersCV.notify_all(); - return; - } - - try - { - if (bufferId.has_value()) - { - TLLM_CHECK(static_cast<size_t>(bufferId.value()) < bufferCount); - { - std::scoped_lock lk(resource.mBuffersMutex); - if (resource.mBufferIndexFlag[bufferId.value()] == 1) - { - resource.mBufferIndexFlag[bufferId.value()] = 2; - } - } - } - TLLM_LOG_ERROR("Poisoned %s cache transfer buffer kind=%s index=%d; process restart is required", direction, - bufferKindName(getBufferKind()), bufferId.value_or(-1)); - } - catch (std::exception const& e) - { - TLLM_LOG_ERROR("Exception while poisoning %s cache transfer buffer kind=%s index=%d: %s", direction, - bufferKindName(getBufferKind()), bufferId.value_or(-1), e.what()); - } - catch (...) - { - TLLM_LOG_ERROR("Unknown exception while poisoning %s cache transfer buffer kind=%s index=%d", direction, - bufferKindName(getBufferKind()), bufferId.value_or(-1)); - } - resource.mBuffersCV.notify_all(); -} - size_t BaseTransBufferManager::getRecvBufferCount() { return mRecvBufferCount; diff --git a/cpp/tensorrt_llm/batch_manager/baseTransBuffer.h b/cpp/tensorrt_llm/batch_manager/baseTransBuffer.h index 88585818d4e5..2cbf9f514bd7 100644 --- a/cpp/tensorrt_llm/batch_manager/baseTransBuffer.h +++ b/cpp/tensorrt_llm/batch_manager/baseTransBuffer.h @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -17,7 +17,6 @@ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -50,9 +49,8 @@ enum class BufferKind : uint8_t class BaseTransBufferManager; /// @brief RAII holder for an index from BaseTransBufferManager::assignBufferIndexFor{Send,Recv}. -/// Releases on destruction (incl. exception unwind). A dynamic-buffer holder has no -/// concrete index, but retains its manager binding so poison() can fail closed. -/// Move-only; call release() on the happy path or detach() when ownership is handed off downstream. +/// Releases on destruction (incl. exception unwind). Move-only; call release() on +/// the happy path or detach() when ownership is handed off downstream. class BufferIndexHolder { public: @@ -84,7 +82,6 @@ class BufferIndexHolder , mIsRecv(other.mIsRecv) { other.mHeld = false; - other.mMgr = nullptr; } BufferIndexHolder& operator=(BufferIndexHolder&& other) noexcept @@ -97,7 +94,6 @@ class BufferIndexHolder mHeld = other.mHeld; mIsRecv = other.mIsRecv; other.mHeld = false; - other.mMgr = nullptr; } return *this; } @@ -112,27 +108,18 @@ class BufferIndexHolder return mHeld; } - [[nodiscard]] bool isBoundTo(BaseTransBufferManager const& manager) const noexcept - { - return mMgr == &manager; - } - /// @brief Relinquish ownership without releasing. Use when a downstream /// owner (e.g. the formatter inside receiveSync) takes over the /// release responsibility on the happy path. std::optional<int> detach() noexcept { mHeld = false; - mMgr = nullptr; return mIndex; } /// @brief Release the slot now and disarm the destructor. Safe to call multiple times. void release() noexcept; - /// @brief Fail-closed release for an exit path where transfer-buffer quiescence is unknown. - void poison() noexcept; - private: BaseTransBufferManager* mMgr{nullptr}; std::optional<int> mIndex{}; @@ -140,8 +127,6 @@ class BufferIndexHolder bool mIsRecv{true}; }; -inline constexpr int64_t kBufferAcquireSliceMs = 100; - /// @brief Base class for cache transfer buffer management. /// Handles buffer pool allocation, index assignment, and slicing. /// Derived classes provide cache-specific size calculations. @@ -154,28 +139,20 @@ class BaseTransBufferManager /// @brief Assign a buffer index for sending. /// @return Assigned buffer index, or nullopt if using dynamic buffers. - std::optional<int> assignBufferIndexForSend( - std::atomic<bool> const* perRequestCancel = nullptr, int64_t waitSliceMs = kBufferAcquireSliceMs); + std::optional<int> assignBufferIndexForSend(); /// @brief Free a buffer index used for sending. /// @param bufferId The buffer index to free. void freeBufferIndexForSend(std::optional<int> bufferId); - /// @brief Poison a send buffer index after an unquiesced transfer exit. - void poisonBufferIndexForSend(std::optional<int> bufferId) noexcept; - /// @brief Assign a buffer index for receiving. /// @return Assigned buffer index, or nullopt if using dynamic buffers. - std::optional<int> assignBufferIndexForRecv( - std::atomic<bool> const* perRequestCancel = nullptr, int64_t waitSliceMs = kBufferAcquireSliceMs); + std::optional<int> assignBufferIndexForRecv(); /// @brief Free a buffer index used for receiving. /// @param bufferId The buffer index to free. void freeBufferIndexForRecv(std::optional<int> bufferId); - /// @brief Poison a receive buffer index after an unquiesced transfer exit. - void poisonBufferIndexForRecv(std::optional<int> bufferId) noexcept; - /// @brief Get or allocate send buffers for cache transfer. /// @param bufferId The assigned buffer ID. /// @param targetNum Number of target sequences. @@ -214,19 +191,13 @@ class BaseTransBufferManager return mMaxNumTokens; } - [[nodiscard]] bool hasPoisonedBuffer() const noexcept - { - return mConcurrenceSendResource.mPoisoned.load(std::memory_order_relaxed) - || mConcurrenceRecvResource.mPoisoned.load(std::memory_order_relaxed); - } - protected: /// @brief Constructor - derived classes call this after computing buffer sizes. /// @param transferBufferSize Size of each transfer buffer in bytes. /// @param dataType Data type for the buffers. /// @param maxNumTokens Optional max tokens for sizing. BaseTransBufferManager( - size_t transferBufferSize, tensorrt_llm::DataType dataType, std::optional<size_t> maxNumTokens = std::nullopt); + size_t transferBufferSize, nvinfer1::DataType dataType, std::optional<size_t> maxNumTokens = std::nullopt); struct ConcurrenceResource { @@ -235,7 +206,6 @@ class BaseTransBufferManager std::mutex mBuffersMutex; std::condition_variable mBuffersCV; std::atomic<int> mConcurrence{0}; - std::atomic<bool> mPoisoned{false}; }; std::tuple<std::vector<runtime::ITensor::SharedPtr>, size_t, bool> getOrAllocateBuffers(std::optional<int> bufferId, @@ -243,12 +213,9 @@ class BaseTransBufferManager runtime::BufferManager const& bufferManagerToUse, ConcurrenceResource& concurrenceResource); void allocateBuffer(); - std::optional<int> assignBufferIndex(ConcurrenceResource& resource, size_t bufferCount, bool onlyUseDynamicBuffer, - std::atomic<bool> const* perRequestCancel = nullptr, int64_t waitSliceMs = kBufferAcquireSliceMs); + std::optional<int> assignBufferIndex(ConcurrenceResource& resource, size_t bufferCount, bool onlyUseDynamicBuffer); void freeBufferIndex( ConcurrenceResource& resource, std::optional<int> bufferId, size_t bufferCount, bool onlyUseDynamicBuffer); - void poisonBufferIndex(ConcurrenceResource& resource, std::optional<int> bufferId, size_t bufferCount, - bool onlyUseDynamicBuffer, char const* direction) noexcept; size_t mPreAllocBufferSize; size_t mRecvBufferCount; @@ -257,7 +224,7 @@ class BaseTransBufferManager bool mOnlyUseDynamicBuffer; bool mUseFabricMemory; size_t mNumberOfElements; - tensorrt_llm::DataType mDataType; + nvinfer1::DataType mDataType; ConcurrenceResource mConcurrenceSendResource; ConcurrenceResource mConcurrenceRecvResource; runtime::BufferManager mBufferManager; diff --git a/cpp/tensorrt_llm/batch_manager/cacheFormatter.cpp b/cpp/tensorrt_llm/batch_manager/cacheFormatter.cpp index 39cd7cc0f8de..e9daca6f5447 100644 --- a/cpp/tensorrt_llm/batch_manager/cacheFormatter.cpp +++ b/cpp/tensorrt_llm/batch_manager/cacheFormatter.cpp @@ -183,7 +183,7 @@ void sendAllBuffers(TransferSession& session, int deviceId, namespace tensorrt_llm::batch_manager::kv_cache_manager { -BlockRange getBlockRangeForSending(BaseKVCacheManager* cacheManager, std::optional<LlmRequest const*> llmRequest, +BlockRange getBlockRangeForSending(BaseKVCacheManager* cacheManager, LlmRequest const& llmRequest, BlockKey const& lastBlockKey, int32_t indexFromEnd, bool recvSideHasCP, SizeType32 ppSize) { auto poolNum = cacheManager->getBlockManager().getNumPools( @@ -197,10 +197,9 @@ BlockRange getBlockRangeForSending(BaseKVCacheManager* cacheManager, std::option || lastBlockKey.uniqueTokens.size() == 0 || recvSideHasCP || ppSize > 1) { // disable reuse path, and vwsa don't support reuse. - TLLM_CHECK_WITH_INFO(llmRequest.has_value(), "LlmRequest required for non-reuse-tree transfer path"); bool needSendAllForWindow = common::getEnvKVCacheTransferAllBlocksForWindow(); - auto blockRange = BlockRange::fromAllBlockIds(*cacheManager, (*llmRequest)->mRequestId); + auto blockRange = BlockRange::fromAllBlockIds(*cacheManager, llmRequest.mRequestId); auto const& windowsMetadata = cacheManager->getBlockManager().getWindowSizesMetadata(); @@ -236,20 +235,16 @@ BlockRange getBlockRangeForSending(BaseKVCacheManager* cacheManager, std::option TLLM_CHECK_WITH_INFO(lastBlockKey.uniqueTokens.size() > 0, "lastBlockKey must be non-empty when reuse is enabled"); - // No request on the reuse-tree path: fall through to the plain lastBlockKey lookup. - if (llmRequest.has_value()) + auto multimodalHashes = llmRequest.getMultimodalHashes(); + bool isMultimodal = multimodalHashes.has_value() && *multimodalHashes && !(*multimodalHashes)->empty(); + if (isMultimodal) { - auto multimodalHashes = (*llmRequest)->getMultimodalHashes(); - bool isMultimodal = multimodalHashes.has_value() && *multimodalHashes && !(*multimodalHashes)->empty(); - if (isMultimodal) - { - auto tokensPerBlock = cacheManager->getBlockManager().getTokensPerBlock(); - auto const usableSize = static_cast<SizeType32>(lastBlockKey.uniqueTokens.size()); - auto blockedUniqueTokens = chopVectorIntoBlocks<UniqueToken>( - lastBlockKey.uniqueTokens, usableSize, tokensPerBlock, /*allowPartial=*/true); - auto blockKeys = buildBlockKeys(blockedUniqueTokens, **llmRequest); - return BlockRange::fromReuseTree(*cacheManager, blockKeys, indexFromEnd); - } + auto tokensPerBlock = cacheManager->getBlockManager().getTokensPerBlock(); + auto const usableSize = static_cast<SizeType32>(lastBlockKey.uniqueTokens.size()); + auto blockedUniqueTokens = chopVectorIntoBlocks<UniqueToken>( + lastBlockKey.uniqueTokens, usableSize, tokensPerBlock, /*allowPartial=*/true); + auto blockKeys = buildBlockKeys(blockedUniqueTokens, llmRequest); + return BlockRange::fromReuseTree(*cacheManager, blockKeys, indexFromEnd); } return BlockRange::fromReuseTree(*cacheManager, lastBlockKey, indexFromEnd); @@ -368,15 +363,11 @@ void CacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& sessio { NVTX3_SCOPED_RANGE(CacheFormatter_format); session.setTime(TransferSession::kTimeFormatter); - auto llmRequest = session.getLlmRequest(); - if (llmRequest.has_value()) - { - TLLM_LOG_DEBUG( - mpi::MpiComm::world().getRank(), "Start sending KV cache for request ID: %ld.", (*llmRequest)->mRequestId); - TLLM_CHECK_WITH_INFO( - (*llmRequest)->mSamplingConfig.beamWidth == 1, "Currently, only beam width 1 is supported."); - } + auto const& llmRequest = session.getLlmRequest(); + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "Start sending KV cache for request ID: %ld.", llmRequest.mRequestId); + TLLM_CHECK_WITH_INFO(llmRequest.mSamplingConfig.beamWidth == 1, "Currently, only beam width 1 is supported."); auto const& connections = session.getConnections(); auto const& selfConfig = session.getSelfState().getCacheState().value(); auto const& destConfig = session.getOtherState().getCacheState().value(); @@ -394,6 +385,7 @@ void CacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& sessio size_t targetNum = pickUpConnections.size(); if (targetNum == 0) { + TLLM_LOG_DEBUG("No targets to send KV cache to for request ID: %ld", llmRequest.mRequestId); return; } @@ -421,15 +413,13 @@ void CacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& sessio SizeType32 const numKvPools = static_cast<SizeType32>(kvWindowSizes.size()); - TLLM_LOG_DEBUG("CacheFormatter::format: allWindowSizes=%zu, kvWindowSizes=%d, numPools=%d, requestId=%s", - allWindowSizes.size(), numKvPools, numPools, - llmRequest.has_value() ? std::to_string((*llmRequest)->mRequestId).c_str() : "<request-free>"); + TLLM_LOG_DEBUG("CacheFormatter::format: allWindowSizes=%zu, kvWindowSizes=%d, numPools=%d, requestId=%lu", + allWindowSizes.size(), numKvPools, numPools, llmRequest.mRequestId); bool layerWise = common::getEnvDisaggLayerwise() && numKvPools == 1; if (layerWise) { - TLLM_CHECK_WITH_INFO(llmRequest.has_value(), "LlmRequest required for layer-wise transfer"); - auto& progress = (*llmRequest)->getContextProgress(); + auto& progress = llmRequest.getContextProgress(); SizeType32 const numLayers = blockManager.getNumLayers(); runtime::ITensor::Shape offset = runtime::ITensor::makeShape({0, 0}); for (SizeType32 layerIdx = 0; layerIdx < numLayers; layerIdx++) @@ -525,11 +515,8 @@ void CacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& sessio } } } - if (llmRequest.has_value()) - { - TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "End the sending of KV cache for the request ID: %ld.", - (*llmRequest)->mRequestId); - } + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "End the sending of KV cache for the request ID: %ld.", + llmRequest.mRequestId); return; } @@ -542,9 +529,7 @@ void CacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& sessio // cache blocks to the corresponding buffer. // 5. send the buffer to the corresponding target. Ideally, we send only once (one buffer) for each target. - auto const* sendCancelFlag - = common::getEnvDisaggEnableInflightCancel() ? &session.getDataContext().getTransferTerminate() : nullptr; - auto cacheBufferId = mCacheTransBufferManager->assignBufferIndexForSend(sendCancelFlag); + auto cacheBufferId = mCacheTransBufferManager->assignBufferIndexForSend(); BufferIndexHolder sendHolder(*mCacheTransBufferManager, cacheBufferId, /*isRecv=*/false); int peerDuplicateHeadFactor = targetInfo.mPeerDupHeadFactor; auto bufferTargetNum = targetNum / peerDuplicateHeadFactor; @@ -624,44 +609,23 @@ void CacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& sessio == inputKvCacheBlocksPerWindow.begin()->second.front()->getDataType()); } - if (sendCancelFlag != nullptr && sendCancelFlag->load(std::memory_order_relaxed)) - { - TLLM_THROW("KV cache transfer cancelled before NIXL submission"); - } - - try - { - sendAllBuffers(session, deviceId, outputSplitCaches, bufferCoverTargetNum, preAllocSendBuffer, - bufferManager, targetInfo, pickUpConnections); - } - catch (...) - { - if (agentConnection != nullptr && common::getEnvDisaggEnableInflightCancel()) - { - sendHolder.poison(); - } - throw; - } + sendAllBuffers(session, deviceId, outputSplitCaches, bufferCoverTargetNum, preAllocSendBuffer, bufferManager, + targetInfo, pickUpConnections); session.setTime(TransferSession::kTimeTransmissions); sendHolder.release(); session.setTime(TransferSession::kTimePostprocess); } - if (llmRequest.has_value()) - { - TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "End the sending of KV cache for the request ID:%ld ", - (*llmRequest)->mRequestId); - } + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "End the sending of KV cache for the request ID:%ld ", llmRequest.mRequestId); } void CacheFormatter::unformat(tensorrt_llm::batch_manager::TransferSession& session) { NVTX3_SCOPED_RANGE(CacheFormatter_unformat); session.setTime(TransferSession::kTimeFormatter); - auto llmRequestOpt = session.getLlmRequest(); - TLLM_CHECK_WITH_INFO(llmRequestOpt.has_value(), "LlmRequest required for receiving KV cache"); - auto const& llmRequest = **llmRequestOpt; + auto const& llmRequest = session.getLlmRequest(); auto const ctxReqId = llmRequest.getContextPhaseParams().value().getReqId(); TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "Start receiving KV cache for request ID: %ld, context request ID: %ld.", llmRequest.mRequestId, ctxReqId); @@ -747,7 +711,12 @@ void CacheFormatter::unformat(tensorrt_llm::batch_manager::TransferSession& sess { NVTX3_SCOPED_RANGE(formatInputRecvBuffer); - auto const dataType = mCacheTransBufferManager->getDataType(); + // TODO(disagg-multi-dtype): pool 0's dtype is treated as canonical for the wire + // transport here. Pools with differing dtypes are rejected up-front in + // CacheTransBufferManager's constructor (see cacheTransBuffer.cpp). When + // per-pool dtype dispatch lands, this single dataType variable must be replaced + // with a per-pool lookup keyed by the source pool of each block. + auto dataType = mCacheManager->getPrimaryPool(0)->getDataType(); bool layerWise = common::getEnvDisaggLayerwise() && numKvPools == 1; if (layerWise) { @@ -906,16 +875,12 @@ void CacheFormatter::unformat(tensorrt_llm::batch_manager::TransferSession& sess if (preAssignedKvId.has_value()) { cacheBufferId = static_cast<int>(*preAssignedKvId); - if (!session.hasReservedRecvBuffer(*mCacheTransBufferManager)) - { - recvHolder = BufferIndexHolder(*mCacheTransBufferManager, cacheBufferId, /*isRecv=*/true); - } } else { cacheBufferId = mCacheTransBufferManager->assignBufferIndexForRecv(); - recvHolder = BufferIndexHolder(*mCacheTransBufferManager, cacheBufferId, /*isRecv=*/true); } + recvHolder = BufferIndexHolder(*mCacheTransBufferManager, cacheBufferId, /*isRecv=*/true); auto [recvSplitCachestmp, bufferCoverTargetNumtmp, onlyUseDynamicBuffer] = mCacheTransBufferManager->getOrAllocateRecvBuffers( cacheBufferId, static_cast<int>(targetNum), bufferEleSizes, bufferManager); @@ -1049,7 +1014,6 @@ void CacheFormatter::unformat(tensorrt_llm::batch_manager::TransferSession& sess recvSplitCaches, outputBuffersPerWindow, destConfig, selfConfig, selfIdx, bufferManager); bufferManager.getStream().synchronize(); - (void) session.releaseReservedRecvBuffer(*mCacheTransBufferManager); recvHolder.release(); } session.setTime(TransferSession::kTimePostprocess); diff --git a/cpp/tensorrt_llm/batch_manager/cacheFormatter.h b/cpp/tensorrt_llm/batch_manager/cacheFormatter.h index 21356f3dd6c0..458cac8d4382 100644 --- a/cpp/tensorrt_llm/batch_manager/cacheFormatter.h +++ b/cpp/tensorrt_llm/batch_manager/cacheFormatter.h @@ -28,6 +28,7 @@ #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include <NvInferRuntimeBase.h> #include <cstddef> #include <cstdint> #include <fstream> @@ -176,7 +177,7 @@ inline std::pair<std::vector<size_t>, std::vector<size_t>> pickRecvConnections(s namespace tensorrt_llm::batch_manager::kv_cache_manager { -BlockRange getBlockRangeForSending(BaseKVCacheManager* cacheManager, std::optional<LlmRequest const*> llmRequest, +BlockRange getBlockRangeForSending(BaseKVCacheManager* cacheManager, LlmRequest const& llmRequest, BlockKey const& lastBlockKey, SizeType32 indexFromEnd, bool recvSideHasCP = false, SizeType32 ppSize = 1); using DataContext = tensorrt_llm::executor::kv_cache::DataContext; diff --git a/cpp/tensorrt_llm/batch_manager/cacheTransBuffer.cpp b/cpp/tensorrt_llm/batch_manager/cacheTransBuffer.cpp index 1e8bbcb766bd..772c9555f0f3 100644 --- a/cpp/tensorrt_llm/batch_manager/cacheTransBuffer.cpp +++ b/cpp/tensorrt_llm/batch_manager/cacheTransBuffer.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -21,78 +21,12 @@ #include "tensorrt_llm/common/opUtils.h" #include "tensorrt_llm/executor/executor.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntimeBase.h> #include <mutex> namespace tensorrt_llm::batch_manager::kv_cache_manager { -namespace -{ - -bool isCachePool(BlockManager const& blockManager, SizeType32 poolIdx) -{ - auto const& pool = blockManager.getPool(poolIdx); - return !pool.containsBlockScales && !pool.containsIndexerKCache; -} - -bool isAttentionCachePool(BlockManager const& blockManager, SizeType32 poolIdx) -{ - return isCachePool(blockManager, poolIdx) - && !LinearAttentionMetadata::hasLinearCache(blockManager.getPoolWindowSize(poolIdx)); -} - -tensorrt_llm::DataType getTransferDataType(KVCacheManager::BaseKVCacheManager* cacheManager, bool transferIndexerKCache) -{ - TLLM_CHECK(cacheManager); - if (transferIndexerKCache) - { - auto const indexerKCachePool = cacheManager->getIndexerKCachePool(); - TLLM_CHECK(indexerKCachePool); - return indexerKCachePool->getDataType(); - } - - auto const& blockManager = cacheManager->getBlockManager(); - std::optional<tensorrt_llm::DataType> cacheDataType; - std::optional<tensorrt_llm::DataType> attentionDataType; - SizeType32 firstPoolIdx = -1; - // Recurrent-state pools have a separate transfer manager and formatter. Only - // attention pools determine the KV transfer-buffer dtype. - for (SizeType32 poolIdx = 0; poolIdx < blockManager.getNumPools(); ++poolIdx) - { - if (!isCachePool(blockManager, poolIdx)) - { - continue; - } - - auto const poolDataType = blockManager.getPrimaryPool(poolIdx)->getDataType(); - if (!cacheDataType.has_value()) - { - cacheDataType = poolDataType; - } - if (!isAttentionCachePool(blockManager, poolIdx)) - { - continue; - } - if (!attentionDataType.has_value()) - { - attentionDataType = poolDataType; - firstPoolIdx = poolIdx; - continue; - } - - TLLM_CHECK_WITH_INFO(poolDataType == attentionDataType.value(), - "Disaggregated KV cache transfer does not yet support attention pools with differing dtypes " - "(pool %d dtype=%d, pool %d dtype=%d). TODO(disagg-multi-dtype): per-pool dtype dispatch in formatter.", - firstPoolIdx, static_cast<int>(attentionDataType.value()), poolIdx, static_cast<int>(poolDataType)); - } - - TLLM_CHECK_WITH_INFO(cacheDataType.has_value(), "Disaggregated KV cache transfer requires a cache pool"); - return attentionDataType.value_or(cacheDataType.value()); -} - -} // namespace - // ============================================================================ // FabricMemory Implementation // ============================================================================ @@ -260,38 +194,39 @@ bool FabricMemory::supportFabricMemory() size_t CacheTransBufferManager::computeTransferBufferSize( KVCacheManager::BaseKVCacheManager* cacheManager, std::optional<size_t> maxNumTokens, bool transferIndexerKCache) { - auto const dataType = getTransferDataType(cacheManager, transferIndexerKCache); - - auto const& blockManager = cacheManager->getBlockManager(); - auto const tokensPerBlock = blockManager.getTokensPerBlock(); - bool hasAttentionCachePool = false; - for (SizeType32 poolIdx = 0; poolIdx < blockManager.getNumPools(); ++poolIdx) + nvinfer1::DataType dataType; + if (transferIndexerKCache) + { + dataType = cacheManager->getIndexerKCachePool()->getDataType(); + } + else { - hasAttentionCachePool |= isAttentionCachePool(blockManager, poolIdx); + dataType = cacheManager->getPrimaryPool(0)->getDataType(); } + + auto tokensPerBlock = cacheManager->getBlockManager().getTokensPerBlock(); size_t bufferSizeFromMaxNumToken = 0; if (maxNumTokens.has_value()) { TLLM_CHECK(maxNumTokens.value() % tokensPerBlock == 0); - auto const dataSize = common::getDTypeSize(dataType); - SizeType32 indexerCacheByteSizePerTokenPerLayer = 0; + auto dataSize = common::getDTypeSize(dataType); + SizeType32 kvCacheByteSizePerTokenPerLayer = 0; if (transferIndexerKCache) { - indexerCacheByteSizePerTokenPerLayer + kvCacheByteSizePerTokenPerLayer = cacheManager->getIndexerKCachePool()->getDimension<-1>() * dataSize / tokensPerBlock; } - for (auto layerId = 0; layerId < blockManager.getNumLayers(); layerId++) + else { - auto const poolIdx = blockManager.getLayerPoolIdx(layerId); - auto const encodedWindowSize = blockManager.getPoolWindowSize(poolIdx); - if (!transferIndexerKCache && hasAttentionCachePool - && LinearAttentionMetadata::hasLinearCache(encodedWindowSize)) - { - continue; - } - - auto const windowSize = static_cast<size_t>(encodedWindowSize); + auto primaryPool = cacheManager->getPrimaryPool(0); + kvCacheByteSizePerTokenPerLayer + = primaryPool->getDimension<-1>() * primaryPool->getDimension<2>() * dataSize / tokensPerBlock; + } + for (auto layerId = 0; layerId < cacheManager->getBlockManager().getNumLayers(); layerId++) + { + auto poolIdx = cacheManager->getBlockManager().getLayerPoolIdx(layerId); + auto windowSize = static_cast<size_t>(cacheManager->getBlockManager().getPoolWindowSize(poolIdx)); auto alignedWindowSize = (windowSize + tokensPerBlock - 1) / tokensPerBlock * tokensPerBlock; auto validTokenNum = (alignedWindowSize < maxNumTokens.value() ? alignedWindowSize : maxNumTokens.value()); if (common::getEnvKVCacheTransferAllBlocksForWindow()) @@ -300,17 +235,7 @@ size_t CacheTransBufferManager::computeTransferBufferSize( } validTokenNum += tokensPerBlock; // add one more block - if (transferIndexerKCache) - { - bufferSizeFromMaxNumToken += validTokenNum * indexerCacheByteSizePerTokenPerLayer; - } - else - { - auto const primaryPool = blockManager.getPrimaryPool(poolIdx); - auto const kvCacheByteSizePerTokenPerLayer - = primaryPool->getDimension<-1>() * primaryPool->getDimension<2>() * dataSize / tokensPerBlock; - bufferSizeFromMaxNumToken += validTokenNum * kvCacheByteSizePerTokenPerLayer; - } + bufferSizeFromMaxNumToken += validTokenNum * kvCacheByteSizePerTokenPerLayer; } } @@ -320,12 +245,36 @@ size_t CacheTransBufferManager::computeTransferBufferSize( CacheTransBufferManager::CacheTransBufferManager( KVCacheManager::BaseKVCacheManager* cacheManager, std::optional<size_t> maxNumTokens, bool transferIndexerKCache) : BaseTransBufferManager(computeTransferBufferSize(cacheManager, maxNumTokens, transferIndexerKCache), - getTransferDataType(cacheManager, transferIndexerKCache), maxNumTokens) + transferIndexerKCache ? cacheManager->getIndexerKCachePool()->getDataType() + : cacheManager->getPrimaryPool(0)->getDataType(), + maxNumTokens) , mCacheManager{cacheManager} , mTransferIndexerKCache{transferIndexerKCache} { // TODO: FP4 dataSize TLLM_CHECK(mCacheManager); + // TODO(disagg-multi-dtype): Per-pool dtype dispatch in formatter / transfer buffer + // not yet implemented. Disagg currently picks pool 0's dtype as the canonical + // transport type (above), so any KV pool with a different dtype would be silently + // miscoerced on the wire. Fail loudly until per-pool dispatch lands. We restrict + // the comparison to KV pools (getNumPools(false, false)) since block-scale and + // indexer-K pools legitimately have their own dtypes and travel through their own + // code paths. + if (!transferIndexerKCache) + { + auto const numKvPools = mCacheManager->getBlockManager().getNumPools( + /*includeBlockScalePools=*/false, /*includeIndexerKCachePools=*/false); + auto const dtype0 = mCacheManager->getPrimaryPool(0)->getDataType(); + for (SizeType32 i = 1; i < numKvPools; ++i) + { + auto const dtypeI = mCacheManager->getPrimaryPool(i)->getDataType(); + TLLM_CHECK_WITH_INFO(dtypeI == dtype0, + "Disaggregated KV cache transfer does not yet support pools with differing dtypes " + "(pool 0 dtype=%d, pool %d dtype=%d). TODO(disagg-multi-dtype): per-pool dtype " + "dispatch in formatter.", + static_cast<int>(dtype0), i, static_cast<int>(dtypeI)); + } + } TLLM_LOG_INFO("CacheTransBufferManager created for KV cache"); } diff --git a/cpp/tensorrt_llm/batch_manager/cacheTransBuffer.h b/cpp/tensorrt_llm/batch_manager/cacheTransBuffer.h index 72f67e8ca3c9..1635c11bc673 100644 --- a/cpp/tensorrt_llm/batch_manager/cacheTransBuffer.h +++ b/cpp/tensorrt_llm/batch_manager/cacheTransBuffer.h @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -74,12 +74,6 @@ class CacheTransBufferManager : public BaseTransBufferManager return mCacheManager; } - /// @brief Get the data type used by KV cache transfer buffers. - [[nodiscard]] tensorrt_llm::DataType getDataType() const noexcept - { - return mDataType; - } - [[nodiscard]] BufferKind getBufferKind() const override { return mTransferIndexerKCache ? BufferKind::kKV_INDEXER : BufferKind::kKV; diff --git a/cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp b/cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp index c4acce47593f..e5a9c8c20273 100644 --- a/cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp +++ b/cpp/tensorrt_llm/batch_manager/cacheTransceiver.cpp @@ -48,22 +48,15 @@ #include "tensorrt_llm/batch_manager/rnnStateManager.h" #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/cache_transmission/mpi_utils/connection.h" #include "tensorrt_llm/executor/dataTransceiverState.h" -#include "tensorrt_llm/executor/serialization.h" #include "tensorrt_llm/executor/serializeUtils.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" #include "tensorrt_llm/runtime/utils/pgUtils.h" #include <algorithm> #include <chrono> #include <cstddef> -#include <filesystem> -#include <fstream> -#include <iomanip> #include <numeric> -#include <random> -#include <sstream> #include <thread> #include <unordered_map> #include <unordered_set> @@ -71,28 +64,6 @@ namespace tensorrt_llm::batch_manager { -namespace -{ - -/// Generate a UUID-like hex string (e.g. "a1b2c3d4-e5f6-7890-abcd-ef1234567890") -/// to uniquely identify a CacheTransceiver instance across gen instances. -std::string generateInstanceId() -{ - // The RNG state is comparatively expensive to construct/seed, so keep one - // per thread instead of building it on every call. - static thread_local std::mt19937_64 gen{std::random_device{}()}; - std::uniform_int_distribution<uint64_t> dis; - uint64_t a = dis(gen); - uint64_t b = dis(gen); - std::ostringstream oss; - oss << std::hex << std::setfill('0') << std::setw(8) << (a >> 32) << "-" << std::setw(4) << ((a >> 16) & 0xFFFF) - << "-" << std::setw(4) << (a & 0xFFFF) << "-" << std::setw(4) << (b >> 48) << "-" << std::setw(12) - << (b & 0xFFFFFFFFFFFF); - return oss.str(); -} - -} // anonymous namespace - std::mutex CacheTransceiver::mDllMutex; namespace @@ -121,57 +92,20 @@ enum class TransferConsensusState : std::uint64_t { kCompleted = 1, kFailed = 2, - kTimedOut = 3, }; struct TransferStateCounts { int completedCount{0}; int failedCount{0}; - int timedOutCount{0}; }; struct TransferConsensusOutcome { std::unordered_set<RequestIdType> completedRequestIds; std::unordered_set<RequestIdType> failedRequestIds; - std::unordered_set<RequestIdType> timedOutRequestIds; }; -template <typename CancelFn> -bool requestCancellationNoThrow(RequestIdType requestId, char const* transferKind, CancelFn&& cancelFn) noexcept -{ - try - { - return cancelFn(); - } - catch (std::exception const& error) - { - TLLM_LOG_ERROR( - "%s cancellation for request %ld failed and will be retried: %s", transferKind, requestId, error.what()); - } - catch (...) - { - TLLM_LOG_ERROR("%s cancellation for request %ld failed with an unknown error and will be retried", transferKind, - requestId); - } - return false; -} - -long getTransferElapsedMs(std::shared_ptr<LlmRequest> const& request, LlmRequest::TimePoint end) -{ - auto const elapsed - = std::chrono::duration_cast<std::chrono::milliseconds>(end - request->getKvCacheTransferStart()); - return static_cast<long>(elapsed.count()); -} - -std::vector<RequestIdType> sortedRequestIds(std::unordered_set<RequestIdType> const& requestIds) -{ - std::vector<RequestIdType> result(requestIds.begin(), requestIds.end()); - std::sort(result.begin(), result.end()); - return result; -} - void appendPackedTransferState( std::vector<std::uint64_t>& packedStates, RequestIdType requestId, TransferConsensusState state) { @@ -211,11 +145,10 @@ std::vector<std::uint64_t> gatherPackedTransferStates( TransferConsensusOutcome reduceTransferStates(std::shared_ptr<CacheTransceiverComm> const& comm, std::unordered_set<RequestIdType> const& completedRequestIds, - std::unordered_set<RequestIdType> const& failedRequestIds, - std::unordered_set<RequestIdType> const& timedOutRequestIds) + std::unordered_set<RequestIdType> const& failedRequestIds) { std::vector<std::uint64_t> localStates; - localStates.reserve((completedRequestIds.size() + failedRequestIds.size() + timedOutRequestIds.size()) * 2); + localStates.reserve((completedRequestIds.size() + failedRequestIds.size()) * 2); for (auto const requestId : completedRequestIds) { if (failedRequestIds.find(requestId) == failedRequestIds.end()) @@ -227,10 +160,6 @@ TransferConsensusOutcome reduceTransferStates(std::shared_ptr<CacheTransceiverCo { appendPackedTransferState(localStates, requestId, TransferConsensusState::kFailed); } - for (auto const requestId : timedOutRequestIds) - { - appendPackedTransferState(localStates, requestId, TransferConsensusState::kTimedOut); - } int const syncSize = (comm != nullptr) ? comm->getSize() : 1; auto const gatheredStates @@ -250,7 +179,6 @@ TransferConsensusOutcome reduceTransferStates(std::shared_ptr<CacheTransceiverCo { case TransferConsensusState::kCompleted: counts.completedCount++; break; case TransferConsensusState::kFailed: counts.failedCount++; break; - case TransferConsensusState::kTimedOut: counts.timedOutCount++; break; } } @@ -258,11 +186,7 @@ TransferConsensusOutcome reduceTransferStates(std::shared_ptr<CacheTransceiverCo for (auto const& [requestId, counts] : stateCounts) { auto const terminalCount = counts.completedCount + counts.failedCount; - if (counts.timedOutCount > 0) - { - outcome.timedOutRequestIds.insert(requestId); - } - if (terminalCount == syncSize && (counts.failedCount > 0 || counts.timedOutCount > 0)) + if (terminalCount == syncSize && counts.failedCount > 0) { outcome.failedRequestIds.insert(requestId); } @@ -277,13 +201,10 @@ TransferConsensusOutcome reduceTransferStates(std::shared_ptr<CacheTransceiverCo TransferConsensusOutcome reduceTransferStates(std::shared_ptr<CacheTransceiverComm> const& firstComm, std::shared_ptr<CacheTransceiverComm> const& secondComm, std::unordered_set<RequestIdType> const& completedRequestIds, - std::unordered_set<RequestIdType> const& failedRequestIds, - std::unordered_set<RequestIdType> const& timedOutRequestIds) + std::unordered_set<RequestIdType> const& failedRequestIds) { - auto const firstOutcome - = reduceTransferStates(firstComm, completedRequestIds, failedRequestIds, timedOutRequestIds); - return reduceTransferStates( - secondComm, firstOutcome.completedRequestIds, firstOutcome.failedRequestIds, firstOutcome.timedOutRequestIds); + auto const firstOutcome = reduceTransferStates(firstComm, completedRequestIds, failedRequestIds); + return reduceTransferStates(secondComm, firstOutcome.completedRequestIds, firstOutcome.failedRequestIds); } void recordLocalTransferOutcome(RequestIdType requestId, std::shared_ptr<LlmRequest> request, bool failed, @@ -323,9 +244,6 @@ std::unique_ptr<BaseCacheTransceiver> CacheTransceiverFactory::createCacheTransc TLLM_LOG_INFO("CacheTransceiver is disabled."); return nullptr; } - TLLM_CHECK_WITH_INFO(!common::getEnvDisaggEnableInflightCancel(), - "TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 is supported only by the PyExecutor C++ NIXL transceiver path; " - "the legacy C++ executor does not provide the required deferred cleanup and poison escalation."); auto backendType = cacheTransceiverConfig.value().getBackendType(); if (backendType.value() == executor::CacheTransceiverConfig::BackendType::DEFAULT) { @@ -374,37 +292,14 @@ std::unique_ptr<BaseCacheTransceiver> CacheTransceiverFactory::createCacheTransc CacheTransceiver::CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheManager, executor::kv_cache::CacheState::ModelConfig const& cacheStateModelCfg, runtime::WorldConfig const& worldConfig, - std::vector<SizeType32> const& attentionLayerNumPerPP, tensorrt_llm::DataType dataType, + std::vector<SizeType32> const& attentionLayerNumPerPP, nvinfer1::DataType dataType, executor::kv_cache::CacheState::AttentionType attentionType, std::optional<executor::CacheTransceiverConfig> cacheTransceiverConfig, - std::vector<SizeType32> const& rnnLayerNumPerPP) + rnn_state_manager::RnnStateManager* rnnStateManager, std::vector<SizeType32> const& rnnLayerNumPerPP) : mCacheTransceiverConfig{cacheTransceiverConfig} + , mRnnStateManager{rnnStateManager} { using tensorrt_llm::batch_manager::kv_cache_manager::CacheFormatter; - TLLM_CHECK_WITH_INFO(mCacheTransceiverConfig.has_value(), "CacheTransceiverConfig is not set."); - auto const backendType = mCacheTransceiverConfig.value().getBackendType(); - TLLM_CHECK_WITH_INFO( - backendType.has_value() && (backendType.value() != executor::CacheTransceiverConfig::BackendType::DEFAULT), - " CacheTransceiverConfig::BackendType is not set."); - if (common::getEnvDisaggEnableInflightCancel()) - { - auto const nixlBackend = common::getEnvNixlBackend(); - TLLM_CHECK_WITH_INFO( - backendType.value() == executor::CacheTransceiverConfig::BackendType::NIXL && nixlBackend == "UCX", - "TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 is experimental and currently supported only with the " - "NIXL cache transceiver and the UCX NIXL backend."); - TLLM_CHECK_WITH_INFO(mCacheTransceiverConfig->getKvTransferTimeoutMs().has_value(), - "TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 requires kv_transfer_timeout_ms to enforce a finite deadline."); - TLLM_CHECK_WITH_INFO(!common::getEnvDisableKVCacheTransferOverlap(), - "TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 requires asynchronous KV cache transfer; " - "TRTLLM_DISABLE_KV_CACHE_TRANSFER_OVERLAP=1 is not supported."); - TLLM_CHECK_WITH_INFO(!common::getEnvDisaggLayerwise(), - "TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 does not support layer-wise KV cache transfer."); - TLLM_CHECK_WITH_INFO(!common::getEnvTryZCopyForKVCacheTransfer(), - "TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL=1 does not support zero-copy KV cache transfer because request " - "blocks cannot be quarantined after an unquiesced cancellation."); - } - if (useMPI()) { mGroupComm = std::make_shared<CacheTransceiverComm>(std::addressof(tensorrt_llm::mpi::MpiComm::session())); @@ -414,89 +309,6 @@ CacheTransceiver::CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheMa mGroupComm = std::make_shared<CacheTransceiverComm>(tensorrt_llm::pg_utils::get_world_pg()); } - // Generate instance ID on rank 0 and broadcast to all ranks in the session - // so every rank in the same gen/ctx instance shares the same ID. - { - if (mGroupComm->getRank() == 0) - { - mInstanceId = generateInstanceId(); - } - if (useMPI()) - { - int len = static_cast<int>(mInstanceId.size()); - tensorrt_llm::mpi::MpiComm::session().bcast(&len, 1, mpi::MpiType::kINT32, 0); - mInstanceId.resize(len); - tensorrt_llm::mpi::MpiComm::session().bcast(mInstanceId.data(), len, mpi::MpiType::kCHAR, 0); - } - else - { - // PG path: rank 0 sends via allgather, others receive. - constexpr int kUuidLen = 36; - std::vector<char> sendBuf(kUuidLen, '\0'); - if (mGroupComm->getRank() == 0) - { - std::copy_n(mInstanceId.begin(), std::min<size_t>(mInstanceId.size(), kUuidLen), sendBuf.begin()); - } - std::vector<char> recvBuf(kUuidLen * mGroupComm->getSize(), '\0'); - mGroupComm->allgather(std::ref(sendBuf), std::ref(recvBuf), {}); - // Take rank 0's segment. - mInstanceId = std::string(recvBuf.begin(), recvBuf.begin() + kUuidLen); - } - } - - // Calibrate steady_clock across ranks so that cross-node allgather - // in batchUpdateKVCacheTransferBW can compare time points. - // globalSteadyClockOffset() reads a single process-global copy shared with - // the nanobind module, so if the Python runtime already calibrated the offset - // (PyExecutor::_set_global_steady_clock_offset) it is visible here and we skip; - // the pure-C++ path performs the calibration below. - // The check-and-set is guarded by a mutex so that CacheTransceiver instances - // constructed concurrently in the same process (e.g. multi-engine serving) do - // not race on the shared offset or issue mismatched collectives. - { - static std::mutex sSteadyClockCalibrationMutex; - std::lock_guard<std::mutex> lock(sSteadyClockCalibrationMutex); - if (!globalSteadyClockOffset().has_value()) - { - using Duration = LlmRequest::Duration; - // Synchronize all ranks immediately before sampling the local clock so - // every rank measures from a consistent point. - if (useMPI()) - { - tensorrt_llm::mpi::MpiComm::session().barrier(); - } - else - { - // CacheTransceiverComm exposes no barrier primitive, so use a cheap - // allgather as a pseudo-barrier for the process-group path. - int64_t const dummy = 0; - std::vector<int64_t> dummyRecv(mGroupComm->getSize(), 0); - mGroupComm->allgather(dummy, std::ref(dummyRecv), {}); - } - auto localNow = std::chrono::steady_clock::now(); - auto localNs = std::chrono::duration_cast<std::chrono::nanoseconds>(localNow.time_since_epoch()).count(); - - // Allgather timestamps from all ranks - std::vector<int64_t> allNs(mGroupComm->getSize(), 0); - if (useMPI()) - { - tensorrt_llm::mpi::MpiComm::session().allgather(&localNs, allNs.data(), 1, mpi::MpiType::kINT64); - } - else - { - mGroupComm->allgather(localNs, std::ref(allNs), {}); - } - - // Offset = rank0's timestamp - my timestamp (same formula as Python) - auto offsetNs = allNs[0] - localNs; - globalSteadyClockOffset() = Duration(offsetNs); - - TLLM_LOG_INFO(mGroupComm->getRank(), - "CacheTransceiver: set global steady clock offset = %.6f sec for rank %d", - static_cast<double>(offsetNs) / 1e9, mGroupComm->getRank()); - } - } - if (worldConfig.isTensorParallel() || worldConfig.isContextParallel()) { mGroupTensorParaComm = std::make_shared<CacheTransceiverComm>( @@ -538,6 +350,12 @@ CacheTransceiver::CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheMa } } bool isMLA = attentionType == executor::kv_cache::CacheState::AttentionType::kMLA; + TLLM_CHECK_WITH_INFO(mCacheTransceiverConfig.has_value(), "CacheTransceiverConfig is not set."); + auto backendType = mCacheTransceiverConfig.value().getBackendType(); + TLLM_CHECK_WITH_INFO( + backendType.has_value() && (backendType.value() != executor::CacheTransceiverConfig::BackendType::DEFAULT), + " CacheTransceiverConfig::BackendType is not set."); + std::optional<size_t> maxNumTokens = mCacheTransceiverConfig.value().getMaxTokensInBuffer(); mCacheTransBufferManagers.push_back( @@ -548,9 +366,28 @@ CacheTransceiver::CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheMa std::make_unique<kv_cache_manager::CacheTransBufferManager>(cacheManager, maxNumTokens, true)); } + // RNN specific setup + if (mRnnStateManager != nullptr) + { + TLLM_LOG_DEBUG("Setting up RNN cache transfer components."); + TLLM_CHECK(!rnnLayerNumPerPP.empty()); + + mRnnCacheTransBufferManager + = std::make_unique<rnn_state_manager::RnnCacheTransBufferManager>(mRnnStateManager, maxNumTokens); + + auto rnnModelCfg = mRnnStateManager->getRnnCacheStateModelConfig(); + + auto const convStateDataType = mRnnStateManager->getConvStateDataType(); + auto const ssmStateDataType = mRnnStateManager->getSsmStateDataType(); + + mCacheState->setRnnConfig(rnnModelCfg, rnnLayerNumPerPP, convStateDataType, ssmStateDataType); + + TLLM_LOG_INFO("RNN cache transfer components initialized."); + } + // Unified pool path (CppMambaHybridCacheManager): build RnnModelConfig from - // LinearAttentionMetadata. Detected by rnnLayerNumPerPP being non-empty. - if (!rnnLayerNumPerPP.empty()) + // LinearAttentionMetadata. Detected by rnnLayerNumPerPP set but no RnnStateManager. + if (mRnnStateManager == nullptr && !rnnLayerNumPerPP.empty()) { auto const& blockManager = cacheManager->getBlockManager(); auto const& linearMeta = blockManager.getLinearAttentionMetadata(); @@ -570,20 +407,20 @@ CacheTransceiver::CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheMa // Pool dtype is UINT8 (raw byte storage), so we cannot use pool->getDataType(). // Only the byte size matters for split/concat kernel stride calculations — the actual // dtype enum is not interpreted numerically, just used for getDTypeSize() dispatch. - auto dtypeFromSize = [](SizeType32 size) -> tensorrt_llm::DataType + auto dtypeFromSize = [](SizeType32 size) -> nvinfer1::DataType { switch (size) { - case 4: return tensorrt_llm::DataType::kFLOAT; - case 2: return tensorrt_llm::DataType::kBF16; - case 1: return tensorrt_llm::DataType::kFP8; + case 4: return nvinfer1::DataType::kFLOAT; + case 2: return nvinfer1::DataType::kBF16; + case 1: return nvinfer1::DataType::kFP8; default: TLLM_THROW("Unsupported RNN state dtype size: %d", size); } }; TLLM_CHECK_WITH_INFO(linearMeta->rnnSsmDtypeSize > 0, "rnnSsmDtypeSize not set in LinearAttentionMetadata"); TLLM_CHECK_WITH_INFO(linearMeta->rnnConvDtypeSize > 0, "rnnConvDtypeSize not set in LinearAttentionMetadata"); - tensorrt_llm::DataType ssmDtype = dtypeFromSize(linearMeta->rnnSsmDtypeSize); - tensorrt_llm::DataType convDtype = dtypeFromSize(linearMeta->rnnConvDtypeSize); + nvinfer1::DataType ssmDtype = dtypeFromSize(linearMeta->rnnSsmDtypeSize); + nvinfer1::DataType convDtype = dtypeFromSize(linearMeta->rnnConvDtypeSize); mCacheState->setRnnConfig(rnnModelCfg, rnnLayerNumPerPP, convDtype, ssmDtype); // Create RnnCacheTransBufferManager for unified pool path. @@ -667,6 +504,11 @@ CacheTransceiver::CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheMa auto makeRnnFormatter = [this, cacheManager]() -> std::unique_ptr<RnnCacheFormatter> { + if (mRnnStateManager != nullptr && mRnnCacheTransBufferManager != nullptr) + { + // Slot-based path (CppMambaCacheManager) + return std::make_unique<RnnCacheFormatter>(mRnnStateManager, mRnnCacheTransBufferManager.get()); + } // Unified pool path (CppMambaHybridCacheManager) if (mCacheState->hasRnnConfig() && mRnnCacheTransBufferManager != nullptr) { @@ -678,10 +520,8 @@ CacheTransceiver::CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheMa auto makeCacheTransferLayer = [&]() { return CacheTransferLayer(*mCacheState, makeFormatter(), makeRnnFormatter()); }; - mCacheSender - = std::make_unique<CacheSender>(mManager.get(), worldConfig.getRank(), makeCacheTransferLayer(), mInstanceId); - mCacheReceiver - = std::make_unique<CacheReceiver>(mManager.get(), worldConfig.getRank(), makeCacheTransferLayer(), mInstanceId); + mCacheSender = std::make_unique<CacheSender>(mManager.get(), worldConfig.getRank(), makeCacheTransferLayer()); + mCacheReceiver = std::make_unique<CacheReceiver>(mManager.get(), worldConfig.getRank(), makeCacheTransferLayer()); // Keep automatic enablement within the currently qualified C++ NIXL/UCX TP1/CP1 pipeline topology. bool const coordinatorTopologyEligible = worldConfig.getPipelineParallelism() > 1 && useMPI() @@ -693,7 +533,7 @@ CacheTransceiver::CacheTransceiver(kv_cache_manager::BaseKVCacheManager* cacheMa TLLM_CHECK(mGroupPipeParaComm != nullptr); constexpr std::uint64_t kCoordinatorProtocolVersion = 1; std::uint64_t const localVersion = coordinatorTopologyEligible ? kCoordinatorProtocolVersion : 0; - bool const cancellationEnabled = common::getEnvDisaggEnableInflightCancel(); + constexpr bool cancellationEnabled = false; std::uint64_t const localProtocolMode = (localVersion << 1) | static_cast<std::uint64_t>(cancellationEnabled); std::vector<std::uint64_t> protocolModes(static_cast<std::size_t>(mGroupPipeParaComm->getSize())); mGroupPipeParaComm->allgather(&localProtocolMode, protocolModes.data(), 1, mpi::MpiType::kUINT64); @@ -734,17 +574,6 @@ void CacheTransceiver::initializeCommState() mCommState = std::addressof(mCacheSender->getCommState()); } -std::vector<char> CacheTransceiver::getSerializedDataTransceiverState() const -{ - TLLM_CHECK(mCommState != nullptr && mCacheState != nullptr); - executor::DataTransceiverState state; - state.setCommState(*mCommState); - state.setCacheState(*mCacheState); - // Only this API marks the state; context responses leave it unset. - state.setIsArbitraryTransferState(true); - return executor::Serialization::serialize(state); -} - void CacheTransceiver::setContextState(LlmRequest* llmRequest) { TLLM_CHECK(llmRequest && llmRequest->isContextOnlyRequest()); @@ -802,37 +631,10 @@ void CacheTransceiver::respondAndSendLayerWise( void CacheTransceiver::requestAndReceiveSync(std::shared_ptr<LlmRequest> llmRequest) { TLLM_CHECK(llmRequest && llmRequest->isGenerationOnlyRequest()); - auto const requestId = llmRequest->mRequestId; - auto const contextRequestId = llmRequest->getContextPhaseParams().value().getReqId(); - TLLM_LOG_DEBUG("Synchronous KV cache receive request %zu, context request %zu waiting for native completion.", - requestId, contextRequestId); - try { auto future = mCacheReceiver->receiveAsync(llmRequest); future.get(); } - catch (std::exception const& err) - { - llmRequest->setState(LlmRequestState::kDISAGG_TRANS_ERROR); - llmRequest->setKvCacheTransferEnd(LlmRequest::getSteadyClockNow()); - TLLM_LOG_ERROR("Synchronous KV cache receive request %zu, context request %zu failed: %s", requestId, - contextRequestId, err.what()); - return; - } - catch (...) - { - llmRequest->setState(LlmRequestState::kDISAGG_TRANS_ERROR); - llmRequest->setKvCacheTransferEnd(LlmRequest::getSteadyClockNow()); - TLLM_LOG_ERROR("Synchronous KV cache receive request %zu, context request %zu failed with an unknown error", - requestId, contextRequestId); - return; - } - if (llmRequest->getState() == LlmRequestState::kDISAGG_TRANS_ERROR) - { - TLLM_LOG_ERROR("Synchronous KV cache receive request %zu, context request %zu completed with an error state.", - requestId, contextRequestId); - return; - } llmRequest->setState(LlmRequestState::kDISAGG_GENERATION_TRANS_COMPLETE); } @@ -886,109 +688,67 @@ std::vector<LlmRequest::RequestIdType> gatherRequestIds( return retData; } -void batchUpdateKVCacheTransferBW( - std::shared_ptr<CacheTransceiverComm> const& comm, std::vector<LlmRequest*> const& requests) +void updateKVCacheTransferBW(std::shared_ptr<CacheTransceiverComm> const& mComm, LlmRequest* request) { - // Key-based merge: each rank serializes (requestId, start, end, size) - // tuples and we use allgatherv so ranks may have different request counts. - // The merge matches by requestId, not by position — this tolerates - // ordering differences and count mismatches across ranks. - namespace su = executor::serialize_utils; - int const worldSize = comm->getSize(); - - // --- Serialize local entries keyed by requestId --- - std::size_t const numReqs = requests.size(); + int worldSize = mComm->getSize(); std::ostringstream oStream; - su::serialize(numReqs, oStream); - for (auto* req : requests) - { - su::serialize(req->getContextPhaseParams().value().getReqId(), oStream); - su::serialize(req->getKvCacheTransferStart(), oStream); - su::serialize(req->getKvCacheTransferEnd(), oStream); - su::serialize(req->getKvCacheSize(), oStream); - } + su::serialize(request->getKvCacheTransferStart(), oStream); + su::serialize(request->getKvCacheTransferEnd(), oStream); auto str = oStream.str(); std::vector<char> sendBuffer(str.begin(), str.end()); - int const sendSize = static_cast<int>(sendBuffer.size()); + auto sendBufferSize = sendBuffer.size(); + auto recvBufferSize = sendBufferSize * worldSize; + std::vector<char> recvBuffer(recvBufferSize); - // --- Step 1: allgather per-rank buffer sizes --- - std::vector<int> recvCounts(worldSize, 0); if (useMPI()) { - comm->allgather(&sendSize, recvCounts.data(), 1, mpi::MpiType::kINT32); + mComm->allgather(sendBuffer.data(), recvBuffer.data(), sendBufferSize, mpi::MpiType::kCHAR); } else { - comm->allgather(sendSize, std::ref(recvCounts), {}); + mComm->allgather(std::ref(sendBuffer), std::ref(recvBuffer), {}); } - // --- Step 2: allgatherv the serialized data --- - std::vector<int> displs(worldSize, 0); - int totalRecvSize = 0; - for (int r = 0; r < worldSize; ++r) + su::VectorWrapBuf<char> strbuf(recvBuffer); + std::istream is(&strbuf); + + auto minStartTime = executor::RequestPerfMetrics::TimePoint::max(); + auto maxEndTime = executor::RequestPerfMetrics::TimePoint::min(); + + for (int rank = 0; rank < worldSize; rank++) { - displs[r] = totalRecvSize; - totalRecvSize += recvCounts[r]; + minStartTime = std::min(su::deserialize<executor::RequestPerfMetrics::TimePoint>(is), minStartTime); + maxEndTime = std::max(su::deserialize<executor::RequestPerfMetrics::TimePoint>(is), maxEndTime); } - std::vector<char> recvBuffer(totalRecvSize, 0); + + // Handle KV cache size separately - gather all sizes to the leader rank + std::size_t localKVCacheSize = request->getKvCacheSize(); + std::vector<std::size_t> allKVCacheSizes(worldSize, 0); if (useMPI()) { - comm->allgatherv(sendBuffer.data(), sendSize, mpi::MpiType::kCHAR, recvBuffer.data(), recvCounts, displs, - mpi::MpiType::kCHAR); + mComm->allgather(&localKVCacheSize, allKVCacheSizes.data(), 1, mpi::MpiType::kUINT64); } else { - comm->allgatherv(std::ref(sendBuffer), std::ref(recvBuffer), recvCounts, {}); + mComm->allgather(&localKVCacheSize, std::ref(allKVCacheSizes), {}); } - // --- Step 3: Deserialize and merge by requestId --- - using TimePoint = executor::RequestPerfMetrics::TimePoint; - using ReqIdType = LlmRequest::RequestIdType; - - struct MergedEntry - { - TimePoint minStart = TimePoint::max(); - TimePoint maxEnd = TimePoint::min(); - std::size_t totalSize = 0; - }; - - std::unordered_map<ReqIdType, MergedEntry> merged; - - su::VectorWrapBuf<char> strbuf(recvBuffer); - std::istream is(&strbuf); - - for (int rank = 0; rank < worldSize; ++rank) + std::size_t totalKVCacheSize = 0; + for (int rank = 0; rank < worldSize; rank++) { - auto rankNumReqs = su::deserialize<std::size_t>(is); - for (std::size_t i = 0; i < rankNumReqs; ++i) - { - auto rid = su::deserialize<ReqIdType>(is); - auto start = su::deserialize<TimePoint>(is); - auto end = su::deserialize<TimePoint>(is); - auto size = su::deserialize<std::size_t>(is); - - auto& entry = merged[rid]; - entry.minStart = std::min(entry.minStart, start); - entry.maxEnd = std::max(entry.maxEnd, end); - entry.totalSize += size; - } + totalKVCacheSize += allKVCacheSizes[rank]; } - // --- Step 4: Update local requests --- - for (auto* req : requests) + // Update the latest KV cache transfer time for leader rank + if (mComm->getRank() == 0) { - auto reqId = req->getContextPhaseParams().value().getReqId(); - auto it = merged.find(reqId); - if (it != merged.end()) - { - req->setKvCacheTransferStart(it->second.minStart); - req->setKvCacheTransferEnd(it->second.maxEnd); - req->setKvCacheSize(it->second.totalSize); - } + request->setKvCacheTransferStart(minStartTime); + request->setKvCacheTransferEnd(maxEndTime); + request->setKvCacheSize(totalKVCacheSize); } } @@ -996,9 +756,6 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( std::optional<int> const& atLeastRequestNum, bool markComplete) { bool const blockAll = !atLeastRequestNum.has_value(); - bool const inflightCancelEnabled = common::getEnvDisaggEnableInflightCancel(); - TLLM_CHECK_WITH_INFO(!inflightCancelEnabled || !blockAll, - "In-flight cancellation requires a finite context-transfer status poll; pass 0 for a nonblocking poll."); std::optional<int> senderFutureTimeoutMs = std::nullopt; if (mCacheTransceiverConfig.has_value()) { @@ -1006,9 +763,8 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( } bool const needsProgress = atLeastRequestNum.value_or(0) > 0; auto const futureWaitInterval = getTransferFutureWaitInterval(senderFutureTimeoutMs, needsProgress); - // Without the opt-in flag, deadline checks remain observe-only. With the - // flag, timeout IDs participate in the same topology consensus as terminal - // outcomes and request cancellation is requested on every nonterminal rank. + // Observe-only: WARN per-request when the wall-clock transfer time exceeds + // kvTransferTimeoutMs. No cancellation, eviction, or state transition. std::optional<int> kvTransferTimeoutMs = std::nullopt; if (mCacheTransceiverConfig.has_value()) { @@ -1064,15 +820,6 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( toCompleteIdSet.insert(request->mRequestId); } - auto recordTimeout = [&](RequestIdType const requestId) - { - bool const inserted = mTimedOutSenderIds.insert(requestId).second; - if (inserted && inflightCancelEnabled && mContextTransferCoordinator) - { - mContextTransferCoordinator->publishTimeout(requestId); - } - return inserted; - }; auto recordOutcome = [&](RequestIdType const requestId, std::shared_ptr<LlmRequest> const& request, bool const failed) { @@ -1091,20 +838,17 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( { auto& [request, future] = *it; auto const requestId = request->mRequestId; - if (kvTransferTimeoutMs.has_value() - && future.wait_for(std::chrono::milliseconds(0)) != std::future_status::ready) + if (kvTransferTimeoutMs.has_value()) { - auto const elapsedMs = getTransferElapsedMs(request, LlmRequest::getSteadyClockNow()); - if (elapsedMs > kvTransferTimeoutMs.value()) + auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>( + LlmRequest::getSteadyClockNow() - request->getKvCacheTransferStart()); + auto elapsedMs = static_cast<long>(elapsed.count()); + if (elapsedMs > kvTransferTimeoutMs.value() && mTimedOutSenderIds.insert(requestId).second) { - if (recordTimeout(requestId)) - { - TLLM_LOG_WARNING( - "Context KV cache transfer for request %ld exceeded configured timeout: " - "elapsed %ld ms > limit %d ms (%s).", - requestId, elapsedMs, kvTransferTimeoutMs.value(), - inflightCancelEnabled ? "requesting cancellation" : "observe-only"); - } + TLLM_LOG_WARNING( + "Context KV cache transfer for request %ld exceeded configured timeout: " + "elapsed %ld ms > limit %d ms (observe-only).", + requestId, elapsedMs, kvTransferTimeoutMs.value()); } } if (blockAll || (toCompleteIdSet.find(requestId) != toCompleteIdSet.end())) @@ -1117,18 +861,6 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( if (status == std::future_status::ready) { future.get(); - if (kvTransferTimeoutMs.has_value()) - { - auto const elapsedMs = getTransferElapsedMs(request, request->getKvCacheTransferEnd()); - if (elapsedMs > kvTransferTimeoutMs.value() && recordTimeout(requestId)) - { - TLLM_LOG_WARNING( - "Context KV cache transfer for request %ld completed after its deadline: " - "elapsed %ld ms > limit %d ms (%s).", - requestId, elapsedMs, kvTransferTimeoutMs.value(), - inflightCancelEnabled ? "failing request" : "observe-only"); - } - } failed = request->getState() == LlmRequestState::kDISAGG_TRANS_ERROR; terminal = true; } @@ -1185,8 +917,6 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( coordinatorOutcome.completedRequestIds.begin(), coordinatorOutcome.completedRequestIds.end()); consensusOutcome.failedRequestIds.insert( coordinatorOutcome.failedRequestIds.begin(), coordinatorOutcome.failedRequestIds.end()); - consensusOutcome.timedOutRequestIds.insert( - coordinatorOutcome.timedOutRequestIds.begin(), coordinatorOutcome.timedOutRequestIds.end()); }; do { @@ -1200,46 +930,13 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( } while (blockAll && consensusOutcome.completedRequestIds.size() + consensusOutcome.failedRequestIds.size() < mSenderRequestsAwaitingConsensus.size()); - - if (inflightCancelEnabled) - { - // A timeout update is transmitted once, but cancellation may be declined transiently. Keep the globally - // observed timeout active so rank-local cancellation is retried on every poll until terminal commit. - consensusOutcome.timedOutRequestIds.insert(mTimedOutSenderIds.begin(), mTimedOutSenderIds.end()); - } } else { - consensusOutcome = reduceTransferStates(syncComm, mGroupPipeParaComm, mCompletedSenderRequestIds, - mFailedSenderRequestIds, inflightCancelEnabled ? mTimedOutSenderIds : std::unordered_set<RequestIdType>{}); + consensusOutcome + = reduceTransferStates(syncComm, mGroupPipeParaComm, mCompletedSenderRequestIds, mFailedSenderRequestIds); } - if (inflightCancelEnabled) - { - for (auto const requestId : consensusOutcome.timedOutRequestIds) - { - // Persist the global timeout even if this rank has not registered its local future yet. The one-shot - // coordinator update must remain actionable when that future appears on a later scheduler poll. - mTimedOutSenderIds.insert(requestId); - auto const futureIt = std::find_if(mSenderFutures.begin(), mSenderFutures.end(), - [requestId](auto const& entry) { return entry.first->mRequestId == requestId; }); - if (futureIt == mSenderFutures.end() - || futureIt->second.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready - || mCancelRequestedSenderIds.find(requestId) != mCancelRequestedSenderIds.end()) - { - continue; - } - if (requestCancellationNoThrow( - requestId, "Context", [&]() { return mCacheSender->cancelRequest(*futureIt->first); })) - { - mCancelRequestedSenderIds.insert(requestId); - } - else - { - TLLM_LOG_DEBUG("Context cancellation for request %ld was not accepted; will retry", requestId); - } - } - } - for (auto const requestId : sortedRequestIds(consensusOutcome.failedRequestIds)) + for (auto const requestId : consensusOutcome.failedRequestIds) { auto const requestIt = mSenderRequestsAwaitingConsensus.find(requestId); if (requestIt == mSenderRequestsAwaitingConsensus.end()) @@ -1249,11 +946,10 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( requestIt->second->setState(LlmRequestState::kDISAGG_TRANS_ERROR); requestsStatus.errorRequestIds.insert(requestId); mTimedOutSenderIds.erase(requestId); - mCancelRequestedSenderIds.erase(requestId); eraseLocalTransferOutcome( requestId, mCompletedSenderRequestIds, mFailedSenderRequestIds, mSenderRequestsAwaitingConsensus); } - for (auto const requestId : sortedRequestIds(consensusOutcome.completedRequestIds)) + for (auto const requestId : consensusOutcome.completedRequestIds) { auto const requestIt = mSenderRequestsAwaitingConsensus.find(requestId); if (requestIt == mSenderRequestsAwaitingConsensus.end()) @@ -1266,7 +962,6 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( requestIt->second->setState(LlmRequestState::kDISAGG_CONTEXT_COMPLETE); } mTimedOutSenderIds.erase(requestId); - mCancelRequestedSenderIds.erase(requestId); eraseLocalTransferOutcome( requestId, mCompletedSenderRequestIds, mFailedSenderRequestIds, mSenderRequestsAwaitingConsensus); } @@ -1277,9 +972,6 @@ RequestStatuses CacheTransceiver::checkContextTransferStatus( void CacheTransceiver::checkGenTransferStatus(std::optional<int> const& atLeastRequestNum) { bool const blockAll = !atLeastRequestNum.has_value(); - bool const inflightCancelEnabled = common::getEnvDisaggEnableInflightCancel(); - TLLM_CHECK_WITH_INFO(!inflightCancelEnabled || !blockAll, - "In-flight cancellation requires a finite generation-transfer status poll; pass 0 for a nonblocking poll."); bool const needsProgress = atLeastRequestNum.value_or(0) > 0; std::optional<int> genTransferPollIntervalMs = std::nullopt; if (mCacheTransceiverConfig.has_value()) @@ -1369,7 +1061,7 @@ void CacheTransceiver::checkGenTransferStatus(std::optional<int> const& atLeastR atLeastRequestNum.value_or(0)); } - // Gen-side mirror of the context deadline/consensus path. + // Observe-only: gen-side mirror of the context-side timeout WARN. std::optional<int> kvTransferTimeoutMs = std::nullopt; if (mCacheTransceiverConfig.has_value()) { @@ -1379,20 +1071,17 @@ void CacheTransceiver::checkGenTransferStatus(std::optional<int> const& atLeastR { auto& request = it->first; auto const requestId = request->mRequestId; - if (kvTransferTimeoutMs.has_value() - && it->second.wait_for(std::chrono::milliseconds(0)) != std::future_status::ready) + if (kvTransferTimeoutMs.has_value()) { - auto const elapsedMs = getTransferElapsedMs(request, LlmRequest::getSteadyClockNow()); - if (elapsedMs > kvTransferTimeoutMs.value()) + auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>( + LlmRequest::getSteadyClockNow() - request->getKvCacheTransferStart()); + auto elapsedMs = static_cast<long>(elapsed.count()); + if (elapsedMs > kvTransferTimeoutMs.value() && mTimedOutRequesterIds.insert(requestId).second) { - if (mTimedOutRequesterIds.insert(requestId).second) - { - TLLM_LOG_WARNING( - "Generation KV cache transfer for request %ld exceeded configured timeout: " - "elapsed %ld ms > limit %d ms (%s).", - requestId, elapsedMs, kvTransferTimeoutMs.value(), - inflightCancelEnabled ? "requesting cancellation" : "observe-only"); - } + TLLM_LOG_WARNING( + "Generation KV cache transfer for request %ld exceeded configured timeout: " + "elapsed %ld ms > limit %d ms (observe-only).", + requestId, elapsedMs, kvTransferTimeoutMs.value()); } } if (blockAll || toCompleteIdSet.find(requestId) != toCompleteIdSet.end()) @@ -1403,19 +1092,14 @@ void CacheTransceiver::checkGenTransferStatus(std::optional<int> const& atLeastR if (status == std::future_status::ready) { it->second.get(); - if (kvTransferTimeoutMs.has_value()) + bool const failed = request->getState() == LlmRequestState::kDISAGG_TRANS_ERROR; + if (failed) { - auto const elapsedMs = getTransferElapsedMs(request, request->getKvCacheTransferEnd()); - if (elapsedMs > kvTransferTimeoutMs.value() && mTimedOutRequesterIds.insert(requestId).second) - { - TLLM_LOG_WARNING( - "Generation KV cache transfer for request %ld completed after its deadline: " - "elapsed %ld ms > limit %d ms (%s).", - requestId, elapsedMs, kvTransferTimeoutMs.value(), - inflightCancelEnabled ? "failing request" : "observe-only"); - } + // The receiver uses the error state as a local transfer-failed signal. + // Keep that signal local until the consensus outcome commits it globally. + request->setState(LlmRequestState::kDISAGG_GENERATION_TRANS_IN_PROGRESS); } - recordLocalTransferOutcome(requestId, request, /*failed=*/false, mCompletedRequesterRequestIds, + recordLocalTransferOutcome(requestId, request, failed, mCompletedRequesterRequestIds, mFailedRequesterRequestIds, mRequesterRequestsAwaitingConsensus); } else if (status == std::future_status::timeout) @@ -1440,12 +1124,6 @@ void CacheTransceiver::checkGenTransferStatus(std::optional<int> const& atLeastR recordLocalTransferOutcome(requestId, request, /*failed=*/true, mCompletedRequesterRequestIds, mFailedRequesterRequestIds, mRequesterRequestsAwaitingConsensus); } - catch (...) - { - TLLM_LOG_ERROR("Unknown error occurred during generation transfer for request %ld", requestId); - recordLocalTransferOutcome(requestId, request, /*failed=*/true, mCompletedRequesterRequestIds, - mFailedRequesterRequestIds, mRequesterRequestsAwaitingConsensus); - } if (useMPI()) { TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), @@ -1467,33 +1145,8 @@ void CacheTransceiver::checkGenTransferStatus(std::optional<int> const& atLeastR } auto const consensusOutcome - = reduceTransferStates(syncComm, mCompletedRequesterRequestIds, mFailedRequesterRequestIds, - inflightCancelEnabled ? mTimedOutRequesterIds : std::unordered_set<RequestIdType>{}); - if (inflightCancelEnabled) - { - for (auto const requestId : consensusOutcome.timedOutRequestIds) - { - auto const futureIt = std::find_if(mRequesterFutures.begin(), mRequesterFutures.end(), - [requestId](auto const& entry) { return entry.first->mRequestId == requestId; }); - if (futureIt == mRequesterFutures.end() - || futureIt->second.wait_for(std::chrono::milliseconds(0)) == std::future_status::ready - || mCancelRequestedRequesterIds.find(requestId) != mCancelRequestedRequesterIds.end()) - { - continue; - } - mTimedOutRequesterIds.insert(requestId); - if (requestCancellationNoThrow( - requestId, "Generation", [&]() { return mCacheReceiver->cancelRequest(*futureIt->first); })) - { - mCancelRequestedRequesterIds.insert(requestId); - } - else - { - TLLM_LOG_DEBUG("Generation cancellation for request %ld was not accepted; will retry", requestId); - } - } - } - for (auto const requestId : sortedRequestIds(consensusOutcome.failedRequestIds)) + = reduceTransferStates(syncComm, mCompletedRequesterRequestIds, mFailedRequesterRequestIds); + for (auto const requestId : consensusOutcome.failedRequestIds) { auto const requestIt = mRequesterRequestsAwaitingConsensus.find(requestId); if (requestIt == mRequesterRequestsAwaitingConsensus.end()) @@ -1502,15 +1155,10 @@ void CacheTransceiver::checkGenTransferStatus(std::optional<int> const& atLeastR } requestIt->second->setState(LlmRequestState::kDISAGG_TRANS_ERROR); mTimedOutRequesterIds.erase(requestId); - mCancelRequestedRequesterIds.erase(requestId); eraseLocalTransferOutcome( requestId, mCompletedRequesterRequestIds, mFailedRequesterRequestIds, mRequesterRequestsAwaitingConsensus); } - - // Collect consensus-completed requests so timing can be synced across ranks in a single - // batched allgather (instead of one collective per request). - std::vector<LlmRequest*> completedRequests; - for (auto const requestId : sortedRequestIds(consensusOutcome.completedRequestIds)) + for (auto const requestId : consensusOutcome.completedRequestIds) { auto const requestIt = mRequesterRequestsAwaitingConsensus.find(requestId); if (requestIt == mRequesterRequestsAwaitingConsensus.end()) @@ -1518,44 +1166,16 @@ void CacheTransceiver::checkGenTransferStatus(std::optional<int> const& atLeastR continue; } requestIt->second->setState(LlmRequestState::kDISAGG_GENERATION_TRANS_COMPLETE); - completedRequests.push_back(requestIt->second.get()); + + // Gather the kv cache transfer time from all workers and update to leader rank. + if (!common::getEnvKVCacheTimeOutputPath().empty()) + { + updateKVCacheTransferBW(syncComm, requestIt->second.get()); + } mTimedOutRequesterIds.erase(requestId); - mCancelRequestedRequesterIds.erase(requestId); eraseLocalTransferOutcome( requestId, mCompletedRequesterRequestIds, mFailedRequesterRequestIds, mRequesterRequestsAwaitingConsensus); } - - // Batch-sync timing across ranks in one allgather (instead of per-request), then write - // the gen-side transfer summary CSV. - if (!completedRequests.empty() && !common::getEnvKVCacheTimeOutputPath().empty()) - { - batchUpdateKVCacheTransferBW(syncComm, completedRequests); - writeGenTransferSummary(completedRequests); - } -} - -void CacheTransceiver::writeGenTransferSummary(std::vector<LlmRequest*> const& completedRequests) -{ - std::lock_guard<std::mutex> lock(mGenTransferSummaryMutex); - if (!mGenTransferSummaryFile.is_open()) - { - namespace fs = std::filesystem; - auto outputPath = fs::path(common::getEnvKVCacheTimeOutputPath()); - fs::create_directories(outputPath); - int rank = useMPI() ? mpi::MpiComm::world().getRank() : tensorrt_llm::pg_utils::get_world_pg()->getRank(); - auto filePath = outputPath / (mInstanceId + "_" + std::to_string(rank) + "_gen_transfer_summary.csv"); - mGenTransferSummaryFile.open(filePath); - TLLM_CHECK_WITH_INFO(mGenTransferSummaryFile.is_open(), "Failed to open gen transfer summary file: %s", - filePath.string().c_str()); - mGenTransferSummaryFile << "RequestID,gen_side_transfer_time(ms),kv_cache_size" << '\n'; - } - for (auto* req : completedRequests) - { - auto reqId = req->getContextPhaseParams().value().getReqId(); - mGenTransferSummaryFile << reqId << "," << req->getKvCacheTransferTimeMS() << "," << req->getKvCacheSize() - << '\n'; - } - mGenTransferSummaryFile << std::flush; } bool CacheTransceiver::checkGenTransferComplete() const @@ -1563,19 +1183,8 @@ bool CacheTransceiver::checkGenTransferComplete() const return mRequesterFutures.empty() && mCompletedRequesterRequestIds.empty() && mFailedRequesterRequestIds.empty(); } -bool CacheTransceiver::hasPoisonedTransferBuffer() const -{ - return std::any_of(mCacheTransBufferManagerPtrs.begin(), mCacheTransBufferManagerPtrs.end(), - [](BaseTransBufferManager const* manager) { return manager != nullptr && manager->hasPoisonedBuffer(); }); -} - bool CacheTransceiver::cancelRequest(std::shared_ptr<LlmRequest> llmRequest) { - if (llmRequest == nullptr) - { - TLLM_LOG_WARNING("Cannot cancel a null KV cache transfer request"); - return false; - } if (llmRequest->isContextOnlyRequest()) { return mCacheSender->cancelRequest(*llmRequest); diff --git a/cpp/tensorrt_llm/batch_manager/cacheTransferLayer.cpp b/cpp/tensorrt_llm/batch_manager/cacheTransferLayer.cpp index b2e0a21537d1..c013fd75c6e4 100644 --- a/cpp/tensorrt_llm/batch_manager/cacheTransferLayer.cpp +++ b/cpp/tensorrt_llm/batch_manager/cacheTransferLayer.cpp @@ -21,7 +21,6 @@ #include "tensorrt_llm/batch_manager/rnnCacheFormatter.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/cache_transmission/agent_utils/connection.h" #include "tensorrt_llm/executor/cache_transmission/cacheSplitConcat.h" @@ -55,7 +54,8 @@ void CacheTransferLayer::validateSupport(executor::DataTransceiverState const& p if (mRnnFormatter && selfHasRnn) { - // Unified pool path (CppMambaHybridCacheManager) uses RnnCacheFormatter. + // Both slot-based (CppMambaCacheManager) and unified pool (CppMambaHybridCacheManager) + // paths now use RnnCacheFormatter. if (peerHasRnn) { TLLM_CHECK_WITH_INFO(mRnnFormatter->inquireSupport(mCacheState, peerState.getCacheState().value()), @@ -122,8 +122,7 @@ void CacheTransferLayer::unformat(TransferSession& session) const } void CacheTransferLayer::setRnnConfig(executor::kv_cache::CacheState::RnnModelConfig rnnModelConfig, - std::vector<SizeType32> rnnLayerNumPerPP, tensorrt_llm::DataType convStateDataType, - tensorrt_llm::DataType ssmStateDataType) + std::vector<SizeType32> rnnLayerNumPerPP, nvinfer1::DataType convStateDataType, nvinfer1::DataType ssmStateDataType) { mCacheState.setRnnConfig( std::move(rnnModelConfig), std::move(rnnLayerNumPerPP), convStateDataType, ssmStateDataType); diff --git a/cpp/tensorrt_llm/batch_manager/cacheTransferLayer.h b/cpp/tensorrt_llm/batch_manager/cacheTransferLayer.h index 48a171a65c9c..0506e98197e6 100644 --- a/cpp/tensorrt_llm/batch_manager/cacheTransferLayer.h +++ b/cpp/tensorrt_llm/batch_manager/cacheTransferLayer.h @@ -18,7 +18,6 @@ #pragma once #include "tensorrt_llm/batch_manager/rnnCacheFormatter.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/runtime/common.h" @@ -80,8 +79,8 @@ class CacheTransferLayer /// @brief Update the RNN config on the internal CacheState. /// Used by CppMambaHybridCacheManager path where RNN config is set after construction. void setRnnConfig(executor::kv_cache::CacheState::RnnModelConfig rnnModelConfig, - std::vector<SizeType32> rnnLayerNumPerPP, tensorrt_llm::DataType convStateDataType, - tensorrt_llm::DataType ssmStateDataType); + std::vector<SizeType32> rnnLayerNumPerPP, nvinfer1::DataType convStateDataType, + nvinfer1::DataType ssmStateDataType); [[nodiscard]] kv_cache_manager::BaseKVCacheManager* getCacheManager() const noexcept; diff --git a/cpp/tensorrt_llm/batch_manager/createNewDecoderRequests.cpp b/cpp/tensorrt_llm/batch_manager/createNewDecoderRequests.cpp index 04c3760be5c7..5c5d3e11a01c 100644 --- a/cpp/tensorrt_llm/batch_manager/createNewDecoderRequests.cpp +++ b/cpp/tensorrt_llm/batch_manager/createNewDecoderRequests.cpp @@ -33,7 +33,7 @@ #include "tensorrt_llm/runtime/utils/mpiUtils.h" #include "tensorrt_llm/runtime/utils/speculativeChoicesUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntimeBase.h> using namespace tensorrt_llm::runtime; @@ -93,7 +93,7 @@ void copySequenceLengths(RequestVector const& contextRequests, DecoderInputBuffe /// @brief Retrieve the embedding bias from the request. This potentially makes a copy of the tensor /// to the appropriate type if the input tensor does not match it. -[[nodiscard]] TensorPtr getEmbeddingBias(tensorrt_llm::DataType logitsType, TensorPtr const& tensor) +[[nodiscard]] TensorPtr getEmbeddingBias(nvinfer1::DataType logitsType, TensorPtr const& tensor) { // Check that embedding bias type is same as logits type. If so, we can return the tensor right away if (tensor->getDataType() == logitsType) @@ -102,7 +102,7 @@ void copySequenceLengths(RequestVector const& contextRequests, DecoderInputBuffe } // Support FP32 input for FP16 embedding bias (in the case of FP8 models) - if (tensor->getDataType() == tensorrt_llm::DataType::kFLOAT && logitsType == tensorrt_llm::DataType::kHALF) + if (tensor->getDataType() == nvinfer1::DataType::kFLOAT && logitsType == nvinfer1::DataType::kHALF) { // Do a deep copy of the tensor to the expected type TLLM_LOG_WARNING( @@ -133,10 +133,10 @@ void copySequenceLengths(RequestVector const& contextRequests, DecoderInputBuffe std::tuple<TensorPtr, std::vector<runtime::SamplingConfig>, std::vector<runtime::ITensor::SharedConstPtr>, std::vector<executor::LookaheadDecodingConfig>> CreateNewDecoderRequests::operator()(runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, - executor::DecodingConfig const& decodingConfig, RequestVector const& contextRequests, - tensorrt_llm::DataType logitsType, DecoderInputBuffers& inputBuffers, runtime::decoder::DecoderState& decoderState, - CudaStream const& runtimeStream, CudaStream const& decoderStream, SizeType32 maxSequenceLength, - SizeType32 beamWidth, OptionalRef<MedusaBuffers const> medusaBuffers) const + executor::DecodingConfig const& decodingConfig, RequestVector const& contextRequests, nvinfer1::DataType logitsType, + DecoderInputBuffers& inputBuffers, runtime::decoder::DecoderState& decoderState, CudaStream const& runtimeStream, + CudaStream const& decoderStream, SizeType32 maxSequenceLength, SizeType32 beamWidth, + OptionalRef<MedusaBuffers const> medusaBuffers) const { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); NVTX3_SCOPED_RANGE(CreateNewDecoderRequests); @@ -235,7 +235,7 @@ void initializeBeamSearch(DecodingInput& dJointInput, DecodingOutput& dJointOutp } void initializeEmbeddingBias(DecodingInput& dJointInput, SizeType32 batchSlot, - std::optional<TensorPtr> const& embeddingBias, tensorrt_llm::DataType logitsType, + std::optional<TensorPtr> const& embeddingBias, nvinfer1::DataType logitsType, runtime::ModelConfig const& modelConfig, BufferManager const& manager) { TensorPtr const embeddingBiasSlice = ITensor::slice(constPointerCast(dJointInput.embeddingBias), batchSlot, 1); @@ -631,7 +631,7 @@ void newRequestSpeculativeDecoding(DecodingInput& jointDecodingInput, DecodingOu std::tuple<std::vector<runtime::ITensor::SharedConstPtr>, std::vector<executor::LookaheadDecodingConfig>> CreateNewDecoderRequests::createDecoderRequests(RequestVector const& finishedContextRequests, TensorPtr const& inputIds, executor::DecodingConfig const& decodingConfig, runtime::decoder::DecoderState& decoderState, - tensorrt_llm::DataType logitsType, runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + nvinfer1::DataType logitsType, runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, runtime::CudaStream const& runtimeStream, runtime::CudaStream const& decoderStream, SizeType32 maxSequenceLength, OptionalRef<MedusaBuffers const> medusaBuffers) const { diff --git a/cpp/tensorrt_llm/batch_manager/dataTransceiver.cpp b/cpp/tensorrt_llm/batch_manager/dataTransceiver.cpp index 0f8ded65613f..93812056abf6 100644 --- a/cpp/tensorrt_llm/batch_manager/dataTransceiver.cpp +++ b/cpp/tensorrt_llm/batch_manager/dataTransceiver.cpp @@ -20,24 +20,21 @@ #include "tensorrt_llm/batch_manager/cacheFormatter.h" #include "tensorrt_llm/batch_manager/common.h" #include "tensorrt_llm/batch_manager/kvCacheUtils.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/common/tllmException.h" #include "tensorrt_llm/common/utils.h" #include "tensorrt_llm/executor/cache_transmission/agent_utils/connection.h" #include "tensorrt_llm/executor/cache_transmission/cacheSplitConcat.h" #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" -#include <algorithm> #include <chrono> #include <future> #include <map> #include <memory> #include <optional> -#include <stdexcept> #include <unordered_map> -#include <variant> namespace tensorrt_llm::batch_manager { @@ -82,11 +79,6 @@ void TransferSession::send(size_t idx, void const* data, size_t size) } catch (std::exception const& e) { - // Request-free (llmRequest-agnostic) transfer: there is no valid ID to attach. - if (mRequest == nullptr) - { - TLLM_THROW("%s", e.what()); - } throw common::RequestSpecificException( __FILE__, __LINE__, e.what(), mRequest->mRequestId, common::RequestErrorCode::kNETWORK_ERROR); } @@ -100,23 +92,15 @@ void TransferSession::recv(size_t idx, void* data, size_t size) } catch (std::exception const& e) { - // Request-free (llmRequest-agnostic) transfer: there is no valid ID to attach. - if (mRequest == nullptr) - { - TLLM_THROW("%s", e.what()); - } throw common::RequestSpecificException( __FILE__, __LINE__, e.what(), mRequest->mRequestId, common::RequestErrorCode::kNETWORK_ERROR); } } -std::optional<LlmRequest const*> TransferSession::getLlmRequest() const +LlmRequest const& TransferSession::getLlmRequest() const { - if (mRequest == nullptr) - { - return std::nullopt; - } - return mRequest; + TLLM_CHECK(mRequest != nullptr); + return *mRequest; } void TransferSession::setLlmRequest(LlmRequest const& llmRequest) @@ -140,53 +124,9 @@ void TransferSession::appendMeasure(LlmRequest::TimePoint start, LlmRequest::Tim } } -void TransferSession::setReservedRecvBuffers(std::vector<BufferIndexHolder> holders) -{ - TLLM_CHECK(mReservedRecvBuffers.empty()); - mReservedRecvBuffers = std::move(holders); -} - -bool TransferSession::hasReservedRecvBuffer(BaseTransBufferManager const& manager) const noexcept -{ - return std::any_of(mReservedRecvBuffers.begin(), mReservedRecvBuffers.end(), - [&manager](BufferIndexHolder const& holder) { return holder.isBoundTo(manager); }); -} - -bool TransferSession::releaseReservedRecvBuffer(BaseTransBufferManager const& manager) noexcept -{ - auto const holderIt = std::find_if(mReservedRecvBuffers.begin(), mReservedRecvBuffers.end(), - [&manager](BufferIndexHolder const& holder) { return holder.isBoundTo(manager); }); - if (holderIt == mReservedRecvBuffers.end()) - { - return false; - } - holderIt->release(); - mReservedRecvBuffers.erase(holderIt); - return true; -} - -void TransferSession::releaseReservedRecvBuffers() noexcept -{ - for (auto& holder : mReservedRecvBuffers) - { - holder.release(); - } - mReservedRecvBuffers.clear(); -} - -void TransferSession::poisonReservedRecvBuffers() noexcept -{ - for (auto& holder : mReservedRecvBuffers) - { - holder.poison(); - } - mReservedRecvBuffers.clear(); -} - void TransferSession::exportMeasure(std::ofstream& outFile, bool isContext) const { - // Request-free transfers are excluded: the exported row is keyed by the LlmRequest. - if (!mTimes || mTimes->measures.empty() || mRequest == nullptr) + if (!mTimes || mTimes->measures.empty()) { return; } @@ -244,7 +184,7 @@ int32_t tagFromRequestId(LlmRequest::RequestIdType requestId) return ((requestId & 0xFFF) << 8) | (kDATA_TAG & 0xFF); } -std::filesystem::path getTransferOutputPath(char const* tag, std::string const& instanceId = "") +std::filesystem::path getTransferOutputPath(char const* tag) { namespace fs = std::filesystem; auto outputPath = common::getEnvKVCacheTimeOutputPath(); @@ -253,9 +193,7 @@ std::filesystem::path getTransferOutputPath(char const* tag, std::string const& auto rank = mpi::MpiComm::world().getRank(); auto path = fs::path(outputPath); fs::create_directories(path); - std::string prefix - = instanceId.empty() ? "rank_" + std::to_string(rank) : instanceId + "_" + std::to_string(rank); - return path / (prefix + "_" + tag + ".csv"); + return path / ("rank_" + std::to_string(rank) + "_" + tag + ".csv"); } return {}; } @@ -292,7 +230,7 @@ RequestInfo::RequestInfo(LlmRequest::RequestIdType requestId, executor::DataTran bool RequestInfo::operator==(RequestInfo const& rhs) const { return mRequestId == rhs.mRequestId && mIndexFromEnd == rhs.mIndexFromEnd && mLastBlockKey == rhs.mLastBlockKey - && mIsArbitraryTransfer == rhs.mIsArbitraryTransfer && mTransState == rhs.mTransState; + && mTransState == rhs.mTransState; } LlmRequest::RequestIdType RequestInfo::getRequestId() const noexcept @@ -311,7 +249,6 @@ void RequestInfo::serialize(RequestInfo const& requestInfo, std::ostream& os) su::serialize(requestInfo.mRequestId, os); su::serialize(requestInfo.mIndexFromEnd, os); su::serialize(requestInfo.mLastBlockKey, os); - su::serialize(requestInfo.mIsArbitraryTransfer, os); su::serialize(requestInfo.mTransState, os); } @@ -321,11 +258,8 @@ RequestInfo RequestInfo::deserialize(std::istream& is) auto requestId = su::deserialize<decltype(mRequestId)>(is); auto indexFromEnd = su::deserialize<decltype(mIndexFromEnd)>(is); auto lastBlockKey = su::deserialize<decltype(mLastBlockKey)>(is); - auto isArbitraryTransfer = su::deserialize<decltype(mIsArbitraryTransfer)>(is); auto transState = su::deserialize<decltype(mTransState)>(is); - auto requestInfo = RequestInfo{requestId, std::move(transState), indexFromEnd, lastBlockKey}; - requestInfo.setIsArbitraryTransfer(isArbitraryTransfer); - return requestInfo; + return RequestInfo{requestId, std::move(transState), indexFromEnd, lastBlockKey}; } std::size_t RequestInfo::serializedSize(RequestInfo const& requestInfo) @@ -335,7 +269,6 @@ std::size_t RequestInfo::serializedSize(RequestInfo const& requestInfo) totalSize += su::serializedSize(requestInfo.mRequestId); totalSize += su::serializedSize(requestInfo.mIndexFromEnd); totalSize += su::serializedSize(requestInfo.mLastBlockKey); - totalSize += su::serializedSize(requestInfo.mIsArbitraryTransfer); totalSize += su::serializedSize(requestInfo.mTransState); return totalSize; } @@ -345,17 +278,16 @@ class CacheSender::Impl public: using RequestIdType = LlmRequest::RequestIdType; - Impl(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer, - std::string instanceId = "") + Impl(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer) : mManager{manager} , mSelfState{cacheLayer.getCacheState(), executor::kv_cache::CommState{manager->getCommState()}} , mCacheTransferLayer{std::move(cacheLayer)} , mBufferManager{std::make_shared<runtime::CudaStream>()} - , mInstanceId{std::move(instanceId)} { TLLM_CHECK(mManager); TLLM_CHECK(mManager->getCommState().getSelfIdx() == selfIndex); TLLM_CUDA_CHECK(cudaGetDevice(&mDeviceId)); + mCurrentRequest = std::nullopt; mResponseFuture = std::async(std::launch::async, &Impl::response, this); int asyncSendThreadNum = common::getEnvKVCacheSendMaxConcurrenceNum(); for (int i = 0; i < asyncSendThreadNum; i++) @@ -371,36 +303,18 @@ class CacheSender::Impl std::promise<void> promise; auto future = promise.get_future(); llmRequest->setKvCacheTransferStart(LlmRequest::getSteadyClockNow()); - if (common::getEnvDisaggEnableInflightCancel()) { - (void) getOrCreateInFlightCancelFlag(llmRequest->mRequestId); - } - { - std::scoped_lock lock(mSenderMutex); - TLLM_CHECK_WITH_INFO( - !mTerminate, "Cannot enqueue request %zu after CacheSender termination", llmRequest->mRequestId); - auto const result - = mReadyResponses.emplace(llmRequest->mRequestId, Response{llmRequest, std::move(promise)}); - TLLM_CHECK_WITH_INFO( - result.second, "Request %zu is already queued for KV cache transfer", llmRequest->mRequestId); + { + std::scoped_lock lkResp(mSenderMutex); + mReadyResponses.emplace(llmRequest->mRequestId, Response{llmRequest, std::move(promise)}); + } + std::unique_lock lkCond(mCondMutex); + mAnyReady = true; } mSenderCv.notify_all(); return future; } - std::shared_ptr<std::atomic<bool>> getOrCreateInFlightCancelFlag(RequestIdType requestId) - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - auto it = mInFlightCancelFlags.find(requestId); - if (it != mInFlightCancelFlags.end()) - { - return it->second; - } - auto flag = std::make_shared<std::atomic<bool>>(false); - mInFlightCancelFlags.emplace(requestId, flag); - return flag; - } - [[nodiscard]] executor::kv_cache::CommState const& getCommState() const { return mSelfState.getCommState().value(); @@ -421,62 +335,21 @@ class CacheSender::Impl void release(LlmRequest::RequestIdType requestId) { + std::unique_lock<std::mutex> lk(mMtxForMap); + auto it = mRequestToSession.find(requestId); + TLLM_CHECK(it != mRequestToSession.end()); + if (!common::getEnvKVCacheTimeOutputPath().empty()) { - std::unique_lock<std::mutex> lk(mMtxForMap); - auto it = mRequestToSession.find(requestId); - TLLM_CHECK(it != mRequestToSession.end()); - if (!common::getEnvKVCacheTimeOutputPath().empty()) + if (!mMeasuresFile.is_open()) { - if (!mMeasuresFile.is_open()) - { - auto outputPath = getTransferOutputPath("send", mInstanceId); - mMeasuresFile.open(outputPath); - TLLM_CHECK_WITH_INFO(mMeasuresFile.is_open(), "Failed to open transfer output file: %s", - outputPath.string().c_str()); - } - it->second.exportMeasure(mMeasuresFile, true); + auto outputPath = getTransferOutputPath("send"); + mMeasuresFile.open(outputPath); + TLLM_CHECK_WITH_INFO( + mMeasuresFile.is_open(), "Failed to open transfer output file: %s", outputPath.string().c_str()); } - mRequestToSession.erase(it); - } - if (common::getEnvDisaggEnableInflightCancel()) - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - mInFlightCancelFlags.erase(requestId); - } - } - - void discardTransferState(LlmRequest::RequestIdType requestId) noexcept - { - try - { - std::unique_lock<std::mutex> lk(mMtxForMap); - mRequestToSession.erase(requestId); - } - catch (std::exception const& e) - { - TLLM_LOG_WARNING("Failed to discard sender session for request %ld: %s", requestId, e.what()); - } - catch (...) - { - TLLM_LOG_WARNING("Failed to discard sender session for request %ld: unknown exception", requestId); - } - if (!common::getEnvDisaggEnableInflightCancel()) - { - return; - } - try - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - mInFlightCancelFlags.erase(requestId); - } - catch (std::exception const& e) - { - TLLM_LOG_WARNING("Failed to discard in-flight cancel flag for request %ld: %s", requestId, e.what()); - } - catch (...) - { - TLLM_LOG_WARNING("Failed to discard in-flight cancel flag for request %ld: unknown exception", requestId); + it->second.exportMeasure(mMeasuresFile, true); } + mRequestToSession.erase(it); } [[nodiscard]] std::optional<RequestInfo> recvRequestInfo() @@ -491,6 +364,7 @@ class CacheSender::Impl : mManager->recvConnect(DataContext{TransceiverTag::kID_TAG, mTerminate}, &id, sizeof(id)); if (connection == nullptr) { + TLLM_LOG_WARNING("recvRequestInfo connection is nullptr, maybe the server is terminating"); return std::nullopt; } @@ -518,25 +392,15 @@ class CacheSender::Impl TLLM_CHECK_WITH_INFO(peerIdx < static_cast<int>(allCounterparts.size()), "Peer rank %d not found in expected counterparts", peerSelfIdx); - std::shared_ptr<std::atomic<bool>> cancelFlag; - if (common::getEnvDisaggEnableInflightCancel()) - { - cancelFlag = getOrCreateInFlightCancelFlag(requestId); - } { std::unique_lock<std::mutex> lk(mMtxForMap); auto it = mRequestToSession.find(requestId); if (it == mRequestToSession.end()) { - auto session = cancelFlag != nullptr - ? TransferSession(std::vector<Connection const*>(allCounterparts.size(), nullptr), - DataContext{tagFromRequestId(requestId), *cancelFlag}, allCounterparts, mSelfState, - info.getTransState(), mBufferManager, info.getIndexFromEnd(), info.getLastBlockKey(), nullptr, - !common::getEnvKVCacheTimeOutputPath().empty()) - : TransferSession(std::vector<Connection const*>(allCounterparts.size(), nullptr), - DataContext{tagFromRequestId(requestId), mTerminate}, allCounterparts, mSelfState, - info.getTransState(), mBufferManager, info.getIndexFromEnd(), info.getLastBlockKey(), nullptr, - !common::getEnvKVCacheTimeOutputPath().empty()); + auto session = TransferSession(std::vector<Connection const*>(allCounterparts.size(), nullptr), + DataContext{tagFromRequestId(requestId), mTerminate}, allCounterparts, mSelfState, + info.getTransState(), mBufferManager, info.getIndexFromEnd(), info.getLastBlockKey(), nullptr, + !common::getEnvKVCacheTimeOutputPath().empty()); session.setTime(TransferSession::kTimeRequestInfo); it = mRequestToSession.emplace(requestId, std::move(session)).first; } @@ -555,60 +419,25 @@ class CacheSender::Impl session = std::addressof(it->second); } session->setLlmRequest(llmRequest); - TLLM_LOG_DEBUG("KV cache transfer request %zu phase=transfer-submit begin.", llmRequest.mRequestId); mCacheTransferLayer.format(*session); - TLLM_LOG_DEBUG("KV cache transfer request %zu phase=transfer-complete end.", llmRequest.mRequestId); llmRequest.setKvCacheTransferEnd(LlmRequest::getSteadyClockNow()); } bool cancelRequest(LlmRequest const& llmRequest) { - bool const inflightCancelEnabled = common::getEnvDisaggEnableInflightCancel(); bool isCancelled = false; - bool isCurrentRequest = false; + std::scoped_lock lkResp(mSenderMutex); + auto it = mReadyResponses.find(llmRequest.mRequestId); + // If the request is not the current request and already in the ready queue, we can cancel it. + if (it != mReadyResponses.end() + && (!mCurrentRequest.has_value() || getCurrentRequestId() != llmRequest.mRequestId)) { - std::scoped_lock lock(mSenderMutex); - auto it = mReadyResponses.find(llmRequest.mRequestId); - if (it != mReadyResponses.end()) - { - isCurrentRequest = mCurrentRequest.has_value() && mCurrentRequest.value() == llmRequest.mRequestId; - // The legacy path cannot interrupt a ready/active transfer, so - // preserve its false return until the opt-in is enabled. - if (!isCurrentRequest || inflightCancelEnabled) - { - mCancelledRequests.insert(llmRequest.mRequestId); - isCancelled = true; - if (inflightCancelEnabled && !isCurrentRequest) - { - // Keep only the request ID as a tombstone so a late peer - // receives ready=false without retaining the request. - failResponse(it->second, - std::make_exception_ptr( - TLLM_REQUEST_EXCEPTION(llmRequest.mRequestId, common::RequestErrorCode::kNETWORK_ERROR, - "Context KV cache request cancelled before a peer was ready for request %zu", - llmRequest.mRequestId))); - mReadyResponses.erase(it); - } - } - } - } - if (inflightCancelEnabled && (!isCancelled || isCurrentRequest)) - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - auto flagIt = mInFlightCancelFlags.find(llmRequest.mRequestId); - if (flagIt != mInFlightCancelFlags.end()) - { - flagIt->second->store(true, std::memory_order_relaxed); - isCancelled = true; - } - } - if (!isCancelled) - { - TLLM_LOG_WARNING("Cannot cancel request %zu", llmRequest.mRequestId); + mCancelledRequests.insert(llmRequest.mRequestId); + isCancelled = true; } else { - mSenderCv.notify_all(); + TLLM_LOG_WARNING("Cannot cancel request %zu", llmRequest.mRequestId); } return isCancelled; } @@ -630,7 +459,8 @@ class CacheSender::Impl { auto* agentConnection = dynamic_cast<executor::kv_cache::AgentConnection const*>(connections.at(i)); TLLM_CHECK(agentConnection); - agentConnection->sendReadySignal(session->getDataContext(), isReady); + agentConnection->sendReadySignal( + executor::kv_cache::DataContext{TransceiverTag::kREADY_SIGNAL_TAG}, isReady); } else { @@ -648,23 +478,10 @@ class CacheSender::Impl private: struct Response { - // An LlmRequest (co-owned until the promise resolves) for normal transfers, or - // just the request id for llmRequest-agnostic reuse-tree transfers. - std::variant<std::shared_ptr<LlmRequest>, RequestIdType> mRequestOrId; + // shared_ptr so this struct co-owns the request until the promise resolves; + // protects worker-side dereferences and the promise itself from premature destruction. + std::shared_ptr<LlmRequest> mRequest; std::promise<void> mPromise; - std::vector<kv_cache_manager::KVCacheBlock::IdType> mPinnedBlockIds; - - [[nodiscard]] LlmRequest* getRequest() const - { - auto const* request = std::get_if<std::shared_ptr<LlmRequest>>(&mRequestOrId); - return request != nullptr ? request->get() : nullptr; - } - - [[nodiscard]] RequestIdType getRequestId() const - { - auto const* request = getRequest(); - return request != nullptr ? request->mRequestId : std::get<RequestIdType>(mRequestOrId); - } }; struct AsyncSendResource @@ -697,398 +514,234 @@ class CacheSender::Impl resp = std::move(resource.mSendQueue.front()); resource.mSendQueue.pop_front(); } - // Read before std::move(resp): argument evaluations are indeterminately sequenced. - auto const requestId = resp.getRequestId(); - sendAndRemoveResponse(requestId, std::move(resp)); + // Sequence the read before the move: argument initializations + // are indeterminately sequenced, so inlining resp.mRequest->... + // alongside std::move(resp) is UB once mRequest is a shared_ptr. + TLLM_CHECK(resp.mRequest != nullptr); + auto const reqId = resp.mRequest->mRequestId; + sendAndRemoveResponse(reqId, std::move(resp)); } } - //! Must not throw: called from noexcept send/failure paths. - void releasePinnedBlocks(Response& response) noexcept - { - if (response.mPinnedBlockIds.empty()) - { - return; - } - try - { - mCacheTransferLayer.getCacheManager()->unpinBlocksById(response.mPinnedBlockIds); - } - catch (std::exception const& err) - { - TLLM_LOG_ERROR("Failed to unpin reuse-tree blocks: %s", err.what()); - } - response.mPinnedBlockIds.clear(); - } - void sendAndRemoveResponse(RequestIdType id, Response resp) noexcept { try { TLLM_CUDA_CHECK(cudaSetDevice(mDeviceId)); - if (auto* llmRequest = resp.getRequest(); llmRequest != nullptr) - { - sendSync(*llmRequest); - } - else - { - // Reuse tree path — no LlmRequest - sendSyncFromReuseTree(id); - } + sendSync(*resp.mRequest); release(id); resp.mPromise.set_value(); } catch (tensorrt_llm::common::RequestSpecificException const& e) { TLLM_LOG_ERROR("Exception in sendAndRemoveResponse: %s ", e.what()); - discardTransferState(id); auto new_exception = TLLM_REQUEST_EXCEPTION(id, e.getErrorCode(), "%s", e.what()); - failResponse(resp, std::make_exception_ptr(new_exception)); + resp.mPromise.set_exception(std::make_exception_ptr(new_exception)); } catch (std::exception const& e) { - auto const exception = std::current_exception(); TLLM_LOG_ERROR("Exception in sendAndRemoveResponse: %s request id: %ld", e.what(), id); - discardTransferState(id); - failResponse(resp, exception); + resp.mPromise.set_exception(std::current_exception()); } - catch (...) - { - auto const exception = std::current_exception(); - TLLM_LOG_ERROR("Unknown exception in sendAndRemoveResponse for request id: %ld", id); - discardTransferState(id); - failResponse(resp, exception); - } - releasePinnedBlocks(resp); } void asyncSendAndRemoveResponse(RequestIdType id, Response resp) noexcept { - try - { - std::unique_lock lock(mAsyncSendResource.mMtxForQueue); - mAsyncSendResource.mSendQueue.emplace_back(std::move(resp)); - mAsyncSendResource.mCVforQueue.notify_one(); - } - catch (std::exception const& err) - { - TLLM_LOG_ERROR("Failed to queue asynchronous KV cache send for request %zu: %s", id, err.what()); - discardTransferState(id); - failResponse(resp, std::current_exception()); - } - catch (...) - { - TLLM_LOG_ERROR("Unknown error while queueing asynchronous KV cache send for request %zu", id); - discardTransferState(id); - failResponse(resp, std::current_exception()); - } + std::unique_lock lk(mAsyncSendResource.mMtxForQueue); + mAsyncSendResource.mSendQueue.emplace_back(std::move(resp)); + mAsyncSendResource.mCVforQueue.notify_one(); } - void sendResponse(RequestIdType reqId) + void sendResponse(std::map<RequestIdType, CacheSender::Impl::Response>::iterator it) { - bool isReady = true; - bool allCounterpartsReady = false; - std::optional<Response> cancelledResponse; + auto reqId = mCurrentRequest.value(); + auto count = --mRemainSendCount[reqId]; + TLLM_CHECK(count >= 0); + if (count == 0) { - std::scoped_lock lock(mSenderMutex); - TLLM_CHECK(mCurrentRequest.has_value() && mCurrentRequest.value() == reqId); - auto responseIt = mReadyResponses.find(reqId); - bool const isCancelled = mCancelledRequests.find(reqId) != mCancelledRequests.end(); - TLLM_CHECK(responseIt != mReadyResponses.end() || isCancelled); - auto countIt = mRemainSendCount.find(reqId); - TLLM_CHECK(countIt != mRemainSendCount.end()); - auto const count = --countIt->second; - TLLM_CHECK(count >= 0); - if (isCancelled && responseIt != mReadyResponses.end()) - { - cancelledResponse.emplace(std::move(responseIt->second)); - mReadyResponses.erase(responseIt); - } - if (count > 0) - { - mCurrentRequest = std::nullopt; - } - else - { - mRemainSendCount.erase(countIt); - isReady = !isCancelled; - allCounterpartsReady = true; - } - } + mRemainSendCount.erase(reqId); - if (cancelledResponse.has_value()) - { - failResponse(*cancelledResponse, - std::make_exception_ptr(TLLM_REQUEST_EXCEPTION(reqId, common::RequestErrorCode::kNETWORK_ERROR, - "KV cache transfer for request %zu was cancelled", reqId))); - } - if (!allCounterpartsReady) - { - return; - } - - // Keep mCurrentRequest set while notifying the peer so cancellation cannot change the decision after it has - // been made. The network operation must not run under mSenderMutex. - sendReadySignal(reqId, isReady); - - Response response; - { - std::scoped_lock lock(mSenderMutex); - auto it = mReadyResponses.find(reqId); - if (isReady) + // Check if the request is cancelled + bool isReady = true; { - TLLM_CHECK(it != mReadyResponses.end()); - response = std::move(it->second); - mReadyResponses.erase(it); + std::scoped_lock lk(mSenderMutex); + if (mCancelledRequests.find(reqId) != mCancelledRequests.end()) + { + isReady = false; + } } - mCancelledRequests.erase(reqId); - mCurrentRequest = std::nullopt; - } + sendReadySignal(reqId, isReady); - if (isReady) - { - if (dynamic_cast<executor::kv_cache::AgentConnectionManager*>(mManager) != nullptr) + if (isReady) { - // Our NIXL implementation only supports recv and send in the same thread. Using ZMQ for the control - // path may avoid this limitation. - sendAndRemoveResponse(reqId, std::move(response)); + if (dynamic_cast<executor::kv_cache::AgentConnectionManager*>(mManager) != nullptr) + { + // our nixl impl seems only support recv and send in the same thread + // if we use zmq as control path, we may avoid this issue + sendAndRemoveResponse(it->first, std::move(it->second)); + } + else + { + // if we send data in another thread, multiple rank may send data for different requests at the same + // time with gen DP case. + asyncSendAndRemoveResponse(it->first, std::move(it->second)); + } + removeResponse(it); } else { - // If we send data in another thread, multiple ranks may send data for different requests at the same - // time with generation attention DP. - asyncSendAndRemoveResponse(reqId, std::move(response)); - } - } - else - { - discardTransferState(reqId); - } - } - - void sendSyncFromReuseTree(RequestIdType requestId) - { - TransferSession* session = nullptr; - { - std::unique_lock<std::mutex> lk(mMtxForMap); - auto it = mRequestToSession.find(requestId); - TLLM_CHECK(it != mRequestToSession.end()); - session = std::addressof(it->second); - } - // READY was already sent by response(); the receiver consumes exactly one per transfer. - mCacheTransferLayer.format(*session); - } + // TODO: if the generation does not require the kv cache, the request will + // not be removed from mCancelledRequests. This should be handled by timeout. + auto const cancelledReqId = mCurrentRequest.value(); + Response cancelledResponse; + { + std::scoped_lock lkResp(mSenderMutex); + auto it = mReadyResponses.find(cancelledReqId); + TLLM_CHECK(it != mReadyResponses.end()); + // Move out before erasing so the promise survives the + // map cleanup and can be resolved (vs. destroyed unfulfilled, + // which would surface as std::future_error: Broken promise). + cancelledResponse = std::move(it->second); + mReadyResponses.erase(it); + mCancelledRequests.erase(cancelledReqId); + mRemainSendCount.erase(cancelledReqId); + } + cancelledResponse.mPromise.set_exception(std::make_exception_ptr( + TLLM_REQUEST_EXCEPTION(cancelledReqId, common::RequestErrorCode::kNETWORK_ERROR, + "KV cache transfer for request %zu was cancelled", cancelledReqId))); + mCurrentRequest = std::nullopt; - // Pin the requested chain in the reuse tree; an empty result means no full match. - // The caller must unpin once the transfer settles. - std::vector<kv_cache_manager::KVCacheBlock::IdType> pinReuseTreeBlocks(RequestIdType requestId) - { - std::unique_lock<std::mutex> lk(mMtxForMap); - auto it = mRequestToSession.find(requestId); - auto const& lastBlockKey = it->second.getLastBlockKey(); - auto* cacheManager = mCacheTransferLayer.getCacheManager(); - auto windowSize = cacheManager->getBlockManager().getWindowSizesMetadata().begin()->first; - std::vector<kv_cache_manager::KVCacheBlock::IdType> pinnedIds; - auto lastBlock = cacheManager->findBlocksInReuseTreeByBlockKey(lastBlockKey, windowSize, pinnedIds); - if (lastBlock == nullptr) - { - return {}; + if (mReadyResponses.empty()) + { + std::unique_lock lk(mCondMutex); + mAnyReady = false; + } + } } - return pinnedIds; + mCurrentRequest = std::nullopt; } void response() noexcept { - std::exception_ptr responseException; try { tensorrt_llm::common::setThreadName("dataTransResp"); TLLM_CUDA_CHECK(cudaSetDevice(mDeviceId)); - while (true) + while (!mTerminate || !mAnyReady) { - if (mTerminate) + if (!mAnyReady) { - break; + std::unique_lock lk(mCondMutex); + mSenderCv.wait(lk, [this]() { return (mAnyReady || mTerminate); }); } - - // Arbitrary transfers arrive without a pre-registered response; do not gate on - // mReadyResponses. - auto requestInfo = recvRequestInfo(); - if (!requestInfo.has_value() || mTerminate || !mManager->isRunning()) + if (mTerminate) { break; } - auto const reqId = requestInfo->getRequestId(); - - if (mRemainSendCount.find(reqId) == mRemainSendCount.end()) - { - mRemainSendCount[reqId] = getCounterpartsCount(reqId); - } - - if (requestInfo->isArbitraryTransfer()) + if (!mReadyResponses.empty()) { - // No LlmRequest will ever be registered; serve from the reuse tree off-thread. + auto requestInfo = recvRequestInfo(); + if (!requestInfo.has_value() || mTerminate || !mManager->isRunning()) { - std::scoped_lock lock(mSenderMutex); - mCurrentRequest = reqId; + return; } - auto countIt = mRemainSendCount.find(reqId); - auto const count = --countIt->second; - TLLM_CHECK(count >= 0); - if (count == 0) + auto reqId = requestInfo->getRequestId(); + { - mRemainSendCount.erase(countIt); - auto pinnedIds = pinReuseTreeBlocks(reqId); - if (pinnedIds.empty()) - { - TLLM_LOG_ERROR( - "Requested blocks do not exist in the source's reuse tree (request id: %lu). Notifying " - "receiver.", - reqId); - sendReadySignal(reqId, false); - discardTransferState(reqId); - } - else - { - sendReadySignal(reqId, true); - std::promise<void> promise; - // Id-only response: the reuse-tree path has no LlmRequest. - Response resp{reqId, std::move(promise), std::move(pinnedIds)}; - if (dynamic_cast<executor::kv_cache::AgentConnectionManager*>(mManager) != nullptr) - { - sendAndRemoveResponse(reqId, std::move(resp)); - } - else - { - asyncSendAndRemoveResponse(reqId, std::move(resp)); - } - } + std::scoped_lock lk(mSenderMutex); + mCurrentRequest = reqId; } + + if (mRemainSendCount.find(reqId) == mRemainSendCount.end()) { - std::scoped_lock lock(mSenderMutex); - mCurrentRequest = std::nullopt; + mRemainSendCount[reqId] = getCounterpartsCount(reqId); } } + auto it = getCurrentResponse(); + if (it != mReadyResponses.end()) + { + sendResponse(it); + } else { - // The RequestInfo may race ahead of sendAsync; wait for the specific response. + auto it = getCurrentResponse(); + while (it == mReadyResponses.end()) { - std::unique_lock lock(mSenderMutex); - mCurrentRequest = reqId; - mSenderCv.wait(lock, - [this, reqId]() - { - return mTerminate || mReadyResponses.find(reqId) != mReadyResponses.end() - || mCancelledRequests.find(reqId) != mCancelledRequests.end(); - }); + std::unique_lock lk(mCondMutex); + mSenderCv.wait(lk, [this]() { return (mAnyReady || mTerminate); }); if (mTerminate) { - mCurrentRequest = std::nullopt; break; } + it = getCurrentResponse(); + } + if (mTerminate || it == mReadyResponses.end()) + { + break; } - sendResponse(reqId); + sendResponse(it); } } } catch (std::exception const& err) { TLLM_LOG_ERROR("Exception in CacheSender response: %s", err.what()); - responseException = std::current_exception(); - } - catch (...) - { - TLLM_LOG_ERROR("Unknown exception in CacheSender response"); - responseException = std::current_exception(); - } - - if (!responseException) - { - responseException - = std::make_exception_ptr(std::runtime_error("CacheSender terminated before response completed")); - } - { - std::scoped_lock lock(mSenderMutex); - mTerminate = true; + for (auto& it : mReadyResponses) + { + it.second.mPromise.set_exception(std::current_exception()); + } } - mSenderCv.notify_all(); - failPendingResponses(responseException); } void terminate() { { - std::scoped_lock lock(mSenderMutex); + std::unique_lock lk(mCondMutex); mTerminate = true; } - if (common::getEnvDisaggEnableInflightCancel()) - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - for (auto& [id, flag] : mInFlightCancelFlags) - { - flag->store(true, std::memory_order_relaxed); - } - } - // Wake the sender loop and make in-flight agent transfers observe termination through - // their per-request cancellation flags. + // We don't have to wait for the future. If another thread is sending data, it won't pay attention + // to the terminate flag. mSenderCv.notify_all(); - if (mResponseFuture.valid()) - { - mResponseFuture.get(); - } - - std::deque<Response> pendingAsyncResponses; - { - std::scoped_lock lock(mAsyncSendResource.mMtxForQueue); - mAsyncSendResource.mTerminate = true; - pendingAsyncResponses.swap(mAsyncSendResource.mSendQueue); - } + mAsyncSendResource.mTerminate = true; mAsyncSendResource.mCVforQueue.notify_all(); for (auto& future : mAsyncSendFutures) { future.get(); } - auto const exception - = std::make_exception_ptr(std::runtime_error("CacheSender terminated before asynchronous send completed")); - for (auto& response : pendingAsyncResponses) + if (mResponseFuture.valid()) { - failResponse(response, exception); + mResponseFuture.get(); } } - void failResponse(Response& response, std::exception_ptr const& exception) noexcept + void removeResponse(std::map<RequestIdType, Response>::iterator it) { - try { - response.mPromise.set_exception(exception); + std::scoped_lock lkResp(mSenderMutex); + mReadyResponses.erase(it); } - catch (std::future_error const& err) + if (mReadyResponses.empty()) { - TLLM_LOG_ERROR("Failed to set CacheSender response exception: %s", err.what()); + std::unique_lock lkCond(mCondMutex); + mAnyReady = false; } - releasePinnedBlocks(response); } - void failPendingResponses(std::exception_ptr const& exception) noexcept + [[nodiscard]] RequestIdType getCurrentRequestId() const { - std::map<RequestIdType, Response> pendingResponses; - { - std::scoped_lock lock(mSenderMutex); - pendingResponses.swap(mReadyResponses); - mCurrentRequest = std::nullopt; - mCancelledRequests.clear(); - mRemainSendCount.clear(); - } - for (auto& entry : pendingResponses) - { - failResponse(entry.second, exception); - } + return mCurrentRequest.value(); + } + + [[nodiscard]] std::map<RequestIdType, Response>::iterator getCurrentResponse() + { + std::scoped_lock lk(mSenderMutex); + return mReadyResponses.find(getCurrentRequestId()); } public: void setRnnConfig(executor::kv_cache::CacheState::RnnModelConfig rnnModelConfig, - std::vector<SizeType32> rnnLayerNumPerPP, tensorrt_llm::DataType convStateDataType, - tensorrt_llm::DataType ssmStateDataType) + std::vector<SizeType32> rnnLayerNumPerPP, nvinfer1::DataType convStateDataType, + nvinfer1::DataType ssmStateDataType) { mCacheTransferLayer.setRnnConfig(rnnModelConfig, rnnLayerNumPerPP, convStateDataType, ssmStateDataType); mSelfState.setCacheState(mCacheTransferLayer.getCacheState()); @@ -1098,9 +751,9 @@ class CacheSender::Impl std::optional<RequestIdType> mCurrentRequest; std::set<LlmRequest::RequestIdType> mCancelledRequests; std::map<RequestIdType, Response> mReadyResponses; - std::mutex mSenderMutex; - std::atomic<bool> mTerminate{false}; - std::condition_variable mSenderCv; + std::mutex mSenderMutex, mCondMutex; + std::atomic<bool> mAnyReady{false}, mTerminate{false}; + std::condition_variable mSenderCv, mResponderCv; std::future<void> mResponseFuture; std::unordered_map<LlmRequest::RequestIdType, int> mRemainSendCount; AsyncSendResource mAsyncSendResource; @@ -1114,21 +767,16 @@ class CacheSender::Impl std::mutex mMtxForMap; runtime::BufferManager mBufferManager; std::ofstream mMeasuresFile; - std::mutex mInFlightCancelMutex; - std::unordered_map<LlmRequest::RequestIdType, std::shared_ptr<std::atomic<bool>>> mInFlightCancelFlags; - std::string mInstanceId; }; class CacheReceiver::Impl { public: - Impl(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer, - std::string instanceId = "") + Impl(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer) : mManager{manager} , mSelfState{cacheLayer.getCacheState(), executor::kv_cache::CommState{manager->getCommState()}} , mCacheTransferLayer{std::move(cacheLayer)} , mBufferManager{std::make_shared<runtime::CudaStream>()} - , mInstanceId{std::move(instanceId)} { TLLM_CHECK(mManager); TLLM_CHECK(mManager->getCommState().getSelfIdx() == selfIndex); @@ -1166,16 +814,9 @@ class CacheReceiver::Impl mRequestFutures.emplace_back(std::move(requestFuture)); } auto& asyncResource = mInstanceToAsyncResource.at(processInfo); - std::shared_ptr<std::atomic<bool>> cancelFlag; - if (common::getEnvDisaggEnableInflightCancel()) - { - cancelFlag = std::make_shared<std::atomic<bool>>(false); - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - mInFlightCancelFlags[llmRequest->mRequestId] = cancelFlag; - } { std::unique_lock<std::mutex> lck(asyncResource->mMtxForQueue); - asyncResource->mRequestsQueue.emplace_back(llmRequest, std::move(promise), cancelFlag); + asyncResource->mRequestsQueue.emplace_back(llmRequest, std::move(promise)); } asyncResource->mCVforQueue.notify_all(); return future; @@ -1188,34 +829,22 @@ class CacheReceiver::Impl void receiveSync(TransferSession& session) { - try + mCacheTransferLayer.unformat(session); + if (!common::getEnvKVCacheTimeOutputPath().empty()) { - mCacheTransferLayer.unformat(session); - if (!common::getEnvKVCacheTimeOutputPath().empty()) + std::unique_lock<std::mutex> lock(mMeasuresFileMutex); + if (!mMeasuresFile.is_open()) { - std::unique_lock<std::mutex> lock(mMeasuresFileMutex); - if (!mMeasuresFile.is_open()) - { - auto outputPath = getTransferOutputPath("recv", mInstanceId); - mMeasuresFile.open(outputPath); - TLLM_CHECK_WITH_INFO(mMeasuresFile.is_open(), "Failed to open transfer output file: %s", - outputPath.string().c_str()); - } - session.exportMeasure(mMeasuresFile, false); + auto outputPath = getTransferOutputPath("recv"); + mMeasuresFile.open(outputPath); + TLLM_CHECK_WITH_INFO( + mMeasuresFile.is_open(), "Failed to open transfer output file: %s", outputPath.string().c_str()); } - session.releaseReservedRecvBuffers(); - } - catch (...) - { - if (common::getEnvDisaggEnableInflightCancel()) - { - session.poisonReservedRecvBuffers(); - } - throw; + session.exportMeasure(mMeasuresFile, false); } } - TransferSession sendRequestInfo(LlmRequest const& llmRequest, std::atomic<bool> const* perRequestCancel = nullptr) + TransferSession sendRequestInfo(LlmRequest const& llmRequest) { uint64_t requestId = llmRequest.getContextPhaseParams().value().getReqId(); auto const& contextState = llmRequest.getDataTransceiverState(); @@ -1255,31 +884,14 @@ class CacheReceiver::Impl requestInfo = RequestInfo(requestId, mSelfState, indexFromEnd, lastBlockKey); } } - // The state's provenance marks llmRequest-agnostic transfers: only - // getSerializedDataTransceiverState sets it; context responses leave it unset. - requestInfo.setIsArbitraryTransfer(contextState.isArbitraryTransferState()); auto* agentConnectionManager = dynamic_cast<executor::kv_cache::AgentConnectionManager*>(mManager); - std::vector<BufferIndexHolder> recvHolders; std::vector<std::optional<size_t>> cacheBufferIds; if (agentConnectionManager) { - auto const* bufferCancel = common::getEnvDisaggEnableInflightCancel() ? perRequestCancel : nullptr; - auto const& managers = agentConnectionManager->getCacheTransBufferManagers(); - recvHolders.reserve(managers.size()); - cacheBufferIds.reserve(managers.size()); - for (auto& cacheTransBufferManager : managers) + for (auto& cacheTransBufferManager : agentConnectionManager->getCacheTransBufferManagers()) { - auto rawIdx = cacheTransBufferManager->assignBufferIndexForRecv(bufferCancel); - recvHolders.emplace_back(*cacheTransBufferManager, rawIdx, /*isRecv=*/true); - if (rawIdx.has_value()) - { - cacheBufferIds.push_back(static_cast<size_t>(rawIdx.value())); - } - else - { - cacheBufferIds.push_back(std::nullopt); - } + cacheBufferIds.push_back(cacheTransBufferManager->assignBufferIndexForRecv()); } TLLM_CHECK(!cacheBufferIds.empty()); } @@ -1308,102 +920,70 @@ class CacheReceiver::Impl allConnections.emplace_back(connection); } - if (common::getEnvDisaggEnableInflightCancel() && perRequestCancel != nullptr - && perRequestCancel->load(std::memory_order_relaxed)) + for (size_t ci = 0; ci < allCounterparts.size(); ci++) { - TLLM_THROW("KV cache receive request cancelled before publishing receive buffers"); - } + auto rank = allCounterparts[ci]; + auto const* connection = connections.at(rank); - try - { - for (size_t ci = 0; ci < allCounterparts.size(); ci++) - { - auto rank = allCounterparts[ci]; - auto const* connection = connections.at(rank); + bool isKvCounterpart + = std::find(kvCounterParts.begin(), kvCounterParts.end(), rank) != kvCounterParts.end(); + bool isRnnCounterpart + = hasRnn && std::find(rnnCounterParts.begin(), rnnCounterParts.end(), rank) != rnnCounterParts.end(); - bool isKvCounterpart - = std::find(kvCounterParts.begin(), kvCounterParts.end(), rank) != kvCounterParts.end(); - bool isRnnCounterpart = hasRnn - && std::find(rnnCounterParts.begin(), rnnCounterParts.end(), rank) != rnnCounterParts.end(); - - if (agentConnectionManager) + if (agentConnectionManager) + { + auto idsForRank = cacheBufferIds; + auto const& managers = agentConnectionManager->getCacheTransBufferManagers(); + for (size_t i = 0; i < idsForRank.size(); i++) { - auto idsForRank = cacheBufferIds; - auto const& managers = agentConnectionManager->getCacheTransBufferManagers(); - for (size_t i = 0; i < idsForRank.size(); i++) + auto kind = managers[i]->getBufferKind(); + bool include = (kind != BufferKind::kRNN) ? isKvCounterpart : isRnnCounterpart; + if (!include) { - auto kind = managers[i]->getBufferKind(); - bool include = (kind != BufferKind::kRNN) ? isKvCounterpart : isRnnCounterpart; - if (!include) - { - idsForRank[i] = std::nullopt; - } + idsForRank[i] = std::nullopt; } + } - int validConnectionIdx = 0; - if (isKvCounterpart) - { - auto kvCpIdx - = std::find(kvCounterParts.begin(), kvCounterParts.end(), rank) - kvCounterParts.begin(); - auto [pickUpIdx, localRankIdx] = mCacheTransferLayer.getKvFormatter()->pickRecvConnections( - allCounterparts.size(), mSelfState.getCacheState().value(), - mSelfState.getCommState().value().getSelfIdx(), destCacheState, allCounterparts); - validConnectionIdx - = std::find(localRankIdx.begin(), localRankIdx.end(), kvCpIdx) - localRankIdx.begin(); - } - else if (isRnnCounterpart) - { - auto rnnTargetInfo = executor::kv_cache::targetIRanksForRnn(destCacheState, - mCacheTransferLayer.getCacheState(), mSelfState.getCommState().value().getSelfIdx()); - auto rnnCpIdx - = std::find(rnnCounterParts.begin(), rnnCounterParts.end(), rank) - rnnCounterParts.begin(); - auto [pickUpIdx, localRankIdx] - = cache_formatter_utils::pickRecvConnections(rnnCounterParts.size(), - mCacheTransferLayer.getCacheState(), mSelfState.getCommState().value().getSelfIdx(), - destCacheState, rnnCounterParts, rnnTargetInfo); - validConnectionIdx - = std::find(localRankIdx.begin(), localRankIdx.end(), rnnCpIdx) - localRankIdx.begin(); - } - - auto* agentConnection = dynamic_cast<executor::kv_cache::AgentConnection const*>(connection); - TLLM_CHECK(agentConnection != nullptr); - - const_cast<executor::kv_cache::AgentConnection*>(agentConnection) - ->sendRequestAndBufferInfo(requestInfo, idsForRank, validConnectionIdx, perRequestCancel); + int validConnectionIdx = 0; + if (isKvCounterpart) + { + auto kvCpIdx + = std::find(kvCounterParts.begin(), kvCounterParts.end(), rank) - kvCounterParts.begin(); + auto [pickUpIdx, localRankIdx] = mCacheTransferLayer.getKvFormatter()->pickRecvConnections( + allCounterparts.size(), mSelfState.getCacheState().value(), + mSelfState.getCommState().value().getSelfIdx(), destCacheState, allCounterparts); + validConnectionIdx + = std::find(localRankIdx.begin(), localRankIdx.end(), kvCpIdx) - localRankIdx.begin(); } - else + else if (isRnnCounterpart) { - sendRequestInfo(connection, requestInfo); + auto rnnTargetInfo = executor::kv_cache::targetIRanksForRnn(destCacheState, + mCacheTransferLayer.getCacheState(), mSelfState.getCommState().value().getSelfIdx()); + auto rnnCpIdx + = std::find(rnnCounterParts.begin(), rnnCounterParts.end(), rank) - rnnCounterParts.begin(); + auto [pickUpIdx, localRankIdx] = cache_formatter_utils::pickRecvConnections(rnnCounterParts.size(), + mCacheTransferLayer.getCacheState(), mSelfState.getCommState().value().getSelfIdx(), + destCacheState, rnnCounterParts, rnnTargetInfo); + validConnectionIdx + = std::find(localRankIdx.begin(), localRankIdx.end(), rnnCpIdx) - localRankIdx.begin(); } - } - auto const& resource = getReceiveCacheResource(llmRequest); - TransferSession session = perRequestCancel != nullptr - ? TransferSession(std::move(allConnections), - DataContext{tagFromRequestId(requestId), *perRequestCancel}, std::move(allCounterparts), mSelfState, - contextState, resource->mBufferManager, requestInfo.getIndexFromEnd(), - requestInfo.getLastBlockKey(), &llmRequest, !common::getEnvKVCacheTimeOutputPath().empty()) - : TransferSession(std::move(allConnections), DataContext{tagFromRequestId(requestId), mTerminate}, - std::move(allCounterparts), mSelfState, contextState, resource->mBufferManager, - requestInfo.getIndexFromEnd(), requestInfo.getLastBlockKey(), &llmRequest, - !common::getEnvKVCacheTimeOutputPath().empty()); - if (!recvHolders.empty()) - { - session.setReservedRecvBuffers(std::move(recvHolders)); + auto* agentConnection = dynamic_cast<executor::kv_cache::AgentConnection const*>(connection); + TLLM_CHECK(agentConnection != nullptr); + + const_cast<executor::kv_cache::AgentConnection*>(agentConnection) + ->sendRequestAndBufferInfo(requestInfo, idsForRank, validConnectionIdx); } - return session; - } - catch (...) - { - if (common::getEnvDisaggEnableInflightCancel()) + else { - for (auto& holder : recvHolders) - { - holder.poison(); - } + sendRequestInfo(connection, requestInfo); } - throw; } + auto const& resource = getReceiveCacheResource(llmRequest); + return TransferSession(std::move(allConnections), DataContext{tagFromRequestId(requestId), mTerminate}, + std::move(allCounterparts), mSelfState, contextState, resource->mBufferManager, + requestInfo.getIndexFromEnd(), requestInfo.getLastBlockKey(), &llmRequest, + !common::getEnvKVCacheTimeOutputPath().empty()); } std::unique_ptr<ReceiveCacheResource> const& getReceiveCacheResource(LlmRequest const& llmRequest) @@ -1442,27 +1022,11 @@ class CacheReceiver::Impl std::string processInfo = kDefaultProcessInfo; if (common::getEnvRequestKVCacheConcurrent()) { - auto const& commState = llmRequest.getDataTransceiverState().getCommState(); - if (!commState.has_value()) - { - TLLM_LOG_WARNING("Cannot cancel request %zu: the request has no data-transceiver communication state", - llmRequest.mRequestId); - return false; - } - processInfo = commState->toString(); - } - - auto const resourceIt = mInstanceToAsyncResource.find(processInfo); - if (resourceIt == mInstanceToAsyncResource.end()) - { - TLLM_LOG_WARNING("Cannot cancel request %zu: receive worker %s is not registered", llmRequest.mRequestId, - processInfo.c_str()); - return false; + processInfo = llmRequest.getDataTransceiverState().getCommState()->toString(); } bool isCancelled = false; - auto& asyncResource = resourceIt->second; - std::optional<LlmRequest::RequestIdType> queuedCancelledReqId; + auto& asyncResource = mInstanceToAsyncResource.at(processInfo); { std::unique_lock<std::mutex> lck(asyncResource->mMtxForQueue); auto it = std::find_if(asyncResource->mRequestsQueue.begin(), asyncResource->mRequestsQueue.end(), @@ -1489,116 +1053,45 @@ class CacheReceiver::Impl } asyncResource->mRequestsQueue.erase(it); isCancelled = true; - queuedCancelledReqId = llmRequest.mRequestId; } - } - if (common::getEnvDisaggEnableInflightCancel() && queuedCancelledReqId.has_value()) - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - mInFlightCancelFlags.erase(*queuedCancelledReqId); - } - if (!isCancelled && common::getEnvDisaggEnableInflightCancel()) - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - auto flagIt = mInFlightCancelFlags.find(llmRequest.mRequestId); - if (flagIt != mInFlightCancelFlags.end()) + else { - flagIt->second->store(true, std::memory_order_relaxed); - isCancelled = true; + TLLM_LOG_WARNING("Cannot cancel request %zu", llmRequest.mRequestId); } } - if (!isCancelled) - { - TLLM_LOG_WARNING("Cannot cancel request %zu", llmRequest.mRequestId); - } return isCancelled; } - enum class ReadySignalResult - { - kReady, - kNotReady, - kMixed, - kCancelled, - }; - - ReadySignalResult receiveReadySignalDetailed(TransferSession& session, std::atomic<bool> const& perRequestCancel) + bool receiveReadySignal(TransferSession& session) { + bool isReadyFinal = true; bool isReady = false; - bool anyReady = false; - bool anyNotReady = false; auto const& connections = session.getConnections(); + for (size_t i = 0; i < connections.size(); i++) { - if (perRequestCancel.load(std::memory_order_relaxed)) - { - return ReadySignalResult::kCancelled; - } auto* agentConnectionManager = dynamic_cast<executor::kv_cache::AgentConnectionManager*>(mManager); if (agentConnectionManager) { auto* agentConnection = dynamic_cast<executor::kv_cache::AgentConnection const*>(connections.at(i)); TLLM_CHECK(agentConnection); - auto ready = agentConnection->recvReadySignalWithStatus( - executor::kv_cache::DataContext{session.getDataContext().getTag(), perRequestCancel}); - if (!ready.has_value()) - { - return ReadySignalResult::kCancelled; - } - isReady = ready.value(); + isReady = agentConnection->recvReadySignal( + executor::kv_cache::DataContext{TransceiverTag::kREADY_SIGNAL_TAG, mTerminate}); } else { connections.at(i)->recv( executor::kv_cache::DataContext{TransceiverTag::kREADY_SIGNAL_TAG}, &isReady, sizeof(isReady)); - if (perRequestCancel.load(std::memory_order_relaxed)) - { - return ReadySignalResult::kCancelled; - } } - anyReady |= isReady; - anyNotReady |= !isReady; + isReadyFinal &= isReady; } - if (anyReady && anyNotReady) - { - return ReadySignalResult::kMixed; - } - return anyReady ? ReadySignalResult::kReady : ReadySignalResult::kNotReady; - } - - bool receiveReadySignal(TransferSession& session) - { - auto const result = receiveReadySignalDetailed(session, mTerminate); - if (result == ReadySignalResult::kNotReady) - { - session.releaseReservedRecvBuffers(); - } - else if (result == ReadySignalResult::kMixed) - { - if (common::getEnvDisaggEnableInflightCancel()) - { - session.poisonReservedRecvBuffers(); - } - else - { - session.releaseReservedRecvBuffers(); - } - } - return result == ReadySignalResult::kReady; + return isReadyFinal; } ~Impl() { mTerminate.store(true); - if (common::getEnvDisaggEnableInflightCancel()) - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - for (auto& [id, flag] : mInFlightCancelFlags) - { - flag->store(true, std::memory_order_relaxed); - } - } for (auto&& [processInfo, asyncResource] : mInstanceToAsyncResource) { asyncResource->mTerminate = true; @@ -1611,105 +1104,29 @@ class CacheReceiver::Impl } private: - void requestSync(LlmRequest& llmRequest, std::atomic<bool> const& perRequestCancel) + void requestSync(LlmRequest& llmRequest) { - auto const requestId = llmRequest.mRequestId; - auto const contextRequestId = llmRequest.getContextPhaseParams().value().getReqId(); - char const* phase = "request-info"; - TLLM_LOG_DEBUG("KV cache receive request %zu, context request %zu started.", requestId, contextRequestId); - if (llmRequest.getKvCacheTransferStart() == LlmRequest::TimePoint{}) - { - llmRequest.setKvCacheTransferStart(LlmRequest::getSteadyClockNow()); - } - - std::optional<TransferSession> session; - try - { - if (perRequestCancel.load(std::memory_order_relaxed) || mTerminate.load(std::memory_order_relaxed)) - { - TLLM_THROW("KV cache receive request %zu cancelled before request-info", requestId); - } - TLLM_CUDA_CHECK(cudaSetDevice(mDeviceId)); - TLLM_LOG_DEBUG("KV cache receive request %zu, context request %zu phase=%s begin.", requestId, - contextRequestId, phase); - auto const* cancelFlag = common::getEnvDisaggEnableInflightCancel() ? &perRequestCancel : nullptr; - session.emplace(sendRequestInfo(llmRequest, cancelFlag)); - session->setTime(TransferSession::kTimeRequestInfo); - TLLM_LOG_DEBUG( - "KV cache receive request %zu, context request %zu phase=%s end.", requestId, contextRequestId, phase); - - phase = "ready-signal"; - TLLM_LOG_DEBUG("KV cache receive request %zu, context request %zu phase=%s begin.", requestId, - contextRequestId, phase); - auto readyResult = receiveReadySignalDetailed(*session, perRequestCancel); - TLLM_LOG_DEBUG("KV cache receive request %zu, context request %zu phase=%s end: result=%d.", requestId, - contextRequestId, phase, static_cast<int>(readyResult)); - if (readyResult == ReadySignalResult::kCancelled) - { - if (common::getEnvDisaggEnableInflightCancel()) - { - session->poisonReservedRecvBuffers(); - } - TLLM_THROW("KV cache receive request %zu cancelled while waiting for the ready signal", requestId); - } - if (readyResult == ReadySignalResult::kNotReady) - { - session->releaseReservedRecvBuffers(); - TLLM_THROW("KV cache receive request %zu was rejected by the context peer", requestId); - } - if (readyResult == ReadySignalResult::kMixed) - { - if (common::getEnvDisaggEnableInflightCancel()) - { - session->poisonReservedRecvBuffers(); - } - else - { - session->releaseReservedRecvBuffers(); - } - TLLM_THROW("KV cache receive request %zu received inconsistent ready signals from its context peers", - requestId); - } - - phase = "transfer-completion-notification"; - TLLM_LOG_DEBUG("KV cache receive request %zu, context request %zu phase=%s begin.", requestId, - contextRequestId, phase); - receiveSync(*session); - TLLM_LOG_DEBUG( - "KV cache receive request %zu, context request %zu phase=%s end.", requestId, contextRequestId, phase); - llmRequest.setKvCacheTransferEnd(LlmRequest::getSteadyClockNow()); - } - catch (std::exception const& err) - { - if (common::getEnvDisaggEnableInflightCancel() && session.has_value()) - { - session->poisonReservedRecvBuffers(); - } - llmRequest.setKvCacheTransferEnd(LlmRequest::getSteadyClockNow()); - TLLM_LOG_ERROR("KV cache receive request %zu, context request %zu failed in phase=%s: %s", requestId, - contextRequestId, phase, err.what()); - throw; - } - catch (...) + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), + "Start calling requestSync for request ID: %zu, context request ID: %zu.", llmRequest.mRequestId, + llmRequest.getContextPhaseParams().value().getReqId()); + llmRequest.setKvCacheTransferStart(LlmRequest::getSteadyClockNow()); + TLLM_CUDA_CHECK(cudaSetDevice(mDeviceId)); + auto session = sendRequestInfo(llmRequest); + session.setTime(TransferSession::kTimeRequestInfo); + bool isReady = receiveReadySignal(session); + if (!isReady) { - if (common::getEnvDisaggEnableInflightCancel() && session.has_value()) - { - session->poisonReservedRecvBuffers(); - } + // Reuse the error state for the cancelled request. + llmRequest.setState(LlmRequestState::kDISAGG_TRANS_ERROR); llmRequest.setKvCacheTransferEnd(LlmRequest::getSteadyClockNow()); - TLLM_LOG_ERROR( - "KV cache receive request %zu, context request %zu failed in phase=%s with an unknown " - "exception", - requestId, contextRequestId, phase); - throw; + return; } + receiveSync(session); + llmRequest.setKvCacheTransferEnd(LlmRequest::getSteadyClockNow()); - TLLM_LOG_DEBUG("KV cache receive request %zu, context request %zu completed.", requestId, contextRequestId); - } - - void requestSync(LlmRequest& llmRequest) - { - requestSync(llmRequest, mTerminate); + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), + "End calling requestSync for request ID: %zu, context request ID: %zu.", llmRequest.mRequestId, + llmRequest.getContextPhaseParams().value().getReqId()); } struct RequestAndPromise @@ -1718,20 +1135,16 @@ class CacheReceiver::Impl // protects worker-side dereferences and the promise itself from premature destruction. std::shared_ptr<LlmRequest> mRequest; std::unique_ptr<std::promise<void>> mPromise; - std::shared_ptr<std::atomic<bool>> mCancelFlag; RequestAndPromise() : mRequest(nullptr) , mPromise(nullptr) - , mCancelFlag(nullptr) { } - RequestAndPromise(std::shared_ptr<LlmRequest> request, std::unique_ptr<std::promise<void>>&& promise, - std::shared_ptr<std::atomic<bool>> cancelFlag) + RequestAndPromise(std::shared_ptr<LlmRequest> request, std::unique_ptr<std::promise<void>>&& promise) : mRequest(std::move(request)) , mPromise(std::move(promise)) - , mCancelFlag(std::move(cancelFlag)) { } @@ -1740,7 +1153,6 @@ class CacheReceiver::Impl RequestAndPromise(RequestAndPromise&& other) noexcept : mRequest(std::move(other.mRequest)) , mPromise(std::move(other.mPromise)) - , mCancelFlag(std::move(other.mCancelFlag)) { } @@ -1756,7 +1168,6 @@ class CacheReceiver::Impl mRequest = std::move(other.mRequest); mPromise = std::move(other.mPromise); - mCancelFlag = std::move(other.mCancelFlag); } return *this; } @@ -1800,9 +1211,7 @@ class CacheReceiver::Impl try { TLLM_CHECK_WITH_INFO(requestAndPromise.mRequest != nullptr, "requestAndPromise.mRequest is null"); - auto const& cancelFlag - = requestAndPromise.mCancelFlag != nullptr ? *requestAndPromise.mCancelFlag : mTerminate; - requestSync(*requestAndPromise.mRequest, cancelFlag); + requestSync(*requestAndPromise.mRequest); requestAndPromise.mPromise->set_value(); } catch (tensorrt_llm::common::RequestSpecificException const& err) @@ -1821,27 +1230,14 @@ class CacheReceiver::Impl requestAndPromise.mRequest->getContextPhaseParams().value().getReqId(), err.what()); requestAndPromise.mPromise->set_exception(std::current_exception()); } - catch (...) - { - TLLM_LOG_ERROR("Unknown exception in CacheReceiver request() loop"); - if (requestAndPromise.mPromise) - { - requestAndPromise.mPromise->set_exception(std::current_exception()); - } - } - if (common::getEnvDisaggEnableInflightCancel() && requestAndPromise.mRequest != nullptr) - { - std::lock_guard<std::mutex> lg(mInFlightCancelMutex); - mInFlightCancelFlags.erase(requestAndPromise.mRequest->mRequestId); - } } } } public: void setRnnConfig(executor::kv_cache::CacheState::RnnModelConfig rnnModelConfig, - std::vector<SizeType32> rnnLayerNumPerPP, tensorrt_llm::DataType convStateDataType, - tensorrt_llm::DataType ssmStateDataType) + std::vector<SizeType32> rnnLayerNumPerPP, nvinfer1::DataType convStateDataType, + nvinfer1::DataType ssmStateDataType) { mCacheTransferLayer.setRnnConfig(rnnModelConfig, rnnLayerNumPerPP, convStateDataType, ssmStateDataType); mSelfState.setCacheState(mCacheTransferLayer.getCacheState()); @@ -1861,9 +1257,6 @@ class CacheReceiver::Impl std::ofstream mMeasuresFile; std::mutex mMeasuresFileMutex; std::atomic<bool> mTerminate{false}; - std::mutex mInFlightCancelMutex; - std::unordered_map<LlmRequest::RequestIdType, std::shared_ptr<std::atomic<bool>>> mInFlightCancelFlags; - std::string mInstanceId; }; void CacheSender::ImplDeleter::operator()(Impl* ptr) @@ -1876,10 +1269,9 @@ void CacheReceiver::ImplDeleter::operator()(Impl* ptr) delete ptr; } -CacheSender::CacheSender(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, - CacheTransferLayer cacheLayer, std::string instanceId) - : mImpl{ - std::unique_ptr<Impl, ImplDeleter>(new Impl(manager, selfIndex, std::move(cacheLayer), std::move(instanceId)))} +CacheSender::CacheSender( + executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer) + : mImpl{std::unique_ptr<Impl, ImplDeleter>(new Impl(manager, selfIndex, std::move(cacheLayer)))} { } @@ -1923,16 +1315,14 @@ void CacheSender::sendReadySignal(LlmRequest::RequestIdType requestId, bool isRe } void CacheSender::setRnnConfig(executor::kv_cache::CacheState::RnnModelConfig rnnModelConfig, - std::vector<SizeType32> rnnLayerNumPerPP, tensorrt_llm::DataType convStateDataType, - tensorrt_llm::DataType ssmStateDataType) + std::vector<SizeType32> rnnLayerNumPerPP, nvinfer1::DataType convStateDataType, nvinfer1::DataType ssmStateDataType) { mImpl->setRnnConfig(std::move(rnnModelConfig), std::move(rnnLayerNumPerPP), convStateDataType, ssmStateDataType); } -CacheReceiver::CacheReceiver(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, - CacheTransferLayer cacheLayer, std::string instanceId) - : mImpl{ - std::unique_ptr<Impl, ImplDeleter>(new Impl(manager, selfIndex, std::move(cacheLayer), std::move(instanceId)))} +CacheReceiver::CacheReceiver( + executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer) + : mImpl{std::unique_ptr<Impl, ImplDeleter>(new Impl(manager, selfIndex, std::move(cacheLayer)))} { } @@ -1964,8 +1354,7 @@ bool CacheReceiver::receiveReadySignal(TransferSession& session) } void CacheReceiver::setRnnConfig(executor::kv_cache::CacheState::RnnModelConfig rnnModelConfig, - std::vector<SizeType32> rnnLayerNumPerPP, tensorrt_llm::DataType convStateDataType, - tensorrt_llm::DataType ssmStateDataType) + std::vector<SizeType32> rnnLayerNumPerPP, nvinfer1::DataType convStateDataType, nvinfer1::DataType ssmStateDataType) { mImpl->setRnnConfig(std::move(rnnModelConfig), std::move(rnnLayerNumPerPP), convStateDataType, ssmStateDataType); } diff --git a/cpp/tensorrt_llm/batch_manager/dataTransceiver.h b/cpp/tensorrt_llm/batch_manager/dataTransceiver.h index 778e5e80c7f4..3362574da902 100644 --- a/cpp/tensorrt_llm/batch_manager/dataTransceiver.h +++ b/cpp/tensorrt_llm/batch_manager/dataTransceiver.h @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -19,18 +19,15 @@ #include <fstream> #include <future> #include <map> -#include <optional> #include <string> #include <vector> -#include "tensorrt_llm/batch_manager/baseTransBuffer.h" #include "tensorrt_llm/batch_manager/cacheTransceiver.h" #include "tensorrt_llm/batch_manager/cacheTransferLayer.h" #include "tensorrt_llm/batch_manager/llmRequest.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/cacheCommunicator.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/executor/serializeUtils.h" @@ -122,7 +119,7 @@ class TransferSession void recv(size_t idx, void* data, size_t size); - [[nodiscard]] std::optional<LlmRequest const*> getLlmRequest() const; + [[nodiscard]] LlmRequest const& getLlmRequest() const; // in CacheSender, the LlmRequest is not available until the sendSync is called void setLlmRequest(LlmRequest const& llmRequest); @@ -131,20 +128,6 @@ class TransferSession void appendMeasure(LlmRequest::TimePoint start, LlmRequest::TimePoint end, size_t size); - /// @brief Transfer ownership of pre-assigned receive buffers to this session. - void setReservedRecvBuffers(std::vector<BufferIndexHolder> holders); - - [[nodiscard]] bool hasReservedRecvBuffer(BaseTransBufferManager const& manager) const noexcept; - - /// @brief Release one formatter's pre-assigned buffer after its receive and postprocessing complete. - bool releaseReservedRecvBuffer(BaseTransBufferManager const& manager) noexcept; - - /// @brief Release all pre-assigned receive buffers after the full receive pipeline completes. - void releaseReservedRecvBuffers() noexcept; - - /// @brief Fail closed when receive-buffer quiescence cannot be established. - void poisonReservedRecvBuffers() noexcept; - // TODO: 1. use global id instead of context request id; 2. export to llm metrics instead of file void exportMeasure(std::ofstream& outFile, bool isContext) const; @@ -177,7 +160,6 @@ class TransferSession runtime::BufferManager const* mBufferManager; LlmRequest const* mRequest; std::unique_ptr<KVCacheTimes> mTimes; - std::vector<BufferIndexHolder> mReservedRecvBuffers; int32_t mIndexFromEnd{0}; BlockKey mLastBlockKey{}; }; @@ -235,17 +217,6 @@ class RequestInfo return mLastBlockKey; } - /// @brief Arbitrary (llmRequest-agnostic) transfer served from the sender's reuse tree. - [[nodiscard]] bool isArbitraryTransfer() const noexcept - { - return mIsArbitraryTransfer; - } - - void setIsArbitraryTransfer(bool isArbitraryTransfer) noexcept - { - mIsArbitraryTransfer = isArbitraryTransfer; - } - /// @brief Serialization. /// @param requestInfo Request information to be serialized. /// @param os The output stream to which the serialization result points. @@ -269,9 +240,6 @@ class RequestInfo // Last block key, used to derive other block keys on receiver BlockKey mLastBlockKey{}; - // True for arbitrary (llmRequest-agnostic) transfers served from the sender's reuse tree. - bool mIsArbitraryTransfer{false}; - // The state of the data transceiver. executor::DataTransceiverState mTransState; }; @@ -283,8 +251,7 @@ class CacheSender /// @param manager The connection manager. /// @param selfIndex The sequential index of the current executor process. /// @param cacheLayer The cache layer bundling all cache states and formatters. - CacheSender(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer, - std::string instanceId = ""); + CacheSender(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer); CacheSender() = default; @@ -323,8 +290,8 @@ class CacheSender /// @brief Update the RNN config on the internal CacheState copies. /// Used by CppMambaHybridCacheManager path where RNN config is set after construction. void setRnnConfig(executor::kv_cache::CacheState::RnnModelConfig rnnModelConfig, - std::vector<SizeType32> rnnLayerNumPerPP, tensorrt_llm::DataType convStateDataType, - tensorrt_llm::DataType ssmStateDataType); + std::vector<SizeType32> rnnLayerNumPerPP, nvinfer1::DataType convStateDataType, + nvinfer1::DataType ssmStateDataType); /// @brief Destructor. virtual ~CacheSender(); @@ -347,8 +314,7 @@ class CacheReceiver /// @param manager The connection manager. /// @param selfIndex The sequential index of the current executor process. /// @param cacheLayer The cache layer bundling all cache states and formatters. - CacheReceiver(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer, - std::string instanceId = ""); + CacheReceiver(executor::kv_cache::ConnectionManager* manager, SizeType32 selfIndex, CacheTransferLayer cacheLayer); CacheReceiver() = default; @@ -375,8 +341,8 @@ class CacheReceiver /// @brief Update the RNN config on the internal CacheState copies. /// Used by CppMambaHybridCacheManager path where RNN config is set after construction. void setRnnConfig(executor::kv_cache::CacheState::RnnModelConfig rnnModelConfig, - std::vector<SizeType32> rnnLayerNumPerPP, tensorrt_llm::DataType convStateDataType, - tensorrt_llm::DataType ssmStateDataType); + std::vector<SizeType32> rnnLayerNumPerPP, nvinfer1::DataType convStateDataType, + nvinfer1::DataType ssmStateDataType); /// @brief Destructor. virtual ~CacheReceiver(); diff --git a/cpp/tensorrt_llm/batch_manager/decoderBuffers.cpp b/cpp/tensorrt_llm/batch_manager/decoderBuffers.cpp index fecc0851d361..fd67bb55e89d 100644 --- a/cpp/tensorrt_llm/batch_manager/decoderBuffers.cpp +++ b/cpp/tensorrt_llm/batch_manager/decoderBuffers.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/batch_manager/decoderBuffers.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/decoderState.h" @@ -71,21 +70,21 @@ DecoderOutputBuffers::DecoderOutputBuffers(SizeType32 maxNumSequences, SizeType3 auto constexpr TRTTokenIdType = runtime::TRTDataType<runtime::TokenIdType>::value; sequenceLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxNumSequences, maxBeamWidth}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxNumSequences, maxBeamWidth}), nvinfer1::DataType::kINT32); - finishedSumHost = BufferManager::pinned(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + finishedSumHost = BufferManager::pinned(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); newOutputTokensHost = BufferManager::pinned(ITensor::makeShape({maxTokensPerStep, maxNumSequences, maxBeamWidth}), TRTTokenIdType); cumLogProbsHost - = BufferManager::pinned(ITensor::makeShape({maxNumSequences, maxBeamWidth}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxNumSequences, maxBeamWidth}), nvinfer1::DataType::kFLOAT); logProbsHost = BufferManager::pinned( - ITensor::makeShape({maxNumSequences, maxBeamWidth, maxSeqLen}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({maxNumSequences, maxBeamWidth, maxSeqLen}), nvinfer1::DataType::kFLOAT); finishReasonsHost - = BufferManager::pinned(ITensor::makeShape({maxNumSequences, maxBeamWidth}), tensorrt_llm::DataType::kUINT8); + = BufferManager::pinned(ITensor::makeShape({maxNumSequences, maxBeamWidth}), nvinfer1::DataType::kUINT8); } void DecoderOutputBuffers::enableLookaheadDecoding(SizeType32 maxNumSequences, SizeType32 maxTokensPerStep) @@ -116,9 +115,9 @@ void DecoderOutputBuffers::setupSpeculativeDecoding( if (speculativeDecodingMode.variableDraftLength()) { nextDraftTokensLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); prevDraftTokensLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); } } } @@ -308,18 +307,17 @@ DecoderSlotAsyncSend::~DecoderSlotAsyncSend() SlotDecoderBuffers::SlotDecoderBuffers(SizeType32 maxBeamWidth, SizeType32 maxSeqLen, BufferManager const& manager) { - outputIds = manager.gpu(ITensor::makeShape({maxBeamWidth, maxSeqLen}), tensorrt_llm::DataType::kINT32); - outputIdsHost - = BufferManager::pinned(ITensor::makeShape({maxBeamWidth, maxSeqLen}), tensorrt_llm::DataType::kINT32); + outputIds = manager.gpu(ITensor::makeShape({maxBeamWidth, maxSeqLen}), nvinfer1::DataType::kINT32); + outputIdsHost = BufferManager::pinned(ITensor::makeShape({maxBeamWidth, maxSeqLen}), nvinfer1::DataType::kINT32); - sequenceLengths = manager.gpu(ITensor::makeShape({maxBeamWidth}), tensorrt_llm::DataType::kINT32); - sequenceLengthsHost = BufferManager::pinned(ITensor::makeShape({maxBeamWidth}), tensorrt_llm::DataType::kINT32); + sequenceLengths = manager.gpu(ITensor::makeShape({maxBeamWidth}), nvinfer1::DataType::kINT32); + sequenceLengthsHost = BufferManager::pinned(ITensor::makeShape({maxBeamWidth}), nvinfer1::DataType::kINT32); - cumLogProbs = manager.gpu(ITensor::makeShape({maxBeamWidth}), tensorrt_llm::DataType::kFLOAT); - cumLogProbsHost = BufferManager::pinned(ITensor::makeShape({maxBeamWidth}), tensorrt_llm::DataType::kFLOAT); + cumLogProbs = manager.gpu(ITensor::makeShape({maxBeamWidth}), nvinfer1::DataType::kFLOAT); + cumLogProbsHost = BufferManager::pinned(ITensor::makeShape({maxBeamWidth}), nvinfer1::DataType::kFLOAT); - logProbs = manager.gpu(ITensor::makeShape({maxBeamWidth, maxSeqLen}), tensorrt_llm::DataType::kFLOAT); - logProbsHost = BufferManager::pinned(ITensor::makeShape({maxBeamWidth, maxSeqLen}), tensorrt_llm::DataType::kFLOAT); + logProbs = manager.gpu(ITensor::makeShape({maxBeamWidth, maxSeqLen}), nvinfer1::DataType::kFLOAT); + logProbsHost = BufferManager::pinned(ITensor::makeShape({maxBeamWidth, maxSeqLen}), nvinfer1::DataType::kFLOAT); } } // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/encoderBuffers.cpp b/cpp/tensorrt_llm/batch_manager/encoderBuffers.cpp new file mode 100644 index 000000000000..56fd393c68d7 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/encoderBuffers.cpp @@ -0,0 +1,560 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "encoderBuffers.h" + +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" + +#include <valarray> + +using namespace tensorrt_llm::runtime; + +namespace tensorrt_llm::batch_manager +{ + +EncoderBuffers::EncoderBuffers( + SizeType32 maxBatchSize, ModelConfig const& modelConfig, WorldConfig const& worldConfig, TllmRuntime const& runtime) +{ + // init empty buffers on cpu/gpu/pinned + init(maxBatchSize, modelConfig, worldConfig, runtime); + + // pre-allocate based on max buffer sizes + // Note: pre-allocation can be done directly instead of empty-->reshape, but it is ok extract the common reshape() + // utility because the buffer shapes can be dynamically set during runtime as well + initBufferSizes(maxBatchSize, modelConfig, worldConfig, runtime); +} + +void EncoderBuffers::init( + SizeType32 maxBatchSize, ModelConfig const& modelConfig, WorldConfig const& worldConfig, TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const& manager = runtime.getBufferManager(); + + auto hiddenStatesType = modelConfig.getDataType(); + + inputFeatures = manager.emptyTensor(MemoryType::kGPU, hiddenStatesType); + inputIds = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + + // in PP, only rank 0 needs the following input fields + if (modelConfig.usePositionEmbedding() && worldConfig.isFirstPipelineParallelRank()) + { + positionIds = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + positionIdsReserved.resize(maxBatchSize * modelConfig.getMaxInputLen()); + std::iota(positionIdsReserved.begin(), positionIdsReserved.end(), 0); + } + if (modelConfig.useTokenTypeEmbedding() && worldConfig.isFirstPipelineParallelRank()) + { + tokenTypeIds = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + tokenTypeIdsReserved.resize(maxBatchSize * modelConfig.getMaxInputLen()); + std::fill(tokenTypeIdsReserved.begin(), tokenTypeIdsReserved.end(), 0); + } + + inputLengths = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + maxInputLength = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + + if (worldConfig.isPipelineParallel()) + { + hiddenStates = manager.emptyTensor(MemoryType::kGPU, hiddenStatesType); + } + if (worldConfig.isLastPipelineParallelRank()) + { + encoderOutput = manager.emptyTensor(MemoryType::kGPU, hiddenStatesType); + } + + if (modelConfig.useLanguageAdapter()) + { + languageAdapterRoutings = manager.emptyTensor(MemoryType::kGPU, TRTDataType<SizeType32>::value); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::initBufferSizes( + SizeType32 maxBatchSize, ModelConfig const& modelConfig, WorldConfig const& worldConfig, TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // get buffer shape based on max values + numRequests = maxBatchSize; + encoderInputLen = maxBatchSize * modelConfig.getMaxInputLen(); + encoderOutputLen = maxBatchSize * modelConfig.getMaxInputLen(); // assume output length <= input length + maxInputLengthInBatch = modelConfig.getMaxInputLen(); + + // update buffer shapes + reshape(runtime, modelConfig, worldConfig); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::updateBufferSizes(RequestVector const& requests, ModelConfig const& modelConfig, + WorldConfig const& worldConfig, TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + numRequests = requests.size(); + encoderInputLen = 0; + encoderOutputLen = 0; + maxInputLengthInBatch = 0; + + // get buffer shape based on actual batched requests + for (auto const& req : requests) + { + encoderInputLen += req->getEncoderInputLen(); + encoderOutputLen += req->getEncoderOutputLen(); + maxInputLengthInBatch + = std::max(maxInputLengthInBatch, req->getEncoderInputLen()); // Decoder input is encoder output + } + + // update buffer shapes + reshape(runtime, modelConfig, worldConfig); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::reshape(TllmRuntime const& runtime, ModelConfig const& modelConfig, WorldConfig const& worldConfig) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + if (modelConfig.isMultiModal()) + { + return; // multimodal models do not need to set position id, etc. or any output tensors + } + + inputIds->reshape(ITensor::makeShape({encoderInputLen})); + if (positionIds) + { + if (modelConfig.isWhisper()) + { + positionIds->reshape(ITensor::makeShape({encoderOutputLen})); + } + else + { + positionIds->reshape(ITensor::makeShape({encoderInputLen})); + } + } + if (tokenTypeIds) + { + tokenTypeIds->reshape(ITensor::makeShape({encoderInputLen})); + } + + inputLengths->reshape(ITensor::makeShape({numRequests})); + maxInputLength->reshape(ITensor::makeShape({maxInputLengthInBatch})); + + if (worldConfig.isPipelineParallel()) + { + hiddenStates->reshape( + ITensor::makeShape({encoderOutputLen, modelConfig.getHiddenSize() * worldConfig.getTensorParallelism()})); + } + if (worldConfig.isLastPipelineParallelRank()) + { + encoderOutput->reshape( + ITensor::makeShape({encoderOutputLen, modelConfig.getHiddenSize() * worldConfig.getTensorParallelism()})); + } + if (modelConfig.useLanguageAdapter()) + { + languageAdapterRoutings->reshape(ITensor::makeShape({encoderInputLen, 1})); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::setFromInputs(RequestVector const& requests, ModelConfig const& modelConfig, + WorldConfig const& worldConfig, TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(encoderBuffersSetFromInputs); + + if (!worldConfig.isFirstPipelineParallelRank()) + { + return; + } + + auto const& manager = runtime.getBufferManager(); + + std::vector<TokenIdType> inputIdsAll; + std::vector<SizeType32> positionIdsAll; + std::vector<SizeType32> tokenTypeIdsAll; + std::vector<SizeType32> inputLengthsAll; + std::vector<SizeType32> languageAdapterRoutingAll; + // use shape to indicates max input length, content is not important + // TODO: change to a scalar value for this from engine side + std::vector<SizeType32> maxInputLengthAll(maxInputLengthInBatch, 0); + + if (requests.front()->getEncoderInputFeatures()) + { + if (modelConfig.isMultiModal()) + { + auto batchedInputShape = requests.front()->getEncoderInputFeatures()->getShape(); // [1, 3, H, W] + batchedInputShape.d[0] = encoderInputLen; // [batch_size, 3, H, W] + inputFeatures->reshape(batchedInputShape); + } + else + { + SizeType32 const featureDim = requests.front()->getEncoderInputFeatures()->getShape().d[1]; + TLLM_LOG_DEBUG("EncoderBuffers::setFromInputs - featureDim = %d", featureDim); + inputFeatures->reshape(ITensor::makeShape({encoderInputLen, featureDim})); + } + } + + SizeType32 offset = 0; + + for (auto const& llmReq : requests) + { + SizeType32 const inputLength = llmReq->getEncoderInputLen(); + SizeType32 const outputLength = llmReq->getEncoderOutputLen(); + if (llmReq->getEncoderInputFeatures()) + { + auto const& reqFeatures + = llmReq + ->getEncoderInputFeatures(); // whisper: [length, featureDim]; Vision: [batch_size, channel, W, H] + TLLM_LOG_DEBUG("EncoderBuffers::setFromInputs - request id = %d, input features length = %d", + llmReq->mRequestId, inputLength); + manager.copy(*reqFeatures, *ITensor::slice(inputFeatures, offset, inputLength)); + offset += inputLength; + } + else + { + auto const& reqTokens = *llmReq->getEncoderTokens().value(); + inputIdsAll.insert(inputIdsAll.end(), reqTokens.begin(), reqTokens.end()); + if (tokenTypeIds) + { + tokenTypeIdsAll.insert( + tokenTypeIdsAll.end(), tokenTypeIdsReserved.begin(), tokenTypeIdsReserved.begin() + inputLength); + } + } + if (positionIds) + { + SizeType32 const length = modelConfig.isWhisper() ? outputLength : inputLength; + positionIdsAll.insert( + positionIdsAll.end(), positionIdsReserved.begin(), positionIdsReserved.begin() + length); + } + if (modelConfig.useLanguageAdapter()) + { + auto const languageAdapterRouting + = llmReq->getLanguageAdapterRouting(modelConfig.getNumLanguages().value(), inputLength); + languageAdapterRoutingAll.insert( + languageAdapterRoutingAll.end(), std::begin(languageAdapterRouting), std::end(languageAdapterRouting)); + } + inputLengthsAll.push_back(inputLength); + } + + // copy inputs from host to device + { + NVTX3_SCOPED_RANGE(bufferCopies); + if (requests.front()->getEncoderTokens()) + { + manager.copy(inputIdsAll.data(), *inputIds); + if (tokenTypeIds) + { + manager.copy(tokenTypeIdsAll.data(), *tokenTypeIds); + } + manager.copy(maxInputLengthAll.data(), *maxInputLength); + } + if (positionIds) + { + manager.copy(positionIdsAll.data(), *positionIds); + } + manager.copy(inputLengthsAll.data(), *inputLengths); + if (modelConfig.useLanguageAdapter()) + { + manager.copy(languageAdapterRoutingAll.data(), *languageAdapterRoutings); + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::fillIOMaps(ModelConfig const& modelConfig, WorldConfig const& worldConfig) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(runtimeBuffersFillIOMaps); + + inputMap.clear(); + outputMap.clear(); + + // inputs + if (modelConfig.isMultiModal()) + { + inputMap.insert_or_assign("input", inputFeatures); + } + else if (modelConfig.isWhisper()) + { + inputMap.insert_or_assign("input_features", inputFeatures); + inputMap.insert_or_assign("input_lengths", inputLengths); + inputMap.insert_or_assign("position_ids", positionIds); + } + else + { + if (worldConfig.isFirstPipelineParallelRank()) + { + inputMap.insert_or_assign("input_ids", inputIds); + if (positionIds) + { + inputMap.insert_or_assign("position_ids", positionIds); + } + if (tokenTypeIds) + { + inputMap.insert_or_assign("token_type_ids", tokenTypeIds); + } + } + else + { + inputMap.insert_or_assign("hidden_states_input", hiddenStates); + } + inputMap.insert_or_assign("input_lengths", inputLengths); + inputMap.insert_or_assign("max_input_length", maxInputLength); + if (modelConfig.useLanguageAdapter()) + { + inputMap.insert_or_assign("language_adapter_routings", languageAdapterRoutings); + } + } + + // outputs + if (worldConfig.isLastPipelineParallelRank()) + { + outputMap.insert_or_assign("encoder_output", encoderOutput); + } + else + { + outputMap.insert_or_assign("hidden_states_output", hiddenStates); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +std::pair<EncoderBuffers::TensorMap const&, EncoderBuffers::TensorMap&> EncoderBuffers::prepareIO( + RequestVector const& requests, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + updateBufferSizes(requests, modelConfig, worldConfig, runtime); + + setFromInputs(requests, modelConfig, worldConfig, runtime); + + fillIOMaps(modelConfig, worldConfig); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + + return {inputMap, outputMap}; +} + +void EncoderBuffers::rearrangeOutputs(RequestVector const& requests, ModelConfig const& modelConfig, + WorldConfig const& worldConfig, TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(encoderBuffersRearrangeOutput); + + auto const& manager = runtime.getBufferManager(); + + SizeType32 offset = 0, size = 0; + + updateReqOutputShape(requests, runtime, worldConfig, modelConfig); + + for (auto const& req : requests) + { + // copy from internal buffer to request-owned external buffers + size = req->getEncoderOutputLen(); + TLLM_LOG_DEBUG("EncoderBuffers::rearrangeOutputs - req: %d, encoderOutput shape = (%d, %d)", req->mClientId, + req->getEncoderOutput()->getShape().d[0], req->getEncoderOutput()->getShape().d[1]); + TLLM_LOG_DEBUG("EncoderBuffers::rearrangeOutputs - req: %d, enc output size = %d", req->mClientId, size); + + if (worldConfig.isPipelineParallel()) + { + manager.copy(*ITensor::slice(hiddenStates, offset, size), *req->getEncoderHiddenStates()); + } + if (worldConfig.isLastPipelineParallelRank()) + { + if (modelConfig.isMultiModal()) + { + manager.copy( + *ITensor::slice(encoderOutput, offset, size), *(req->getPromptEmbeddingTableMutable().value())); + } + else + { + manager.copy(*ITensor::slice(encoderOutput, offset, size), *req->getEncoderOutput()); + } + } + offset += size; + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::updateReqOutputShape(RequestVector const& requests, TllmRuntime const& runtime, + WorldConfig const& worldConfig, ModelConfig const& modelConfig) +{ + auto const& manager = runtime.getBufferManager(); + + for (auto const& req : requests) + { + if (modelConfig.isMultiModal()) + { + auto shape = encoderOutput->getShape(); // [batch_size, prompt_vocab_size, feature_dim] + shape.d[0] = req->getEncoderOutputLen(); + req->getPromptEmbeddingTableMutable() = manager.emptyTensor(MemoryType::kGPU, encoderOutput->getDataType()); + req->getPromptEmbeddingTableMutable().value()->reshape(shape); + req->setPromptVocabSize(shape.d[1]); + // TODO: extra ids for kv cache reuse + } + else + { + auto encOutLen = req->getEncoderOutputLen(); + // update request-owned external buffer for each request + if (worldConfig.isPipelineParallel()) + { + req->getEncoderHiddenStates()->reshape( + ITensor::makeShape({encOutLen, modelConfig.getHiddenSize() * worldConfig.getTensorParallelism()})); + } + if (worldConfig.isLastPipelineParallelRank()) + { + req->getEncoderOutput()->reshape( + ITensor::makeShape({encOutLen, modelConfig.getHiddenSize() * worldConfig.getTensorParallelism()})); + } + } + } +} + +void EncoderBuffers::create(SizeType32 maxBatchSize, ModelConfig const& modelConfig, TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const& manager = runtime.getBufferManager(); + + inputLengths = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + maxInputLength = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + + hiddenSize = modelConfig.getEncoderHiddenSize(); // full hidden size + // assume encoder & decoder use the same data type + encoderOutput = manager.emptyTensor(MemoryType::kGPU, modelConfig.getDataType()); + encoderOutputReserved = manager.gpu(ITensor::makeShape({1, hiddenSize}), modelConfig.getDataType()); + + crossKvCacheGen = manager.gpu(ITensor::makeShape({1}), nvinfer1::DataType::kBOOL); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::setMaxBufferSizes(SizeType32 maxBatchSize, runtime::ModelConfig const& modelConfig) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + numRequests = maxBatchSize; + encoderInputLen = maxBatchSize * modelConfig.getMaxEncoderLen(); + encoderOutputLen = maxBatchSize * modelConfig.getMaxEncoderLen(); + maxInputLengthInBatch = modelConfig.getMaxEncoderLen(); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::setBufferSizes(RequestVector const& contextRequests, RequestVector const& genRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + numRequests = 0; /// total number of requests that need encoder information (context requests + + /// generation requests * beam width) + encoderInputLen = 0; + encoderOutputLen = 0; + maxInputLengthInBatch = 1; /// maximum encoder length in a batch + + for (auto const& llmReq : contextRequests) + { + numRequests += 1; + encoderInputLen += llmReq->getEncoderInputLen(); + encoderOutputLen += llmReq->getEncoderOutputLen(); + maxInputLengthInBatch = std::max(maxInputLengthInBatch, llmReq->getEncoderInputLen()); + } + + for (auto const& llmReq : genRequests) + { + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + numRequests += reqBeamWidth; // tile by beam width + maxInputLengthInBatch = std::max(maxInputLengthInBatch, llmReq->getEncoderInputLen()); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::reshape() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + inputLengths->reshape(ITensor::makeShape({numRequests})); + maxInputLength->reshape(ITensor::makeShape({maxInputLengthInBatch})); + encoderOutput->reshape(ITensor::makeShape({encoderOutputLen, hiddenSize})); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::fill( + RequestVector const& ctxRequests, RequestVector const& genRequests, runtime::BufferManager const& manager) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(encoderBufferCopies); + + std::vector<SizeType32> inputLengthsAll; + std::vector<SizeType32> maxInputLengthAll(maxInputLength->getShape().d[0], 0); + + SizeType32 offset = 0, size = 0; + for (auto const& requests : {ctxRequests, genRequests}) + { + for (auto const& llmReq : requests) + { + // 1. only ctx requests should gather the encoder output + // 2. only gen requests should tile encoder input lengths info by beam width + bool isCtx = llmReq->isContextInitState(); + if (isCtx) + { + size = llmReq->getEncoderOutputLen(); + auto const encoderOutputSlice = runtime::ITensor::slice(encoderOutput, offset, size); + manager.copy(*llmReq->getEncoderOutput(), *encoderOutputSlice); + offset += size; + + inputLengthsAll.emplace_back(size); + } + else + { + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + std::fill_n(std::back_inserter(inputLengthsAll), reqBeamWidth, + llmReq->getEncoderOutputLen()); // although encoder output is not needed, gen phase still needs the + // encoder length info for cross kv cache. Also tile by beam width + } + } + } + manager.copy(inputLengthsAll.data(), *inputLengths); + manager.copy(maxInputLengthAll.data(), *maxInputLength); + // crossKvCacheGen unused in engine for now, use default tensor + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EncoderBuffers::insertInputTensors(TensorMap& inputMap) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + inputMap.insert_or_assign("encoder_output", encoderOutput); + inputMap.insert_or_assign("encoder_input_lengths", inputLengths); + inputMap.insert_or_assign("encoder_max_input_length", maxInputLength); + inputMap.insert_or_assign("cross_kv_cache_gen", crossKvCacheGen); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/encoderBuffers.h b/cpp/tensorrt_llm/batch_manager/encoderBuffers.h new file mode 100644 index 000000000000..64d416280f21 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/encoderBuffers.h @@ -0,0 +1,140 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +namespace tensorrt_llm::batch_manager +{ + +class EncoderBuffers +{ +public: + using SizeType32 = tensorrt_llm::runtime::SizeType32; + using ITensor = tensorrt_llm::runtime::ITensor; + using TensorPtr = runtime::ITensor::SharedPtr; + using TensorMap = runtime::StringPtrMap<runtime::ITensor>; + using ModelConfig = runtime::ModelConfig; + using WorldConfig = runtime::WorldConfig; + using TllmRuntime = runtime::TllmRuntime; + + TensorPtr inputIds; + TensorPtr positionIds = nullptr; + TensorPtr tokenTypeIds = nullptr; + + TensorPtr inputLengths; // [numEncoderRequests] + TensorPtr maxInputLength; // [maxInputLengthInBatch] + + // intermediate states in pipeline parallelism + TensorPtr hiddenStates; // [numTokens, hiddenSize] + + // features for multimodal encoders (audio, image, etc.) + TensorPtr + inputFeatures; // [totalNumOfFeatures, featureDim] if remove_padding else [batchSize, featureDim, featureLength] + + // language adapter routing information for encoders if language adapter is presented. + TensorPtr languageAdapterRoutings; // [numTokens, numLanguages] + + // encoder output + TensorPtr encoderOutput; // [numEncoderTokens, hiddenSize] + + // output buffer owned by llmRequest, such that it's per-request output buffer + // encoderBuffers class can init and reshape each buffer, without maintaining a list/set of inflight buffers + // TODO in progress: to support BS>1 encoder, need (1) internal scratch space tensors to save the contiguous + // batched output (2) copy from CONTIGUOUS scratch tensor to individual request's DISCRETE output tensor after + // execution To standardize the implementation, for both BS=1 and BS>1, we use internal buffer to store BS=1/BS>1 + // results, and copy to request's external buffers. For BS=1, this introduces a redundancy copy, but ok for now. + + EncoderBuffers() = default; + EncoderBuffers(SizeType32 maxBatchSize, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + TllmRuntime const& runtime); + + std::pair<EncoderBuffers::TensorMap const&, EncoderBuffers::TensorMap&> prepareIO(RequestVector const& requests, + ModelConfig const& modelConfig, WorldConfig const& worldConfig, TllmRuntime const& runtime); + + void rearrangeOutputs(RequestVector const& requests, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + TllmRuntime const& runtime); + + //! @brief set shape of individual request's encoder output (Ptuning embedding table if multimodal) + void updateReqOutputShape(RequestVector const& requests, TllmRuntime const& runtime, WorldConfig const& worldConfig, + ModelConfig const& modelConfig); + +private: + SizeType32 numRequests{}; + SizeType32 encoderInputLen{}; + SizeType32 encoderOutputLen{}; + SizeType32 maxInputLengthInBatch{}; // max input length in a batch + + // prefilled with deterministic values to avoid runtime creation + std::vector<SizeType32> positionIdsReserved; + std::vector<SizeType32> tokenTypeIdsReserved; + + // engine I/O + TensorMap inputMap; + TensorMap outputMap; + + void init(SizeType32 maxBatchSize, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + TllmRuntime const& runtime); + + //! @brief pre-allocate max buffer sizes during init + void initBufferSizes(SizeType32 maxBatchSize, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + TllmRuntime const& runtime); + + //! @brief update actual buffer usage of requests during runtime + void updateBufferSizes(RequestVector const& requests, ModelConfig const& modelConfig, + WorldConfig const& worldConfig, TllmRuntime const& runtime); + + void reshape(TllmRuntime const& runtime, ModelConfig const& modelConfig, WorldConfig const& worldConfig); + + void setFromInputs(RequestVector const& requests, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + TllmRuntime const& runtime); + + void fillIOMaps(ModelConfig const& modelConfig, WorldConfig const& worldConfig); + + // additional members that are Encoder-Decoder specific +private: + TensorPtr encoderOutputReserved; // [1, hiddenSize], dummy tensor for gen phase + TensorPtr crossKvCacheGen; // [1] + SizeType32 hiddenSize; // full hidden size (after multiplying tensor parallelism) + +public: + void create(SizeType32 maxBatchSize, ModelConfig const& modelConfig, TllmRuntime const& runtime); + + SizeType32 getMaxInputLengthInBatch() const + { + return maxInputLengthInBatch; + }; + + void setMaxBufferSizes(SizeType32 maxBatchSize, runtime::ModelConfig const& modelConfig); + + void setBufferSizes(RequestVector const& contextRequests, RequestVector const& genRequests); + + void reshape(); + + void fill( + RequestVector const& ctxRequests, RequestVector const& genRequests, runtime::BufferManager const& manager); + + void insertInputTensors(TensorMap& inputMap); +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/guidedDecoder.cpp b/cpp/tensorrt_llm/batch_manager/guidedDecoder.cpp new file mode 100644 index 000000000000..cb2264ec8003 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/guidedDecoder.cpp @@ -0,0 +1,224 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/guidedDecoder.h" +#include "tensorrt_llm/batch_manager/decoderBuffers.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/common/envUtils.h" +#include "tensorrt_llm/kernels/logitsBitmask.h" + +#include <nlohmann/json.hpp> +#include <xgrammar/xgrammar.h> + +using namespace tensorrt_llm::runtime; + +namespace tensorrt_llm::batch_manager +{ + +GuidedDecoder::GuidedDecoder(executor::GuidedDecodingConfig const& guidedDecodingConfig, SizeType32 maxNumSequences, + SizeType32 vocabSizePadded, nvinfer1::DataType logitsDtype, BufferManager const& runtimeBufferManager) + : mGuidedDecodingBackend{guidedDecodingConfig.getBackend()} + , mMaxNumSequences{maxNumSequences} + , mVocabSizePadded{vocabSizePadded} + , mBitmaskSize{common::ceilDiv(mVocabSizePadded, 32)} + , mLogitsDtype{logitsDtype} + , mCopyBufferManager{std::make_shared<CudaStream>()} +{ + TLLM_CHECK_WITH_INFO(mGuidedDecodingBackend != executor::GuidedDecodingConfig::GuidedDecodingBackend::kLLGUIDANCE, + "LLGuidance is not supported for guided decoding in C++ runtime."); + if (mGuidedDecodingBackend == executor::GuidedDecodingConfig::GuidedDecodingBackend::kXGRAMMAR) + { + mXGrammarMatchers.resize(mMaxNumSequences); + xgrammar::VocabType vocabType = xgrammar::VocabType::RAW; + bool addPrefixSpace = false; + auto const& tokenizerStr = guidedDecodingConfig.getTokenizerStr(); + if (tokenizerStr) + { + auto const& metadata = xgrammar::TokenizerInfo::DetectMetadataFromHF(tokenizerStr.value()); + auto const& metadataJson = nlohmann::json::parse(metadata); + vocabType = metadataJson.at("vocab_type").template get<xgrammar::VocabType>(); + addPrefixSpace = metadataJson.at("add_prefix_space").template get<bool>(); + } + auto const& tokenizerInfo = xgrammar::TokenizerInfo(guidedDecodingConfig.getEncodedVocab().value(), vocabType, + mVocabSizePadded, guidedDecodingConfig.getStopTokenIds(), addPrefixSpace); + + auto const cacheLimitGb = common::getFloatEnv("XGRAMMAR_CACHE_LIMIT_GB"); + mXGrammarCompiler = std::make_shared<xgrammar::GrammarCompiler>(tokenizerInfo, /*max_threads=*/8, + /*cache_enabled=*/true, + /*cache_limit_bytes=*/static_cast<long long>(cacheLimitGb.value_or(1.0f) * 1024 * 1024 * 1024)); + + auto const logitsPtrDtype = BufferDataType{mLogitsDtype, false, true}; + auto constexpr bitmaskDtype = TRTDataType<BitmaskT>::value; + auto constexpr bitmaskPtrDtype = TRTDataType<BitmaskT*>::value; + + mLogitsBitmask = runtimeBufferManager.gpu(ITensor::makeShape({mMaxNumSequences, mBitmaskSize}), bitmaskDtype); + mLogitsBitmaskHost = BufferManager::pinned(ITensor::makeShape({mMaxNumSequences, mBitmaskSize}), bitmaskDtype); + mLogitsBitmaskPtrVec = runtimeBufferManager.gpu(ITensor::makeShape({mMaxNumSequences}), bitmaskPtrDtype); + mLogitsBitmaskPtrVecHost = BufferManager::pinned(ITensor::makeShape({mMaxNumSequences}), bitmaskPtrDtype); + mLogitsPtrVec = runtimeBufferManager.gpu(ITensor::makeShape({mMaxNumSequences}), logitsPtrDtype); + mLogitsPtrVecHost = BufferManager::pinned(ITensor::makeShape({mMaxNumSequences}), logitsPtrDtype); + } +} + +void GuidedDecoder::build(ScheduledRequests const& scheduledRequests) +{ + if (mGuidedDecodingBackend == executor::GuidedDecodingConfig::GuidedDecodingBackend::kXGRAMMAR) + { + for (auto const& requests : {scheduledRequests.contextRequests, scheduledRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + auto const& guidedDecodingParams = llmReq->getGuidedDecodingParams(); + if (!guidedDecodingParams.has_value()) + { + continue; + } + auto const seqSlot = llmReq->mSeqSlot.value(); + if (llmReq->isContextInitState() && llmReq->isFirstContextChunk()) + { + // The request is in the first context forward step (considering kv cache reuse). + auto const& guideType = guidedDecodingParams->getGuideType(); + auto const& guide = guidedDecodingParams->getGuide(); + switch (guideType) + { + case executor::GuidedDecodingParams::GuideType::kJSON: + { + mXGrammarMatchers.at(seqSlot) = std::make_shared<xgrammar::GrammarMatcher>( + mXGrammarCompiler->CompileBuiltinJSONGrammar()); + break; + } + case executor::GuidedDecodingParams::GuideType::kJSON_SCHEMA: + { + mXGrammarMatchers.at(seqSlot) = std::make_shared<xgrammar::GrammarMatcher>( + mXGrammarCompiler->CompileJSONSchema(guide.value())); + break; + } + case executor::GuidedDecodingParams::GuideType::kREGEX: + { + mXGrammarMatchers.at(seqSlot) = std::make_shared<xgrammar::GrammarMatcher>( + mXGrammarCompiler->CompileRegex(guide.value())); + break; + } + case executor::GuidedDecodingParams::GuideType::kEBNF_GRAMMAR: + { + mXGrammarMatchers.at(seqSlot) = std::make_shared<xgrammar::GrammarMatcher>( + mXGrammarCompiler->CompileGrammar(guide.value())); + break; + } + case executor::GuidedDecodingParams::GuideType::kSTRUCTURAL_TAG: + { + mXGrammarMatchers.at(seqSlot) = std::make_shared<xgrammar::GrammarMatcher>( + mXGrammarCompiler->CompileStructuralTag(guide.value())); + break; + } + default: + { + TLLM_THROW("Unsupported guide type."); + } + } + } + else if (llmReq->isGenerationInProgressState()) + { + // The request is in a generation forward step. + // Currently, guided decoding does not support with beam search. + mXGrammarMatchers.at(seqSlot)->AcceptToken(llmReq->getLastTokens(0)); + } + else + { + continue; + } + + // Fill the bitmask on host and asynchorously copy to device using mCopyBufferManager. + auto const logitsBitmask = ITensor::at(mLogitsBitmask, {seqSlot}); + auto const logitsBitmaskHost = ITensor::at(mLogitsBitmaskHost, {seqSlot}); + + std::array<int64_t, 1> bitmaskShape{mBitmaskSize}; + DLTensor logitsBitmaskDlt{logitsBitmaskHost->data(), DLDevice{kDLCPU, 0}, 1, DLDataType{kDLInt, 32, 1}, + bitmaskShape.data(), nullptr, 0}; + mXGrammarMatchers.at(seqSlot)->FillNextTokenBitmask(&logitsBitmaskDlt); + mCopyBufferManager.copy(*logitsBitmaskHost, *logitsBitmask); + } + } + } +} + +void GuidedDecoder::execute(DecoderInputBuffers const& decoderInputBuffers, BufferManager const& runtimeBufferManager) +{ + auto const& stream = runtimeBufferManager.getStream(); + + // Wait for mCopyBufferManager finishing the H2D copy of logitsBitmask + // TODO(enweiz): Move the H2D copy of logitsBitmaskPtrVec to buildGuidedDecoding. + // This may not bring too much perf gain because of the small size of logitsBitmaskPtrVec. + // TODO(enweiz): For chunked context, we currently build mask cache at the first context chunk, and apply + // the mask at the last context chunk. So, ideally we should sync the stream at the last context chunk. + CudaEvent event{}; + mCopyBufferManager.getStream().record(event); + stream.wait(event); + + if (mGuidedDecodingBackend == executor::GuidedDecodingConfig::GuidedDecodingBackend::kXGRAMMAR + && !decoderInputBuffers.decoderRequests.empty()) + { + SizeType32 batchIdx{0}; + for (size_t requestIdx = 0; requestIdx < decoderInputBuffers.decoderRequests.size(); ++requestIdx) + { + auto const& llmReq = decoderInputBuffers.decoderRequests.at(requestIdx); + + auto const& guidedDecodingParams = llmReq->getGuidedDecodingParams(); + if (guidedDecodingParams.has_value()) + { + auto const seqSlot = llmReq->mSeqSlot.value(); + + auto const& logits = decoderInputBuffers.decoderLogits.at(requestIdx); + auto const logitsBitmask = ITensor::at(mLogitsBitmask, {seqSlot}); + + // Use void* to unify the code for different mLogitsDtype + *reinterpret_cast<void**>(ITensor::at(mLogitsPtrVecHost, {batchIdx})->data()) = logits->data(); + *reinterpret_cast<void**>(ITensor::at(mLogitsBitmaskPtrVecHost, {batchIdx})->data()) + = logitsBitmask->data(); + + ++batchIdx; + } + } + if (batchIdx > 0) + { + runtimeBufferManager.copy( + *ITensor::slice(mLogitsPtrVecHost, 0, batchIdx), *ITensor::slice(mLogitsPtrVec, 0, batchIdx)); + runtimeBufferManager.copy(*ITensor::slice(mLogitsBitmaskPtrVecHost, 0, batchIdx), + *ITensor::slice(mLogitsBitmaskPtrVec, 0, batchIdx)); + + auto logitsBitmaskPtrVec = bufferCast<BitmaskT const*>(*mLogitsBitmaskPtrVec); + if (mLogitsDtype == nvinfer1::DataType::kFLOAT) + { + auto logitsPtrVec = bufferCast<float*>(*mLogitsPtrVec); + tensorrt_llm::kernels::invokeLogitsBitmask<float>( + logitsPtrVec, logitsBitmaskPtrVec, batchIdx, mVocabSizePadded, stream.get()); + } + else if (mLogitsDtype == nvinfer1::DataType::kHALF) + { + auto logitsPtrVec = bufferCast<half*>(*mLogitsPtrVec); + tensorrt_llm::kernels::invokeLogitsBitmask<half>( + logitsPtrVec, logitsBitmaskPtrVec, batchIdx, mVocabSizePadded, stream.get()); + } + else + { + TLLM_THROW("Unsupported logits data type."); + } + } + } +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/handleContextLogits.cpp b/cpp/tensorrt_llm/batch_manager/handleContextLogits.cpp new file mode 100644 index 000000000000..6f4a541ffcbb --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/handleContextLogits.cpp @@ -0,0 +1,176 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/handleContextLogits.h" + +#include "tensorrt_llm/batch_manager/decoderBuffers.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/medusaBuffers.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/runtimeKernels.h" +#include "tensorrt_llm/runtime/utils/debugUtils.h" + +namespace tr = tensorrt_llm::runtime; +namespace tru = tensorrt_llm::runtime::utils; + +namespace tensorrt_llm::batch_manager +{ + +using BufferManager = tensorrt_llm::runtime::BufferManager; +using TensorPtr = runtime::ITensor::SharedPtr; +using ITensor = runtime::ITensor; +using SizeType32 = tensorrt_llm::runtime::SizeType32; + +namespace +{ + +//! @brief Copy logits from context phase to beginning of generation logits. +//! @details Usually, this concerns logits of 1 token. In speculative decoding this concerns draftLen + 1 tokens. +void copyLastContextLogits(TensorPtr const& contextLogits, LlmRequest& llmReq, BufferManager const& bufferManager) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + auto const numLogits = contextLogits->getShape().d[0]; + for (int beam = 0; beam < llmReq.getBeamWidthByIter(); beam++) + { + // [beamWidth, mMaxNewTokens, vocabSizePadded] -> [numLogits, vocabSizePadded] + auto beamHostTensorPtr = ITensor::slice(llmReq.getGenerationLogitsHost(), {beam, 0}, numLogits); + bufferManager.copy(*contextLogits, *beamHostTensorPtr); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void setupMedusaLogits(std::vector<TensorPtr>& medusaLogitsHeads, TensorPtr const& medusaLogitsDevice, + SizeType32 medusaHeads, SizeType32 logitsIndex, SizeType32 numLogits) +{ + for (SizeType32 hi = 0; hi < medusaHeads; ++hi) + { + TensorPtr logitsHead = ITensor::slice(medusaLogitsDevice, hi, 1); + logitsHead->squeeze(0); + medusaLogitsHeads[hi] = ITensor::slice(logitsHead, logitsIndex, numLogits); + } +} + +} // namespace + +SizeType32 HandleContextLogits::operator()(DecoderInputBuffers& inputBuffers, RequestVector const& contextRequests, + tr::ITensor::SharedPtr const& logits, std::vector<tr::SizeType32> const& numContextLogitsVec, + tr::ModelConfig const& modelConfig, tr::BufferManager const& manager, + OptionalRef<MedusaBuffers> medusaBuffers) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(HandleContextLogits); + + auto& decoderRequests = inputBuffers.decoderRequests; + decoderRequests.clear(); + decoderRequests.reserve(contextRequests.size()); + auto& allDecoderLogits = inputBuffers.decoderLogits; + allDecoderLogits.clear(); + allDecoderLogits.reserve(contextRequests.size()); + + SizeType32 batchIndex{0}; + SizeType32 logitsIndex{0}; + // Copy logits into decoderBuffers.logits + for (auto const& llmReq : contextRequests) + { + auto const numContextLogits = numContextLogitsVec.at(batchIndex); + auto const draftLength = llmReq->isLastContextChunk() ? llmReq->getNumDraftTokens() : 0; + + TLLM_LOG_DEBUG("logitsIndex: %d", logitsIndex); + TLLM_LOG_DEBUG("numContextLogits %d", numContextLogits); + TLLM_LOG_DEBUG("draftLength: %d", draftLength); + + if (modelConfig.computeContextLogits()) + { + // Since the computational graph has been modified, only the last token is needed. + TLLM_CHECK_WITH_INFO(!modelConfig.getSpeculativeDecodingMode().isMedusa() + && !modelConfig.getSpeculativeDecodingMode().isLookaheadDecoding(), + "Return context logits is not supported with Medusa and Lookahead decoding"); + + if (llmReq->getReturnContextLogits()) + { + if (llmReq->getPrepopulatedPromptLen() > 0) + { + TLLM_LOG_WARNING( + "Because of KV cache reuse, not all context logits could be produced for request %lu.", + llmReq->mRequestId); + } + TensorPtr contextLogitsDeviceView = ITensor::slice(logits, logitsIndex, numContextLogits); + TensorPtr contextLogitsHostView = ITensor::slice( + llmReq->getContextLogitsHost(), llmReq->getContextCurrentPosition(), numContextLogits); + // Copy to host directly + manager.copy(*contextLogitsDeviceView, *contextLogitsHostView); + } + } + logitsIndex += numContextLogits + draftLength; + + // Get the logits from the last context token and draft tokens + auto const numDecoderLogits = 1 + draftLength; + auto const seqSlot = llmReq->mSeqSlot.value(); + TensorPtr logitsView = ITensor::slice(logits, logitsIndex - numDecoderLogits, numDecoderLogits); + + if (modelConfig.getSpeculativeDecodingMode().hasDraftLogits()) + { + auto& medusaLogitsHeads = inputBuffers.predictedDraftLogits.at(seqSlot); + TLLM_CHECK(medusaBuffers); + setupMedusaLogits(medusaLogitsHeads, medusaBuffers->medusaLogitsDevice, + modelConfig.getSpeculativeDecodingModule().getMaxDraftPathLen(), logitsIndex - numDecoderLogits, + numDecoderLogits); + } + + // Save the last token logits of context into generation logits or + // save the accepted token logits from target model + if (llmReq->getReturnGenerationLogits()) + { + copyLastContextLogits(logitsView, *llmReq, manager); + } + + TLLM_CHECK_DEBUG_WITH_INFO(tru::tensorHasInvalid<float>(*logitsView, manager, "logits") == false, + "Found invalid number (NaN or Inf) in logits"); + + if (llmReq->isLastContextChunk()) + { + TensorPtr decoderLogits; + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + if (reqBeamWidth > 1) + { + // Tile logits of context requests + auto const& logitsShape = logitsView->getShape(); + auto const logitsType = logitsView->getDataType(); + decoderLogits = manager.gpu(ITensor::makeShape({reqBeamWidth, logitsShape.d[1]}), logitsType); + tensorrt_llm::runtime::kernels::tileTensor( + *decoderLogits, *logitsView, reqBeamWidth, manager.getStream()); + decoderLogits->unsqueeze(0); + } + else + { + decoderLogits = logitsView; + decoderLogits->unsqueeze(1); + } + decoderRequests.push_back(llmReq); + allDecoderLogits.emplace_back(std::move(decoderLogits)); + } + + ++batchIndex; + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return logitsIndex; +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/handleGenerationLogits.cpp b/cpp/tensorrt_llm/batch_manager/handleGenerationLogits.cpp new file mode 100644 index 000000000000..e2a7486b050a --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/handleGenerationLogits.cpp @@ -0,0 +1,161 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/handleGenerationLogits.h" + +#include "tensorrt_llm/batch_manager/decoderBuffers.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/medusaBuffers.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" +#include "tensorrt_llm/batch_manager/utils/inflightBatchingUtils.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/utils/debugUtils.h" + +namespace tr = tensorrt_llm::runtime; +namespace tru = tensorrt_llm::runtime::utils; + +namespace tensorrt_llm::batch_manager +{ + +using BufferManager = tensorrt_llm::runtime::BufferManager; +using TensorPtr = runtime::ITensor::SharedPtr; +using ITensor = runtime::ITensor; +using SizeType32 = tensorrt_llm::runtime::SizeType32; + +namespace +{ + +//! @brief Copy logits from generation phase under streaming mode. +void copyStreamingGenerationLogits(BufferManager const& bufferManager, LlmRequest& llmReq) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + // If llmRequest is streaming, directly copy to host. + // Only one token's logits needs to be copied each time. + TLLM_CHECK(llmReq.getGenerationLogitsFragmentsSize() == 1); + + SizeType32 numGenerationToken = llmReq.getMaxBeamNumTokens() - llmReq.mPromptLen; + TensorPtr const& generationLogitsHost + = llmReq.getGenerationLogitsHost(); // [mMaxNewTokens (or 1), beamWidth, vocabSizePadded] + + TensorPtr hostTensorPtr + = ITensor::slice(generationLogitsHost, numGenerationToken, 1); // [1, beamWidth, vocabSizePadded] + TensorPtr deviceTensorPtr = *(llmReq.getGenerationLogitsFragments().begin()); + + bufferManager.copy(*deviceTensorPtr, *hostTensorPtr); + llmReq.clearGenerationLogitsFragments(); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void setupMedusaLogits(std::vector<TensorPtr>& medusaLogitsHeads, TensorPtr const& medusaLogitsDevice, + SizeType32 medusaHeads, SizeType32 logitsIndex, SizeType32 numLogits) +{ + for (SizeType32 hi = 0; hi < medusaHeads; ++hi) + { + TensorPtr logitsHead = ITensor::slice(medusaLogitsDevice, hi, 1); + logitsHead->squeeze(0); + medusaLogitsHeads[hi] = ITensor::slice(logitsHead, logitsIndex, numLogits); + } +} + +} // namespace + +void HandleGenerationLogits::operator()(DecoderInputBuffers& inputBuffers, RequestVector const& generationRequests, + tr::ITensor::SharedPtr const& logits, tr::SizeType32 logitsIndex, tr::ModelConfig const& modelConfig, + tr::BufferManager const& manager, OptionalRef<RuntimeBuffers> genRuntimeBuffers, + OptionalRef<MedusaBuffers> medusaBuffers) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(HandleGenerationLogits); + + auto& decoderRequests = inputBuffers.decoderRequests; + decoderRequests.reserve(decoderRequests.size() + generationRequests.size()); + auto& allDecoderLogits = inputBuffers.decoderLogits; + allDecoderLogits.reserve(allDecoderLogits.size() + generationRequests.size()); + + for (auto const& llmReq : generationRequests) + { + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + auto const seqSlot = llmReq->mSeqSlot.value(); + + auto const draftLength = llmReq->getNumDraftTokens(); + auto const numLogits = draftLength + reqBeamWidth; + + TLLM_CHECK(draftLength == 0 || reqBeamWidth == 1); + + TLLM_LOG_DEBUG("logitsIndex: %d", logitsIndex); + TLLM_LOG_DEBUG("draftLength: %d", draftLength); + TLLM_LOG_DEBUG("reqBeamWidth: %d", reqBeamWidth); + + // genRuntimeBuffers.logits shape: [numGen*reqBeamWidth, vocabSize] + // logitsView shape: [numLogits, vocabSize] + TensorPtr logitsView = ITensor::slice(logits, logitsIndex, numLogits); + TLLM_CHECK_DEBUG_WITH_INFO(tru::tensorHasInvalid<float>(*logitsView, manager, "logits") == false, + "Found invalid number (NaN or Inf) in logits"); + + TLLM_CHECK(llmReq->isGenerationInProgressState()); + TensorPtr decoderLogits; + if (reqBeamWidth > 1) + { + decoderLogits = logitsView; + decoderLogits->unsqueeze(0); + } + else + { + decoderLogits = logitsView; + decoderLogits->unsqueeze(1); + } + decoderRequests.push_back(llmReq); + allDecoderLogits.emplace_back(std::move(decoderLogits)); + + if (llmReq->getReturnGenerationLogits()) + { + TLLM_CHECK_WITH_INFO(modelConfig.getSpeculativeDecodingMode().isNone() + || modelConfig.getSpeculativeDecodingMode().isDraftTokensExternal(), + "Only speculative decoding with external draft tokens supports returning generation logits"); + + // Push into fragments vector + llmReq->addGenerationLogitsFragment(logitsView); + TLLM_CHECK( + llmReq->getGenerationLogitsFragmentsSize() <= RuntimeBuffers::GenerationLogitsCache::kCACHE_LENGTH); + if (llmReq->isStreaming()) + { + copyStreamingGenerationLogits(manager, *llmReq); + } + // Copy back to host for every kCACHE_LENGTH steps to mitigate GPU memory pressure + else if (llmReq->getGenerationLogitsFragmentsSize() == RuntimeBuffers::GenerationLogitsCache::kCACHE_LENGTH) + { + TLLM_CHECK(genRuntimeBuffers); + auto constexpr beforeDecoder = true; + utils::copyGenerationLogits(genRuntimeBuffers->generationLogitsCache, manager, *llmReq, beforeDecoder); + } + } + if (modelConfig.getSpeculativeDecodingMode().hasDraftLogits()) + { + auto& medusaLogitsHeads = inputBuffers.predictedDraftLogits.at(seqSlot); + TLLM_CHECK(medusaBuffers); + setupMedusaLogits(medusaLogitsHeads, medusaBuffers->medusaLogitsDevice, + modelConfig.getSpeculativeDecodingModule().getMaxDraftPathLen(), logitsIndex, draftLength); + } + logitsIndex += numLogits; + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp b/cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp index 8c1ffb70e372..c4d265289a71 100644 --- a/cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp +++ b/cpp/tensorrt_llm/batch_manager/kvCacheManager.cpp @@ -27,7 +27,6 @@ #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/kernels/kvCacheIndex.h" #include "tensorrt_llm/runtime/common.h" @@ -587,7 +586,7 @@ std::map<SizeType32, float> BlockManager::calculateWindowSizeToShare( BlockManager::BlockManager(std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, CudaStreamPtr stream, SizeType32 maxSequenceLength, SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, - tensorrt_llm::DataType dtype, SizeType32 sinkBubbleLength, SizeType32 chunkSize, CacheType cacheType, + nvinfer1::DataType dtype, SizeType32 sinkBubbleLength, SizeType32 chunkSize, CacheType cacheType, std::optional<executor::RetentionPriority> secondaryOffloadMinPriority, std::shared_ptr<KVCacheEventManager> eventManager, bool enablePartialReuse, bool copyOnPartialReuse, std::shared_ptr<kv_connector::KvCacheConnectorManager> kvCacheConnectorManager, @@ -741,7 +740,7 @@ BlockManager::BlockManager(std::vector<SizeType32> const& numKvHeadsPerLayer, Si "Maybe you tried changing either of them to an std::unordered_map?"); } -WindowBlockManager::WindowBlockManager(tensorrt_llm::DataType dtype, SizeType32 windowSize, +WindowBlockManager::WindowBlockManager(nvinfer1::DataType dtype, SizeType32 windowSize, std::vector<SizeType32> const& managedLayers, std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, bool isSWA, SizeType32 blocksInPrimaryPool, SizeType32 blocksInSecondaryPool, SizeType32 maxNumSequences, std::shared_ptr<runtime::CudaStream> stream, @@ -841,7 +840,7 @@ WindowBlockManager::WindowBlockManager(tensorrt_llm::DataType dtype, SizeType32 // to specify FP4 related parameters (scale dtypes, etc)? This can also be passed // in the constructor. constexpr SizeType32 kQuantBlockSizeNVFP4 = 16; - if (dtype == tensorrt_llm::DataType::kFP4) + if (dtype == nvinfer1::DataType::kFP4) { createBlockScalePools(kQuantBlockSizeNVFP4); } @@ -937,7 +936,6 @@ bool BlockManager::verifyQueueIntegrity(SizeType32 windowSize) const bool WindowBlockManager::verifyQueueIntegrity() const { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); return mEvictionPolicy->verifyQueueIntegrity(); } @@ -1094,7 +1092,7 @@ void BlockManager::allocatePools(bool useUvm) void WindowBlockManager::allocatePools(bool useUvm) { - constexpr tensorrt_llm::DataType kScaleDtypeNVFP4 = tensorrt_llm::DataType::kFP8; + constexpr nvinfer1::DataType kScaleDtypeNVFP4 = nvinfer1::DataType::kFP8; bool const requestFabricMemory = tc::getEnvKVCachePoolUseFabricMemory(); bool const fabricMemorySupported = FabricMemory::supportFabricMemory(); @@ -1120,21 +1118,21 @@ void WindowBlockManager::allocatePools(bool useUvm) auto blockSize = pool.blockSize; auto poolDtype = pool.containsBlockScales ? kScaleDtypeNVFP4 : mDataType; #ifdef ENABLE_FP4 - auto const poolIsFP4 = poolDtype == tensorrt_llm::DataType::kFP4; + auto const poolIsFP4 = poolDtype == nvinfer1::DataType::kFP4; #else auto const poolIsFP4 = false; #endif if (poolIsFP4) { - poolDtype = tensorrt_llm::DataType::kINT8; + poolDtype = nvinfer1::DataType::kINT8; } if (pool.containsIndexerKCache) { - poolDtype = tensorrt_llm::DataType::kUINT8; + poolDtype = nvinfer1::DataType::kUINT8; } - tensorrt_llm::Dims cacheShape = isRecurrentState() + nvinfer1::Dims cacheShape = isRecurrentState() ? ITensor::makeShape({pool.numLayers, mNumPrimaryBlocks, mKVFactor, blockSize}) : ITensor::makeShape({mNumPrimaryBlocks, pool.numLayers, mKVFactor, blockSize}); pool.layerFirstLayout = isRecurrentState(); @@ -1168,7 +1166,7 @@ void WindowBlockManager::allocatePools(bool useUvm) if (mNumSecondaryBlocks > 0) { - tensorrt_llm::Dims cacheShapeOffload = isRecurrentState() + nvinfer1::Dims cacheShapeOffload = isRecurrentState() ? ITensor::makeShape({pool.numLayers, mNumSecondaryBlocks, mKVFactor, blockSize}) : ITensor::makeShape({mNumSecondaryBlocks, pool.numLayers, mKVFactor, blockSize}); TLLM_LOG_DEBUG("[%s] Allocating secondary pool with %d blocks for %d layers with %d kv heads", @@ -1220,7 +1218,6 @@ void BlockManager::startScheduling() void WindowBlockManager::startScheduling() { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); mSchedulingNumFreeBlocks = mEvictionPolicy->getNumFreeBlocks(kPrimaryLevel); for (auto& [requestId, slotAllocatedBlocks] : mAllocatedBlocksPerSeq) { @@ -1240,7 +1237,6 @@ void WindowBlockManager::freeLeafBlock(BlockPtr const& block) void WindowBlockManager::releaseSubtree(BlockPtr const& block) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); // Iterative pre-order DFS over `block` and all its descendants. Collect // first, then detach in reverse order: cascade-prune in freeLeafBlock() // removes empty parent nodes from the trie, so leaves must be detached @@ -1278,7 +1274,6 @@ BlockPtr WindowBlockManager::getFreeBlock(GenerationRequest& sequence, executor: std::optional<std::chrono::milliseconds> durationMs, executor::KvCacheTransferMode mode, std::string const& directory, bool wantPlaceholder) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); // eviction policy get free primary block auto [block, canOffload] = mEvictionPolicy->getFreeBlock(kPrimaryLevel, wantPlaceholder); if (block->getUniqueTokens().empty()) @@ -1349,7 +1344,7 @@ BlockPtr WindowBlockManager::getFreeBlock(GenerationRequest& sequence, executor: return block; } -void WindowBlockManager::setOffsets(tk::KVCacheIndex* offsetsPtr, tensorrt_llm::Dims const& offsetsShape, +void WindowBlockManager::setOffsets(tk::KVCacheIndex* offsetsPtr, nvinfer1::Dims const& offsetsShape, SizeType32 beamIdx, SizeType32 blockIdx, KVCacheBlock::IdType blockId) const { auto constexpr kIdx = 0; @@ -1387,7 +1382,7 @@ void WindowBlockManager::setOffsets(tk::KVCacheIndex* offsetsPtr, tensorrt_llm:: } } -void BlockManager::setOffsets(tk::KVCacheIndex* offsetsPtr, tensorrt_llm::Dims const& offsetsShape, SizeType32 beamIdx, +void BlockManager::setOffsets(tk::KVCacheIndex* offsetsPtr, nvinfer1::Dims const& offsetsShape, SizeType32 beamIdx, SizeType32 blockIdx, KVCacheBlock::IdType blockId, SizeType32 windowSize) const { mWindowBlockManagers.at(windowSize).setOffsets(offsetsPtr, offsetsShape, beamIdx, blockIdx, blockId); @@ -1402,7 +1397,6 @@ void BlockManager::onboardBlock(GenerationRequest& sequence, BlockPtr const& off void WindowBlockManager::onboardBlock(GenerationRequest& sequence, BlockPtr const& offloadBlock, executor::KvCacheTransferMode mode, std::string const& directory) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); if (!offloadBlock->isPlaceholder() && !offloadBlock->isPrimary()) { auto block = getFreeBlock( @@ -1431,7 +1425,6 @@ void BlockManager::offloadBlock( void WindowBlockManager::offloadBlock( BlockPtr const& block, executor::KvCacheTransferMode mode, std::string const& directory) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); // The current default behavior is to offload the out-of-window block // to secondary block pool to allow more free primary blocks for reuse. // However, such behavior does not take account whether the offloaded @@ -2137,19 +2130,6 @@ bool WindowBlockManager::blockInRadixTree(BlockPtr const& block) } std::shared_ptr<KVCacheBlock> WindowBlockManager::findBlocksInReuseTreeByBlockKey(BlockKey const& blockKey) -{ - std::vector<KVCacheBlock::IdType> unusedPinnedBlockIds; - return findBlocksInReuseTreeByBlockKeyImpl(blockKey, /*pinBlocks=*/false, unusedPinnedBlockIds); -} - -std::shared_ptr<KVCacheBlock> WindowBlockManager::findBlocksInReuseTreeByBlockKey( - BlockKey const& blockKey, std::vector<KVCacheBlock::IdType>& pinnedBlockIds) -{ - return findBlocksInReuseTreeByBlockKeyImpl(blockKey, /*pinBlocks=*/true, pinnedBlockIds); -} - -std::shared_ptr<KVCacheBlock> WindowBlockManager::findBlocksInReuseTreeByBlockKeyImpl( - BlockKey const& blockKey, bool pinBlocks, std::vector<KVCacheBlock::IdType>& pinnedBlockIds) { std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); auto blockedUniqueTokens @@ -2162,42 +2142,7 @@ std::shared_ptr<KVCacheBlock> WindowBlockManager::findBlocksInReuseTreeByBlockKe blockKeys.emplace_back(blockKey.usesExtraIds, blockKey.loraTaskId, blockedUniqueTokensList, blockKey.extraKeys, blockKey.cacheSalt); } - auto searchRoot = mCachedBlocksRoot; - std::vector<BlockPtr> pinnedInScope; - for (auto const& blockKey : blockKeys) - { - auto [partialMatch, numMatched, matchingBlock] = searchRoot != nullptr - ? searchRoot->findMatchingBlock(blockKey, true, true) - : std::make_tuple(false, 0, nullptr); - - // A prefix-only match lacks KV for the requested tail, and pinning (transfer) - // lookups read primary-pool buffers directly, so offloaded and placeholder blocks - // are misses too (isPlaceholder first: isPrimary asserts on placeholders). - bool const fullyMatched - = matchingBlock != nullptr && numMatched == static_cast<SizeType32>(blockKey.uniqueTokens.size()); - bool const transferable - = fullyMatched && (!pinBlocks || (!matchingBlock->isPlaceholder() && matchingBlock->isPrimary())); - if (!transferable) - { - // Roll back pins taken during the partial walk. - for (auto const& block : pinnedInScope) - { - unpinBlock(block); - } - pinnedBlockIds.clear(); - return nullptr; - } - - if (pinBlocks) - { - pinBlock(matchingBlock); - pinnedInScope.push_back(matchingBlock); - pinnedBlockIds.push_back(matchingBlock->getBlockId()); - } - - searchRoot = std::move(matchingBlock); - } - return searchRoot; + return searchReuseTree(blockKeys); } std::shared_ptr<KVCacheBlock> WindowBlockManager::findBlocksInReuseTreeByBlockKeys( @@ -2246,7 +2191,6 @@ void BlockManager::refreshBlocks() void WindowBlockManager::refreshBlocks() { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); mEvictionPolicy->refresh(); mTransferManager->syncTransfers(); } @@ -2377,7 +2321,6 @@ bool BlockManager::copyLinearAttentionBlock(GenerationRequest& sequence, LlmRequ bool WindowBlockManager::tryAllocatePlaceholderForLinearAttention(GenerationRequest& sequence, bool shareAmongBeams) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); auto const beamWidth = sequence.getBeamWidth(); auto const newBlockIdx = sequence.getCacheBlockIds(mWindowSize).at(0).size(); // The first block is not a placeholder. @@ -2467,7 +2410,6 @@ bool WindowBlockManager::tryAllocatePlaceholderForLinearAttention(GenerationRequ void WindowBlockManager::allocateBlock(GenerationRequest& sequence, bool shareAmongBeams) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); auto const beamWidth = sequence.getBeamWidth(); auto const requiredBlocks = shareAmongBeams ? 1 : beamWidth; @@ -2749,7 +2691,16 @@ std::pair<SizeType32, std::vector<KVCacheBlock::IdType>> WindowBlockManager::sto if (pinBlocks) { - pinBlock(prevBlock); + // If the block has no refs it sits in the eviction policy's free + // queue. Claim it first so that the later unpinBlocksById / + // releaseBlock cycle does not create a duplicate queue entry. + // Pass the block's existing priority and duration so that + // claimBlock does not clear its retention/expiry metadata. + if (!prevBlock->hasRefs()) + { + mEvictionPolicy->claimBlock(prevBlock, prevBlock->getPriority(), prevBlock->getDurationMs()); + } + prevBlock->incRefCount(); pinnedBlockIds.push_back(prevBlock->getBlockId()); } } @@ -2784,7 +2735,6 @@ void BlockManager::replaceSharedBlock(GenerationRequest& sequence, SizeType32 wi void WindowBlockManager::replaceSharedBlock(GenerationRequest& sequence, SizeType32 blockIdx) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); auto const requestId = sequence.getRequestId(); auto const beamWidth = sequence.getBeamWidth(); auto& allocatedBlocks = mAllocatedBlocksPerSeq.at(requestId); @@ -2836,7 +2786,6 @@ void BlockManager::releaseLastBlock(GenerationRequest& sequence, SizeType32 wind void WindowBlockManager::releaseLastBlock(GenerationRequest& sequence) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); if (isRecurrentState()) { // In recurrent state, the last block always contains the current state and should not be released. @@ -2885,7 +2834,6 @@ void WindowBlockManager::releaseLastBlock(GenerationRequest& sequence) [[nodiscard]] SizeType32 WindowBlockManager::getNumFreeBlocks() const { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); auto const numFree = mEvictionPolicy->getNumFreeBlocks(kPrimaryLevel); TLLM_CHECK_WITH_INFO(numFree <= getMaxNumBlocks(), "%s::getNumFreeBlocks - primary free block count (%d) exceeds total block count (%d). " @@ -2897,7 +2845,6 @@ void WindowBlockManager::releaseLastBlock(GenerationRequest& sequence) [[nodiscard]] SizeType32 WindowBlockManager::getNumFreeSecondaryBlocks() const noexcept { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); return mEvictionPolicy->getNumFreeBlocks(kSecondaryLevel); } @@ -3028,41 +2975,18 @@ void BlockManager::unpinBlocksById(std::vector<KVCacheBlock::IdType> const& bloc firstManager.unpinBlocksById(blockIds); } -void WindowBlockManager::pinBlock(BlockPtr const& block) -{ - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); - // Claim free blocks out of the eviction queue first, keeping their retention metadata. - if (!block->hasRefs()) - { - mEvictionPolicy->claimBlock(block, block->getPriority(), block->getDurationMs()); - } - block->incRefCount(); -} - -void WindowBlockManager::unpinBlock(BlockPtr const& block) -{ - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); - block->decRefCount(); - if (!block->hasRefs()) - { - mEvictionPolicy->releaseBlock(block); - } -} - void WindowBlockManager::pinBlocks(GenerationRequest& sequence) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); auto const requestId = sequence.getRequestId(); auto& allocatedBlocks = mAllocatedBlocksPerSeq.at(requestId); for (auto& block : allocatedBlocks) { - pinBlock(block); + block->incRefCount(); } } void WindowBlockManager::unpinBlocksById(std::vector<KVCacheBlock::IdType> const& blockIds) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); if (blockIds.empty()) { return; @@ -3075,7 +2999,11 @@ void WindowBlockManager::unpinBlocksById(std::vector<KVCacheBlock::IdType> const auto block = mAllBlocksById[blockId]; if (block && block->getBlockId() != KVCacheBlock::kCachedBlocksRootId) { - unpinBlock(block); + block->decRefCount(); + if (!block->hasRefs()) + { + mEvictionPolicy->releaseBlock(block); + } } } } @@ -3091,7 +3019,6 @@ void BlockManager::storeNewBlock(GenerationRequest& sequence, OptionalRef<LlmReq void WindowBlockManager::storeNewBlock(GenerationRequest& sequence, OptionalRef<LlmRequest const> llmRequest) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); auto constexpr beamIdx = 0; auto const& uniqueTokens = llmRequest->getUniqueTokens(beamIdx); @@ -3198,7 +3125,6 @@ std::vector<KVCacheBlock::IdType> WindowBlockManager::storeBlocksForReuse( std::optional<KVCacheBlock::IdType> WindowBlockManager::releaseBlocks( GenerationRequest& sequence, OptionalRef<LlmRequest const> llmRequest) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); auto const requestId = sequence.getRequestId(); TLLM_LOG_DEBUG("%s::releaseBlocks - requestId=%lu, llmRequest.id=%s", mLogPrefix.c_str(), requestId, llmRequest.has_value() ? std::to_string(llmRequest->mRequestId).c_str() : "null"); @@ -3256,16 +3182,9 @@ std::optional<KVCacheBlock::IdType> WindowBlockManager::releaseBlocks( // mRefCount==0 and are silently ignored by EvictionPolicy::releaseBlock(). if (!block->hasRefs()) { - auto const isDetached = block->isDetached(); - if (isDetached) - { - // Detached blocks have no reusable hash chain. Drop the stale owning link before recycling the block; - // otherwise front-queue reuse can join completed request chains into an unbounded ownership chain. - block->setPrevBlockInSeq(nullptr); - } // Send block to front of free queue if it has no reusable state, // so detached blocks are evicted before blocks cached for reuse. - mEvictionPolicy->releaseBlock(block, /*toFront=*/isDetached); + mEvictionPolicy->releaseBlock(block, /*toFront = */ block->isDetached()); } } // Remove stored block ids in sequence @@ -3303,7 +3222,7 @@ void WindowBlockManager::schedulingReleaseBlocks(RequestIdType requestId) KVCacheManager::KVCacheManager(SizeType32 numLayers, SizeType32 numKvHeads, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, - SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, + SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkTokenLength, int64_t stream, runtime::SizeType32 maxSequenceLength, SizeType32 chunkSize, bool enableBlockReuse, CacheType cacheType, bool enablePartialReuse, bool copyOnPartialReuse, bool enableIndexerKCache, SizeType32 indexerKCacheQuantBlockSize, SizeType32 indexerKCacheIndexHeadDim, @@ -3320,7 +3239,7 @@ KVCacheManager::KVCacheManager(SizeType32 numLayers, SizeType32 numKvHeads, Size KVCacheManager::KVCacheManager(std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, - SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, + SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkTokenLength, int64_t stream, runtime::SizeType32 maxSequenceLength, SizeType32 chunkSize, bool enableBlockReuse, CacheType cacheType, std::optional<executor::RetentionPriority> secondaryOffloadMinPriority, std::shared_ptr<KVCacheEventManager> eventManager, bool enablePartialReuse, bool copyOnPartialReuse, @@ -3339,7 +3258,7 @@ KVCacheManager::KVCacheManager(std::vector<SizeType32> const& numKvHeadsPerLayer KVCacheManager::KVCacheManager(std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, - SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, + SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkTokenLength, CudaStreamPtr stream, runtime::SizeType32 maxSequenceLength, SizeType32 chunkSize, bool enableBlockReuse, CacheType cacheType, std::optional<executor::RetentionPriority> secondaryOffloadMinPriority, std::shared_ptr<KVCacheEventManager> eventManager, bool enablePartialReuse, bool copyOnPartialReuse, @@ -3381,7 +3300,7 @@ KVCacheManager::KVCacheManager(std::vector<SizeType32> const& numKvHeadsPerLayer KVCacheManager::KVCacheManager(SizeType32 numLayers, SizeType32 numKvHeads, SizeType32 sizePerHead, SizeType32 tokensPerBlock, BlocksPerWindow const& blocksPerWindow, SizeType32 maxNumSequences, - SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, tensorrt_llm::DataType dtype, + SizeType32 maxBeamWidth, std::vector<SizeType32> const& maxAttentionWindowVec, nvinfer1::DataType dtype, SizeType32 sinkTokenLength, CudaStreamPtr stream, runtime::SizeType32 maxSequenceLength, SizeType32 chunkSize, bool enableBlockReuse, CacheType cacheType, std::optional<executor::RetentionPriority> secondaryOffloadMinPriority, std::shared_ptr<KVCacheEventManager> eventManager, bool enablePartialReuse, bool copyOnPartialReuse, @@ -3414,7 +3333,7 @@ void KVCacheManager::allocatePools(bool useUvm) // a future per-window override map can mix precisions inside a single manager. auto const poolDataType = primaryPool->getDataType(); #ifdef ENABLE_FP4 - auto const isFp4 = poolDataType == tensorrt_llm::DataType::kFP4; + auto const isFp4 = poolDataType == nvinfer1::DataType::kFP4; #else auto const isFp4 = false; #endif @@ -3785,7 +3704,6 @@ bool KVCacheManager::copyLinearAttentionBlockBatch(std::vector<std::shared_ptr<L void WindowBlockManager::detachFrontBlock(GenerationRequest& sequence) { - std::lock_guard<std::recursive_mutex> lock(mLookupTree->getMutex()); // streamLLM is not supported at the moment. The out of window block will // always be the 0th block. TLLM_CHECK_WITH_INFO( @@ -4068,28 +3986,6 @@ tle::RetentionPriority KVCacheManager::getPriorityByBlockId(KVCacheBlock::IdType } } -std::vector<kernels::KVCacheIndex::UnderlyingType> KVCacheManager::getMemoryPoolBlockIndicesByBlockIds( - std::vector<KVCacheBlock::IdType> const& blockIds, SizeType32 windowSize) const -{ - std::vector<kernels::KVCacheIndex::UnderlyingType> indices; - indices.reserve(blockIds.size()); - for (auto const blockId : blockIds) - { - BlockPtr const& block = mBlockManager.getBlockById(blockId, windowSize); - TLLM_CHECK_WITH_INFO(block != nullptr, "Block not found (blockId=%d, windowSize=%d)", blockId, windowSize); - // Invariant, not a recoverable condition: blocks referenced here are held by a live - // request, allocation onboards offloaded blocks, and offload only selects free blocks. - // A secondary block therefore indicates a block-lifetime bug, and the returned index - // (pool flag stripped) would silently alias a primary slot. - TLLM_CHECK_WITH_INFO(block->isPrimary(), - "Block is not in the primary pool (blockId=%d, windowSize=%d); the returned index is only " - "valid for primary-pool blocks", - blockId, windowSize); - indices.push_back(block->getMemoryPoolBlockIndex()); - } - return indices; -} - SizeType32 KVCacheManager::copyBlockOffsets(ITensor& output, SizeType32 outputSlotOffset, RequestIdType requestId) const { auto const& sequence = getSequence(requestId); @@ -4358,7 +4254,7 @@ std::map<SizeType32, float> computeWindowSizeShares( } // namespace BlocksPerWindow BaseKVCacheManager::calculateMaxNumBlocks(executor::KvCacheConfig const& config, - tensorrt_llm::DataType dtype, std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, + nvinfer1::DataType dtype, std::vector<SizeType32> const& numKvHeadsPerLayer, SizeType32 sizePerHead, SizeType32 tokensPerBlock, WorldConfig const& worldConfig, std::map<SizeType32, std::vector<SizeType32>> const& windowSizeToLayers, uint64_t allottedPrimaryMemBytes, uint64_t allottedSecondaryMemBytes, size_t extraCostMemory, SizeType32 kvFactor, SizeType32 maxBatchSize, diff --git a/cpp/tensorrt_llm/batch_manager/kvCacheTransferManager.cpp b/cpp/tensorrt_llm/batch_manager/kvCacheTransferManager.cpp index 33ad5b4a1675..c28d1e476137 100644 --- a/cpp/tensorrt_llm/batch_manager/kvCacheTransferManager.cpp +++ b/cpp/tensorrt_llm/batch_manager/kvCacheTransferManager.cpp @@ -22,7 +22,6 @@ #include "tensorrt_llm/batch_manager/kvCacheEventManager.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/kernels/kvCachePartialCopy.h" #include "tensorrt_llm/runtime/bufferManager.h" @@ -164,8 +163,8 @@ void KVCacheTransferManager::copyBlock(BlockPtr const& src, BlockPtr const& dst, // If no partial tokens or if the dataType is not supported for partial copy, copy entire block. // Note that nvfp4 kv cache SFs use an interleaved layout, so we need to copy the entire block. - if (numTokensToCopy <= 0 || srcPtr->getDataType() == tensorrt_llm::DataType::kINT4 - || srcPtr->getDataType() == tensorrt_llm::DataType::kFP4 || containsBlockScales) + if (numTokensToCopy <= 0 || srcPtr->getDataType() == nvinfer1::DataType::kINT4 + || srcPtr->getDataType() == nvinfer1::DataType::kFP4 || containsBlockScales) { // For partial copy not implemented with these data types, // just do a full copy. @@ -462,8 +461,8 @@ std::size_t KVCacheTransferManager::computeBlockTransferBytes( // Mirror the logic in copyBlock: a partial copy only happens when numTokensToCopy > 0, // the data type supports it (not kINT4/kFP4), not block scales, and numTokensToCopy < tokensPerBlock. - bool const isPartialCopy = numTokensToCopy > 0 && dataType != tensorrt_llm::DataType::kINT4 - && dataType != tensorrt_llm::DataType::kFP4 && !pool.containsBlockScales + bool const isPartialCopy = numTokensToCopy > 0 && dataType != nvinfer1::DataType::kINT4 + && dataType != nvinfer1::DataType::kFP4 && !pool.containsBlockScales && numTokensToCopy < pool.tokensPerBlock; if (isPartialCopy) diff --git a/cpp/tensorrt_llm/batch_manager/llmRequest.cpp b/cpp/tensorrt_llm/batch_manager/llmRequest.cpp index d5466b4a7539..e51fad8ba149 100644 --- a/cpp/tensorrt_llm/batch_manager/llmRequest.cpp +++ b/cpp/tensorrt_llm/batch_manager/llmRequest.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -22,16 +22,6 @@ namespace tensorrt_llm::batch_manager { -// Single, process-global storage for the steady-clock offset. Keeping it in this -// translation unit (rather than as an inline-static member reachable from every -// shared object) guarantees that libtensorrt_llm.so and the nanobind extension -// module observe the same value once either side calibrates it. -std::optional<std::chrono::steady_clock::duration>& globalSteadyClockOffset() -{ - static std::optional<std::chrono::steady_clock::duration> offset{std::nullopt}; - return offset; -} - template <typename TTensor, typename TStream> runtime::SizeType32 GenericLlmRequest<TTensor, TStream>::getBeamWidthByIter(bool const forNextIteration) { diff --git a/cpp/tensorrt_llm/batch_manager/logitsPostProcessor.cpp b/cpp/tensorrt_llm/batch_manager/logitsPostProcessor.cpp new file mode 100644 index 000000000000..95b324f0f2ec --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/logitsPostProcessor.cpp @@ -0,0 +1,88 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/logitsPostProcessor.h" + +#include "tensorrt_llm/batch_manager/decoderBuffers.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/runtime/iTensor.h" + +namespace tr = tensorrt_llm::runtime; + +namespace tensorrt_llm::batch_manager +{ + +using TensorPtr = runtime::ITensor::SharedPtr; +using ITensor = runtime::ITensor; +using SizeType32 = tensorrt_llm::runtime::SizeType32; + +bool LogitsPostProcessor::operator()(DecoderInputBuffers& inputBuffers, bool replicateLogitsPostProcessor, + tr::WorldConfig const& worldConfig, CudaStreamPtr const& stream, + std::optional<LogitsPostProcessorBatched> const& logitsPostProcessorBatched) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(LogitsPostProcessor); + + // Arguments for batched processor + std::vector<LlmRequest::RequestIdType> reqIdsVec; + std::vector<LlmRequest::TensorPtr> logitsVec; + std::vector<std::reference_wrapper<LlmRequest::BeamTokens const>> beamTokensVec; + std::vector<std::optional<LlmRequest::RequestIdType>> clientIdsVec; + + bool logitsPostProcessorIsApplied = false; + for (size_t batchIdx = 0; batchIdx < inputBuffers.decoderRequests.size(); ++batchIdx) + { + auto const& llmReq = inputBuffers.decoderRequests.at(batchIdx); + auto& logits = inputBuffers.decoderLogits.at(batchIdx); + + // Invoke non-batched processor or collect arguments for batched processor + if (llmReq->mLogitsPostProcessor) + { + logitsPostProcessorIsApplied = true; + if (replicateLogitsPostProcessor || worldConfig.isFirstTensorParallelRank()) + { + (*llmReq->mLogitsPostProcessor)( + llmReq->mRequestId, logits, llmReq->getTokens(), stream, llmReq->mClientId); + } + } + else if (llmReq->mApplyLogitsPostProcessorBatched) + { + reqIdsVec.push_back(llmReq->mRequestId); + logitsVec.push_back(logits); + beamTokensVec.emplace_back(llmReq->getTokens()); + clientIdsVec.push_back(llmReq->mClientId); + } + } + + // Invoke batched processor + if (!reqIdsVec.empty()) + { + logitsPostProcessorIsApplied = true; + if (replicateLogitsPostProcessor || worldConfig.isFirstTensorParallelRank()) + { + (*logitsPostProcessorBatched)(reqIdsVec, logitsVec, beamTokensVec, stream, clientIdsVec); + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + + return logitsPostProcessorIsApplied; +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/loraBuffers.cpp b/cpp/tensorrt_llm/batch_manager/loraBuffers.cpp new file mode 100644 index 000000000000..b67b72f6c49a --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/loraBuffers.cpp @@ -0,0 +1,109 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "loraBuffers.h" + +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/runtime/loraUtils.h" + +namespace tensorrt_llm::batch_manager +{ + +LoraBuffers::LoraBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, runtime::TllmRuntime const& tllmRuntime, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig) +{ + auto const localNbLayers + = modelConfig.getNbAttentionLayers(worldConfig.getPipelineParallelism(), worldConfig.getPipelineParallelRank()); + auto const firstLayerId = worldConfig.getPipelineParallelRank() * localNbLayers; + + auto nbModelConfigs = static_cast<SizeType32>(modelConfig.getLoraModules().size()); + + // there are 3 pointers: LoRA A, LoRA B, and a DoRA magnitude (null if not DoRA) + auto loraWeightsPtrsShape + = runtime::ITensor::makeShape({nbModelConfigs, localNbLayers, maxBatchSize * maxBeamWidth, 3}); + auto loraAdapterSizesShape + = runtime::ITensor::makeShape({nbModelConfigs, localNbLayers, maxBatchSize * maxBeamWidth}); + + auto firstModuleName = std::string(modelConfig.getLoraModules().front().name()); + auto ptrsFieldName = firstModuleName + "_lora_weights_pointers_" + std::to_string(firstLayerId); + auto rankFieldName = firstModuleName + "_lora_ranks_" + std::to_string(firstLayerId); + auto weightsPtrDtype = tllmRuntime.getEngine().getTensorDataType(ptrsFieldName.c_str()); + auto ranksDtype = tllmRuntime.getEngine().getTensorDataType(rankFieldName.c_str()); + + mLoraManager.create(modelConfig); + + mLoraWeightsPointersHost = runtime::BufferManager::pinned(loraWeightsPtrsShape, weightsPtrDtype); + mLoraAdapterSizesHost = runtime::BufferManager::pinned(loraAdapterSizesShape, ranksDtype); +} + +void LoraBuffers::fill(RequestVector const& contextRequests, RequestVector const& genRequests, + PeftTable const& peftTable, runtime::BufferManager const& manager, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig) +{ + manager.setZero(*mLoraWeightsPointersHost); + manager.setZero(*mLoraAdapterSizesHost); + + SizeType32 batchIdx{0}; + for (auto const& requests : {contextRequests, genRequests}) + { + for (auto const& llmReq : requests) + { + auto const optReqLoraWeights = llmReq->getLoraWeights(); + auto const optReqLoraConfig = llmReq->getLoraConfig(); + + auto const isContextRequest = llmReq->isContextInitState(); + auto const beamWidth = isContextRequest ? 1 : llmReq->mSamplingConfig.beamWidth; + auto const peftIt = peftTable.find(llmReq->mRequestId); + if (peftIt != peftTable.end()) + { + auto const& peftValues = peftIt->second; + if (!peftValues.empty()) + { + mLoraManager.fillInputTensors(mLoraWeightsPointersHost, mLoraAdapterSizesHost, peftIt->second, + batchIdx, beamWidth, modelConfig, worldConfig); + } + } + ++batchIdx; + } + } +} + +void LoraBuffers::validate(std::optional<std::uint64_t> const& optTaskId, + std::optional<TensorPtr> const& optReqLoraWeights, std::optional<TensorPtr> const& optReqLoraConfig, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig) +{ + runtime::lora::loraValidateRequestTensors(optTaskId, optReqLoraWeights, optReqLoraConfig, modelConfig, worldConfig); +} + +void LoraBuffers::insertInputTensors(TensorMap& inputTensors, TensorPtr weightsPtrs, TensorPtr adapterSizes, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig) const +{ + mLoraManager.insertInputTensors(inputTensors, weightsPtrs, adapterSizes, modelConfig, worldConfig); +} + +void LoraBuffers::reshape(SizeType32 numSequences) +{ + auto weightsPtrsShape = mLoraWeightsPointersHost->getShape(); + weightsPtrsShape.d[2] = numSequences; + mLoraWeightsPointersHost->reshape(weightsPtrsShape); + + auto adapterSizesShape = mLoraAdapterSizesHost->getShape(); + adapterSizesShape.d[2] = numSequences; + mLoraAdapterSizesHost->reshape(adapterSizesShape); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/loraBuffers.h b/cpp/tensorrt_llm/batch_manager/loraBuffers.h new file mode 100644 index 000000000000..3ba68995518f --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/loraBuffers.h @@ -0,0 +1,61 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/loraManager.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +namespace tensorrt_llm::batch_manager +{ + +class LoraBuffers +{ +public: + using SizeType32 = tensorrt_llm::runtime::SizeType32; + using PeftTable = runtime::LoraManager::PeftTable; + using TensorPtr = runtime::ITensor::SharedPtr; + using TensorMap = runtime::StringPtrMap<runtime::ITensor>; + + TensorPtr mLoraWeightsPointersHost; + TensorPtr mLoraAdapterSizesHost; + + runtime::LoraManager mLoraManager; + + LoraBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, runtime::TllmRuntime const& tllmRuntime, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig); + + static void validate(std::optional<std::uint64_t> const& optTaskId, + std::optional<TensorPtr> const& optReqLoraWeights, std::optional<TensorPtr> const& optReqLoraConfig, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig); + + void fill(RequestVector const& contextRequests, RequestVector const& genRequests, PeftTable const& peftTable, + runtime::BufferManager const& manager, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig); + + void insertInputTensors(TensorMap& inputTensors, TensorPtr weightsPtrs, TensorPtr adapterSizes, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig) const; + + void reshape(SizeType32 numSequences); +}; +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.cpp b/cpp/tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.cpp new file mode 100644 index 000000000000..3e494a6383ec --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.cpp @@ -0,0 +1,198 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.h" +#include "tensorrt_llm/batch_manager/decoderBuffers.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/runtime/decoderState.h" +#include "tensorrt_llm/runtime/iGptDecoderBatched.h" + +namespace tr = tensorrt_llm::runtime; + +namespace tensorrt_llm::batch_manager +{ +using SizeType32 = MakeDecodingBatchInputOutput::SizeType32; +using TensorPtr = MakeDecodingBatchInputOutput::TensorPtr; + +void MakeDecodingBatchInputOutput::createDecoderBatchInputs(DecoderInputBuffers& inputBuffers, + std::vector<SizeType32> const& activeSlots, runtime::decoder::DecoderState const& decoderState) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const& numDecodingEngineTokens = decoderState.getNumDecodingEngineTokens(); + auto const& maxDecodingEngineTokens = decoderState.getMaxDecodingEngineTokens(); + auto const& maxDecodingDecoderTokens = decoderState.getMaxDecodingDecoderTokens(); + auto const maxDecoderSteps = common::ceilDiv(maxDecodingEngineTokens, maxDecodingDecoderTokens); + + auto& batchSlots = inputBuffers.forwardBatchSlots; + auto& decoderLogits = inputBuffers.decoderLogits; + + for (SizeType32 step = 0; step < maxDecoderSteps; ++step) + { + batchSlots.at(step)->resize(activeSlots.size()); + } + + auto constexpr singleRequest = 1; + + std::vector<SizeType32> batchSizes(maxDecoderSteps); + std::vector<std::vector<tr::ITensor::SharedConstPtr>> batchLogits(maxDecoderSteps); + auto maxActiveDecoderSteps = 1; + for (size_t batchIdx = 0; batchIdx < activeSlots.size(); ++batchIdx) + { + auto const slot = activeSlots.at(batchIdx); + auto const& logits = decoderLogits.at(batchIdx); + + auto const numDecoderSteps = common::ceilDiv(numDecodingEngineTokens.at(slot), maxDecodingDecoderTokens); + maxActiveDecoderSteps = std::max(maxActiveDecoderSteps, numDecoderSteps); + for (SizeType32 step = 0; step < numDecoderSteps; ++step) + { + auto batchSlotsRange = tr::BufferRange<SizeType32>(*batchSlots.at(step)); + batchSlotsRange[batchSizes[step]] = slot; + batchSizes[step]++; + auto logitsSlice = tr::ITensor::slice(logits, step, singleRequest); + batchLogits[step].emplace_back(std::move(logitsSlice)); + } + } + + for (SizeType32 step = 0; step < maxDecoderSteps; ++step) + { + batchSlots.at(step)->resize(batchSizes[step]); + } + batchLogits.resize(maxActiveDecoderSteps); + + inputBuffers.maxDecoderSteps = maxActiveDecoderSteps; + inputBuffers.batchLogits = batchLogits; + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +namespace +{ + +std::pair<std::vector<SizeType32>, std::vector<SizeType32>> getActiveSlots(RequestVector const& decoderRequests) +{ + std::vector<SizeType32> activeSlots; + std::vector<SizeType32> generationSteps; + for (auto const& llmReq : decoderRequests) + { + activeSlots.push_back(llmReq->mSeqSlot.value()); + generationSteps.push_back(llmReq->getDecodingIter()); + } + + return {activeSlots, generationSteps}; +} + +//! @brief Sets inputs for explicit draft tokens. +void setExplicitDraftTokensInputs(tr::DecodingInput& dInput, RuntimeBuffers const& fusedRuntimeBuffers) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK(fusedRuntimeBuffers.mExplicitDraftTokensBuffers); + auto const& explicitDraftTokensInputs = fusedRuntimeBuffers.mExplicitDraftTokensBuffers->engineOutputs; + auto const& explicitDraftTokensLastInputs = fusedRuntimeBuffers.mExplicitDraftTokensBuffers->engineInputs; + + dInput.explicitDraftTokensInputs = tr::DecodingInput::ExplicitDraftTokensInputs(); + dInput.explicitDraftTokensInputs->nextDraftTokens = explicitDraftTokensInputs.nextDraftTokens; + dInput.explicitDraftTokensInputs->nextFlatTokens = explicitDraftTokensInputs.nextFlatTokens; + dInput.explicitDraftTokensInputs->nextDraftIndices = explicitDraftTokensInputs.nextDraftIndices; + dInput.explicitDraftTokensInputs->nextDraftProbs = explicitDraftTokensInputs.nextDraftProbs; + dInput.explicitDraftTokensInputs->lastDraftTokens = explicitDraftTokensLastInputs.draftTokens; + dInput.explicitDraftTokensInputs->lastDraftIndices = explicitDraftTokensLastInputs.draftIndices; + dInput.explicitDraftTokensInputs->lastPositionIdsBase = explicitDraftTokensLastInputs.positionIdsBase; + dInput.explicitDraftTokensInputs->masks = explicitDraftTokensInputs.masks; + dInput.explicitDraftTokensInputs->packedPositionIds = explicitDraftTokensInputs.packedPositionIds; + dInput.explicitDraftTokensInputs->bestPathLengths = explicitDraftTokensInputs.bestPathLengths; + dInput.explicitDraftTokensInputs->bestPathIndices = explicitDraftTokensInputs.bestPathIndices; + dInput.explicitDraftTokensInputs->nextGenerationLengths = explicitDraftTokensInputs.nextGenerationLengths; + dInput.explicitDraftTokensInputs->lastGenerationLengths = explicitDraftTokensLastInputs.generationLengths; + dInput.explicitDraftTokensInputs->maxGenLengthDevice = explicitDraftTokensInputs.maxGenToken; + // Slots in request order + dInput.explicitDraftTokensInputs->seqSlots = fusedRuntimeBuffers.seqSlots; + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +//! @brief Sets inputs for eagle decoding. +void setEagleInputs(tr::DecodingInput& dInput, RuntimeBuffers const& fusedRuntimeBuffers) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK(fusedRuntimeBuffers.mEagleBuffers); + auto const& eagleInputs = fusedRuntimeBuffers.mEagleBuffers->engineOutputs; + auto const& eagleLastInputs = fusedRuntimeBuffers.mEagleBuffers->engineInputs; + + dInput.eagleInputs = tr::DecodingInput::EagleInputs(); + dInput.eagleInputs->nextDraftTokens = eagleInputs.nextDraftTokens; + dInput.eagleInputs->nextDraftLens = eagleInputs.nextDraftLens; + dInput.eagleInputs->nextDraftPaths = eagleInputs.nextDraftPaths; + dInput.eagleInputs->lastDraftTokens = eagleLastInputs.draftTokens; + dInput.eagleInputs->lastDraftLens = eagleLastInputs.draftLens; + dInput.eagleInputs->lastDraftPaths = eagleLastInputs.draftPaths; + dInput.eagleInputs->acceptedTokens = eagleInputs.acceptedTokens; + dInput.eagleInputs->acceptedLens = eagleInputs.acceptedLens; + dInput.eagleInputs->acceptedPathIds = eagleInputs.acceptedPaths; + dInput.eagleInputs->chunkedContextNextTokens = eagleInputs.chunkedContextNextTokens; + // Slots in request order + dInput.eagleInputs->seqSlots = fusedRuntimeBuffers.seqSlots; + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +} // namespace + +void MakeDecodingBatchInputOutput::operator()(DecoderInputBuffers& inputBuffers, + runtime::decoder::DecoderState& decoderState, runtime::ModelConfig const& modelConfig, + OptionalRef<RuntimeBuffers> fusedRuntimeBuffers) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto [activeSlots, generationSteps] = getActiveSlots(inputBuffers.decoderRequests); + + createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); + + auto const maxBeamWidth = decoderState.getMaxBeamWidth(); + if (maxBeamWidth > 1) + { + // For Variable-Beam-Width-Search + decoderState.getJointDecodingInput().generationSteps = generationSteps; + } + + if (modelConfig.getSpeculativeDecodingMode().hasDraftLogits()) + { + decoderState.getJointDecodingInput().medusaInputs->medusaLogits = inputBuffers.predictedDraftLogits; + } + + if (modelConfig.getSpeculativeDecodingMode().isExplicitDraftTokens()) + { + TLLM_CHECK(fusedRuntimeBuffers); + // requires mCtxGenFusion == true + setExplicitDraftTokensInputs(decoderState.getJointDecodingInput(), *fusedRuntimeBuffers); + } + else if (modelConfig.getSpeculativeDecodingMode().isEagle()) + { + TLLM_CHECK(fusedRuntimeBuffers); + // requires mCtxGenFusion == true + setEagleInputs(decoderState.getJointDecodingInput(), *fusedRuntimeBuffers); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/medusaBuffers.cpp b/cpp/tensorrt_llm/batch_manager/medusaBuffers.cpp index 32935e683b83..eb40208739cf 100644 --- a/cpp/tensorrt_llm/batch_manager/medusaBuffers.cpp +++ b/cpp/tensorrt_llm/batch_manager/medusaBuffers.cpp @@ -17,10 +17,99 @@ #include "tensorrt_llm/batch_manager/medusaBuffers.h" #include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/medusaModule.h" +#include "tensorrt_llm/runtime/utils/speculativeChoicesUtils.h" namespace tensorrt_llm::batch_manager { +MedusaBuffers::MedusaBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, runtime::BufferManager const& manager, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + executor::DecodingConfig const& decodingConfig, runtime::TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + TLLM_CHECK_WITH_INFO(maxBeamWidth == 1, "Medusa does not support beam search"); + + auto const& engine = runtime.getEngine(); + + auto const maxNumSequences = maxBatchSize; + + auto const medusaModule = std::dynamic_pointer_cast<tensorrt_llm::runtime::MedusaModule const>( + modelConfig.getSpeculativeDecodingModulePtr()); + + auto const medusaHeads = medusaModule->getMaxDraftPathLen(); + auto const maxPathLen = medusaModule->getMaxPathLen(); // medusaHeads + 1 + auto const maxMedusaTokens = medusaModule->getMaxDecodingDraftTokens(); + auto const maxDecodingTokens = medusaModule->getMaxDecodingTokens(); // maxMedusaTokens + 1 + auto const numPackedMasks = medusaModule->getNumPackedMasks(); + + auto const vocabSizePadded = modelConfig.getVocabSizePadded(worldConfig.getSize()); + + if (worldConfig.isLastPipelineParallelRank()) + { + auto logitsType = engine.getTensorDataType("medusa_logits"); + medusaLogitsDevice = manager.gpu( + ITensor::makeShape({medusaHeads, maxBatchSize, maxDecodingTokens, vocabSizePadded}), logitsType); + } + + // Note: reserved for variable sequence length support. + medusaGenerationLengthsHost + = runtime::BufferManager::pinned(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); + // TODO: pack batch and tokensPerStep into one dim to support variable sequence length without padddings. + attentionPackedMaskHost = runtime::BufferManager::pinned( + ITensor::makeShape({maxNumSequences, maxDecodingTokens, numPackedMasks}), nvinfer1::DataType::kINT32); + medusaPositionOffsetsHost = runtime::BufferManager::pinned( + ITensor::makeShape({maxNumSequences, maxDecodingTokens}), nvinfer1::DataType::kINT32); + medusaTreeIdsHost = runtime::BufferManager::pinned( + ITensor::makeShape({maxNumSequences, maxMedusaTokens}), nvinfer1::DataType::kINT32); + medusaPathsHost = runtime::BufferManager::pinned( + ITensor::makeShape({maxNumSequences, maxDecodingTokens, maxPathLen}), nvinfer1::DataType::kINT32); + + TensorPtr medusaPositionOffsetsHostSlice = ITensor::slice(medusaPositionOffsetsHost, 0, 1); + medusaPositionOffsetsHostSlice->squeeze(0); + TensorPtr medusaTreeIdsHostSlice = ITensor::slice(medusaTreeIdsHost, 0, 1); + medusaTreeIdsHostSlice->squeeze(0); + TensorPtr medusaPathsHostSlice = ITensor::slice(medusaPathsHost, 0, 1); + medusaPathsHostSlice->squeeze(0); + TensorPtr attentionPackedMaskHostSlice = ITensor::slice(attentionPackedMaskHost, 0, 1); + attentionPackedMaskHostSlice->squeeze(0); + + // Init buffers for 1 request + auto const& choices = decodingConfig.getMedusaChoices().value_or(medusaModule->getMedusaChoices()); + runtime::utils::initTensorsFromChoices(*medusaModule, choices, mTopKs, medusaGenerationLengthsHost, + medusaPositionOffsetsHostSlice, medusaTreeIdsHostSlice, medusaPathsHostSlice, attentionPackedMaskHostSlice); + + auto scatterToBatch = [maxBatchSize, &manager](TensorPtr& data) + { + auto srcSlice = ITensor::slice(data, 0, 1); + // Populate data from the 1st request to the other requests in the batch + for (SizeType32 bi = 1; bi < maxBatchSize; ++bi) + { + auto dstSlice = ITensor::slice(data, bi, 1); + manager.copy(*srcSlice, *dstSlice); + } + }; + + scatterToBatch(medusaPositionOffsetsHost); + scatterToBatch(medusaTreeIdsHost); + scatterToBatch(medusaPathsHost); + scatterToBatch(attentionPackedMaskHost); + + // Copy buffers to device + // 1st dimension of packed mask is num_total_generation_tokens now (packed without paddings). + attentionPackedMaskHost->reshape(ITensor::makeShape({maxNumSequences * maxDecodingTokens, numPackedMasks})); + attentionPackedMaskDevice = manager.copyFrom(*attentionPackedMaskHost, runtime::MemoryType::kGPU); + medusaGenerationLengthsDevice = manager.copyFrom(*medusaGenerationLengthsHost, runtime::MemoryType::kGPU); + medusaPositionOffsetsDevice = manager.copyFrom(*medusaPositionOffsetsHost, runtime::MemoryType::kGPU); + medusaTreeIdsDevice = manager.copyFrom(*medusaTreeIdsHost, runtime::MemoryType::kGPU); + medusaPathsDevice = manager.copyFrom(*medusaPathsHost, runtime::MemoryType::kGPU); + + // use speculative decoding buffer + medusaUseSpecDecoding = manager.cpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + runtime::bufferCast<SizeType32>(*medusaUseSpecDecoding)[0] = 1; + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + void MedusaBuffers::reshape(SizeType32 /* numCtxSequences */, SizeType32 numGenSequences, SizeType32 tokensPerStep) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); diff --git a/cpp/tensorrt_llm/batch_manager/microBatchScheduler.cpp b/cpp/tensorrt_llm/batch_manager/microBatchScheduler.cpp index 2807f33463a1..3e8fca0be052 100644 --- a/cpp/tensorrt_llm/batch_manager/microBatchScheduler.cpp +++ b/cpp/tensorrt_llm/batch_manager/microBatchScheduler.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -230,9 +230,8 @@ void MicroBatchScheduler::setCtxRequestsChunkSize<MicroBatchScheduler::ContextCh // Assigns chunk sizes to context requests under the kFORCE_CHUNK policy. // -// Requests with expected snapshot points advance to their next absolute snapshot position. -// Otherwise, every request consumes its full remaining context. -// Capacity and context-length truncation are rounded down to a chunk-unit boundary. +// Every request is assigned exactly min(contextRemainingLength, chunkUnitSize) tokens. +// Requests whose chunk would push the running total past ctxTokensCapacity are zeroed. // // This policy is designed for linear attention state caching, so reusable KV-cache tokens are NOT // calculated because it's not supported yet. @@ -249,36 +248,16 @@ void MicroBatchScheduler::setCtxRequestsChunkSize<MicroBatchScheduler::ContextCh SizeType32 totalTokens{0}; for (auto& llmReq : contextsToBeChunked) { - SizeType32 chunkSize = llmReq->getContextRemainingLength(); - auto const& expectedSnapshotPoints = llmReq->getExpectedSnapshotPoints(); - if (!expectedSnapshotPoints.empty()) - { - auto const currentPosition = llmReq->getContextCurrentPosition(); - std::optional<SizeType32> nextSnapshotPoint; - for (auto const point : expectedSnapshotPoints) - { - if (point > currentPosition && (!nextSnapshotPoint || point < nextSnapshotPoint.value())) - { - nextSnapshotPoint = point; - } - } - chunkSize = nextSnapshotPoint - ? std::max<SizeType32>(0, std::min(nextSnapshotPoint.value(), llmReq->getPromptLen()) - currentPosition) - : llmReq->getContextRemainingLength(); - } - - if (maxContextLength && chunkSize > maxContextLength.value()) + SizeType32 const chunkSize = std::min(llmReq->getContextRemainingLength(), chunkUnitSize); + if (ctxTokensCapacity && totalTokens + chunkSize > ctxTokensCapacity.value()) { - chunkSize = maxContextLength.value() / chunkUnitSize * chunkUnitSize; + llmReq->setContextChunkSize(0); } - if (ctxTokensCapacity && totalTokens + chunkSize > ctxTokensCapacity.value()) + else { - auto const remainingCapacity = std::max<SizeType32>(0, ctxTokensCapacity.value() - totalTokens); - chunkSize = std::min(chunkSize, remainingCapacity) / chunkUnitSize * chunkUnitSize; + llmReq->setContextChunkSize(chunkSize); + totalTokens += llmReq->getContextChunkSize(); } - - llmReq->setContextChunkSize(chunkSize); - totalTokens += llmReq->getContextChunkSize(); } } @@ -288,9 +267,8 @@ void MicroBatchScheduler::setCtxRequestsChunkSize<MicroBatchScheduler::ContextCh // kEQUAL_PROGRESS — all requests advance together one chunkUnitSize at a time. // kFIRST_COME_FIRST_SERVED — requests are served greedily in order until the budget // is exhausted. -// kFORCE_CHUNK — requests advance to the next expected snapshot point, or consume -// the remaining context when none are configured; budget is charged -// at face value (no reuse discount). +// kFORCE_CHUNK — every request gets exactly min(remaining, chunkUnitSize) +// tokens; budget is charged at face value (no reuse discount). // // EQUAL_PROGRESS and FIRST_COME_FIRST_SERVED are compute-aware: tokens covered by the // reusable KV-cache prefix are not charged against ctxTokensCapacity. @@ -458,7 +436,7 @@ std::tuple<RequestVector, RequestVector> MicroBatchScheduler::operator()(Request allContextRequestsFit = false; } - // FORCE_CHUNK must always run boundary selection even when all contexts fit. + // For FORCE_CHUNK policy, always re-chunk regardless of whether all contexts fit. if (mCtxChunkConfig && mCtxChunkConfig.value().chunkingPolicy == ContextChunkingPolicy::kFORCE_CHUNK) { allContextRequestsFit = false; diff --git a/cpp/tensorrt_llm/batch_manager/mlaCacheFormatter.cpp b/cpp/tensorrt_llm/batch_manager/mlaCacheFormatter.cpp index 2631df9aa9a2..ad55455ad3c2 100644 --- a/cpp/tensorrt_llm/batch_manager/mlaCacheFormatter.cpp +++ b/cpp/tensorrt_llm/batch_manager/mlaCacheFormatter.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -124,13 +124,9 @@ void MLACacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& ses { NVTX3_SCOPED_RANGE(MLACacheFormatter_format); session.setTime(TransferSession::kTimeFormatter); - auto llmRequest = session.getLlmRequest(); - if (llmRequest.has_value()) - { - TLLM_LOG_DEBUG( - mpi::MpiComm::world().getRank(), "Start sending KV cache for request ID: %ld.", (*llmRequest)->mRequestId); - TLLM_CHECK_WITH_INFO((*llmRequest)->mSamplingConfig.beamWidth == 1, "Currently only supports beam width 1."); - } + auto const& llmRequest = session.getLlmRequest(); + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "Start sending KV cache for request ID: %ld.", llmRequest.mRequestId); auto const& selfConfig = session.getSelfState().getCacheState().value(); auto const& destConfig = session.getOtherState().getCacheState().value(); auto const selfIdx = session.getSelfState().getCommState().value().getSelfIdx(); @@ -138,6 +134,7 @@ void MLACacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& ses auto const& lastBlockKey = session.getLastBlockKey(); auto const& connections = session.getConnections(); auto& bufferManager = session.getBufferManager(); + TLLM_CHECK_WITH_INFO(llmRequest.mSamplingConfig.beamWidth == 1, "Currently only supports beam width 1."); TLLM_CHECK(!connections.empty()); if (!needSendCache(selfConfig, destConfig, selfIdx)) { @@ -148,6 +145,7 @@ void MLACacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& ses auto targetNum = pickUpConnections.size(); if (targetNum == 0) { + TLLM_LOG_DEBUG("No targets to send KV cache to for request ID: %ld", llmRequest.mRequestId); return; } @@ -221,11 +219,8 @@ void MLACacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& ses } } - if (llmRequest.has_value()) - { - TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "End the sending of KV cache for the request ID: %ld.", - (*llmRequest)->mRequestId); - } + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "End the sending of KV cache for the request ID: %ld.", + llmRequest.mRequestId); return; } @@ -258,9 +253,7 @@ void MLACacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& ses return bufferSizeForTarget; }; auto bufferEleSizes = getBufferSizeForTarget(); - auto const* sendCancelFlag - = common::getEnvDisaggEnableInflightCancel() ? &session.getDataContext().getTransferTerminate() : nullptr; - auto cacheBufferId = mCacheTransBufferManagers[transferIndexerKCache]->assignBufferIndexForSend(sendCancelFlag); + auto cacheBufferId = mCacheTransBufferManagers[transferIndexerKCache]->assignBufferIndexForSend(); BufferIndexHolder sendHolder( *mCacheTransBufferManagers[transferIndexerKCache], cacheBufferId, /*isRecv=*/false); auto result = mCacheTransBufferManagers[transferIndexerKCache]->getOrAllocateSendBuffers( @@ -353,84 +346,63 @@ void MLACacheFormatter::format(tensorrt_llm::batch_manager::TransferSession& ses session.appendMeasure(startTime, endTime, outputSplitCaches.at(cacheIdx)->getSizeInBytes()); }; - if (sendCancelFlag != nullptr && sendCancelFlag->load(std::memory_order_relaxed)) + if (pickUpConnections.size() > 1) { - TLLM_THROW("MLA cache transfer cancelled before NIXL submission"); - } - - try - { - if (pickUpConnections.size() > 1) + if (!common::getEnvEnableReceiveKVCacheParallel()) { - if (!common::getEnvEnableReceiveKVCacheParallel()) + TLLM_LOG_DEBUG("Disable parallel receiving of the KV cache."); + for (size_t i = 0; i < pickUpConnections.size(); i++) { - TLLM_LOG_DEBUG("Disable parallel receiving of the KV cache."); - for (size_t i = 0; i < pickUpConnections.size(); i++) - { - sendBufferFun(deviceId, pickUpConnections[i]); - } + sendBufferFun(deviceId, pickUpConnections[i]); } - else - { - // concurrency num - auto concurrencyNum - = std::min(std::max(static_cast<size_t>(1), bufferCoverTargetNum), pPDomainSize * cPDomainSize); + } + else + { + // concurrency num + auto concurrencyNum + = std::min(std::max(static_cast<size_t>(1), bufferCoverTargetNum), pPDomainSize * cPDomainSize); - auto remainSendNum = pickUpConnections.size(); + auto remainSendNum = pickUpConnections.size(); - while (remainSendNum > 0) + while (remainSendNum > 0) + { + auto sendConcurrencyNum = std::min(remainSendNum, concurrencyNum); + std::vector<std::future<void>> futures; + futures.reserve(sendConcurrencyNum); + for (size_t i = 0; i < sendConcurrencyNum; i++) { - auto sendConcurrencyNum = std::min(remainSendNum, concurrencyNum); - std::vector<std::future<void>> futures; - futures.reserve(sendConcurrencyNum); - for (size_t i = 0; i < sendConcurrencyNum; i++) - { - size_t idx = i + (pickUpConnections.size() - remainSendNum); - size_t connIdx = pickUpConnections[idx]; - TLLM_CHECK(idx < pickUpConnections.size()); - TLLM_CHECK(connIdx < session.getConnections().size()); - futures.push_back(std::async(std::launch::async, sendBufferFun, deviceId, connIdx)); - } - for (auto& future : futures) - { - future.get(); - } - remainSendNum -= sendConcurrencyNum; + size_t idx = i + (pickUpConnections.size() - remainSendNum); + size_t connIdx = pickUpConnections[idx]; + TLLM_CHECK(idx < pickUpConnections.size()); + TLLM_CHECK(connIdx < session.getConnections().size()); + futures.push_back(std::async(std::launch::async, sendBufferFun, deviceId, connIdx)); + } + for (auto& future : futures) + { + future.get(); } + remainSendNum -= sendConcurrencyNum; } } - else - { - sendBufferFun(deviceId, pickUpConnections[0]); - } } - catch (...) + else { - if (agentConnection != nullptr && common::getEnvDisaggEnableInflightCancel()) - { - sendHolder.poison(); - } - throw; + sendBufferFun(deviceId, pickUpConnections[0]); } sendHolder.release(); } session.setTime(TransferSession::kTimeTransmissions); session.setTime(TransferSession::kTimePostprocess); - if (llmRequest.has_value()) - { - TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "End the sending of KV cache for the request ID: %ld.", - (*llmRequest)->mRequestId); - } + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "End the sending of KV cache for the request ID: %ld.", llmRequest.mRequestId); } void MLACacheFormatter::unformat(tensorrt_llm::batch_manager::TransferSession& session) { NVTX3_SCOPED_RANGE(MLACacheFormatter_unformat); session.setTime(TransferSession::kTimeFormatter); - auto llmRequestOpt = session.getLlmRequest(); - TLLM_CHECK_WITH_INFO(llmRequestOpt.has_value(), "LlmRequest required for receiving KV cache"); - auto const& llmRequest = **llmRequestOpt; + auto const& llmRequest = session.getLlmRequest(); TLLM_CHECK_WITH_INFO(llmRequest.mSamplingConfig.beamWidth == 1, "Currently only supports beam width 1."); auto const ctxReqId = llmRequest.getContextPhaseParams().value().getReqId(); TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), @@ -535,18 +507,13 @@ void MLACacheFormatter::unformat(tensorrt_llm::batch_manager::TransferSession& s if (preAssignedId.has_value()) { cacheBufferId = static_cast<int>(*preAssignedId); - if (!session.hasReservedRecvBuffer(*mCacheTransBufferManagers[transferIndexerKCache])) - { - recvHolder = BufferIndexHolder( - *mCacheTransBufferManagers[transferIndexerKCache], cacheBufferId, /*isRecv=*/true); - } } else { cacheBufferId = mCacheTransBufferManagers[transferIndexerKCache]->assignBufferIndexForRecv(); - recvHolder = BufferIndexHolder( - *mCacheTransBufferManagers[transferIndexerKCache], cacheBufferId, /*isRecv=*/true); } + recvHolder + = BufferIndexHolder(*mCacheTransBufferManagers[transferIndexerKCache], cacheBufferId, /*isRecv=*/true); auto targetNum = pickUpConnections.size(); @@ -694,7 +661,6 @@ void MLACacheFormatter::unformat(tensorrt_llm::batch_manager::TransferSession& s bufferManager.getStream().synchronize(); } - (void) session.releaseReservedRecvBuffer(*mCacheTransBufferManagers[transferIndexerKCache]); recvHolder.release(); } session.setTime(TransferSession::kTimePostprocess); diff --git a/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp b/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp index 89cc475b82ee..0bf9a989fd65 100644 --- a/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp +++ b/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp @@ -30,7 +30,7 @@ #include "tensorrt_llm/runtime/workerPool.h" #include "tensorrt_llm/runtime/worldConfig.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cstdint> #include <limits> @@ -52,8 +52,7 @@ PeftTaskNotCachedException::PeftTaskNotCachedException(std::string const& msg) PeftTaskNotCachedException::~PeftTaskNotCachedException() noexcept = default; std::pair<uint64_t, uint64_t> PeftCacheManager::getMaxNumSlots(PeftCacheManagerConfig const& config, - tensorrt_llm::DataType dataType, uint64_t pageWidth, uint64_t max1dModSize, - runtime::BufferManager const& bufferManager) + nvinfer1::DataType dataType, uint64_t pageWidth, uint64_t max1dModSize, runtime::BufferManager const& bufferManager) { TLLM_LOG_DEBUG("max1dModeSize=%llu", max1dModSize); TLLM_LOG_DEBUG("pageWidth=%llu", pageWidth); diff --git a/cpp/tensorrt_llm/batch_manager/promptTuningBuffers.cpp b/cpp/tensorrt_llm/batch_manager/promptTuningBuffers.cpp new file mode 100644 index 000000000000..1cf73a2c0d21 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/promptTuningBuffers.cpp @@ -0,0 +1,323 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/promptTuningBuffers.h" + +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/common/nvtxUtils.h" + +namespace tensorrt_llm::batch_manager +{ +using SizeType32 = tensorrt_llm::runtime::SizeType32; +using TensorPtr = runtime::ITensor::SharedPtr; + +PromptTuningBuffers::PromptTuningBuffers(SizeType32 maxBatchSize, runtime::BufferManager const& manager, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig) +{ + auto maxPromptEmbeddingTableSize = modelConfig.getMaxPromptEmbeddingTableSize(); + auto const hiddenSize = modelConfig.getHiddenSize() * worldConfig.getTensorParallelism(); + + // vocabSize and mMaxPromptVocabSize + mPromptTuningParams.vocabSize = manager.gpu(runtime::ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + mMaxPromptVocabSize = maxPromptEmbeddingTableSize / maxBatchSize; + + auto promptVocabSizeHost + = runtime::BufferManager::pinned(runtime::ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + auto promptVocabSizeHostData = runtime::bufferCast<SizeType32>(*promptVocabSizeHost); + promptVocabSizeHostData[0] = mMaxPromptVocabSize; + manager.copy(*promptVocabSizeHost, *mPromptTuningParams.vocabSize); + + // embeddingTable + mPromptTuningParams.embeddingTable = manager.gpu( + runtime::ITensor::makeShape({maxPromptEmbeddingTableSize, hiddenSize}), modelConfig.getDataType()); + + // tasks + mPromptTuningParams.tasks = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); +} + +PromptTuningBuffers::PromptTuningBuffers(SizeType32 maxBatchSize, runtime::BufferManager const& manager, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, bool promptTableOffloading) +{ + auto maxPromptEmbeddingTableSize = modelConfig.getMaxPromptEmbeddingTableSize(); + auto const hiddenSize = modelConfig.getHiddenSize() * worldConfig.getTensorParallelism(); + + // vocabSize and mMaxPromptVocabSize + mPromptTuningParams.vocabSize = manager.gpu(runtime::ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + mMaxPromptVocabSize = maxPromptEmbeddingTableSize / maxBatchSize; + mPromptTableOffloading = promptTableOffloading; + + auto promptVocabSizeHost + = runtime::BufferManager::pinned(runtime::ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + auto promptVocabSizeHostData = runtime::bufferCast<SizeType32>(*promptVocabSizeHost); + promptVocabSizeHostData[0] = mMaxPromptVocabSize; + manager.copy(*promptVocabSizeHost, *mPromptTuningParams.vocabSize); + + // embeddingTable + mPromptTuningParams.embeddingTable = manager.gpu( + runtime::ITensor::makeShape({maxPromptEmbeddingTableSize, hiddenSize}), modelConfig.getDataType()); + + // tasks + mPromptTuningParams.tasks = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); +} + +void PromptTuningBuffers::validate( + std::optional<TensorPtr> const& optReqPromptEmbeddingTable, std::optional<SizeType32> const& optReqPromptVocabSize) +{ + // Need to copy request embeddingTable to promptEmbeddingTable + if (optReqPromptEmbeddingTable.has_value()) + { + + auto reqPromptEmbeddingTable = optReqPromptEmbeddingTable.value(); + auto reqPromptVocabSize = optReqPromptVocabSize.value(); + + if (reqPromptVocabSize > mMaxPromptVocabSize) + { + std::string errStr = "Prompt vocab size" + std::to_string(reqPromptVocabSize) + + " is larger than max prompt vocab size of " + std::to_string(mMaxPromptVocabSize) + + ". Max prompt vocab size is computed from max_prompt_embedding_table_size / max_batch_size. "; + TLLM_LOG_ERROR(errStr); + throw std::runtime_error(errStr); + } + else + { + // Check that type matches model weights + if (reqPromptEmbeddingTable->getDataType() != mPromptTuningParams.embeddingTable->getDataType()) + { + std::string errStr = "Request embedding table data type doesn't match model weight data type."; + TLLM_LOG_ERROR(errStr); + throw std::runtime_error(errStr); + } + + if (reqPromptEmbeddingTable->getShape().d[1] != reqPromptVocabSize) + { + std::string errStr + = "First dimension of request embedding table is expected to be equal to prompt vocab size"; + TLLM_LOG_ERROR(errStr); + throw std::runtime_error(errStr); + } + } + } +} + +void PromptTuningBuffers::fill(RequestVector const& contextRequests, RequestVector const& genRequests, + runtime::BufferManager const& manager, bool packed) +{ + NVTX3_SCOPED_RANGE_WITH_NAME(range, "PromptTuningBuffers::fill"); + + auto const numContextRequests = static_cast<SizeType32>(contextRequests.size()); + + std::vector<SizeType32> reqBeamWidths; + std::vector<SizeType32> reqPromptLengths; + mPromptTuningParams.promptTuningEnabled.clear(); + + SizeType32 batchIdx{0}; + for (auto const& requests : {contextRequests, genRequests}) + { + for (auto const& llmReq : requests) + { + reqBeamWidths.push_back(llmReq->mSamplingConfig.beamWidth); + if (batchIdx < numContextRequests) + { + SizeType32 numContextTokens = 0; + auto const draftLength = llmReq->isLastContextChunk() ? llmReq->getNumDraftTokens() : 0; + auto const contextChunkSize = llmReq->getContextChunkSize(); + numContextTokens += contextChunkSize + draftLength; + reqPromptLengths.push_back(numContextTokens); + } + + std::optional<TensorPtr> optReqPromptEmbeddingTable = std::nullopt; + std::optional<SizeType32> optReqPromptVocabSize = std::nullopt; + + if (mPromptTableOffloading) + { + optReqPromptEmbeddingTable = getChunkPtableBuffer(getChunkPtableCurrentIndex()); + optReqPromptVocabSize = getChunkPtableBufferSliceSize(getChunkPtableCurrentIndex(), batchIdx); + } + else + { + optReqPromptEmbeddingTable = llmReq->getPromptEmbeddingTable(); + optReqPromptVocabSize = llmReq->getPromptVocabSize(); + } + + mPromptTuningParams.promptTuningEnabled.push_back(optReqPromptEmbeddingTable.has_value()); + + // If context request & has embedding table, validate it + if (optReqPromptEmbeddingTable.has_value()) + { + // If a context request, validate prompt tensors and move to GPU + if (batchIdx < numContextRequests) + { + if (mPromptTableOffloading) + { + // Need to slice the ptable since we don't need the entire buffer + // The size depends on optReqPromptVocabSize which stores how many fake prompts are in the chunk + auto slicedPtable = runtime::ITensor::slice( + optReqPromptEmbeddingTable.value(), 0, optReqPromptVocabSize.value()); + slicedPtable->unsqueeze(0); + optReqPromptEmbeddingTable = std::move(slicedPtable); + } + else + { + // Move to GPU + llmReq->movePromptEmbeddingTableToGpu(manager); + optReqPromptEmbeddingTable = llmReq->getPromptEmbeddingTable(); + } + + // Validate the table, prompt_vocab_size + validate(optReqPromptEmbeddingTable, optReqPromptVocabSize); + } + + auto const reqPromptEmbeddingTable = optReqPromptEmbeddingTable.value(); + auto const reqPromptVocabSize = optReqPromptVocabSize.value(); + + // TODO: Use invokeCopyBatch to avoid multiple bs1 copies + // Copy into large prompt embedding table + TensorPtr reqPromptEmbeddingTableView = runtime::ITensor::view(reqPromptEmbeddingTable); + reqPromptEmbeddingTableView->squeeze(0); + auto const promptEmbeddingTableSlice = runtime::ITensor::slice( + mPromptTuningParams.embeddingTable, batchIdx * mMaxPromptVocabSize, reqPromptVocabSize); + manager.copy(*reqPromptEmbeddingTable, *promptEmbeddingTableSlice); + // TODO: src: 2007040 (llmReq->getPromptEmbeddingTable()) != dst: 1003520 (reqPromptVocabSize) + // (original shape passed from + // python == 196 * 5120, fp16) + // VILA mode 1 , 2 images in one request + } + ++batchIdx; + } + } + + auto const batchSize = batchIdx; + std::vector<SizeType32> tasksHostVec(batchSize); + std::iota(tasksHostVec.begin(), tasksHostVec.end(), 0); + + // Create a tensor that wraps the vector and convert unique_ptr to shared_ptr + auto tasksHost = std::shared_ptr<runtime::ITensor>( + runtime::ITensor::wrap(tasksHostVec, runtime::ITensor::makeShape({batchSize})).release()); + + mPromptTuningParams.fillTasksTensor( + tasksHost, batchSize, numContextRequests, reqBeamWidths, reqPromptLengths, manager, packed); +} + +void PromptTuningBuffers::initializeChunkPtableBuffers(runtime::BufferManager const& manager, + runtime::ModelConfig const& modelConfig, SizeType32 contextChunkSize, std::shared_ptr<LlmRequest> const& llmReq) +{ + if (mChunkPtableInitialized) + { + return; + } + + std::array<TensorPtr, 2> buffers; + std::vector<std::vector<SizeType32>> startPositions(2); + for (int i = 0; i < 2; i++) + { + startPositions[i].emplace_back(0); + auto memType = llmReq->getPromptEmbeddingTable().value()->getDataType(); + buffers[i] = manager.gpu(runtime::ITensor::makeShape({contextChunkSize, modelConfig.getHiddenSize()}), memType); + } + + mChunkPtableBuffers = std::move(buffers); + mChunkPtableBufferStartPositions = std::move(startPositions); + + mChunkPtableCurrentIndex = 0; + mChunkPtableInitialized = true; +} + +void PromptTuningBuffers::switchChunkPtableBuffer() +{ + mChunkPtableCurrentIndex = 1 - mChunkPtableCurrentIndex; + clearBufferStartPositions(mChunkPtableCurrentIndex); +} + +size_t PromptTuningBuffers::getChunkPtableCurrentIndex() +{ + return mChunkPtableCurrentIndex; +} + +TensorPtr& PromptTuningBuffers::getChunkPtableBuffer(size_t index) +{ + if (!mChunkPtableBuffers.has_value()) + { + TLLM_THROW("Chunk ptable buffers not initialized"); + } + if (!mChunkPtableBuffers.value()[index]) + { + TLLM_THROW("Chunk ptable buffer at index %zu is null", index); + } + return mChunkPtableBuffers.value()[index]; +} + +SizeType32 PromptTuningBuffers::getChunkPtableBufferSliceSize(size_t index, size_t batchIdx) +{ + if (!mChunkPtableBufferStartPositions.has_value()) + { + return 0; + } + + if (batchIdx + 1 >= mChunkPtableBufferStartPositions.value()[index].size()) + { + TLLM_THROW("Batch index %zu + 1 out of bounds for buffer %zu (size: %zu)", batchIdx, index, + mChunkPtableBufferStartPositions.value()[index].size()); + } + + return mChunkPtableBufferStartPositions.value()[index][batchIdx + 1] + - mChunkPtableBufferStartPositions.value()[index][batchIdx]; +} + +SizeType32 PromptTuningBuffers::getChunkPtableBufferStartPosition(size_t index, size_t batchIdx) +{ + if (!mChunkPtableBufferStartPositions.has_value()) + { + return 0; + } + + if (batchIdx >= mChunkPtableBufferStartPositions.value()[index].size()) + { + TLLM_THROW("Batch index %zu out of bounds for buffer %zu (size: %zu)", batchIdx, index, + mChunkPtableBufferStartPositions.value()[index].size()); + } + + // For first batch, return the value directly + if (batchIdx == 0) + { + return mChunkPtableBufferStartPositions.value()[index][0]; + } + + // For other batches, return difference from previous position + return mChunkPtableBufferStartPositions.value()[index][batchIdx] + - mChunkPtableBufferStartPositions.value()[index][batchIdx - 1]; +} + +void PromptTuningBuffers::updateBufferStartPosition(size_t index, SizeType32 numRows) +{ + if (!mChunkPtableBufferStartPositions.has_value()) + { + return; + } + auto& positions = mChunkPtableBufferStartPositions.value()[index]; + positions.push_back(positions.back() + numRows); +} + +void PromptTuningBuffers::clearBufferStartPositions(size_t index) +{ + if (mChunkPtableBufferStartPositions.has_value()) + { + mChunkPtableBufferStartPositions.value()[index].clear(); + mChunkPtableBufferStartPositions.value()[index].emplace_back(0); + } +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/rnnCacheFormatter.cpp b/cpp/tensorrt_llm/batch_manager/rnnCacheFormatter.cpp index b7ceb727efa3..05742af17a28 100644 --- a/cpp/tensorrt_llm/batch_manager/rnnCacheFormatter.cpp +++ b/cpp/tensorrt_llm/batch_manager/rnnCacheFormatter.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -20,9 +20,9 @@ #include "tensorrt_llm/batch_manager/dataTransceiver.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/batch_manager/kvCacheUtils.h" +#include "tensorrt_llm/batch_manager/rnnStateManager.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/dataType.h" -#include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/nvtxUtils.h" #include "tensorrt_llm/executor/cache_transmission/agent_utils/connection.h" @@ -33,6 +33,15 @@ namespace tensorrt_llm::batch_manager { using CacheState = executor::kv_cache::CacheState; +RnnCacheFormatter::RnnCacheFormatter(rnn_state_manager::RnnStateManager* rnnStateManager, + rnn_state_manager::RnnCacheTransBufferManager* rnnCacheTransBufferManager) + : mRnnStateManager{rnnStateManager} + , mRnnCacheTransBufferManager{rnnCacheTransBufferManager} +{ + TLLM_CHECK(mRnnStateManager != nullptr); + TLLM_CHECK(mRnnCacheTransBufferManager != nullptr); +} + RnnCacheFormatter::RnnCacheFormatter(kv_cache_manager::BaseKVCacheManager* kvCacheManager, rnn_state_manager::RnnCacheTransBufferManager* rnnCacheTransBufferManager) : mRnnCacheTransBufferManager{rnnCacheTransBufferManager} @@ -43,13 +52,475 @@ RnnCacheFormatter::RnnCacheFormatter(kv_cache_manager::BaseKVCacheManager* kvCac } void RnnCacheFormatter::format(TransferSession& session) +{ + if (isUnifiedPoolMode()) + { + formatUnifiedPoolMode(session); + } + else + { + formatSlotMode(session); + } +} + +void RnnCacheFormatter::unformat(TransferSession& session) +{ + if (isUnifiedPoolMode()) + { + unformatUnifiedPoolMode(session); + } + else + { + unformatSlotMode(session); + } +} + +void RnnCacheFormatter::formatSlotMode(TransferSession& session) +{ + NVTX3_SCOPED_RANGE(RnnCacheFormatter_format); + session.setTime(TransferSession::kTimeFormatter); + + auto const& llmRequest = session.getLlmRequest(); + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "Start sending RNN state for request ID: %ld.", llmRequest.mRequestId); + TLLM_CHECK_WITH_INFO(llmRequest.mSamplingConfig.beamWidth == 1, "Currently, only beam width 1 is supported."); + + auto const& connections = session.getConnections(); + auto const& selfConfig = session.getSelfState().getCacheState().value(); + auto const& destConfig = session.getOtherState().getCacheState().value(); + auto const selfIdx = session.getSelfState().getCommState().value().getSelfIdx(); + auto& bufferManager = session.getBufferManager(); + + auto targetInfo = executor::kv_cache::targetIRanksForRnn(destConfig, selfConfig, selfIdx); + if (!cache_formatter_utils::needSendCache(selfConfig, destConfig, selfIdx, targetInfo)) + { + return; + } + + auto pickUpConnections = cache_formatter_utils::pickSendConnections( + connections.size(), selfConfig, selfIdx, destConfig, session.getCounterPartRanks(), targetInfo); + auto const targetNum = pickUpConnections.size(); + if (targetNum == 0) + { + TLLM_LOG_DEBUG("No targets to send RNN state to for request ID: %ld", llmRequest.mRequestId); + return; + } + + auto const slotIdx = mRnnStateManager->getCacheIndex(llmRequest.mRequestId); + int deviceId; + TLLM_CUDA_CHECK(cudaGetDevice(&deviceId)); + + auto const& selfParallel = selfConfig.getParallelConfig(); + auto const selfTPNum = selfParallel.mTensorParallelism; + auto const selfPPRank = selfIdx / selfTPNum; + auto const& selfLayersPerPP = selfConfig.getRnnCacheState().mLayerNumPerPP; + SizeType32 const numLocalLayers = selfLayersPerPP[selfPPRank]; + + if (common::getEnvTryZCopyForKVCacheTransfer() && destConfig == selfConfig) + { + TLLM_LOG_DEBUG("Try using zero-copy for the RNN cache."); + NVTX3_SCOPED_RANGE(RnnZeroCopySend); + + TLLM_CHECK(pickUpConnections.size() == 1); + + TLLM_CUDA_CHECK(cudaSetDevice(deviceId)); + for (size_t i = 0; i < pickUpConnections.size(); i++) + { + for (SizeType32 layer = 0; layer < numLocalLayers; layer++) + { + + // Get conv state for this layer: shape is [maxBatchSize, convDim, dConv-1] + auto convState = mRnnStateManager->getConvStates(mRnnStateManager->getGlobalLayerNum(layer)); + // Slice out the specific slot: shape becomes [convDim, dConv-1] + auto slotConv = runtime::ITensor::slice(convState, slotIdx, 1); + slotConv->squeeze(0); + + // Receive conv state + // llmRequest.updateKvCacheSize(slotConv->getSizeInBytes()); + session.send(pickUpConnections[i], slotConv->data(), slotConv->getSizeInBytes()); + + // Get SSM state for this layer: shape is [maxBatchSize, numHeads, headDim, dState] + auto ssmState = mRnnStateManager->getSsmStates(mRnnStateManager->getGlobalLayerNum(layer)); + // Slice out the specific slot: shape becomes [numHeads, headDim, dState] + auto slotSsm = runtime::ITensor::slice(ssmState, slotIdx, 1); + slotSsm->squeeze(0); + + // Receive SSM state + // llmRequest.updateKvCacheSize(slotSsm->getSizeInBytes()); + session.send(pickUpConnections[i], slotSsm->data(), slotSsm->getSizeInBytes()); + } + } + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "End the sending of RNN cache for the request ID: %ld.", + llmRequest.mRequestId); + + return; + } + + // Calculate buffer sizes for each target + // Each target gets: conv states + ssm states for overlapping layers + auto const& modelConfig = selfConfig.getRnnModelConfig(); + auto const maxBatchSize = mRnnStateManager->getMaxBatchSize(); + int const selfTPSizePerDPGroup = selfConfig.getParallelConfig().mEnableAttentionDP + ? selfTPNum / selfConfig.getParallelConfig().mDPsize + : selfTPNum; + SizeType32 convDimLocal = modelConfig.mConvDimSize / selfTPSizePerDPGroup; + SizeType32 numHeadsLocal = modelConfig.mNumHeads / selfTPSizePerDPGroup; + + size_t convBytesPerLayer + = convDimLocal * (modelConfig.mDConv - 1) * common::getDTypeSize(selfConfig.getConvStateDataType()); + convBytesPerLayer = (convBytesPerLayer + 15) & ~static_cast<size_t>(15); + size_t ssmBytesPerLayer = numHeadsLocal * modelConfig.mHeadDim * modelConfig.mDState + * common::getDTypeSize(selfConfig.getSsmStateDataType()); + + int peerDuplicateHeadFactor = targetInfo.mPeerDupHeadFactor; + auto bufferTargetNum = targetNum / peerDuplicateHeadFactor; + + std::vector<size_t> bufferSizesPerTarget(targetNum, 0); + + for (size_t i = 0; i < targetNum; i++) + { + SizeType32 layersForTarget = targetInfo.getPeerPPDomainLayerNum(static_cast<SizeType32>(i)); + bufferSizesPerTarget[i] = layersForTarget * (convBytesPerLayer + ssmBytesPerLayer) * peerDuplicateHeadFactor + / targetInfo.mDomainTPSize; + } + + auto cacheBufferId = mRnnCacheTransBufferManager->assignBufferIndexForSend(); + auto allocationResult = mRnnCacheTransBufferManager->getOrAllocateSendBuffers( + cacheBufferId, static_cast<int>(bufferTargetNum), bufferSizesPerTarget, bufferManager); + auto& outputBuffers = std::get<0>(allocationResult); + auto& bufferCoverTargetNum = std::get<1>(allocationResult); + auto& onlyUseDynamicBuffer = std::get<2>(allocationResult); + + TLLM_CHECK(cacheBufferId.has_value() || onlyUseDynamicBuffer); + + auto const* agentConnection + = dynamic_cast<executor::kv_cache::AgentConnection const*>(connections[pickUpConnections[0]]); + if (agentConnection != nullptr) + { + TLLM_CHECK_WITH_INFO(bufferCoverTargetNum == bufferTargetNum, "Agent needs all RNN send buffers pre-allocated"); + TLLM_CHECK(onlyUseDynamicBuffer == false); + } + + std::vector<runtime::ITensor::SharedPtr> inputConvBlocks; + std::vector<runtime::ITensor::SharedPtr> inputSsmBlocks; + + auto convStates = mRnnStateManager->getConvStates(); // [numLocalLayers, maxBatchSize, convDim, dConv-1] + auto ssmStates = mRnnStateManager->getSsmStates(); // [numLocalLayers, maxBatchSize, numHeads, headDim, dState] + + inputConvBlocks.push_back(convStates); + inputSsmBlocks.push_back(ssmStates); + + tensorrt_llm::executor::rnn_cache::splitRnnConvStateDispatch( + inputConvBlocks, outputBuffers, slotIdx, maxBatchSize, destConfig, selfConfig, selfIdx, bufferManager); + + // Conv and SSM use same output buffer. So need to track convBytesPerLayer to compute offset. + tensorrt_llm::executor::rnn_cache::splitRnnSsmStateDispatch(inputSsmBlocks, outputBuffers, slotIdx, maxBatchSize, + convBytesPerLayer, destConfig, selfConfig, selfIdx, bufferManager); + + bufferManager.getStream().synchronize(); + session.setTime(TransferSession::kTimePreprocess); + + auto preAllocSendBuffer = mRnnCacheTransBufferManager->getSendBuffer(cacheBufferId); + + sendAllBuffers(session, deviceId, outputBuffers, bufferCoverTargetNum, preAllocSendBuffer, bufferManager, + targetInfo, pickUpConnections); + + session.setTime(TransferSession::kTimeTransmissions); + + mRnnCacheTransBufferManager->freeBufferIndexForSend(cacheBufferId); + session.setTime(TransferSession::kTimePostprocess); + + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "End sending RNN state for request ID: %ld.", llmRequest.mRequestId); +} + +void RnnCacheFormatter::unformatSlotMode(TransferSession& session) +{ + NVTX3_SCOPED_RANGE(RnnCacheFormatter_unformat); + session.setTime(TransferSession::kTimeFormatter); + + auto& llmRequest = session.getLlmRequest(); + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "Start receiving RNN state for request ID: %ld.", llmRequest.mRequestId); + TLLM_CHECK_WITH_INFO(llmRequest.mSamplingConfig.beamWidth == 1, "Currently, only beam width 1 is supported."); + + auto const& connections = session.getConnections(); + auto const& selfConfig = session.getSelfState().getCacheState().value(); + auto const& destConfig = session.getOtherState().getCacheState().value(); + auto const selfIdx = session.getSelfState().getCommState().value().getSelfIdx(); + auto& bufferManager = session.getBufferManager(); + + auto sourceInfo = executor::kv_cache::targetIRanksForRnn(destConfig, selfConfig, selfIdx); + int deviceId; + TLLM_CUDA_CHECK(cudaGetDevice(&deviceId)); + + auto pickRecvConnResult = cache_formatter_utils::pickRecvConnections( + connections.size(), selfConfig, selfIdx, destConfig, session.getCounterPartRanks(), sourceInfo); + auto pickUpConnections = std::get<0>(pickRecvConnResult); + auto localRankIndices = std::get<1>(pickRecvConnResult); + auto const sourceNum = pickUpConnections.size(); + + if (sourceNum == 0) + { + TLLM_LOG_DEBUG("No sources to receive RNN state from for request ID: %ld", llmRequest.mRequestId); + return; + } + + if (common::getEnvDisaggLayerwise()) + { + TLLM_LOG_ERROR("Layer-wise RNN cache transfer is not supported yet"); + return; + } + + // Since allocation happens earlier + auto const slotIdx = mRnnStateManager->getCacheIndex(llmRequest.mRequestId); + + auto const& selfParallel = selfConfig.getParallelConfig(); + auto const selfTPNum = selfParallel.mTensorParallelism; + auto const selfPPRank = selfIdx / selfTPNum; + auto const& selfLayersPerPP = selfConfig.getRnnCacheState().mLayerNumPerPP; + SizeType32 const numLocalLayers = selfLayersPerPP[selfPPRank]; + + if (common::getEnvTryZCopyForKVCacheTransfer() && destConfig == selfConfig) + { + TLLM_LOG_DEBUG("try zcopy for RNN cache"); + NVTX3_SCOPED_RANGE(RnnZeroCopyRecv); + + TLLM_CHECK(sourceNum == 1); + + TLLM_CUDA_CHECK(cudaSetDevice(deviceId)); + for (size_t i = 0; i < sourceNum; i++) + { + for (SizeType32 layer = 0; layer < numLocalLayers; layer++) + { + + // Get conv state for this layer: shape is [maxBatchSize, convDim, dConv-1] + auto convState = mRnnStateManager->getConvStates(mRnnStateManager->getGlobalLayerNum(layer)); + // Slice out the specific slot: shape becomes [convDim, dConv-1] + auto slotConv = runtime::ITensor::slice(convState, slotIdx, 1); + slotConv->squeeze(0); + + // Send conv state + // llmRequest.updateKvCacheSize(slotConv->getSizeInBytes()); + session.recv(pickUpConnections[i], slotConv->data(), slotConv->getSizeInBytes()); + + // Get SSM state for this layer: shape is [maxBatchSize, numHeads, headDim, dState] + auto ssmState = mRnnStateManager->getSsmStates(mRnnStateManager->getGlobalLayerNum(layer)); + // Slice out the specific slot: shape becomes [numHeads, headDim, dState] + auto slotSsm = runtime::ITensor::slice(ssmState, slotIdx, 1); + slotSsm->squeeze(0); + + // Send SSM state + // llmRequest.updateKvCacheSize(slotSsm->getSizeInBytes()); + session.recv(pickUpConnections[i], slotSsm->data(), slotSsm->getSizeInBytes()); + } + } + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "End receiving RNN cache for request ID: %ld.", llmRequest.mRequestId); + return; + } + + // Calculate buffer sizes + auto const& modelConfig = selfConfig.getRnnModelConfig(); + int const selfTPSizePerDPGroup = selfParallel.mEnableAttentionDP ? selfTPNum / selfParallel.mDPsize : selfTPNum; + SizeType32 selfConvDimLocal = modelConfig.mConvDimSize / selfTPSizePerDPGroup; + int const selfNumHeadsLocal = modelConfig.mNumHeads / selfTPSizePerDPGroup; + + size_t convBytesPerLayer + = selfConvDimLocal * (modelConfig.mDConv - 1) * common::getDTypeSize(selfConfig.getConvStateDataType()); + convBytesPerLayer = (convBytesPerLayer + 15) & ~static_cast<size_t>(15); + size_t ssmBytesPerLayer = selfNumHeadsLocal * modelConfig.mHeadDim * modelConfig.mDState + * common::getDTypeSize(selfConfig.getSsmStateDataType()); + + std::vector<size_t> bufferSizesPerSource(sourceNum, 0); + size_t validTpSources = sourceNum / sourceInfo.mDomainPPSize; + + // Compute source conv bytes for SSM offset + size_t sourceConvBytesPerLayer = convBytesPerLayer / validTpSources; + + for (size_t i = 0; i < sourceNum; i++) + { + SizeType32 layersFromSource = sourceInfo.getPeerPPDomainLayerNum(static_cast<SizeType32>(localRankIndices[i])); + bufferSizesPerSource[i] = layersFromSource * (convBytesPerLayer + ssmBytesPerLayer) / validTpSources; + } + + // Allocate receive buffers + size_t remainNoCoverSourceNum = 0; + size_t bufferCoverSourceNum = 0; + std::optional<int> cacheBufferId = std::nullopt; + + auto preAssignedRnnId + = connections[pickUpConnections[0]]->getPreAssignedBufferId(static_cast<uint8_t>(BufferKind::kRNN)); + if (preAssignedRnnId.has_value()) + { + cacheBufferId = static_cast<int>(*preAssignedRnnId); + } + else + { + cacheBufferId = mRnnCacheTransBufferManager->assignBufferIndexForRecv(); + } + + auto allocationResult = mRnnCacheTransBufferManager->getOrAllocateRecvBuffers( + cacheBufferId, static_cast<int>(sourceNum), bufferSizesPerSource, bufferManager); + auto& recvBuffers = std::get<0>(allocationResult); + auto& bufferCoverSourceNumTmp = std::get<1>(allocationResult); + auto& onlyUseDynamicBuffer = std::get<2>(allocationResult); + + TLLM_CHECK(cacheBufferId.has_value() || onlyUseDynamicBuffer); + + if (preAssignedRnnId.has_value()) + { + TLLM_CHECK_WITH_INFO(bufferCoverSourceNumTmp == sourceNum, "Agent needs all RNN recv buffers pre-allocated"); + TLLM_CHECK(onlyUseDynamicBuffer == false); + } + + bufferCoverSourceNum = bufferCoverSourceNumTmp; + remainNoCoverSourceNum = sourceNum > bufferCoverSourceNum ? sourceNum - bufferCoverSourceNum : 0; + + bufferManager.getStream().synchronize(); + session.setTime(TransferSession::kTimePreprocess); + + // Get pre-allocated buffer for chunked receive + runtime::ITensor::SharedPtr preAllocRecvBuffer = nullptr; + if (cacheBufferId.has_value()) + { + preAllocRecvBuffer = mRnnCacheTransBufferManager->getRecvBuffer(cacheBufferId); + TLLM_CHECK(preAllocRecvBuffer != nullptr); + } + + auto recvBufferFun = [&](int devId, size_t srcIdx) + { + NVTX3_SCOPED_RANGE(recvBufferFun); + TLLM_CUDA_CHECK(cudaSetDevice(devId)); + TLLM_CHECK(recvBuffers.size() > srcIdx); + auto startTime = LlmRequest::getSteadyClockNow(); + size_t size = 0; + + if (srcIdx >= remainNoCoverSourceNum) + { + // Fast path: buffer is pre-allocated, receive directly + auto& buffer = recvBuffers[srcIdx]; + size = buffer->getSizeInBytes(); + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), " start recv srcIdx: %lu size:%lu", srcIdx, buffer->getSizeInBytes()); + session.recv(pickUpConnections[srcIdx], buffer->data(), buffer->getSizeInBytes()); + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), " recv srcIdx: %lu size:%lu", srcIdx, buffer->getSizeInBytes()); + } + else + { + // Slow path: chunked receive for buffers that couldn't be pre-allocated + auto recvBufferIdx = bufferCoverSourceNum == 0 ? 0 : srcIdx % bufferCoverSourceNum + remainNoCoverSourceNum; + auto recvBufferUsed = bufferCoverSourceNum == 0 ? preAllocRecvBuffer : recvBuffers[recvBufferIdx]; + + size_t remainRecvSize = recvBuffers[srcIdx]->getSize(); + size_t needRecvSize = recvBuffers[srcIdx]->getSize(); + + while (remainRecvSize > 0) + { + TLLM_CHECK(recvBufferUsed != nullptr); + auto recvBufferEleSize = recvBufferUsed->getSize(); + auto recvSize = std::min(remainRecvSize, recvBufferEleSize); + auto recvSlice = runtime::ITensor::slice(recvBufferUsed, 0, recvSize); + auto copySlice = runtime::ITensor::slice(recvBuffers[srcIdx], needRecvSize - remainRecvSize, recvSize); + size += recvSlice->getSizeInBytes(); + session.recv(pickUpConnections[srcIdx], recvSlice->data(), recvSlice->getSizeInBytes()); + // Use cudaMemcpyAsync since we're copying bytes + TLLM_CUDA_CHECK(cudaMemcpyAsync(copySlice->data(), recvSlice->data(), recvSlice->getSizeInBytes(), + cudaMemcpyDeviceToDevice, bufferManager.getStream().get())); + bufferManager.getStream().synchronize(); + remainRecvSize -= recvSize; + } + } + + auto endTime = LlmRequest::getSteadyClockNow(); + session.appendMeasure(startTime, endTime, size); + }; + + // Dispatch receives (sequential or parallel based on env var) + if (sourceNum > 1) + { + if (!common::getEnvEnableReceiveKVCacheParallel()) + { + TLLM_LOG_DEBUG("Sequential receive for RNN cache."); + for (size_t i = 0; i < sourceNum; i++) + { + recvBufferFun(deviceId, i); + } + } + else + { + // Parallel receive with controlled concurrency + auto concurrencyNum = std::min(std::max(static_cast<size_t>(1), bufferCoverSourceNum), sourceNum); + auto remainRecvNum = sourceNum; + + while (remainRecvNum > 0) + { + auto recvConcurrencyNum = std::min(remainRecvNum, concurrencyNum); + + // Avoid leaving a tiny remainder + if (remainRecvNum > concurrencyNum && remainRecvNum < (2 * concurrencyNum)) + { + recvConcurrencyNum = remainRecvNum - concurrencyNum; + } + + std::vector<std::future<void>> futures; + futures.reserve(recvConcurrencyNum); + for (size_t i = 0; i < recvConcurrencyNum; i++) + { + size_t idx = i + (sourceNum - remainRecvNum); + TLLM_CHECK(idx < sourceNum); + futures.push_back(std::async(std::launch::async, recvBufferFun, deviceId, idx)); + } + for (auto& future : futures) + { + future.get(); + } + remainRecvNum -= recvConcurrencyNum; + } + } + } + else + { + recvBufferFun(deviceId, 0); + } + session.setTime(TransferSession::kTimeTransmissions); + + // Unpack received buffers into RNN states + std::vector<runtime::ITensor::SharedPtr> outputConvBlocks; + std::vector<runtime::ITensor::SharedPtr> outputSsmBlocks; + + auto const maxBatchSize = mRnnStateManager->getMaxBatchSize(); + auto convStates = mRnnStateManager->getConvStates(); // [numLocalLayers, maxBatchSize, convDim, dConv-1] + auto ssmStates = mRnnStateManager->getSsmStates(); // [numLocalLayers, maxBatchSize, numHeads, headDim, dState] + + outputConvBlocks.push_back(convStates); + outputSsmBlocks.push_back(ssmStates); + + tensorrt_llm::executor::rnn_cache::concatRnnConvStateDispatch( + recvBuffers, outputConvBlocks, slotIdx, maxBatchSize, destConfig, selfConfig, selfIdx, bufferManager); + + tensorrt_llm::executor::rnn_cache::concatRnnSsmStateDispatch(recvBuffers, outputSsmBlocks, slotIdx, maxBatchSize, + sourceConvBytesPerLayer, destConfig, selfConfig, selfIdx, bufferManager); + + bufferManager.getStream().synchronize(); + + if (cacheBufferId.has_value()) + { + mRnnCacheTransBufferManager->freeBufferIndexForRecv(cacheBufferId); + } + session.setTime(TransferSession::kTimePostprocess); + + TLLM_LOG_DEBUG( + mpi::MpiComm::world().getRank(), "End receiving RNN state for request ID: %ld.", llmRequest.mRequestId); +} + +void RnnCacheFormatter::formatUnifiedPoolMode(TransferSession& session) { NVTX3_SCOPED_RANGE(RnnCacheFormatter_formatUnifiedPool); session.setTime(TransferSession::kTimeFormatter); - auto llmRequestOpt = session.getLlmRequest(); - TLLM_CHECK_WITH_INFO(llmRequestOpt.has_value(), "LlmRequest required for RNN state transfer"); - auto const& llmRequest = **llmRequestOpt; + auto const& llmRequest = session.getLlmRequest(); TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "Start sending unified pool RNN state for request ID: %ld.", llmRequest.mRequestId); TLLM_CHECK_WITH_INFO(llmRequest.mSamplingConfig.beamWidth == 1, "Currently, only beam width 1 is supported."); @@ -81,7 +552,7 @@ void RnnCacheFormatter::format(TransferSession& session) bool const recvSideHasCP = destConfig.getParallelConfig().mContextParallelism > 1; auto const indexFromEnd = session.getIndexFromEnd(); auto blockRange = kv_cache_manager::getBlockRangeForSending( - mKvCacheManager, llmRequestOpt, lastBlockKey, indexFromEnd, recvSideHasCP, ppSize); + mKvCacheManager, llmRequest, lastBlockKey, indexFromEnd, recvSideHasCP, ppSize); auto const& blockIdsPerWindow = blockRange.getBlockIdsPerWindow(); auto const allWindowSizes = blockRange.getWindowSizes(); @@ -172,26 +643,11 @@ void RnnCacheFormatter::format(TransferSession& session) bufferSizesPerTarget[t] = ssmBufBytes + convBufBytes; } - auto const* sendCancelFlag = common::getEnvDisaggEnableInflightCancel() - ? &session.getDataContext().getTransferTerminate() - : nullptr; - auto cacheBufferId = mRnnCacheTransBufferManager->assignBufferIndexForSend(sendCancelFlag); - BufferIndexHolder sendHolder(*mRnnCacheTransBufferManager, cacheBufferId, /*isRecv=*/false); + auto cacheBufferId = mRnnCacheTransBufferManager->assignBufferIndexForSend(); auto allocationResult = mRnnCacheTransBufferManager->getOrAllocateSendBuffers( cacheBufferId, static_cast<int>(bufferTargetNum), bufferSizesPerTarget, bufferManager); auto& outputBuffers = std::get<0>(allocationResult); auto& bufferCoverTargetNum = std::get<1>(allocationResult); - auto& onlyUseDynamicBuffer = std::get<2>(allocationResult); - TLLM_CHECK(cacheBufferId.has_value() || onlyUseDynamicBuffer); - auto const* agentConnection = rnnSendConns.empty() - ? nullptr - : dynamic_cast<executor::kv_cache::AgentConnection const*>(connections[rnnSendConns[0]]); - if (agentConnection != nullptr) - { - TLLM_CHECK_WITH_INFO(bufferCoverTargetNum == bufferTargetNum, - "Agent needs all unified-pool RNN send buffers pre-allocated"); - TLLM_CHECK(onlyUseDynamicBuffer == false); - } // Split each outputBuffer into SSM and conv portions. std::vector<runtime::ITensor::SharedPtr> ssmOutputBuffers(numTargets); @@ -217,30 +673,15 @@ void RnnCacheFormatter::format(TransferSession& session) bufferManager.getStream().synchronize(); session.setTime(TransferSession::kTimePreprocess); - // Send buffers to targets. + // Send (same protocol as slot mode). int deviceId; TLLM_CUDA_CHECK(cudaGetDevice(&deviceId)); auto preAllocSendBuffer = mRnnCacheTransBufferManager->getSendBuffer(cacheBufferId); - if (sendCancelFlag != nullptr && sendCancelFlag->load(std::memory_order_relaxed)) - { - TLLM_THROW("Unified-pool RNN cache transfer cancelled before NIXL submission"); - } - try - { - sendAllBuffers(session, deviceId, outputBuffers, bufferCoverTargetNum, preAllocSendBuffer, - bufferManager, targetInfo, rnnSendConns); - } - catch (...) - { - if (agentConnection != nullptr && common::getEnvDisaggEnableInflightCancel()) - { - sendHolder.poison(); - } - throw; - } + sendAllBuffers(session, deviceId, outputBuffers, bufferCoverTargetNum, preAllocSendBuffer, bufferManager, + targetInfo, rnnSendConns); session.setTime(TransferSession::kTimeTransmissions); - sendHolder.release(); + mRnnCacheTransBufferManager->freeBufferIndexForSend(cacheBufferId); } } @@ -248,14 +689,12 @@ void RnnCacheFormatter::format(TransferSession& session) llmRequest.mRequestId); } -void RnnCacheFormatter::unformat(TransferSession& session) +void RnnCacheFormatter::unformatUnifiedPoolMode(TransferSession& session) { NVTX3_SCOPED_RANGE(RnnCacheFormatter_unformatUnifiedPool); session.setTime(TransferSession::kTimeFormatter); - auto llmRequestOpt = session.getLlmRequest(); - TLLM_CHECK_WITH_INFO(llmRequestOpt.has_value(), "LlmRequest required for RNN state transfer"); - auto const& llmRequest = **llmRequestOpt; + auto const& llmRequest = session.getLlmRequest(); TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "Start receiving unified pool RNN state for request ID: %ld.", llmRequest.mRequestId); TLLM_CHECK_WITH_INFO(llmRequest.mSamplingConfig.beamWidth == 1, "Currently, only beam width 1 is supported."); @@ -380,23 +819,17 @@ void RnnCacheFormatter::unformat(TransferSession& session) bufferSizesPerSource[t] = ssmBufBytes + convBufBytes; } - // Use pre-assigned buffer ID from NIXL connection if available. + // Use pre-assigned buffer ID from NIXL connection if available (same as slot mode). std::optional<int> cacheBufferId = std::nullopt; - BufferIndexHolder recvHolder; auto preAssignedRnnId = connections[rnnRecvConns[0]]->getPreAssignedBufferId(static_cast<uint8_t>(BufferKind::kRNN)); if (preAssignedRnnId.has_value()) { cacheBufferId = static_cast<int>(*preAssignedRnnId); - if (!session.hasReservedRecvBuffer(*mRnnCacheTransBufferManager)) - { - recvHolder = BufferIndexHolder(*mRnnCacheTransBufferManager, cacheBufferId, /*isRecv=*/true); - } } else { cacheBufferId = mRnnCacheTransBufferManager->assignBufferIndexForRecv(); - recvHolder = BufferIndexHolder(*mRnnCacheTransBufferManager, cacheBufferId, /*isRecv=*/true); } auto allocationResult = mRnnCacheTransBufferManager->getOrAllocateRecvBuffers( @@ -443,12 +876,10 @@ void RnnCacheFormatter::unformat(TransferSession& session) bufferManager.getStream().synchronize(); - recvHolder.release(); + mRnnCacheTransBufferManager->freeBufferIndexForRecv(cacheBufferId); } } - (void) session.releaseReservedRecvBuffer(*mRnnCacheTransBufferManager); - TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), "End receiving unified pool RNN state for request ID: %ld.", llmRequest.mRequestId); } diff --git a/cpp/tensorrt_llm/batch_manager/rnnCacheTransBuffer.cpp b/cpp/tensorrt_llm/batch_manager/rnnCacheTransBuffer.cpp index f9c04200e8d2..8fa9508cbefe 100644 --- a/cpp/tensorrt_llm/batch_manager/rnnCacheTransBuffer.cpp +++ b/cpp/tensorrt_llm/batch_manager/rnnCacheTransBuffer.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -21,13 +21,58 @@ #include "tensorrt_llm/common/dataType.h" #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include <algorithm> namespace tensorrt_llm::batch_manager::rnn_state_manager { +size_t RnnCacheTransBufferManager::computeTransferBufferSize( + RnnStateManager* rnnStateManager, std::optional<size_t> maxNumTokens) +{ + SizeType32 numLocalLayers = rnnStateManager->getNumLocalLayers(); + + // Get the tensor for one layer to determine per-slot dimensions + // Conv state shape per layer: [maxBatchSize, convDim_local, dConv-1] + // SSM state shape per layer: [maxBatchSize, numHeads_local, headDim, dState] + // The tensors are shaped [maxBatchSize, ...], so one slot = total_size / maxBatchSize + auto convState = rnnStateManager->getConvStates(rnnStateManager->getGlobalLayerNum(0)); // Get first layer's tensor + auto ssmState = rnnStateManager->getSsmStates(rnnStateManager->getGlobalLayerNum(0)); + + auto convShape = convState->getShape(); + auto ssmShape = ssmState->getShape(); + + // Compute elements per slot per layer (divide total volume by batch size) + size_t convElemsPerSlotPerLayer = runtime::ITensor::volume(convShape) / convShape.d[0]; + size_t ssmElemsPerSlotPerLayer = runtime::ITensor::volume(ssmShape) / ssmShape.d[0]; + + size_t convDtypeSize = common::getDTypeSize(rnnStateManager->getConvStateDataType()); + size_t ssmDtypeSize = common::getDTypeSize(rnnStateManager->getSsmStateDataType()); + + size_t convBytesPerSlotPerLayer = convElemsPerSlotPerLayer * convDtypeSize; + size_t ssmBytesPerSlotPerLayer = ssmElemsPerSlotPerLayer * ssmDtypeSize; + + size_t bufferSizePerSlot = numLocalLayers * (convBytesPerSlotPerLayer + ssmBytesPerSlotPerLayer); + + TLLM_LOG_DEBUG( + "RNN computeTransferBufferSize: numLocalLayers=%d, convBytesPerLayer=%lu, ssmBytesPerLayer=%lu, " + "totalPerSlot=%lu", + numLocalLayers, convBytesPerSlotPerLayer, ssmBytesPerSlotPerLayer, bufferSizePerSlot); + + return bufferSizePerSlot > 0 ? bufferSizePerSlot : common::getEnvMemSizeForKVCacheTransferBuffer(); +} + +RnnCacheTransBufferManager::RnnCacheTransBufferManager( + RnnStateManager* rnnStateManager, std::optional<size_t> maxNumTokens) + : BaseTransBufferManager(computeTransferBufferSize(rnnStateManager, maxNumTokens), + nvinfer1::DataType::kUINT8, // Use byte buffer for mixed dtypes + maxNumTokens) + , mRnnStateManager{rnnStateManager} +{ + TLLM_CHECK(mRnnStateManager != nullptr); + TLLM_LOG_INFO("RnnCacheTransBufferManager created for RNN cache"); +} + size_t RnnCacheTransBufferManager::computeTransferBufferSizeFromPool( kv_cache_manager::BaseKVCacheManager* kvCacheManager, executor::kv_cache::CacheState const& cacheState, std::optional<size_t> maxNumTokens) @@ -96,7 +141,8 @@ size_t RnnCacheTransBufferManager::computeTransferBufferSizeFromPool( RnnCacheTransBufferManager::RnnCacheTransBufferManager(kv_cache_manager::BaseKVCacheManager* kvCacheManager, executor::kv_cache::CacheState const& cacheState, std::optional<size_t> maxNumTokens) : BaseTransBufferManager(computeTransferBufferSizeFromPool(kvCacheManager, cacheState, maxNumTokens), - tensorrt_llm::DataType::kUINT8, maxNumTokens) + nvinfer1::DataType::kUINT8, maxNumTokens) + , mRnnStateManager{nullptr} { TLLM_CHECK(kvCacheManager != nullptr); TLLM_LOG_INFO("RnnCacheTransBufferManager created for unified pool RNN cache"); diff --git a/cpp/tensorrt_llm/batch_manager/rnnCacheTransBuffer.h b/cpp/tensorrt_llm/batch_manager/rnnCacheTransBuffer.h index 124525184815..e6ffa06db994 100644 --- a/cpp/tensorrt_llm/batch_manager/rnnCacheTransBuffer.h +++ b/cpp/tensorrt_llm/batch_manager/rnnCacheTransBuffer.h @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -18,6 +18,7 @@ #pragma once #include "tensorrt_llm/batch_manager/baseTransBuffer.h" +#include "tensorrt_llm/batch_manager/rnnStateManager.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/runtime/bufferManager.h" @@ -42,6 +43,11 @@ class RnnCacheTransBufferManager : public BaseTransBufferManager using SizeType32 = tensorrt_llm::runtime::SizeType32; using CacheState = executor::kv_cache::CacheState; + /// @brief Constructor for slot-based path (CppMambaCacheManager with RnnStateManager). + /// @param rnnStateManager Pointer to the RNN state manager. + /// @param maxNumTokens Optional maximum number of tokens for buffer sizing. + RnnCacheTransBufferManager(RnnStateManager* rnnStateManager, std::optional<size_t> maxNumTokens = std::nullopt); + /// @brief Constructor for unified pool path (CppMambaHybridCacheManager). /// Computes buffer sizes from the KV cache manager's recurrent state pool metadata. /// @param kvCacheManager Pointer to the KV cache manager with unified pool. @@ -57,15 +63,26 @@ class RnnCacheTransBufferManager : public BaseTransBufferManager static size_t preAllocBufferSize( size_t rnnStateSizeBytes, std::optional<executor::CacheTransceiverConfig> const& cacheTransceiverConfig); + /// @brief Get the RNN state manager. + [[nodiscard]] RnnStateManager* getRnnStateManager() const noexcept + { + return mRnnStateManager; + } + [[nodiscard]] BufferKind getBufferKind() const override { return BufferKind::kRNN; } private: + /// @brief Compute transfer buffer size from RNN state configuration. + static size_t computeTransferBufferSize(RnnStateManager* rnnStateManager, std::optional<size_t> maxNumTokens); + /// @brief Compute transfer buffer size from unified pool metadata. static size_t computeTransferBufferSizeFromPool(kv_cache_manager::BaseKVCacheManager* kvCacheManager, executor::kv_cache::CacheState const& cacheState, std::optional<size_t> maxNumTokens); + + RnnStateManager* mRnnStateManager{nullptr}; }; } // namespace tensorrt_llm::batch_manager::rnn_state_manager diff --git a/cpp/tensorrt_llm/batch_manager/rnnStateBuffers.cpp b/cpp/tensorrt_llm/batch_manager/rnnStateBuffers.cpp new file mode 100644 index 000000000000..6fc7977ef8f1 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/rnnStateBuffers.cpp @@ -0,0 +1,78 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "rnnStateBuffers.h" + +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/rnnStateManager.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" + +using namespace tensorrt_llm::runtime; + +namespace tensorrt_llm::batch_manager +{ + +RnnStateBuffers::RnnStateBuffers(SizeType32 maxBatchSize, runtime::TllmRuntime const& runtime) +{ + auto const& manager = runtime.getBufferManager(); + + slotMappingHost = BufferManager::cpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + slotMappingDevice = manager.gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); +} + +void RnnStateBuffers::reshape(SizeType32 numSequences) +{ + slotMappingHost->reshape(ITensor::makeShape({numSequences})); + slotMappingDevice->reshape(ITensor::makeShape({numSequences})); +} + +void RnnStateBuffers::fillSlotMappings( + RequestVector const& contextRequests, rnn_state_manager::RnnStateManager* rnnStateManager) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(rnnStateBuffersFillSlotMappings); + + SizeType32 batchIdx{0}; + for (auto const& llmReq : contextRequests) + { + auto const seqSlot = llmReq->mSeqSlot.value(); + auto const reqBeamWidth = llmReq->mSamplingConfig.beamWidth; + rnnStateManager->fillSlotMapping(*slotMappingHost, batchIdx, seqSlot, reqBeamWidth); + ++batchIdx; + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void RnnStateBuffers::copySlotMappingH2D(runtime::TllmRuntime const& runtime) +{ + auto const& manager = runtime.getBufferManager(); + manager.copy(*slotMappingHost, *slotMappingDevice); +} + +void RnnStateBuffers::getBuffers(TensorMap& inputBuffers) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(rnnStateBuffersGetBuffers); + + inputBuffers.insert_or_assign("slot_mapping", slotMappingDevice); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/rnnStateBuffers.h b/cpp/tensorrt_llm/batch_manager/rnnStateBuffers.h new file mode 100644 index 000000000000..e25df47382a1 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/rnnStateBuffers.h @@ -0,0 +1,58 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/runtime/iTensor.h" + +namespace tensorrt_llm::runtime +{ +class TllmRuntime; +} // namespace tensorrt_llm::runtime + +namespace tensorrt_llm::batch_manager +{ + +namespace rnn_state_manager +{ +class RnnStateManager; +} + +class RnnStateBuffers +{ +public: + using SizeType32 = tensorrt_llm::runtime::SizeType32; + using TensorPtr = runtime::ITensor::SharedPtr; + using TensorMap = runtime::StringPtrMap<runtime::ITensor>; + + // others should be in rnnStateManager, we only need slotMapping here. + TensorPtr slotMappingHost; // [batch_size] + TensorPtr slotMappingDevice; // [batch_size] + + RnnStateBuffers(SizeType32 maxBatchSize, runtime::TllmRuntime const& runtime); + + void reshape(SizeType32 numSequences); + + void fillSlotMappings(RequestVector const& contextRequests, rnn_state_manager::RnnStateManager* rnnStateManager); + + void copySlotMappingH2D(runtime::TllmRuntime const& runtime); + + void getBuffers(TensorMap& inputBuffers) const; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/rnnStateManager.cpp b/cpp/tensorrt_llm/batch_manager/rnnStateManager.cpp index 7d032a268fdd..7608079fb396 100644 --- a/cpp/tensorrt_llm/batch_manager/rnnStateManager.cpp +++ b/cpp/tensorrt_llm/batch_manager/rnnStateManager.cpp @@ -17,7 +17,6 @@ #include "tensorrt_llm/batch_manager/rnnStateManager.h" #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/cudaStream.h" #include "tensorrt_llm/runtime/utils/runtimeUtils.h" @@ -81,7 +80,7 @@ RnnStateManager::RnnStateManager(SizeType32 maxNumSequences, tensorrt_llm::runti {localNbLayers, mMaxNumSequences * mBeamSlotsPerSequence, convKernel - 1, rnnConvDimSize}); mDtype = dataType; - mSsmCacheDtype = tensorrt_llm::DataType::kFLOAT; + mSsmCacheDtype = nvinfer1::DataType::kFLOAT; // Store RNN model config for CacheTransceiver mDState = stateSize; @@ -118,7 +117,7 @@ RnnStateManager::RnnStateManager(SizeType32 maxNumSequences, tensorrt_llm::runti RnnStateManager::RnnStateManager(SizeType32 dState, SizeType32 dConv, SizeType32 numHeads, SizeType32 nGroups, SizeType32 headDim, SizeType32 maxBatchSize, WorldConfig const& worldConfig, int64_t stream, - tensorrt_llm::DataType dtype, tensorrt_llm::DataType ssmCacheDtype, std::vector<SizeType32> const& ppLayers, + nvinfer1::DataType dtype, nvinfer1::DataType ssmCacheDtype, std::vector<SizeType32> const& ppLayers, SizeType32 numLayers) : mMaxNumSequences(maxBatchSize) , mMaxBeamWidth{1} @@ -298,12 +297,12 @@ RnnStateManager::TensorPtr RnnStateManager::getSsmStates() const return pagedRnnStates; } -tensorrt_llm::DataType RnnStateManager::getConvStateDataType() const noexcept +nvinfer1::DataType RnnStateManager::getConvStateDataType() const noexcept { return mDtype; } -tensorrt_llm::DataType RnnStateManager::getSsmStateDataType() const noexcept +nvinfer1::DataType RnnStateManager::getSsmStateDataType() const noexcept { return mSsmCacheDtype; } diff --git a/cpp/tensorrt_llm/batch_manager/runtimeBuffers.cpp b/cpp/tensorrt_llm/batch_manager/runtimeBuffers.cpp new file mode 100644 index 000000000000..ea5b9b06a96e --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/runtimeBuffers.cpp @@ -0,0 +1,1029 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" + +#include "tensorrt_llm/batch_manager/encoderBuffers.h" +#include "tensorrt_llm/batch_manager/kvCacheManager.h" +#include "tensorrt_llm/batch_manager/loraBuffers.h" +#include "tensorrt_llm/batch_manager/medusaBuffers.h" +#include "tensorrt_llm/batch_manager/promptTuningBuffers.h" +#include "tensorrt_llm/batch_manager/rnnStateBuffers.h" +#include "tensorrt_llm/batch_manager/rnnStateManager.h" +#include "tensorrt_llm/batch_manager/transformerBuffers.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/common/stlUtils.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/decoderState.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/runtimeKernels.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" + +#include <algorithm> +#include <iterator> +#include <memory> +#include <numeric> +#include <vector> + +using namespace tensorrt_llm::runtime; + +namespace tensorrt_llm::batch_manager +{ + +RuntimeBuffers::RuntimeBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, + std::vector<SizeType32> const& maxAttentionWindowVec, SizeType32 maxAttentionWindow, SizeType32 sinkTokenLen, + TllmRuntime const& runtime, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + executor::DecodingConfig const& decodingConfig, bool gatherGenerationLogits, std::optional<SizeType32> maxNumTokens, + std::optional<std::vector<executor::AdditionalModelOutput>> const& additionalModelOutputs, + bool promptTableOffloadingParam) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + promptTableOffloading = promptTableOffloadingParam; + + create(maxBatchSize, maxBeamWidth, maxAttentionWindowVec, maxAttentionWindow, sinkTokenLen, runtime, modelConfig, + worldConfig, decodingConfig, gatherGenerationLogits, additionalModelOutputs); + + // pre-allocate + setMaxBufferSizes(maxBatchSize, maxBeamWidth, modelConfig, maxNumTokens); + reshape(runtime, modelConfig, worldConfig, gatherGenerationLogits); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +RuntimeBuffers::~RuntimeBuffers() = default; + +void RuntimeBuffers::create(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, + std::vector<SizeType32> const& maxAttentionWindowVec, SizeType32 maxAttentionWindow, SizeType32 sinkTokenLen, + TllmRuntime const& runtime, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + executor::DecodingConfig const& decodingConfig, bool gatherGenerationLogits, + std::optional<std::vector<executor::AdditionalModelOutput>> const& additionalModelOutputs) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const& manager = runtime.getBufferManager(); + auto const& engine = runtime.getEngine(); + + if (modelConfig.isTransformerBased()) + { + transformerBuffers = std::make_unique<TransformerBuffers>(maxBatchSize, maxBeamWidth, maxAttentionWindowVec, + maxAttentionWindow, sinkTokenLen, runtime, modelConfig, worldConfig); + } + if (modelConfig.isRnnBased()) + { + rnnStateBuffers = std::make_unique<RnnStateBuffers>(maxBatchSize, runtime); + } + + auto constexpr nvTokenIdType = TRTDataType<TokenIdType>::value; + inputsIds = manager.emptyTensor(MemoryType::kGPU, nvTokenIdType); + + mropeRotaryCosSin = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kFLOAT); + mropePositionDeltas = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + + if (worldConfig.isLastPipelineParallelRank()) + { + auto const logitsType = engine.getTensorDataType(batch_manager::RuntimeBuffers::kLogitsTensorName); + logits = manager.emptyTensor(MemoryType::kGPU, logitsType); + } + + // TODO: check which tensors can be allocated as pinned for max size + requestTypes = manager.emptyTensor(MemoryType::kCPU, TRTDataType<runtime::RequestType>::value); + + contextLengthsHost = manager.emptyTensor(MemoryType::kCPU, nvinfer1::DataType::kINT32); + contextLengthsDevice = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + sequenceLengthsHost = manager.emptyTensor(MemoryType::kCPU, nvinfer1::DataType::kINT32); + sequenceLengthsDevice = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + + lastTokenIdsHost = manager.emptyTensor(MemoryType::kCPU, nvinfer1::DataType::kINT32); + lastTokenIdsDevice = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + logitsIdsHost = manager.emptyTensor(MemoryType::kCPU, nvinfer1::DataType::kINT32); + + inputsIds = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + + if (worldConfig.isPipelineParallel()) + { + hiddenStates = manager.emptyTensor(MemoryType::kGPU, modelConfig.getDataType()); + } + + auto const maxBatchSizeShape = ITensor::makeShape({maxBatchSize}); + seqSlots = tensorrt_llm::runtime::BufferManager::pinnedPool(maxBatchSizeShape, nvinfer1::DataType::kINT32); + seqSlotsDevice = manager.gpu(maxBatchSizeShape, nvinfer1::DataType::kINT32); + + cacheIndirDecoderIOBatchedCopySrcOffsets + = tensorrt_llm::runtime::BufferManager::pinnedPool(maxBatchSizeShape, nvinfer1::DataType::kINT64); + cacheIndirDecoderIOBatchedCopyDstOffsets + = tensorrt_llm::runtime::BufferManager::pinnedPool(maxBatchSizeShape, nvinfer1::DataType::kINT64); + cacheIndirDecoderIOBatchedCopySizes + = tensorrt_llm::runtime::BufferManager::pinnedPool(maxBatchSizeShape, nvinfer1::DataType::kINT64); + mCacheIndirDecoderIOBatchedCopySrcOffsetsSliceDevice = manager.gpu(maxBatchSizeShape, nvinfer1::DataType::kINT64); + mCacheIndirDecoderIOBatchedCopyDstOffsetsSliceDevice = manager.gpu(maxBatchSizeShape, nvinfer1::DataType::kINT64); + mCacheIndirDecoderIOBatchedCopyCopySizesDevice = manager.gpu(maxBatchSizeShape, nvinfer1::DataType::kINT64); + + // Pre-allocate buffer for saving generation logits for model w/o draft tokens + if (gatherGenerationLogits + && (modelConfig.getSpeculativeDecodingMode().isDraftTokensExternal() + || modelConfig.getSpeculativeDecodingMode().isNone()) + && worldConfig.isLastPipelineParallelRank()) + { + auto const vocabSizePadded = modelConfig.getVocabSizePadded(worldConfig.getSize()); + auto const logitsType = engine.getTensorDataType(batch_manager::RuntimeBuffers::kLogitsTensorName); + + generationLogitsCache.transposedLogits = manager.gpu( + ITensor::makeShape({maxBeamWidth, GenerationLogitsCache::kCACHE_LENGTH, vocabSizePadded}), logitsType); + generationLogitsCache.logits = manager.gpu( + ITensor::makeShape({GenerationLogitsCache::kCACHE_LENGTH, maxBatchSize * maxBeamWidth, vocabSizePadded}), + logitsType); + + generationLogitsCache.fragmentPointerDevice = manager.gpu( + ITensor::makeShape({maxBatchSize, GenerationLogitsCache::kCACHE_LENGTH}), nvinfer1::DataType::kINT64); + generationLogitsCache.fragmentPointerHost = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxBatchSize, GenerationLogitsCache::kCACHE_LENGTH}), nvinfer1::DataType::kINT64); + } + + if (modelConfig.useCrossAttention()) + { + encoderBuffers = std::make_unique<EncoderBuffers>(); + encoderBuffers->create(maxBatchSize, modelConfig, runtime); + } + + if (modelConfig.usePromptTuning()) + { + promptTuningBuffers = std::make_unique<PromptTuningBuffers>( + maxBatchSize, manager, modelConfig, worldConfig, promptTableOffloading); + } + + if (modelConfig.useLoraPlugin()) + { + loraBuffers = std::make_unique<LoraBuffers>(maxBatchSize, maxBeamWidth, runtime, modelConfig, worldConfig); + } + + if (modelConfig.getSpeculativeDecodingMode().isMedusa()) + { + mMedusaBuffers = std::make_unique<MedusaBuffers>( + maxBatchSize, maxBeamWidth, manager, modelConfig, worldConfig, decodingConfig, runtime); + } + else if (modelConfig.getSpeculativeDecodingMode().isLookaheadDecoding()) + { + mLookaheadBuffers = std::make_unique<runtime::LookaheadRuntimeBuffers>( + maxBatchSize, maxBeamWidth, manager, modelConfig, worldConfig, decodingConfig, runtime); + } + else if (modelConfig.getSpeculativeDecodingMode().isExplicitDraftTokens()) + { + mExplicitDraftTokensBuffers = std::make_unique<runtime::ExplicitDraftTokensBuffers>( + maxBatchSize, maxBeamWidth, manager, modelConfig, worldConfig); + } + else if (modelConfig.getSpeculativeDecodingMode().isEagle()) + { + mEagleBuffers = std::make_unique<runtime::EagleBuffers>( + maxBatchSize, maxBeamWidth, manager, modelConfig, worldConfig, decodingConfig); + } + + if (modelConfig.useLanguageAdapter()) + { + languageAdapterRoutings = manager.emptyTensor(MemoryType::kGPU, TRTDataType<SizeType32>::value); + } + + for (auto const& output : additionalModelOutputs.value_or(std::vector<executor::AdditionalModelOutput>{})) + { + auto const& engine = runtime.getEngine(); + auto const dataType = engine.getTensorDataType(output.name.c_str()); + mAdditionalOutputTensors.emplace(output.name, manager.emptyTensor(runtime::MemoryType::kGPU, dataType)); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void RuntimeBuffers::setMaxBufferSizes(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, + runtime::ModelConfig const& modelConfig, std::optional<SizeType32> maxNumRuntimeTokens) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // `maxNumSequences` is reached when all requests are in generation + numContextRequests = 0; + numGenRequests = maxBatchSize; + numGenSequences = maxBatchSize * maxBeamWidth; + + auto const maxDraftTokens = modelConfig.getMaxDecodingDraftTokens(); + // Draft-Tokens and Beam-Search are mutually exclusive + numLogits = maxBatchSize * std::max(1 + maxDraftTokens, maxBeamWidth); + auto const maxNumModelTokens = modelConfig.getMaxNumTokens(); + auto const maxNumContextTokens = maxBatchSize * modelConfig.getMaxInputLen(); + auto const maxNumGenTokens = numLogits; + // For pre-allocation + numContextTokens = 0; // Set in `setBufferSizes` rather than here for `computeContextLogits` + numGenTokens + = maxNumRuntimeTokens.value_or(maxNumModelTokens.value_or(std::max(maxNumContextTokens, maxNumGenTokens))); + + if (modelConfig.useCrossAttention()) + { + encoderBuffers->setMaxBufferSizes(maxBatchSize, modelConfig); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void RuntimeBuffers::setBufferSizes(RequestVector const& contextRequests, RequestVector const& genRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(runtimeBuffersSetBufferSizes); + + // set context sizes + numContextRequests = static_cast<SizeType32>(contextRequests.size()); + auto numContextLogits = numContextRequests; + numContextTokens = 0; + maxContextLength = 0; + for (auto const& llmReq : contextRequests) + { + auto const draftLength = llmReq->isLastContextChunk() ? llmReq->getNumDraftTokens() : 0; + numContextLogits += draftLength; + + auto const contextChunkSize = llmReq->getContextChunkSize(); + numContextTokens += contextChunkSize + draftLength; + if (maxContextLength < llmReq->mPromptLen) + { + maxContextLength = llmReq->mPromptLen; + } + } + + // set generation sizes + numGenRequests = static_cast<SizeType32>(genRequests.size()); + numGenSequences = 0; + numGenTokens = 0; + for (auto const& llmReq : genRequests) + { + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + numGenSequences += reqBeamWidth; + auto const draftLen = llmReq->getNumDraftTokens(); + numGenTokens += draftLen + reqBeamWidth; + } + + numLogits = numContextLogits + numGenTokens; + + if (encoderBuffers) + { + encoderBuffers->setBufferSizes(contextRequests, genRequests); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void RuntimeBuffers::reshape(TllmRuntime const& runtime, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + bool gatherGenerationLogits) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(runtimeBuffersReshape); + + if (worldConfig.isLastPipelineParallelRank()) + { + auto const vocabSizePadded = modelConfig.getVocabSizePadded(worldConfig.getSize()); + + if (modelConfig.computeContextLogits() && (numContextRequests > 0)) + { + // Only when need to return context logits, and there are new requests will execute context phase, + // logits buffer need to be re-allocated with size of [numContextTokens + numGenSequences, vocabSizePadded] + auto const& engine = runtime.getEngine(); + auto const& manager = runtime.getBufferManager(); + auto const logitsType = engine.getTensorDataType(kLogitsTensorName); + logits = manager.gpu(ITensor::makeShape({numContextTokens + numGenSequences, vocabSizePadded}), logitsType); + } + else if (gatherGenerationLogits && modelConfig.getSpeculativeDecodingMode().isNone()) + { + // If need to return generation logits, re-point the logit buffer to avoid overwrite, + // so we could write back GenerationLogitsCache::kCACHE_LENGTH steps' logits together + // logits shape: [1, maxBatchSize * maxBeamWidth, vocabSizePadded] + // which is large enough to cover both numContextRequests and numGenSequences + logits = ITensor::slice(generationLogitsCache.logits, generationLogitsCache.offset, 1); + generationLogitsCache.offset = (generationLogitsCache.offset + 1) % GenerationLogitsCache::kCACHE_LENGTH; + logits->squeeze(0); + } + else + { + logits->reshape(ITensor::makeShape({numLogits, vocabSizePadded})); + } + } + + auto const numSequences = getNumSequences(); + auto const numSequencesShape = ITensor::makeShape({numSequences}); + requestTypes->reshape(numSequencesShape); + contextLengthsHost->reshape(numSequencesShape); + contextLengthsDevice->reshape(numSequencesShape); + sequenceLengthsHost->reshape(numSequencesShape); + sequenceLengthsDevice->reshape(numSequencesShape); + + auto const numLogitsShape = ITensor::makeShape({numLogits}); + lastTokenIdsHost->reshape(numLogitsShape); + lastTokenIdsDevice->reshape(numLogitsShape); + logitsIdsHost->reshape(numLogitsShape); + + if (transformerBuffers) + { + transformerBuffers->reshape(numSequences, numContextTokens + numGenTokens); + } + + if (rnnStateBuffers) + { + rnnStateBuffers->reshape(numSequences); + } + + if (modelConfig.useCrossAttention()) + { + encoderBuffers->reshape(); + } + + if (modelConfig.useLoraPlugin()) + { + loraBuffers->reshape(numSequences); + } + + if (mMedusaBuffers) + { + mMedusaBuffers->reshape( + numContextRequests, numGenRequests, modelConfig.getSpeculativeDecodingModulePtr()->getMaxDecodingTokens()); + } + + if (mLookaheadBuffers && modelConfig.getSpeculativeDecodingMode().isLookaheadDecoding()) + { + mLookaheadBuffers->reshape( + numContextRequests, numGenRequests, modelConfig.getSpeculativeDecodingModulePtr()->getMaxDecodingTokens()); + } + + if (mExplicitDraftTokensBuffers) + { + mExplicitDraftTokensBuffers->reshape(numContextRequests, numGenRequests, modelConfig); + } + + if (mEagleBuffers) + { + mEagleBuffers->reshape(numContextRequests, numGenRequests, modelConfig); + } + + auto const numRequests = getNumRequests(); + auto const numRequestsShape = ITensor::makeShape({numRequests}); + seqSlots->reshape(numRequestsShape); + seqSlotsDevice->reshape(numRequestsShape); + + auto const numTokens = getNumTokens(); + inputsIds->reshape(ITensor::makeShape({numTokens})); + + if (modelConfig.useMrope()) + { + auto const mropeRotaryCosSinSize = modelConfig.getMaxPositionEmbeddings() * modelConfig.getRotaryEmbeddingDim(); + mropeRotaryCosSin->reshape(ITensor::makeShape({numSequences, mropeRotaryCosSinSize})); + mropePositionDeltas->reshape(ITensor::makeShape({numSequences, 1})); + } + + if (worldConfig.isPipelineParallel()) + { + auto const hiddenSize = (!modelConfig.getPpReduceScatter() || worldConfig.isFirstPipelineParallelRank()) + ? modelConfig.getHiddenSize() * worldConfig.getTensorParallelism() + : modelConfig.getHiddenSize(); + + auto const hiddenStatesShape = ITensor::makeShape({numTokens, hiddenSize}); + hiddenStates->reshape(hiddenStatesShape); + } + + if (modelConfig.useLanguageAdapter()) + { + languageAdapterRoutings->reshape(ITensor::makeShape({numTokens, 1})); + } + + for (auto const& outputTensor : mAdditionalOutputTensors) + { + auto const& [name, tensor] = outputTensor; + auto const& engine = runtime.getEngine(); + auto shape = engine.getTensorShape(name.c_str()); + TLLM_CHECK_WITH_INFO( + shape.d[0] == -1, "First dimension of additional output tensor '%s' must be dynamic", name.c_str()); + shape.d[0] = numTokens; + tensor->reshape(shape); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void RuntimeBuffers::prepareBuffersForCudaGraph(SizeType32 maxSequenceLength) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(prepareBuffersForCudaGraph); + + TLLM_CHECK(numContextRequests == 0); + + if (transformerBuffers) + { + // Set pastKeyValueLength for graph capturing. This way we will capture graph with + // maxKvCacheLengthRounded rounded to the next kKV_CACHE_LEN_CUDA_GRAPH_ROUND_SIZE. + // MMHA will launch excessive amount of blocks and some of them will exit early during the actual launch. + // We can reuse the same graph for the next kKV_CACHE_LEN_CUDA_GRAPH_ROUND_SIZE iterations. + + // make sure the size does not overflow the max allowed pastKvCacheLength + auto const pastKvCacheLength = std::min(maxSequenceLength - 1, maxKvCacheLengthRounded); + + auto* pastKeyValueLengthsPtr = bufferCast<SizeType32>(*transformerBuffers->pastKeyValueLengths); + std::fill_n(pastKeyValueLengthsPtr, getNumSequences(), pastKvCacheLength); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void RuntimeBuffers::setFromInputs(RequestVector const& contextRequests, RequestVector const& genRequests, + SizeType32 maxBeamWidth, SizeType32 maxAttentionWindow, runtime::decoder::DecoderState const& decoderState, + kv_cache_manager::BaseKVCacheManager* kvCacheManagerPtr, + kv_cache_manager::BaseKVCacheManager* crossKvCacheManagerPtr, + rnn_state_manager::RnnStateManager* rnnStateManagerPtr, PeftTable const& peftTable, + runtime::TllmRuntime const& runtime, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig, bool trtOverlap, OptionalRef<runtime::ITensor const> newOutputTokens) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(runtimeBuffersSetFromInputs); + + auto const& manager = runtime.getBufferManager(); + auto const& stream = runtime.getStream(); + + // Fill requestTypes + { + auto* hostRequestTypes = bufferCast<runtime::RequestType>(*requestTypes); + std::fill_n(hostRequestTypes, numContextRequests, runtime::RequestType::kCONTEXT); + std::fill_n(hostRequestTypes + numContextRequests, numGenSequences, runtime::RequestType::kGENERATION); + } + + SizeType32 totalInputSize = 0; + std::vector<TokenIdType> inputHost; + std::vector<SizeType32> positionIdsHost; + std::vector<SizeType32> positionIdsHostRow2; + std::vector<SizeType32> mropePositionDeltasHost; + std::vector<SizeType32> languageAdapterRoutingsHost; + + auto* contextLengthsHostPtr = bufferCast<SizeType32>(*contextLengthsHost); + auto* sequenceLengthsHostPtr = bufferCast<SizeType32>(*sequenceLengthsHost); + auto* pastKeyValueLengthsPtr + = transformerBuffers ? bufferCast<SizeType32>(*transformerBuffers->pastKeyValueLengths) : nullptr; + SizeType32 totalNumLogits{0}; + auto* logitsIdsHostPtr = bufferCast<SizeType32>(*logitsIdsHost); + bool const isChatGlm = modelConfig.getModelVariant() == ModelConfig::ModelVariant::kChatGlm; + bool const isGlm = modelConfig.getModelVariant() == ModelConfig::ModelVariant::kGlm; + auto const mropeRotaryCosSinSize = modelConfig.getMaxPositionEmbeddings() * modelConfig.getRotaryEmbeddingDim(); + + { + NVTX3_SCOPED_RANGE(seqSlotsLoop); + auto* seqSlotIndices = bufferCast<SizeType32>(*seqSlots); + + SizeType32 batchIdx{0}; + for (auto const& requests : {contextRequests, genRequests}) + { + for (auto const& llmReq : requests) + { + // Get position of the current sequence in the decoder + auto const seqSlot = llmReq->mSeqSlot.value(); + seqSlotIndices[batchIdx] = seqSlot; + ++batchIdx; + } + } + + TLLM_CHECK(seqSlots->getSize() == static_cast<std::size_t>(batchIdx)); + manager.copy(*seqSlots, *seqSlotsDevice); + } + + // context preparation loop + if (!contextRequests.empty()) + { + NVTX3_SCOPED_RANGE(contextPrepareLoop); + numContextLogits.resize(contextRequests.size()); + + SizeType32 batchIdx{0}; + for (auto const& llmReq : contextRequests) + { + TLLM_CHECK_WITH_INFO(llmReq->isContextInitState() || llmReq->isDisaggGenerationTransmissionComplete(), + "The request should be in context phase or disaggregated generation tranmissionComplete phase."); + TLLM_CHECK_WITH_INFO( + llmReq->getMaxNumGeneratedTokens() == 0, "Context request should not have generated tokens."); + + auto const& reqTokens = llmReq->getTokens(0); + auto const& draftTokens = llmReq->getDraftTokens(); + auto const draftLength = llmReq->getNumDraftTokens(); + auto const& positionIds = llmReq->getPositionIds(); + + auto const contextChunkSize = llmReq->getContextChunkSize(); + auto const beginCompute = llmReq->getContextCurrentPosition(); + auto const endCompute = beginCompute + contextChunkSize; + inputHost.insert(inputHost.end(), reqTokens.begin() + beginCompute, reqTokens.begin() + endCompute); + + logitsIdsHostPtr[totalNumLogits++] = contextChunkSize; + numContextLogits.at(batchIdx) = modelConfig.computeContextLogits() ? contextChunkSize : 1; + + if (llmReq->isLastContextChunk()) + { + inputHost.insert(inputHost.end(), draftTokens->begin(), draftTokens->end()); + std::fill_n(logitsIdsHostPtr + totalNumLogits, draftLength, 1); + totalNumLogits += draftLength; + } + auto const inputLength = contextChunkSize + (llmReq->isLastContextChunk() ? draftLength : 0); + contextLengthsHostPtr[batchIdx] = inputLength; + auto const sequenceLen = inputLength + llmReq->getContextCurrentPosition(); + sequenceLengthsHostPtr[batchIdx] = sequenceLen; + + if (static_cast<bool>(pastKeyValueLengthsPtr)) + { + pastKeyValueLengthsPtr[batchIdx] = beginCompute + inputLength; + } + + if (positionIds.has_value()) + { + TLLM_CHECK_WITH_INFO(!(isChatGlm || isGlm), "ChatGLM-6B and Glm only use the default initialization"); + positionIdsHost.insert(positionIdsHost.end(), positionIds.value()->begin() + beginCompute, + positionIds.value()->begin() + endCompute); + } + else + { + if (isChatGlm) + { + // Specialize for ChatGLM-6B with 2D-Position-Embedding + positionIdsHost.resize(totalInputSize + inputLength); + std::iota(std::begin(positionIdsHost) + totalInputSize, std::end(positionIdsHost), 0); + positionIdsHost.back() = positionIdsHost.back() - 1; + + positionIdsHostRow2.resize(totalInputSize + inputLength); + positionIdsHostRow2.back() = 1; + } + else if (isGlm) + { + // Specialize for GLM-10B with 2D-Position-Embedding and special value of the mask id position + auto start = inputHost.begin() + totalInputSize; + auto end = start + inputLength; + auto it = std::find_if( + start, end, [](SizeType32 id) { return id == 50260 || id == 50263 || id == 50264; }); + llmReq->mMaskPosition = (it != end) ? std::distance(start, it) : maxContextLength; + + positionIdsHost.resize(totalInputSize + inputLength); + std::iota(std::begin(positionIdsHost) + totalInputSize, std::end(positionIdsHost), 0); + positionIdsHost.back() = llmReq->mMaskPosition; + + positionIdsHostRow2.resize(totalInputSize + inputLength); + positionIdsHostRow2.back() = 1; + } + else + { + // Other models + positionIdsHost.resize(totalInputSize + inputLength); + std::iota(std::begin(positionIdsHost) + totalInputSize, + std::begin(positionIdsHost) + totalInputSize + inputLength, beginCompute); + } + } + if (modelConfig.useMrope()) + { + auto optMropeRotaryCosSin = llmReq->getMropeRotaryCosSin().value(); + TLLM_CHECK_WITH_INFO(optMropeRotaryCosSin->getShape().d[0] == mropeRotaryCosSinSize, + "Provided MropeRotarySinCos is %ld and expected is %d.\n", optMropeRotaryCosSin->getShape().d[0], + int(mropeRotaryCosSinSize)); + + auto const mropeRotaryCosSinCtx = ITensor::slice(mropeRotaryCosSin, batchIdx, 1); + manager.copy(*optMropeRotaryCosSin, *mropeRotaryCosSinCtx); + } + + if (modelConfig.useLanguageAdapter()) + { + auto const languageAdapterRouting = llmReq->getLanguageAdapterRouting( + modelConfig.getNumLanguages().value(), endCompute - beginCompute); + languageAdapterRoutingsHost.insert(languageAdapterRoutingsHost.end(), + std::begin(languageAdapterRouting), std::end(languageAdapterRouting)); + } + totalInputSize += inputLength; + ++batchIdx; + } + + if (rnnStateBuffers) + { + rnnStateBuffers->fillSlotMappings(contextRequests, rnnStateManagerPtr); + } + } + + // generation preparation loop + if (!genRequests.empty()) + { + NVTX3_SCOPED_RANGE(genPrepareLoop); + + auto const numContextRequests = static_cast<SizeType32>(contextRequests.size()); + auto numSequences = numContextRequests; + for (auto const& llmReq : genRequests) + { + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + auto const draftLength = llmReq->getNumDraftTokens(); + auto const& draftTokens = llmReq->getDraftTokens(); + auto const numLogits = draftLength + reqBeamWidth; + TLLM_CHECK(draftLength == 0 || reqBeamWidth == 1); + + auto const promptLen = llmReq->mPromptLen; + auto const sequenceLen + = promptLen + llmReq->getMaxNumGeneratedTokens() + static_cast<SizeType32>(trtOverlap); + auto const& positionIds = llmReq->getPositionIds(); + for (int beam = 0; beam < reqBeamWidth; ++beam) + { + auto const numTokens = llmReq->getNumTokens(beam) + static_cast<SizeType32>(trtOverlap); + // TODO: can this be removed completely? + if (!trtOverlap) + { + auto const lastToken = llmReq->getLastTokens(beam); + inputHost.push_back(lastToken); + if (draftLength > 0) + { + inputHost.insert(inputHost.end(), draftTokens->begin(), draftTokens->end()); + } + } + + // If model updates generation position ids do not append them here. + if (!modelConfig.getSpeculativeDecodingMode().updatesPositionIds()) + { + if (positionIds.has_value()) + { + TLLM_CHECK_WITH_INFO( + !(isChatGlm || isGlm), "ChatGLM-6B and Glm only use the default initialization"); + auto last_context_position_id = positionIds.value()->back(); + positionIdsHost.push_back( + static_cast<SizeType32>(last_context_position_id + sequenceLen - promptLen)); + } + else + { + if (isChatGlm) // ChatGLM-6B + { + positionIdsHost.push_back(static_cast<SizeType32>(promptLen - 2)); + positionIdsHostRow2.push_back(static_cast<SizeType32>(sequenceLen - promptLen + 1)); + } + else if (isGlm) + { + positionIdsHost.push_back(llmReq->mMaskPosition); + positionIdsHostRow2.push_back(static_cast<SizeType32>(sequenceLen - promptLen + 1)); + } + else // GPT / ChatGLM2-6B / ChatGLM3-6B / BART + { + // positionIds is just the size of tokens -1 + positionIdsHost.push_back(numTokens - 1); + } + } + } + + if (modelConfig.useMrope()) + { + auto optMropePositionDeltas = llmReq->getMropePositionDeltas().value(); + mropePositionDeltasHost.push_back(optMropePositionDeltas); + } + + if (modelConfig.useLanguageAdapter()) + { + // Generation requests only have one token per sequence + auto const languageAdapterRouting + = llmReq->getLanguageAdapterRouting(modelConfig.getNumLanguages().value(), 1); + languageAdapterRoutingsHost.insert(languageAdapterRoutingsHost.end(), + std::begin(languageAdapterRouting), std::end(languageAdapterRouting)); + } + } + + if (static_cast<bool>(pastKeyValueLengthsPtr)) + { + SizeType32 pastKeyValueLength = sequenceLen - 1; + std::fill_n(pastKeyValueLengthsPtr + numSequences, reqBeamWidth, pastKeyValueLength); + } + totalInputSize += numLogits; + + std::fill_n(logitsIdsHostPtr + totalNumLogits, numLogits, 1); + + totalNumLogits += numLogits; + + if (rnnStateBuffers) + { + auto const seqSlot = llmReq->mSeqSlot.value(); + auto& rnnStateManager = *rnnStateManagerPtr; + rnnStateManager.fillSlotMapping(*rnnStateBuffers->slotMappingHost, numSequences, seqSlot, reqBeamWidth); + } + numSequences += reqBeamWidth; + } + + if (transformerBuffers && maxBeamWidth > 1) + { + transformerBuffers->copyCacheIndirection(genRequests, decoderState.getCacheIndirectionOutput(), stream); + } + + numSequences = numContextRequests; + for (auto const& llmReq : genRequests) + { + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + auto const draftLength = llmReq->getNumDraftTokens(); + + auto const contextQLength = llmReq->mPromptLen + draftLength; + auto const sequenceLen + = contextQLength + llmReq->getMaxNumGeneratedTokens() + static_cast<SizeType32>(trtOverlap); + + std::fill_n(contextLengthsHostPtr + numSequences, reqBeamWidth, contextQLength); + std::fill_n(sequenceLengthsHostPtr + numSequences, reqBeamWidth, sequenceLen); + numSequences += reqBeamWidth; + } + if (modelConfig.getSpeculativeDecodingMode().isLookaheadDecoding()) + { + // copy from lookahead decoding buffer + mLookaheadBuffers->setFromInputs(numContextRequests, numGenRequests, *requestTypes, *seqSlots, + decoderState.getLookaheadBuffers(), runtime, modelConfig, worldConfig); + } + } + + // check skipCrossAttnBlocks + if (transformerBuffers && modelConfig.skipCrossAttnBlocks()) + { + bool isSkipCrossAttn = true; + for (auto const& requests : {contextRequests, genRequests}) + { + for (auto const& llmReq : requests) + { + bool tmpValue = false; + if (llmReq->getSkipCrossAttnBlocks() != nullptr) + { + manager.copy(*llmReq->getSkipCrossAttnBlocks(), &tmpValue); + } + isSkipCrossAttn &= tmpValue; + } + } + transformerBuffers->copySkipCrossAttnBlocks(isSkipCrossAttn, runtime); + } + + if (isChatGlm || isGlm) + { + positionIdsHost.reserve(totalInputSize * 2); + positionIdsHost.insert(positionIdsHost.end(), positionIdsHostRow2.begin(), positionIdsHostRow2.end()); + } + + if (modelConfig.useCrossAttention()) + { + encoderBuffers->fill(contextRequests, genRequests, manager); + } + if (modelConfig.usePromptTuning()) + { + promptTuningBuffers->fill(contextRequests, genRequests, manager, modelConfig.usePackedInput()); + } + if (modelConfig.useLoraPlugin()) + { + loraBuffers->fill(contextRequests, genRequests, peftTable, manager, modelConfig, worldConfig); + } + if (modelConfig.useMrope()) + { + if (!mropePositionDeltasHost.empty()) + { + auto mropePositionDeltasGen = ITensor::slice(mropePositionDeltas, 0, numGenSequences); + manager.copy(mropePositionDeltasHost.data(), *mropePositionDeltasGen); + } + } + + { + NVTX3_SCOPED_RANGE(bufferCopies); + if (trtOverlap) + { + auto contextInputsIds = ITensor::slice(inputsIds, 0, numContextTokens); + manager.copy(inputHost.data(), *contextInputsIds); + + if (!genRequests.empty()) + { + auto generationInputsIds = ITensor::slice(inputsIds, numContextTokens); + auto seqSlotsDeviceSlice = ITensor::slice(seqSlotsDevice, numContextRequests); + runtime::kernels::invokeGatherBatch( + *generationInputsIds, *newOutputTokens, *seqSlotsDeviceSlice, maxBeamWidth, stream); + } + } + else + { + manager.copy(inputHost.data(), *inputsIds); + } + // In generation phase, device ptr of context lengths need to be tiled. + manager.copy(*contextLengthsHost, *contextLengthsDevice); + manager.copy(*sequenceLengthsHost, *sequenceLengthsDevice); + auto const logitsIdsHostRange = BufferRange<SizeType32>(*logitsIdsHost); + auto lastTokenIdsHostRange = BufferRange<SizeType32>(*lastTokenIdsHost); + common::stl_utils::inclusiveScan( + logitsIdsHostRange.begin(), logitsIdsHostRange.end(), lastTokenIdsHostRange.begin()); + manager.copy(*lastTokenIdsHost, *lastTokenIdsDevice); + if (transformerBuffers) + { + TensorPtr decoderPositionIds = modelConfig.getSpeculativeDecodingMode().isLookaheadDecoding() + ? mLookaheadBuffers->positionIdsDevice + : nullptr; + transformerBuffers->copyPositionIds(runtime, positionIdsHost, isChatGlm || isGlm, decoderPositionIds); + } + if (rnnStateBuffers) + { + rnnStateBuffers->copySlotMappingH2D(runtime); + } + if (modelConfig.useLanguageAdapter()) + { + manager.copy(languageAdapterRoutingsHost.data(), *languageAdapterRoutings); + } + } + + if (transformerBuffers && static_cast<bool>(kvCacheManagerPtr)) + { + transformerBuffers->copyKvBlockOffsets( + contextRequests, genRequests, kvCacheManagerPtr, crossKvCacheManagerPtr, manager); + } + + if (modelConfig.useCrossAttention()) + { + transformerBuffers->copyCrossAttentionMasks(contextRequests, genRequests, contextLengthsDevice, + encoderBuffers->inputLengths, maxContextLength, encoderBuffers->getMaxInputLengthInBatch(), runtime); + } + + maxKvCacheLengthRounded = 0; + if (static_cast<bool>(pastKeyValueLengthsPtr)) + { + auto const maxKvCacheLength + = *std::max_element(pastKeyValueLengthsPtr, pastKeyValueLengthsPtr + getNumSequences()); + // Round up kv cache length + maxKvCacheLengthRounded = common::ceilDiv(maxKvCacheLength, kKV_CACHE_LEN_CUDA_GRAPH_ROUND_SIZE) + * kKV_CACHE_LEN_CUDA_GRAPH_ROUND_SIZE; + } + + if (modelConfig.getSpeculativeDecodingMode().needsDecoderPrologue()) + { + if (modelConfig.getSpeculativeDecodingMode().isExplicitDraftTokens()) + { + prepareExplicitDraftTokenBuffers( + decoderState.getExplicitDraftTokensBuffers(), runtime, modelConfig, worldConfig); + } + if (modelConfig.getSpeculativeDecodingMode().isEagle()) + { + prepareEagleBuffers( + contextRequests, genRequests, decoderState.getEagleBuffers(), runtime, modelConfig, worldConfig); + } + } + + sync_check_cuda_error(stream.get()); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void RuntimeBuffers::prepareExplicitDraftTokenBuffers( + runtime::ExplicitDraftTokensBuffers::Inputs const& explicitDraftTokensBuffers, TllmRuntime const& runtime, + ModelConfig const& modelConfig, WorldConfig const& worldConfig) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK(mExplicitDraftTokensBuffers); + + mExplicitDraftTokensBuffers->setFromInputs(numContextRequests, numGenRequests, *requestTypes, *seqSlots, + explicitDraftTokensBuffers, *transformerBuffers->positionIds, modelConfig, worldConfig, + runtime.getBufferManager(), runtime.getStream()); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void RuntimeBuffers::prepareEagleBuffers(RequestVector const& contextRequests, RequestVector const& genRequests, + runtime::EagleBuffers::Inputs const& eagleBuffers, TllmRuntime const& runtime, ModelConfig const& modelConfig, + WorldConfig const& worldConfig) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK(mEagleBuffers); + + mEagleBuffers->setFromInputs(contextRequests, genRequests, *requestTypes, *seqSlots, eagleBuffers, + runtime.getBufferManager(), modelConfig, worldConfig); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +std::tuple<SizeType32, RuntimeBuffers::TensorMap const&, RuntimeBuffers::TensorMap&> RuntimeBuffers::prepareStep( + RequestVector const& contextRequests, RequestVector const& genRequests, SizeType32 maxBeamWidth, + SizeType32 maxAttentionWindow, runtime::decoder::DecoderState const& decoderState, + kv_cache_manager::BaseKVCacheManager* kvCacheManager, kv_cache_manager::BaseKVCacheManager* crossKvCacheManager, + rnn_state_manager::RnnStateManager* rnnStateManager, PeftTable const& peftTable, TllmRuntime const& runtime, + ModelConfig const& modelConfig, WorldConfig const& worldConfig, bool gatherGenerationLogits, bool trtOverlap, + OptionalRef<runtime::ITensor const> newOutputTokens) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(runtimeBuffersPrepareStep); + + setBufferSizes(contextRequests, genRequests); + reshape(runtime, modelConfig, worldConfig, gatherGenerationLogits); + + setFromInputs(contextRequests, genRequests, maxBeamWidth, maxAttentionWindow, decoderState, kvCacheManager, + crossKvCacheManager, rnnStateManager, peftTable, runtime, modelConfig, worldConfig, trtOverlap, + newOutputTokens); + + fillIOMaps(modelConfig, worldConfig); + + auto const numTokens = getNumTokens(); + auto const optProfileId = runtime.getOptProfileId(numTokens, ModelConfig::getOptProfilesSplitPoints()); + setContextIndex(optProfileId); + TLLM_LOG_DEBUG("numTokens: %d, optProfileId: %d", numTokens, optProfileId); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return {optProfileId, inputMap, outputMap}; +} + +void RuntimeBuffers::fillIOMaps(ModelConfig const& modelConfig, WorldConfig const& worldConfig) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(runtimeBuffersFillIOMaps); + + inputMap.clear(); + outputMap.clear(); + + if (transformerBuffers) + { + transformerBuffers->getBuffers(inputMap, outputMap, modelConfig); + } + if (rnnStateBuffers) + { + rnnStateBuffers->getBuffers(inputMap); + } + + if (worldConfig.isLastPipelineParallelRank()) + { + // feed a view to TensorRT runtime so reshaping does not change logits buffer + outputMap.insert_or_assign(kLogitsTensorName, ITensor::view(logits)); + } + else + { + outputMap.insert_or_assign(kHiddenStatesOutputTensorName, hiddenStates); + } + + if (worldConfig.isFirstPipelineParallelRank()) + { + inputMap.insert_or_assign(kInputIdsTensorName, inputsIds); + } + else + { + inputMap.insert_or_assign(kHiddenStatesInputTensorName, hiddenStates); + } + + inputMap.insert_or_assign(kLastTokenIdsTensorName, lastTokenIdsDevice); + + inputMap.insert_or_assign(kHostRequestTypesTensorName, requestTypes); + // In the generation phase, we still pass context lengths. + inputMap.insert_or_assign(kContextLengthsTensorName, contextLengthsDevice); + inputMap.insert_or_assign(kHostContextLengthsTensorName, contextLengthsHost); + inputMap.insert_or_assign(kSequenceLengthsTensorName, sequenceLengthsDevice); + + if (modelConfig.useCrossAttention()) + { + encoderBuffers->insertInputTensors(inputMap); + } + if (modelConfig.usePromptTuning()) + { + auto const& promptTuningParams = promptTuningBuffers->mPromptTuningParams; + inputMap.insert_or_assign(kPromptEmbeddingTableTensorName, promptTuningParams.embeddingTable); + inputMap.insert_or_assign(kTasksTensorName, promptTuningParams.tasks); + inputMap.insert_or_assign(kPromptVocabSizeTensorName, promptTuningParams.vocabSize); + } + if (modelConfig.useMrope()) + { + + inputMap.insert_or_assign(kMRopeRotaryCosSinTensorName, mropeRotaryCosSin); + inputMap.insert_or_assign(kMRopePositionDeltasTensorName, mropePositionDeltas); + } + if (modelConfig.useLoraPlugin()) + { + loraBuffers->insertInputTensors(inputMap, loraBuffers->mLoraWeightsPointersHost, + loraBuffers->mLoraAdapterSizesHost, modelConfig, worldConfig); + } + if (modelConfig.useLanguageAdapter()) + { + inputMap.insert_or_assign("language_adapter_routings", languageAdapterRoutings); + } + + if (mMedusaBuffers) + { + mMedusaBuffers->insertInputTensors(inputMap, outputMap, worldConfig); + } + if (mLookaheadBuffers) + { + mLookaheadBuffers->insertInputTensors(inputMap, outputMap, worldConfig); + } + if (mExplicitDraftTokensBuffers) + { + mExplicitDraftTokensBuffers->insertInputTensors(inputMap, outputMap, worldConfig); + } + if (mEagleBuffers) + { + mEagleBuffers->insertInputTensors(inputMap, outputMap, worldConfig); + } + + for (auto const& outputTensor : mAdditionalOutputTensors) + { + outputMap.insert_or_assign(outputTensor.first, outputTensor.second); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/transformerBuffers.cpp b/cpp/tensorrt_llm/batch_manager/transformerBuffers.cpp new file mode 100644 index 000000000000..4f81c8926682 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/transformerBuffers.cpp @@ -0,0 +1,679 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/transformerBuffers.h" + +#include "tensorrt_llm/batch_manager/kvCacheManager.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/kernels/attentionMask.h" +#include "tensorrt_llm/kernels/contextFusedMultiHeadAttention/fmhaPackedMask.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/runtimeKernels.h" +#include "tensorrt_llm/runtime/tllmBuffers.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" +#include <cstdint> + +using namespace tensorrt_llm::runtime; +namespace tk = tensorrt_llm::kernels; + +namespace tensorrt_llm::batch_manager +{ + +TransformerBuffers::TransformerBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, + std::vector<SizeType32> const& maxAttentionWindowVec, SizeType32 maxAttentionWindow, SizeType32 sinkTokenLen, + runtime::TllmRuntime const& runtime, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig) + : maxInputLen(modelConfig.getMaxInputLen()) + , maxEncoderOutputLen(modelConfig.getMaxEncoderLen()) +{ + auto const& manager = runtime.getBufferManager(); + auto const& engine = runtime.getEngine(); + + positionIds = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + + auto const localNbAttnLayers + = modelConfig.getNbAttentionLayers(worldConfig.getPipelineParallelism(), worldConfig.getPipelineParallelRank()); + // find the index of the first attention layer in the current rank + auto const firstLayerId = modelConfig.countLowerRankLayers(runtime::ModelConfig::LayerType::kATTENTION, + worldConfig.getPipelineParallelism(), worldConfig.getPipelineParallelRank()); + + cacheIndirection + = manager.gpu(ITensor::makeShape({maxBatchSize, maxBeamWidth, maxAttentionWindow}), nvinfer1::DataType::kINT32); + + if (!modelConfig.getMaxNumTokens().has_value()) + { + TLLM_THROW("Model must configure a max number of tokens."); + } + maxNumTokens = modelConfig.getMaxNumTokens().value(); + + if (modelConfig.isKVCacheEnabled()) + { + auto const kvCacheBlockOffsetsType = engine.getTensorDataType("kv_cache_block_offsets"); + kvCacheBlockOffsetsHost = manager.emptyTensor(MemoryType::kPINNEDPOOL, kvCacheBlockOffsetsType); + kvCacheBlockOffsetsDevice = manager.emptyTensor(MemoryType::kGPU, kvCacheBlockOffsetsType); + + if (modelConfig.useCrossAttention()) + { + crossKvCacheBlockOffsetsHost = manager.emptyTensor(MemoryType::kPINNEDPOOL, kvCacheBlockOffsetsType); + crossKvCacheBlockOffsetsDevice = manager.emptyTensor(MemoryType::kGPU, kvCacheBlockOffsetsType); + crossAttentionMaskDevice = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kBOOL); + crossAttentionMaskPinnedHost = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxNumTokens, maxEncoderOutputLen}), nvinfer1::DataType::kBOOL); + crossAttentionPackedMaskDevice = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + crossAttentionCuQSeqLensDevice = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + crossAttentionPackedMaskCuMaskRowsDevice + = manager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); + + // Pinned memory for batch copy of attention masks. + // There will be paddings in the dim1, so copy it by tokens. + crossAttentionMaskCopySrcOffsets = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxNumTokens}), nvinfer1::DataType::kINT64); + crossAttentionMaskCopyDstOffsets = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxNumTokens}), nvinfer1::DataType::kINT64); + crossAttentionMaskCopySizes = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxNumTokens}), nvinfer1::DataType::kINT64); + } + } + + fillValuesAlt = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + fillValuesAltDevice = manager.gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + seqSlotsAlt = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + seqSlotsAltDevice = manager.gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + + cacheIndirBatchedCopySrcOffsets = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT64); + cacheIndirBatchedCopyDstOffsets = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT64); + cacheIndirBatchedCopySizes = tensorrt_llm::runtime::BufferManager::pinnedPool( + ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT64); + skipCrossAttnBlocks + = tensorrt_llm::runtime::BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kBOOL); + + pastKeyValueLengths = manager.emptyTensor(MemoryType::kCPU, nvinfer1::DataType::kINT32); + + maxAttentionWindows = BufferManager::cpu(ITensor::makeShape({localNbAttnLayers}), nvinfer1::DataType::kINT32); + auto* maxAttentionWindowsPtr = bufferCast<SizeType32>(*maxAttentionWindows); + auto const attentionWindowLength = maxAttentionWindowVec.size(); + for (SizeType32 i = 0; i < localNbAttnLayers; ++i) + { + maxAttentionWindowsPtr[i] = maxAttentionWindowVec[(firstLayerId + i) % attentionWindowLength]; + } + + sinkTokenLengths = BufferManager::cpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + bufferCast<SizeType32>(*sinkTokenLengths)[0] = sinkTokenLen; + + contextProgressHost = BufferManager::cpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT64); + bufferCast<int64_t>(*contextProgressHost)[0] = 0; + + if (modelConfig.useGemmAllReducePlugin() && worldConfig.isTensorParallel()) + { + nvinfer1::DataType ARType = modelConfig.getGemmAllReduceDtype(); + + auto hiddenSize = modelConfig.getHiddenSize() * worldConfig.getTensorParallelism(); + + auto tpGroup = worldConfig.getTensorParallelGroup(); + std::set<int> tpGroupSet(tpGroup.begin(), tpGroup.end()); + + auto outputDims = ITensor::makeShape({modelConfig.getMaxNumTokens().value() * hiddenSize}); + + gemmAllReduceOutput = std::make_shared<MulticastTensor>(outputDims, ARType, tpGroupSet); + } +} + +void TransformerBuffers::reshape(SizeType32 numSequences, SizeType32 numInputTokens) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + pastKeyValueLengths->reshape(ITensor::makeShape({numSequences})); + + if (kvCacheBlockOffsetsHost) + { + auto cacheBlockOffsetsShape = kvCacheBlockOffsetsHost->getShape(); + if (cacheBlockOffsetsShape.nbDims > 0) + { + cacheBlockOffsetsShape.d[1] = numSequences; + kvCacheBlockOffsetsHost->reshape(cacheBlockOffsetsShape); + kvCacheBlockOffsetsDevice->reshape(cacheBlockOffsetsShape); + } + else + { + TLLM_LOG_DEBUG("kvCacheBlockOffsets not allocated yet"); + } + } + + if (crossKvCacheBlockOffsetsHost) + { + TLLM_CHECK_WITH_INFO( + crossKvCacheBlockOffsetsDevice, "crossKvCacheBlockOffsetsDevice is empty for model with cross attention!"); + auto crossCacheBlockOffsetsShape = crossKvCacheBlockOffsetsHost->getShape(); + if (crossCacheBlockOffsetsShape.nbDims > 0) + { + crossCacheBlockOffsetsShape.d[1] = numSequences; + crossKvCacheBlockOffsetsHost->reshape(crossCacheBlockOffsetsShape); + crossKvCacheBlockOffsetsDevice->reshape(crossCacheBlockOffsetsShape); + } + else + { + TLLM_LOG_DEBUG("crossKvCacheBlockOffsets not allocated yet"); + } + } + + if (crossAttentionMaskDevice) + { + auto crossAttentionMaskShape = crossAttentionMaskDevice->getShape(); + if (crossAttentionMaskShape.nbDims > 0) + { + crossAttentionMaskShape.d[0] = numInputTokens; + crossAttentionMaskDevice->reshape(crossAttentionMaskShape); + crossAttentionMaskPinnedHost->reshape(crossAttentionMaskShape); + crossAttentionMaskCopySrcOffsets->reshape(ITensor::makeShape({numInputTokens})); + crossAttentionMaskCopyDstOffsets->reshape(ITensor::makeShape({numInputTokens})); + crossAttentionMaskCopySizes->reshape(ITensor::makeShape({numInputTokens})); + } + else + { + TLLM_LOG_DEBUG("crossAttentionMaskDevice not allocated yet"); + } + } + + if (crossAttentionPackedMaskDevice) + { + auto crossAttentionMaskPackedShape = crossAttentionPackedMaskDevice->getShape(); + if (crossAttentionMaskPackedShape.nbDims > 0) + { + crossAttentionMaskPackedShape.d[0] = numInputTokens; + crossAttentionPackedMaskDevice->reshape(crossAttentionMaskPackedShape); + } + else + { + TLLM_LOG_DEBUG("crossAttentionPackedMaskDevice not allocated yet"); + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TransformerBuffers::reshapeKvTensors(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, SizeType32 maxBlocksPerSeq, + kv_cache_manager::CacheType kvCacheType, SizeType32 numPools, BufferManager const& manager) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // allocate with max shape during init + if (kvCacheType == kv_cache_manager::CacheType::kSELF) + { + auto const cacheBlockOffsetsShape + = ITensor::makeShape({numPools, maxBatchSize * maxBeamWidth, 2, maxBlocksPerSeq}); + + kvCacheBlockOffsetsHost->reshape(cacheBlockOffsetsShape); + manager.setZero(*kvCacheBlockOffsetsHost); + + kvCacheBlockOffsetsDevice->reshape(cacheBlockOffsetsShape); + manager.setZero(*kvCacheBlockOffsetsDevice); + } + else if (kvCacheType == kv_cache_manager::CacheType::kCROSS) + { + auto const crossCacheBlockOffsetsShape + = ITensor::makeShape({numPools, maxBatchSize * maxBeamWidth, 2, maxBlocksPerSeq}); + + crossKvCacheBlockOffsetsHost->reshape(crossCacheBlockOffsetsShape); + manager.setZero(*crossKvCacheBlockOffsetsHost); + + crossKvCacheBlockOffsetsDevice->reshape(crossCacheBlockOffsetsShape); + manager.setZero(*crossKvCacheBlockOffsetsDevice); + + crossAttentionMaskDevice->reshape(ITensor::makeShape({maxNumTokens, maxEncoderOutputLen})); + manager.setZero(*crossAttentionMaskDevice); + manager.setZero(*crossAttentionMaskPinnedHost); + + // Only context attention needs this, so allocate it by shape [maxBatchSize, maxInputLen, maxEncoderOutputLen]. + auto [packedMaskM, packedMaskN] = tk::roundUpPackedMaskMNDims(maxInputLen, maxEncoderOutputLen); + crossAttentionPackedMaskDevice->reshape(ITensor::makeShape({maxBatchSize * packedMaskM, packedMaskN})); + manager.setZero(*crossAttentionPackedMaskDevice); + + crossAttentionCuQSeqLensDevice->reshape(ITensor::makeShape({maxBatchSize + 1})); + manager.setZero(*crossAttentionCuQSeqLensDevice); + + crossAttentionPackedMaskCuMaskRowsDevice->reshape(ITensor::makeShape({maxBatchSize + 1})); + manager.setZero(*crossAttentionPackedMaskCuMaskRowsDevice); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TransformerBuffers::getBuffers( + TensorMap& inputBuffers, TensorMap& outputBuffers, runtime::ModelConfig const& modelConfig) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(transformerBuffersGetBuffers); + + inputBuffers.insert_or_assign(kPositionIdsTensorName, positionIds); + inputBuffers.insert_or_assign(kHostPastKeyValueLengthsTensorName, pastKeyValueLengths); + inputBuffers.insert_or_assign(kCacheIndirectionsTensorName, cacheIndirection); + inputBuffers.insert_or_assign(kHostSinkTokenLengthTensorName, sinkTokenLengths); + + inputBuffers.insert_or_assign(kHostMaxAttentionWindowSizesTensorName, maxAttentionWindows); + inputBuffers.insert_or_assign(kKvCacheBlockOffsetsTensorName, kvCacheBlockOffsetsDevice); + inputBuffers.insert_or_assign(kHostKvCacheBlockOffsetsTensorName, kvCacheBlockOffsetsHost); + inputBuffers.insert_or_assign(kHostContextProgressTensorName, contextProgressHost); + + if (crossKvCacheBlockOffsetsHost) + { + inputBuffers.insert_or_assign(kCrossKvCacheBlockOffsetsTensorName, crossKvCacheBlockOffsetsDevice); + inputBuffers.insert_or_assign(kHostCrossKvCacheBlockOffsetsTensorName, crossKvCacheBlockOffsetsHost); + inputBuffers.insert_or_assign(kHostCrossKvCachePoolPointersTensorName, crossKvCacheBlockPoolPointers); + inputBuffers.insert_or_assign(kHostCrossKvCachePoolMappingTensorName, crossKvCacheBlockPoolMapping); + inputBuffers.insert_or_assign(kCrossAttentionMaskTensorName, crossAttentionMaskDevice); + inputBuffers.insert_or_assign(kCrossAttentionPackedMaskTensorName, crossAttentionPackedMaskDevice); + } + + if (skipCrossAttnBlocks) + { + inputBuffers.insert_or_assign(kSkipCrossAttentionBlocksTensorName, skipCrossAttnBlocks); + } + + if (modelConfig.useGemmAllReducePlugin()) + { + for (int idx = 0; idx < modelConfig.getNbAttentionLayers() * 2; ++idx) + { + // XXX (xsimmons): this is a bit hacky as it assumes + // 2x RowLinear layers per attention block. + // This will be fixed soon when I remove coupling between model + // and runtime. + auto gemmARViewUC = gemmAllReduceOutput->getTensorView(MulticastTensorView::ViewType::kUNICAST); + auto gemmARViewMC = gemmAllReduceOutput->getTensorView(MulticastTensorView::ViewType::kMULTICAST); + auto gemmARViewIpc = gemmAllReduceOutput->getTensorView(MulticastTensorView::ViewType::kIPC_LIST); + + outputBuffers.insert_or_assign("gemm_allreduce_uc_out_" + std::to_string(idx), gemmARViewUC); + outputBuffers.insert_or_assign("gemm_allreduce_mc_out_" + std::to_string(idx), gemmARViewMC); + outputBuffers.insert_or_assign("gemm_allreduce_ipc_out_" + std::to_string(idx), gemmARViewIpc); + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TransformerBuffers::copyPositionIds(runtime::TllmRuntime const& runtime, + std::vector<SizeType32> const& positionIdsHost, bool isChatGlm, TensorPtr const& decoderPositionIds) +{ + auto const& manager = runtime.getBufferManager(); + if (isChatGlm) + { + positionIds->reshape(ITensor::makeShape({2, static_cast<int>(positionIdsHost.size()) / 2})); + manager.copy(positionIdsHost.data(), *positionIds); + } + else if (decoderPositionIds == nullptr) + { + positionIds->reshape(ITensor::makeShape({static_cast<int>(positionIdsHost.size())})); + manager.copy(positionIdsHost.data(), *positionIds); + } + else + { + // concat context phase and generation phase positionIds. + auto const contextPositionIdsLen = static_cast<ITensor::DimType64>(positionIdsHost.size()); + auto const generationPositionIdsLen = ITensor::volume(decoderPositionIds->getShape()); + positionIds->reshape(ITensor::makeShape({contextPositionIdsLen + generationPositionIdsLen})); + manager.copy(positionIdsHost.data(), *ITensor::slice(positionIds, 0, contextPositionIdsLen)); + manager.copy(*decoderPositionIds, *ITensor::slice(positionIds, contextPositionIdsLen)); + } +} + +void TransformerBuffers::copyKvBlockOffsets(RequestVector const& contextRequests, RequestVector const& genRequests, + kv_cache_manager::BaseKVCacheManager const* kvCacheManager, + kv_cache_manager::BaseKVCacheManager const* crossKvCacheManager, BufferManager const& manager) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(copyKvBlockOffsets); + + auto const& cudaStream = manager.getStream(); + + SizeType32 constexpr contextBeamWidth{1}; + SizeType32 numSequences{0}; + SizeType32 maxBlockCount{0}; + SizeType32 maxCrossBlockCount{0}; + for (auto const& requests : {contextRequests, genRequests}) + { + for (auto const& llmReq : requests) + { + auto const requestId = llmReq->mRequestId; + auto const isContextRequest = llmReq->isContextInitState(); + auto const beamWidth = isContextRequest ? contextBeamWidth : llmReq->getBeamWidthByIter(); + auto const maxBeamBlockCount + = kvCacheManager->copyBlockOffsets(*kvCacheBlockOffsetsHost, numSequences, requestId); + maxBlockCount = std::max(maxBlockCount, maxBeamBlockCount); + if (crossKvCacheBlockOffsetsHost) + { + auto const maxCrossBeamBlockCount + = crossKvCacheManager->copyBlockOffsets(*crossKvCacheBlockOffsetsHost, numSequences, requestId); + maxCrossBlockCount = std::max(maxCrossBlockCount, maxCrossBeamBlockCount); + } + numSequences += beamWidth; + } + } + + // requests' block offsets collected as [totalNumSequences, 2, maxBlocksPerSeq], copy to device + auto copyOffsetsToDevice = [&cudaStream](TensorPtr& offsetsHost, TensorPtr& offsetsDevice, SizeType32 maxBlockCount) + { + // shape should be [totalNumSequences, 2, maxBlocksPerSeq] + auto const& offsetsShape = offsetsHost->getShape(); + auto const maxBlocksPerSeq = offsetsShape.d[3]; + auto const offsetsTypeSize = tensorrt_llm::common::getDTypeSize(offsetsHost->getDataType()); + auto const copyPitch = maxBlocksPerSeq * offsetsTypeSize; + auto const copyHeight = offsetsShape.d[0] * offsetsShape.d[1] * offsetsShape.d[2]; + auto const copyWidth = maxBlockCount * offsetsTypeSize; + auto* srcPtr = bufferCast<tk::KVCacheIndex>(*offsetsHost); + auto* dstPtr = bufferCast<tk::KVCacheIndex>(*offsetsDevice); + + TLLM_CUDA_CHECK(cudaMemcpy2DAsync( + dstPtr, copyPitch, srcPtr, copyPitch, copyWidth, copyHeight, cudaMemcpyHostToDevice, cudaStream.get())); + }; + + copyOffsetsToDevice(kvCacheBlockOffsetsHost, kvCacheBlockOffsetsDevice, maxBlockCount); + if (crossKvCacheBlockOffsetsHost) + { + copyOffsetsToDevice(crossKvCacheBlockOffsetsHost, crossKvCacheBlockOffsetsDevice, maxCrossBlockCount); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TransformerBuffers::copyCacheIndirection( + RequestVector const& genRequests, TensorPtr const& decoderCacheIndirectionOutput, CudaStream const& stream) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(copyCacheIndirection); + + auto const numGenerationRequests = genRequests.size(); + + auto batchedCopySrcOffsets = BufferRange<SizeType64>(*cacheIndirBatchedCopySrcOffsets); + auto batchedCopyDstOffsets = BufferRange<SizeType64>(*cacheIndirBatchedCopyDstOffsets); + auto batchedCopySizes = BufferRange<SizeType64>(*cacheIndirBatchedCopySizes); + + auto cacheIndirShape = decoderCacheIndirectionOutput->getShape(); + + // At present, all requests of a batch must have the same beam width in one generation step (or they will not + // be batched together). So, the beam width of the first request is taken here to reshape the buffer. + // Corresponding changes must be done if Diverse-Beam-Width-Search (DBWS, requests with diverse beam width in + // a batch in one generation step) is supported in the future. + auto reqBeamWidth = genRequests[0]->getBeamWidthByIter(); + + // Get size of copying from shape of `CacheIndirectionOutput` + cacheIndirShape.d[0] = 1; + cacheIndirShape.d[1] = reqBeamWidth; // Use beam width of current step rather than max beam width as dst offset + auto const copySize = static_cast<SizeType64>(ITensor::volume(cacheIndirShape)); + + std::transform(genRequests.begin(), genRequests.end(), batchedCopySrcOffsets.begin(), + [copySize](auto const& llmReq) { return llmReq->mSeqSlot.value() * copySize; }); + std::generate_n( + batchedCopyDstOffsets.begin(), numGenerationRequests, [copySize, i = 0]() mutable { return (i++) * copySize; }); + std::fill_n(batchedCopySizes.begin(), numGenerationRequests, copySize); + + auto const batchedCopySrcOffsetsSlice = ITensor::slice(cacheIndirBatchedCopySrcOffsets, 0, numGenerationRequests); + auto const batchedCopyDstOffsetsSlice = ITensor::slice(cacheIndirBatchedCopyDstOffsets, 0, numGenerationRequests); + auto const batchedCopySizesSlice = ITensor::slice(cacheIndirBatchedCopySizes, 0, numGenerationRequests); + runtime::kernels::invokeCopyBatch(*decoderCacheIndirectionOutput, *cacheIndirection, *batchedCopySrcOffsetsSlice, + *batchedCopyDstOffsetsSlice, *batchedCopySizesSlice, copySize, stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TransformerBuffers::copyCrossAttentionMasks(RequestVector const& contextRequests, RequestVector const& genRequests, + TensorPtr const& decoderContextLengthsDevice, TensorPtr const& encoderInputLengths, + SizeType32 maxDecoderContextLength, SizeType32 maxEncoderInputLengthInBatch, TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + auto const& manager = runtime.getBufferManager(); + + // Reshape the tensor to make sure the dim1 matches maxEncoderInputLengthInBatch. + auto crossAttentionMaskShape = crossAttentionMaskDevice->getShape(); + crossAttentionMaskShape.d[1] = maxEncoderInputLengthInBatch; + crossAttentionMaskDevice->reshape(crossAttentionMaskShape); + // Set crossAttentionMask to true by default if it is not provided. + manager.setMem(*crossAttentionMaskDevice, 1); + + // Check if all context requests have cross attention mask. + bool allContextCrossAttentionMaskProvided = true; + for (auto const& llmReq : contextRequests) + { + auto const& crossAttentionMaskRequest = llmReq->getCrossAttentionMask(); + if (bufferCastOrNull<bool>(crossAttentionMaskRequest) == nullptr) + { + allContextCrossAttentionMaskProvided = false; + break; + } + } + // If not all requests have cross attention mask, let us create the default ones. + auto const& stream = runtime.getStream(); + if (!allContextCrossAttentionMaskProvided) + { + TLLM_LOG_WARNING("Default padding attention mask will be used as not all requests have cross attention mask."); + tk::AttentionMaskParams<bool> attentionMaskParams; + memset((void*) &attentionMaskParams, 0, sizeof(attentionMaskParams)); + // Set parameters. + attentionMaskParams.mask = bufferCastOrNull<bool>(crossAttentionMaskDevice); + attentionMaskParams.cuQSeqLens = bufferCastOrNull<SizeType32>(crossAttentionCuQSeqLensDevice); + attentionMaskParams.actualQSeqLens = bufferCastOrNull<SizeType32>(decoderContextLengthsDevice); + attentionMaskParams.actualKvSeqLens = bufferCastOrNull<SizeType32>(encoderInputLengths); + attentionMaskParams.attentionMaskType = tk::AttentionMaskType::PADDING; + attentionMaskParams.batchSize = static_cast<SizeType32>(contextRequests.size()); + attentionMaskParams.maxQSeqLen = maxDecoderContextLength; + attentionMaskParams.maxKvSeqLen = maxEncoderInputLengthInBatch; + // Launch the kernel. + tk::invokeBuildAttentionMask(attentionMaskParams, stream.get()); + sync_check_cuda_error(stream.get()); + } + // Use the first request's cross attention mask tensor's pointer address as the primary source pointer. + auto const& attentionMaskSrc = !contextRequests.empty() ? contextRequests[0]->getCrossAttentionMask() + : genRequests[0]->getCrossAttentionMask(); + bool const* primarySrcPtr = bufferCastOrNull<bool>(attentionMaskSrc); + + // Pinned-memory buffer preparation for batch copy. + auto batchedCopySrcOffsets = BufferRange<SizeType64>(*crossAttentionMaskCopySrcOffsets); + auto batchedCopyDstOffsets = BufferRange<SizeType64>(*crossAttentionMaskCopyDstOffsets); + auto batchedCopySizes = BufferRange<SizeType64>(*crossAttentionMaskCopySizes); + // Requests with cross-attention-mask don't need to copy. + manager.setZero(*crossAttentionMaskCopySizes); + sync_check_cuda_error(stream.get()); + + SizeType32 numTokens = 0; + SizeType32 numCopiedTokens = 0; + bool* pinnedMemPtr = bufferCastOrNull<bool>(crossAttentionMaskPinnedHost); + for (auto const& llmReq : contextRequests) + { + auto const& crossAttentionMaskRequest = llmReq->getCrossAttentionMask(); + auto const position = llmReq->getContextCurrentPosition(); + auto const size = llmReq->getContextChunkSize(); + if (bufferCastOrNull<bool>(crossAttentionMaskRequest) != nullptr) + { + auto memType = crossAttentionMaskRequest->getMemoryType(); + auto const crossAttentionMaskRequestDim0 + = static_cast<SizeType64>(crossAttentionMaskRequest->getShape().d[0]); + auto const crossAttentionMaskRequestDim1 + = static_cast<SizeType64>(crossAttentionMaskRequest->getShape().d[1]); + TLLM_LOG_DEBUG("copyCrossAttentionMasks (shape [%d, %d]) from contextRequests position %d chunkSize %d", + crossAttentionMaskRequestDim0, crossAttentionMaskRequestDim1, position, size); + if ((position + size - 1) >= crossAttentionMaskRequestDim0) + { + TLLM_LOG_WARNING( + "The provided crossAttentionMask input is not complete for context phases, the last row " + "will be " + "used by default."); + } + // copy it to pinned memory if it is a cpu tensor. + if (memType == MemoryType::kCPU) + { + TLLM_LOG_DEBUG("CrossAttentionMask tensor is on CPU."); + auto const copiedPosition + = std::min(crossAttentionMaskRequestDim0 - 1, static_cast<SizeType64>(position)); + auto const copiedSize + = std::min(crossAttentionMaskRequestDim0 - copiedPosition, static_cast<SizeType64>(size)); + SizeType64 inputMaskOffset = (copiedPosition * crossAttentionMaskRequestDim1); + SizeType64 inputMaskSize = (copiedSize * crossAttentionMaskRequestDim1); + std::memcpy( + pinnedMemPtr, bufferCastOrNull<bool>(crossAttentionMaskRequest) + inputMaskOffset, inputMaskSize); + pinnedMemPtr += inputMaskSize; + for (SizeType32 tokenId = position; tokenId < position + size; tokenId++) + { + SizeType64 tokenIdInPinnedMem + = std::min(copiedSize - 1, static_cast<SizeType64>(tokenId - position)); + batchedCopySrcOffsets.begin()[numCopiedTokens] + = (pinnedMemPtr - primarySrcPtr) + tokenIdInPinnedMem * crossAttentionMaskRequestDim1; + batchedCopyDstOffsets.begin()[numCopiedTokens] + = numTokens * static_cast<SizeType64>(maxEncoderInputLengthInBatch); + batchedCopySizes.begin()[numCopiedTokens] = crossAttentionMaskRequestDim1; + numCopiedTokens++; + numTokens++; + } + } + else + { + TLLM_LOG_DEBUG("CrossAttentionMask tensor is on GPU."); + for (SizeType32 tokenId = position; tokenId < position + size; tokenId++) + { + batchedCopySrcOffsets.begin()[numCopiedTokens] + = static_cast<SizeType64>(bufferCastOrNull<bool>(crossAttentionMaskRequest) - primarySrcPtr) + + std::min(crossAttentionMaskRequestDim0 - 1, static_cast<SizeType64>(tokenId)) + * crossAttentionMaskRequestDim1; + batchedCopyDstOffsets.begin()[numCopiedTokens] + = numTokens * static_cast<SizeType64>(maxEncoderInputLengthInBatch); + batchedCopySizes.begin()[numCopiedTokens] = crossAttentionMaskRequestDim1; + numCopiedTokens++; + numTokens++; + } + } + } + else + { + numTokens += size; + TLLM_LOG_WARNING( + "CrossAttentionMask is not provided for the request. Default padding attention mask will be " + "created."); + } + } + sync_check_cuda_error(stream.get()); + + for (auto const& llmReq : genRequests) + { + auto const promptLen = llmReq->mPromptLen; + auto const decodingIter = llmReq->getDecodingIter(); + auto const& crossAttentionMaskRequest = llmReq->getCrossAttentionMask(); + if (bufferCastOrNull<bool>(crossAttentionMaskRequest) != nullptr) + { + auto const memType = crossAttentionMaskRequest->getMemoryType(); + auto const crossAttentionMaskRequestDim0 + = static_cast<SizeType64>(crossAttentionMaskRequest->getShape().d[0]); + auto const crossAttentionMaskRequestDim1 + = static_cast<SizeType64>(crossAttentionMaskRequest->getShape().d[1]); + TLLM_LOG_DEBUG("copyCrossAttentionMasks (shape [%d, %d]) from genRequests decodingIter %d", + crossAttentionMaskRequestDim0, crossAttentionMaskRequestDim1, decodingIter); + if (promptLen + decodingIter - 1 >= crossAttentionMaskRequestDim0) + { + TLLM_LOG_WARNING( + "The provided crossAttentionMask input is not complete for generation phases, the last row " + "will be " + "used by default."); + } + // copy it to pinned memory if it is a cpu tensor. + if (memType == MemoryType::kCPU) + { + TLLM_LOG_DEBUG("CrossAttentionMask tensor is on CPU."); + SizeType64 copiedPosition = std::min( + crossAttentionMaskRequestDim0 - 1, static_cast<SizeType64>(promptLen + decodingIter - 1)); + SizeType64 inputMaskOffset = (copiedPosition * crossAttentionMaskRequestDim1); + SizeType64 inputMaskSize = crossAttentionMaskRequestDim1; + std::memcpy( + pinnedMemPtr, bufferCastOrNull<bool>(crossAttentionMaskRequest) + inputMaskOffset, inputMaskSize); + pinnedMemPtr += inputMaskSize; + batchedCopySrcOffsets.begin()[numCopiedTokens] = static_cast<SizeType64>(pinnedMemPtr - primarySrcPtr); + batchedCopyDstOffsets.begin()[numCopiedTokens] + = numTokens * static_cast<SizeType64>(maxEncoderInputLengthInBatch); + batchedCopySizes.begin()[numCopiedTokens] = crossAttentionMaskRequestDim1; + } + else + { + TLLM_LOG_DEBUG("CrossAttentionMask tensor is on GPU."); + batchedCopySrcOffsets.begin()[numCopiedTokens] + = static_cast<SizeType64>(bufferCastOrNull<bool>(crossAttentionMaskRequest) - primarySrcPtr) + + std::min(crossAttentionMaskRequestDim0 - 1, static_cast<SizeType64>(promptLen + decodingIter - 1)) + * crossAttentionMaskRequestDim1; + batchedCopyDstOffsets.begin()[numCopiedTokens] + = numTokens * static_cast<SizeType64>(maxEncoderInputLengthInBatch); + batchedCopySizes.begin()[numCopiedTokens] = crossAttentionMaskRequestDim1; + } + numCopiedTokens++; + numTokens++; + } + else + { + numTokens++; + TLLM_LOG_WARNING( + "CrossAttentionMask is not provided for the generation request. Full valid attentionMask will " + "be used " + "by default."); + } + } + sync_check_cuda_error(stream.get()); + + // Copy all requests' attention mask in one kernel. + if (attentionMaskSrc != nullptr) + { + crossAttentionMaskCopySrcOffsets->reshape(ITensor::makeShape({numCopiedTokens})); + crossAttentionMaskCopyDstOffsets->reshape(ITensor::makeShape({numCopiedTokens})); + crossAttentionMaskCopySizes->reshape(ITensor::makeShape({numCopiedTokens})); + runtime::kernels::invokeCopyBatch(*attentionMaskSrc, *crossAttentionMaskDevice, + *crossAttentionMaskCopySrcOffsets, *crossAttentionMaskCopyDstOffsets, *crossAttentionMaskCopySizes, + maxEncoderInputLengthInBatch, stream); + } + sync_check_cuda_error(stream.get()); + + // The packed mask is only needed by context requests now. + if (!contextRequests.empty()) + { + // Set the parameters for creating packed mask for context FMHA. + tk::PackedMaskParams<bool> maskParams{}; + maskParams.maskInput = bufferCastOrNull<bool>(crossAttentionMaskDevice); + maskParams.cuQSeqLens = bufferCastOrNull<SizeType32>(crossAttentionCuQSeqLensDevice); + maskParams.packedMask = bufferCastOrNull<uint32_t>(crossAttentionPackedMaskDevice); + maskParams.cuMaskRows = bufferCastOrNull<SizeType32>(crossAttentionPackedMaskCuMaskRowsDevice); + maskParams.actualQSeqLens = bufferCastOrNull<SizeType32>(decoderContextLengthsDevice); + maskParams.actualKvSeqLens = bufferCastOrNull<SizeType32>(encoderInputLengths); + maskParams.batchSize = contextRequests.size(); + maskParams.maxQSeqLen = maxDecoderContextLength; + maskParams.maxKvSeqLen = maxEncoderInputLengthInBatch; + maskParams.attentionMaskType = tk::ContextAttentionMaskType::CUSTOM_MASK; + maskParams.validPosVal = true; + + // Launch the pack mask kernel. + tk::invokeBuildPackedMask(maskParams, stream.get()); + sync_check_cuda_error(stream.get()); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TransformerBuffers::copySkipCrossAttnBlocks(bool const& _skipCrossAttnBlocks, runtime::TllmRuntime const& runtime) +{ + auto const& manager = runtime.getBufferManager(); + manager.copy(&_skipCrossAttnBlocks, *skipCrossAttnBlocks); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/trtEncoderModel.cpp b/cpp/tensorrt_llm/batch_manager/trtEncoderModel.cpp new file mode 100644 index 000000000000..0d7dbfde42e6 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/trtEncoderModel.cpp @@ -0,0 +1,618 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "trtEncoderModel.h" +#include "encoderBuffers.h" +#include "tensorrt_llm/batch_manager/capacityScheduler.h" +#include "tensorrt_llm/batch_manager/microBatchScheduler.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" +#include "tensorrt_llm/runtime/utils/runtimeUtils.h" + +#include <algorithm> +#include <cstddef> +#include <vector> + +using namespace tensorrt_llm::runtime; +using namespace tensorrt_llm::mpi; + +namespace tensorrt_llm::batch_manager +{ + +TrtEncoderModel::TrtEncoderModel(runtime::ModelConfig const& modelConfig, WorldConfig const& worldConfig, + runtime::RawEngine const& rawEngine, std::shared_ptr<nvinfer1::ILogger> logger, + executor::ExecutorConfig const& executorConfig) + : TrtGptModel(modelConfig, worldConfig, executorConfig) + , mModelConfig{modelConfig} + , mWorldConfig{worldConfig} + , mDevice{runtime::utils::initDevice(worldConfig)} + , mLogger{logger ? std::move(logger) : std::make_shared<TllmLogger>()} + , mRuntime{std::make_shared<TllmRuntime>( + rawEngine, mLogger.get(), executorConfig.getUseGpuDirectStorage(), executorConfig.getGpuWeightsPercent())} + , mNumMicroBatches{1} + , mNumBuffers{mNumMicroBatches} + , mCopyBufferManager{std::make_shared<CudaStream>()} +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK_WITH_INFO( + !mWorldConfig.isPipelineParallel(), "Pipeline parallelism is currently not supported for encoder models."); + + createRuntimeContexts(); + + createBuffers(); + + if (mWorldConfig.isPipelineParallel()) + { + auto const& commSession = COMM_SESSION; + mMpiCommPipelinePara = std::make_shared<tensorrt_llm::mpi::MpiComm>( + commSession.split(mWorldConfig.getTensorParallelRank(), mWorldConfig.getPipelineParallelRank())); + } + + mMicroBatchScheduledRequests.resize(mNumMicroBatches); + // mEncoderWaitEvents.resize(mNumMicroBatches); + + // set noScheduleUntilState to LlmRequestState::kENCODER_INIT for encoder model + // when null kv cache manager is given, request scheduler will use MaxRequests as capacity scheduler, i.e. no + // handling of maximizing utilization or pause/evict + // TODO: finer control on encoder requests scheduling + mCapacityScheduler = std::make_unique<tensorrt_llm::batch_manager::CapacityScheduler>( + getMaxBatchSize() * mNumMicroBatches, executorConfig.getSchedulerConfig().getCapacitySchedulerPolicy(), + /*hasKvCacheManager=*/false, /*twoStepsLookAhead=*/false, + /*noScheduleUntilState=*/LlmRequestState::kENCODER_INIT, + /*noScheduleAfterState=*/LlmRequestState::kCONTEXT_INIT, + /*enablePrefixAwareScheduling=*/executorConfig.getSchedulerConfig().getEnablePrefixAwareScheduling()); + + mMicroBatchScheduler = std::make_unique<tensorrt_llm::batch_manager::MicroBatchScheduler>( + std::nullopt, mModelConfig.getMaxInputLen(), LlmRequestState::kENCODER_INIT, LlmRequestState::kCONTEXT_INIT); + + mHiddenSize = modelConfig.getHiddenSize(); + + mMaxInputLen = mModelConfig.getMaxInputLen(); + TLLM_LOG_INFO("TRTEncoderModel mMaxInputLen: reset to %d from build config.", mMaxInputLen); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +BufferManager const& TrtEncoderModel::getBufferManager() const +{ + return mRuntime->getBufferManager(); +} + +BufferManager::CudaStreamPtr TrtEncoderModel::getRuntimeStreamPtr() const +{ + return mRuntime->getStreamPtr(); +} + +nvinfer1::DataType TrtEncoderModel::getTensorDataType(std::string const& name) const +{ + auto const& engine = mRuntime->getEngine(); + return engine.getTensorDataType(name.c_str()); +} + +nvinfer1::Dims TrtEncoderModel::getTensorShape(std::string const& name) const +{ + auto const& engine = mRuntime->getEngine(); + return engine.getTensorShape(name.c_str()); +} + +void TrtEncoderModel::getCurrentIterationStats(executor::IterationStats& stats) const +{ + stats.iter = mIterCounter; +} + +void TrtEncoderModel::getCurrentRequestStats(executor::RequestStatsPerIteration& stats) const +{ + stats.iter = mIterCounter; +} + +executor::DebugTensorsPerIteration TrtEncoderModel::getCurrentDebugTensors() const +{ + executor::DebugTensorsPerIteration debugTensors; + debugTensors.iter = mIterCounter; + + TLLM_LOG_WARNING("TrtEncoderModel doesn't support getting debug tensors."); + + return debugTensors; +} + +void TrtEncoderModel::setLayerProfiler() +{ + TLLM_CHECK(mRuntime); + mRuntime->setLayerProfiler(); +} + +std::string TrtEncoderModel::getLayerProfileInfo() const +{ + TLLM_CHECK(mRuntime); + return mRuntime->getLayerProfileInfo(); +} + +void TrtEncoderModel::createRuntimeContexts() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + mRuntime->clearContexts(); + auto const numProfiles = mRuntime->getNbProfiles(); + TLLM_CHECK_WITH_INFO(numProfiles == 1, "Encoder only expects one optimization profile"); + for (auto i = 0; i < numProfiles; ++i) + { + mRuntime->addContext(i); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::executeContext(SizeType32 runtimeContextId) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(executeContext); + auto enqueueSuccessful = mRuntime->executeContext(runtimeContextId); + if (!enqueueSuccessful) + { + throw std::runtime_error("Executing TRT engine failed!"); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::createBuffers() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + for (SizeType32 i = 0; i < mNumBuffers; ++i) + { + mBuffers.emplace_back( + std::make_shared<EncoderBuffers>(getMaxBatchSize(), mModelConfig, mWorldConfig, *mRuntime)); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::executeBatch(ScheduledRequests const& scheduledRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(executeBatch); + + // encoder model only have one optimization profile for now, so no optimization profile switch + SizeType32 optProfileIndex = 0; + auto const bufferId = getBufferId(); + if (!scheduledRequests.contextRequests.empty()) + { + // engine I/O + auto [inputMap, outputMap] + = mBuffers[bufferId]->prepareIO(scheduledRequests.contextRequests, mModelConfig, mWorldConfig, *mRuntime); + mRuntime->setInputTensors(optProfileIndex, inputMap); + mRuntime->setOutputTensors(optProfileIndex, outputMap); + + // engine run + executeContext(optProfileIndex); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::rearrangeOutputs(ScheduledRequests const& scheduledRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(rearrangeOutputs); + + auto const bufferId = getBufferId(); + if (!scheduledRequests.contextRequests.empty()) + { + mBuffers[bufferId]->rearrangeOutputs(scheduledRequests.contextRequests, mModelConfig, mWorldConfig, *mRuntime); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::forwardSync() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE_WITH_NAME(range, "TrtEncoderModel::forwardSync"); + + auto const device = mWorldConfig.getDevice(); + TLLM_CUDA_CHECK(cudaSetDevice(device)); + + auto& currRequests = mMicroBatchScheduledRequests.at(mMicroBatchId); + // auto& encoderWaitEvent = mEncoderWaitEvents.at(mMicroBatchId); + + if (!currRequests.empty()) + { + if (!mWorldConfig.isPipelineParallel() || !mWorldConfig.isLastPipelineParallelRank()) + { + // TLLM_CHECK_WITH_INFO(mEncStepAsyncSndHdl.get() == nullptr, "encoderSync handle must be nullptr."); + // // Wait for encoding for requests in flight for the current micro batch + // mEncStepAsyncSndHdl = encoderSync(currRequests, encoderWaitEvent); + } + else + { + } + + NVTX3_SCOPED_RANGE(pauseFlaggedCurrRequests); + for (auto const& requests : {currRequests.contextRequests}) + { + for (auto const& llmReq : requests) + { + auto const reqId = llmReq->mRequestId; + mInflightReqIds.erase(reqId); + TLLM_LOG_DEBUG("request ID %u removed from ENCODER inflight set", reqId); + + // If a request in encoder phase had been flagged to be paused, pause it right away + if (mReqIdsToPause.find(reqId) != mReqIdsToPause.end()) + { + terminateRequest(llmReq, true); + mReqIdsToPause.erase(reqId); + } + } + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::forwardAsync(RequestList const& activeRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE_WITH_NAME(range, "TrtEncoderModel::ForwardAsync"); + auto const device = mWorldConfig.getDevice(); + TLLM_CUDA_CHECK(cudaSetDevice(device)); + + try + { + auto& currRequests = mMicroBatchScheduledRequests.at(mMicroBatchId); + // auto& encoderWaitEvent = mEncoderWaitEvents.at(mMicroBatchId); + + // Get a new set of requests for encoder + // The scheduler will not include any requests that are already in flight for encoder models + // TODO: add pause handling logic + TLLM_LOG_DEBUG("Running ENCODER request scheduler"); + + auto [fittingRequests, fittingDisaggeGenInitReuqests, requestsToPause] = (*mCapacityScheduler)(activeRequests); + + TLLM_CHECK_WITH_INFO( + fittingDisaggeGenInitReuqests.empty(), "Disaggregated servering is not support by encoder model."); + + std::tie(currRequests.contextRequests, std::ignore) = (*mMicroBatchScheduler)( + fittingRequests, mInflightReqIds, getMaxBatchSize(), mModelConfig.getMaxNumTokens()); + + { + NVTX3_SCOPED_RANGE(pauseRequestsFlaggedByScheduler); + // Loop over requests flagged to be paused, and if not in flight pause it right away + for (auto const& llmReq : requestsToPause) + { + auto const reqId = llmReq->mRequestId; + if (mInflightReqIds.find(reqId) == mInflightReqIds.end()) + { + // Not in flight, can terminate right away + terminateRequest(llmReq, true); + } + else + { + // In flight, add to set for pausing later + mReqIdsToPause.insert(reqId); + } + } + } + + TLLM_CHECK(currRequests.size() <= static_cast<size_t>(getMaxBatchSize())); + + if (!currRequests.empty()) + { + TLLM_LOG_DEBUG("Running ENCODER model with batch size: %u", currRequests.size()); + { + NVTX3_SCOPED_RANGE(updateInflightReqIds); + // Add to set of requests in flight + for (auto const& requests : {currRequests.contextRequests}) + { + for (auto const& llmReq : requests) + { + TLLM_LOG_DEBUG("request ID %u added to ENCODER inflight set", llmReq->mRequestId); + mInflightReqIds.insert(llmReq->mRequestId); + } + } + } + + executeBatch(currRequests); + + sync_check_cuda_error(mRuntime->getStream().get()); + + rearrangeOutputs(currRequests); + + sync_check_cuda_error(mRuntime->getStream().get()); + + // encoderWaitEvent = encoderStepAsync(currRequests); + + for (auto const& requests : {currRequests.contextRequests}) + { + for (auto const& llmReq : requests) + { + if (llmReq->isEncoderInitState()) + { + llmReq->setState(LlmRequestState::kCONTEXT_INIT); + TLLM_LOG_DEBUG("request ID: %u finishes encoder phase", llmReq->mRequestId); + } + } + } + } + + // TODO: PP handling + if (!currRequests.empty()) + { + if (mWorldConfig.isPipelineParallel() && mWorldConfig.isLastPipelineParallelRank()) + { + // TLLM_CHECK_WITH_INFO(mEncStepAsyncSndHdl.get() == nullptr, "decoderSync handle must be nullptr."); + // Wait for encoding for requests in flight for the current micro batch + // mEncStepAsyncSndHdl = encoderSync(currRequests, encoderWaitEvent); + } + } + + // Update the micro batch ID + mMicroBatchId = (mMicroBatchId + 1) % mNumMicroBatches; + } + // In case of error, we need to free the batch slot associated with those requests + catch (std::exception const& e) + { + for (auto const& llmReq : activeRequests) + { + terminateRequest(llmReq); + } + throw; + } + + ++mIterCounter; + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::terminateRequest(std::shared_ptr<LlmRequest> const& llmReq, bool pause) +{ + // For encoder-only models, just change req state here. might need to do more when using an asynced forward + // For enc-dec models, only remove cross kv cache after decoder + // genenration has finished + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + if (llmReq->isEncoderInitState()) + { + llmReq->setState(LlmRequestState::kCONTEXT_INIT); + } + else + { + TLLM_LOG_DEBUG("Non-encoder request terminated in encoder model: id %lu", llmReq->mRequestId); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::terminateRequestSync( + std::shared_ptr<LlmRequest> const& llmReq, executor::FinishReason finishReason) +{ + terminateRequest(llmReq, false); + llmReq->finishByReason(finishReason); + llmReq->clearGeneratedTokens(); +} + +void TrtEncoderModel::fillEncoderOutputSync(RequestVector const& requestList, TensorMap outputTensors) +{ + auto const totalTokensNb = outputTensors["encoder_output"]->getShape().d[0]; + auto const encoderOutputDtype = mRuntime->getEngine().getTensorDataType("encoder_output"); + SizeType32 const bytesPerValue = (encoderOutputDtype == nvinfer1::DataType::kFLOAT) ? 4 : 2; + std::vector<std::byte> encoderOutputHost( + totalTokensNb * mHiddenSize * bytesPerValue * mWorldConfig.getTensorParallelism()); + TLLM_CHECK_WITH_INFO(encoderOutputHost.size() > 0, "Encoder output size is 0!"); + getBufferManager().copy(*(outputTensors["encoder_output"]), reinterpret_cast<void*>(encoderOutputHost.data())); + getBufferManager().getStream().synchronize(); // TODO: change engine call to async to improve perf. Also + // need to store output buffers, cuda events, etc. + + auto encoderOutputHostPtr = encoderOutputHost.data(); + for (auto const& llmReq : requestList) + { + SizeType32 const seqLen = llmReq->getEncoderOutputLen(); + TensorPtr currentEncoderOutput + = mCopyBufferManager.copyFrom(reinterpret_cast<half const*>(encoderOutputHostPtr), + ITensor::makeShape({seqLen, mHiddenSize * mWorldConfig.getTensorParallelism()}), MemoryType::kCPU); + llmReq->setEncoderOutputHost(currentEncoderOutput); + encoderOutputHostPtr += seqLen * mHiddenSize * bytesPerValue * mWorldConfig.getTensorParallelism(); + + if (llmReq->isEncoderInitState()) + { + llmReq->setState(LlmRequestState::kCONTEXT_INIT); + } + else + { + TLLM_LOG_DEBUG("Non-encoder request terminated in encoder model: id %lu", llmReq->mRequestId); + } + } +} + +void TrtEncoderModel::executeBatch(RequestVector const& requestList) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(executeBatch); + + auto const modelName = mModelConfig.getModelName(); + TLLM_CHECK_WITH_INFO(modelName == "EncoderModel" || modelName == "WhisperEncoder", "Model not supported."); + TensorMap inputTensors; + TensorMap outputTensors; + TensorPtr rankOutput; + + std::vector<TokenIdType> inputIdsHost; + std::vector<SizeType32> positionIdsHost; + SizeType32 totalOutputLength = 0; + SizeType32 totalInputLength = 0; + std::vector<SizeType32> inputLengthsHost; + std::vector<std::byte> inputFeaturesHost; + + inputLengthsHost.reserve(requestList.size()); + SizeType32 maxInputLengthHost = 0; + + for (auto const& llmReq : requestList) + { + SizeType32 length = 0; + if (mModelConfig.getModelName() == "EncoderModel") + { + auto const& reqTokens = *(llmReq->getEncoderTokens().value()); + length = reqTokens.size(); + + inputIdsHost.insert(inputIdsHost.end(), reqTokens.begin(), reqTokens.end()); + maxInputLengthHost = std::max(maxInputLengthHost, static_cast<SizeType32>(length)); + } + else if (mModelConfig.getModelName() == "WhisperEncoder") + { + auto const& reqFeatures = llmReq->getEncoderInputFeatures(); // [length, featureDim] + length = reqFeatures->getShape().d[0]; + + auto const curFeatureBytes = reqFeatures->getSizeInBytes(); + auto const srcPtr = reinterpret_cast<std::byte*>(reqFeatures->data()); + inputFeaturesHost.insert(inputFeaturesHost.end(), srcPtr, srcPtr + curFeatureBytes); + } + positionIdsHost.reserve(positionIdsHost.size() + length); + auto const newReqPosBegin = positionIdsHost.end(); + positionIdsHost.resize(positionIdsHost.size() + length); + std::iota(newReqPosBegin, positionIdsHost.end(), 0); + + totalOutputLength += llmReq->getEncoderOutputLen(); + totalInputLength += length; + inputLengthsHost.push_back(length); + } + + TensorPtr hiddenStatesInput; + TensorPtr inputLengths = getBufferManager().copyFrom( + inputLengthsHost, ITensor::makeShape({static_cast<SizeType32>(inputLengthsHost.size())}), MemoryType::kGPU); + inputTensors.emplace("input_lengths", inputLengths); + + if (mModelConfig.getModelName() == "EncoderModel") + { + // use shape of maxInputLength to indicates max length, content is not important + TensorPtr maxInputLength + = getBufferManager().gpu(ITensor::makeShape({maxInputLengthHost}), nvinfer1::DataType::kINT32); + inputTensors.emplace("max_input_length", maxInputLength); + } + + // engine outputs + rankOutput = getBufferManager().gpu( + ITensor::makeShape({totalOutputLength, mHiddenSize * mWorldConfig.getTensorParallelism()}), + mModelConfig.getDataType()); + + if (mWorldConfig.isFirstPipelineParallelRank()) + { + if (mModelConfig.getModelName() == "EncoderModel") + { + // Engine inputs + TensorPtr inputIds + = getBufferManager().copyFrom(inputIdsHost, ITensor::makeShape({totalInputLength}), MemoryType::kGPU); + TensorPtr positionIds = getBufferManager().copyFrom( + positionIdsHost, ITensor::makeShape({totalInputLength}), MemoryType::kGPU); + inputTensors.emplace("input_ids", inputIds); + inputTensors.emplace("position_ids", positionIds); + } + else if (mModelConfig.getModelName() == "WhisperEncoder") + { + auto inputFeaturesHostPtr = inputFeaturesHost.data(); + auto const featureDim = requestList.front()->getEncoderInputFeatures()->getShape().d[1]; + auto const dtype = requestList.front()->getEncoderInputFeatures()->getDataType(); + TensorPtr inputFeatures = getBufferManager().gpu(ITensor::makeShape({totalInputLength, featureDim}), dtype); + getBufferManager().copy( + reinterpret_cast<void const*>(inputFeaturesHostPtr), *inputFeatures, runtime::MemoryType::kCPU); + TensorPtr positionIds = getBufferManager().copyFrom( + positionIdsHost, ITensor::makeShape({totalOutputLength}), MemoryType::kGPU); + inputTensors.emplace("input_features", inputFeatures); + inputTensors.emplace("position_ids", positionIds); + } + } + else + { + SizeType32 length = mModelConfig.getModelName() == "WhisperEncoder" ? totalOutputLength : totalInputLength; + hiddenStatesInput + = getBufferManager().gpu(ITensor::makeShape({length, mHiddenSize * mWorldConfig.getTensorParallelism()}), + mModelConfig.getDataType()); + + inputTensors.emplace("hidden_states_input", hiddenStatesInput); + } + + auto const outputName = mWorldConfig.isLastPipelineParallelRank() ? "encoder_output" : "hidden_states_output"; + outputTensors.emplace(outputName, rankOutput); + + // Set input / output tensors to context, encoder model only have one context + mRuntime->setInputTensors(0, inputTensors); + mRuntime->setOutputTensors(0, outputTensors); + + executeContext(0); + + // copy encoder output to llmRequest, if last PP rank + // dispatch result to each llmReq, only needed by the last PP rank + // TODO: more dtypes support + if (mWorldConfig.isLastPipelineParallelRank()) + { + fillEncoderOutputSync(requestList, outputTensors); + } + else + { + getBufferManager().getStream().synchronize(); + } + + // Update the micro batch ID for next microbatches + mMicroBatchId = (mMicroBatchId + 1) % mWorldConfig.getPipelineParallelism(); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::forward(RequestVector& activeRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + auto const device = mWorldConfig.getDevice(); + TLLM_CUDA_CHECK(cudaSetDevice(device)); + + try + { + if (activeRequests.empty()) + { + return; + } + + executeBatch(activeRequests); + } + catch (std::exception const& e) + { + for (auto& req : activeRequests) + { + terminateRequest(req); + } + throw; + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtEncoderModel::setLogitsPostProcessorBatched( + std::optional<LogitsPostProcessorBatched> logitsPostProcessorBatched) +{ + TLLM_CHECK_WITH_INFO(!logitsPostProcessorBatched.has_value(), "TrtEncoderModel does not use logits processor."); +} + +void TrtEncoderModel::setReplicateLogitsPostProcessor(bool replicateLogitsPostProcessor) +{ + TLLM_THROW("TrtEncoderModel does not use logits processor."); +} + +bool TrtEncoderModel::getReplicateLogitsPostProcessor() const +{ + TLLM_THROW("TrtEncoderModel does not use logits processor."); +} + +TrtEncoderModel::~TrtEncoderModel() = default; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/trtEncoderModel.h b/cpp/tensorrt_llm/batch_manager/trtEncoderModel.h new file mode 100644 index 000000000000..31f7d3d0c89b --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/trtEncoderModel.h @@ -0,0 +1,205 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "trtGptModel.h" + +#include <NvInferRuntime.h> + +namespace tensorrt_llm::runtime +{ +class TllmRuntime; +class NcclCommunicator; +} // namespace tensorrt_llm::runtime + +namespace tensorrt_llm::batch_manager +{ +class CapacityScheduler; +class MicroBatchScheduler; +class EncoderBuffers; + +class TrtEncoderModel : public TrtGptModel +{ +public: + using SizeType32 = tensorrt_llm::runtime::SizeType32; + using TokenIdType = tensorrt_llm::runtime::TokenIdType; + using BufferManager = tensorrt_llm::runtime::BufferManager; + using TensorMap = runtime::StringPtrMap<runtime::ITensor>; + using TensorPtr = runtime::ITensor::SharedPtr; + + TrtEncoderModel(runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + runtime::RawEngine const& rawEngine, std::shared_ptr<nvinfer1::ILogger> logger, + executor::ExecutorConfig const& executorConfig); + + ~TrtEncoderModel() override; + + void terminateRequest(std::shared_ptr<LlmRequest> const& llmRequest, bool pause = false) override; + void terminateRequestSync( + std::shared_ptr<LlmRequest> const& llmRequest, executor::FinishReason finishReason) override; + + void forward(RequestVector& activeRequests); + + void forwardSync() override; + + void forwardAsync(RequestList const& activeRequests) override; + + [[nodiscard]] runtime::BufferManager const& getBufferManager() const override; + [[nodiscard]] runtime::BufferManager::CudaStreamPtr getRuntimeStreamPtr() const override; + + runtime::ModelConfig const& getModelConfig() const override + { + return mModelConfig; + } + + [[nodiscard]] bool getGatherGenerationLogits() const override + { + return getModelConfig().computeGenerationLogits(); + } + + runtime::WorldConfig const& getWorldConfig() const override + { + return mWorldConfig; + } + + [[nodiscard]] SizeType32 getHiddenSize() const override + { + return mHiddenSize; + } + + [[nodiscard]] SizeType32 getMaxInputLen() const override + { + return mMaxInputLen; + } + + [[nodiscard]] SizeType32 getNumMicroBatches() const override + { + return mNumMicroBatches; + } + + [[nodiscard]] nvinfer1::DataType getLogitDataType() const override + { + return getModelConfig().getDataType(); + } + + nvinfer1::DataType getTensorDataType(std::string const& name) const override; + nvinfer1::Dims getTensorShape(std::string const& name) const override; + + [[nodiscard]] TrtGptModelType getModelType() const override + { + throw std::runtime_error("TrtEncoderModel does not have model type."); // FIXME: + } + + [[nodiscard]] executor::IterationType getIterCounter() const noexcept override + { + return mIterCounter; + } + + void updatePeftCache(std::shared_ptr<LlmRequest> const& /*llmRequest*/) override + { + throw std::runtime_error("TrtEncoderModel does not have Peft Cache."); + } + + void getCurrentIterationStats(executor::IterationStats& stats) const override; + void getCurrentRequestStats(executor::RequestStatsPerIteration& stats) const override; + [[nodiscard]] executor::DebugTensorsPerIteration getCurrentDebugTensors() const override; + + void setLayerProfiler() override; + std::string getLayerProfileInfo() const override; + + void setLogitsPostProcessorBatched(std::optional<LogitsPostProcessorBatched> logitsPostProcessorBatched) override; + void setReplicateLogitsPostProcessor(bool replicateLogitsPostProcessor) override; + [[nodiscard]] bool getReplicateLogitsPostProcessor() const override; + + void resetIterationStats() override {} + + [[nodiscard]] SizeType32 getMaxCapacityBatchSize(SizeType32 inputLength, SizeType32 outputLength) const override + { + return 0; + }; + +protected: + std::shared_ptr<kv_cache_manager::BaseKVCacheManager> getKVCacheManager() override + { + throw std::runtime_error("TrtEncoderModel does not have KVCache."); + } + + [[nodiscard]] std::shared_ptr<kv_cache_manager::BaseKVCacheManager const> getKVCacheManager() const override + { + throw std::runtime_error("TrtEncoderModel does not have KVCache."); + } + + [[nodiscard]] std::shared_ptr<BasePeftCacheManager> getPeftCacheManager() override + { + throw std::runtime_error("TrtEncoderModel does not use PEFT."); + } + + [[nodiscard]] std::shared_ptr<BasePeftCacheManager const> getPeftCacheManager() const override + { + throw std::runtime_error("TrtEncoderModel does not use PEFT."); + } + +private: + [[nodiscard]] SizeType32 getBufferId() const + { + return mMicroBatchId; + } + + void createRuntimeContexts(); + void executeContext(SizeType32 runtimeContextId); + void createBuffers(); + void executeBatch(RequestVector const& requestList); + void executeBatch(ScheduledRequests const& scheduledRequests); + void rearrangeOutputs(ScheduledRequests const& scheduledRequests); + void createCustomAllReduceWorkspace(); + void fillEncoderOutputSync(RequestVector const& requestList, TensorMap outputTensors); + + runtime::ModelConfig const mModelConfig; + runtime::WorldConfig const mWorldConfig; + int mDevice{-1}; + std::shared_ptr<tensorrt_llm::mpi::MpiComm> mMpiCommPipelinePara; + + std::shared_ptr<nvinfer1::ILogger> mLogger; + std::shared_ptr<runtime::TllmRuntime> mRuntime; + + SizeType32 mMicroBatchId{0}; + + // TODO: Add runtime buffers for async PP + std::vector<std::shared_ptr<EncoderBuffers>> mBuffers; + + SizeType32 mNumMicroBatches; + SizeType32 mNumBuffers; + + std::vector<ScheduledRequests> mMicroBatchScheduledRequests; + ReqIdsSet mInflightReqIds; + ReqIdsSet mReqIdsToPause; + + std::unique_ptr<tensorrt_llm::batch_manager::CapacityScheduler const> mCapacityScheduler; + std::unique_ptr<tensorrt_llm::batch_manager::MicroBatchScheduler const> mMicroBatchScheduler; + + SizeType32 mHiddenSize; // already divided by Tensor Parallelism + SizeType32 mMaxInputLen; // WAR for max_input_len == max_seq_len at all circumstances + + runtime::BufferManager mCopyBufferManager; + + // Iteration counter used to distinguish debug output + executor::IterationType mIterCounter{0}; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/trtGptModel.h b/cpp/tensorrt_llm/batch_manager/trtGptModel.h new file mode 100644 index 000000000000..54ad36b13895 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/trtGptModel.h @@ -0,0 +1,339 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/peftCacheManager.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/stlUtils.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/model.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +#include <memory> + +namespace tc = tensorrt_llm::common; + +namespace tensorrt_llm::batch_manager +{ +enum class TrtGptModelType +{ + InflightBatching, + InflightFusedBatching +}; + +class LlmRequest; + +namespace kv_cache_manager +{ +class BaseKVCacheManager; +} // namespace kv_cache_manager + +class TrtGptModel : public executor::Model +{ +public: + using SizeType32 = tensorrt_llm::runtime::SizeType32; + + TrtGptModel(runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + executor::ExecutorConfig const& executorConfig) + : mMaxBatchSize{executorConfig.getMaxBatchSize().value_or(modelConfig.getMaxBatchSize())} + , mMaxBeamWidth{executorConfig.getMaxBeamWidth()} + , mMaxSequenceLen{modelConfig.getMaxSequenceLen()} + , mMaxDraftLen{modelConfig.getMaxDecodingDraftTokens()} + , mVocabSizePadded{modelConfig.getVocabSizePadded(worldConfig.getSize())} + , mNormalizeLogProbs{executorConfig.getNormalizeLogProbs()} + , mEnableTrtOverlap{executorConfig.getEnableTrtOverlap()} + , mCudaGraphMode{executorConfig.getExtendedRuntimePerfKnobConfig().getCudaGraphMode()} + { + TLLM_CHECK_WITH_INFO(mMaxBeamWidth <= modelConfig.getMaxBeamWidth(), + "Runtime configured max beam width (%d) must not exceed engine max beam width (%d)", mMaxBeamWidth, + modelConfig.getMaxBeamWidth()); + TLLM_CHECK_WITH_INFO(mMaxBatchSize <= modelConfig.getMaxBatchSize(), + "Runtime configured max batch size (%d) must not exceed engine max batch size (%d)", mMaxBatchSize, + modelConfig.getMaxBatchSize()); + if (executorConfig.getEnableTrtOverlap()) + { + if (mMaxBeamWidth > 1) + { + mEnableTrtOverlap = false; + TLLM_LOG_WARNING( + "TRT overlap is not supported with beam search (maxBeamWidth is set to %d) and will be disabled.", + mMaxBeamWidth); + } + if (!modelConfig.getSpeculativeDecodingMode().isNone()) + { + mEnableTrtOverlap = false; + TLLM_LOG_WARNING("TRT overlap is not supported with speculative decoding and will be disabled."); + } + } + + mMaxAttentionWindow = 0; + if (executorConfig.getKvCacheConfig().getMaxAttentionWindowVec().has_value()) + { + bool warning = false; + auto const& maxAttentionWindowVec = executorConfig.getKvCacheConfig().getMaxAttentionWindowVec(); + for (int maxAttenWin : maxAttentionWindowVec.value()) + { + mMaxAttentionWindowVec.push_back(std::min(maxAttenWin, mMaxSequenceLen)); + mMaxAttentionWindow = std::max(mMaxAttentionWindow, mMaxAttentionWindowVec.back()); + if (maxAttenWin > mMaxSequenceLen) + { + warning = true; + } + TLLM_CHECK_WITH_INFO(mMaxAttentionWindowVec.back() > 0, + "Attention window sizes (elements in maxAttentionWindowVec) must be > 0"); + } + if (warning) + { + TLLM_LOG_WARNING( + "The value of maxAttentionWindow cannot exceed mMaxSequenceLen. " + "Therefore, it has been adjusted to match the value of mMaxSequenceLen."); + } + } + else + { + mMaxAttentionWindowVec.push_back(mMaxSequenceLen); + mMaxAttentionWindow = mMaxSequenceLen; + } + + mSinkTokenLen = executorConfig.getKvCacheConfig().getSinkTokenLength().has_value() + ? executorConfig.getKvCacheConfig().getSinkTokenLength().value() + : 0; + + mMaxNumSequences = mMaxBatchSize * worldConfig.getPipelineParallelism(); + + auto const numTotalAttenLayers = modelConfig.getNbAttentionLayers(); + auto const numRepeatsAttenWindow = numTotalAttenLayers / mMaxAttentionWindowVec.size(); + auto const numRemainsAttenWindow = numTotalAttenLayers % mMaxAttentionWindowVec.size(); + std::string attenWindowRemainInfo = numRemainsAttenWindow > 0 + ? " + " + tc::arr2str(mMaxAttentionWindowVec.data(), numRemainsAttenWindow) + : ""; + + TLLM_LOG_INFO("TRTGptModel maxNumSequences: %d", mMaxNumSequences); + TLLM_LOG_INFO("TRTGptModel maxBatchSize: %d", mMaxBatchSize); + TLLM_LOG_INFO("TRTGptModel maxBeamWidth: %d", mMaxBeamWidth); + TLLM_LOG_INFO("TRTGptModel maxSequenceLen: %d", mMaxSequenceLen); + TLLM_LOG_INFO("TRTGptModel maxDraftLen: %d", mMaxDraftLen); + TLLM_LOG_INFO("TRTGptModel mMaxAttentionWindowSize: %s * %d%s", tc::vec2str(mMaxAttentionWindowVec).c_str(), + numRepeatsAttenWindow, attenWindowRemainInfo.c_str()); + TLLM_LOG_INFO("TRTGptModel enableTrtOverlap: %d", mEnableTrtOverlap); + TLLM_LOG_INFO("TRTGptModel normalizeLogProbs: %d", mNormalizeLogProbs); + + mMaxNumTokens = modelConfig.getMaxNumTokens(); + if (executorConfig.getMaxNumTokens().has_value() && mMaxNumTokens) + { + if (executorConfig.getMaxNumTokens().value() > mMaxNumTokens.value()) + { + TLLM_LOG_WARNING( + "Runtime configured max num tokens (%d) is larger than model max num tokens (%d) and will be " + "ignored.", + executorConfig.getMaxNumTokens().value(), mMaxNumTokens.value()); + } + else + { + mMaxNumTokens = executorConfig.getMaxNumTokens(); + } + } + if (mMaxNumTokens) + { + TLLM_LOG_INFO("TRTGptModel maxNumTokens: %d", mMaxNumTokens.value()); + } + + if (executorConfig.getEnableChunkedContext()) + { + mMaxInputLen = mMaxSequenceLen - 1; + TLLM_LOG_INFO( + "TRTGptModel maxInputLen: %d = maxSequenceLen - 1 since chunked context is enabled", mMaxInputLen); + TLLM_LOG_INFO( + "TRTGptModel If model type is encoder, maxInputLen would be reset in trtEncoderModel to maxInputLen: " + "%d = maxSequenceLen.", + mMaxSequenceLen); + } + else if (modelConfig.getContextFMHA() && modelConfig.usePackedInput()) + { + TLLM_CHECK_WITH_INFO( + mMaxNumTokens, "Max number of tokens has to be set for context FMHA and usePackedInput case."); + mMaxInputLen = std::min(mMaxSequenceLen - 1, mMaxNumTokens.value()); + TLLM_LOG_INFO( + "TRTGptModel maxInputLen: %d = min(maxSequenceLen - 1, maxNumTokens) since context FMHA " + "and usePackedInput are enabled", + mMaxInputLen); + TLLM_LOG_INFO( + "TRTGptModel If model type is encoder, maxInputLen would be reset in trtEncoderModel to maxInputLen: " + "min(maxSequenceLen, maxNumTokens)."); + } + else + { + mMaxInputLen = modelConfig.getMaxInputLen(); + TLLM_LOG_INFO("TRTGptModel maxInputLen: %d = max_input_len (in trtllm-build args)", mMaxInputLen); + } + + using tensorrt_llm::common::stl_utils::toString; + + TLLM_LOG_INFO("Capacity Scheduler Policy: %s", + toString(executorConfig.getSchedulerConfig().getCapacitySchedulerPolicy()).c_str()); + TLLM_LOG_INFO("Context Chunking Scheduler Policy: %s", + toString(executorConfig.getSchedulerConfig().getContextChunkingPolicy()).c_str()); + } + + [[nodiscard]] std::optional<SizeType32> getMaxNumTokens() const + { + return mMaxNumTokens; + } + + [[nodiscard]] SizeType32 getMaxNumSequences() const override + { + return mMaxNumSequences; + } + + [[nodiscard]] SizeType32 getMaxBatchSize() const + { + return mMaxBatchSize; + } + + [[nodiscard]] SizeType32 getMaxInputLen() const override + { + return mMaxInputLen; + } + + [[nodiscard]] SizeType32 getHiddenSize() const override + { + return getModelConfig().getHiddenSize(); + }; + + [[nodiscard]] SizeType32 getMaxSequenceLen() const override + { + return mMaxSequenceLen; + } + + [[nodiscard]] virtual TrtGptModelType getModelType() const = 0; + + [[nodiscard]] SizeType32 getVocabSizePadded() const override + { + return mVocabSizePadded; + } + + [[nodiscard]] SizeType32 getMaxDraftLen() const override + { + return mMaxDraftLen; + } + + [[nodiscard]] SizeType32 getOperatingBeamWidth() const override + { + return mMaxBeamWidth; + } + + [[nodiscard]] bool hasSpeculativeDecodingFastLogits() const noexcept override + { + return false; + } + + [[nodiscard]] bool hasGuidedDecoder() const noexcept override + { + return false; + } + + virtual void setLayerProfiler() = 0; + [[nodiscard]] virtual std::string getLayerProfileInfo() const = 0; + + [[nodiscard]] bool hasKVCacheManager() const + { + return getKVCacheManager() != nullptr; + } + +protected: + [[nodiscard]] SizeType32 getMaxBeamWidth() const + { + return mMaxBeamWidth; + } + + [[nodiscard]] std::vector<SizeType32> getMaxAttentionWindowVec() const + { + return mMaxAttentionWindowVec; + } + + [[nodiscard]] SizeType32 getMaxAttentionWindow() const + { + return mMaxAttentionWindow; + } + + [[nodiscard]] SizeType32 getSinkTokenLen() const + { + return mSinkTokenLen; + } + + [[nodiscard]] bool isNormalizeLogProbs() const + { + return mNormalizeLogProbs; + } + + [[nodiscard]] bool isTrtOverlap() const + { + return mEnableTrtOverlap; + } + + [[nodiscard]] bool isCudaGraphMode() const + { + return mCudaGraphMode; + } + + void setMaxAttentionWindowVec(std::vector<SizeType32> const& maxAttentionWindowVec) + { + TLLM_CHECK_WITH_INFO(maxAttentionWindowVec.size() == mMaxAttentionWindowVec.size(), + "The size of maxAttentionWindowVec must match the size of mMaxAttentionWindowVec"); + mMaxAttentionWindowVec = maxAttentionWindowVec; + mMaxAttentionWindow = *std::max_element(std::begin(mMaxAttentionWindowVec), std::end(mMaxAttentionWindowVec)); + } + + void setMaxSequenceLen(SizeType32 maxSequenceLen) + { + mMaxSequenceLen = maxSequenceLen; + } + + void setMaxInputLen(SizeType32 maxInputLen) + { + mMaxInputLen = maxInputLen; + } + + [[nodiscard]] std::shared_ptr<kv_cache_manager::BaseKVCacheManager> getKVCacheManager() override = 0; + [[nodiscard]] std::shared_ptr<kv_cache_manager::BaseKVCacheManager const> getKVCacheManager() const override = 0; + + [[nodiscard]] virtual std::shared_ptr<BasePeftCacheManager> getPeftCacheManager() = 0; + [[nodiscard]] virtual std::shared_ptr<BasePeftCacheManager const> getPeftCacheManager() const = 0; + +private: + std::optional<SizeType32> mMaxNumTokens; + SizeType32 mMaxNumSequences; + SizeType32 mMaxBatchSize; + SizeType32 mMaxBeamWidth; + SizeType32 mMaxInputLen; + SizeType32 mMaxSequenceLen; + SizeType32 mMaxDraftLen; + + SizeType32 mVocabSizePadded; + std::vector<SizeType32> mMaxAttentionWindowVec; + SizeType32 mMaxAttentionWindow; + SizeType32 mSinkTokenLen; + + bool mNormalizeLogProbs; + bool mEnableTrtOverlap; + bool mCudaGraphMode; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/trtGptModelFactory.h b/cpp/tensorrt_llm/batch_manager/trtGptModelFactory.h new file mode 100644 index 000000000000..bd4d7c767378 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/trtGptModelFactory.h @@ -0,0 +1,98 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/trtGptModel.h" +#include "tensorrt_llm/batch_manager/trtGptModelInflightBatching.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/runtime/gptJsonConfig.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +#include <NvInferPlugin.h> + +#include <memory> +#include <optional> + +namespace tensorrt_llm::batch_manager +{ + +class TrtGptModelFactory +{ +public: + using SizeType32 = tensorrt_llm::runtime::SizeType32; + + static std::shared_ptr<TrtGptModel> create(std::filesystem::path const& trtEnginePath, TrtGptModelType modelType, + executor::ExecutorConfig const& executorConfig, bool isLeaderInOrchMode) + { + auto const jsonConfig = runtime::GptJsonConfig::parse(trtEnginePath / "config.json"); + auto const& deviceIds = executorConfig.getParallelConfig().value_or(executor::ParallelConfig()).getDeviceIds(); + auto const worldConfig = getWorldConfig(jsonConfig, deviceIds); + auto const enginePath = trtEnginePath / jsonConfig.engineFilename(worldConfig); + + auto const& modelConfig = jsonConfig.getModelConfig(); + return create( + runtime::RawEngine(enginePath), modelConfig, worldConfig, modelType, executorConfig, isLeaderInOrchMode); + } + + static std::shared_ptr<TrtGptModel> create(std::filesystem::path const& trtEnginePath, TrtGptModelType modelType, + runtime::GptJsonConfig const& jsonConfig, runtime::WorldConfig const& worldConfig, + executor::ExecutorConfig const& executorConfig, bool isLeaderInOrchMode) + { + auto const enginePath = trtEnginePath / jsonConfig.engineFilename(worldConfig); + auto const& modelConfig = jsonConfig.getModelConfig(); + return create( + runtime::RawEngine(enginePath), modelConfig, worldConfig, modelType, executorConfig, isLeaderInOrchMode); + } + + static std::shared_ptr<TrtGptModel> create(runtime::RawEngine const& rawEngine, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, TrtGptModelType modelType, + executor::ExecutorConfig const& executorConfig, bool isLeaderInOrchMode) + { + auto logger = std::make_shared<runtime::TllmLogger>(); + auto const device = worldConfig.getDevice(); + auto const rank = worldConfig.getRank(); + TLLM_LOG_INFO("Rank %d is using GPU %d", rank, device); + TLLM_CUDA_CHECK(cudaSetDevice(device)); + + if ((modelType == TrtGptModelType::InflightBatching) || (modelType == TrtGptModelType::InflightFusedBatching)) + { + executor::ExecutorConfig const& fixedExecutorConfig + = TrtGptModelInflightBatching::executorConfigIsValid(modelConfig, executorConfig) + ? executorConfig + : TrtGptModelInflightBatching::fixExecutorConfig(modelConfig, executorConfig); + bool const ctxGenFusion = modelType == TrtGptModelType::InflightFusedBatching; + return std::make_shared<TrtGptModelInflightBatching>( + logger, modelConfig, worldConfig, rawEngine, ctxGenFusion, fixedExecutorConfig, isLeaderInOrchMode); + } + + throw std::runtime_error("Invalid modelType in trtGptModelFactory"); + } + +private: + static runtime::WorldConfig getWorldConfig( + runtime::GptJsonConfig const& json, std::optional<std::vector<SizeType32>> const& deviceIds) + { + return runtime::WorldConfig::mpi(json.getGpusPerNode(), json.getTensorParallelism(), + json.getPipelineParallelism(), json.getContextParallelism(), deviceIds); + } +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.cpp b/cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.cpp new file mode 100644 index 000000000000..7a0d78beb8a0 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.cpp @@ -0,0 +1,3136 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "trtGptModelInflightBatching.h" + +#include "tensorrt_llm/batch_manager/allocateKvCache.h" +#include "tensorrt_llm/batch_manager/assignReqSeqSlots.h" +#include "tensorrt_llm/batch_manager/cacheTransceiver.h" +#include "tensorrt_llm/batch_manager/capacityScheduler.h" +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/batch_manager/contextProgress.h" +#include "tensorrt_llm/batch_manager/createNewDecoderRequests.h" +#include "tensorrt_llm/batch_manager/decoderBuffers.h" +#include "tensorrt_llm/batch_manager/disaggTransferAdmissionController.h" +#include "tensorrt_llm/batch_manager/guidedDecoder.h" +#include "tensorrt_llm/batch_manager/handleContextLogits.h" +#include "tensorrt_llm/batch_manager/handleGenerationLogits.h" +#include "tensorrt_llm/batch_manager/kvCacheEventManager.h" +#include "tensorrt_llm/batch_manager/kvCacheManager.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/logitsPostProcessor.h" +#include "tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.h" +#include "tensorrt_llm/batch_manager/microBatchScheduler.h" +#include "tensorrt_llm/batch_manager/pauseRequests.h" +#include "tensorrt_llm/batch_manager/peftCacheManager.h" +#include "tensorrt_llm/batch_manager/promptTuningBuffers.h" +#include "tensorrt_llm/batch_manager/rnnStateManager.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" +#include "tensorrt_llm/batch_manager/sequenceSlotManager.h" +#include "tensorrt_llm/batch_manager/transformerBuffers.h" +#include "tensorrt_llm/batch_manager/updateDecoderBuffers.h" +#include "tensorrt_llm/batch_manager/utils/debugUtils.h" +#include "tensorrt_llm/batch_manager/utils/inflightBatchingUtils.h" +#include "tensorrt_llm/batch_manager/utils/logitsThread.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/envUtils.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/common/timestampUtils.h" +#include "tensorrt_llm/kernels/decodingCommon.h" +#include "tensorrt_llm/layers/defaultDecodingParams.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/gptDecoderBatched.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/ipcUtils.h" +#include "tensorrt_llm/runtime/lookaheadModule.h" +#include "tensorrt_llm/runtime/memoryCounters.h" +#include "tensorrt_llm/runtime/runtimeKernels.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tensorrt_llm/runtime/utils/runtimeUtils.h" + +#include <algorithm> +#include <cstddef> +#include <cstring> +#include <memory> +#include <numeric> +#include <optional> +#include <stdexcept> +#include <thread> +#include <utility> +#include <vector> + +using namespace tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; +namespace tk = tensorrt_llm::kernels; + +using tensorrt_llm::batch_manager::CacheTransceiverFactory; + +namespace tensorrt_llm::batch_manager +{ + +std::map<SizeType32, SizeType32> TrtGptModelInflightBatching::calculateCacheSizePerTokenForDisagg( + ModelConfig const& modelConfig, WorldConfig const& worldConfig, + std::vector<SizeType32> const& maxAttentionWindowVec, bool isCrossAttention, SizeType32 kvFactor) +{ + // These are the number of attention layers on this PP rank. + auto const numLocalAttnLayers + = modelConfig.getNbAttentionLayers(worldConfig.getPipelineParallelism(), worldConfig.getPipelineParallelRank()); + // These are the number of attention layers on all previous PP ranks. + auto const numLowerRankAttnLayers = modelConfig.countLowerRankLayers(ModelConfig::LayerType::kATTENTION, + worldConfig.getPipelineParallelism(), worldConfig.getPipelineParallelRank()); + // Use global ranks of attention layers to lookup from maxAttentionWindowVec. + auto const startAttnLayerId = numLowerRankAttnLayers; + auto const endAttnLayerId = numLowerRankAttnLayers + numLocalAttnLayers; + auto const numNonUniqueWindowSizes = static_cast<SizeType32>(maxAttentionWindowVec.size()); + std::map<SizeType32, std::vector<SizeType32>> uniqueWindowSizeToLayers; + for (SizeType32 layerIdx = startAttnLayerId; layerIdx < endAttnLayerId; layerIdx++) + { + // maxAttentionWindowVec may or may not be stretched to the length of numLayers yet. + // If not stretched yet, we cycle through the window sizes. + auto const windowSize = maxAttentionWindowVec.at(layerIdx % numNonUniqueWindowSizes); + uniqueWindowSizeToLayers[windowSize].push_back(layerIdx); + } + std::map<SizeType32, SizeType32> cacheSizeBytesPerTokenPerWindow; + for (auto const& [windowSize, globalLayerIds] : uniqueWindowSizeToLayers) + { + auto const nkvh = modelConfig.getNumKvHeadsForGivenLayers(globalLayerIds, isCrossAttention); + auto const sumLocalHeads = std::reduce(nkvh.cbegin(), nkvh.cend()); + auto const cacheSizePerToken = sumLocalHeads * kvFactor * modelConfig.getSizePerHead(); + auto const cacheSizeBytesPerToken = cacheSizePerToken * BufferDataType(modelConfig.getKvDataType()).getSize(); + cacheSizeBytesPerTokenPerWindow[windowSize] = cacheSizeBytesPerToken; + } + + return cacheSizeBytesPerTokenPerWindow; +}; + +bool TrtGptModelInflightBatching::executorConfigIsValid( + ModelConfig const& modelConfig, executor::ExecutorConfig const& executorConfig) +{ + // Make sure logic in this function matches fixExecutorConfig + if (executorConfig.getKvCacheConfig().getEnableBlockReuse()) + { + if (!modelConfig.getPagedContextFMHA()) + { + return false; + } + // Context logits cannot be returned for reused tokens, so disable reuse + if (modelConfig.computeContextLogits()) + { + return false; + } + } + return true; +} + +executor::ExecutorConfig TrtGptModelInflightBatching::fixExecutorConfig( + ModelConfig const& modelConfig, executor::ExecutorConfig const& executorConfig) +{ + // Make sure logic in this function matches executorConfigIsValid + if (executorConfig.getKvCacheConfig().getEnableBlockReuse()) + { + auto kvCacheConfig = executorConfig.getKvCacheConfig(); + + if (!modelConfig.getPagedContextFMHA()) + { + TLLM_LOG_WARNING( + "Fixing executorConfig: KV cache reuse disabled because model was not built with paged context FMHA " + "support"); + kvCacheConfig.setEnableBlockReuse(false); + } + if (modelConfig.computeContextLogits()) + { + TLLM_LOG_WARNING( + "Fixing executorConfig: KV cache reuse disabled because model was built to return context logits"); + kvCacheConfig.setEnableBlockReuse(false); + } + + auto fixedExecutorConfig = executor::ExecutorConfig(executorConfig); + fixedExecutorConfig.setKvCacheConfig(kvCacheConfig); + return fixedExecutorConfig; + } + return executorConfig; +} + +TrtGptModelInflightBatching::TrtGptModelInflightBatching(std::shared_ptr<nvinfer1::ILogger> logger, + ModelConfig const& modelConfig, WorldConfig const& worldConfig, RawEngine const& rawEngine, bool ctxGenFusion, + executor::ExecutorConfig const& executorConfig, bool isLeaderInOrchMode) + : TrtGptModel(modelConfig, worldConfig, executorConfig) + , mModelConfig(modelConfig) + , mWorldConfig(worldConfig) + , mDevice{runtime::utils::initDevice(worldConfig)} + , mDecodingConfig{executorConfig.getDecodingConfig().value_or(executor::DecodingConfig{})} + , mExtendedRuntimePerfKnobConfig{executorConfig.getExtendedRuntimePerfKnobConfig()} + , mDebugConfig{executorConfig.getDebugConfig()} + , mAdditionalModelOutputs{worldConfig.isLastPipelineParallelRank() ? executorConfig.getAdditionalModelOutputs() + : std::nullopt} + , mLogger{logger ? std::move(logger) : std::make_shared<TllmLogger>()} + , mRuntime{std::make_unique<TllmRuntime>(rawEngine, mLogger.get(), executorConfig.getUseGpuDirectStorage(), + executorConfig.getGpuWeightsPercent(), modelConfig.useShapeInference())} + , mCopyBufferManager{std::make_shared<CudaStream>()} + , mCtxGenFusion(ctxGenFusion) + , mOperatingBeamWidth{getMaxBeamWidth()} + , mGatherGenerationLogits{executorConfig.getGatherGenerationLogits()} + , mPromptTableOffloading{executorConfig.getPromptTableOffloading()} + , mIsLeaderInOrchMode{isLeaderInOrchMode} +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_LOG_INFO("gatherContextLogits: %d", mModelConfig.computeContextLogits()); + TLLM_LOG_INFO("gatherGenerationLogits: %d", getGatherGenerationLogits()); + + if (!(mModelConfig.supportsInflightBatching())) + { + throw std::runtime_error( + "TrtGptModelInflightBatching requires GPT attention/Mamba Conv 1d plugin with " + "packed input and paged KV cache."); + } + if (mWorldConfig.isTensorParallel()) + { + mRuntime->initializeUserBuffer(mWorldConfig, mModelConfig.getMaxBatchSize(), mModelConfig.getMaxBeamWidth(), + mModelConfig.getMaxSequenceLen(), mModelConfig.getHiddenSize(), getMaxNumTokens()); + } + if (mWorldConfig.isPipelineParallel()) + { + mNumMicroBatches = mWorldConfig.getPipelineParallelism(); + } + else + { + mNumMicroBatches = isTrtOverlap() ? 2 : 1; + } + + mNumBuffers = (mCtxGenFusion ? 1 : 2) * mNumMicroBatches; + + auto const& kvCacheConfig = executorConfig.getKvCacheConfig(); + + if (mModelConfig.getSpeculativeDecodingMode().isDraftTokensExternal()) + { + TLLM_CHECK_WITH_INFO(kvCacheConfig.getEnableBlockReuse(), + "KV cache block reuse must be enabled for speculative decoding target model"); + } + + if (mCtxGenFusion) + { + TLLM_CHECK_WITH_INFO(!mModelConfig.isRnnBased(), "RNN based model doesn't support context generation fusion."); + TLLM_CHECK_WITH_INFO( + mModelConfig.isTransformerBased(), "Only transformer based model support context generation fusion now."); + } + + if (mModelConfig.getSpeculativeDecodingMode().isLookaheadDecoding()) + { + mSeamlessLADMaxDraftLen = modelConfig.getMaxDecodingDraftTokens(); + // TODO: enable it when speculativeDecodingMode is None and run with '--lookahead_config' + mUseSeamlessLookahead = false; + } + + setupSpeculativeDecodingModule(mDecodingConfig); + + if (mWorldConfig.isLastPipelineParallelRank() && executorConfig.getGuidedDecodingConfig()) + { + mGuidedDecoder = std::make_unique<GuidedDecoder>(executorConfig.getGuidedDecodingConfig().value(), + getMaxNumSequences(), mModelConfig.getVocabSizePadded(mWorldConfig.getSize()), + mModelConfig.getLogitsDtype(), mRuntime->getBufferManager()); + } + + createRuntimeContexts(); + + if (mWorldConfig.isTensorParallel()) + { + createCustomAllReduceWorkspace(); + } + + if (mModelConfig.isTransformerBased()) + { + createRuntimePerfKnobsTensor(mExtendedRuntimePerfKnobConfig); + } + + auto& memCounter = MemoryCounters::getInstance(); + auto const gpuUsage1 = memCounter.getGpu(); + createBuffers(mDecodingConfig, mAdditionalModelOutputs); + auto const gpuUsage2 = memCounter.getGpu(); + TLLM_LOG_INFO("[MemUsageChange] Allocated %s GPU memory for runtime buffers.", + memCounter.bytesToString(gpuUsage2 - gpuUsage1).c_str()); + + createDecoder(mDecodingConfig.getDecodingMode()); + auto const gpuUsage3 = memCounter.getGpu(); + TLLM_LOG_INFO("[MemUsageChange] Allocated %s GPU memory for decoder.", + memCounter.bytesToString(gpuUsage3 - gpuUsage2).c_str()); + + if (modelConfig.getManageWeightsType() != ModelConfig::ManageWeightsType::kDisabled) + { + mRuntime->loadManagedWeights(rawEngine, worldConfig.getLocalRank()); + } + + if (mModelConfig.useLoraPlugin()) + { + auto const peftCacheManagerConfig + = PeftCacheManagerConfig(executorConfig.getPeftCacheConfig().value_or(executor::PeftCacheConfig())); + mPeftCacheManager = std::make_shared<PeftCacheManager>( + peftCacheManagerConfig, mModelConfig, mWorldConfig, mRuntime->getBufferManager()); + } + else + { + mPeftCacheManager = std::make_shared<NoOpPeftCacheManager>(); + } + + if (mModelConfig.isRnnBased()) + { + createRnnStateManager(); + } + if (mModelConfig.isTransformerBased() && modelConfig.isKVCacheEnabled()) + { + auto cacheTransceiverConfig + = executorConfig.getCacheTransceiverConfig().value_or(executor::CacheTransceiverConfig()); + + auto const cacheSizeBytesPerTokenPerWindow = calculateCacheSizePerTokenForDisagg( + mModelConfig, mWorldConfig, getMaxAttentionWindowVec(), mModelConfig.useCrossAttention(), 2); + auto cacheTransPreAllocaSize = kv_cache_manager::CacheTransBufferManager::preAllocBufferSize( + cacheSizeBytesPerTokenPerWindow, mModelConfig.getTokensPerBlock(), cacheTransceiverConfig); + + auto const [freePrimaryMemBytes, freeSecondaryMemBytes] + = BaseKVCacheManager::calculateFreeMemBytes(mRuntime->getBufferManager(), kvCacheConfig); + if (mModelConfig.useCrossAttention()) + { + TLLM_CHECK_WITH_INFO(kvCacheConfig.getCrossKvCacheFraction().has_value(), + "Must set crossKvCacheFraction for encoder-decoder model"); + auto const crossKvCacheFraction = kvCacheConfig.getCrossKvCacheFraction().value(); + mKvCacheManager = createKvCacheManager(kvCacheConfig, KvCacheType::kSELF, + freePrimaryMemBytes * (1.0f - crossKvCacheFraction), + freeSecondaryMemBytes * (1.0f - crossKvCacheFraction), cacheTransPreAllocaSize, + executorConfig.getFailFastOnAttentionWindowTooLarge()); + mCrossKvCacheManager = createKvCacheManager(kvCacheConfig, KvCacheType::kCROSS, + freePrimaryMemBytes * crossKvCacheFraction, freeSecondaryMemBytes * crossKvCacheFraction, + cacheTransPreAllocaSize, executorConfig.getFailFastOnAttentionWindowTooLarge()); + TLLM_LOG_INFO("This is an Encoder-Decoder model, set %0.1f cross KV cache fraction based on the config.", + crossKvCacheFraction); + } + else + { + TLLM_CHECK_WITH_INFO(!kvCacheConfig.getCrossKvCacheFraction().has_value(), + "Do not set crossKvCacheFraction for decoder-only model"); + mKvCacheManager = createKvCacheManager(kvCacheConfig, KvCacheType::kSELF, freePrimaryMemBytes, + freeSecondaryMemBytes, cacheTransPreAllocaSize, executorConfig.getFailFastOnAttentionWindowTooLarge()); + } + + mCacheTransceiver + = CacheTransceiverFactory::createCacheTransceiver(mKvCacheManager.get(), mModelConfig, mWorldConfig, + executor::kv_cache::CacheState::AttentionType::kDEFAULT, executorConfig.getCacheTransceiverConfig()); + mDisaggTransferAdmissionController = std::make_unique<DisaggTransferAdmissionController>( + cacheTransceiverConfig.getMaxTokensInBuffer(), mModelConfig.getTokensPerBlock()); + } + + if (mModelConfig.getSpeculativeDecodingMode().needsKVCacheRewind()) + { + TLLM_CHECK_WITH_INFO( + mModelConfig.isKVCacheEnabled(), "When needsKVCacheRewind() returns true, KV cache needs to be enabled."); + auto const& blockManager = mKvCacheManager->getBlockManager(); + + TLLM_CHECK_WITH_INFO(blockManager.getNumPools() == 1, + "Rewinding KV cache blocks for models with multiple pools is not supported"); + + // Two "redundant" checks given the pool size check above, but those below don't rely on an implementation + // detail I guess. + TLLM_CHECK_WITH_INFO( + !blockManager.isVariableWindow(), "Rewinding KV cache blocks for variable SWA models isn't supported"); + auto const maxBlocksPerSeq = blockManager.getMaxBlockPerSeqWhenSingleWindowSize(); + + // TODO(oargov): VGQA is not supported, assume all layers have the same num_kv_heads + TLLM_CHECK_WITH_INFO( + !blockManager.isVariableGQA(), "Rewinding KV cache blocks for variable GQA models isn't supported"); + auto const numKvHeads = mModelConfig.getNbKvHeads(0); + + mRewindInputs = RewindInputs{maxBlocksPerSeq, /*isUseOneMoreBlock*/ false, numKvHeads}; + } + + if (mWorldConfig.isPipelineParallel()) + { + mAsyncSendWaitThread = std::make_unique<tensorrt_llm::mpi::MpiWaitThread>( + "asyncSendWaitThread", + [this]() + { + mDecStepAsyncSndHdls.clear(); + mDecSlotAsyncSndHdls.clear(); + }, + [this]() { TLLM_CUDA_CHECK(cudaSetDevice(mWorldConfig.getDevice())); }); + + auto const& commSession = COMM_SESSION; + mMpiCommPipelinePara = std::make_unique<tensorrt_llm::mpi::MpiComm>( + commSession.split(mWorldConfig.getTensorParallelRank(), mWorldConfig.getPipelineParallelRank())); + mDecSlotAsyncSndHdls.reserve(getMaxBatchSize()); + } + if (mWorldConfig.isTensorParallel()) + { + auto const& commSession = COMM_SESSION; + mMpiCommTensorPara = std::make_unique<tensorrt_llm::mpi::MpiComm>( + commSession.split(mWorldConfig.getPipelineParallelRank(), mWorldConfig.getTensorParallelRank())); + } + + mSeqSlotManager + = std::make_shared<SequenceSlotManager>(getMaxNumSequences(), executorConfig.getMaxSeqIdleMicroseconds()); + + mMicroBatchScheduledRequests.resize(mNumMicroBatches); + mDecoderFinishedEvents.resize(mNumMicroBatches); + mPeftTables.resize(mNumMicroBatches); + + if (modelConfig.isRnnBased()) + { + TLLM_CHECK_WITH_INFO(modelConfig.getMaxBeamWidth() == 1, "RNN based model doesn't support beam search now."); + TLLM_CHECK_WITH_INFO( + !executorConfig.getEnableChunkedContext(), "RNN based model doesn't support Chunked Context now."); + TLLM_CHECK_WITH_INFO( + modelConfig.getSpeculativeDecodingMode().isNone(), "RNN based model doesn't support speculative decoding."); + } + + std::optional<batch_scheduler::ContextChunkingConfig> ctxChunkConfig; + if (executorConfig.getEnableChunkedContext()) + { + TLLM_CHECK_WITH_INFO(modelConfig.isKVCacheEnabled() && mModelConfig.getPagedContextFMHA(), + "Chunked context requires context FMHA, paged kv_cache and paged context FMHA all enabled at the same " + "time."); + SizeType32 chunkUnitSize = mKvCacheManager->getTokensPerBlock(); + // If sliding window attention is used, then make sure the unit size aligns with the paged context fmha's kv + // step size. + if (getMaxInputLen() > getMaxAttentionWindow()) // TODO(nhaber): minAttentionWindow + { + chunkUnitSize = std::max(/* maxKvStepSizeInFmha */ 256, chunkUnitSize); + TLLM_LOG_INFO("ChunkUnitSize is set to %d as sliding window attention is used.", chunkUnitSize); + } + ctxChunkConfig = batch_scheduler::ContextChunkingConfig{ + executorConfig.getSchedulerConfig().getContextChunkingPolicy().value_or( + executor::ContextChunkingPolicy::kFIRST_COME_FIRST_SERVED), + chunkUnitSize}; + } + + auto maxNumTokens = getMaxNumTokens(); + TLLM_CHECK_WITH_INFO(maxNumTokens, "Max number of tokens is not set in model config."); + + // Max context size is limited by `max_num_tokens` for chunked-context or context-FMHA, + // or by `max_input_len` of the model. + auto const maxContextLength = (executorConfig.getEnableChunkedContext() || mModelConfig.getContextFMHA()) + ? maxNumTokens + : std::make_optional<SizeType32>(mModelConfig.getMaxInputLen()); + + mMaxBatchSizeTunerRecommended = 0; + mMaxBatchSizeRuntime = getMaxBatchSize(); + mMaxNumTokensStatic = maxNumTokens; + mMaxNumTokensTunerRecommended = 0; + mMaxNumTokensRuntime = maxNumTokens; + + if (mKvCacheManager && ctxChunkConfig) + { + TLLM_CHECK_WITH_INFO(ctxChunkConfig.value().chunkUnitSize % mKvCacheManager->getTokensPerBlock() == 0, + "To prevent cache fragmentation, the context chunk unit size (%d) should be divisible by the number of " + "tokens per kv-cache block (%d).", + ctxChunkConfig.value().chunkUnitSize, mKvCacheManager->getTokensPerBlock()); + } + + mCapacityScheduler = std::make_unique<CapacityScheduler>(getMaxNumSequences(), + executorConfig.getSchedulerConfig().getCapacitySchedulerPolicy(), mKvCacheManager != nullptr, + /*twoStepsLookAhead=*/mWorldConfig.isPipelineParallel(), + /*noScheduleUntilState=*/LlmRequestState::kCONTEXT_INIT, + /*noScheduleAfterState=*/LlmRequestState::kGENERATION_COMPLETE, + /*enablePrefixAwareScheduling=*/executorConfig.getSchedulerConfig().getEnablePrefixAwareScheduling()); + + mMicroBatchScheduler = std::make_unique<MicroBatchScheduler>(ctxChunkConfig, maxContextLength); + + if (ctxChunkConfig) + { + if (maxContextLength) + { + ctxChunkConfig.value().chunkUnitSize + = std::min(ctxChunkConfig.value().chunkUnitSize, maxContextLength.value()); + } + TLLM_CHECK_WITH_INFO(ctxChunkConfig.value().chunkUnitSize > 0, + "Context chunk size (%d) must be a positive integer.", maxContextLength.value()); + } + else + { + if (maxContextLength && maxNumTokens) + { + TLLM_CHECK_WITH_INFO(maxContextLength.value() <= maxNumTokens.value(), + "Without enabling chunked context, the max context length (%d) needs to be less than or equal to the " + "max number of tokens (%d).", + maxContextLength.value(), maxNumTokens.value()); + } + } + + mPauseRequests = std::make_unique<PauseRequests>(getMaxInputLen()); + mAssignReqSeqSlots = std::make_unique<AssignReqSeqSlots>(); + mAllocateKvCache = std::make_unique<AllocateKvCache>(); + + if (isCudaGraphMode()) + { + // Limit cuda graph cache size. Depending on the model one graph is 4-10MB of GPU memory. + SizeType32 cudaGraphCacheSize + = std::min(getMaxBatchSize(), std::max(mExtendedRuntimePerfKnobConfig.getCudaGraphCacheSize(), 1)); + // We can't have common cache for all microbatches as cuda graph is tied to the memory pointers of the runtime + // buffers. + mCudaGraphExecutorCaches.resize(mNumBuffers, utils::CudaGraphExecutorCache(cudaGraphCacheSize)); + } + + mSpeculativeDecodingFastLogits + = executorConfig.getSpecDecConfig().has_value() && executorConfig.getSpecDecConfig()->fastLogits; + if (mSpeculativeDecodingFastLogits && modelConfig.getSpeculativeDecodingMode().isNone() && mIsLeaderInOrchMode) + { + mDraftModelSendLogitsThread + = std::make_unique<std::thread>(&utils::draftModelSendLogitsThread, mDevice, &mDraftModelThreadShouldExit, + &mDraftRequestsWaitingToSendLogits, &mDraftRequestsDoneSendingLogits, &mDraftRequestsMtx); + } + + mCreateNewDecoderRequests = std::make_unique<CreateNewDecoderRequests>( + mSpeculativeDecodingFastLogits, mIsLeaderInOrchMode, isNormalizeLogProbs()); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +TrtGptModelInflightBatching::~TrtGptModelInflightBatching() +{ + if (mCacheTransceiver) + { + mCacheTransceiver->checkContextTransferStatus(1, true); + TLLM_CHECK_WITH_INFO(mCacheTransceiver->checkGenTransferComplete(), "Generation transfer not complete"); + } + if (mAsyncSendWaitThread) + { + mAsyncSendWaitThread.reset(nullptr); + } + if (mDraftModelSendLogitsThread) + { + mDraftModelThreadShouldExit = true; + mDraftModelSendLogitsThread->join(); + mDraftModelSendLogitsThread.reset(nullptr); + } +} + +void TrtGptModelInflightBatching::setupSpeculativeDecodingModule(executor::DecodingConfig const& decodingConfig) +{ + if (mModelConfig.getSpeculativeDecodingMode().isExplicitDraftTokens() + || mModelConfig.getSpeculativeDecodingMode().isEagle()) + { + TLLM_CHECK_WITH_INFO(mCtxGenFusion, "Current speculative decoding mode requires context-gen fusion IFB"); + } + + if (mModelConfig.getSpeculativeDecodingMode().isLookaheadDecoding() && decodingConfig.getLookaheadDecodingConfig()) + { + // FIXME choose defaults + auto maxLookaheadConfig = decodingConfig.getLookaheadDecodingConfig().value(); + + SizeType32 maxDraftTokens{0}; + SizeType32 maxDraftPathLen{0}; + std::tie(std::ignore, std::ignore, maxDraftTokens, maxDraftPathLen) + = maxLookaheadConfig.calculateSpeculativeResource(); + TLLM_CHECK(maxDraftTokens <= mModelConfig.getMaxDecodingDraftTokens()); + mModelConfig.getSpeculativeDecodingModulePtr()->setMaxDraftTokens(maxDraftTokens); + mModelConfig.getSpeculativeDecodingModulePtr()->setMaxDraftPathLen(maxDraftPathLen); + + auto lookaheadModulePtr + = std::dynamic_pointer_cast<runtime::LookaheadModule>(mModelConfig.getSpeculativeDecodingModulePtr()); + lookaheadModulePtr->setExecutionConfig(maxLookaheadConfig); + } +} + +void TrtGptModelInflightBatching::reshapeKvTensors(OffsetTableDimensions const& dims) +{ + TLLM_CHECK(mBuffers.size() == static_cast<size_t>(mNumBuffers)); + auto const& manager = mRuntime->getBufferManager(); + for (auto& buffers : mBuffers) + { + TLLM_CHECK(buffers->transformerBuffers); + // any method that operates on transformerBuffers must distinguish between self and cross cache, because + // transformerBuffers is not managed by KVCacheManager same rule applies to kv pool pointers below + buffers->transformerBuffers->reshapeKvTensors( + getMaxBatchSize(), mOperatingBeamWidth, dims.maxBlocksPerSeq, dims.cacheType, dims.numPools, manager); + } +} + +using BlocksPerWindow = std::map<SizeType32, std::tuple<SizeType32, SizeType32>>; + +std::pair<BlocksPerWindow, std::vector<SizeType32>> +TrtGptModelInflightBatching::clampWindowSizesToFitAtLeastOneSequence( + BlocksPerWindow const& blocksPerWindow, bool const failFastOnAttentionWindowTooLarge) +{ + // At this point, we can only validate that the cheapest sequence in terms of kv-cache resources still fits. More + // validation is needed on a per-request basis, once the prompt / output lengths and the actual beam width are + // known. + auto const promptLength = getMaxInputLen(); + auto const outputLength + = getMaxSequenceLen() - promptLength; // This makes it the best case scenario, as context tokens are 'cheaper' + // in terms of kv-cache resources on average. + auto const sinkTokenLength = getSinkTokenLen(); + auto const maxBeamWidth = getMaxBeamWidth(); + auto const tokensPerBlock = mModelConfig.getTokensPerBlock(); + auto const& oldMaxAttentionWindowVec = getMaxAttentionWindowVec(); + std::vector<SizeType32> newMaxAttentionWindowVec; + BlocksPerWindow newBlocksPerWindow; + + newMaxAttentionWindowVec.reserve(oldMaxAttentionWindowVec.size()); + for (auto const windowSize : oldMaxAttentionWindowVec) + { + auto const bestCaseBlockRequirements = kv_cache_manager::KVCacheManager::calculateMaxBlockRequirements( + promptLength, outputLength, sinkTokenLength, windowSize, maxBeamWidth, tokensPerBlock); + auto const [numPrimaryBlocks, numSecondaryBlocks] = blocksPerWindow.at(windowSize); + if (bestCaseBlockRequirements > numPrimaryBlocks) + { + auto const newMaxAttentionWindow = KVCacheManager::calculateMaxAttentionWindow( + promptLength, outputLength, sinkTokenLength, numPrimaryBlocks, maxBeamWidth, tokensPerBlock); + newMaxAttentionWindowVec.push_back(newMaxAttentionWindow); + newBlocksPerWindow[newMaxAttentionWindow] = std::make_tuple(numPrimaryBlocks, numSecondaryBlocks); + } + else + { + newMaxAttentionWindowVec.push_back(windowSize); + newBlocksPerWindow[windowSize] = std::make_tuple(numPrimaryBlocks, numSecondaryBlocks); + } + } + if (newMaxAttentionWindowVec == getMaxAttentionWindowVec()) + { + return {blocksPerWindow, newMaxAttentionWindowVec}; + } + TLLM_LOG_WARNING("maxAttentionWindowVec too large to fit at least one sequence in kvCache. Old: %s, New: %s", + common::vec2str(getMaxAttentionWindowVec()).c_str(), common::vec2str(newMaxAttentionWindowVec).c_str()); + + if (failFastOnAttentionWindowTooLarge) + { + throw std::runtime_error( + "Attention window too large to fit even a single sequence in the KV cache. Failing fast rather than " + "attempting an adjustment of the window sizes. " + "Old: " + + common::vec2str(getMaxAttentionWindowVec()) + ", New: " + common::vec2str(newMaxAttentionWindowVec)); + } + + setMaxAttentionWindowVec(newMaxAttentionWindowVec); + if (getMaxSequenceLen() > getMaxAttentionWindow()) + { + TLLM_LOG_WARNING("maxSequenceLen is reduced to maxAttentionWindow: %d", getMaxAttentionWindow()); + setMaxSequenceLen(getMaxAttentionWindow()); + if (getMaxInputLen() > getMaxSequenceLen() - 1) + { + setMaxInputLen(getMaxSequenceLen() - 1); + TLLM_LOG_WARNING("maxInputLen is reduced to %d", getMaxInputLen()); + } + } + // createBuffers depends on: + // maxAttentionWindow; maxAttentionWindowVec; maxSequenceLen; + // TODO: This is problematic, as createBuffers edits the state of trtGptModelInflightBatching, but + // what if there are different window values for cross+self etc. in encoder+decoder scenario... + createBuffers(mDecodingConfig, mAdditionalModelOutputs); + createDecoder(mDecodingConfig.getDecodingMode()); + return {newBlocksPerWindow, newMaxAttentionWindowVec}; +} + +std::unique_ptr<kv_cache_manager::KVCacheManager> TrtGptModelInflightBatching::createKvCacheManager( + KvCacheConfig const& kvCacheConfig, KvCacheType kvCacheType, uint64_t freePrimaryMemBytes, + uint64_t freeSecondaryMemBytes, size_t extraCostMemory, bool const failFastOnAttentionWindowTooLarge) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + bool isCrossAttention = kvCacheType == KvCacheType::kCROSS; + TLLM_CHECK_WITH_INFO( + mModelConfig.isTransformerBased(), "KvCacheManager is only needed by transformer based model."); + + auto const tokensPerBlock = mModelConfig.getTokensPerBlock(); + auto const kvDtype = mModelConfig.getKvDataType(); + + // init KV cache block manager + auto [numKvHeadsPerLayerBegin, numKvHeadsPerLayerEnd] = mModelConfig.getNumKvHeadsPerLayerLocalRange( + mWorldConfig.getPipelineParallelism(), mWorldConfig.getPipelineParallelRank(), isCrossAttention); + auto numKvHeadsPerLayer = std::vector<SizeType32>(numKvHeadsPerLayerBegin, numKvHeadsPerLayerEnd); + + auto maxAttentionWindowVec = getMaxAttentionWindowVec(); + if (kvCacheType != KvCacheType::kSELF) // TODO(nhaber): more foolproof way of initing cross-kvcache-manager + { + maxAttentionWindowVec = std::vector<SizeType32>{mModelConfig.getMaxEncoderLen()}; + } + + auto const numLayers = static_cast<SizeType32>(numKvHeadsPerLayer.size()); + auto const windowSizeToLayers = KVCacheManager::groupLayersByWindowSize(maxAttentionWindowVec, numLayers); + auto const sizePerHead = mModelConfig.getSizePerHead(); + auto blocksPerWindow = KVCacheManager::calculateMaxNumBlocks(kvCacheConfig, kvDtype, numKvHeadsPerLayer, + sizePerHead, tokensPerBlock, mWorldConfig, windowSizeToLayers, freePrimaryMemBytes, freeSecondaryMemBytes, + extraCostMemory, 2, getMaxBatchSize()); + + // now we check if any of the window sizes is too large for at least one sequence to fit in kvCache + // this can happen if e.g. maxSeqLen is deduced from the model and is too large + // and user also didn't provide maxAttentionWindow, which leads it to be equal to maxSeqLen + if (kvCacheType == KvCacheType::kSELF) + { + std::tie(blocksPerWindow, maxAttentionWindowVec) + = clampWindowSizesToFitAtLeastOneSequence(blocksPerWindow, failFastOnAttentionWindowTooLarge); + } + + if (kvCacheType == KvCacheType::kCROSS && kvCacheConfig.getEnableBlockReuse()) + { + TLLM_LOG_INFO( + "Cross KV cache does not support reuse because cross attention depends on encoder and decoder input ids. " + "Thus, KV cache reuse is disabled for cross KV cache."); + } + auto const enableBlockReuse = kvCacheType == KvCacheType::kSELF ? kvCacheConfig.getEnableBlockReuse() : false; + + auto kvCacheManager = std::make_unique<KVCacheManager>(numKvHeadsPerLayer, sizePerHead, tokensPerBlock, + blocksPerWindow, getMaxNumSequences(), getMaxBeamWidth(), maxAttentionWindowVec, kvDtype, getSinkTokenLen(), + mRuntime->getStreamPtr(), + kvCacheType == KvCacheType::kCROSS ? mModelConfig.getMaxEncoderLen() : getMaxSequenceLen(), + getMaxNumTokens().value(), enableBlockReuse, kvCacheType, kvCacheConfig.getSecondaryOffloadMinPriority(), + kvCacheConfig.getEventBufferMaxSize() > 0 + ? std::make_unique<kv_cache_manager::KVCacheEventManager>(kvCacheConfig.getEventBufferMaxSize()) + : nullptr, + kvCacheConfig.getEnablePartialReuse(), kvCacheConfig.getCopyOnPartialReuse()); + + reshapeKvTensors(kvCacheManager->getOffsetTableDimensions()); + + kvCacheManager->allocatePools(kvCacheConfig.getUseUvm()); + + TensorMap inputBuffers; + TensorPtr poolPointers = kvCacheManager->getBlockPoolPointers(); + TensorPtr poolMapping = kvCacheManager->getLayerToPoolMapping(); + + if (kvCacheType == KvCacheType::kSELF) + { + inputBuffers.insert_or_assign("host_kv_cache_pool_pointers", std::move(poolPointers)); + inputBuffers.insert_or_assign("host_kv_cache_pool_mapping", std::move(poolMapping)); + } + else + { + inputBuffers.insert_or_assign("host_cross_kv_cache_pool_pointers", std::move(poolPointers)); + inputBuffers.insert_or_assign("host_cross_kv_cache_pool_mapping", std::move(poolMapping)); + } + mRuntime->setStaticInputTensors(inputBuffers); + + // Emit the `created` event + kvCacheManager->flushIterationEvents(); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return kvCacheManager; +} + +void TrtGptModelInflightBatching::createRnnStateManager() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK_WITH_INFO(mModelConfig.isRnnBased(), "RnnStateManager is only needed by RNN based model."); + + mRnnStateManager = std::make_unique<RnnStateManager>( + getMaxNumSequences(), mModelConfig, mWorldConfig, mRuntime->getBufferManager()); + + TensorMap inputBuffers; + mRnnStateManager->getPtrBuffers(inputBuffers, mModelConfig, mWorldConfig); + mRuntime->setStaticInputTensors(inputBuffers); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::createCustomAllReduceWorkspace() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + TLLM_CHECK(mWorldConfig.isTensorParallel()); + + auto const& manager = mRuntime->getBufferManager(); + auto const hiddenSize = mModelConfig.getHiddenSize(); + + mAllReduceBuffers = std::make_unique<AllReduceBuffers>(getMaxBatchSize(), getMaxBeamWidth(), getMaxSequenceLen(), + hiddenSize, manager, mWorldConfig, mRuntime->isUserBufferEnabled()); + + TensorMap inputBuffers; + inputBuffers.insert_or_assign("all_reduce_workspace", mAllReduceBuffers->mAllReduceCommPtrs); + mRuntime->setStaticInputTensors(inputBuffers); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::createRuntimePerfKnobsTensor( + executor::ExtendedRuntimePerfKnobConfig const& extendedRuntimePerfKnobConfig) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + SizeType32 constexpr perfKnobSize{16}; + mExtendedRuntimePerfKnobsHost = BufferManager::cpu(ITensor::makeShape({perfKnobSize}), nvinfer1::DataType::kINT64); + auto* runtimePerfKnobsHostPtr = bufferCast<int64_t>(*mExtendedRuntimePerfKnobsHost); + std::fill_n(runtimePerfKnobsHostPtr, perfKnobSize, -1); + SizeType32 multiBlockModeVal = extendedRuntimePerfKnobConfig.getMultiBlockMode() ? 1 : 0; + SizeType32 enableContextFMHAFP32AccVal = extendedRuntimePerfKnobConfig.getEnableContextFMHAFP32Acc() ? 1 : 0; + runtimePerfKnobsHostPtr[0] = multiBlockModeVal; + runtimePerfKnobsHostPtr[1] = enableContextFMHAFP32AccVal; + + TensorMap inputBuffers; + inputBuffers.insert_or_assign("host_runtime_perf_knobs", mExtendedRuntimePerfKnobsHost); + mRuntime->setStaticInputTensors(inputBuffers); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::terminateRequest(LlmRequestPtr const& llmReq, bool pause) +{ + utils::terminateRequest( + *mSeqSlotManager, *llmReq, getMaxInputLen(), mKvCacheManager, mCrossKvCacheManager, mPeftCacheManager, pause); +} + +void TrtGptModelInflightBatching::terminateRequestSync( + LlmRequestPtr const& llmRequest, executor::FinishReason finishReason) +{ + TLLM_LOG_DEBUG("Registering termination for request %lu with finish reason %d", llmRequest->mRequestId, + static_cast<int>(finishReason)); + mReqIdsToTerminate.try_emplace(llmRequest->mRequestId, finishReason); +} + +TrtGptModelInflightBatching::IterationStatsIFB TrtGptModelInflightBatching::fillIterationStats( + ScheduledRequests const& scheduledRequests, RequestVector const& requestsToPause) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(fillIterationStats); + + IterationStatsIFB iterationStatsIfb{mMicroBatchId}; + iterationStatsIfb.numCtxRequests = scheduledRequests.contextRequests.size(); + iterationStatsIfb.numGenRequests = scheduledRequests.generationRequests.size(); + iterationStatsIfb.avgNumDecodedTokensPerIter = 0; + + auto const contextBufferId = mCtxGenFusion ? getFusedBufferId() : getContextBufferId(); + auto const& buffers = mBuffers.at(contextBufferId); + iterationStatsIfb.numCtxTokens = buffers->getNumContextTokens(); + + for (auto const& llmReq : scheduledRequests.contextRequests) + { + iterationStatsIfb.scheduledRequests.insert(llmReq->mRequestId); + } + for (auto const& llmReq : scheduledRequests.generationRequests) + { + iterationStatsIfb.scheduledRequests.insert(llmReq->mRequestId); + iterationStatsIfb.avgNumDecodedTokensPerIter += llmReq->getAvgDecodedTokensPerIter(); + } + if (iterationStatsIfb.numGenRequests > 0) + { + iterationStatsIfb.avgNumDecodedTokensPerIter /= iterationStatsIfb.numGenRequests; + TLLM_LOG_DEBUG( + "iterationStatsIfb.avgNumDecodedTokensPerIter = %.2f", iterationStatsIfb.avgNumDecodedTokensPerIter); + } + for (auto const& llmReq : requestsToPause) + { + iterationStatsIfb.pausedRequests.insert(llmReq->mRequestId); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return iterationStatsIfb; +} + +void TrtGptModelInflightBatching::forwardSync() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE_WITH_NAME(range, "TrtGptModelInflightBatching::forwardSync"); + + TLLM_CUDA_CHECK(cudaSetDevice(mWorldConfig.getDevice())); + + if (!mWorldConfig.isLastPipelineParallelRank()) + { + mAsyncSendWaitThread->waitStop(); + } + + auto& currRequests = mMicroBatchScheduledRequests.at(mMicroBatchId); + + if (!currRequests.empty()) + { + if (!mWorldConfig.isPipelineParallel() || !mWorldConfig.isLastPipelineParallelRank()) + { + for (auto& hdl : mDecStepAsyncSndHdls) + { + TLLM_CHECK_WITH_INFO(hdl.get() == nullptr, "decoderSync handle must be nullptr."); + } + // Wait for decoding for requests in flight for the current micro batch + auto& decoderWaitEvent = mDecoderFinishedEvents.at(mMicroBatchId); + mDecStepAsyncSndHdls = decoderSync(currRequests, decoderWaitEvent); + decoderWaitEvent.reset(); + + if (!mWorldConfig.isLastPipelineParallelRank()) + { + mAsyncSendWaitThread->notifyStart(); + } + } + else + { + for (auto const& requests : {currRequests.contextRequests, currRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + for (SizeType32 beam = 0; beam < llmReq->mSamplingConfig.beamWidth; ++beam) + { + llmReq->setNumPreDecodedTokens(0, beam); + } + if (llmReq->isGenerationToCompleteState()) + { + llmReq->setState(LlmRequestState::kGENERATION_COMPLETE); + terminateRequest(llmReq); + } + } + } + } + + (*mPauseRequests)(currRequests.generationRequests, mInflightReqIds, mReqIdsToPause, true, *mSeqSlotManager, + mKvCacheManager, mCrossKvCacheManager, mPeftCacheManager); + + if (!mReqIdsToTerminate.empty()) + { + for (auto const& requests : {currRequests.contextRequests, currRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + if (mReqIdsToTerminate.count(llmReq->mRequestId) != 0U) + { + if (!llmReq->isGenerationCompleteState()) + { + TLLM_LOG_DEBUG("Terminating request %lu with finish reason %d", llmReq->mRequestId, + static_cast<int>(mReqIdsToTerminate[llmReq->mRequestId])); + terminateRequest(llmReq); + llmReq->finishByReason(mReqIdsToTerminate[llmReq->mRequestId]); + llmReq->clearGeneratedTokens(); + } + mReqIdsToTerminate.erase(llmReq->mRequestId); + } + } + } + } + + // Terminate draft requests whose logits have been sent by the background thread. + { + RequestVector doneSending; + { + std::lock_guard<std::mutex> lk(mDraftRequestsMtx); + doneSending.swap(mDraftRequestsDoneSendingLogits); + } + for (auto const& llmReq : doneSending) + { + terminateRequest(llmReq); + } + } + + // Finished context requests have been moved to generationRequests by moveFinishedContextRequestsToGeneration + for (auto const& llmReq : currRequests.generationRequests) + { + // If a context-only request is finished, send its KV cache and mark it. + if (llmReq->isContextOnlyRequest() && llmReq->isContextFinished()) + { + // TODO: skip if sending layer-wise + { + TLLM_CHECK_WITH_INFO(mCacheTransceiver, + "Disaggregated serving is not enabled, please check the configuration of " + "cacheTransceiverConfig."); + mCacheTransceiver->respondAndSendAsync(llmReq); + } + mSeqSlotManager->freeSequenceSlot(llmReq->mRequestId); + } + } + } + // report profile data + auto const bufferId = getFusedBufferId(); + auto const contextId = mBuffers[bufferId]->getContextIndex(); + if (mRuntime->hasLayerProfiler(contextId)) + { + mRuntime->reportToProfiler(contextId); + } + if (mCacheTransceiver) + { + mCacheTransceiver->checkContextTransferStatus(0, true); + } + ++mIterCounter; + + if (mKvCacheManager) + { + mKvCacheManager->flushIterationEvents(); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::storeContextBlocks(std::shared_ptr<LlmRequest> const& llmReq) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // TMJ - Note + // Make context blocks reusable immediately after context phase finishes. + // For chunked contexts, this occurs in step that processes last context chunk. + // isLastContextChunk() is always true for non-chunked contexts. + // This check is made in code that calls storeContextBlocks, so omitted here. + if (mKvCacheManager) + { + mKvCacheManager->storeContextBlocks(*llmReq); + } + if (mCrossKvCacheManager) + { + mCrossKvCacheManager->storeContextBlocks(*llmReq); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::storeNewBlock(std::shared_ptr<LlmRequest> const& llmReq) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // TMJ - Note + // Make context blocks reusable immediately after each generation step. + + if (mKvCacheManager) + { + mKvCacheManager->storeNewBlock(*llmReq); + } + if (mCrossKvCacheManager) + { + mCrossKvCacheManager->storeNewBlock(*llmReq); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::resetIterationStats() +{ + mLastIterationStatsIFB = IterationStatsIFB{mMicroBatchId}; +} + +void TrtGptModelInflightBatching::forwardAsync(RequestList const& activeRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE_WITH_NAME(range, "TrtGptModelInflightBatching::forwardAsync"); + + TLLM_CUDA_CHECK(cudaSetDevice(mWorldConfig.getDevice())); + + try + { + verifyRequests(activeRequests); + if (mModelConfig.isTransformerBased() && getKVCacheManager() && mCacheTransceiver) + { + checkDisaggGenTransferStatus(activeRequests); + } + auto& currRequests = mMicroBatchScheduledRequests.at(mMicroBatchId); + + // Get a new set of requests for that context + // The scheduler will not include any requests that are (i) still in encoder state if encoder-decoder models OR + // (ii) already in flight for decoder models + TLLM_LOG_DEBUG("Running DECODER request scheduler"); + auto [fittingRequests, fittingDisaggGenInitRequests, requestsToPause] + = (*mCapacityScheduler)(activeRequests, mKvCacheManager, mPeftCacheManager, mCrossKvCacheManager); + // Remove from fitting requests the requests that cannot be scheduled due to disagg KV cache transfer + bool waitForDisaggGenTransferProgress = false; + if (mModelConfig.isTransformerBased() && getKVCacheManager() && mCacheTransceiver) + { + if (mDisaggTransferAdmissionController && mDisaggTransferAdmissionController->enabled() + && !fittingDisaggGenInitRequests.empty()) + { + auto admissionResult + = mDisaggTransferAdmissionController->select(activeRequests, fittingDisaggGenInitRequests); + waitForDisaggGenTransferProgress = admissionResult.isBlockedByActiveTransfers(); + if (admissionResult.deferredRequestCount > 0) + { + TLLM_LOG_DEBUG( + "Disagg transfer admission deferred %zu requests; active transfer blocks=%zu, admitted " + "transfer blocks=%zu, budget=%zu", + admissionResult.deferredRequestCount, admissionResult.activeTransferBlocks, + admissionResult.admittedTransferBlocks, + mDisaggTransferAdmissionController->getMaxTransferBlocks().value_or(0)); + } + fittingDisaggGenInitRequests = std::move(admissionResult.admittedRequests); + } + prepareDisaggGenInitRequests(activeRequests, fittingDisaggGenInitRequests); + } + if (fittingRequests.empty() && fittingDisaggGenInitRequests.empty()) + { + TLLM_LOG_WARNING( + "CapacityScheduler didn't schedule any requests in iteration %lu, " + "probably because of insufficient resources such as KV cache, " + "will try wait for KV cache transfer to complete", + mIterCounter); + if (mCacheTransceiver) + { + if (waitForDisaggGenTransferProgress) + { + TLLM_LOG_DEBUG("Waiting for generation KV cache transfer progress to free disagg admission budget"); + mCacheTransceiver->checkGenTransferStatus(1); + } + else + { + mCacheTransceiver->checkContextTransferStatus(1, true); + // will free kvCache in next iteration. + } + } + } + std::tie(currRequests.contextRequests, currRequests.generationRequests) + = (*mMicroBatchScheduler)(fittingRequests, mInflightReqIds, mMaxBatchSizeRuntime, mMaxNumTokensRuntime); + TLLM_CHECK(currRequests.size() <= static_cast<size_t>(getMaxBatchSize())); + + (*mPauseRequests)(requestsToPause, mInflightReqIds, mReqIdsToPause, false, *mSeqSlotManager, mKvCacheManager, + mCrossKvCacheManager, mPeftCacheManager); + + if (mUseSeamlessLookahead) + { + changeSpecDecMode(currRequests); + } + + if (!currRequests.empty()) + { + TLLM_LOG_DEBUG("Running DECODER model with batch size: %lu", currRequests.size()); + // For overlap don't store inflight requests, so they are not skipped in scheduler + if (!isTrtOverlap()) + { + NVTX3_SCOPED_RANGE(updateInflightReqIds); + // Add requests to in-flight set, so they can be skipped in other micro batches + for (auto const& llmReq : currRequests.contextRequests) + { + // Context requests that are chunking are not added to inflight set, so they are scheduled in the + // next micro batch. + if (llmReq->isLastContextChunk()) + { + TLLM_LOG_DEBUG( + "Context request with ID %lu added to DECODER model inflight set", llmReq->mRequestId); + mInflightReqIds.insert(llmReq->mRequestId); + } + } + for (auto const& llmReq : currRequests.generationRequests) + { + TLLM_LOG_DEBUG( + "Generation request with ID %lu added to DECODER model inflight set", llmReq->mRequestId); + mInflightReqIds.insert(llmReq->mRequestId); + } + } + + (*mAssignReqSeqSlots)(*mSeqSlotManager, currRequests.contextRequests, currRequests.generationRequests); + + if (mKvCacheManager) + { + (*mAllocateKvCache)(*mKvCacheManager, currRequests.contextRequests, currRequests.generationRequests, + mModelConfig, mCrossKvCacheManager); + } + + mPeftTables.at(mMicroBatchId) + = mPeftCacheManager->ensureBatch(currRequests.contextRequests, currRequests.generationRequests, true); + + // Do decoder setup before context phase if model needs to setup buffers for the context phase. + if (mModelConfig.getSpeculativeDecodingMode().needsDecoderPrologue()) + { + auto const contextBufferId = mCtxGenFusion ? getFusedBufferId() : getContextBufferId(); + setupDecoderStep(currRequests.contextRequests, *mBuffers.at(contextBufferId), + mDecoderInputBuffers.at(getFusedBufferId())); + // WAR: Sync to ensure that the decoder setup is complete before the context phase starts. + // Without this, there may be a race condition between the decoder setup and the context phase + // which also leads to spurious test failure in trtGptModelRealDecoderTest. + mRuntime->getStream().synchronize(); + } + else + { + prepareDistGenBufferAndDecoder(currRequests.generationRequests); + } + sync_check_cuda_error(mRuntime->getStream().get()); + + executeBatch(currRequests); + if (mWorldConfig.isLastPipelineParallelRank() && mGuidedDecoder) + { + // XGrammar: build maskcache for context requests and perform maskgen for all requests + // These need to be overlapped with the kernel execution of forward step + mGuidedDecoder->build(currRequests); + } + + sync_check_cuda_error(mRuntime->getStream().get()); + + // Postpone decoder setup if model does not need to setup buffers for the context phase. + if (!mModelConfig.getSpeculativeDecodingMode().needsDecoderPrologue()) + { + auto const contextBufferId = mCtxGenFusion ? getFusedBufferId() : getContextBufferId(); + setupDecoderStep(currRequests.contextRequests, *mBuffers.at(contextBufferId), + mDecoderInputBuffers.at(getFusedBufferId())); + } + + sync_check_cuda_error(mRuntime->getStream().get()); + + if (isTrtOverlap()) + { + // WAR: Because the decoder is not stateless (yet) a sync is needed between + // decoder execution and next decoder step preparation. + auto const prevMicroBatchId = getPrevMicroBatchId(mMicroBatchId); + auto& prevDecoderFinishedEvent = mDecoderFinishedEvents.at(prevMicroBatchId); + if (prevDecoderFinishedEvent) + { + prevDecoderFinishedEvent->synchronize(); + } + } + + auto& decoderFinishedEvent = mDecoderFinishedEvents.at(mMicroBatchId); + TLLM_CHECK_WITH_INFO(!decoderFinishedEvent.has_value(), "decoderFinishedEvent must be nullopt."); + decoderFinishedEvent = mWorldConfig.isLastPipelineParallelRank() + ? std::make_optional(decoderStepAsync(currRequests)) + : std::nullopt; + + sync_check_cuda_error(mRuntime->getStream().get()); + + mLastIterationStatsIFB = fillIterationStats(currRequests, requestsToPause); + for (auto const& requests : {currRequests.contextRequests, currRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + if (llmReq->isContextInitState()) + { + llmReq->moveToNextContextChunk(); + if (llmReq->getContextRemainingLength() == 0) + { + TLLM_LOG_DEBUG("[RANK %d] request with ID %lu finishes decoder ctx phase", + COMM_SESSION.getRank(), llmReq->mRequestId); + + llmReq->setState(LlmRequestState::kGENERATION_IN_PROGRESS); + + // for encoder-decoder models, free encoder output buffers after decoder context phase is + // completed + if (llmReq->getEncoderTokens().has_value()) + { + llmReq->freeEncoderOutputBuffers(); + } + storeContextBlocks(llmReq); + + if (isTrtOverlap() && llmReq->willCompleteNextIteration()) + { + // This prohibits the request from being scheduled for another iteration if only one + // iteration is expected. + llmReq->setState(LlmRequestState::kGENERATION_TO_COMPLETE); + } + } + } + else if (llmReq->isGenerationInProgressState()) + { + storeNewBlock(llmReq); + TLLM_LOG_DEBUG("request with ID %lu forwards a step in decoder gen phase", llmReq->mRequestId); + } + } + } + + utils::moveFinishedContextRequestsToGeneration(currRequests); + } + else + { + mLastIterationStatsIFB = IterationStatsIFB{mMicroBatchId}; + } + + if (mWorldConfig.isPipelineParallel() && mWorldConfig.isLastPipelineParallelRank()) + { + mAsyncSendWaitThread->waitStop(); + if (!currRequests.empty()) + { + for (auto& hdl : mDecStepAsyncSndHdls) + { + TLLM_CHECK_WITH_INFO(hdl.get() == nullptr, "decoderSync handle must be nullptr."); + } + // Wait for decoding for requests in flight for the current micro batch + auto& decoderFinishedEvent = mDecoderFinishedEvents.at(mMicroBatchId); + mDecStepAsyncSndHdls = decoderSync(currRequests, decoderFinishedEvent); + decoderFinishedEvent.reset(); + + mAsyncSendWaitThread->notifyStart(); + } + } + + // Update the micro batch ID + mMicroBatchId = getNextMicroBatchId(mMicroBatchId); + } + // In case of error, we need to free the batch slot associated with those requests + catch (std::exception const&) + { + try + { + for (auto const& llmReq : activeRequests) + { + // Remove from mInflightReqIds so changeBeamWidth can proceed on the next iteration. + // terminateRequest frees seqSlot/KV cache but does not clean up mInflightReqIds. + mInflightReqIds.erase(llmReq->mRequestId); + terminateRequest(llmReq); + } + // Force buffer/decoder reset to clean up any partial state from the aborted batch + // (e.g. partially-filled cross-KV block offsets from mid-context-chunk processing). + // Guard on mInflightReqIds.empty(): in pipeline-parallel multi-micro-batch mode, + // other micro-batches may still have requests tracked here; changeBeamWidth asserts + // emptiness so we skip the reset and let the next successful forwardAsync iteration + // perform it when the set is clear. + if (mWorldConfig.isLastPipelineParallelRank() && mInflightReqIds.empty()) + { + changeBeamWidth(mOperatingBeamWidth); + } + } + catch (std::exception const& e) + { + TLLM_LOG_ERROR("forwardAsync catch-all catch block that runs `terminateRequest` has failed with:"); + TLLM_LOG_EXCEPTION(e); + TLLM_LOG_ERROR("Rethrowing *outer* exception:"); + } + throw; + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::setRuntimeBatchSize(SizeType32 runtimeMaxBatchSize) +{ + mMaxBatchSizeTunerRecommended = runtimeMaxBatchSize; + mMaxBatchSizeRuntime = std::min(getMaxBatchSize(), runtimeMaxBatchSize); +} + +SizeType32 TrtGptModelInflightBatching::getRuntimeBatchSize() const +{ + return mMaxBatchSizeRuntime; +} + +void TrtGptModelInflightBatching::setRuntimeMaxNumTokens(SizeType32 runtimeMaxNumTokens) +{ + mMaxNumTokensTunerRecommended = runtimeMaxNumTokens; + mMaxNumTokensRuntime + = (mMaxNumTokensStatic) ? std::min(mMaxNumTokensStatic.value(), runtimeMaxNumTokens) : runtimeMaxNumTokens; +} + +void TrtGptModelInflightBatching::updatePeftCache(std::shared_ptr<LlmRequest> const& llmRequest) +{ + mPeftCacheManager->addRequestPeft(llmRequest, true); +} + +runtime::BufferManager const& TrtGptModelInflightBatching::getBufferManager() const +{ + return mRuntime->getBufferManager(); +} + +BufferManager::CudaStreamPtr TrtGptModelInflightBatching::getRuntimeStreamPtr() const +{ + return mRuntime->getStreamPtr(); +} + +void TrtGptModelInflightBatching::executeContext(SizeType32 runtimeContextId, SizeType32 bufferId) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(executeContext); + + auto const& currBatchState = mBuffers[bufferId]->getBatchState(); + + bool hasCudaGraph = false; + // If batch state is context only, do not capture/launch graph and execute the engine as is. + if (isCudaGraphMode() && !currBatchState.isAnyContext()) + { + auto cudaGraphOpt = mCudaGraphExecutorCaches[bufferId].get(currBatchState); + // If graph exists for current batch state, launch it. + if (cudaGraphOpt.has_value()) + { + hasCudaGraph = true; + } + } + + // If there is no graph for current state, execute the engine. + if (!hasCudaGraph) + { + auto enqueueSuccessful = mRuntime->executeContext(runtimeContextId); + if (!enqueueSuccessful) + { + throw std::runtime_error("Executing TRT engine failed!"); + } + } + else + { + // Launch graph. + auto cudaGraphOpt = mCudaGraphExecutorCaches[bufferId].get(currBatchState); + cudaGraphOpt.value()->launch(mRuntime->getStream()); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::setLayerProfiler() +{ + mRuntime->setLayerProfiler(); +} + +std::string TrtGptModelInflightBatching::getLayerProfileInfo() const +{ + return mRuntime->getLayerProfileInfo(); +} + +void TrtGptModelInflightBatching::verifyRequests(RequestList const& activeRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(verifyRequests); + + if (activeRequests.empty()) + { + return; + } + + auto const& firstRequest = activeRequests.front(); + auto const firstRequestId = firstRequest->mRequestId; + auto const firstBeamWidth = firstRequest->mSamplingConfig.beamWidth; + + for (auto const& llmReq : activeRequests) + { + auto const beamWidth = llmReq->mSamplingConfig.beamWidth; + auto const draftLength = llmReq->getNumDraftTokens(); + auto const maxDraftLength = mModelConfig.getMaxDecodingDraftTokens(); + + TLLM_CHECK_WITH_INFO(beamWidth == 1 || draftLength == 0, "Can't use speculative decoding with beam search."); + TLLM_CHECK_WITH_INFO(draftLength <= maxDraftLength, + "Number of draft tokens (%d) is larger than maximum number of draft tokens (%d)", draftLength, + maxDraftLength); + + // FIXME: Remove this check when varying beam width is supported + { + TLLM_CHECK_WITH_INFO(beamWidth == firstBeamWidth, + "All active requests must have same beam width, " + "but request %lu with beam width %d differs from first request %lu with beam width %d", + llmReq->mRequestId, beamWidth, firstRequestId, firstBeamWidth); + } + } + + if (firstBeamWidth != mOperatingBeamWidth) + { + changeBeamWidth(firstBeamWidth); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::executeBatch(ScheduledRequests const& scheduledRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(executeBatch); + + if (!mCtxGenFusion) + { + if (!scheduledRequests.contextRequests.empty()) + { + auto const bufferId = getContextBufferId(); + executeStep(scheduledRequests.contextRequests, {}, bufferId); + } + if (!scheduledRequests.generationRequests.empty()) + { + auto const bufferId = getGenerationBufferId(); + executeStep({}, scheduledRequests.generationRequests, bufferId); + } + } + else + { + auto const bufferId = getFusedBufferId(); + executeStep(scheduledRequests.contextRequests, scheduledRequests.generationRequests, bufferId); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::createRuntimeContexts() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + mRuntime->clearContexts(); + auto const numProfiles = mRuntime->getNbProfiles(); + for (auto i = 0; i < numProfiles; ++i) + { + mRuntime->addContext(i); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +namespace +{ +// TODO: move this somewhere else? +/** + * This function logic is also implemented in tensorrt_llm/_torch/pyexecutor/_util.py get_decoding_mode(). + */ +executor::DecodingMode getDecodingMode(SpeculativeDecodingMode specDecodingMode, + std::optional<executor::DecodingMode> const& decodingModeOpt, runtime::SizeType32 const beamWidth) +{ + auto getDefaultDecodingMode = [beamWidth](std::optional<executor::DecodingMode> const& decodingModeOpt) + { + if (decodingModeOpt.has_value() && !decodingModeOpt->isAuto()) + { + return decodingModeOpt.value(); + } + return (beamWidth == 1) ? executor::DecodingMode::TopKTopP() : executor::DecodingMode::BeamSearch(); + }; + + auto decodingMode = getDefaultDecodingMode(decodingModeOpt); + // Variable-Beam-Width-Search (special mode of Beam-Search) is enabled. + if (decodingMode.isBeamSearch() && decodingMode.isUseVariableBeamWidthSearch()) + { + TLLM_LOG_INFO("Variable-Beam-Width-Search is enabled"); + } + // Overwrite decoding mode when beam width is one. + if (beamWidth == 1 && decodingMode.isBeamSearch()) + { + TLLM_LOG_WARNING( + "Beam width is set to 1, but decoding mode is BeamSearch. Overwriting decoding mode to TopKTopP."); + decodingMode = executor::DecodingMode::TopKTopP(); + } + // Overwrite decoding mode when Medusa is used. + if (specDecodingMode.isMedusa() && !decodingMode.isMedusa()) + { + TLLM_LOG_WARNING("Model is Medusa, but decoding mode is not Medusa. Overwriting decoding mode to Medusa."); + decodingMode = executor::DecodingMode::Medusa(); + } + // Overwrite decoding mode when Medusa is not used. + if (!specDecodingMode.isMedusa() && decodingMode.isMedusa()) + { + TLLM_LOG_WARNING("Model is not Medusa, but decoding mode is Medusa. Overwriting decoding mode."); + decodingMode = getDefaultDecodingMode(decodingModeOpt); + } + // Overwrite decoding mode when lookahead decoding is used. + if (specDecodingMode.isLookaheadDecoding() && !decodingMode.isLookahead()) + { + TLLM_LOG_WARNING( + "Model is Lookahead, but decoding mode is not Lookahead. Overwriting decoding mode to Lookahead."); + decodingMode = executor::DecodingMode::Lookahead(); + } + // Overwrite decoding mode when lookahead decoding is not used. + if (!specDecodingMode.isLookaheadDecoding() && decodingMode.isLookahead()) + { + TLLM_LOG_WARNING( + "Model is not built with Lookahead decoding, but decoding mode is Lookahead. Overwriting decoding " + "mode."); + decodingMode = getDefaultDecodingMode(decodingModeOpt); + } + // Overwrite decoding mode when 'explicit draft tokens' is used. + if (specDecodingMode.isExplicitDraftTokens() && !decodingMode.isExplicitDraftTokens()) + { + TLLM_LOG_WARNING( + "Model is built with 'explicit draft tokens' decoding, but decoding mode is something else. Overwriting " + "decoding mode."); + decodingMode = executor::DecodingMode::ExplicitDraftTokens(); + } + // Overwrite decoding mode when 'explicit draft tokens' is not used. + if (!specDecodingMode.isExplicitDraftTokens() && decodingMode.isExplicitDraftTokens()) + { + TLLM_LOG_WARNING( + "Model is not built with 'explicit draft tokens' decoding, but decoding mode is set to it. Overwriting " + "decoding " + "mode to default."); + decodingMode = getDefaultDecodingMode(decodingModeOpt); + } + // Overwrite decoding mode when EAGLE is used. + if (specDecodingMode.isEagle() && !decodingMode.isEagle()) + { + TLLM_LOG_WARNING("Model is Eagle, but decoding mode is not Eagle. Overwriting decoding mode to Eagle."); + decodingMode = executor::DecodingMode::Eagle(); + } + // Overwrite decoding mode when Eagle is not used. + if (!specDecodingMode.isEagle() && decodingMode.isEagle()) + { + TLLM_LOG_WARNING("Model is not Eagle, but decoding mode is Eagle. Overwriting decoding mode."); + decodingMode = getDefaultDecodingMode(decodingModeOpt); + } + if (specDecodingMode.isDraftTokensExternal()) + { + TLLM_LOG_WARNING("Overwriting decoding mode to external draft token"); + decodingMode = executor::DecodingMode::ExternalDraftTokens(); + } + TLLM_LOG_DEBUG("DecodingMode: %s", decodingMode.getName()); + return decodingMode; +} +} // namespace + +void TrtGptModelInflightBatching::createDecoder(std::optional<executor::DecodingMode> const& decodingModeOpt) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + mDecoderState = std::make_unique<runtime::decoder::DecoderState>(); + + if (mWorldConfig.isLastPipelineParallelRank()) + { + auto decoderType = mRuntime->getEngine().getTensorDataType("logits"); + + auto const decodingMode + = getDecodingMode(mModelConfig.getSpeculativeDecodingMode(), decodingModeOpt, mOperatingBeamWidth); + + if (decodingMode.isExplicitDraftTokens()) + { + // There are no logits in Explicit draft tokens model. + decoderType = mModelConfig.getDataType(); + // Decoder is not instantiated for bf16. We use half to get the same data size + // and explicitly pass dtype to redrafter that has bf16 kernels. + if (decoderType == nvinfer1::DataType::kBF16) + { + decoderType = nvinfer1::DataType::kHALF; + } + } + + mDecoder = std::make_unique<runtime::GptDecoderBatched>(mRuntime->getStreamPtr()); + mDecoder->setup( + decodingMode, getMaxNumSequences(), mOperatingBeamWidth, decoderType, mModelConfig, mWorldConfig); + + mDecoderState->setup(getMaxNumSequences(), mOperatingBeamWidth, getMaxAttentionWindow(), getSinkTokenLen(), + getMaxSequenceLen(), decoderType, mModelConfig, mWorldConfig, mRuntime->getBufferManager()); + + if (!mModelConfig.getSpeculativeDecodingMode().isNone()) + { + mDecoderState->setupSpeculativeDecoding(mModelConfig.getSpeculativeDecodingMode(), + mModelConfig.getMaxDecodingTokens(), decoderType, mModelConfig, mWorldConfig, + mRuntime->getBufferManager()); + } + } + else + { + mDecoderState->setupCacheIndirection( + getMaxNumSequences(), mOperatingBeamWidth, getMaxAttentionWindow(), mRuntime->getBufferManager()); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::createBuffers(executor::DecodingConfig const& decodingConfig, + std::optional<std::vector<executor::AdditionalModelOutput>> const& additionalModelOutputs) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + mBuffers.clear(); + for (SizeType32 i = 0; i < mNumBuffers; ++i) + { + mBuffers.emplace_back( + std::make_unique<RuntimeBuffers>(getMaxBatchSize(), mOperatingBeamWidth, getMaxAttentionWindowVec(), + getMaxAttentionWindow(), getSinkTokenLen(), *mRuntime, mModelConfig, mWorldConfig, decodingConfig, + getGatherGenerationLogits(), getMaxNumTokens(), additionalModelOutputs, mPromptTableOffloading)); + } + + mDecoderInputBuffers.clear(); + mDecoderOutputBuffers.clear(); + for (SizeType32 i = 0; i < mNumMicroBatches; ++i) + { + mDecoderInputBuffers.emplace_back( + getMaxBatchSize(), mModelConfig.getMaxDecodingTokens(), mRuntime->getBufferManager()); + mDecoderInputBuffers.back().setupMedusaLogits(getMaxNumSequences(), mModelConfig); + mDecoderOutputBuffers.emplace_back(getMaxNumSequences(), mOperatingBeamWidth, getMaxSequenceLen(), + mModelConfig.getMaxDecodingTokens(), mRuntime->getBufferManager()); + mDecoderOutputBuffers.back().setupSpeculativeDecoding( + getMaxNumSequences(), mModelConfig.getMaxDecodingTokens(), mModelConfig); + } + + mSlotDecoderBuffers.clear(); + for (SizeType32 i = 0; i < getMaxNumSequences(); ++i) + { + mSlotDecoderBuffers.emplace_back(std::make_unique<SlotDecoderBuffers>( + mOperatingBeamWidth, getMaxSequenceLen(), mRuntime->getBufferManager())); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::prepareDisaggGenInitRequests( + RequestList const& activeRequests, RequestVector& newGenReqs) +{ + NVTX3_SCOPED_RANGE(prepareDisaggGenInitRequests); + + // Allocate KV cache by treating them as context requests + (*mAllocateKvCache)(*mKvCacheManager, newGenReqs, {}, mModelConfig, mCrossKvCacheManager); + + // Initiate KV cache transfer + auto timeStart = std::chrono::steady_clock::now(); + + if (tc::getEnvDisaggBenchmarkGenOnly()) + { + TLLM_LOG_DEBUG("Disaggregated generation only benchmark mode is enabled"); + for (auto& req : newGenReqs) + { + req->setState(LlmRequestState::kDISAGG_GENERATION_TRANS_COMPLETE); + } + return; + } + + auto const genInitReqNum = std::count_if(activeRequests.begin(), activeRequests.end(), + [](auto const& req) { return req->isDisaggGenerationInitState(); }); + + // Loop over the new disagg gen requests and trigger receive of KV cache + for (auto& newGenReq : newGenReqs) + { + TLLM_CHECK_WITH_INFO( + mCacheTransceiver, "Disaggregated serving is not enabled, please check the configuration."); + if (common::getEnvDisableKVCacheTransferOverlap()) + { + mCacheTransceiver->requestAndReceiveSync(newGenReq); + } + else + { + mCacheTransceiver->requestAndReceiveAsync(newGenReq); + } + } + if (!common::getEnvDisableKVCacheTransferOverlap()) + { + auto const blockTransfer = std::all_of(activeRequests.begin(), activeRequests.end(), + [](auto const& req) { return req->isDisaggGenerationTransmissionInProgress(); }); + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), + "newGenReqs.size():%ld requests, activeRequests.size():%ld allTransferInProgress:%d original " + "gen_only_requests_num:%ld", + newGenReqs.size(), activeRequests.size(), blockTransfer, genInitReqNum); + mCacheTransceiver->checkGenTransferStatus(0); + auto timeEnd = std::chrono::steady_clock::now(); + auto duration = std::chrono::duration<float, std::milli>(timeEnd - timeStart).count(); + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), + "receiveDisaggGenCache time:%f ms, " + "blockTransfer:%d,genInitReqNum:%ld,newGenReqs.size():%ld,activeRequests.size():%ld", + duration, blockTransfer, genInitReqNum, newGenReqs.size(), activeRequests.size()); + } + + return; +} + +void TrtGptModelInflightBatching::checkDisaggGenTransferStatus(RequestList const& activeRequests) +{ + NVTX3_SCOPED_RANGE(checkDisaggGenTransferStatus); + + if (common::getEnvDisableKVCacheTransferOverlap()) + { + return; + } + + auto timeStart = std::chrono::steady_clock::now(); + + // TODO: + auto const needCheck = std::any_of(activeRequests.begin(), activeRequests.end(), + [](auto const& req) { return req->isDisaggGenerationTransmissionInProgress(); }); + + if (needCheck) + { + mCacheTransceiver->checkGenTransferStatus(0); + + auto timeEnd = std::chrono::steady_clock::now(); + auto duration = std::chrono::duration<float, std::milli>(timeEnd - timeStart).count(); + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), + "no Prepare checkDisaggGenTransferStatus time:%f ms, " + "needCheck:%d,activeRequests.size():%ld", + duration, needCheck, activeRequests.size()); + } +} + +void TrtGptModelInflightBatching::prepareDistGenBufferAndDecoder(RequestVector const& generationRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // set decoderStep for disagg_generation + RequestVector cacheTransCompleteRequests; + for (auto const& request : generationRequests) + { + if (request->isDisaggGenerationTransmissionComplete()) + { + cacheTransCompleteRequests.push_back((request)); + } + } + if (!cacheTransCompleteRequests.empty()) + { + auto timeStart = std::chrono::steady_clock::now(); + auto const bufferId = getFusedBufferId(); + auto& runtimeBuffers = *mBuffers[bufferId]; + runtimeBuffers.prepareStep(cacheTransCompleteRequests, {}, getMaxBeamWidth(), getMaxAttentionWindow(), + *mDecoderState, mKvCacheManager.get(), mCrossKvCacheManager.get(), mRnnStateManager.get(), + mPeftTables[mMicroBatchId], *mRuntime, mModelConfig, mWorldConfig, getGatherGenerationLogits(), + isTrtOverlap()); + auto const contextBufferId = mCtxGenFusion ? getFusedBufferId() : getContextBufferId(); + setupDecoderStep( + cacheTransCompleteRequests, *mBuffers.at(contextBufferId), mDecoderInputBuffers.at(getFusedBufferId())); + sync_check_cuda_error(mRuntime->getStream().get()); + auto timeEnd = std::chrono::steady_clock::now(); + auto duration = std::chrono::duration<float, std::milli>(timeEnd - timeStart).count(); + TLLM_LOG_DEBUG(mpi::MpiComm::world().getRank(), + "prepareDistGenBufferAndDecoder time:%f ms , cacheTransCompleteRequests.size():%ld", duration, + cacheTransCompleteRequests.size()); + } + for (auto& request : cacheTransCompleteRequests) + { + request->setState(LlmRequestState::kGENERATION_IN_PROGRESS); + request->setContextCurrentPosition(request->mPromptLen); + request->setDecodingIter(1); + auto const reqBeamWidth = request->mSamplingConfig.beamWidth; + auto firstGenTokens = request->getContextPhaseParams().value().getFirstGenTokens(); + for (SizeType32 beam = 0; beam < reqBeamWidth; ++beam) + { + request->addNewToken(firstGenTokens.at(beam), beam); + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::debugIOTensors(RequestVector const& contextRequests, + RequestVector const& generationRequests, TensorMap const& inputMap, TensorMap const& outputMap) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + TLLM_CHECK(mDebugConfig); + + auto const& manager = mRuntime->getBufferManager(); + auto requestIds = utils::collectRequestIds(contextRequests, generationRequests); + + if (mDebugConfig->getDebugTensorsMaxIterations() > 0) + { + mLastIterationDebugTensors.clear(); + mLastIterationDebugTensors = utils::storeIOTensors(*mDebugConfig, requestIds, inputMap, outputMap, manager); + } + else + { + utils::dumpIOTensors(*mDebugConfig, mIterCounter, requestIds, inputMap, outputMap, mWorldConfig, manager); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +std::tuple<SizeType32, runtime::StringPtrMap<runtime::ITensor> const&, runtime::StringPtrMap<runtime::ITensor>&> +TrtGptModelInflightBatching::prepareBuffers( + RequestVector const& contextRequests, RequestVector const& generationRequests, SizeType32 bufferId) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(prepareBuffers); + + auto& runtimeBuffers = *mBuffers.at(bufferId); + + auto allNewTokens = mWorldConfig.isLastPipelineParallelRank() + ? RuntimeBuffers::OptionalRef<runtime::ITensor const>(mDecoderState->getAllNewTokens()) + : std::nullopt; + + auto [optProfileId, inputMap, outputMap] = runtimeBuffers.prepareStep(contextRequests, generationRequests, + mOperatingBeamWidth, getMaxAttentionWindow(), *mDecoderState, mKvCacheManager.get(), mCrossKvCacheManager.get(), + mRnnStateManager.get(), mPeftTables[bufferId], *mRuntime, mModelConfig, mWorldConfig, + getGatherGenerationLogits(), isTrtOverlap(), allNewTokens); + + // For Variable-Beam-Width-Search + mRuntime->setCurrentBeamWidths( + tensorrt_llm::batch_manager::utils::getRequestBeamWidths(contextRequests, generationRequests)); + + mRuntime->setInputTensors(optProfileId, inputMap); + mRuntime->setOutputTensors(optProfileId, outputMap); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return {optProfileId, inputMap, outputMap}; +} + +void TrtGptModelInflightBatching::prepareGraph(SizeType32 bufferId, SizeType32 optProfileId) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(prepareGraph); + + auto const nextBatchState = mBuffers[bufferId]->getBatchState(); + auto cudaGraphOpt = mCudaGraphExecutorCaches[bufferId].get(nextBatchState); + // If graph is not found in the cache, capture it. + if (!cudaGraphOpt.has_value()) + { + // We need to prepare some tensors once again to properly set values for graph capture. + // Graph capture requires setting some tensors (e.g. past_kv_len) + // to the round_up(max_kv_cache_len, kKV_CACHE_LEN_CUDA_GRAPH_ROUND_SIZE) + // in order to capture the kernels with the large enough grid. + mBuffers[bufferId]->prepareBuffersForCudaGraph(getMaxSequenceLen()); + + auto cudaGraph = std::make_shared<utils::CudaGraphExecutor>(); + cudaGraph->prepareNextGraph(mRuntime, optProfileId); + mCudaGraphExecutorCaches[bufferId].put(nextBatchState, cudaGraph); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::executeStep( + RequestVector const& contextRequests, RequestVector const& generationRequests, SizeType32 bufferId) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE_WITH_NAME(range, + "executeStep: " + std::to_string(contextRequests.size()) + " ctx reqs, " + + std::to_string(generationRequests.size()) + " gen reqs"); + + if (mPromptTableOffloading) + { + prefetchNextPromptTableChunk(contextRequests, /* isFirstChunk */ true, bufferId); + } + + auto [optProfileId, inputMap, outputMap] = prepareBuffers(contextRequests, generationRequests, bufferId); + + if (mBuffers[bufferId]->transformerBuffers) + { + // Creation of context progress, or remains nullptr if not needed + std::shared_ptr<ContextProgress> progress = nullptr; + RequestVector layerWiseRequests; + if (common::getEnvDisaggLayerwise()) + { + for (auto const& request : contextRequests) + { + bool const enableLayerWise = request->isContextOnlyRequest() && request->isLastContextChunk(); + if (enableLayerWise) + { + layerWiseRequests.push_back(request); + } + } + } + // TODO: support layer-wise cross kv cache in encoder-decoder models + if (!layerWiseRequests.empty() && !mModelConfig.useCrossAttention()) + { + int const numLayers = mModelConfig.getNbAttentionLayers( + mWorldConfig.getPipelineParallelism(), mWorldConfig.getPipelineParallelRank()); + progress = std::make_shared<ContextProgress>(numLayers); + } + bufferCast<void*>(*mBuffers[bufferId]->transformerBuffers->contextProgressHost)[0] = progress.get(); + if (progress) + { + TLLM_CHECK_WITH_INFO(mCacheTransceiver, + "Disaggregated serving is not enabled, please check the configuration of cacheTransceiverConfig."); + mCacheTransceiver->respondAndSendLayerWise(layerWiseRequests, progress); + } + } + + if (mPromptTableOffloading) + { + prefetchNextPromptTableChunk(contextRequests, /* isFirstChunk */ false, bufferId); + } + + executeContext(optProfileId, bufferId); + + // If batch state has any context request, do not capture this graph. + if (isCudaGraphMode() && contextRequests.empty()) + { + // Capture graph of current batch state during engine execution. + // This is based on the assumptions that + // a) We can hide CPU graph capture behind the GPU engine execution. + // b) Batch size in the next iterations won't change and we can reuse the graph multiple times. + prepareGraph(bufferId, optProfileId); + } + + if (mDebugConfig) + { + debugIOTensors(contextRequests, generationRequests, inputMap, outputMap); + } + + if (mAdditionalModelOutputs.has_value() && !mAdditionalModelOutputs.value().empty()) + { + utils::copyAdditionalOutputs( + mAdditionalModelOutputs.value(), contextRequests, generationRequests, outputMap, getBufferManager()); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::setupDecoderStep( + RequestVector const& contextRequests, RuntimeBuffers const& buffers, DecoderInputBuffers& inputBuffers) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(setupDecoderStep); + + if (mWorldConfig.isLastPipelineParallelRank() && !contextRequests.empty()) + { + auto const logitsType = mRuntime->getEngine().getTensorDataType("logits"); + + auto [batchSlots, samplingConfigs, lookaheadPrompt, lookaheadAlgoConfigs] + = (*mCreateNewDecoderRequests)(mModelConfig, mWorldConfig, mDecodingConfig, contextRequests, logitsType, + inputBuffers, *mDecoderState, mRuntime->getStream(), *mDecoder->getDecoderStream(), getMaxSequenceLen(), + mOperatingBeamWidth, buffers.mMedusaBuffers); + + auto const localBatchSize = batchSlots->getSize(); + if (localBatchSize > 0) + { + auto samplingConfig = SamplingConfig(samplingConfigs); + mDecoder->getUnderlyingDecoder().setup(samplingConfig, localBatchSize, batchSlots, + {mDecoderState->getJointDecodingOutput()}, mModelConfig.getDataType(), lookaheadPrompt, + lookaheadAlgoConfigs); + + auto const& stream = mDecoder->getDecoderStream(); + CudaEvent event{}; + stream->record(event); + mRuntime->getStreamPtr()->wait(event); + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::postProcessRequest( + LlmRequest& llmReq, std::vector<SizeType32> const& numDroppedTokens) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + auto const seqSlot = llmReq.mSeqSlot.value(); + auto const reqBeamWidth = llmReq.getBeamWidthByIter(true); + auto const& bufferManager = getBufferManager(); + + if (llmReq.getReturnGenerationLogits() && !llmReq.getGenerationLogitsFragments().empty()) + { + TLLM_CHECK(!llmReq.isStreaming()); + auto const genBufferId = mCtxGenFusion ? getFusedBufferId() : getGenerationBufferId(); + auto& genRuntimeBuffers = *mBuffers.at(genBufferId); + + auto constexpr beforeDecoder = false; + utils::copyGenerationLogits( + genRuntimeBuffers.generationLogitsCache, bufferManager, llmReq, beforeDecoder, numDroppedTokens); + + bufferManager.getStream().synchronize(); + } + + if (reqBeamWidth == 1) + { + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return; + } + + // Update mDecoderBuffers->slotOutputIdsHost and synchronize + getDecoderSlotHostOutputs(seqSlot, llmReq.returnLogProbs(), llmReq.mSamplingConfig, llmReq.isStreaming()); + + auto const* outputIdsHostData = bufferCast<TokenIdType>(*mSlotDecoderBuffers[seqSlot]->outputIdsHost); + auto const* sequenceLengthsHostData = bufferCast<SizeType32>(*mSlotDecoderBuffers[seqSlot]->sequenceLengthsHost); + auto const* cumLogProbsHostData = bufferCast<float>(*mSlotDecoderBuffers[seqSlot]->cumLogProbsHost); + auto logProbsHost = mSlotDecoderBuffers[seqSlot]->logProbsHost; + auto const* logProbsHostData = bufferCast<float>(*logProbsHost); + + auto const& outputIdsShape = mSlotDecoderBuffers[seqSlot]->outputIdsHost->getShape(); + auto const maxSeqLength = outputIdsShape.d[1]; + + std::vector<std::vector<TokenIdType>> generatedTokens(reqBeamWidth); + for (SizeType32 beam = 0; beam < reqBeamWidth; ++beam) + { + auto const* const begin = outputIdsHostData + tc::flat_index2(beam, llmReq.mPromptLen, maxSeqLength); + auto const generatedLength = sequenceLengthsHostData[beam] - llmReq.mPromptLen; + auto const* const end = begin + generatedLength; + generatedTokens[beam].assign(begin, end); + + if (llmReq.returnLogProbs()) + { + llmReq.setCumLogProb(cumLogProbsHostData[beam], beam); + + auto const beginLogProbsOffset = reqBeamWidth == 1 ? llmReq.mPromptLen : 0; + auto const* const begin = logProbsHostData + beam * logProbsHost->getShape().d[1] + beginLogProbsOffset; + auto const* const end = begin + generatedLength; + LlmRequest::VecLogProbs logProbs(begin, end); + llmReq.setLogProbs(logProbs, beam); + } + } + + // store the generated tokens into the mTokensGathered buffer + llmReq.setGeneratedTokens(generatedTokens); + + if (llmReq.getReturnGenerationLogits() && llmReq.getGenerationLogitsHost() + && mWorldConfig.isLastPipelineParallelRank()) + { + reorderGenerationLogitsForBeamSearch( + llmReq, seqSlot, reqBeamWidth, maxSeqLength, outputIdsHostData, sequenceLengthsHostData); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::reorderGenerationLogitsForBeamSearch(LlmRequest& llmReq, SizeType32 seqSlot, + SizeType32 reqBeamWidth, SizeType32 maxSeqLength, TokenIdType const* outputIdsHostData, + SizeType32 const* sequenceLengthsHostData) +{ + // Reorder generation logits to match the gathered (finalized) beam ordering. + // During generation, logits are stored indexed by beam SLOT position. After beam search + // finalization (gatherTree), output_ids are reordered by tracing parentIds to reconstruct + // the correct beam paths. However, generation_logits are NOT reordered by gatherTree. + // We fix this here by tracing parentIds on the host to build the beam-slot mapping, + // then reindexing the logits accordingly. + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const promptLen = llmReq.mPromptLen; + + // Copy parentIds and ids (ungathered step IDs) from GPU to temporary host buffers. + // parentIds[slot][t] = the parent slot of beam slot `slot` at position t. + // ids[slot][t] = the token in beam slot `slot` at position t (before gather). + auto parentIdsDevice = ITensor::at(mDecoderState->getParentIds(), {seqSlot}); + auto idsDevice = mDecoderState->getIds(seqSlot); + + auto parentIdsHost = runtime::BufferManager::pinnedPool(parentIdsDevice->getShape(), nvinfer1::DataType::kINT32); + auto idsHost = runtime::BufferManager::pinnedPool(idsDevice->getShape(), nvinfer1::DataType::kINT32); + + mCopyBufferManager.copy(*parentIdsDevice, *parentIdsHost); + mCopyBufferManager.copy(*idsDevice, *idsHost); + mCopyBufferManager.getStream().synchronize(); + + auto const* parentIdsData = bufferCast<TokenIdType>(*parentIdsHost); + auto const* idsData = bufferCast<TokenIdType>(*idsHost); + + // For each final beam b, find the beam slot at the last generated step, then + // trace back through parentIds to build the slot trace for every generation step. + // slotTrace[beam][genStep] = the beam slot that produced the logits at that step. + auto const generationLogitsHost = llmReq.getGenerationLogitsHost(); + auto const& logitsShape = generationLogitsHost->getShape(); + // Non-streaming shape: [beamWidth, maxNewTokens, vocabSizePadded] + TLLM_CHECK_WITH_INFO(logitsShape.d[0] == reqBeamWidth, + "Generation logits beam dimension (%ld) does not match beam width (%d).", logitsShape.d[0], reqBeamWidth); + auto const maxNewTokens = logitsShape.d[1]; + auto const vocabSizePadded = logitsShape.d[2]; + + std::vector<std::vector<SizeType32>> slotTrace(reqBeamWidth, std::vector<SizeType32>(maxNewTokens, 0)); + bool anyReorderNeeded = false; + + for (SizeType32 beam = 0; beam < reqBeamWidth; ++beam) + { + auto const seqLen = sequenceLengthsHostData[beam]; + auto const genLen = seqLen - promptLen; + if (genLen <= 0) + { + continue; + } + + // Find the starting beam slot at the last generated step by matching the + // backtracked token sequence against the gathered (finalized) output. + SizeType32 startSlot = -1; + for (SizeType32 s = 0; s < reqBeamWidth; ++s) + { + SizeType32 slot = s; + bool matches = true; + for (SizeType32 t = seqLen - 1; t >= promptLen; --t) + { + if (idsData[slot * maxSeqLength + t] != outputIdsHostData[beam * maxSeqLength + t]) + { + matches = false; + break; + } + if (t > promptLen) + { + slot = parentIdsData[slot * maxSeqLength + t]; + } + } + if (matches) + { + startSlot = s; + break; + } + } + + TLLM_CHECK_WITH_INFO(startSlot >= 0, + "Could not determine beam slot mapping for beam %d during generation logits reordering.", beam); + + // Build the slot trace: slotTrace[beam][g] = the pre-reassignment slot whose + // logits correspond to generation step g of this beam. + // + // The model runs BEFORE beam search reassigns beams to slots, so + // generationLogits[slot][g] was produced by the pre-reassignment slot — + // i.e. the slot the beam occupied in the *previous* step. + // parentIds[postSlot][promptLen+g] gives exactly that pre-reassignment slot, + // so taking the parentIds lookup before storing (rather than after) yields + // the correct source slot in a single pass. + SizeType32 slot = startSlot; + for (SizeType32 t = seqLen - 1; t >= promptLen; --t) + { + slot = parentIdsData[slot * maxSeqLength + t]; + slotTrace[beam][t - promptLen] = slot; + } + + // Check if any reordering is actually needed for this beam + auto& slotTraceIds = slotTrace[beam]; + anyReorderNeeded |= std::any_of( + slotTraceIds.begin(), slotTraceIds.begin() + genLen, [beam](SizeType32 s) { return s != beam; }); + } + + // Reorder the generation logits in-place using a per-step temporary buffer. + if (anyReorderNeeded) + { + auto const logitsDataType = generationLogitsHost->getDataType(); + auto const elemSize = runtime::BufferDataType(logitsDataType).getSize(); + auto const stepSize = static_cast<size_t>(vocabSizePadded) * elemSize; + + // Temp buffer for one generation step across all beams: [beamWidth, vocabSizePadded] + auto tempLogits + = runtime::BufferManager::pinnedPool(ITensor::makeShape({reqBeamWidth, vocabSizePadded}), logitsDataType); + + auto* logitsPtr = static_cast<uint8_t*>(generationLogitsHost->data()); + auto* tempPtr = static_cast<uint8_t*>(tempLogits->data()); + + std::vector<SizeType32> genLens(reqBeamWidth); + SizeType32 maxGenLen = 0; + for (SizeType32 b = 0; b < reqBeamWidth; ++b) + { + genLens[b] = std::max(SizeType32{0}, sequenceLengthsHostData[b] - promptLen); + maxGenLen = std::max(maxGenLen, genLens[b]); + } + + for (SizeType32 g = 0; g < maxGenLen; ++g) + { + // Check if any beam that generated this step needs reordering + bool stepNeedsReorder = false; + for (SizeType32 b = 0; b < reqBeamWidth; ++b) + { + if (g < genLens[b] && slotTrace[b][g] != b) + { + stepNeedsReorder = true; + break; + } + } + if (!stepNeedsReorder) + { + continue; + } + + // Copy all beams' logits at this step to the temp buffer + for (SizeType32 b = 0; b < reqBeamWidth; ++b) + { + // logits layout: [beamWidth, maxNewTokens, vocabSizePadded] + auto const offset = (static_cast<size_t>(b) * maxNewTokens + g) * stepSize; + std::memcpy(tempPtr + static_cast<size_t>(b) * stepSize, logitsPtr + offset, stepSize); + } + + // Reorder: logits[b][g] = temp[slotTrace[b][g]] + for (SizeType32 b = 0; b < reqBeamWidth; ++b) + { + if (g >= genLens[b]) + { + continue; + } + auto const dstOffset = (static_cast<size_t>(b) * maxNewTokens + g) * stepSize; + auto const srcSlot = slotTrace[b][g]; + std::memcpy(logitsPtr + dstOffset, tempPtr + static_cast<size_t>(srcSlot) * stepSize, stepSize); + } + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::getDecoderSlotHostOutputs( + SizeType32 seqSlot, bool returnLogProbs, SamplingConfig const& samplingConfig, bool streaming) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + if (mWorldConfig.isLastPipelineParallelRank()) + { + auto event = mDecoder->finalize(*mDecoderState, seqSlot, samplingConfig, streaming); + // Make sure that postprocessing is done before copying outputIds + mCopyBufferManager.getStream().wait(event.get()); + + auto sequenceLengths = mDecoderState->getSequenceLengths(seqSlot); + auto outputIds = mDecoderState->getGatheredIds(seqSlot); + auto cumLogProbs = mDecoderState->getCumLogProbs(seqSlot); + auto logProbs = mDecoderState->getLogProbs(seqSlot); + + mCopyBufferManager.copy(*sequenceLengths, *mSlotDecoderBuffers[seqSlot]->sequenceLengths); + mCopyBufferManager.copy(*outputIds, *mSlotDecoderBuffers[seqSlot]->outputIds); + if (returnLogProbs) + { + mCopyBufferManager.copy(*cumLogProbs, *mSlotDecoderBuffers[seqSlot]->cumLogProbs); + mCopyBufferManager.copy(*logProbs, *mSlotDecoderBuffers[seqSlot]->logProbs); + } + + if (mWorldConfig.isPipelineParallel()) + { + // Make sure that postprocessing is done before sending outputIds + event.synchronize(); + + auto const peerSend = 0; + mDecSlotAsyncSndHdls.emplace_back(std::make_unique<DecoderSlotAsyncSend>( + outputIds, sequenceLengths, cumLogProbs, logProbs, returnLogProbs, *mMpiCommPipelinePara, peerSend)); + } + } + else + { + auto const peerRecv = mWorldConfig.getPipelineParallelRank() == 0 ? mWorldConfig.getPipelineParallelism() - 1 + : mWorldConfig.getPipelineParallelRank() - 1; + DecoderSlotAsyncSend::recv(*mSlotDecoderBuffers[seqSlot], returnLogProbs, *mMpiCommPipelinePara, peerRecv); + + auto const peerSend = mWorldConfig.getPipelineParallelRank() + 1; + if (peerSend != mWorldConfig.getPipelineParallelism() - 1) + { + mDecSlotAsyncSndHdls.emplace_back(std::make_unique<DecoderSlotAsyncSend>( + *mSlotDecoderBuffers[seqSlot], returnLogProbs, *mMpiCommPipelinePara, peerSend)); + } + } + sync_check_cuda_error(mRuntime->getStream().get()); + + // Here copy stream is synchronized after receiving decoderSlotOutputIdsView either by copy or by receive + // before copying to host on copy stream + runtime::CudaEvent beforeEvent{}; + mRuntime->getStreamPtr()->record(beforeEvent); + mCopyBufferManager.getStream().wait(beforeEvent); + mCopyBufferManager.copy(*mSlotDecoderBuffers[seqSlot]->outputIds, *mSlotDecoderBuffers[seqSlot]->outputIdsHost); + mCopyBufferManager.copy( + *mSlotDecoderBuffers[seqSlot]->sequenceLengths, *mSlotDecoderBuffers[seqSlot]->sequenceLengthsHost); + + if (returnLogProbs) + { + mCopyBufferManager.copy( + *mSlotDecoderBuffers[seqSlot]->cumLogProbs, *mSlotDecoderBuffers[seqSlot]->cumLogProbsHost); + mCopyBufferManager.copy(*mSlotDecoderBuffers[seqSlot]->logProbs, *mSlotDecoderBuffers[seqSlot]->logProbsHost); + } + + // Make sure copy is done before continuing on host + mCopyBufferManager.getStream().synchronize(); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +namespace +{ +// Check if one of the request needs log probs, need to get from decoder and communicate +bool batchReturnLogProbs(ScheduledRequests const& scheduledRequests) +{ + auto pred = [](auto const& llmReq) { return llmReq->returnLogProbs(); }; + return std::any_of(scheduledRequests.contextRequests.begin(), scheduledRequests.contextRequests.end(), pred) + || std::any_of(scheduledRequests.generationRequests.begin(), scheduledRequests.generationRequests.end(), pred); +} +} // namespace + +runtime::CudaEvent TrtGptModelInflightBatching::decoderStepAsync(ScheduledRequests const& scheduledRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(decoderStepAsync); + + auto& decoderInputBuffers = mDecoderInputBuffers.at(getFusedBufferId()); + + auto const contextBufferId = mCtxGenFusion ? getFusedBufferId() : getContextBufferId(); + auto& contextRuntimeBuffers = mBuffers.at(contextBufferId); + auto const logitsIndex = (*mHandleContextLogits)(decoderInputBuffers, scheduledRequests.contextRequests, + contextRuntimeBuffers->logits, contextRuntimeBuffers->numContextLogits, mModelConfig, + mRuntime->getBufferManager(), contextRuntimeBuffers->mMedusaBuffers); + + auto const genLogitsIndex = mCtxGenFusion ? logitsIndex : 0; + auto const genBufferId = mCtxGenFusion ? getFusedBufferId() : getGenerationBufferId(); + auto& genRuntimeBuffers = mBuffers.at(genBufferId); + (*mHandleGenerationLogits)(decoderInputBuffers, scheduledRequests.generationRequests, genRuntimeBuffers->logits, + genLogitsIndex, mModelConfig, mRuntime->getBufferManager(), *genRuntimeBuffers, + genRuntimeBuffers->mMedusaBuffers); + + if (mOperatingBeamWidth > 1) + { + copyCacheIndirectionFromOutputsToInputs(scheduledRequests, genBufferId); + } + + mLogitsPostProcessorIsApplied = (*mLogitsPostProcessor)(decoderInputBuffers, mReplicateLogitsPostProcessor, + mWorldConfig, mRuntime->getStreamPtr(), mLogitsPostProcessorBatched); + + if (mGuidedDecoder) + { + mGuidedDecoder->execute(decoderInputBuffers, mRuntime->getBufferManager()); + } + + auto const fusedBufferId = getFusedBufferId(); + auto& fusedRuntimeBuffers = mBuffers.at(fusedBufferId); + + (*mMakeDecodingBatchInputOutput)(decoderInputBuffers, *mDecoderState, mModelConfig, *fusedRuntimeBuffers); + + auto decoderFinishEvent = mDecoder->forwardAsync(*mDecoderState, decoderInputBuffers); + + auto const returnLogProbs = batchReturnLogProbs(scheduledRequests); + auto updateDecoderBuffersEvent = (*mUpdateDecoderBuffers)(mModelConfig, mDecoderOutputBuffers.at(fusedBufferId), + mRuntime->getBufferManager(), *mDecoderState, returnLogProbs, decoderFinishEvent); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return updateDecoderBuffersEvent; +} + +void TrtGptModelInflightBatching::copyCacheIndirectionFromOutputsToInputs( + ScheduledRequests const& scheduledRequests, SizeType32 genBufferId) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(copyCacheIndirectionFromOutputsToInputs); + + auto& genRuntimeBuffers = *mBuffers.at(genBufferId); + auto* srcOffsetsPtr = bufferCast<SizeType64>(*genRuntimeBuffers.cacheIndirDecoderIOBatchedCopySrcOffsets); + auto* dstOffsetsPtr = bufferCast<SizeType64>(*genRuntimeBuffers.cacheIndirDecoderIOBatchedCopyDstOffsets); + auto* copySizesPtr = bufferCast<SizeType64>(*genRuntimeBuffers.cacheIndirDecoderIOBatchedCopySizes); + + // Only `cacheIndirShape.d[2]` is used + auto const& cacheIndirShape = mDecoderState->getCacheIndirectionOutput()->getShape(); + auto const maxBeamWidth = cacheIndirShape.d[1]; + auto const maxAttentionWindow = cacheIndirShape.d[2]; + auto const slotOffset = maxBeamWidth * maxAttentionWindow; + + SizeType32 batchIdx{0}; + SizeType64 maxCopySize{0}; + auto& manager = mRuntime->getBufferManager(); + for (auto const& requests : {scheduledRequests.contextRequests, scheduledRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + auto const seqSlot = llmReq->mSeqSlot.value(); + auto const copySize = reqBeamWidth * maxAttentionWindow; + srcOffsetsPtr[batchIdx] = seqSlot * slotOffset; + dstOffsetsPtr[batchIdx] = seqSlot * slotOffset; + copySizesPtr[batchIdx] = copySize; + maxCopySize = std::max(maxCopySize, copySize); + batchIdx++; + } + } + if (batchIdx != 0) + { + auto const srcOffsetsSlice + = ITensor::slice(genRuntimeBuffers.cacheIndirDecoderIOBatchedCopySrcOffsets, 0, batchIdx); + auto const srcOffsetsSliceDeviceSlice + = ITensor::slice(genRuntimeBuffers.mCacheIndirDecoderIOBatchedCopySrcOffsetsSliceDevice, 0, batchIdx); + manager.copy(srcOffsetsSlice->data(), *srcOffsetsSliceDeviceSlice, + runtime::MemoryType::kGPU); // Explicitly move to device for faster access. + auto const dstOffsetsSlice + = ITensor::slice(genRuntimeBuffers.cacheIndirDecoderIOBatchedCopyDstOffsets, 0, batchIdx); + auto const dstOffsetsSliceDeviceSlice + = ITensor::slice(genRuntimeBuffers.mCacheIndirDecoderIOBatchedCopyDstOffsetsSliceDevice, 0, batchIdx); + manager.copy(dstOffsetsSlice->data(), *dstOffsetsSliceDeviceSlice, + runtime::MemoryType::kGPU); // Explicitly move to device for faster access. + auto const sizesSlice = ITensor::slice(genRuntimeBuffers.cacheIndirDecoderIOBatchedCopySizes, 0, batchIdx); + auto const copySizesDeviceSlice + = ITensor::slice(genRuntimeBuffers.mCacheIndirDecoderIOBatchedCopyCopySizesDevice, 0, batchIdx); + manager.copy(sizesSlice->data(), *copySizesDeviceSlice); // Explicitly move to device for faster access. + runtime::kernels::invokeCopyBatch(*mDecoderState->getCacheIndirectionOutput(), + *mDecoderState->getCacheIndirectionInput(), *srcOffsetsSliceDeviceSlice, *dstOffsetsSliceDeviceSlice, + *copySizesDeviceSlice, maxCopySize, manager.getStream()); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +std::vector<std::unique_ptr<DecoderStepAsyncSend>> TrtGptModelInflightBatching::communicateDecoderBuffers( + bool returnLogProbs) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(communicateDecoderBuffers); + + auto& decoderOutputBuffers = mDecoderOutputBuffers.at(getFusedBufferId()); + + std::vector<std::unique_ptr<DecoderStepAsyncSend>> asyncHandles; + if (mWorldConfig.isLastPipelineParallelRank()) + { + if (broadcastPostDecoder()) + { + DecoderStepAsyncSend::bcast(decoderOutputBuffers, *mDecoderState, returnLogProbs, mOperatingBeamWidth, + mModelConfig.getSpeculativeDecodingMode().needsKVCacheRewind(), *mMpiCommTensorPara, 0); + } + + if (mWorldConfig.isPipelineParallel()) + { + auto const peerSend = 0; + asyncHandles.emplace_back(std::make_unique<DecoderStepAsyncSend>(decoderOutputBuffers, *mDecoderState, + returnLogProbs, mOperatingBeamWidth, mModelConfig.getSpeculativeDecodingMode().needsKVCacheRewind(), + *mMpiCommPipelinePara, peerSend)); + } + } + else + { + auto const peerRecv = mWorldConfig.isFirstPipelineParallelRank() ? mWorldConfig.getPipelineParallelism() - 1 + : mWorldConfig.getPipelineParallelRank() - 1; + DecoderStepAsyncSend::recv(decoderOutputBuffers, *mDecoderState, returnLogProbs, mOperatingBeamWidth, + mModelConfig.getSpeculativeDecodingMode().needsKVCacheRewind(), *mMpiCommPipelinePara, peerRecv); + auto const peerSend = mWorldConfig.getPipelineParallelRank() + 1; + if (peerSend != mWorldConfig.getPipelineParallelism() - 1) + { + asyncHandles.emplace_back(std::make_unique<DecoderStepAsyncSend>(decoderOutputBuffers, *mDecoderState, + returnLogProbs, mOperatingBeamWidth, mModelConfig.getSpeculativeDecodingMode().needsKVCacheRewind(), + *mMpiCommPipelinePara, peerSend)); + } + } + TLLM_CHECK_WITH_INFO(asyncHandles.size() <= 2, "Up to two decoder step async handles expected"); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return asyncHandles; +} + +void TrtGptModelInflightBatching::updateRequests(ScheduledRequests const& scheduledRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(updateRequests); + + auto const& decoderOutputBuffers = mDecoderOutputBuffers.at(getFusedBufferId()); + + auto const hostNewOutputTokensShape = decoderOutputBuffers.newOutputTokensHost->getShape(); + auto const* const hostNewOutputTokensData + = bufferCast<TokenIdType const>(*decoderOutputBuffers.newOutputTokensHost); + auto const* const sequenceLengthsHostData = bufferCast<SizeType32 const>(*decoderOutputBuffers.sequenceLengthsHost); + auto const* const decoderFinishedSumPtr = bufferCast<SizeType32 const>(*decoderOutputBuffers.finishedSumHost); + auto const* const cumLogProbsPtr = bufferCast<float const>(*decoderOutputBuffers.cumLogProbsHost); + auto const* const logProbsPtr = bufferCast<float const>(*decoderOutputBuffers.logProbsHost); + auto const* const finishReasonsHostData + = bufferCast<kernels::FinishedState>(*decoderOutputBuffers.finishReasonsHost); + + // Update only requests that ran through the decoder + for (auto const& llmReq : scheduledRequests.generationRequests) + { + if (llmReq->isGenerationCompleteState()) + { + continue; + } + auto const reqBeamWidth = llmReq->getBeamWidthByIter(true); + auto const seqSlot = llmReq->mSeqSlot.value(); + auto const currentNumOfTokens = llmReq->getMaxBeamNumTokens(); + + // Save the accepted token logits from target model + if (mModelConfig.getSpeculativeDecodingMode().isDraftTokensExternal() && llmReq->getReturnGenerationLogits() + && llmReq->hasDraftTokens()) + { + TLLM_CHECK_WITH_INFO(reqBeamWidth == 1, "Speculative decoding only works for beam width == 1"); + + SizeType32 numAcceptedTokens + = sequenceLengthsHostData[seqSlot * mOperatingBeamWidth + 0] - llmReq->getMaxBeamNumTokens(); + + auto const& generationLogitsHost = llmReq->getGenerationLogitsHost(); + auto shape = generationLogitsHost->getShape(); + shape.d[1] = numAcceptedTokens; + generationLogitsHost->reshape(shape); + } + + std::vector<SizeType32> numNewTokens(reqBeamWidth); + std::vector<SizeType32> numDroppedTokens(reqBeamWidth); + + // numGeneratedTokens is the number of tokens generated by the decoder. + // Some tokens might be dropped due to end token or rejected draft tokens. + auto const numGeneratedTokens = llmReq->getNumDraftTokens() + 1; + + for (SizeType32 beam = 0; beam < reqBeamWidth; ++beam) + { + // Sequence length is only advanced for accepted tokens. + auto const seqLen = sequenceLengthsHostData[seqSlot * mOperatingBeamWidth + beam]; + // Actual number of tokens that should be added to the request. + auto const numNewOutputTokens = seqLen - llmReq->getNumTokens(beam); + if (reqBeamWidth == 1) + { + TLLM_CHECK_WITH_INFO(numGeneratedTokens >= numNewOutputTokens, + "numNewOutputTokens must not be greater than numGeneratedTokens: " + "numGeneratedTokens %d < numNewOutputTokens %d", + numGeneratedTokens, numNewOutputTokens); + } + numNewTokens[beam] = std::min(numGeneratedTokens, numNewOutputTokens); + numDroppedTokens[beam] = numGeneratedTokens - numNewTokens[beam]; + for (SizeType32 step = 0; step < numNewTokens[beam]; ++step) + { + auto const newTokenIdx = tc::flat_index(hostNewOutputTokensShape.d, step, seqSlot, beam); + auto const newToken = hostNewOutputTokensData[newTokenIdx]; + llmReq->addNewToken(newToken, beam); + TLLM_LOG_DEBUG("request ID %ld beam %d newToken %d", llmReq->mRequestId, beam, newToken); + + if (llmReq->returnLogProbs()) + { + auto const cumLogProb = cumLogProbsPtr[seqSlot * mOperatingBeamWidth + beam]; + llmReq->setCumLogProb(cumLogProb, beam); + + auto const beginLogProbsOffset = reqBeamWidth == 1 ? llmReq->mPromptLen : 0; + SizeType32 offset + = (seqSlot * mOperatingBeamWidth + beam) * getMaxSequenceLen() + beginLogProbsOffset; + auto const generatedLength = seqLen - llmReq->mPromptLen; + std::vector<float> logProbs(logProbsPtr + offset, logProbsPtr + offset + generatedLength); + llmReq->setLogProbs(logProbs, beam); + } + } + + auto const finishReason = finishReasonsHostData[seqSlot * mOperatingBeamWidth + beam]; + llmReq->setFinishedReason(finishReason.toFinishReason(), beam); + + TLLM_LOG_DEBUG("[RANK %d] decoderSync: request ID %lu beam %d tokens %s finished %d", + COMM_SESSION.getRank(), llmReq->mRequestId, beam, common::vec2str(llmReq->getTokens(beam)).c_str(), + static_cast<int>(finishReason.toFinishReason())); + } + + // Set number of tokens predicted per runtime iteration. Will be > 1 for speculative decoding. + llmReq->updateNumTokensPerIteration(llmReq->getMaxBeamNumTokens() - currentNumOfTokens, mModelConfig); + + // Fill new draft tokens for the next step + if (decoderFinishedSumPtr[seqSlot] != reqBeamWidth + && (mModelConfig.getSpeculativeDecodingMode().predictsDraftTokens() + || mModelConfig.getSpeculativeDecodingMode().needsKVCacheRewind())) + { + auto const maxDraftTokensLen = mModelConfig.getMaxDecodingDraftTokens(); + auto prevDraftTokensLen = llmReq->getNumDraftTokens(); + + // We overallocate KV cache for EAGLE to the maxDecodingTokens + maxPathLen in order to fit both + // Base model verification (needs up to maxDecodingTokens) and + // Drafter (needs up to maxPathLen of accepted tokens and maxDecodingDraftTokens for new draft tokens). + if (mModelConfig.getSpeculativeDecodingMode().isEagle()) + { + prevDraftTokensLen = mModelConfig.getSpeculativeDecodingModule().getMaxDecodingTokens() + + mModelConfig.getSpeculativeDecodingModule().getMaxPathLen() - 1; + } + + auto nextDraftTokensLen = mModelConfig.getSpeculativeDecodingModule().getMaxDecodingDraftTokens(); + if (mModelConfig.getSpeculativeDecodingMode().variableDraftLength()) + { + auto const* const nextDraftTokensLengthsHostData + = bufferCast<SizeType32 const>(*decoderOutputBuffers.nextDraftTokensLengthsHost); + nextDraftTokensLen = nextDraftTokensLengthsHostData[seqSlot]; + } + TLLM_CHECK(nextDraftTokensLen <= maxDraftTokensLen); + + auto const* const nextDraftTokensHostData + = bufferCast<TokenIdType const>(*decoderOutputBuffers.nextDraftTokensHost); + auto draftTokensShared + = std::make_shared<std::vector<TokenIdType>>(nextDraftTokensHostData + seqSlot * maxDraftTokensLen, + nextDraftTokensHostData + seqSlot * maxDraftTokensLen + nextDraftTokensLen); + + llmReq->setDraftTokens(draftTokensShared); + + // For all phases except context that does not have draft tokens + if (!llmReq->isGenerationCompleteState() && prevDraftTokensLen != 0 + && mModelConfig.getSpeculativeDecodingMode().needsKVCacheRewind()) + { + // -1 here is for current 'main' token + auto const acceptedTokensLen = llmReq->getMaxBeamNumTokens() - currentNumOfTokens - 1; + auto const rewindLength = prevDraftTokensLen - acceptedTokensLen; + + TLLM_LOG_DEBUG("request ID %lu (seqSlot %d): accepted %d of %d draft tokens, rewind %d tokens", + llmReq->mRequestId, seqSlot, acceptedTokensLen, prevDraftTokensLen, rewindLength); + TLLM_CHECK(0 <= acceptedTokensLen && acceptedTokensLen <= prevDraftTokensLen); + + // At this point, KV cache rows are already gathered and moved to the right location. + // We can safely rewind (draft - accepted) tokens + mKvCacheManager->rewindKVCache(llmReq->mRequestId, rewindLength); + } + } + + // Terminate if request has finished or if it is speculative decoding target model + if (decoderFinishedSumPtr[seqSlot] == reqBeamWidth + || (mModelConfig.getSpeculativeDecodingMode().isDraftTokensExternal() && llmReq->hasDraftTokens())) + { + postProcessRequest(*llmReq, numDroppedTokens); + + if (!mWorldConfig.isPipelineParallel() || !mWorldConfig.isLastPipelineParallelRank()) + { + if (llmReq->getReturnGenerationLogits() && mSpeculativeDecodingFastLogits && mIsLeaderInOrchMode) + { + std::lock_guard<std::mutex> lk(mDraftRequestsMtx); + mDraftRequestsWaitingToSendLogits.push_back(llmReq); + } + else + { + terminateRequest(llmReq); + } + llmReq->setState(LlmRequestState::kGENERATION_COMPLETE); + } + else + { + llmReq->setState(LlmRequestState::kGENERATION_TO_COMPLETE); + } + } + else + { + // gather tokens in the case of streaming and beam search + if (llmReq->isStreaming() && llmReq->mSamplingConfig.beamWidth > 1) + { + postProcessRequest(*llmReq, numDroppedTokens); + } + if (llmReq->isContextInitState()) + { + llmReq->setState(LlmRequestState::kGENERATION_IN_PROGRESS); + } + + if (isTrtOverlap() && llmReq->willCompleteNextIteration()) + { + // This state prohibits the request from being scheduled for another iteration. It assumes that the next + // iteration has already been scheduled and the request can finish in the next call to updateRequests(). + llmReq->setState(LlmRequestState::kGENERATION_TO_COMPLETE); + } + } + + if (llmReq->getReturnPerfMetrics()) + { + llmReq->updatePerfMetrics(mIterCounter); + } + + llmReq->advanceDecodingIter(); + + if (mWorldConfig.isPipelineParallel() && mWorldConfig.isLastPipelineParallelRank()) + { + for (SizeType32 beam = 0; beam < reqBeamWidth; ++beam) + { + llmReq->setNumPreDecodedTokens(numNewTokens[beam], beam); + } + } + } + + if (mModelConfig.getSpeculativeDecodingMode().needsKVCacheRewind()) + { + SizeType32 numSequences{0}; + for (auto const& requests : {scheduledRequests.contextRequests, scheduledRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + auto const reqBeamWidth = llmReq->mSamplingConfig.beamWidth; + numSequences += reqBeamWidth; + } + } + + TLLM_CHECK_WITH_INFO(mCtxGenFusion, "Current speculative decoding mode requires context-gen fusion IFB"); + rewindKVCacheBlocks(numSequences); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +std::vector<std::unique_ptr<DecoderStepAsyncSend>> TrtGptModelInflightBatching::decoderSync( + ScheduledRequests const& scheduledRequests, std::optional<runtime::CudaEvent> const& decoderFinishEvent) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(decoderSync); + + if (mWorldConfig.isLastPipelineParallelRank()) + { + decoderFinishEvent->synchronize(); + } + + auto const returnLogProbs = batchReturnLogProbs(scheduledRequests); + auto asyncHandles = communicateDecoderBuffers(returnLogProbs); + + updateRequests(scheduledRequests); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return asyncHandles; +} + +void TrtGptModelInflightBatching::rewindKVCacheBlocks(SizeType32 numSequences) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + auto const bufferId = getFusedBufferId(); + auto& runtimeBuffers = *mBuffers.at(bufferId); + auto& decoderOutputBuffers = mDecoderOutputBuffers.at(bufferId); + + auto localNbLayers = mModelConfig.getNbAttentionLayers( + mWorldConfig.getPipelineParallelism(), mWorldConfig.getPipelineParallelRank()); + if (mWorldConfig.isLastPipelineParallelRank() && mModelConfig.getSpeculativeDecodingMode().isEagle()) + { + // Do not correct the last kv caches, which are for EagleNet drafter. Those KV caches are managed separately. + auto eagleModulePtr + = std::dynamic_pointer_cast<runtime::EagleModule>(mModelConfig.getSpeculativeDecodingModulePtr()); + localNbLayers -= eagleModulePtr->getNumTransformerLayers(); + } + + auto const tokensPerBlock = mModelConfig.getTokensPerBlock(); + auto const elemSize = BufferDataType(mModelConfig.getKvDataType()).getSize(); + auto const sizeInBytesPerKVHead = mModelConfig.getSizePerHead() * elemSize; + + auto const poolPointers = mKvCacheManager->getBlockPoolPointers(); + auto* const* pointerArrayPtr = bufferCast<void*>(*poolPointers); + auto const* offsetArrayPtr + = bufferCast<tk::KVCacheIndex>(*runtimeBuffers.transformerBuffers->kvCacheBlockOffsetsDevice); + + auto commonRewindLen = mModelConfig.getSpeculativeDecodingModule().getMaxDecodingDraftTokens(); + SizeType32 const* rewindLens = nullptr; + if (mModelConfig.getSpeculativeDecodingMode().variableDraftLength()) + { + commonRewindLen = 0; + rewindLens = bufferCast<SizeType32 const>(*decoderOutputBuffers.prevDraftTokensLengthsHost); + } + + tensorrt_llm::runtime::kernels::invokeUpdateKVBlockArrayDraftTokenLocation( + *mDecoderState->getAcceptedLengthsCumSum(), *mDecoderState->getAcceptedPackedPaths(), + *runtimeBuffers.sequenceLengthsDevice, pointerArrayPtr, offsetArrayPtr, localNbLayers, numSequences, + mRewindInputs.numKvHeads, sizeInBytesPerKVHead, commonRewindLen, rewindLens, *runtimeBuffers.seqSlots, + getMaxAttentionWindow(), mRewindInputs.maxBlocksPerSeq, tokensPerBlock, mRewindInputs.isUseOneMoreBlock, + mRuntime->getStreamPtr()->get()); + + sync_check_cuda_error(mRuntime->getStream().get()); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +nvinfer1::DataType TrtGptModelInflightBatching::getLogitDataType() const +{ + return mModelConfig.getLogitsDtype(); +} + +TrtGptModelInflightBatching::SizeType32 TrtGptModelInflightBatching::numCachedCudaGraphs() const +{ + return std::accumulate(mCudaGraphExecutorCaches.begin(), mCudaGraphExecutorCaches.end(), SizeType32{0}, + [](SizeType32 sum, auto const& cache) { return sum + cache.size(); }); +} + +void TrtGptModelInflightBatching::changeBeamWidth(SizeType32 beamWidth) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK(mInflightReqIds.empty()); + + TLLM_CHECK_WITH_INFO(beamWidth <= getMaxBeamWidth(), + "Requested beam width %d is larger than configured max beam width %d", beamWidth, getMaxBeamWidth()); + TLLM_LOG_DEBUG("Changing operating beam width from %d to %d", mOperatingBeamWidth, beamWidth); + mOperatingBeamWidth = beamWidth; + + if (isCudaGraphMode()) + { + for (auto& cache : mCudaGraphExecutorCaches) + { + cache.clear(); + } + } + createBuffers(mDecodingConfig, mAdditionalModelOutputs); + createDecoder(mDecodingConfig.getDecodingMode()); + + if (static_cast<bool>(mKvCacheManager)) + { + auto const dims = mKvCacheManager->getOffsetTableDimensions(); + reshapeKvTensors(dims); + } + if (static_cast<bool>(mCrossKvCacheManager)) + { + auto const dims = mCrossKvCacheManager->getOffsetTableDimensions(); + reshapeKvTensors(dims); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::changeSpecDecMode(ScheduledRequests const& scheduledRequests) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + if ((!mModelConfig.getSpeculativeDecodingMode().isLookaheadDecoding() + && !mModelConfig.getSpeculativeDecodingMode().isNone()) + || scheduledRequests.empty() || mSeamlessLADMaxDraftLen == 0 || getGatherGenerationLogits() + || mModelConfig.isRnnBased()) + { + return; + } + + bool canUseLookahead = false; + auto maxNumRequestForLad = mDecodingConfig.getLookaheadDecodingMaxNumRequest(); + SizeType32 numRequests = scheduledRequests.contextRequests.size() + scheduledRequests.generationRequests.size(); + if (numRequests > maxNumRequestForLad) + { + if (mModelConfig.getSpeculativeDecodingMode().isLookaheadDecoding()) + { + canUseLookahead = false; + } + else + { + return; + } + } + { + bool useTopKTopP = false; + bool useBanWords = false; + bool useTempAccVocabPenalties = false; // use temperature and penalties that need to accumulate #vocab. + SizeType32 beamWidth = 1; + for (auto const& requests : {scheduledRequests.contextRequests, scheduledRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + useTopKTopP |= !(llmReq->mSamplingConfig.useDefaultValues( + llmReq->mSamplingConfig.topK, layers::DefaultDecodingParams::getTopK()) + || llmReq->mSamplingConfig.useDefaultValues(llmReq->mSamplingConfig.topK, 1)); + useTopKTopP |= !llmReq->mSamplingConfig.useDefaultValues( + llmReq->mSamplingConfig.topP, layers::DefaultDecodingParams::getTopP()); + useBanWords |= llmReq->getBadWordsList().has_value(); + useBanWords |= !llmReq->mSamplingConfig.useDefaultValues( + llmReq->mSamplingConfig.noRepeatNgramSize, layers::DefaultDecodingParams::getNoRepeatNgramSize()); + useTempAccVocabPenalties |= !llmReq->mSamplingConfig.useDefaultValues( + llmReq->mSamplingConfig.temperature, layers::DefaultDecodingParams::getTemperature()); + useTempAccVocabPenalties |= !llmReq->mSamplingConfig.useDefaultValues( + llmReq->mSamplingConfig.repetitionPenalty, layers::DefaultDecodingParams::getRepetitionPenalty()); + useTempAccVocabPenalties |= !llmReq->mSamplingConfig.useDefaultValues( + llmReq->mSamplingConfig.presencePenalty, layers::DefaultDecodingParams::getPresencePenalty()); + useTempAccVocabPenalties |= !llmReq->mSamplingConfig.useDefaultValues( + llmReq->mSamplingConfig.frequencyPenalty, layers::DefaultDecodingParams::getFrequencyPenalty()); + beamWidth = llmReq->mSamplingConfig.beamWidth; + if (useTopKTopP || useBanWords || useTempAccVocabPenalties || beamWidth > 1) + { + break; + } + } + canUseLookahead = !(useTopKTopP || useBanWords || useTempAccVocabPenalties || beamWidth > 1); + } + } + + // Change speculative decoding mode + auto const bufferId = mCtxGenFusion + ? getFusedBufferId() + : (!scheduledRequests.contextRequests.empty() ? getContextBufferId() : getGenerationBufferId()); + // TODO: enable lookahead for generation requests. + bool canChangeToLookahead = scheduledRequests.generationRequests.empty(); + if (mModelConfig.getSpeculativeDecodingMode().isNone() && canUseLookahead && canChangeToLookahead) + { + // None -> Lookahead + mModelConfig.enableSeamlessLookaheadDecoding(mSeamlessLADMaxDraftLen); + mDecodingConfig.enableSeamlessLookaheadDecoding(); + setupSpeculativeDecodingModule(mDecodingConfig); + mBuffers.at(bufferId)->mLookaheadBuffers->enableLookaheadDecoding( + getMaxBatchSize(), mModelConfig.getMaxDecodingTokens()); + mDecoderOutputBuffers.at(getFusedBufferId()) + .enableLookaheadDecoding(getMaxNumSequences(), mModelConfig.getMaxDecodingTokens()); + createDecoder(mDecodingConfig.getDecodingMode()); + } + else if (mModelConfig.getSpeculativeDecodingMode().isLookaheadDecoding() + && (!canUseLookahead || numRequests > maxNumRequestForLad)) + { + // Lookahead -> None + mModelConfig.disableSeamlessLookaheadDecoding(); + mDecodingConfig.setDecodingMode(executor::DecodingMode::Auto()); + mBuffers.at(bufferId)->mLookaheadBuffers->disableLookaheadDecoding(); + mDecoderOutputBuffers.at(getFusedBufferId()).disableLookaheadDecoding(getMaxNumSequences()); + mDecoder->disableLookahead( + scheduledRequests.generationRequests, mDecoderInputBuffers.at(getFusedBufferId()).setupBatchSlots); + mDecoderState->disableLookahead(scheduledRequests.generationRequests); + for (auto const& llmReq : scheduledRequests.generationRequests) + { + if (llmReq->getNumDraftTokens() > 0) + { + llmReq->discardDraftTokens(llmReq->getNumDraftTokens()); + } + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TrtGptModelInflightBatching::getCurrentIterationStats(executor::IterationStats& stats) const +{ + stats.iter = mIterCounter; + + // Max batch size and max num tokens can be tuned at runtime + stats.maxBatchSizeStatic = getMaxBatchSize(); + stats.maxBatchSizeTunerRecommended = mMaxBatchSizeTunerRecommended; + stats.maxBatchSizeRuntime = mMaxBatchSizeRuntime; + stats.maxNumTokensStatic = mMaxNumTokensStatic.value_or(0); + stats.maxNumTokensTunerRecommended = mMaxNumTokensTunerRecommended; + stats.maxNumTokensRuntime = mMaxNumTokensRuntime.value_or(0); + + // KVCacheManager statistics + auto const& kvCacheManager = getKVCacheManager(); + if (kvCacheManager) + { + executor::KvCacheStats kvStats{}; + auto kvCacheStats = kvCacheManager->getKvCacheStats(); + kvStats.maxNumBlocks = kvCacheStats.maxNumBlocks; + kvStats.freeNumBlocks = kvCacheStats.freeNumBlocks; + kvStats.usedNumBlocks = kvCacheStats.usedNumBlocks; + kvStats.tokensPerBlock = kvCacheStats.toksPerBlock; + kvStats.allocTotalBlocks = kvCacheStats.allocTotalBlocks; + kvStats.allocNewBlocks = kvCacheStats.allocNewBlocks; + kvStats.reusedBlocks = kvCacheStats.reusedBlocks; + kvStats.missedBlocks = kvCacheStats.missedBlocks; + kvStats.cacheHitRate = kvCacheStats.cacheHitRate; + stats.kvCacheStats = kvStats; + } + auto const& crossKvCacheManager = getCrossKVCacheManager(); + if (crossKvCacheManager) + { + executor::KvCacheStats kvStats{}; + auto kvCacheStats = crossKvCacheManager->getKvCacheStats(); + kvStats.maxNumBlocks = kvCacheStats.maxNumBlocks; + kvStats.freeNumBlocks = kvCacheStats.freeNumBlocks; + kvStats.usedNumBlocks = kvCacheStats.usedNumBlocks; + kvStats.tokensPerBlock = kvCacheStats.toksPerBlock; + kvStats.allocTotalBlocks = kvCacheStats.allocTotalBlocks; + kvStats.allocNewBlocks = kvCacheStats.allocNewBlocks; + kvStats.reusedBlocks = kvCacheStats.reusedBlocks; + kvStats.missedBlocks = kvCacheStats.missedBlocks; + kvStats.cacheHitRate = kvCacheStats.cacheHitRate; + stats.crossKvCacheStats = kvStats; + } + executor::InflightBatchingStats modelStats{}; + modelStats.numScheduledRequests = mLastIterationStatsIFB.scheduledRequests.size(); + modelStats.numContextRequests = mLastIterationStatsIFB.numCtxRequests; + modelStats.numGenRequests = mLastIterationStatsIFB.numGenRequests; + modelStats.numPausedRequests = mLastIterationStatsIFB.pausedRequests.size(); + modelStats.avgNumDecodedTokensPerIter = mLastIterationStatsIFB.avgNumDecodedTokensPerIter; + modelStats.numCtxTokens = mLastIterationStatsIFB.numCtxTokens; + modelStats.microBatchId = mLastIterationStatsIFB.microBatchId; + stats.inflightBatchingStats = modelStats; +} + +void TrtGptModelInflightBatching::getCurrentRequestStats(executor::RequestStatsPerIteration& stats) const +{ + stats.iter = mIterCounter; + for (auto& requestStat : stats.requestStats) + { + requestStat.scheduled + = mLastIterationStatsIFB.scheduledRequests.count(static_cast<RequestIdType>(requestStat.id)); + requestStat.paused = mLastIterationStatsIFB.pausedRequests.count(static_cast<RequestIdType>(requestStat.id)); + } +} + +executor::DebugTensorsPerIteration TrtGptModelInflightBatching::getCurrentDebugTensors() const +{ + executor::DebugTensorsPerIteration debugTensors; + debugTensors.iter = mIterCounter; + + for (auto const& [name, tensor] : mLastIterationDebugTensors) + { + debugTensors.debugTensors.emplace(name, executor::detail::ofITensor(tensor)); + } + + return debugTensors; +} + +nvinfer1::DataType TrtGptModelInflightBatching::getTensorDataType(std::string const& name) const +{ + auto const& engine = mRuntime->getEngine(); + return engine.getTensorDataType(name.c_str()); +} + +nvinfer1::Dims TrtGptModelInflightBatching::getTensorShape(std::string const& name) const +{ + auto const& engine = mRuntime->getEngine(); + return engine.getTensorShape(name.c_str()); +} + +SizeType32 TrtGptModelInflightBatching::getMaxCapacityBatchSize(SizeType32 inputLength, SizeType32 outputLength) const +{ + return mKvCacheManager->getMaxCapacityBatchSize(inputLength, outputLength); +} + +/* + * Manages prefetching of prompt table chunks using a double-buffer strategy + * + * Function Flow: + * 1. First Chunk Processing (isFirstChunk == true): + * - Uses blocking prefetch on main runtime stream + * - Ensures initial data is ready before computation starts + * + * 2. Subsequent Chunks (isFirstChunk == false): + * - Uses non-blocking prefetch on separate copy stream + * - Overlaps data transfer with computation + * + * Synchronization: + * - First prefetch: No wait needed (fresh start) + * - Later prefetches: Wait for previous copy to complete + * - Uses mPtableCopyDoneEvent to track completion + * + * Key Functions: + * 1. prefetchNextPromptTableChunk: + * - Calls the correct function based on position in code (before or after prepareBuffers()) + * - Waits for previous copy to complete if not the first chunk + * + * 2. remapInputTokensForPromptTable: + * - Identifies tokens that need prompt table embeddings (tokens that are greater than vocabSize) + * - Remaps IDs to match chunked prompt table layout + * + * 3. copyPromptTableToGpuInChunk: + * - Handles actual transfer from CPU pinned memory to GPU + * - Uses appropriate buffer manager based on isFirstChunk + */ +void TrtGptModelInflightBatching::prefetchNextPromptTableChunk( + RequestVector const& contextRequests, bool isFirstChunk, SizeType32 bufferId) +{ + auto& promptTuningBuffers = mBuffers[bufferId]->promptTuningBuffers; + + if (!isFirstChunk) + { + // Only switch buffer after prepareBuffer() + promptTuningBuffers->switchChunkPtableBuffer(); + } + + SizeType32 contextId = 0; + for (auto const& llmReq : contextRequests) + { + if (llmReq->isFirstContextChunk() && isFirstChunk) + { + // For first chunk: Blocking prefetch on runtime stream to ensure data is ready + remapInputTokensForPromptTable(llmReq, true, bufferId, contextId); + } + else if (!isFirstChunk) // prefetching for subsequent chunks + { + // For the first prefetch chunk, don't need to wait for previous prefetch to complete + // For subsequent chunks: Need to wait for previous prefetch to complete + if (!llmReq->isFirstContextChunk()) + { + mRuntime->getBufferManager().getStream().wait(mPtableCopyDoneEvent); + } + + // Non-blocking prefetch on copy stream to prepare next chunk in pong buffer + if (llmReq->getContextRemainingLength() > 0) + { + remapInputTokensForPromptTable(llmReq, false, bufferId, contextId); + } + } + + ++contextId; + } +} + +void TrtGptModelInflightBatching::remapInputTokensForPromptTable( + std::shared_ptr<LlmRequest> const& llmReq, bool isFirstChunk, SizeType32 bufferId, SizeType32 contextId) +{ + NVTX3_SCOPED_RANGE_WITH_NAME(range, "remapInputTokensForPromptTable"); + auto& promptTuningBuffers = mBuffers[bufferId]->promptTuningBuffers; + auto const chunkSize = llmReq->getContextChunkSize(); + auto& inputTokensMutable = llmReq->getTokensMutable(0); + auto vocabSize = mModelConfig.getVocabSize(); + + if (isFirstChunk) + { + promptTuningBuffers->initializeChunkPtableBuffers( + mRuntime->getBufferManager(), mModelConfig, chunkSize, llmReq); + } + + size_t processChunkSize; + size_t beginPos; + + if (!isFirstChunk) + { + processChunkSize = std::min(chunkSize, llmReq->getContextRemainingLength() - chunkSize); + } + else + { + processChunkSize = std::min(chunkSize, llmReq->getContextRemainingLength()); + } + + if (!isFirstChunk) + { + // For prefetching next chunk + if (llmReq->getContextRemainingLength() - chunkSize <= 0) + { + promptTuningBuffers->updateBufferStartPosition(promptTuningBuffers->getChunkPtableCurrentIndex(), 0); + return; // No more chunks to prefetch + } + beginPos = llmReq->getContextCurrentPosition() + chunkSize; + } + else + { + // For current chunk + beginPos = llmReq->getContextCurrentPosition(); + } + + TLLM_CHECK_WITH_INFO(beginPos + processChunkSize <= inputTokensMutable.size(), + "Invalid chunk access: beginPos(%zu) + processChunkSize(%zu) > totalSize(%zu)", beginPos, processChunkSize, + inputTokensMutable.size()); + + auto inputTokensChunk = inputTokensMutable.begin() + beginPos; + std::vector<SizeType32> outOfVocabTokens; + SizeType32 ptableTokenId = vocabSize; + for (size_t i = 0; i < processChunkSize; i++) + { + if (inputTokensChunk[i] >= vocabSize) + { + outOfVocabTokens.push_back(inputTokensChunk[i]); + inputTokensChunk[i] = ptableTokenId++; + } + } + + copyPromptTableToGpuInChunk(llmReq, outOfVocabTokens, isFirstChunk, bufferId, contextId); +} + +void TrtGptModelInflightBatching::copyPromptTableToGpuInChunk(std::shared_ptr<LlmRequest> const& llmReq, + std::vector<int32_t> const& outOfVocabTokens, bool isFirstChunk, SizeType32 bufferId, SizeType32 contextId) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE_WITH_NAME(range, "copyPromptTableToGpuInChunk"); + auto& promptTuningBuffers = mBuffers[bufferId]->promptTuningBuffers; + + if (outOfVocabTokens.empty()) + { + return; + } + + auto const& promptTable = llmReq->getPromptEmbeddingTable(); + TLLM_CHECK_WITH_INFO(promptTable.has_value(), "promptTable is empty but there's fake_prompt"); + TLLM_CHECK_WITH_INFO(promptTable.value() != nullptr, "promptTable value is null but there's fake_prompt"); + + auto currentBufferManager = isFirstChunk ? mRuntime->getBufferManager() : mCopyBufferManager; + auto const hiddenSize = mModelConfig.getHiddenSize(); + auto numRows = outOfVocabTokens.size(); + std::size_t sliceSize = static_cast<size_t>(numRows * hiddenSize); + auto currentIndex = promptTuningBuffers->getChunkPtableCurrentIndex(); + + // Calculate the offset based on current position + size_t srcOffset = llmReq->mPtableCurrentPosition * hiddenSize; + size_t dstOffset = promptTuningBuffers->getChunkPtableBufferStartPosition(currentIndex, contextId); + + auto gpuBuffer = promptTuningBuffers->getChunkPtableBuffer(currentIndex); + + // First view as 1D tensor of elements + auto totalElements = promptTable.value()->getSize(); + auto table1D = runtime::ITensor::view( + promptTable.value(), runtime::ITensor::makeShape({static_cast<int64_t>(totalElements)})); + + TLLM_CHECK_WITH_INFO(srcOffset + sliceSize <= totalElements, + "Buffer bounds violation: Trying to access up to %zu elements but buffer only has %zu elements (offset: %zu, " + "slice size: %zu)", + srcOffset + sliceSize, totalElements, srcOffset, sliceSize); + + auto table1DShared = runtime::ITensor::SharedPtr(table1D.release()); + auto pTableView = runtime::ITensor::slice(table1DShared, srcOffset, sliceSize); + + auto gpuBufferSlice = runtime::ITensor::slice(gpuBuffer, dstOffset, numRows); + + currentBufferManager.copy(*pTableView, *gpuBufferSlice); + + promptTuningBuffers->updateBufferStartPosition(currentIndex, outOfVocabTokens.size()); + + llmReq->mPtableCurrentPosition += outOfVocabTokens.size(); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.h b/cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.h new file mode 100644 index 000000000000..d6550281a758 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.h @@ -0,0 +1,639 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/batch_manager/kvCacheType.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tensorrt_llm/runtime/worldConfig.h" +#include "trtGptModel.h" + +#include <NvInferRuntime.h> + +namespace tensorrt_llm::runtime +{ +class TllmRuntime; +class GptDecoderBatched; +class AllReduceBuffers; +class NcclCommunicator; +class SpeculativeDecodingMode; + +namespace decoder +{ +class DecoderState; +} // namespace decoder + +namespace decoder_batch +{ +class Input; +class Output; +} // namespace decoder_batch + +} // namespace tensorrt_llm::runtime + +namespace tensorrt_llm::mpi +{ +class MpiWaitThread; +} // namespace tensorrt_llm::mpi + +namespace tensorrt_llm::batch_manager +{ +class BaseCacheTransceiver; +} + +namespace tensorrt_llm::batch_manager +{ + +namespace kv_cache_manager +{ +class KVCacheManager; +struct OffsetTableDimensions; +} // namespace kv_cache_manager + +namespace rnn_state_manager +{ +class RnnStateManager; +} // namespace rnn_state_manager + +class SequenceSlotManager; +class DecoderStepAsyncSend; +class DecoderSlotAsyncSend; +class DecoderInputBuffers; +class DecoderOutputBuffers; +class SlotDecoderBuffers; +class LlmRequest; +class RuntimeBuffers; +class BasePeftCacheManager; +class GuidedDecoder; +class TrtGptModelTest; + +// Algorithms +class CapacityScheduler; +class DisaggTransferAdmissionController; +class MicroBatchScheduler; +class PauseRequests; +class AssignReqSeqSlots; +class AllocateKvCache; +class HandleContextLogits; +class HandleGenerationLogits; +class GenerateRequestOptions; +class LogitsPostProcessor; +class MakeDecodingBatchInputOutput; +class CreateNewDecoderRequests; +class UpdateDecoderBuffers; + +namespace utils +{ +class CudaGraphExecutorCache; +} // namespace utils + +struct RewindInputs +{ + SizeType32 maxBlocksPerSeq; + bool isUseOneMoreBlock; + SizeType32 numKvHeads; +}; + +class TrtGptModelInflightBatching : public TrtGptModel +{ + using BaseKVCacheManager = kv_cache_manager::BaseKVCacheManager; + using OffsetTableDimensions = kv_cache_manager::OffsetTableDimensions; + using KVCacheManager = kv_cache_manager::KVCacheManager; + using KvCacheType = kv_cache_manager::CacheType; + using KvCacheConfig = executor::KvCacheConfig; + using RnnStateManager = rnn_state_manager::RnnStateManager; + using LlmRequestPtr = std::shared_ptr<batch_manager::LlmRequest>; + +public: + class IterationStatsIFB + { + public: + explicit IterationStatsIFB(SizeType32 microBatchId) + : microBatchId{microBatchId} + { + } + + SizeType32 microBatchId; + SizeType32 numCtxRequests{}; + SizeType32 numGenRequests{}; + SizeType32 numCtxTokens{}; + float avgNumDecodedTokensPerIter{}; + ReqIdsSet scheduledRequests; + ReqIdsSet pausedRequests; + }; + + using SizeType32 = tensorrt_llm::runtime::SizeType32; + using TokenIdType = tensorrt_llm::runtime::TokenIdType; + using BufferManager = tensorrt_llm::runtime::BufferManager; + using PeftTable = PeftCacheManager::PeftTable; + using TensorMap = runtime::StringPtrMap<runtime::ITensor>; + using TensorPtr = runtime::ITensor::SharedPtr; + + TrtGptModelInflightBatching(std::shared_ptr<nvinfer1::ILogger> logger, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig, runtime::RawEngine const& rawEngine, bool ctxGenFusion, + executor::ExecutorConfig const& executorConfig, bool isLeaderInOrchMode); + + ~TrtGptModelInflightBatching() override; + + /// @brief Calculate the cache size per token for the disaggregated serving. + /// @param modelConfig Model configuration. + /// @param worldConfig World configuration. + /// @param maxAttentionWindowVec Maximum attention window vector. (may have fewer elements than numLayers, in which + /// case it cycles) + /// @param isCrossAttention Whether the attention is cross attention. + /// @param kvFactor KV factor. + /// @return Cache size per token for the disaggregated layers. Note that window size is not included in the result + /// here. + [[nodiscard]] static std::map<SizeType32, SizeType32> calculateCacheSizePerTokenForDisagg( + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + std::vector<SizeType32> const& maxAttentionWindowVec, bool isCrossAttention, SizeType32 kvFactor); + + void terminateRequest(LlmRequestPtr const& llmRequest, bool pause = false) override; + + /// @brief Terminate request in the next forwardSync call that includes the request. + /// @details This function does not terminate requests immediately. It will add the requests to the + /// mReqIdsToTerminate set. The requests will be terminated in the next forwardSync call that + /// includes the request in the batch. + void terminateRequestSync(LlmRequestPtr const& llmRequest, executor::FinishReason finishReason) override; + + /// @brief Function that waits for the decoding of requests in flight. + /// When the requests have finished or using speculative decoding, the state of requests + /// will become LlmRequestState::kGENERATION_COMPLETE. Else, it will be set to + /// LlmRequestState::kGENERATION_IN_PROGRESS. + void forwardSync() override; + + /// @brief Function that tries to advance the active requests. + /// Depending on resources available, it's possible that not all requests will get advanced. + /// Requests that may be in state LlmRequestState::kCONTEXT_INIT become + /// LlmRequestState::kGENERATION_IN_PROGRESS or LlmRequestState::kGENERATION_TO_COMPLETE. + /// @param activeRequests The list of request to try to advance. + void forwardAsync(RequestList const& activeRequests) override; + + /// @brief Override the runtime batch size for the model + void setRuntimeBatchSize(SizeType32 runtimeMaxBatchSize) override; + + /// @brief Get the runtime batch size for the model + [[nodiscard]] SizeType32 getRuntimeBatchSize() const override; + + /// @brief Override the runtime max num tokens for the model + void setRuntimeMaxNumTokens(SizeType32 runtimeMaxNumTokens) override; + + void updatePeftCache(std::shared_ptr<LlmRequest> const& llmRequest) override; + + [[nodiscard]] IterationStatsIFB getLastIterationStats() const + { + return mLastIterationStatsIFB; + } + + [[nodiscard]] TrtGptModelType getModelType() const override + { + return mCtxGenFusion ? TrtGptModelType::InflightFusedBatching : TrtGptModelType::InflightBatching; + }; + + [[nodiscard]] runtime::BufferManager const& getBufferManager() const override; + [[nodiscard]] runtime::BufferManager::CudaStreamPtr getRuntimeStreamPtr() const override; + + void getCurrentIterationStats(executor::IterationStats& stats) const override; + void getCurrentRequestStats(executor::RequestStatsPerIteration& stats) const override; + [[nodiscard]] executor::DebugTensorsPerIteration getCurrentDebugTensors() const override; + + [[nodiscard]] executor::IterationType getIterCounter() const noexcept override + { + return mIterCounter; + } + + [[nodiscard]] static bool executorConfigIsValid( + runtime::ModelConfig const& modelConfig, executor::ExecutorConfig const& executorConfig); + [[nodiscard]] static executor::ExecutorConfig fixExecutorConfig( + runtime::ModelConfig const& modelConfig, executor::ExecutorConfig const& executorConfig); + + void prepareDisaggGenInitRequests(RequestList const& activeRequests, RequestVector& newGenReques); + void checkDisaggGenTransferStatus(RequestList const& activeRequests); + void prepareDistGenBufferAndDecoder(RequestVector const& generationRequests); + + void resetIterationStats() override; + + runtime::SpeculativeDecodingMode getSpeculativeDecodingMode() const noexcept + { + return mModelConfig.getSpeculativeDecodingMode(); + } + + [[nodiscard]] SizeType32 numCachedCudaGraphs() const; + +private: + friend class TrtGptModelTest; + + [[nodiscard]] SizeType32 getContextBufferId() const + { + return mMicroBatchId; + } + + [[nodiscard]] SizeType32 getGenerationBufferId() const + { + return mNumMicroBatches + mMicroBatchId; + } + + [[nodiscard]] SizeType32 getFusedBufferId() const + { + return mMicroBatchId; + } + + [[nodiscard]] SizeType32 getNextMicroBatchId(SizeType32 bufferId) const + { + return (bufferId + 1) % mNumMicroBatches; + } + + [[nodiscard]] SizeType32 getPrevMicroBatchId(SizeType32 bufferId) const + { + return (bufferId + mNumMicroBatches - 1) % mNumMicroBatches; + } + + //! @brief Store full kv cache blocks contributed by req. + //! These blocks become reusable from next step. + void storeContextBlocks(std::shared_ptr<LlmRequest> const& req); + + //! @brief Store newest kv cache block for reuse. + //! The block become reusable from next step. + void storeNewBlock(std::shared_ptr<LlmRequest> const& req); + + //! @brief Set LayerProfiler to collect performance per layer. + void setLayerProfiler() override; + + //! @brief Print profile information per layer. + std::string getLayerProfileInfo() const override; + + std::tuple<SizeType32, TensorMap const&, TensorMap&> prepareBuffers( + RequestVector const& contextRequests, RequestVector const& generationRequests, SizeType32 bufferId); + + //! @brief Capture graph of current batch state during engine execution. + //! This is based on the assumptions that + //! a) We can hide CPU graph capture behind the GPU engine execution. + //! b) Batch size in the next iterations won't change and we can reuse the graph multiple times. + void prepareGraph(SizeType32 bufferId, SizeType32 optProfileId); + + void executeContext(SizeType32 runtimeContextId, SizeType32 bufferId); + void executeBatch(ScheduledRequests const& scheduledRequests); + void executeStep( + RequestVector const& contextRequests, RequestVector const& generationRequests, SizeType32 bufferId); + + void debugIOTensors(RequestVector const& contextRequests, RequestVector const& generationRequests, + TensorMap const& inputMap, TensorMap const& outputMap); + + void createRuntimeContexts(); + void createDecoder(std::optional<executor::DecodingMode> const& decodingModeOpt); + void createBuffers(executor::DecodingConfig const& decodingConfig, + std::optional<std::vector<executor::AdditionalModelOutput>> const& additionalModelOutputs); + std::unique_ptr<KVCacheManager> createKvCacheManager(KvCacheConfig const& kvCacheConfig, KvCacheType kvCacheType, + uint64_t freePrimaryMemBytes, uint64_t freeSecondaryMemBytes, size_t extraCostMemory, + bool const failFastOnAttentionWindowTooLarge = false); + void createRnnStateManager(); + void createCustomAllReduceWorkspace(); + void createRuntimePerfKnobsTensor(executor::ExtendedRuntimePerfKnobConfig const& extendedRuntimePerfKnobConfig); + + /// @brief Verify draft token length and beam width of all active requests. + /// May change operating beam width if all requests agree on same beam width. + void verifyRequests(RequestList const& activeRequests); + + /// @brief Change the operating beam width. + /// Only possible if no requests are currently in-flight. + /// @param beamWidth New operating beam width. Must be smaller than initial maxBeamWidth. + void changeBeamWidth(SizeType32 beamWidth); + + SizeType32 getOperatingBeamWidth() const override + { + return mOperatingBeamWidth; + } + + /// @details Should be called after setting up the current batch in executeBatch to get the correct number of + /// context tokens. + IterationStatsIFB fillIterationStats( + ScheduledRequests const& scheduledRequests, RequestVector const& requestsToPause); + + /// @brief Function that sets up the TensorRT execution context that is going to be used for execution. If multiple + /// TensorRT optimization profiles are built in the engine, it selects the corresponding context that is going to be + /// used, and prepares the input and output tensors so that both buffers and the context is ready for the execution. + /// @return The TensorRT execution context index that has been setup. + void setupContext( + RequestVector const& contextRequests, RequestVector const& generationRequests, SizeType32 bufferId); + + void setupDecoderStep( + RequestVector const& contextRequests, RuntimeBuffers const& buffers, DecoderInputBuffers& inputBuffers); + runtime::CudaEvent decoderStepAsync(ScheduledRequests const& scheduledRequests); + std::vector<std::unique_ptr<DecoderStepAsyncSend>> decoderSync( + ScheduledRequests const& scheduledRequests, std::optional<runtime::CudaEvent> const& decoderFinishEvent); + + std::vector<std::unique_ptr<DecoderStepAsyncSend>> communicateDecoderBuffers(bool returnLogProbs); + void updateRequests(ScheduledRequests const& scheduledRequests); + + /// @brief It gathers the logits if they need to be returned, calls getDecoderSlotHostOutputs, + /// and overwrites the llmRequest tokens buffer. + /// Called either on request finishing, or at every step when doing beam search and streaming. + void postProcessRequest(LlmRequest& llmReq, std::vector<SizeType32> const& numDroppedTokens); + /// @brief Reorders generation logits to match finalized beam paths after gatherTree. + /// During beam search, logits are stored by beam slot. After finalization, output_ids are + /// reordered by parentIds, but logits are not. This method traces parentIds on the host + /// to build the slot mapping and reindexes the logits accordingly. + void reorderGenerationLogitsForBeamSearch(LlmRequest& llmReq, SizeType32 seqSlot, SizeType32 reqBeamWidth, + SizeType32 maxSeqLength, TokenIdType const* outputIdsHostData, SizeType32 const* sequenceLengthsHostData); + /// @brief Calls gatherTree (via finalize) and transmits the received data across ranks if PP>1 + void getDecoderSlotHostOutputs( + SizeType32 seqSlot, bool returnLogProbs, runtime::SamplingConfig const& samplingConfig, bool streaming); + void rewindKVCacheBlocks(SizeType32 numSequences); + void setupSpeculativeDecodingModule(executor::DecodingConfig const& decodingConfig); + + /// @brief Copies the content of the cache indirection outputs to the cache indirection inputs. + /// @param[in] scheduledRequests The requests to copy the cache indirections for. + /// @param[in] genBufferId The id of the generation buffers for those requests. + void copyCacheIndirectionFromOutputsToInputs(ScheduledRequests const& scheduledRequests, SizeType32 genBufferId); + + [[nodiscard]] bool getGatherGenerationLogits() const override + { + return getModelConfig().computeGenerationLogits() || mGatherGenerationLogits; + } + + [[nodiscard]] runtime::ModelConfig const& getModelConfig() const override + { + return mModelConfig; + } + + [[nodiscard]] runtime::WorldConfig const& getWorldConfig() const override + { + return mWorldConfig; + } + + [[nodiscard]] SizeType32 getNumMicroBatches() const override + { + return mNumMicroBatches; + } + + [[nodiscard]] nvinfer1::DataType getLogitDataType() const override; + + [[nodiscard]] nvinfer1::DataType getTensorDataType(std::string const& name) const override; + + [[nodiscard]] nvinfer1::Dims getTensorShape(std::string const& name) const override; + + void reshapeKvTensors(OffsetTableDimensions const& dims); + + [[nodiscard]] bool hasSpeculativeDecodingFastLogits() const noexcept override + { + return mSpeculativeDecodingFastLogits; + } + + [[nodiscard]] bool hasGuidedDecoder() const noexcept override + { + return static_cast<bool>(mGuidedDecoder); + } + + using BlocksPerWindow = std::map<SizeType32, std::tuple<SizeType32, SizeType32>>; + /// @brief Based on the KV-cache manager's capacity and configuration, we adjust the maximum supported attention + /// window. + /// + /// @param blocksPerWindow map of window size to number of blocks. + /// @param failFastOnAttentionWindowTooLarge if true, the function will report a runtime error if the attention + /// window is too large to fit even a single sequence in the KV cache. + /// @return pair of new blocks per window and new maxAttentionWindowVec + [[nodiscard]] std::pair<BlocksPerWindow, std::vector<SizeType32>> clampWindowSizesToFitAtLeastOneSequence( + BlocksPerWindow const& blocksPerWindow, bool const failFastOnAttentionWindowTooLarge = false); + + /// @brief Change the speculative decoding mode. + void changeSpecDecMode(ScheduledRequests const& scheduledRequests); + + void prefetchNextPromptTableChunk(RequestVector const& contextRequests, bool isFirstChunk, SizeType32 bufferId); + + void remapInputTokensForPromptTable( + std::shared_ptr<LlmRequest> const& llmReq, bool isCurrentChunk, SizeType32 bufferId, SizeType32 contextId); + + void copyPromptTableToGpuInChunk(std::shared_ptr<LlmRequest> const& llmReq, + std::vector<int32_t> const& outOfVocabTokens, bool useCurrentBuffer, SizeType32 bufferId, SizeType32 contextId); + +protected: + std::shared_ptr<BaseKVCacheManager> getKVCacheManager() override + { + return mKvCacheManager; + } + + [[nodiscard]] std::shared_ptr<BaseKVCacheManager const> getKVCacheManager() const override + { + return mKvCacheManager; + } + + std::shared_ptr<BaseKVCacheManager> getCrossKVCacheManager() + { + return mCrossKvCacheManager; + } + + [[nodiscard]] std::shared_ptr<BaseKVCacheManager const> getCrossKVCacheManager() const + { + return mCrossKvCacheManager; + } + + [[nodiscard]] std::shared_ptr<BasePeftCacheManager> getPeftCacheManager() override + { + return mPeftCacheManager; + } + + [[nodiscard]] std::shared_ptr<BasePeftCacheManager const> getPeftCacheManager() const override + { + return mPeftCacheManager; + } + + void setLogitsPostProcessorBatched(std::optional<LogitsPostProcessorBatched> logitsPostProcessorBatched) override + { + mLogitsPostProcessorBatched = logitsPostProcessorBatched; + } + + void setReplicateLogitsPostProcessor(bool replicateLogitsPostProcessor) override + { + mReplicateLogitsPostProcessor = replicateLogitsPostProcessor; + } + + [[nodiscard]] bool getReplicateLogitsPostProcessor() const override + { + return mReplicateLogitsPostProcessor; + } + + SizeType32 getMaxCapacityBatchSize(SizeType32 inputLength, SizeType32 outputLength) const override; + +private: + /******************** Configs ********************/ + // Parameters of the model (TRT engine) + runtime::ModelConfig mModelConfig; + // Parameters of the execution environment + runtime::WorldConfig mWorldConfig; + // Device ID of this instance + int mDevice{-1}; + // Config for (speculative) decoding + executor::DecodingConfig mDecodingConfig; + // Performance knobs for the engine. + executor::ExtendedRuntimePerfKnobConfig mExtendedRuntimePerfKnobConfig; + TensorPtr mExtendedRuntimePerfKnobsHost; + // Config for debugging output + std::optional<executor::DebugConfig> mDebugConfig; + // List of additional outputs for each request + std::optional<std::vector<executor::AdditionalModelOutput>> mAdditionalModelOutputs; + + /******************** Components ********************/ + std::shared_ptr<nvinfer1::ILogger> mLogger; + // Runner for the TRT engine. The engine produces logits. + std::unique_ptr<runtime::TllmRuntime> mRuntime; + // Decoder that generates new tokens from the logits. + std::unique_ptr<runtime::GptDecoderBatched> mDecoder; + // Decoder state for all requests + std::unique_ptr<runtime::decoder::DecoderState> mDecoderState; + // Synchronization handles for decoder + std::vector<std::optional<runtime::CudaEvent>> mDecoderFinishedEvents; + + // Manager that maps requests to slots + std::shared_ptr<SequenceSlotManager> mSeqSlotManager; + // KV cache manager for attention layers (optional) + std::shared_ptr<BaseKVCacheManager> mKvCacheManager; + // KV cache manager for cross attention in enc-dec models (optional) + std::shared_ptr<BaseKVCacheManager> mCrossKvCacheManager = nullptr; + // RNN state manager for recurrent layers (optional) + std::unique_ptr<RnnStateManager> mRnnStateManager; + // PEFT cache manager for LoRA tasks (optional) + std::shared_ptr<BasePeftCacheManager> mPeftCacheManager; + // BufferManager using a separate stream for async copy operations. + runtime::BufferManager mCopyBufferManager; + // Event for async data transfers + runtime::CudaEvent mPtableCopyDoneEvent; + + /******************** Logits Post-Processor ********************/ + std::optional<LogitsPostProcessorBatched> mLogitsPostProcessorBatched; + bool mReplicateLogitsPostProcessor{true}; + // Set if any request invoked a logits processor in current step + bool mLogitsPostProcessorIsApplied{false}; + + constexpr bool broadcastPostDecoder() + { + return mWorldConfig.isTensorParallel() && !mReplicateLogitsPostProcessor && mLogitsPostProcessorIsApplied; + } + + std::unique_ptr<tensorrt_llm::batch_manager::GuidedDecoder> mGuidedDecoder; + + /******************** Pipeline parallelism ********************/ + std::unique_ptr<tensorrt_llm::mpi::MpiComm> mMpiCommPipelinePara; + std::vector<std::unique_ptr<DecoderStepAsyncSend>> mDecStepAsyncSndHdls; + std::vector<std::unique_ptr<DecoderSlotAsyncSend>> mDecSlotAsyncSndHdls; + std::unique_ptr<tensorrt_llm::mpi::MpiWaitThread> mAsyncSendWaitThread; + + /******************** Tensor parallelism ********************/ + std::unique_ptr<tensorrt_llm::mpi::MpiComm> mMpiCommTensorPara; + std::unique_ptr<runtime::AllReduceBuffers> mAllReduceBuffers; + + /******************** Runtime parameters ********************/ + // Flag to select fused or unfused context+generation execution + bool mCtxGenFusion; + // ID of current micro batch, changes after each iteration + SizeType32 mMicroBatchId{0}; + // Number of micro batches. Multiple batches are used for overlapping setup and execution, + // and in pipeline parallelism. + SizeType32 mNumMicroBatches; + // Number of buffers to be added to mBuffers. + SizeType32 mNumBuffers; + // Current operating beam width. Can be changed with changeBeamWidth function. + SizeType32 mOperatingBeamWidth; + // Runtime batch size optimized during execution for microBatchScheduler: + /// The max batch size recommended by the dynamic tuner + SizeType32 mMaxBatchSizeTunerRecommended; + /// The min of mMaxBatchSize and mMaxBatchSizeTunerRecommended + SizeType32 mMaxBatchSizeRuntime; + // Runtime max num tokens optimized during execution for microBatchScheduler: + /// Build time max num tokens + std::optional<SizeType32> mMaxNumTokensStatic; + /// The max num tokens recommended by the dynamic tuner + SizeType32 mMaxNumTokensTunerRecommended; + /// The min of mMaxNumTokens and mMaxNumTokensTunerRecommended + std::optional<SizeType32> mMaxNumTokensRuntime; + // Controls if generation logits should be gathered, so that returnGenerationLogits can be requested. + bool mGatherGenerationLogits{false}; + // offloading and prefetching the prompt tuning table (only effective in chunked prefill mode) + bool mPromptTableOffloading; + + /******************** Buffers ********************/ + // Buffers for each micro batch. Unfused path (mCtxGenFusion==false) uses two times the buffers. + std::vector<std::unique_ptr<RuntimeBuffers>> mBuffers; + // Decoder input buffers for each micro batch. + std::vector<DecoderInputBuffers> mDecoderInputBuffers; + // Decoder output buffers for each micro batch. + std::vector<DecoderOutputBuffers> mDecoderOutputBuffers; + // Buffers for each slot in the decoder + std::vector<std::unique_ptr<SlotDecoderBuffers>> mSlotDecoderBuffers; + // PEFT table for each micro batch + std::vector<PeftTable> mPeftTables; + + /******************** Book keeping ********************/ + // List of requests in each micro batch + std::vector<ScheduledRequests> mMicroBatchScheduledRequests; + // Set of in-flight requests of *all* micro batches + ReqIdsSet mInflightReqIds; + // Requests that should be terminated (requested from outside the model) + std::unordered_map<RequestIdType, executor::FinishReason> mReqIdsToTerminate; + // Requests that the scheduler selected to be paused + ReqIdsSet mReqIdsToPause; + // Stats collected in last iteration + IterationStatsIFB mLastIterationStatsIFB{-1}; + // Iteration counter used to distinguish debug output + executor::IterationType mIterCounter{0}; + // Debug tensors of last itreation + TensorMap mLastIterationDebugTensors; + // Cuda graph instances for each microbatch. + std::vector<utils::CudaGraphExecutorCache> mCudaGraphExecutorCaches; + + /******************** Cache transceiver ********************/ + std::unique_ptr<BaseCacheTransceiver> mCacheTransceiver; + std::unique_ptr<DisaggTransferAdmissionController> mDisaggTransferAdmissionController; + + /******************** Spec dec ***********************/ + std::unique_ptr<std::thread> mDraftModelSendLogitsThread; + bool mSpeculativeDecodingFastLogits; + std::atomic<bool> mDraftModelThreadShouldExit{false}; + bool mIsLeaderInOrchMode{false}; + // List of completed draft requests which logits will need to be sent to the target model. + // Guarded by mDraftRequestsMtx (shared with the background logits sender thread). + RequestVector mDraftRequestsWaitingToSendLogits; + // Draft requests whose logits have been sent — pending termination by main thread. + // Guarded by mDraftRequestsMtx. + RequestVector mDraftRequestsDoneSendingLogits; + std::mutex mDraftRequestsMtx; + SizeType32 mSeamlessLADMaxDraftLen{0}; + bool mUseSeamlessLookahead{false}; + RewindInputs mRewindInputs; + + /******************** Algorithms ********************/ + // Algorithms are reentrant, they are assigned a state at + // construction time and it is not modified through execution, hence they are const. + // Schedulers that select which requests to run in each iteration + std::unique_ptr<tensorrt_llm::batch_manager::CapacityScheduler const> mCapacityScheduler; + std::unique_ptr<tensorrt_llm::batch_manager::MicroBatchScheduler const> mMicroBatchScheduler; + std::unique_ptr<tensorrt_llm::batch_manager::PauseRequests const> mPauseRequests; + std::unique_ptr<tensorrt_llm::batch_manager::AssignReqSeqSlots const> mAssignReqSeqSlots; + std::unique_ptr<tensorrt_llm::batch_manager::AllocateKvCache const> mAllocateKvCache; + std::unique_ptr<tensorrt_llm::batch_manager::HandleContextLogits const> mHandleContextLogits; + std::unique_ptr<tensorrt_llm::batch_manager::HandleGenerationLogits const> mHandleGenerationLogits; + std::unique_ptr<tensorrt_llm::batch_manager::LogitsPostProcessor const> mLogitsPostProcessor; + std::unique_ptr<tensorrt_llm::batch_manager::MakeDecodingBatchInputOutput const> mMakeDecodingBatchInputOutput; + std::unique_ptr<tensorrt_llm::batch_manager::CreateNewDecoderRequests const> mCreateNewDecoderRequests; + std::unique_ptr<tensorrt_llm::batch_manager::UpdateDecoderBuffers const> mUpdateDecoderBuffers; +}; + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/updateDecoderBuffers.cpp b/cpp/tensorrt_llm/batch_manager/updateDecoderBuffers.cpp new file mode 100644 index 000000000000..ead120135f3a --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/updateDecoderBuffers.cpp @@ -0,0 +1,78 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/updateDecoderBuffers.h" +#include "tensorrt_llm/batch_manager/decoderBuffers.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/runtime/decoderState.h" +#include "tensorrt_llm/runtime/iTensor.h" + +namespace tensorrt_llm::batch_manager +{ + +using BufferManager = tensorrt_llm::runtime::BufferManager; +using TensorPtr = runtime::ITensor::SharedPtr; +using ITensor = runtime::ITensor; +using SizeType32 = tensorrt_llm::runtime::SizeType32; + +runtime::CudaEvent UpdateDecoderBuffers::operator()(runtime::ModelConfig const& modelConfig, + DecoderOutputBuffers& decoderOutputBuffers, runtime::BufferManager const& copyBufferManager, + runtime::decoder::DecoderState const& decoderState, bool returnLogProbs, + runtime::CudaEvent const& decoderFinishEvent) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(updateDecoderBuffers); + + // Chain copy after decoder event, using a different stream + copyBufferManager.getStream().wait(decoderFinishEvent); + + copyBufferManager.copy(*decoderState.getAllNewTokens(), *decoderOutputBuffers.newOutputTokensHost); + copyBufferManager.copy(*decoderState.getSequenceLengths(), *decoderOutputBuffers.sequenceLengthsHost); + + auto const finishedSumDevice = decoderState.getFinishedSum(); + copyBufferManager.copy(*finishedSumDevice, *decoderOutputBuffers.finishedSumHost); + auto const finishReasonsDevice = decoderState.getFinishReasons(); + copyBufferManager.copy(*finishReasonsDevice, *decoderOutputBuffers.finishReasonsHost); + + if (returnLogProbs) + { + copyBufferManager.copy(*decoderState.getCumLogProbs(), *decoderOutputBuffers.cumLogProbsHost); + copyBufferManager.copy(*decoderState.getLogProbs(), *decoderOutputBuffers.logProbsHost); + } + + if (modelConfig.getSpeculativeDecodingMode().predictsDraftTokens()) + { + // TODO: keep data on device for next iteration + copyBufferManager.copy(*decoderState.getNextDraftTokens(), *decoderOutputBuffers.nextDraftTokensHost); + + if (modelConfig.getSpeculativeDecodingMode().variableDraftLength()) + { + copyBufferManager.copy( + *decoderState.getNextDraftTokensLengths(), *decoderOutputBuffers.nextDraftTokensLengthsHost); + copyBufferManager.copy( + *decoderState.getPrevDraftTokensLengths(), *decoderOutputBuffers.prevDraftTokensLengthsHost); + } + } + + runtime::CudaEvent copyEvent{}; + copyBufferManager.getStream().record(copyEvent); + // Store the event for later sync. Sync stream before calling next decoder. Sync host before updating requests. + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return copyEvent; +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tensorrt_llm/batch_manager/utils/debugUtils.h b/cpp/tensorrt_llm/batch_manager/utils/debugUtils.h index c041c7a71de8..e4732a75f649 100644 --- a/cpp/tensorrt_llm/batch_manager/utils/debugUtils.h +++ b/cpp/tensorrt_llm/batch_manager/utils/debugUtils.h @@ -23,6 +23,11 @@ #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/worldConfig.h" +namespace tensorrt_llm::runtime +{ +class TllmRuntime; +} // namespace tensorrt_llm::runtime + namespace tensorrt_llm::batch_manager::utils { diff --git a/cpp/tensorrt_llm/batch_manager/utils/inflightBatchingUtils.cpp b/cpp/tensorrt_llm/batch_manager/utils/inflightBatchingUtils.cpp index a3e54a6b0f9b..416235f347b8 100644 --- a/cpp/tensorrt_llm/batch_manager/utils/inflightBatchingUtils.cpp +++ b/cpp/tensorrt_llm/batch_manager/utils/inflightBatchingUtils.cpp @@ -16,6 +16,7 @@ */ #include "inflightBatchingUtils.h" +#include "tensorrt_llm/runtime/runtimeKernels.h" namespace tensorrt_llm::batch_manager::utils { @@ -87,6 +88,170 @@ void moveFinishedContextRequestsToGeneration(ScheduledRequests& scheduledRequest TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); } +void copyGenerationLogits(RuntimeBuffers::GenerationLogitsCache& generationLogitsCache, + runtime::BufferManager const& bufferManager, LlmRequest& llmReq, bool beforeDecoder, + std::vector<SizeType32> const& numDroppedTokens) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK_WITH_INFO( + !beforeDecoder || numDroppedTokens.empty(), "numDroppedTokens are only possible after decoder."); + + auto const reqBeamWidth = llmReq.getBeamWidthByIter(); + TLLM_CHECK_WITH_INFO(numDroppedTokens.empty() || numDroppedTokens.size() == static_cast<size_t>(reqBeamWidth), + "Dropped tokens have to be defined for all beams."); + + auto const fragmentSize = llmReq.getGenerationLogitsFragmentsSize(); + + // Merge logits fragments on device. getFragmentPointerSlot() returns the matching host and + // device rows for the current workIdx and advances the index atomically, so concurrent flushes + // for different requests in the same batch never clobber each other's pointer arrays. + auto const& transposeBufferPtr = generationLogitsCache.transposedLogits; + auto [cachePointerHost, cachePointerDevice] = generationLogitsCache.getFragmentPointerSlot(); + tensorrt_llm::runtime::kernels::mergeLogitsFragments(bufferManager, *transposeBufferPtr, + llmReq.getGenerationLogitsFragments(), *cachePointerDevice, *cachePointerHost, 0, 1, reqBeamWidth, + bufferManager.getStream(), 0); + llmReq.clearGenerationLogitsFragments(); + + // Copy logits to host + for (SizeType32 beam = 0; beam < reqBeamWidth; beam++) + { + auto const droppedSize = !numDroppedTokens.empty() ? numDroppedTokens.at(beam) : 0; + // Ignore logits of dropped tokens + auto const beamFragmentSize = fragmentSize - droppedSize; + // If this function is called before the decoder, the request does not contain the generated token of the + // current iteration, so we add 1 to the number of tokens. + auto const numGenerationToken + = static_cast<SizeType32>(beforeDecoder) + llmReq.getNumTokens(beam) - llmReq.mPromptLen; + auto const hostOffset = numGenerationToken - beamFragmentSize; + + // [beamWidth, GENERATION_LOGITS_BUFFER_LENGTH, vocabSizePadded] -> [beamFragmentSize, vocabSizePadded] + auto beamDeviceTensorPtr = ITensor::slice(transposeBufferPtr, {beam, 0}, beamFragmentSize); + // [beamWidth, mMaxNewTokens, vocabSizePadded] -> [beamFragmentSize, vocabSizePadded] + auto beamHostTensorPtr = ITensor::slice(llmReq.getGenerationLogitsHost(), {beam, hostOffset}, beamFragmentSize); + bufferManager.copy(*beamDeviceTensorPtr, *beamHostTensorPtr); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +namespace +{ + +std::pair<TensorPtr const&, bool> findOutputTensor(std::string const& outputTensorName, + std::vector<executor::AdditionalModelOutput> const& additionalModelOutputs, + RuntimeBuffers::TensorMap const& outputMap, bool isContext) +{ + auto const aoIter = std::find_if(additionalModelOutputs.cbegin(), additionalModelOutputs.cend(), + [&outputTensorName](auto const& ao) { return ao.name == outputTensorName; }); + TLLM_CHECK_WITH_INFO(aoIter != additionalModelOutputs.cend(), "Additional %s output tensor not found: %s", + isContext ? "context" : "generation", outputTensorName.c_str()); + + auto const gatherContext = aoIter->gatherContext; + if (isContext) + { + TLLM_CHECK_WITH_INFO( + gatherContext, "Additional context output tensor not gathered: %s", outputTensorName.c_str()); + } + + auto const tensorIt = outputMap.find(outputTensorName); + TLLM_CHECK_WITH_INFO(tensorIt != outputMap.end(), "Additional %s output tensor not found: %s", + isContext ? "context" : "generation", outputTensorName.c_str()); + + return {tensorIt->second, gatherContext}; +} + +} // namespace + +void copyAdditionalOutputs(std::vector<executor::AdditionalModelOutput> const& additionalModelOutputs, + RequestVector const& contextRequests, RequestVector const& generationRequests, + RuntimeBuffers::TensorMap const& outputMap, runtime::BufferManager const& manager) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // One index shared across all output tensors that have gatherContext + SizeType32 srcTensorIndexWithContext{0}; + // One index shared across all output tensors that do not have gatherContext + SizeType32 srcTensorIndexWithoutContext{0}; + + for (auto const& llmReq : contextRequests) + { + auto numContextTokens = llmReq->getContextChunkSize(); + for (auto const& outputTensor : llmReq->getAdditionalContextOutputs()) + { + auto const& [tensor, gatherContext] + = findOutputTensor(outputTensor.first, additionalModelOutputs, outputMap, true); + + auto const srcTensorIndex = srcTensorIndexWithContext; + auto srcView = ITensor::slice(tensor, srcTensorIndex, numContextTokens); + auto dstView = ITensor::slice(outputTensor.second, llmReq->getContextCurrentPosition(), numContextTokens); + manager.copy(*srcView, *dstView); + } + srcTensorIndexWithContext += numContextTokens; + srcTensorIndexWithoutContext += 1; + + // Copy output of last token to generation outputs + if (llmReq->isLastContextChunk()) + { + for (auto const& outputTensor : llmReq->getAdditionalGenerationOutputs()) + { + auto const& [tensor, gatherContext] + = findOutputTensor(outputTensor.first, additionalModelOutputs, outputMap, false); + + auto const srcTensorIndex = gatherContext ? srcTensorIndexWithContext : srcTensorIndexWithoutContext; + auto srcView = ITensor::slice(tensor, srcTensorIndex - 1, 1); + for (SizeType32 beam = 0; beam < llmReq->getBeamWidthByIter(); beam++) + { + auto dstView = ITensor::slice(outputTensor.second, {beam, 0}, 1); + manager.copy(*srcView, *dstView); + } + } + } + } + + for (auto const& llmReq : generationRequests) + { + auto const reqBeamWidth = llmReq->getBeamWidthByIter(); + for (auto const& outputTensor : llmReq->getAdditionalGenerationOutputs()) + { + auto const& [tensor, gatherContext] + = findOutputTensor(outputTensor.first, additionalModelOutputs, outputMap, false); + + auto const srcTensorIndex = gatherContext ? srcTensorIndexWithContext : srcTensorIndexWithoutContext; + for (SizeType32 beam = 0; beam < reqBeamWidth; beam++) + { + auto const generatedLength = llmReq->getNumTokens(beam) - llmReq->getPromptLen(); + TLLM_CHECK(generatedLength >= 1); + auto srcView = ITensor::slice(tensor, srcTensorIndex + beam, 1); + auto dstView = ITensor::slice(outputTensor.second, {beam, generatedLength}, 1); + manager.copy(*srcView, *dstView); + } + } + srcTensorIndexWithContext += reqBeamWidth; + srcTensorIndexWithoutContext += reqBeamWidth; + } + + // Check final indices + for (auto const& outputTensor : additionalModelOutputs) + { + auto const& outputTensorName = outputTensor.name; + auto const gatherContext = outputTensor.gatherContext; + + auto const tensorIt = outputMap.find(outputTensorName); + TLLM_CHECK_WITH_INFO( + tensorIt != outputMap.end(), "Additional output tensor not found: %s", outputTensorName.c_str()); + + auto const& outputShape = tensorIt->second->getShape(); + auto const outputSize = outputShape.d[0]; + auto const finalIndex = gatherContext ? srcTensorIndexWithContext : srcTensorIndexWithoutContext; + + TLLM_CHECK_WITH_INFO(finalIndex == outputSize, "Additional %s output tensor final index mismatch %d != %ld: %s", + gatherContext ? "context" : "generation", finalIndex, outputSize, outputTensorName.c_str()); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + void terminateRequest(SequenceSlotManager& seqSlotManager, LlmRequest& llmReq, SizeType32 maxInputLen, OptionalRef<kv_cache_manager::BaseKVCacheManager> kvCacheManager, OptionalRef<kv_cache_manager::BaseKVCacheManager> crossKvCacheManager, @@ -135,4 +300,109 @@ std::vector<SizeType32> getRequestBeamWidths( return beamWidths; } +void CudaGraphExecutor::create(cudaGraph_t const& graph) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + assert(mInstance == nullptr); + TLLM_CUDA_CHECK(cudaGraphInstantiate(&mInstance, graph, nullptr, nullptr, 0)); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void CudaGraphExecutor::uploadToStream(runtime::CudaStream const& stream) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + assert(hasInstance()); + TLLM_CUDA_CHECK(cudaGraphUpload(mInstance, stream.get())); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void CudaGraphExecutor::launch(runtime::CudaStream const& stream) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + TLLM_CUDA_CHECK(cudaGraphLaunch(mInstance, stream.get())); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +bool CudaGraphExecutor::update(cudaGraph_t const& graph) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + return cudaGraphExecUpdate(mInstance, graph, nullptr) != cudaSuccess; +} + +void CudaGraphExecutor::clear() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + if (mInstance != nullptr) + { + TLLM_CUDA_CHECK(cudaGraphExecDestroy(mInstance)); + mInstance = nullptr; + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void CudaGraphExecutor::prepareNextGraph(std::unique_ptr<runtime::TllmRuntime>& runtime, SizeType32 nextContextId) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + auto& stream = runtime->getStream(); + + cudaGraph_t nextGraph; + TLLM_CUDA_CHECK(cudaStreamBeginCapture(stream.get(), cudaStreamCaptureModeThreadLocal)); + runtime->executeContext(nextContextId); + TLLM_CUDA_CHECK(cudaStreamEndCapture(stream.get(), &nextGraph)); + + if (hasInstance()) + { + if (update(nextGraph)) + { + clear(); + create(nextGraph); + } + } + else + { + create(nextGraph); + } + + TLLM_CUDA_CHECK(cudaGraphDestroy(nextGraph)); + uploadToStream(stream); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +std::optional<std::shared_ptr<CudaGraphExecutor>> CudaGraphExecutorCache::get(BatchState const& state) +{ + auto it = mMap.find(state); + if (it == mMap.end()) + { + return std::nullopt; + } + mCache.splice(mCache.begin(), mCache, it->second); + return it->second->second; +} + +void CudaGraphExecutorCache::put(BatchState const& state, std::shared_ptr<CudaGraphExecutor> const& value) +{ + auto it = mMap.find(state); + if (it != mMap.end()) + { + mCache.erase(it->second); + } + mCache.emplace_front(state, value); + mMap[state] = mCache.begin(); + + if (static_cast<runtime::SizeType32>(mMap.size()) > mCapacity) + { + auto lastState = mCache.back().first; + mCache.pop_back(); + mMap.erase(lastState); + } +} + +void CudaGraphExecutorCache::clear() +{ + // Releasing the shared_ptrs runs ~CudaGraphExecutor, which calls + // cudaGraphExecDestroy on each cached instance. + mMap.clear(); + mCache.clear(); +} + } // namespace tensorrt_llm::batch_manager::utils diff --git a/cpp/tensorrt_llm/batch_manager/utils/inflightBatchingUtils.h b/cpp/tensorrt_llm/batch_manager/utils/inflightBatchingUtils.h index 374ae398f781..fe0c4e505218 100644 --- a/cpp/tensorrt_llm/batch_manager/utils/inflightBatchingUtils.h +++ b/cpp/tensorrt_llm/batch_manager/utils/inflightBatchingUtils.h @@ -20,6 +20,7 @@ #include "tensorrt_llm/batch_manager/common.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/batch_manager/peftCacheManager.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" #include "tensorrt_llm/batch_manager/sequenceSlotManager.h" #include "tensorrt_llm/common/optionalRef.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -49,6 +50,17 @@ void sortRequests(RequestVector& contextRequests, RequestVector& generationReque //! @param scheduledRequests The scheduled context and generation requests. void moveFinishedContextRequestsToGeneration(ScheduledRequests& scheduledRequests); +//! @param beforeDecoder Whether the function is called before the decoder. If it is true, correct the output offset. +//! @param numDroppedTokens The number of dropped tokens for each beam (e.g. when the requests finished early). +//! Generation logits for dropped tokens are ignored. +void copyGenerationLogits(RuntimeBuffers::GenerationLogitsCache& generationLogitsCache, + runtime::BufferManager const& bufferManager, LlmRequest& llmReq, bool beforeDecoder, + std::vector<SizeType32> const& numDroppedTokens = {}); + +void copyAdditionalOutputs(std::vector<executor::AdditionalModelOutput> const& additionalModelOutputs, + RequestVector const& contextRequests, RequestVector const& generationRequests, + RuntimeBuffers::TensorMap const& outputMap, runtime::BufferManager const& manager); + void terminateRequest(SequenceSlotManager& seqSlotManager, LlmRequest& llmRequest, SizeType32 maxInputLen, OptionalRef<kv_cache_manager::BaseKVCacheManager> kvCacheManager = std::nullopt, OptionalRef<kv_cache_manager::BaseKVCacheManager> crossKvCacheManager = std::nullopt, @@ -56,4 +68,66 @@ void terminateRequest(SequenceSlotManager& seqSlotManager, LlmRequest& llmReques std::vector<SizeType32> getRequestBeamWidths( RequestVector const& contextRequests, RequestVector const& generationRequests); + +class CudaGraphExecutor +{ +public: + CudaGraphExecutor() = default; + + ~CudaGraphExecutor() + { + try + { + clear(); + } + catch (std::exception& e) + { + TLLM_LOG_EXCEPTION(e); + } + } + + bool hasInstance() const + { + return mInstance != nullptr; + } + + void clear(); + void prepareNextGraph(std::unique_ptr<runtime::TllmRuntime>& runtime, SizeType32 nextContextId); + void launch(runtime::CudaStream const& stream); + +private: + void create(cudaGraph_t const& graph); + bool update(cudaGraph_t const& graph); + void uploadToStream(runtime::CudaStream const& stream); + + cudaGraphExec_t mInstance = nullptr; +}; + +class CudaGraphExecutorCache +{ + /// @brief LRU cache to store cuda graph instances. +public: + explicit CudaGraphExecutorCache(runtime::SizeType32 capacity) + : mCapacity(capacity) + { + } + + std::optional<std::shared_ptr<CudaGraphExecutor>> get(BatchState const& state); + + void put(BatchState const& state, std::shared_ptr<CudaGraphExecutor> const& value); + + void clear(); + + [[nodiscard]] runtime::SizeType32 size() const noexcept + { + return static_cast<runtime::SizeType32>(mCache.size()); + } + +private: + using BatchStateGraphExecutorPair = std::pair<BatchState, std::shared_ptr<CudaGraphExecutor>>; + using GraphExecutorLruCache = std::list<BatchStateGraphExecutorPair>; + SizeType32 mCapacity; + GraphExecutorLruCache mCache; + std::unordered_map<BatchState, GraphExecutorLruCache::iterator, BatchStateHash> mMap; +}; } // namespace tensorrt_llm::batch_manager::utils diff --git a/cpp/tensorrt_llm/batch_manager/utils/logitsThread.cpp b/cpp/tensorrt_llm/batch_manager/utils/logitsThread.cpp index 4f978a302187..941c1b655073 100644 --- a/cpp/tensorrt_llm/batch_manager/utils/logitsThread.cpp +++ b/cpp/tensorrt_llm/batch_manager/utils/logitsThread.cpp @@ -19,7 +19,6 @@ #include "tensorrt_llm/batch_manager/llmRequest.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/runtime/utils/mpiTags.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" @@ -125,7 +124,7 @@ void draftModelSendLogitsThread(int device, std::atomic<bool>* draftModelThreadS } void targetModelReceiveLogits(runtime::ITensor::SharedPtr& draftLogitsHost, - executor::SpeculativeDecodingFastLogitsInfo const& fastLogitsInfo, tensorrt_llm::DataType logitsDtype) + executor::SpeculativeDecodingFastLogitsInfo const& fastLogitsInfo, nvinfer1::DataType logitsDtype) { #if ENABLE_MULTI_DEVICE auto const& worldComm = tensorrt_llm::mpi::MpiComm::world(); diff --git a/cpp/tensorrt_llm/batch_manager/utils/logitsThread.h b/cpp/tensorrt_llm/batch_manager/utils/logitsThread.h index 7af3b1762d20..637f8a850610 100644 --- a/cpp/tensorrt_llm/batch_manager/utils/logitsThread.h +++ b/cpp/tensorrt_llm/batch_manager/utils/logitsThread.h @@ -18,7 +18,6 @@ #pragma once #include "tensorrt_llm/batch_manager/common.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -47,6 +46,6 @@ void draftModelSendLogitsThread(int device, std::atomic<bool>* draftModelThreadS std::mutex* draftRequestsMtx); void targetModelReceiveLogits(runtime::ITensor::SharedPtr& draftLogitsHost, - executor::SpeculativeDecodingFastLogitsInfo const& fastLogitsInfo, tensorrt_llm::DataType logitsDtype); + executor::SpeculativeDecodingFastLogitsInfo const& fastLogitsInfo, nvinfer1::DataType logitsDtype); } // namespace tensorrt_llm::batch_manager::utils diff --git a/cpp/tensorrt_llm/common/attentionOp.cpp b/cpp/tensorrt_llm/common/attentionOp.cpp index 762ed7b54164..b589b2a97bbd 100644 --- a/cpp/tensorrt_llm/common/attentionOp.cpp +++ b/cpp/tensorrt_llm/common/attentionOp.cpp @@ -22,7 +22,6 @@ #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/memoryUtils.h" #include "tensorrt_llm/common/sageQuant.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/decoderMaskedMultiheadAttention.h" #include "tensorrt_llm/kernels/decoderMaskedMultiheadAttention/cascadeAttentionKernel.h" #include "tensorrt_llm/kernels/flashMLA/flash_mla.h" @@ -217,7 +216,6 @@ bool AttentionOp::convertMMHAParamsToXQAParams(tensorrt_llm::kernels::XQAParams& // Medusa mode will have multiple query tokens. xqaParams.multi_query_tokens = mIsSpecDecodingEnabled && mUseSpecDecoding; xqaParams.is_spec_dec_tree = mIsSpecDecTree; - xqaParams.force_prepare_spec_dec_tree_mask = mForcePrepareSpecDecTreeMask; xqaParams.layer_idx = generationsParams.layer_idx; if (mKVCacheQuantMode.hasInt8KvCache()) @@ -760,8 +758,8 @@ size_t AttentionOp::getFmhaMultiCtasKvScratchSize() const noexcept return partialStatsSize + partialOSize; } -size_t AttentionOp::getWorkspaceSizeForContext(tensorrt_llm::DataType type, int32_t max_num_seq, - int32_t input_seq_length, int32_t cross_kv_length, int32_t max_num_tokens, int32_t total_kv_len) const noexcept +size_t AttentionOp::getWorkspaceSizeForContext(nvinfer1::DataType type, int32_t max_num_seq, int32_t input_seq_length, + int32_t cross_kv_length, int32_t max_num_tokens, int32_t total_kv_len) const noexcept { if (max_num_tokens == 0) { @@ -913,7 +911,7 @@ size_t AttentionOp::getWorkspaceSizeForContext(tensorrt_llm::DataType type, int3 return context_workspace_size; } -size_t AttentionOp::getWorkspaceSizeForGeneration(tensorrt_llm::DataType type, int32_t max_num_seq, +size_t AttentionOp::getWorkspaceSizeForGeneration(nvinfer1::DataType type, int32_t max_num_seq, int32_t max_attention_window_size, int32_t max_num_tokens, int32_t max_blocks_per_sequence) const noexcept { if (max_num_tokens == 0) @@ -1132,12 +1130,6 @@ int AttentionOp::mlaGeneration( tllmRunnerParams.oPtr = reinterpret_cast<void*>(params.context_buf); tllmRunnerParams.oSfPtr = generation_params.context_buf_sf; - if (params.dsv4_epilogue_fusion.enabled) - { - tllmRunnerParams.mDsv4EpilogueFusion.enabled = true; - tllmRunnerParams.mDsv4EpilogueFusion.cosSinCache = params.dsv4_epilogue_fusion.cos_sin_cache; - tllmRunnerParams.mDsv4EpilogueFusion.scaleBufM = params.dsv4_epilogue_fusion.scale_buf_m; - } // softmax stats if needed tllmRunnerParams.softmaxStatsPtr = generation_params.softmax_stats; @@ -2026,12 +2018,6 @@ int AttentionOp::enqueueContext(EnqueueContextParams<T> const& params, cudaStrea // Only use [totalLength, h / cpSize, Dh]. fmhaParams.outputPtr = mCpSize > 1 ? workspaceViews.gatherOutBuffer : params.context_buf; fmhaParams.outputSfPtr = params.context_buf_sf; - if (params.mla_param != nullptr && params.mla_param->dsv4_epilogue_fusion.enabled) - { - fmhaParams.dsv4EpilogueFusion.enabled = true; - fmhaParams.dsv4EpilogueFusion.cosSinCache = params.mla_param->dsv4_epilogue_fusion.cos_sin_cache; - fmhaParams.dsv4EpilogueFusion.scaleBufM = params.mla_param->dsv4_epilogue_fusion.scale_buf_m; - } fmhaParams.attentionSinksPtr = params.attention_sinks; fmhaParams.packedMaskPtr = params.attention_packed_mask; if constexpr (std::is_same_v<KVCacheBuffer, KVBlockArray>) @@ -2820,7 +2806,7 @@ int AttentionOp::initialize() noexcept if (mEnableContextFMHA) { mEnableContextFMHA = false; - if (!(mType == tensorrt_llm::DataType::kHALF || mType == tensorrt_llm::DataType::kBF16)) + if (!(mType == nvinfer1::DataType::kHALF || mType == nvinfer1::DataType::kBF16)) { TLLM_LOG_WARNING("Fall back to unfused MHA because of unsupported data type."); } @@ -2865,7 +2851,7 @@ int AttentionOp::initialize() noexcept "mFP8ContextFMHA must enable if FP4 KV cache is enabled"); TLLM_CHECK(isRoPE() == (mRotaryEmbeddingDim != 0)); - TLLM_CHECK_WITH_INFO((mSM >= 80) || (mType != tensorrt_llm::DataType::kBF16), + TLLM_CHECK_WITH_INFO((mSM >= 80) || (mType != nvinfer1::DataType::kBF16), "Unsupported data type, pre SM 80 GPUs do not support bfloat16"); // Pre-check whether the head size is supported by MMHA. @@ -2917,11 +2903,11 @@ int AttentionOp::initialize() noexcept // Pre-checked during constructing. Data_type data_type, data_type_kv; - if (mType == tensorrt_llm::DataType::kHALF) + if (mType == nvinfer1::DataType::kHALF) { data_type = DATA_TYPE_FP16; } - else if (mType == tensorrt_llm::DataType::kBF16) + else if (mType == nvinfer1::DataType::kBF16) { data_type = DATA_TYPE_BF16; } @@ -2980,10 +2966,6 @@ int AttentionOp::initialize() noexcept fmhaParams.dataTypeKv = DATA_TYPE_E4M3; fmhaParams.dataTypeOut = DATA_TYPE_BF16; } - if (mFusesDsv4InvRopeFp8Quant) - { - fmhaParams.dataTypeOut = DATA_TYPE_E4M3; - } // TODO: remove forceFp32Acc from MHARunnerFixedParams after adding host_runtime_perf_knobs to // bertAttentionPlugin input tensors, so that we can change mLaunchParams.force_fp32_acc value in runtime. fmhaParams.forceFp32Acc = false; @@ -3058,7 +3040,6 @@ int AttentionOp::initialize() noexcept fmhaParams.scaleAlibi = isAliBiWithScale(); fmhaParams.useSparseMLA = useSparseMLA(); fmhaParams.useTllmGenSparseAttention = useTllmGenSparseAttention(); - fmhaParams.fusesDsv4InvRopeFp8Quant = mFusesDsv4InvRopeFp8Quant; // SageAttention: set block sizes for sage quantization. if (useSageAttn) @@ -3081,13 +3062,13 @@ int AttentionOp::initialize() noexcept Data_type kvDataType = DATA_TYPE_FP32; Data_type outputDataType = DATA_TYPE_FP32; - if (mType == tensorrt_llm::DataType::kHALF) + if (mType == nvinfer1::DataType::kHALF) { qDataType = DATA_TYPE_FP16; kvDataType = DATA_TYPE_FP16; outputDataType = DATA_TYPE_FP16; } - else if (mType == tensorrt_llm::DataType::kBF16) + else if (mType == nvinfer1::DataType::kBF16) { qDataType = DATA_TYPE_BF16; kvDataType = DATA_TYPE_BF16; @@ -3103,14 +3084,9 @@ int AttentionOp::initialize() noexcept qDataType = DATA_TYPE_E4M3; kvDataType = DATA_TYPE_E4M3; } - if (mFusesDsv4InvRopeFp8Quant) - { - outputDataType = DATA_TYPE_E4M3; - } // Instantiate the mTllmGenFMHARunner used for MLA - mTllmGenFMHARunner.reset(new TllmGenFmhaRunner( - qDataType, kvDataType, kvDataType, outputDataType, 0, 0, 0, 0, mFusesDsv4InvRopeFp8Quant)); + mTllmGenFMHARunner.reset(new TllmGenFmhaRunner(qDataType, kvDataType, kvDataType, outputDataType)); } else if (mIsGenerationMLA && !mUseGenFlashMLA) { @@ -3177,7 +3153,7 @@ int AttentionOp::initialize() noexcept } mEnableXQA = (mEnableXQA || mIsSpecDecodingEnabled) - && (mType == tensorrt_llm::DataType::kHALF || mType == tensorrt_llm::DataType::kBF16) && mUseKVCache; + && (mType == nvinfer1::DataType::kHALF || mType == nvinfer1::DataType::kBF16) && mUseKVCache; if (mEnableXQA) { @@ -3187,12 +3163,12 @@ int AttentionOp::initialize() noexcept fixedParams.isMLA = mIsGenerationMLA; // TODO: support more combinations. // Update Q and O dtype. - if (mType == tensorrt_llm::DataType::kHALF) + if (mType == nvinfer1::DataType::kHALF) { fixedParams.inputDataType = DATA_TYPE_FP16; fixedParams.outputDataType = DATA_TYPE_FP16; } - else if (mType == tensorrt_llm::DataType::kBF16) + else if (mType == nvinfer1::DataType::kBF16) { fixedParams.inputDataType = DATA_TYPE_BF16; fixedParams.outputDataType = DATA_TYPE_BF16; @@ -3260,6 +3236,10 @@ int AttentionOp::initialize() noexcept reserveSemaphoreArray(mNbMultiBlockSemaphores); } + if (isBuilding()) + { + return 0; + } #if ENABLE_MULTI_DEVICE if (mCpSize > 1 && COMM_SESSION.getSize() > 1) { diff --git a/cpp/tensorrt_llm/common/attentionOp.h b/cpp/tensorrt_llm/common/attentionOp.h index 438489348577..5fcf901f8a79 100644 --- a/cpp/tensorrt_llm/common/attentionOp.h +++ b/cpp/tensorrt_llm/common/attentionOp.h @@ -20,7 +20,6 @@ #include "tensorrt_llm/common/cublasMMWrapper.h" #include "tensorrt_llm/common/opUtils.h" #include "tensorrt_llm/common/quantization.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h" #include "tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm.h" #include "tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQARunner.h" @@ -57,11 +56,10 @@ class AttentionOp [[nodiscard]] size_t getFmhaMultiCtasKvScratchSize() const noexcept; [[nodiscard]] int getHeadSize(bool checkInit = true) const; [[nodiscard]] int getMaxNumSeqLenTile(int batch_beam_size = 1) const; - [[nodiscard]] size_t getWorkspaceSizeForContext(tensorrt_llm::DataType type, int32_t nbReq, - int32_t max_input_length, int32_t cross_kv_length = 0, int32_t max_num_tokens = 0, - int32_t total_kv_len = 0) const noexcept; + [[nodiscard]] size_t getWorkspaceSizeForContext(nvinfer1::DataType type, int32_t nbReq, int32_t max_input_length, + int32_t cross_kv_length = 0, int32_t max_num_tokens = 0, int32_t total_kv_len = 0) const noexcept; // total_num_seq is the sum of beam_width for multiple requests - [[nodiscard]] size_t getWorkspaceSizeForGeneration(tensorrt_llm::DataType type, int32_t total_num_seq, + [[nodiscard]] size_t getWorkspaceSizeForGeneration(nvinfer1::DataType type, int32_t total_num_seq, int32_t max_attention_window_size, int32_t max_num_tokens, int32_t max_blocks_per_sequence) const noexcept; template <typename T> @@ -183,14 +181,14 @@ class AttentionOp if (this->context_lengths && batch_size > 0) { ss << "context_lengths: " - << *(runtime::ITensor::wrap((void*) this->context_lengths, tensorrt_llm::DataType::kINT32, + << *(runtime::ITensor::wrap((void*) this->context_lengths, nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))) << std::endl; } if (this->sequence_lengths && batch_size > 0) { ss << "sequence_lengths: " - << *(runtime::ITensor::wrap((void*) this->sequence_lengths, tensorrt_llm::DataType::kINT32, + << *(runtime::ITensor::wrap((void*) this->sequence_lengths, nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))) << std::endl; } @@ -480,7 +478,7 @@ class AttentionOp int mTpSize = 1; int mTpRank = 0; bool mUnfuseQkvGemm = false; - tensorrt_llm::DataType mType; + nvinfer1::DataType mType; int32_t mMaxContextLength = 0; int32_t mMaxSeqLen = 0; int32_t mMaxNumRequests = 0; @@ -503,7 +501,6 @@ class AttentionOp int32_t mSpecDecodingMaxGenerationLength = 1; // Static spec-dec tree length used by FMHA autotuning. int32_t mSpecDecodingTargetMaxGenLen = 0; - bool mForcePrepareSpecDecTreeMask = false; bool mIsMLAEnabled = false; bool mIsGenerationMLA = false; bool mUseGenFlashMLA = false; @@ -536,8 +533,6 @@ class AttentionOp // Whether to fuse FP4 quant into attention kernel. bool mFuseFp4Quant = false; - // Whether to fuse DSv4 inverse-RoPE + FP8 output quant into trtllm-gen FMHA. - bool mFusesDsv4InvRopeFp8Quant = false; kernels::SparseAttentionParams mRuntimeSparseAttentionParams; @@ -574,14 +569,14 @@ class AttentionOp mCrossAttention, mMaxDistance, mPosShiftEnabled, mPagedContextFMHA, mFP8ContextFMHA, mFP8AttenOutput, mFP8ContextMLA, mFP8GenerationMLA, mChunkPrefillBufferBatchSize, mDenseContextFMHA, mHasFullAttentionMask, mIsSpecDecodingEnabled, mUseSpecDecoding, mIsSpecDecTree, mSpecDecodingIsGenerationLengthVariable, - mSpecDecodingMaxGenerationLength, mSpecDecodingTargetMaxGenLen, mForcePrepareSpecDecTreeMask, mIsMLAEnabled, - mIsGenerationMLA, mUseGenFlashMLA, mUseSparseAttention, mUseTllmGenSparseAttentionPaged, - mUseTllmGenSparseAttention, mMLAParams.data(), mCpSize, mCpRank, mCpGroup, mNumAttnHeads, mNumAttnKVHeads, - mNumKVHeadsOrigin, mAttnTpSize, mAttnTpRank, mAttnCpSize, mAttnCpRank, mUlyssesMQABroadcast, - mEnableContextFMHA, mFMHAForceFP32Acc, mMultiBlockMode, mEnableXQA, mUseKVCache, mSkipAttn, mFuseFp4Quant, - mFusesDsv4InvRopeFp8Quant, mNbMultiBlockSemaphores, mAttentionChunkSize.value_or(-1), - mSkipSoftmaxThresholdScaleFactorPrefill, mSkipSoftmaxThresholdScaleFactorDecode, mSageAttnNumEltsPerBlkQ, - mSageAttnNumEltsPerBlkK, mSageAttnNumEltsPerBlkV, mSageAttnQkInt8); + mSpecDecodingMaxGenerationLength, mSpecDecodingTargetMaxGenLen, mIsMLAEnabled, mIsGenerationMLA, + mUseGenFlashMLA, mUseSparseAttention, mUseTllmGenSparseAttentionPaged, mUseTllmGenSparseAttention, + mMLAParams.data(), mCpSize, mCpRank, mCpGroup, mNumAttnHeads, mNumAttnKVHeads, mNumKVHeadsOrigin, + mAttnTpSize, mAttnTpRank, mAttnCpSize, mAttnCpRank, mUlyssesMQABroadcast, mEnableContextFMHA, + mFMHAForceFP32Acc, mMultiBlockMode, mEnableXQA, mUseKVCache, mSkipAttn, mFuseFp4Quant, + mNbMultiBlockSemaphores, mAttentionChunkSize.value_or(-1), mSkipSoftmaxThresholdScaleFactorPrefill, + mSkipSoftmaxThresholdScaleFactorDecode, mSageAttnNumEltsPerBlkQ, mSageAttnNumEltsPerBlkK, + mSageAttnNumEltsPerBlkV, mSageAttnQkInt8); }; private: diff --git a/cpp/tensorrt_llm/common/cudaFp8Utils.cu b/cpp/tensorrt_llm/common/cudaFp8Utils.cu index 03aae114c94f..9cd0227739d8 100644 --- a/cpp/tensorrt_llm/common/cudaFp8Utils.cu +++ b/cpp/tensorrt_llm/common/cudaFp8Utils.cu @@ -421,45 +421,6 @@ __global__ void computeFP8QuantizeScale(T_S* quant_ptr, const T_W* weights, cons } } -// Vectorized PER_TENSOR amax→scale for bf16 input: 128-bit loads (8 bf16/thread), -// block reduce, one atomicMax per block. Paired with a numel-sized grid -// (scaleMatrixVecGridSize) it launches only as many blocks as the input needs, so -// a skinny decode activation ([64, 2048]) contends ~16 blocks on the scale address -// instead of the scalar path's fixed grid(1024) — whose ~1024-way single-address -// atomicMax plus ~900 idle blocks give a data-independent ~6us floor. Mirrors the -// existing vectorized PER_TENSOR *apply* kernel (scaleMatrixPerTensorVec). -template <typename T_S> -__global__ void computeFP8QuantizeScalePerTensorVec(T_S* quant_ptr, __nv_bfloat16 const* weights, int64_t const numel) -{ - constexpr float min_scaling_factor = 1.0f / (FP8_E4M3_MAX * 512.f); - int64_t const vecElements = numel / kVecSize; - int64_t const stride = static_cast<int64_t>(blockDim.x) * gridDim.x; - float max = 0.f; - for (int64_t vi = threadIdx.x + static_cast<int64_t>(blockIdx.x) * blockDim.x; vi < vecElements; vi += stride) - { - float4 raw = *reinterpret_cast<float4 const*>(weights + vi * kVecSize); - __nv_bfloat162 const* pairs = reinterpret_cast<__nv_bfloat162 const*>(&raw); -#pragma unroll - for (int p = 0; p < kPairsPerVec; ++p) - { - float2 const f2 = __bfloat1622float2(pairs[p]); - max = fmaxf(max, fmaxf(fabsf(f2.x), fabsf(f2.y))); - } - } - // scalar tail (numel not divisible by 8) - int64_t const tailStart = vecElements * kVecSize; - for (int64_t i = tailStart + threadIdx.x + static_cast<int64_t>(blockIdx.x) * blockDim.x; i < numel; i += stride) - { - max = fmaxf(max, fabsf(static_cast<float>(weights[i]))); - } - max = blockReduceMax<float>(max); - if (threadIdx.x == 0) - { - auto const scale = (T_S) std::max(max / FP8_E4M3_MAX, min_scaling_factor); - atomicMaxExtd(quant_ptr, scale); - } -} - template <typename T_S, typename T_W> void invokeComputeFP8QuantizeScale(T_S* quant_ptr, const T_W* weights, const int64_t numel, const int64_t lda, QuantizeMode quantize_mode, cudaStream_t stream) @@ -480,34 +441,11 @@ void invokeComputeFP8QuantizeScale(T_S* quant_ptr, const T_W* weights, const int } else if (quantize_mode == QuantizeMode::PER_TENSOR) { + dim3 block(1024); + dim3 grid(1024); cudaMemsetAsync(quant_ptr, 0, sizeof(T_S), stream); sync_check_cuda_error(stream); - // Size the grid to the input (scaleMatrixVecGridSize), NOT a fixed 1024, so - // small activations launch only a handful of blocks — this is the fix for - // the ~6us fixed floor at skinny decode shapes. For 16B-aligned bf16 input - // use the vectorized amax kernel; otherwise the scalar kernel, still with a - // numel-sized grid. - dim3 const block(CTA_SIZE); - dim3 const grid(static_cast<unsigned int>(scaleMatrixVecGridSize(numel))); - bool const aligned = (reinterpret_cast<uintptr_t>(weights) % 16 == 0); - if constexpr (std::is_same_v<T_W, __nv_bfloat16>) - { - if (aligned) - { - computeFP8QuantizeScalePerTensorVec<<<grid, block, 0, stream>>>( - quant_ptr, reinterpret_cast<__nv_bfloat16 const*>(weights), numel); - } - else - { - computeFP8QuantizeScale<QuantizeMode::PER_TENSOR> - <<<grid, block, 0, stream>>>(quant_ptr, weights, numel, lda); - } - } - else - { - computeFP8QuantizeScale<QuantizeMode::PER_TENSOR> - <<<grid, block, 0, stream>>>(quant_ptr, weights, numel, lda); - } + computeFP8QuantizeScale<QuantizeMode::PER_TENSOR><<<grid, block, 0, stream>>>(quant_ptr, weights, numel, lda); } sync_check_cuda_error(stream); } @@ -597,28 +535,11 @@ void invokeComputeScalesAndQuantizeMatrix(T_OUT* output, T_S* quant_ptr, const T } else if (quantize_mode == QuantizeMode::PER_TENSOR) { + dim3 block(1024); + dim3 grid(1024); cudaMemsetAsync(quant_ptr, 0, sizeof(T_S), stream); sync_check_cuda_error(stream); - // Size the amax grid to the input (scaleMatrixVecGridSize), NOT a fixed 1024, - // so skinny decode activations launch only a handful of blocks — this is the - // fix for the ~6us fixed floor. bf16 (16B-aligned) uses the vectorized amax - // kernel; otherwise the scalar kernel, still numel-sized. Then apply as before. - dim3 const block(CTA_SIZE); - dim3 const grid(static_cast<unsigned int>(scaleMatrixVecGridSize(numel))); - bool const aligned = (reinterpret_cast<uintptr_t>(input) % 16 == 0); - if constexpr (std::is_same_v<T_IN, __nv_bfloat16>) - { - if (aligned) - computeFP8QuantizeScalePerTensorVec<<<grid, block, 0, stream>>>( - quant_ptr, reinterpret_cast<__nv_bfloat16 const*>(input), numel); - else - computeFP8QuantizeScale<QuantizeMode::PER_TENSOR> - <<<grid, block, 0, stream>>>(quant_ptr, input, numel, lda); - } - else - { - computeFP8QuantizeScale<QuantizeMode::PER_TENSOR><<<grid, block, 0, stream>>>(quant_ptr, input, numel, lda); - } + computeFP8QuantizeScale<QuantizeMode::PER_TENSOR><<<grid, block, 0, stream>>>(quant_ptr, input, numel, lda); sync_check_cuda_error(stream); invokeQuantizeMatrix(output, quant_ptr, input, numel, lda, quantize_mode, stream); } diff --git a/cpp/tensorrt_llm/common/envUtils.cpp b/cpp/tensorrt_llm/common/envUtils.cpp index 9dcf6c81634e..efef0362f596 100644 --- a/cpp/tensorrt_llm/common/envUtils.cpp +++ b/cpp/tensorrt_llm/common/envUtils.cpp @@ -401,12 +401,6 @@ bool getEnvTryZCopyForKVCacheTransfer() return zcopyForSysmmetricKVCache; } -bool getEnvDisaggEnableInflightCancel() -{ - static bool const enabled = getBoolEnv("TRTLLM_DISAGG_ENABLE_INFLIGHT_CANCEL"); - return enabled; -} - bool getEnvForceDeterministic() { static bool const forceDeterministic = getBoolEnv("FORCE_DETERMINISTIC"); @@ -517,10 +511,10 @@ uint16_t getEnvNixlPort() return nixlPort; } -bool getEnvNixlDisableCoalesce() +bool getEnvNixlEnableCoalesce() { - static bool const disableCoalesce = getBoolEnv("TRTLLM_NIXL_DISABLE_COALESCE"); - return disableCoalesce; + static bool const enableCoalesce = getBoolEnv("TRTLLM_NIXL_ENABLE_COALESCE"); + return enableCoalesce; } bool getEnvDisaggBenchmarkGenOnly() diff --git a/cpp/tensorrt_llm/common/envUtils.h b/cpp/tensorrt_llm/common/envUtils.h index 13ad0399d574..649813253772 100644 --- a/cpp/tensorrt_llm/common/envUtils.h +++ b/cpp/tensorrt_llm/common/envUtils.h @@ -115,9 +115,6 @@ std::string const& getEnvKVCacheTimeOutputPath(); bool getEnvTryZCopyForKVCacheTransfer(); -// Opt-in for disaggregated KV transfer in-flight cancellation and fail-closed transfer-buffer quarantine. -bool getEnvDisaggEnableInflightCancel(); - // Force deterministic behavior for all kernels. bool getEnvForceDeterministic(); @@ -151,8 +148,7 @@ bool getEnvKVCachePoolUseFabricMemory(); uint16_t getEnvNixlPort(); -// Whether to disable coalescing of contiguous NIXL transfer descriptors (coalescing is on by default). -bool getEnvNixlDisableCoalesce(); +bool getEnvNixlEnableCoalesce(); bool getEnvDisaggBenchmarkGenOnly(); diff --git a/cpp/tensorrt_llm/common/opUtils.cpp b/cpp/tensorrt_llm/common/opUtils.cpp index 560750c2ba93..ff9b57cdd099 100644 --- a/cpp/tensorrt_llm/common/opUtils.cpp +++ b/cpp/tensorrt_llm/common/opUtils.cpp @@ -17,7 +17,6 @@ #include "tensorrt_llm/common/opUtils.h" #include "tensorrt_llm/common/ncclUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/ipcNvlsMemory.h" #include "tensorrt_llm/runtime/utils/mpiTags.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" @@ -34,18 +33,18 @@ TRTLLM_NAMESPACE_BEGIN #if ENABLE_MULTI_DEVICE -std::unordered_map<tensorrt_llm::DataType, ncclDataType_t>* getDtypeMap() +std::unordered_map<nvinfer1::DataType, ncclDataType_t>* getDtypeMap() { - static std::unordered_map<tensorrt_llm::DataType, ncclDataType_t> dtypeMap = { - {tensorrt_llm::DataType::kFLOAT, ncclFloat32}, - {tensorrt_llm::DataType::kHALF, ncclFloat16}, - {tensorrt_llm::DataType::kBF16, ncclBfloat16}, - {tensorrt_llm::DataType::kFP8, ncclInt8}, - {tensorrt_llm::DataType::kBOOL, ncclInt8}, - {tensorrt_llm::DataType::kINT32, ncclInt32}, - {tensorrt_llm::DataType::kINT64, ncclInt64}, - {tensorrt_llm::DataType::kUINT8, ncclUint8}, - {tensorrt_llm::DataType::kINT8, ncclInt8}, + static std::unordered_map<nvinfer1::DataType, ncclDataType_t> dtypeMap = { + {nvinfer1::DataType::kFLOAT, ncclFloat32}, + {nvinfer1::DataType::kHALF, ncclFloat16}, + {nvinfer1::DataType::kBF16, ncclBfloat16}, + {nvinfer1::DataType::kFP8, ncclInt8}, + {nvinfer1::DataType::kBOOL, ncclInt8}, + {nvinfer1::DataType::kINT32, ncclInt32}, + {nvinfer1::DataType::kINT64, ncclInt64}, + {nvinfer1::DataType::kUINT8, ncclUint8}, + {nvinfer1::DataType::kINT8, ncclInt8}, }; return &dtypeMap; } diff --git a/cpp/tensorrt_llm/common/opUtils.h b/cpp/tensorrt_llm/common/opUtils.h index 22169843a9f5..72e5a5ea3e09 100644 --- a/cpp/tensorrt_llm/common/opUtils.h +++ b/cpp/tensorrt_llm/common/opUtils.h @@ -21,7 +21,7 @@ #include "tensorrt_llm/common/cublasMMWrapper.h" #include "tensorrt_llm/common/workspace.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cublasLt.h> #include <cublas_v2.h> #include <cuda_runtime.h> @@ -62,14 +62,14 @@ void read(char const*& buffer, T& val) buffer += sizeof(T); } -inline cudaDataType_t trtToCublasDtype(tensorrt_llm::DataType type) +inline cudaDataType_t trtToCublasDtype(nvinfer1::DataType type) { switch (type) { - case tensorrt_llm::DataType::kFLOAT: return CUDA_R_32F; - case tensorrt_llm::DataType::kHALF: return CUDA_R_16F; + case nvinfer1::DataType::kFLOAT: return CUDA_R_32F; + case nvinfer1::DataType::kHALF: return CUDA_R_16F; #if defined(NV_TENSORRT_MAJOR) && NV_TENSORRT_MAJOR >= 9 - case tensorrt_llm::DataType::kBF16: return CUDA_R_16BF; + case nvinfer1::DataType::kBF16: return CUDA_R_16BF; #endif default: TLLM_THROW("Not supported data type for cuBLAS"); } @@ -185,6 +185,13 @@ struct hash void const* getCommSessionHandle(); } // namespace common::op +inline bool isBuilding() +{ + auto constexpr key = "IS_BUILDING"; + auto const val = getenv(key); + return val != nullptr && std::string(val) == "1"; +} + #if ENABLE_MULTI_DEVICE #define NCCLCHECK(cmd) \ do \ @@ -207,7 +214,7 @@ void const* getCommSessionHandle(); } \ } while (0) -std::unordered_map<tensorrt_llm::DataType, ncclDataType_t>* getDtypeMap(); +std::unordered_map<nvinfer1::DataType, ncclDataType_t>* getDtypeMap(); std::shared_ptr<ncclComm_t> getComm(std::set<int> const& group); diff --git a/cpp/tensorrt_llm/common/safetensors.cpp b/cpp/tensorrt_llm/common/safetensors.cpp index 8bd91ccfbd51..9171f79e44e5 100644 --- a/cpp/tensorrt_llm/common/safetensors.cpp +++ b/cpp/tensorrt_llm/common/safetensors.cpp @@ -18,7 +18,7 @@ #include "nlohmann/json.hpp" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cstdint> #include <fstream> #include <map> @@ -30,7 +30,7 @@ TRTLLM_NAMESPACE_BEGIN namespace common::safetensors { -using tensorrt_llm::DataType; +using nvinfer1::DataType; static DataType convertDataTypeStrToEnum(std::string const& str) { diff --git a/cpp/tensorrt_llm/common/safetensors.h b/cpp/tensorrt_llm/common/safetensors.h index bdecf95e5909..e31225f1be24 100644 --- a/cpp/tensorrt_llm/common/safetensors.h +++ b/cpp/tensorrt_llm/common/safetensors.h @@ -18,7 +18,7 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cstdint> #include <map> #include <memory> @@ -34,13 +34,13 @@ class INdArray [[nodiscard]] virtual void const* data() const = 0; [[nodiscard]] virtual int ndim() const = 0; [[nodiscard]] virtual std::vector<int64_t> const& dims() const = 0; - [[nodiscard]] virtual tensorrt_llm::DataType dtype() const = 0; + [[nodiscard]] virtual nvinfer1::DataType dtype() const = 0; - [[nodiscard]] tensorrt_llm::Dims trtDims() const + [[nodiscard]] nvinfer1::Dims trtDims() const { - tensorrt_llm::Dims dims; + nvinfer1::Dims dims; dims.nbDims = ndim(); - TLLM_CHECK(dims.nbDims <= tensorrt_llm::Dims::MAX_DIMS); + TLLM_CHECK(dims.nbDims <= nvinfer1::Dims::MAX_DIMS); memset(dims.d, 0, sizeof(dims.d)); for (int i = 0; i < dims.nbDims; ++i) { diff --git a/cpp/tensorrt_llm/executor/CMakeLists.txt b/cpp/tensorrt_llm/executor/CMakeLists.txt index 358494e9989c..ca1ab298d7f4 100644 --- a/cpp/tensorrt_llm/executor/CMakeLists.txt +++ b/cpp/tensorrt_llm/executor/CMakeLists.txt @@ -27,7 +27,9 @@ set(SRCS contextPhaseParams.cpp debugConfig.cpp decodingConfig.cpp + executor.cpp executorConfig.cpp + executorImpl.cpp executorKVCacheEventManager.cpp extendedRuntimePerfKnobConfig.cpp guidedDecodingConfig.cpp @@ -50,6 +52,7 @@ set(SRCS response.cpp samplingConfig.cpp dynamicBatchConfig.cpp + dynamicBatchTuner.cpp schedulerConfig.cpp serialization.cpp speculativeDecodingConfig.cpp @@ -57,6 +60,7 @@ set(SRCS types.cpp requestUtils.cpp contextPhaseParams.cpp + disaggServerUtil.cpp cacheTransceiverConfig.cpp) if(NOT WIN32) diff --git a/cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.cpp b/cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.cpp index 2be557ac1f54..a21d470b898a 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.cpp +++ b/cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.cpp @@ -189,45 +189,8 @@ void AgentConnection::send(DataContext const& ctx, void const* data, size_t size NotificationInfo notificationInfo{syncInfo}; std::stringstream ss; NotificationInfo::serialize(notificationInfo, ss); - bool const inflightCancelEnabled = common::getEnvDisaggEnableInflightCancel(); - TransferState transferState; - if (!inflightCancelEnabled) - { - transferState = status->wait(); - } - else - { - static constexpr int64_t kCancelPollTimeoutMs = 100; - transferState = TransferState::kIN_PROGRESS; - while (transferState == TransferState::kIN_PROGRESS) - { - transferState = status->wait(kCancelPollTimeoutMs); - if (transferState == TransferState::kIN_PROGRESS - && ctx.getTransferTerminate().load(std::memory_order_relaxed)) - { - bool const released = status->release(); - TLLM_LOG_WARNING( - "AgentConnection::send cancelled while transfer was in progress (ctx tag=%d, remote=%s, " - "releaseAccepted=%d)", - ctx.getTag(), mRemoteAgentName.c_str(), released); - TLLM_CHECK_WITH_INFO( - released, "AgentConnection::send cancel could not release the backend transfer handle"); - TLLM_THROW("AgentConnection::send cancelled mid-transfer"); - } - } - } + TransferState transferState = status->wait(); TLLM_CHECK_WITH_INFO(transferState == TransferState::kSUCCESS, "AgentConnection::send failed"); - if (inflightCancelEnabled && ctx.getTransferTerminate().load(std::memory_order_relaxed)) - { - bool const released = status->release(); - TLLM_LOG_WARNING( - "AgentConnection::send cancelled after transfer completed but before notify (ctx tag=%d, remote=%s, " - "releaseAccepted=%d)", - ctx.getTag(), mRemoteAgentName.c_str(), released); - TLLM_CHECK_WITH_INFO( - released, "AgentConnection::send pre-notify cancel could not release the backend transfer handle"); - TLLM_THROW("AgentConnection::send cancelled pre-notify"); - } // TODO: there is a bug in request_with_notify https://github.com/ai-dynamo/nixl/pull/252 mAgentConnectionManager->getAgent()->notifySyncMessage(mRemoteAgentName, ss.str()); } @@ -236,19 +199,11 @@ void AgentConnection::recv(DataContext const& ctx, void* data, size_t size) cons { NotificationSyncInfo syncInfo{mAgentName, ctx}; - bool const received - = mAgentConnectionManager->waitForSyncInfo(mRemoteAgentName, syncInfo, ctx.getTransferTerminate()); - if (common::getEnvDisaggEnableInflightCancel()) - { - TLLM_CHECK_WITH_INFO(received, - "AgentConnection::recv ended before receiving sync notification (ctx tag=%d, remote=%s)", ctx.getTag(), - mRemoteAgentName.c_str()); - } + mAgentConnectionManager->waitForSyncInfo(mRemoteAgentName, syncInfo, ctx.getTransferTerminate()); } void AgentConnection::sendRequestAndBufferInfo(batch_manager::RequestInfo& requestInfo, - std::vector<std::optional<size_t>> const& cacheBufferIds, int connectionIdx, - std::atomic<bool> const* perRequestCancel) + std::vector<std::optional<size_t>> const& cacheBufferIds, int connectionIdx) { TLLM_CHECK(!common::getEnvTryZCopyForKVCacheTransfer()); @@ -300,11 +255,6 @@ void AgentConnection::sendRequestAndBufferInfo(batch_manager::RequestInfo& reque std::stringstream ss; NotificationInfo notificationInfo{requestAndBufferInfo}; NotificationInfo::serialize(notificationInfo, ss); - if (common::getEnvDisaggEnableInflightCancel() && perRequestCancel != nullptr - && perRequestCancel->load(std::memory_order_relaxed)) - { - TLLM_THROW("sendRequestAndBufferInfo cancelled before notify"); - } mAgentConnectionManager->getAgent()->notifySyncMessage(mRemoteAgentName, ss.str()); } @@ -341,17 +291,9 @@ void AgentConnection::sendReadySignal(DataContext const& ctx, bool isReady) cons } bool AgentConnection::recvReadySignal(DataContext const& ctx) const -{ - return recvReadySignalWithStatus(ctx).value_or(false); -} - -std::optional<bool> AgentConnection::recvReadySignalWithStatus(DataContext const& ctx) const { ReadySignalInfo readySignalInfo{mAgentName, ctx, false}; - if (!mAgentConnectionManager->waitForReadySignal(mRemoteAgentName, readySignalInfo, ctx.getTransferTerminate())) - { - return std::nullopt; - } + mAgentConnectionManager->waitForReadySignal(mRemoteAgentName, readySignalInfo, ctx.getTransferTerminate()); return readySignalInfo.mIsReady; } @@ -750,7 +692,7 @@ int AgentConnectionManager::getDeviceId() const } template <typename NotificationType> -bool AgentConnectionManager::waitForNotification( +void AgentConnectionManager::waitForNotification( std::string const& remoteAgentName, NotificationType& expectedInfo, std::atomic<bool> const& terminateFlag) { while (!terminateFlag.load()) @@ -758,7 +700,7 @@ bool AgentConnectionManager::waitForNotification( if (!mIsRunning) { - return false; + return; } updateUnhandledNotifications(); std::scoped_lock lock(mNotificationMutex); @@ -791,7 +733,7 @@ bool AgentConnectionManager::waitForNotification( { it = mUnhandledNotifications.erase(it); } - return true; + return; } } } @@ -811,7 +753,7 @@ bool AgentConnectionManager::waitForNotification( { it = mUnhandledNotifications.erase(it); } - return true; + return; } } } @@ -831,25 +773,24 @@ bool AgentConnectionManager::waitForNotification( } } } - return false; } // Explicit template instantiations -template bool AgentConnectionManager::waitForNotification<NotificationSyncInfo>( +template void AgentConnectionManager::waitForNotification<NotificationSyncInfo>( std::string const& remoteAgentName, NotificationSyncInfo& expectedInfo, std::atomic<bool> const& terminateFlag); -template bool AgentConnectionManager::waitForNotification<ReadySignalInfo>( +template void AgentConnectionManager::waitForNotification<ReadySignalInfo>( std::string const& remoteAgentName, ReadySignalInfo& expectedInfo, std::atomic<bool> const& terminateFlag); -bool AgentConnectionManager::waitForSyncInfo( +void AgentConnectionManager::waitForSyncInfo( std::string const& remoteAgentName, NotificationSyncInfo& syncInfo, std::atomic<bool> const& terminateFlag) { - return waitForNotification(remoteAgentName, syncInfo, terminateFlag); + waitForNotification(remoteAgentName, syncInfo, terminateFlag); } -bool AgentConnectionManager::waitForReadySignal( +void AgentConnectionManager::waitForReadySignal( std::string const& remoteAgentName, ReadySignalInfo& readySignalInfo, std::atomic<bool> const& terminateFlag) { - return waitForNotification(remoteAgentName, readySignalInfo, terminateFlag); + waitForNotification(remoteAgentName, readySignalInfo, terminateFlag); } std::string const& AgentConnectionManager::getAgentName() const diff --git a/cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.h b/cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.h index 410eff0248c1..25283d1341a1 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.h +++ b/cpp/tensorrt_llm/executor/cache_transmission/agent_utils/connection.h @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -260,15 +260,13 @@ class AgentConnection : public Connection void send(DataContext const& ctx, void const* data, size_t size) const override; void recv(DataContext const& ctx, void* data, size_t size) const override; void sendRequestAndBufferInfo(batch_manager::RequestInfo& requestInfo, - std::vector<std::optional<size_t>> const& cacheBufferIds, int validConnectionIdx, - std::atomic<bool> const* perRequestCancel = nullptr); - void setSenderState(std::vector<MemoryDesc> cacheReceiverBufferDescs, int validSegmentIdx, + std::vector<std::optional<size_t>> const& cacheBufferIds, int validConnectionIdx); + void setSenderState(std::vector<MemoryDesc> cacheReceiverBufferDescs, int valideSegmentIdx, std::vector<std::pair<size_t, size_t>> offsetRatios, std::vector<uint8_t> bufferKinds); void setHasLoadRemoteAgent(bool hasLoadRemoteAgent); [[nodiscard]] bool hasLoadRemoteAgent() const; void sendReadySignal(DataContext const& ctx, bool isReady) const; bool recvReadySignal(DataContext const& ctx) const; - std::optional<bool> recvReadySignalWithStatus(DataContext const& ctx) const; void activateBuffer(uint8_t kind) const override; [[nodiscard]] std::optional<size_t> getPreAssignedBufferId(uint8_t kind) const override; @@ -322,11 +320,11 @@ class AgentConnectionManager : public ConnectionManager [[nodiscard]] std::string const& getAgentName() const; template <typename NotificationType> - bool waitForNotification( + void waitForNotification( std::string const& remoteAgentName, NotificationType& expectedInfo, std::atomic<bool> const& terminateFlag); - bool waitForSyncInfo( + void waitForSyncInfo( std::string const& remoteAgentName, NotificationSyncInfo& syncInfo, std::atomic<bool> const& terminateFlag); - bool waitForReadySignal( + void waitForReadySignal( std::string const& remoteAgentName, ReadySignalInfo& readySignalInfo, std::atomic<bool> const& terminateFlag); [[nodiscard]] bool isRunning() const override; diff --git a/cpp/tensorrt_llm/executor/cache_transmission/cacheSplitConcat.cu b/cpp/tensorrt_llm/executor/cache_transmission/cacheSplitConcat.cu index d57ed7530e28..6e8c68d7efa7 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/cacheSplitConcat.cu +++ b/cpp/tensorrt_llm/executor/cache_transmission/cacheSplitConcat.cu @@ -20,7 +20,6 @@ #include "tensorrt_llm/common/cudaFp8Utils.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/reduceKernelUtils.cuh" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/executor/tensor.h" #include "tensorrt_llm/executor/types.h" @@ -28,6 +27,7 @@ #include "tensorrt_llm/runtime/iBuffer.h" #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include <NvInferRuntimeBase.h> #include <cstddef> #include <cstdint> #include <sstream> @@ -467,7 +467,7 @@ void concatKVCache(runtime::ITensor::SharedPtr* inputBlocks, int inputBlockNum, blockInfos[outputBlockNum * inputAllRankNum + oi] = fillBlockInfo(oCacheState, outputBlocks[oi], oRank); } runtime::BufferManager::IBufferPtr blockInfosDeviceBuffer - = bufferManager.gpu(sizeof(BlockInfo<T>) * (blockInfos.size()), tensorrt_llm::DataType::kUINT8); + = bufferManager.gpu(sizeof(BlockInfo<T>) * (blockInfos.size()), nvinfer1::DataType::kUINT8); bufferManager.copy((blockInfos.data()), *blockInfosDeviceBuffer, runtime::MemoryType::kCPU); BlockInfo<T>* iBlockInfoDevice = static_cast<BlockInfo<T>*>(blockInfosDeviceBuffer->data()); @@ -594,7 +594,7 @@ void concatKVCacheDispatch(runtime::ITensor::SharedPtr* inputBlocks, int inputBl } } -tensorrt_llm::Dims makeShapeFromCacheState(kv_cache::CacheState const& cacheState) +nvinfer1::Dims makeShapeFromCacheState(kv_cache::CacheState const& cacheState) { int64_t blockSize = static_cast<int64_t>(cacheState.getModelConfig().mNbKvHeadsPerLayer[0] @@ -1130,8 +1130,8 @@ void splitKVCache(std::map<SizeType32, std::vector<runtime::ITensor::SharedPtr>> std::vector<SizeType32> layersInWindow; size_t cacheBlockSizeSum = 0; size_t inputBlockLayerNumSum = 0; - auto cacheDataType = isIndexerKCache ? tensorrt_llm::DataType::kUINT8 - : kVCacheBlocksPerWindow.begin()->second.front()->getDataType(); + auto cacheDataType + = isIndexerKCache ? nvinfer1::DataType::kUINT8 : kVCacheBlocksPerWindow.begin()->second.front()->getDataType(); for (auto const& [window, blocks] : kVCacheBlocksPerWindow) { @@ -1170,7 +1170,7 @@ void splitKVCache(std::map<SizeType32, std::vector<runtime::ITensor::SharedPtr>> bool const isWindow = windowSizes.size() > 1; runtime::BufferManager::IBufferPtr PtrsDeviceBuffer - = bufferManager.gpu(cachePtrs.size(), tensorrt_llm::DataType::kINT64); + = bufferManager.gpu(cachePtrs.size(), nvinfer1::DataType::kINT64); TLLM_CHECK(PtrsDeviceBuffer->getSizeInBytes() == cachePtrs.size() * sizeof(T*)); bufferManager.copy(cachePtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); @@ -1182,7 +1182,7 @@ void splitKVCache(std::map<SizeType32, std::vector<runtime::ITensor::SharedPtr>> windowInfoHostBuffer.insert(windowInfoHostBuffer.end(), blockNumInwindow.begin(), blockNumInwindow.end()); windowInfoHostBuffer.insert(windowInfoHostBuffer.end(), layersInWindow.begin(), layersInWindow.end()); - windowInfoDeviceBuffer = bufferManager.gpu(windowInfoHostBuffer.size(), tensorrt_llm::DataType::kINT32); + windowInfoDeviceBuffer = bufferManager.gpu(windowInfoHostBuffer.size(), nvinfer1::DataType::kINT32); bufferManager.copy(windowInfoHostBuffer.data(), *windowInfoDeviceBuffer, runtime::MemoryType::kCPU); for (auto layerNum : layersInWindow) @@ -1404,8 +1404,8 @@ void splitKVCacheDispatch(std::map<SizeType32, std::vector<runtime::ITensor::Sha bool isIndexerKCache) { TLLM_CHECK(!kVCacheBlocksPerWindow.empty()); - auto dataType = isIndexerKCache ? tensorrt_llm::DataType::kUINT8 - : kVCacheBlocksPerWindow.begin()->second.front()->getDataType(); + auto dataType + = isIndexerKCache ? nvinfer1::DataType::kUINT8 : kVCacheBlocksPerWindow.begin()->second.front()->getDataType(); auto dataSize = tensorrt_llm::common::getDTypeSize(dataType); switch (dataSize) @@ -1513,7 +1513,7 @@ void concatKVCache(std::vector<runtime::ITensor::SharedPtr> const& inputSplitBlo } cachePtrs.insert(cachePtrs.end(), prefixLayerNum.begin(), prefixLayerNum.end()); runtime::BufferManager::IBufferPtr PtrsDeviceBuffer - = bufferManager.gpu(cachePtrs.size(), tensorrt_llm::DataType::kINT64); + = bufferManager.gpu(cachePtrs.size(), nvinfer1::DataType::kINT64); TLLM_CHECK(PtrsDeviceBuffer->getSizeInBytes() == cachePtrs.size() * sizeof(uint64_t)); bufferManager.copy(cachePtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); bool const isWindow = windowSizes.size() > 1; @@ -1525,7 +1525,7 @@ void concatKVCache(std::vector<runtime::ITensor::SharedPtr> const& inputSplitBlo windowInfoHostBuffer.insert(windowInfoHostBuffer.end(), blockNumInwindow.begin(), blockNumInwindow.end()); windowInfoHostBuffer.insert(windowInfoHostBuffer.end(), layersInWindow.begin(), layersInWindow.end()); - windowInfoDeviceBuffer = bufferManager.gpu(windowInfoHostBuffer.size(), tensorrt_llm::DataType::kINT32); + windowInfoDeviceBuffer = bufferManager.gpu(windowInfoHostBuffer.size(), nvinfer1::DataType::kINT32); bufferManager.copy(windowInfoHostBuffer.data(), *windowInfoDeviceBuffer, runtime::MemoryType::kCPU); } constexpr int subWarpSize = 8; diff --git a/cpp/tensorrt_llm/executor/cache_transmission/cacheSplitConcat.h b/cpp/tensorrt_llm/executor/cache_transmission/cacheSplitConcat.h index 80816bc5c3a0..c7036b219612 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/cacheSplitConcat.h +++ b/cpp/tensorrt_llm/executor/cache_transmission/cacheSplitConcat.h @@ -27,7 +27,7 @@ #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntimeBase.h> namespace tensorrt_llm::executor::kv_cache { @@ -78,7 +78,7 @@ void concatKVCacheDispatch(runtime::ITensor::SharedPtr* inputBlocks, int inputBl runtime::ITensor::SharedPtr* outputBlocks, int outputBlockNum, int selfRank, kv_cache::CacheState const& selfCacheState, runtime::BufferManager const& bufferManager); -tensorrt_llm::Dims makeShapeFromCacheState(kv_cache::CacheState const& cacheState); +nvinfer1::Dims makeShapeFromCacheState(kv_cache::CacheState const& cacheState); void splitKVCacheDispatch(std::map<SizeType32, std::vector<runtime::ITensor::SharedPtr>> const& kVCacheBlocksPerWindow, std::vector<runtime::ITensor::SharedPtr>& ouputSplitBlocks, kv_cache::CacheState const& peerCacheState, @@ -147,7 +147,7 @@ void concatRnnSsmStateDispatch(std::vector<runtime::ITensor::SharedPtr> const& i void splitUnifiedPoolSsmDispatch(runtime::ITensor::SharedPtr const& pool, std::vector<SizeType32> const& realBlockIndices, std::vector<runtime::ITensor::SharedPtr>& outputSplitBlocks, kv_cache::CacheState const& destCacheState, kv_cache::CacheState const& selfCacheState, int selfIdx, - size_t ssmBytes, size_t blockSizeBytes, tensorrt_llm::DataType ssmDataType, + size_t ssmBytes, size_t blockSizeBytes, nvinfer1::DataType ssmDataType, runtime::BufferManager const& bufferManager); /** @@ -156,7 +156,7 @@ void splitUnifiedPoolSsmDispatch(runtime::ITensor::SharedPtr const& pool, void splitUnifiedPoolConvDispatch(runtime::ITensor::SharedPtr const& pool, std::vector<SizeType32> const& realBlockIndices, std::vector<runtime::ITensor::SharedPtr>& outputSplitBlocks, kv_cache::CacheState const& destCacheState, kv_cache::CacheState const& selfCacheState, int selfIdx, - size_t ssmBytes, size_t blockSizeBytes, tensorrt_llm::DataType convDataType, + size_t ssmBytes, size_t blockSizeBytes, nvinfer1::DataType convDataType, runtime::BufferManager const& bufferManager); /** @@ -165,7 +165,7 @@ void splitUnifiedPoolConvDispatch(runtime::ITensor::SharedPtr const& pool, void concatUnifiedPoolSsmDispatch(runtime::ITensor::SharedPtr const& pool, std::vector<SizeType32> const& realBlockIndices, std::vector<runtime::ITensor::SharedPtr> const& inputSplitBlocks, kv_cache::CacheState const& srcCacheState, kv_cache::CacheState const& selfCacheState, int selfIdx, size_t ssmBytes, - size_t blockSizeBytes, tensorrt_llm::DataType ssmDataType, runtime::BufferManager const& bufferManager); + size_t blockSizeBytes, nvinfer1::DataType ssmDataType, runtime::BufferManager const& bufferManager); /** * @brief Concat conv state from per-source buffers into unified pool blocks (section-aware). @@ -173,6 +173,6 @@ void concatUnifiedPoolSsmDispatch(runtime::ITensor::SharedPtr const& pool, void concatUnifiedPoolConvDispatch(runtime::ITensor::SharedPtr const& pool, std::vector<SizeType32> const& realBlockIndices, std::vector<runtime::ITensor::SharedPtr> const& inputSplitBlocks, kv_cache::CacheState const& srcCacheState, kv_cache::CacheState const& selfCacheState, int selfIdx, size_t ssmBytes, - size_t blockSizeBytes, tensorrt_llm::DataType convDataType, runtime::BufferManager const& bufferManager); + size_t blockSizeBytes, nvinfer1::DataType convDataType, runtime::BufferManager const& bufferManager); } // namespace tensorrt_llm::executor::rnn_cache diff --git a/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cpp b/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cpp index 040979e7197d..a987ac108f3b 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cpp +++ b/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -261,35 +261,28 @@ NB_MODULE(tensorrt_llm_transfer_agent_binding, m) nb::class_<kvc::NixlTransferAgent, kvc::BaseTransferAgent>(m, "NixlTransferAgent") .def(nb::init<kvc::BaseAgentConfig const&>(), nb::arg("config"), nb::call_guard<nb::gil_scoped_release>()) .def("shutdown", &kvc::NixlTransferAgent::shutdown, nb::call_guard<nb::gil_scoped_release>()) - .def("register_memory", &kvc::NixlTransferAgent::registerMemory, nb::arg("descs"), - nb::call_guard<nb::gil_scoped_release>()) - .def("deregister_memory", &kvc::NixlTransferAgent::deregisterMemory, nb::arg("descs"), - nb::call_guard<nb::gil_scoped_release>()) + .def("register_memory", &kvc::NixlTransferAgent::registerMemory, nb::arg("descs")) + .def("deregister_memory", &kvc::NixlTransferAgent::deregisterMemory, nb::arg("descs")) .def("load_remote_agent", nb::overload_cast<std::string const&, kvc::AgentDesc const&>(&kvc::NixlTransferAgent::loadRemoteAgent), - nb::arg("name"), nb::arg("agent_desc"), nb::call_guard<nb::gil_scoped_release>()) + nb::arg("name"), nb::arg("agent_desc")) .def("load_remote_agent_by_connection", nb::overload_cast<std::string const&, kvc::ConnectionInfoType const&>( &kvc::NixlTransferAgent::loadRemoteAgent), - nb::arg("name"), nb::arg("connection_info"), nb::call_guard<nb::gil_scoped_release>()) - .def("get_local_agent_desc", &kvc::NixlTransferAgent::getLocalAgentDesc, - nb::call_guard<nb::gil_scoped_release>()) - .def("get_local_connection_info", &kvc::NixlTransferAgent::getLocalConnectionInfo, - nb::call_guard<nb::gil_scoped_release>()) - .def("invalidate_remote_agent", &kvc::NixlTransferAgent::invalidateRemoteAgent, nb::arg("name"), - nb::call_guard<nb::gil_scoped_release>()) + nb::arg("name"), nb::arg("connection_info")) + .def("get_local_agent_desc", &kvc::NixlTransferAgent::getLocalAgentDesc) + .def("get_local_connection_info", &kvc::NixlTransferAgent::getLocalConnectionInfo) + .def("invalidate_remote_agent", &kvc::NixlTransferAgent::invalidateRemoteAgent, nb::arg("name")) .def( "submit_transfer_requests", [](kvc::NixlTransferAgent& self, kvc::TransferRequest const& request) { return self.submitTransferRequests(request).release(); }, nb::arg("request"), nb::rv_policy::take_ownership, nb::call_guard<nb::gil_scoped_release>(), nb::keep_alive<0, 1>()) - .def("notify_sync_message", &kvc::NixlTransferAgent::notifySyncMessage, nb::arg("name"), - nb::arg("sync_message"), nb::call_guard<nb::gil_scoped_release>()) - .def("get_notified_sync_messages", &kvc::NixlTransferAgent::getNotifiedSyncMessages, - nb::call_guard<nb::gil_scoped_release>()) - .def("check_remote_descs", &kvc::NixlTransferAgent::checkRemoteDescs, nb::arg("name"), nb::arg("memory_descs"), - nb::call_guard<nb::gil_scoped_release>()); + .def( + "notify_sync_message", &kvc::NixlTransferAgent::notifySyncMessage, nb::arg("name"), nb::arg("sync_message")) + .def("get_notified_sync_messages", &kvc::NixlTransferAgent::getNotifiedSyncMessages) + .def("check_remote_descs", &kvc::NixlTransferAgent::checkRemoteDescs, nb::arg("name"), nb::arg("memory_descs")); #endif // NOTE: MooncakeTransferAgent/MooncakeTransferStatus class bindings are intentionally diff --git a/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp b/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp index 5ab589d7ca75..563b6e6f2a95 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp +++ b/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -321,7 +321,6 @@ void NixlHelper::posixFileToGpuFallback(MemoryDescs const& memoryDescs, FileDesc NixlTransferStatus::NixlTransferStatus(std::weak_ptr<nixlAgent> agent, nixlXferReqH* handle) : mWeakAgent{std::move(agent)} , mHandle{handle} - , mSynchronizeHandleAccess{common::getEnvDisaggEnableInflightCancel()} { TLLM_CHECK(!mWeakAgent.expired()); TLLM_CHECK(mHandle); @@ -329,13 +328,19 @@ NixlTransferStatus::NixlTransferStatus(std::weak_ptr<nixlAgent> agent, nixlXferR NixlTransferStatus::~NixlTransferStatus() noexcept { + if (mHandle == nullptr) + { + return; + } + // Skip release if the owning agent was reset; the underlying nixlXferReqH is already gone. + auto agent = mWeakAgent.lock(); + if (!agent) + { + return; + } try { - if (!release()) - { - TLLM_LOG_WARNING( - "NIXL transfer handle release failed during destruction; backend handle may remain active"); - } + agent->releaseXferReq(mHandle); } catch (std::exception const& e) { @@ -347,13 +352,179 @@ NixlTransferStatus::~NixlTransferStatus() noexcept } } +[[nodiscard]] MemoryDescs NixlHelper::coalesceMemoryDescs(MemoryDescs const& descs) +{ + auto const& descVec = descs.getDescs(); + + // If empty or single element, return as-is + if (descVec.size() <= 1) + { + return descs; + } + + size_t const numDescs = descVec.size(); + + // Create index array and sort by address + std::vector<size_t> sortedIndices(numDescs); + std::iota(sortedIndices.begin(), sortedIndices.end(), 0); + + std::sort(sortedIndices.begin(), sortedIndices.end(), + [&descVec](size_t lhs, size_t rhs) + { + // Sort by deviceId first, then by address + if (descVec[lhs].getDeviceId() != descVec[rhs].getDeviceId()) + { + return descVec[lhs].getDeviceId() < descVec[rhs].getDeviceId(); + } + return descVec[lhs].getAddr() < descVec[rhs].getAddr(); + }); + + std::vector<MemoryDesc> coalesced; + coalesced.reserve(numDescs); + + // Start with the first entry + size_t firstIdx = sortedIndices[0]; + uintptr_t currentAddr = descVec[firstIdx].getAddr(); + size_t currentLen = descVec[firstIdx].getLen(); + uint32_t currentDeviceId = descVec[firstIdx].getDeviceId(); + + for (size_t idx = 1; idx < numDescs; ++idx) + { + size_t sortedIdx = sortedIndices[idx]; + auto const& desc = descVec[sortedIdx]; + + // Check if current can be coalesced with previous + bool isContiguous = (currentAddr + currentLen == desc.getAddr()) && (currentDeviceId == desc.getDeviceId()); + + if (isContiguous) + { + // Coalesce: extend the current region + currentLen += desc.getLen(); + } + else + { + // Cannot coalesce: save the current region and start a new one + coalesced.emplace_back(currentAddr, currentLen, currentDeviceId); + + currentAddr = desc.getAddr(); + currentLen = desc.getLen(); + currentDeviceId = desc.getDeviceId(); + } + } + + // Add the last region + coalesced.emplace_back(currentAddr, currentLen, currentDeviceId); + + TLLM_LOG_DEBUG("NixlHelper::coalesceMemoryDescs: coalesced %zu -> %zu entries", descVec.size(), coalesced.size()); + + return MemoryDescs{descs.getType(), std::move(coalesced)}; +} + +[[nodiscard]] std::pair<MemoryDescs, MemoryDescs> NixlHelper::coalesceTransferDescs( + TransferDescs const& srcDescs, TransferDescs const& dstDescs) +{ + auto const& srcVec = srcDescs.getDescs(); + auto const& dstVec = dstDescs.getDescs(); + + // If sizes don't match or empty, return as-is + if (srcVec.size() != dstVec.size() || srcVec.empty()) + { + return {srcDescs, dstDescs}; + } + + size_t const numDescs = srcVec.size(); + + // Create index array and sort by src address + // This allows us to find contiguous regions even if the original order is scattered + std::vector<size_t> sortedIndices(numDescs); + std::iota(sortedIndices.begin(), sortedIndices.end(), 0); + + std::sort(sortedIndices.begin(), sortedIndices.end(), + [&srcVec](size_t lhs, size_t rhs) + { + // Sort by deviceId first, then by address + if (srcVec[lhs].getDeviceId() != srcVec[rhs].getDeviceId()) + { + return srcVec[lhs].getDeviceId() < srcVec[rhs].getDeviceId(); + } + return srcVec[lhs].getAddr() < srcVec[rhs].getAddr(); + }); + + std::vector<MemoryDesc> coalescedSrc; + std::vector<MemoryDesc> coalescedDst; + coalescedSrc.reserve(numDescs); + coalescedDst.reserve(numDescs); + + // Start with the first entry (using sorted order) + size_t firstIdx = sortedIndices[0]; + uintptr_t currentSrcAddr = srcVec[firstIdx].getAddr(); + size_t currentSrcLen = srcVec[firstIdx].getLen(); + uint32_t currentSrcDeviceId = srcVec[firstIdx].getDeviceId(); + + uintptr_t currentDstAddr = dstVec[firstIdx].getAddr(); + size_t currentDstLen = dstVec[firstIdx].getLen(); + uint32_t currentDstDeviceId = dstVec[firstIdx].getDeviceId(); + + for (size_t idx = 1; idx < numDescs; ++idx) + { + size_t sortedIdx = sortedIndices[idx]; + auto const& src = srcVec[sortedIdx]; + auto const& dst = dstVec[sortedIdx]; + + // Check if current src and dst can be coalesced with previous + bool srcContiguous + = (currentSrcAddr + currentSrcLen == src.getAddr()) && (currentSrcDeviceId == src.getDeviceId()); + bool dstContiguous + = (currentDstAddr + currentDstLen == dst.getAddr()) && (currentDstDeviceId == dst.getDeviceId()); + + if (srcContiguous && dstContiguous) + { + // Coalesce: extend the current region + currentSrcLen += src.getLen(); + currentDstLen += dst.getLen(); + } + else + { + // Cannot coalesce: save the current region and start a new one + coalescedSrc.emplace_back(currentSrcAddr, currentSrcLen, currentSrcDeviceId); + coalescedDst.emplace_back(currentDstAddr, currentDstLen, currentDstDeviceId); + + currentSrcAddr = src.getAddr(); + currentSrcLen = src.getLen(); + currentSrcDeviceId = src.getDeviceId(); + + currentDstAddr = dst.getAddr(); + currentDstLen = dst.getLen(); + currentDstDeviceId = dst.getDeviceId(); + } + } + + // Don't forget to add the last region + coalescedSrc.emplace_back(currentSrcAddr, currentSrcLen, currentSrcDeviceId); + coalescedDst.emplace_back(currentDstAddr, currentDstLen, currentDstDeviceId); + + TLLM_LOG_DEBUG( + "NixlHelper::coalesceTransferDescs: coalesced %zu -> %zu transfer entries", srcVec.size(), coalescedSrc.size()); + + return {MemoryDescs{srcDescs.getType(), std::move(coalescedSrc)}, + MemoryDescs{dstDescs.getType(), std::move(coalescedDst)}}; +} + TransferState NixlTransferStatus::wait(int64_t timeout_ms) const { auto startTime = std::chrono::steady_clock::now(); while (true) { - auto const status = queryStatus(); + auto agent = mWeakAgent.lock(); + if (!agent) + { + // Owning agent was reset; report failure so callers don't deref a null status. + mLastStatus.store(static_cast<int>(NIXL_ERR_INVALID_PARAM), std::memory_order_relaxed); + return TransferState::kFAILURE; + } + auto status = agent->getXferStatus(mHandle); + mLastStatus.store(static_cast<int>(status), std::memory_order_relaxed); if (status == NIXL_SUCCESS) { return TransferState::kSUCCESS; @@ -395,64 +566,15 @@ std::string NixlTransferStatus::getLastStatusStr() const [[nodiscard]] bool NixlTransferStatus::isCompleted() const { - return queryStatus() == NIXL_SUCCESS; -} - -nixl_status_t NixlTransferStatus::queryStatus() const -{ - auto const query = [this]() - { - if (mHandle == nullptr) - { - mLastStatus.store(static_cast<int>(NIXL_ERR_INVALID_PARAM), std::memory_order_relaxed); - return NIXL_ERR_INVALID_PARAM; - } - auto agent = mWeakAgent.lock(); - if (!agent) - { - // Owning agent was reset; report failure so callers don't deref a null status. - mLastStatus.store(static_cast<int>(NIXL_ERR_INVALID_PARAM), std::memory_order_relaxed); - return NIXL_ERR_INVALID_PARAM; - } - auto const status = agent->getXferStatus(mHandle); - mLastStatus.store(static_cast<int>(status), std::memory_order_relaxed); - return status; - }; - - if (mSynchronizeHandleAccess) - { - std::lock_guard<std::mutex> lock(mHandleMutex); - return query(); - } - return query(); -} - -[[nodiscard]] bool NixlTransferStatus::release() -{ - std::lock_guard<std::mutex> lock(mHandleMutex); - if (mHandle == nullptr) - { - return true; - } - auto agent = mWeakAgent.lock(); if (!agent) { - mHandle = nullptr; mLastStatus.store(static_cast<int>(NIXL_ERR_INVALID_PARAM), std::memory_order_relaxed); - return true; + return false; } - - auto status = agent->releaseXferReq(mHandle); + auto status = agent->getXferStatus(mHandle); mLastStatus.store(static_cast<int>(status), std::memory_order_relaxed); - if (status == NIXL_SUCCESS) - { - mHandle = nullptr; - return true; - } - - TLLM_LOG_WARNING("NIXL releaseXferReq failed with status: %s", nixlEnumStrings::statusStr(status).c_str()); - return false; + return status == NIXL_SUCCESS; } NixlTransferAgent::NixlTransferAgent(BaseAgentConfig const& config) @@ -535,8 +657,12 @@ void NixlTransferAgent::registerMemory(RegisterDescs const& descs) auto detectedRegionMap = VmmDescSplitter::detectVramRegionMap(descs); mLocalVramRegionInfo.merge(detectedRegionMap); + // Coalesce contiguous memory regions to reduce registration overhead (disabled by default) + // Set TRTLLM_NIXL_ENABLE_COALESCE=1 to enable this optimization + auto coalescedDescs = common::getEnvNixlEnableCoalesce() ? NixlHelper::coalesceMemoryDescs(splitDescs) : splitDescs; + nixl_status_t status; - status = mRawAgent->registerMem(NixlHelper::convertRegDlist(splitDescs), &mExtraParams); + status = mRawAgent->registerMem(NixlHelper::convertRegDlist(coalescedDescs), &mExtraParams); TLLM_CHECK(status == NIXL_SUCCESS); std::string localMD; @@ -551,8 +677,12 @@ void NixlTransferAgent::deregisterMemory(RegisterDescs const& descs) // Split using per-region registry info to match what was registered auto splitDescs = VmmDescSplitter::splitDescsWithRegionMap(descs, mLocalVramRegionInfo); + // Coalesce contiguous memory regions to match what was registered (disabled by default) + // Set TRTLLM_NIXL_ENABLE_COALESCE=1 to enable this optimization + auto coalescedDescs = common::getEnvNixlEnableCoalesce() ? NixlHelper::coalesceMemoryDescs(splitDescs) : splitDescs; + nixl_status_t status; - status = mRawAgent->deregisterMem(NixlHelper::convertRegDlist(splitDescs), &mExtraParams); + status = mRawAgent->deregisterMem(NixlHelper::convertRegDlist(coalescedDescs), &mExtraParams); TLLM_CHECK(status == NIXL_SUCCESS); // Remove entries from registry @@ -577,7 +707,7 @@ void NixlTransferAgent::loadRemoteAgent(std::string const& name, AgentDesc const name == remoteName, "loadRemoteAgent gets error agent name: %s != %s", name.c_str(), remoteName.c_str()); // Store remote VMM region info for chunk boundary calculations in - // VmmDescSplitter::splitAndCoalesceTransferDescs. Per-agent map because different remote agents may have + // VmmDescSplitter::splitTransferDescsWithRegionMaps. Per-agent map because different remote agents may have // overlapping virtual addresses. auto const& regions = agentDesc.getVramRegions(); if (!regions.empty()) @@ -598,13 +728,14 @@ AgentDesc NixlTransferAgent::getLocalAgentDesc() nixl_status_t status = mRawAgent->getLocalMD(nixlBlob); TLLM_CHECK(status == NIXL_SUCCESS); - // Pack ALL local region info (VMM multi-chunk and single-allocation alike) so remote agents can - // compute chunk boundaries and never coalesce transfer descs across separately registered regions. + // Pack local VMM region info so remote agents can compute chunk boundaries. std::vector<VramRegionMeta> regions; - regions.reserve(mLocalVramRegionInfo.size()); for (auto const& [base, info] : mLocalVramRegionInfo) { - regions.push_back({base, info.totalLen, info.chunkSize}); + if (info.chunkSize > 0) + { + regions.push_back({base, info.totalLen, info.chunkSize}); + } } return AgentDesc{nixlBlob, std::move(regions)}; @@ -642,25 +773,32 @@ void NixlTransferAgent::invalidateRemoteAgent(std::string const& name) { reqParams.hasNotif = false; } - // Split transfer descriptors at VMM chunk boundaries to match registered memory, then coalesce - // contiguous pieces. A coalesced descriptor never crosses a chunk boundary or a registered - // region boundary on either side, so every descriptor still falls within a single registered - // memory region on both local and remote sides. Set TRTLLM_NIXL_DISABLE_COALESCE=1 to fall back - // to split-only descriptors. Find remote agent's region map (empty map if not found — e.g. the - // peer's AgentDesc carried no region info; addresses missing from a map are never coalesced, - // so an empty remote map degrades to split-only rather than risking merges across unknown - // remote chunk/registration boundaries). + // Split transfer descriptors at VMM chunk boundaries to match registered memory. + // Both src and dst are split at chunk boundaries to ensure each descriptor + // falls within a single registered memory region on both local and remote sides. + // Find remote agent's VMM region map (empty map if not found). static VramRegionMap const kEmptyMap; auto remoteIt = mRemoteVramRegionInfo.find(request.getRemoteName()); auto const& remoteRegionMap = (remoteIt != mRemoteVramRegionInfo.end()) ? remoteIt->second : kEmptyMap; - auto [xferSrc, xferDst] = VmmDescSplitter::splitAndCoalesceTransferDescs(request.getSrcDescs(), - request.getDstDescs(), mLocalVramRegionInfo, remoteRegionMap, !common::getEnvNixlDisableCoalesce()); + auto [splitSrc, splitDst] = VmmDescSplitter::splitTransferDescsWithRegionMaps( + request.getSrcDescs(), request.getDstDescs(), mLocalVramRegionInfo, remoteRegionMap); + // Coalesce contiguous memory regions to reduce transfer count (disabled by default) + // This matches the coalescing done during registerMemory() + // Set TRTLLM_NIXL_ENABLE_COALESCE=1 to enable this optimization + if (common::getEnvNixlEnableCoalesce()) + { + NVTX3_SCOPED_RANGE(coalesceTransferDescs_CreateXferReq); + auto [coalescedSrc, coalescedDst] = NixlHelper::coalesceTransferDescs(splitSrc, splitDst); + status + = mRawAgent->createXferReq(NixlHelper::convert(request.getOp()), NixlHelper::convertXferDist(coalescedSrc), + NixlHelper::convertXferDist(coalescedDst), request.getRemoteName(), handle, &reqParams); + } + else { - NVTX3_SCOPED_RANGE(createXferReq); - status = mRawAgent->createXferReq(NixlHelper::convert(request.getOp()), NixlHelper::convertXferDist(xferSrc), - NixlHelper::convertXferDist(xferDst), request.getRemoteName(), handle, &reqParams); + status = mRawAgent->createXferReq(NixlHelper::convert(request.getOp()), NixlHelper::convertXferDist(splitSrc), + NixlHelper::convertXferDist(splitDst), request.getRemoteName(), handle, &reqParams); } TLLM_CHECK_WITH_INFO(status == NIXL_SUCCESS, diff --git a/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.h b/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.h index 31e62fd6d822..a8f3a0e71005 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.h +++ b/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.h @@ -21,7 +21,6 @@ #include "tensorrt_llm/executor/transferAgent.h" #include <atomic> #include <memory> -#include <mutex> #include <shared_mutex> #include <thread> @@ -39,6 +38,21 @@ struct NixlHelper [[nodiscard]] static nixl_xfer_dlist_t convertXferDist(FileDescs const& descs); static void posixGpuToFileFallback(MemoryDescs const& memoryDesc, FileDescs const& fileDescs); static void posixFileToGpuFallback(MemoryDescs const& memoryDesc, FileDescs const& fileDescs); + + /// @brief Coalesce contiguous memory regions to reduce memory registration overhead. + /// Adjacent memory regions with the same deviceId will be merged into a single region. + /// @param descs Memory descriptors to coalesce + /// @return Coalesced MemoryDescs + [[nodiscard]] static MemoryDescs coalesceMemoryDescs(MemoryDescs const& descs); + + /// @brief Coalesce contiguous memory regions in src and dst to reduce transfer count. + /// If src[i] and src[i+1] are contiguous, and dst[i] and dst[i+1] are also contiguous + /// (with same deviceId), they will be merged into a single transfer. + /// @param srcDescs Source memory descriptors + /// @param dstDescs Destination memory descriptors + /// @return Pair of coalesced (src, dst) MemoryDescs + [[nodiscard]] static std::pair<MemoryDescs, MemoryDescs> coalesceTransferDescs( + TransferDescs const& srcDescs, TransferDescs const& dstDescs); }; class NixlTransferStatus final : public TransferStatus @@ -59,17 +73,11 @@ class NixlTransferStatus final : public TransferStatus [[nodiscard]] int getLastStatus() const noexcept; [[nodiscard]] std::string getLastStatusStr() const; - [[nodiscard]] bool release() override; - private: - [[nodiscard]] nixl_status_t queryStatus() const; - // weak_ptr so the status outliving the owning agent is safe (lock() returns null after reset). std::weak_ptr<nixlAgent> mWeakAgent; nixlXferReqH* mHandle{}; mutable std::atomic<int> mLastStatus{0}; - bool const mSynchronizeHandleAccess; - mutable std::mutex mHandleMutex; }; class NixlTransferAgent final : public BaseTransferAgent diff --git a/cpp/tensorrt_llm/executor/cache_transmission/rnnCacheSplitConcat.cu b/cpp/tensorrt_llm/executor/cache_transmission/rnnCacheSplitConcat.cu index f464c41aadc9..b697f2eeb3d6 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/rnnCacheSplitConcat.cu +++ b/cpp/tensorrt_llm/executor/cache_transmission/rnnCacheSplitConcat.cu @@ -24,7 +24,6 @@ #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/memoryUtils.h" #include "tensorrt_llm/common/reduceKernelUtils.cuh" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/executor/tensor.h" #include "tensorrt_llm/executor/types.h" @@ -32,6 +31,7 @@ #include "tensorrt_llm/runtime/iBuffer.h" #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include <NvInferRuntimeBase.h> #include <cstddef> #include <cstdint> #include <sstream> @@ -366,7 +366,7 @@ void splitRnnConvState(std::vector<runtime::ITensor::SharedPtr> const& inputConv // Allocate and copy pointer array to device runtime::BufferManager::IBufferPtr PtrsDeviceBuffer - = bufferManager.gpu(cachePtrs.size(), tensorrt_llm::DataType::kINT64); + = bufferManager.gpu(cachePtrs.size(), nvinfer1::DataType::kINT64); TLLM_CHECK(PtrsDeviceBuffer->getSizeInBytes() == cachePtrs.size() * sizeof(uint64_t)); bufferManager.copy(cachePtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); @@ -497,7 +497,7 @@ void splitRnnSsmState(std::vector<runtime::ITensor::SharedPtr> const& inputSsmBl cachePtrs.insert(cachePtrs.end(), prefixLayerNum.begin(), prefixLayerNum.end()); runtime::BufferManager::IBufferPtr PtrsDeviceBuffer - = bufferManager.gpu(cachePtrs.size(), tensorrt_llm::DataType::kINT64); + = bufferManager.gpu(cachePtrs.size(), nvinfer1::DataType::kINT64); TLLM_CHECK(PtrsDeviceBuffer->getSizeInBytes() == cachePtrs.size() * sizeof(uint64_t)); bufferManager.copy(cachePtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); @@ -625,7 +625,7 @@ void concatRnnConvState(std::vector<runtime::ITensor::SharedPtr> const& inputSpl cachePtrs.insert(cachePtrs.end(), prefixLayerNum.begin(), prefixLayerNum.end()); runtime::BufferManager::IBufferPtr PtrsDeviceBuffer - = bufferManager.gpu(cachePtrs.size(), tensorrt_llm::DataType::kINT64); + = bufferManager.gpu(cachePtrs.size(), nvinfer1::DataType::kINT64); TLLM_CHECK(PtrsDeviceBuffer->getSizeInBytes() == cachePtrs.size() * sizeof(uint64_t)); bufferManager.copy(cachePtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); @@ -747,7 +747,7 @@ void concatRnnSsmState(std::vector<runtime::ITensor::SharedPtr> const& inputSpli cachePtrs.insert(cachePtrs.end(), prefixLayerNum.begin(), prefixLayerNum.end()); runtime::BufferManager::IBufferPtr PtrsDeviceBuffer - = bufferManager.gpu(cachePtrs.size(), tensorrt_llm::DataType::kINT64); + = bufferManager.gpu(cachePtrs.size(), nvinfer1::DataType::kINT64); TLLM_CHECK(PtrsDeviceBuffer->getSizeInBytes() == cachePtrs.size() * sizeof(uint64_t)); bufferManager.copy(cachePtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); @@ -964,7 +964,8 @@ void concatRnnSsmStateDispatch(std::vector<runtime::ITensor::SharedPtr> const& i // SSM portion: [numHeads, headDim, dState] — split by heads // Conv portion: [section0_dim, section1_dim, ...] x [dConv-1] — section-aware split // -// The input/output pointer arrays follow the pattern: input pointers → output pointers → prefixLayerNum. +// The input/output pointer arrays follow the same pattern as the existing +// RnnStateManager kernels: input pointers → output pointers → prefixLayerNum. /** * @brief Kernel to split SSM state from unified pool blocks to per-target buffers. @@ -1348,7 +1349,7 @@ void splitUnifiedPoolSsm(runtime::ITensor::SharedPtr const& pool, std::vector<Si } allPtrs.insert(allPtrs.end(), prefixLayerNum.begin(), prefixLayerNum.end()); - auto PtrsDeviceBuffer = bufferManager.gpu(allPtrs.size(), tensorrt_llm::DataType::kINT64); + auto PtrsDeviceBuffer = bufferManager.gpu(allPtrs.size(), nvinfer1::DataType::kINT64); bufferManager.copy(allPtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); T const** inputPtrsDev = static_cast<T const**>(PtrsDeviceBuffer->data()); @@ -1481,10 +1482,10 @@ void splitUnifiedPoolConv(runtime::ITensor::SharedPtr const& pool, std::vector<S sectionInfo.insert(sectionInfo.end(), sectionDimsDomainTP.begin(), sectionDimsDomainTP.end()); sectionInfo.insert(sectionInfo.end(), sectionOffsetsLocal.begin(), sectionOffsetsLocal.end()); - auto PtrsDeviceBuffer = bufferManager.gpu(allPtrs.size(), tensorrt_llm::DataType::kINT64); + auto PtrsDeviceBuffer = bufferManager.gpu(allPtrs.size(), nvinfer1::DataType::kINT64); bufferManager.copy(allPtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); - auto sectionInfoBuffer = bufferManager.gpu(sectionInfo.size(), tensorrt_llm::DataType::kINT32); + auto sectionInfoBuffer = bufferManager.gpu(sectionInfo.size(), nvinfer1::DataType::kINT32); bufferManager.copy(sectionInfo.data(), *sectionInfoBuffer, runtime::MemoryType::kCPU); T const** inputPtrsDev = static_cast<T const**>(PtrsDeviceBuffer->data()); @@ -1606,7 +1607,7 @@ void concatUnifiedPoolSsm(runtime::ITensor::SharedPtr const& pool, std::vector<S } allPtrs.insert(allPtrs.end(), prefixLayerNum.begin(), prefixLayerNum.end()); - auto PtrsDeviceBuffer = bufferManager.gpu(allPtrs.size(), tensorrt_llm::DataType::kINT64); + auto PtrsDeviceBuffer = bufferManager.gpu(allPtrs.size(), nvinfer1::DataType::kINT64); bufferManager.copy(allPtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); T** outputPtrsDev = static_cast<T**>(PtrsDeviceBuffer->data()); @@ -1731,10 +1732,10 @@ void concatUnifiedPoolConv(runtime::ITensor::SharedPtr const& pool, std::vector< sectionInfo.insert(sectionInfo.end(), sectionDimsDomainTP.begin(), sectionDimsDomainTP.end()); sectionInfo.insert(sectionInfo.end(), sectionOffsetsLocal.begin(), sectionOffsetsLocal.end()); - auto PtrsDeviceBuffer = bufferManager.gpu(allPtrs.size(), tensorrt_llm::DataType::kINT64); + auto PtrsDeviceBuffer = bufferManager.gpu(allPtrs.size(), nvinfer1::DataType::kINT64); bufferManager.copy(allPtrs.data(), *PtrsDeviceBuffer, runtime::MemoryType::kCPU); - auto sectionInfoBuffer = bufferManager.gpu(sectionInfo.size(), tensorrt_llm::DataType::kINT32); + auto sectionInfoBuffer = bufferManager.gpu(sectionInfo.size(), nvinfer1::DataType::kINT32); bufferManager.copy(sectionInfo.data(), *sectionInfoBuffer, runtime::MemoryType::kCPU); T** outputPtrsDev = static_cast<T**>(PtrsDeviceBuffer->data()); @@ -1810,8 +1811,7 @@ void concatUnifiedPoolConv(runtime::ITensor::SharedPtr const& pool, std::vector< void splitUnifiedPoolSsmDispatch(runtime::ITensor::SharedPtr const& pool, std::vector<SizeType32> const& realBlockIndices, std::vector<runtime::ITensor::SharedPtr>& outputSplitBlocks, kv_cache::CacheState const& destCacheState, kv_cache::CacheState const& selfCacheState, int selfIdx, - size_t ssmBytes, size_t blockSizeBytes, tensorrt_llm::DataType ssmDataType, - runtime::BufferManager const& bufferManager) + size_t ssmBytes, size_t blockSizeBytes, nvinfer1::DataType ssmDataType, runtime::BufferManager const& bufferManager) { auto dataSize = tensorrt_llm::common::getDTypeSize(ssmDataType); switch (dataSize) @@ -1835,7 +1835,7 @@ void splitUnifiedPoolSsmDispatch(runtime::ITensor::SharedPtr const& pool, void splitUnifiedPoolConvDispatch(runtime::ITensor::SharedPtr const& pool, std::vector<SizeType32> const& realBlockIndices, std::vector<runtime::ITensor::SharedPtr>& outputSplitBlocks, kv_cache::CacheState const& destCacheState, kv_cache::CacheState const& selfCacheState, int selfIdx, - size_t ssmBytes, size_t blockSizeBytes, tensorrt_llm::DataType convDataType, + size_t ssmBytes, size_t blockSizeBytes, nvinfer1::DataType convDataType, runtime::BufferManager const& bufferManager) { auto dataSize = tensorrt_llm::common::getDTypeSize(convDataType); @@ -1860,7 +1860,7 @@ void splitUnifiedPoolConvDispatch(runtime::ITensor::SharedPtr const& pool, void concatUnifiedPoolSsmDispatch(runtime::ITensor::SharedPtr const& pool, std::vector<SizeType32> const& realBlockIndices, std::vector<runtime::ITensor::SharedPtr> const& inputSplitBlocks, kv_cache::CacheState const& srcCacheState, kv_cache::CacheState const& selfCacheState, int selfIdx, size_t ssmBytes, - size_t blockSizeBytes, tensorrt_llm::DataType ssmDataType, runtime::BufferManager const& bufferManager) + size_t blockSizeBytes, nvinfer1::DataType ssmDataType, runtime::BufferManager const& bufferManager) { auto dataSize = tensorrt_llm::common::getDTypeSize(ssmDataType); switch (dataSize) @@ -1884,7 +1884,7 @@ void concatUnifiedPoolSsmDispatch(runtime::ITensor::SharedPtr const& pool, void concatUnifiedPoolConvDispatch(runtime::ITensor::SharedPtr const& pool, std::vector<SizeType32> const& realBlockIndices, std::vector<runtime::ITensor::SharedPtr> const& inputSplitBlocks, kv_cache::CacheState const& srcCacheState, kv_cache::CacheState const& selfCacheState, int selfIdx, size_t ssmBytes, - size_t blockSizeBytes, tensorrt_llm::DataType convDataType, runtime::BufferManager const& bufferManager) + size_t blockSizeBytes, nvinfer1::DataType convDataType, runtime::BufferManager const& bufferManager) { auto dataSize = tensorrt_llm::common::getDTypeSize(convDataType); switch (dataSize) diff --git a/cpp/tensorrt_llm/executor/cache_transmission/transferAgent.cpp b/cpp/tensorrt_llm/executor/cache_transmission/transferAgent.cpp index 147c99b37329..fd60651dc6a0 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/transferAgent.cpp +++ b/cpp/tensorrt_llm/executor/cache_transmission/transferAgent.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -19,12 +19,9 @@ #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/executor/serializeUtils.h" -#include <algorithm> #include <cuda.h> #include <dlfcn.h> -#include <numeric> #include <sstream> -#include <tuple> namespace tensorrt_llm::executor::kv_cache { @@ -155,9 +152,8 @@ MemoryDescs VmmDescSplitter::splitDescsWithRegionMap(MemoryDescs const& descs, V return MemoryDescs{descs.getType(), std::move(result)}; } -std::pair<MemoryDescs, MemoryDescs> VmmDescSplitter::splitAndCoalesceTransferDescs(MemoryDescs const& srcDescs, - MemoryDescs const& dstDescs, VramRegionMap const& localRegionMap, VramRegionMap const& remoteRegionMap, - bool enableCoalesce) +std::pair<MemoryDescs, MemoryDescs> VmmDescSplitter::splitTransferDescsWithRegionMaps(MemoryDescs const& srcDescs, + MemoryDescs const& dstDescs, VramRegionMap const& localRegionMap, VramRegionMap const& remoteRegionMap) { if (srcDescs.getType() != MemoryType::kVRAM) return {srcDescs, dstDescs}; @@ -165,143 +161,54 @@ std::pair<MemoryDescs, MemoryDescs> VmmDescSplitter::splitAndCoalesceTransferDes auto const& srcVec = srcDescs.getDescs(); auto const& dstVec = dstDescs.getDescs(); TLLM_CHECK(srcVec.size() == dstVec.size()); - if (srcVec.empty()) - return {srcDescs, dstDescs}; - - // Sort pair indices by (src deviceId, src addr) so pairs that are contiguous in memory become - // adjacent, maximizing coalescing regardless of input order. Pairs are independent transfers, - // so reordering is safe. The sort only serves coalescing; with it disabled, keep input order. - std::vector<size_t> order(srcVec.size()); - std::iota(order.begin(), order.end(), 0); - if (enableCoalesce) - { - std::sort(order.begin(), order.end(), - [&srcVec](size_t lhs, size_t rhs) - { - if (srcVec[lhs].getDeviceId() != srcVec[rhs].getDeviceId()) - { - return srcVec[lhs].getDeviceId() < srcVec[rhs].getDeviceId(); - } - return srcVec[lhs].getAddr() < srcVec[rhs].getAddr(); - }); - } - - std::vector<MemoryDesc> outSrc, outDst; - outSrc.reserve(srcVec.size()); - outDst.reserve(dstVec.size()); - // Region info of the last emitted piece. Invariant: every emitted desc lies within a single - // chunk on both sides, so a merge is legal iff the new piece is contiguous, in the same region, - // and does not start on a chunk boundary (starting on a boundary means the merge would cross it). - size_t prevSrcChunkSize = 0, prevDstChunkSize = 0; - uintptr_t prevSrcBase = 0, prevDstBase = 0; + std::vector<MemoryDesc> splitSrc, splitDst; + splitSrc.reserve(srcVec.size()); + splitDst.reserve(dstVec.size()); - auto emitPiece - = [&](uintptr_t srcAddr, uintptr_t dstAddr, size_t len, uint32_t srcDev, uint32_t dstDev, size_t srcChunkSize, - uintptr_t srcBase, size_t dstChunkSize, uintptr_t dstBase, bool regionsKnown) - { - if (enableCoalesce && regionsKnown && !outSrc.empty()) - { - auto const& lastSrc = outSrc.back(); - auto const& lastDst = outDst.back(); - bool contiguous = lastSrc.getAddr() + lastSrc.getLen() == srcAddr && lastSrc.getDeviceId() == srcDev - && lastDst.getAddr() + lastDst.getLen() == dstAddr && lastDst.getDeviceId() == dstDev; - bool sameSrcRegion = srcChunkSize == prevSrcChunkSize && srcBase == prevSrcBase; - bool sameDstRegion = dstChunkSize == prevDstChunkSize && dstBase == prevDstBase; - bool srcWithinChunk = srcChunkSize == 0 || (srcAddr - srcBase) % srcChunkSize != 0; - bool dstWithinChunk = dstChunkSize == 0 || (dstAddr - dstBase) % dstChunkSize != 0; - if (contiguous && sameSrcRegion && sameDstRegion && srcWithinChunk && dstWithinChunk) - { - outSrc.back() = MemoryDesc{lastSrc.getAddr(), lastSrc.getLen() + len, srcDev}; - outDst.back() = MemoryDesc{lastDst.getAddr(), lastDst.getLen() + len, dstDev}; - return; - } - } - outSrc.emplace_back(srcAddr, len, srcDev); - outDst.emplace_back(dstAddr, len, dstDev); - prevSrcChunkSize = srcChunkSize; - prevSrcBase = srcBase; - prevDstChunkSize = dstChunkSize; - prevDstBase = dstBase; - }; - - // One-entry region cache per side: after sorting, consecutive pairs almost always fall in the - // same region (typically one KV pool), so the O(log R) map lookup is skipped on cache hits. - struct RegionCache - { - uintptr_t base = 0; - size_t totalLen = 0; - size_t chunkSize = 0; - bool valid = false; - }; - - // Returns {chunkSize, regionBase, found}. A miss means the address is not covered by any - // region metadata (e.g. the peer did not send its region info). Two misses both look like - // {0, 0} yet may belong to two distinct regions, so pieces with a missed lookup on either - // side are never merged — a merge could silently cross a chunk or registration boundary. - auto cachedLookup = [](uintptr_t addr, VramRegionMap const& regionMap, RegionCache& cache) + for (size_t i = 0; i < srcVec.size(); ++i) { - if (cache.valid && addr >= cache.base && addr - cache.base < cache.totalLen) - { - return std::tuple<size_t, uintptr_t, bool>{cache.chunkSize, cache.base, true}; - } - auto it = regionMap.upper_bound(addr); - if (it != regionMap.begin()) + auto [srcChunkSize, srcBase] = lookupChunkInfo(srcVec[i].getAddr(), localRegionMap); + auto [dstChunkSize, dstBase] = lookupChunkInfo(dstVec[i].getAddr(), remoteRegionMap); + + // If neither side is multi-chunk VMM, no splitting is needed. + if (srcChunkSize == 0 && dstChunkSize == 0) { - --it; - if (addr >= it->first && addr - it->first < it->second.totalLen) - { - cache = {it->first, it->second.totalLen, it->second.chunkSize, true}; - return std::tuple<size_t, uintptr_t, bool>{cache.chunkSize, cache.base, true}; - } + splitSrc.push_back(srcVec[i]); + splitDst.push_back(dstVec[i]); + continue; } - return std::tuple<size_t, uintptr_t, bool>{0, 0, false}; - }; - - RegionCache srcCache, dstCache; - size_t numPieces = 0; - for (size_t idx : order) - { - auto const& src = srcVec[idx]; - auto const& dst = dstVec[idx]; - auto [srcChunkSize, srcBase, srcFound] = cachedLookup(src.getAddr(), localRegionMap, srcCache); - auto [dstChunkSize, dstBase, dstFound] = cachedLookup(dst.getAddr(), remoteRegionMap, dstCache); - uintptr_t srcAddr = src.getAddr(); - uintptr_t dstAddr = dst.getAddr(); - size_t remaining = src.getLen(); + uintptr_t srcAddr = srcVec[i].getAddr(); + uintptr_t dstAddr = dstVec[i].getAddr(); + size_t remaining = srcVec[i].getLen(); while (remaining > 0) { size_t srcPieceSize = remaining; if (srcChunkSize > 0) { - srcPieceSize = srcChunkSize - static_cast<size_t>((srcAddr - srcBase) % srcChunkSize); + size_t srcOffsetInChunk = static_cast<size_t>((srcAddr - srcBase) % srcChunkSize); + srcPieceSize = srcChunkSize - srcOffsetInChunk; } size_t dstPieceSize = remaining; if (dstChunkSize > 0) { - dstPieceSize = dstChunkSize - static_cast<size_t>((dstAddr - dstBase) % dstChunkSize); + size_t dstOffsetInChunk = static_cast<size_t>((dstAddr - dstBase) % dstChunkSize); + dstPieceSize = dstChunkSize - dstOffsetInChunk; } size_t pieceSize = std::min({remaining, srcPieceSize, dstPieceSize}); - emitPiece(srcAddr, dstAddr, pieceSize, src.getDeviceId(), dst.getDeviceId(), srcChunkSize, srcBase, - dstChunkSize, dstBase, srcFound && dstFound); + splitSrc.emplace_back(srcAddr, pieceSize, srcVec[i].getDeviceId()); + splitDst.emplace_back(dstAddr, pieceSize, dstVec[i].getDeviceId()); srcAddr += pieceSize; dstAddr += pieceSize; remaining -= pieceSize; - ++numPieces; } } - if (outSrc.size() != srcVec.size()) - { - TLLM_LOG_DEBUG("VmmDescSplitter::splitAndCoalesceTransferDescs: %zu pairs -> %zu pieces -> %zu transfers", - srcVec.size(), numPieces, outSrc.size()); - } - - return {MemoryDescs{srcDescs.getType(), std::move(outSrc)}, MemoryDescs{dstDescs.getType(), std::move(outDst)}}; + return {MemoryDescs{srcDescs.getType(), std::move(splitSrc)}, MemoryDescs{dstDescs.getType(), std::move(splitDst)}}; } MemoryDescs VmmDescSplitter::splitVmmDescs(MemoryDescs const& descs, size_t& detectedChunkSize) diff --git a/cpp/tensorrt_llm/executor/cache_transmission/ucx_utils/ucxCacheCommunicator.cpp b/cpp/tensorrt_llm/executor/cache_transmission/ucx_utils/ucxCacheCommunicator.cpp index 3cd388625640..4ad1e7bffc86 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/ucx_utils/ucxCacheCommunicator.cpp +++ b/cpp/tensorrt_llm/executor/cache_transmission/ucx_utils/ucxCacheCommunicator.cpp @@ -611,41 +611,23 @@ UcxConnection::ConnectionIdType UcxConnectionManager::getNewConnectionId(std::sh Connection const* UcxConnectionManager::recvConnect(DataContext const& ctx, void* data, size_t size) { - // Co-owned by the completion callback: the terminate path returns while the - // cancelled request may still write the buffer and fire the callback. - struct RecvState - { - std::vector<char> mBuffer; - std::promise<void> mPromise; - }; + std::vector<char> buffer(size + sizeof(UcxConnection::ConnectionIdType)); + std::promise<void> promise; + std::future<void> future = promise.get_future(); + auto completionCallback = [&](ucs_status_t, ucxx::RequestCallbackUserData) -> void { promise.set_value(); }; - auto state = std::make_shared<RecvState>(); - state->mBuffer.resize(size + sizeof(UcxConnection::ConnectionIdType)); - std::future<void> future = state->mPromise.get_future(); - auto completionCallback - = [state](ucs_status_t, ucxx::RequestCallbackUserData) -> void { state->mPromise.set_value(); }; - - std::shared_ptr<ucxx::Request> req = mWorkersPool.front()->tagRecv(state->mBuffer.data(), state->mBuffer.size(), - ucxx::Tag(ctx.getTag()), ucxx::TagMask(0xFFFFFFFF), false, completionCallback); + std::shared_ptr<ucxx::Request> req = mWorkersPool.front()->tagRecv( + buffer.data(), buffer.size(), ucxx::Tag(ctx.getTag()), ucxx::TagMask(0xFFFFFFFF), false, completionCallback); if (!req->isCompleted()) { - // Poll with timeout to allow checking the terminate flag - auto const& terminate = ctx.getTransferTerminate(); - while (future.wait_for(std::chrono::milliseconds(100)) != std::future_status::ready) - { - if (terminate.load()) - { - req->cancel(); - return nullptr; - } - } + future.get(); } TLLM_CHECK_WITH_INFO(req->isCompleted(), "recv SendConnectionId should be completed"); req->checkError(); - memcpy(data, state->mBuffer.data(), size); + memcpy(data, buffer.data(), size); UcxConnection::ConnectionIdType connectionId - = *reinterpret_cast<UcxConnection::ConnectionIdType*>(state->mBuffer.data() + size); + = *reinterpret_cast<UcxConnection::ConnectionIdType*>(buffer.data() + size); std::scoped_lock lock(mConnectionsMutex, mConnectionFuturesMutex); TLLM_CHECK_WITH_INFO(mConnectionFutures.find(connectionId) != mConnectionFutures.end(), "connectionFuture not found In recvConnect connectionId : %lu , worldRank: %d", connectionId, mRank); @@ -660,7 +642,7 @@ Connection const* UcxConnectionManager::recvConnect(DataContext const& ctx, void TLLM_CHECK(!mConnections[connectionId]->isFromRequester()); TLLM_LOG_DEBUG(mRank, "recvConnect connectionId: %lu , sendIDData:%lu", connectionId, - *reinterpret_cast<uint64_t*>(state->mBuffer.data())); + *reinterpret_cast<uint64_t*>(buffer.data())); return mConnections[connectionId].get(); } diff --git a/cpp/tensorrt_llm/executor/disaggServerUtil.cpp b/cpp/tensorrt_llm/executor/disaggServerUtil.cpp new file mode 100644 index 000000000000..6be2e4fb8ae8 --- /dev/null +++ b/cpp/tensorrt_llm/executor/disaggServerUtil.cpp @@ -0,0 +1,555 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/executor/disaggServerUtil.h" +#include "tensorrt_llm/common/utils.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include <mutex> + +namespace tensorrt_llm::executor::disagg_executor +{ + +class DisaggExecutorOrchestrator::Impl +{ +public: + Impl(std::vector<std::filesystem::path> const& ctxEnginePaths, + std::vector<std::filesystem::path> const& genEnginePaths, + std::vector<texec::ExecutorConfig> const& ctxExecutorConfigs, + std::vector<texec::ExecutorConfig> const& genExecutorConfigs, bool hasContextAwaitThreads, + bool hasGenAwaitThreads) + : mhasContextAwaitThreads(hasContextAwaitThreads) + , mhasGenAwaitThreads(hasGenAwaitThreads) + { + TLLM_CHECK(ctxEnginePaths.size() == ctxExecutorConfigs.size()); + TLLM_CHECK(genEnginePaths.size() == genExecutorConfigs.size()); + TLLM_CHECK(!(ctxEnginePaths.empty() || genEnginePaths.empty())); + int worldRank = tensorrt_llm::mpi::MpiComm::world().getRank(); + mIsOrchestrator = (worldRank == 0); + auto contextNum = ctxEnginePaths.size(); + mContextReqIdToGlobalId = std::vector<std::unordered_map<IdType, IdType>>(contextNum); + mContextMapMutexs = std::vector<std::mutex>(contextNum); + auto genNum = genEnginePaths.size(); + mGenerationReqIdToGlobalId = std::vector<std::unordered_map<IdType, IdType>>(genNum); + mGenerationMapMutexs = std::vector<std::mutex>(genNum); + + for (size_t cN = 0; cN < contextNum; cN++) + { + mContextExecutors.push_back(std::make_unique<texec::Executor>( + ctxEnginePaths[cN], texec::ModelType::kDECODER_ONLY, ctxExecutorConfigs[cN])); + } + + for (size_t gN = 0; gN < genNum; gN++) + { + mGenerationExecutors.push_back(std::make_unique<texec::Executor>( + genEnginePaths[gN], texec::ModelType::kDECODER_ONLY, genExecutorConfigs[gN])); + } + + if (mIsOrchestrator) + { + if (mhasContextAwaitThreads) + { + for (size_t contextIdx = 0; contextIdx < contextNum; contextIdx++) + { + mContextThreads.emplace_back( + [this, contextIdx]() { this->waitResponseAndAppendThreadFun(true, contextIdx); }); + } + } + if (mhasGenAwaitThreads) + { + + for (size_t genIdx = 0; genIdx < genNum; genIdx++) + { + mGenerationThreads.emplace_back( + [this, genIdx]() { this->waitResponseAndAppendThreadFun(false, genIdx); }); + } + } + } + tensorrt_llm::mpi::MpiComm::world().barrier(); + } + + std::vector<IdType> enqueueContext(std::vector<texec::Request> const& requests, + std::optional<int> selectContextId = std::nullopt, bool batch = false) + { + + std::vector<IdType> globalReqIds; + for (auto const& request : requests) + { + globalReqIds.push_back(generatedGlobalId()); + TLLM_CHECK(request.getRequestType() == tensorrt_llm::executor::RequestType::REQUEST_TYPE_CONTEXT_ONLY); + } + + if (batch) + { + size_t contextId = selectContextId.has_value() ? selectContextId.value() : selectContextExecutor(); + auto contextReqIds = mContextExecutors[contextId]->enqueueRequests(requests); + { + std::scoped_lock<std::mutex> lock{mContextMapMutexs[contextId]}; + for (size_t i = 0; i < requests.size(); ++i) + { + mContextReqIdToGlobalId[contextId][contextReqIds[i]] = globalReqIds[i]; + } + } + } + else + { + for (size_t i = 0; i < requests.size(); ++i) + { + size_t contextId = selectContextId.has_value() ? selectContextId.value() : selectContextExecutor(); + + auto contextReqId = mContextExecutors[contextId]->enqueueRequest(requests[i]); + { + std::scoped_lock<std::mutex> lock{mContextMapMutexs[contextId]}; + mContextReqIdToGlobalId[contextId][contextReqId] = globalReqIds[i]; + } + } + } + return globalReqIds; + } + + void enqueueGeneration(std::vector<texec::Request> const& requests, std::vector<IdType> const& globalRequestIds, + std::optional<int> selectGenIdx = std::nullopt, bool batch = false) + { + + TLLM_CHECK(globalRequestIds.size() == requests.size()); + + for (auto const& request : requests) + { + + TLLM_CHECK(request.getRequestType() == tensorrt_llm::executor::RequestType::REQUEST_TYPE_GENERATION_ONLY); + } + if (batch) + { + size_t genIdx = selectGenIdx.has_value() ? selectGenIdx.value() : selectGenerationExecutor(); + auto genReqIds = mGenerationExecutors[genIdx]->enqueueRequests(requests); + { + std::scoped_lock<std::mutex> lock{mGenerationMapMutexs[genIdx]}; + for (size_t i = 0; i < requests.size(); ++i) + { + mGenerationReqIdToGlobalId[genIdx][genReqIds[i]] = globalRequestIds[i]; + } + } + } + else + { + for (size_t i = 0; i < requests.size(); ++i) + { + size_t genIdx = selectGenIdx.has_value() ? selectGenIdx.value() : selectGenerationExecutor(); + + auto genReqId = mGenerationExecutors[genIdx]->enqueueRequest(requests[i]); + { + std::scoped_lock<std::mutex> lock{mGenerationMapMutexs[genIdx]}; + mGenerationReqIdToGlobalId[genIdx][genReqId] = globalRequestIds[i]; + } + } + } + } + + std::vector<ResponseWithId> awaitContextResponses( + std::optional<int> contextIdx, std::optional<std::chrono::milliseconds> const& timeout) + { + + std::vector<ResponseWithId> responses; + + if (mhasContextAwaitThreads) + { + + std::unique_lock<std::mutex> lock(mResponsesContextMtx); + auto pred = [&mShutdown = mShutdown, &resp = this->mContextResponses]() -> bool + { return !resp.empty() || mShutdown; }; + auto storeResponses = [&resp = this->mContextResponses, &responses]() + { + responses = std::move(resp); + resp.clear(); + }; + if (timeout) + { + if (mContextResponsesCV.wait_for(lock, timeout.value(), pred)) + { + storeResponses(); + } + } + else + { + mContextResponsesCV.wait(lock, pred); + storeResponses(); + } + TLLM_CHECK_WITH_INFO( + !contextIdx.has_value(), "contextIdx should not be provided when mhasContextAwaitThreads is true"); + + return responses; + } + + if (contextIdx.has_value()) + { + TLLM_CHECK(!mhasContextAwaitThreads); + auto responseFromExecutor = mContextExecutors[contextIdx.value()]->awaitResponses(timeout); + for (auto&& resp : responseFromExecutor) + { + + auto reqId = resp.getRequestId(); + IdType globalId{0}; + + { + std::scoped_lock<std::mutex> lock{mContextMapMutexs.at(contextIdx.value())}; + globalId = mContextReqIdToGlobalId.at(contextIdx.value()).at(reqId); + } + TLLM_CHECK(globalId != 0); + responses.emplace_back(std::move(resp), globalId); + } + return responses; + } + TLLM_CHECK(timeout.has_value()); + auto timeouP = timeout.value() / mContextExecutors.size(); + for (size_t ci = 0; ci < mContextExecutors.size(); ci++) + { + auto responseFromExecutor = mContextExecutors.at(ci)->awaitResponses(timeouP); + for (auto&& resp : responseFromExecutor) + { + auto reqId = resp.getRequestId(); + IdType globalId{0}; + + { + std::scoped_lock<std::mutex> lock{mContextMapMutexs.at(ci)}; + globalId = mContextReqIdToGlobalId.at(ci).at(reqId); + } + TLLM_CHECK(globalId != 0); + responses.emplace_back(std::move(resp), globalId); + } + } + + return responses; + }; + + std::vector<ResponseWithId> awaitGenerationResponses( + std::optional<int> genIdx, std::optional<std::chrono::milliseconds> const& timeout) + { + + std::vector<ResponseWithId> responses; + + if (mhasGenAwaitThreads) + { + + std::unique_lock<std::mutex> lock(mResponseGenerationMtx); + auto pred = [&mShutdown = mShutdown, &resp = this->mGenerationResponses]() -> bool + { return !resp.empty() || mShutdown; }; + auto storeResponses = [&resp = this->mGenerationResponses, &responses]() + { + responses = std::move(resp); + resp.clear(); + }; + if (timeout) + { + if (mGenerationResponsesCv.wait_for(lock, timeout.value(), pred)) + { + storeResponses(); + } + } + else + { + mGenerationResponsesCv.wait(lock, pred); + storeResponses(); + } + TLLM_CHECK_WITH_INFO(!genIdx.has_value(), "genIdx should not be provided when mhasGenAwaitThreads is true"); + return responses; + } + + if (genIdx.has_value()) + { + TLLM_CHECK(!mhasGenAwaitThreads); + auto responseFromExecutor = mGenerationExecutors[genIdx.value()]->awaitResponses(timeout); + for (auto&& resp : responseFromExecutor) + { + + auto reqId = resp.getRequestId(); + IdType globalId{0}; + + { + std::scoped_lock<std::mutex> lock{mGenerationMapMutexs.at(genIdx.value())}; + globalId = mGenerationReqIdToGlobalId.at(genIdx.value()).at(reqId); + } + TLLM_CHECK(globalId != 0); + responses.emplace_back(std::move(resp), globalId); + } + return responses; + } + TLLM_CHECK(timeout.has_value()); + auto timeouP = timeout.value() / mGenerationExecutors.size(); + + for (size_t gi = 0; gi < mGenerationExecutors.size(); gi++) + { + auto responseFromExecutor = mGenerationExecutors.at(gi)->awaitResponses(timeouP); + for (auto&& resp : responseFromExecutor) + { + + auto reqId = resp.getRequestId(); + IdType globalId{0}; + + { + std::scoped_lock<std::mutex> lock{mGenerationMapMutexs.at(gi)}; + globalId = mGenerationReqIdToGlobalId.at(gi).at(reqId); + } + TLLM_CHECK(globalId != 0); + responses.emplace_back(std::move(resp), globalId); + } + } + + return responses; + }; + + [[nodiscard]] bool canEnqueue() const + { + return mIsOrchestrator; + } + + [[nodiscard]] std::vector<std::unique_ptr<texec::Executor>> const& getContextExecutors() const + { + return mContextExecutors; + } + + [[nodiscard]] std::vector<std::unique_ptr<texec::Executor>> const& getGenExecutors() const + { + return mGenerationExecutors; + } + + ~Impl() + { + + mShutdown = true; + + mContextResponsesCV.notify_all(); + mGenerationResponsesCv.notify_all(); + for (auto&& executor : mContextExecutors) + { + executor->shutdown(); + } + for (auto&& executor : mGenerationExecutors) + { + executor->shutdown(); + } + + if (mIsOrchestrator) + { + if (mhasContextAwaitThreads) + { + for (auto&& contextThread : mContextThreads) + { + if (contextThread.joinable()) + { + contextThread.join(); + } + } + } + if (mhasGenAwaitThreads) + { + for (auto&& genThread : mGenerationThreads) + { + if (genThread.joinable()) + { + genThread.join(); + } + } + } + } + } + +private: + IdType generatedGlobalId() + { + return (++mLastId % UINT64_MAX); + }; + + size_t selectContextExecutor() + { + static size_t selectContextId = 0; + auto contextId = (selectContextId++) % mContextExecutors.size(); + if (selectContextId >= mContextExecutors.size()) + { + selectContextId = 0; + } + return contextId; + } + + size_t selectGenerationExecutor() + { + static size_t selectGenerationId = 0; + auto generationIdx = (selectGenerationId++) % mGenerationExecutors.size(); + if (selectGenerationId >= mGenerationExecutors.size()) + { + selectGenerationId = 0; + } + return generationIdx; + } + + void appendNewContextResponse(std::vector<ResponseWithId>&& newResponses) + { + { + std::scoped_lock<std::mutex> lock(mResponsesContextMtx); + for (auto&& response : newResponses) + { + mContextResponses.emplace_back(std::move(response)); + } + } + mContextResponsesCV.notify_all(); + } + + void appendNewGenerationResponse(std::vector<ResponseWithId>&& newResponses) + { + { + std::scoped_lock<std::mutex> lock(mResponseGenerationMtx); + for (auto&& response : newResponses) + { + mGenerationResponses.emplace_back(std::move(response)); + } + } + mGenerationResponsesCv.notify_all(); + } + + void waitResponseAndAppendThreadFun(bool isContext, int executorIdx) + { + + tensorrt_llm::common::setThreadName("waitResponseAndAppendThreadFun"); + + auto& executor = isContext ? mContextExecutors[executorIdx] : mGenerationExecutors[executorIdx]; + + while (!mShutdown) + { + auto responses = executor->awaitResponses(); + + if (responses.empty()) + { + continue; + } + std::vector<ResponseWithId> responseWithIds; + if (isContext) + { + for (auto&& response : responses) + { + auto reqId = response.getRequestId(); + IdType globalId{0}; + + { + std::scoped_lock<std::mutex> lock{mContextMapMutexs.at(executorIdx)}; + globalId = mContextReqIdToGlobalId.at(executorIdx).at(reqId); + } + TLLM_CHECK(globalId != 0); + responseWithIds.emplace_back(std::move(response), globalId); + } + if (responseWithIds.size() > 0) + { + appendNewContextResponse(std::move(responseWithIds)); + } + } + else + { + + for (auto&& response : responses) + { + auto reqId = response.getRequestId(); + IdType globalId{0}; + + { + std::scoped_lock<std::mutex> lock{mGenerationMapMutexs.at(executorIdx)}; + globalId = mGenerationReqIdToGlobalId.at(executorIdx).at(reqId); + } + TLLM_CHECK(globalId != 0); + responseWithIds.emplace_back(std::move(response), globalId); + } + if (responseWithIds.size() > 0) + { + appendNewGenerationResponse(std::move(responseWithIds)); + } + } + } + }; + + std::vector<std::unique_ptr<texec::Executor>> mContextExecutors; + std::vector<std::unique_ptr<texec::Executor>> mGenerationExecutors; + std::vector<std::thread> mContextThreads; + std::vector<std::thread> mGenerationThreads; + + std::atomic<IdType> mLastId{0}; + std::vector<std::unordered_map<IdType, IdType>> mContextReqIdToGlobalId; + std::vector<std::unordered_map<IdType, IdType>> mGenerationReqIdToGlobalId; + std::vector<std::mutex> mContextMapMutexs; + std::vector<std::mutex> mGenerationMapMutexs; + std::vector<ResponseWithId> mContextResponses; + std::condition_variable mContextResponsesCV; + std::mutex mResponsesContextMtx; + + std::vector<ResponseWithId> mGenerationResponses; + std::condition_variable mGenerationResponsesCv; + std::mutex mResponseGenerationMtx; + std::atomic<bool> mShutdown{false}; + std::atomic<bool> mhasContextAwaitThreads{false}; + std::atomic<bool> mhasGenAwaitThreads{false}; + bool mIsOrchestrator{false}; +}; + +DisaggExecutorOrchestrator::DisaggExecutorOrchestrator(std::vector<std::filesystem::path> const& ctxEnginePaths, + std::vector<std::filesystem::path> const& genEnginePaths, + std::vector<executor::ExecutorConfig> const& ctxExecutorConfigs, + std::vector<executor::ExecutorConfig> const& genExecutorConfigs, bool hasContextAwaitThreads, + bool hasGenAwaitThreads) + : mImpl(std::make_unique<DisaggExecutorOrchestrator::Impl>(ctxEnginePaths, genEnginePaths, ctxExecutorConfigs, + genExecutorConfigs, hasContextAwaitThreads, hasGenAwaitThreads)) +{ +} + +std::vector<IdType> DisaggExecutorOrchestrator::enqueueContext( + std::vector<texec::Request> const& requests, std::optional<int> selectContextId, bool batch) +{ + return mImpl->enqueueContext(requests, selectContextId, batch); +} + +void DisaggExecutorOrchestrator::enqueueGeneration(std::vector<texec::Request> const& requests, + std::vector<IdType> const& globalRequestIds, std::optional<int> selectGenIdx, bool batch) +{ + mImpl->enqueueGeneration(requests, globalRequestIds, selectGenIdx, batch); +} + +std::vector<ResponseWithId> DisaggExecutorOrchestrator::awaitContextResponses( + std::optional<std::chrono::milliseconds> const& timeout, std::optional<int> contextIdx) +{ + return mImpl->awaitContextResponses(contextIdx, timeout); +} + +std::vector<ResponseWithId> DisaggExecutorOrchestrator::awaitGenerationResponses( + std::optional<std::chrono::milliseconds> const& timeout, std::optional<int> genIdx) +{ + return mImpl->awaitGenerationResponses(genIdx, timeout); +} + +bool DisaggExecutorOrchestrator::canEnqueue() const +{ + return mImpl->canEnqueue(); +}; + +std::vector<std::unique_ptr<texec::Executor>> const& DisaggExecutorOrchestrator::getContextExecutors() const +{ + return mImpl->getContextExecutors(); +} + +std::vector<std::unique_ptr<texec::Executor>> const& DisaggExecutorOrchestrator::getGenExecutors() const +{ + return mImpl->getGenExecutors(); +} + +DisaggExecutorOrchestrator::~DisaggExecutorOrchestrator() = default; + +} // namespace tensorrt_llm::executor::disagg_executor diff --git a/cpp/tensorrt_llm/executor/dynamicBatchTuner.cpp b/cpp/tensorrt_llm/executor/dynamicBatchTuner.cpp new file mode 100644 index 000000000000..b7cd49f5430b --- /dev/null +++ b/cpp/tensorrt_llm/executor/dynamicBatchTuner.cpp @@ -0,0 +1,113 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/executor/dynamicBatchTuner.h" +#include "tensorrt_llm/common/logger.h" +#include <cmath> + +namespace +{ +using namespace tensorrt_llm::executor; + +void updateStats(SizeType32 value, std::deque<SizeType32>& stats, int64_t& sum, SizeType32 windowSize) +{ + while (static_cast<SizeType32>(stats.size()) >= windowSize) + { + sum -= stats.front(); + stats.pop_front(); + } + stats.push_back(value); + sum += value; +} +} // namespace + +namespace tensorrt_llm::executor +{ + +DynamicBatchTuner::DynamicBatchTuner(DynamicBatchConfig const& config) + : mEnableBatchSizeTuning(config.getEnableBatchSizeTuning()) + , mEnableMaxNumTokensTuning(config.getEnableMaxNumTokensTuning()) + , mDynamicBatchMovingAverageWindow(config.getDynamicBatchMovingAverageWindow()) + , mBatchSizeTable(config.getBatchSizeTable()) +{ + TLLM_CHECK_WITH_INFO(!mBatchSizeTable.empty(), "Batch size table is empty."); + for (size_t i = 1; i < mBatchSizeTable.size(); ++i) + { + TLLM_CHECK_WITH_INFO(mBatchSizeTable[i - 1].first < mBatchSizeTable[i].first, + "Batch size table is not sorted in ascending order."); + } +} + +void DynamicBatchTuner::updateStats(SizeType32 inputLength, SizeType32 outputLength) +{ + ::updateStats(inputLength, mInputLengthStats, mInputLengthSum, mDynamicBatchMovingAverageWindow); + ::updateStats(outputLength, mOutputLengthStats, mOutputLengthSum, mDynamicBatchMovingAverageWindow); +} + +double DynamicBatchTuner::getAverageInputLength() const +{ + return mInputLengthStats.empty() ? 0 : static_cast<double>(mInputLengthSum) / mInputLengthStats.size(); +} + +double DynamicBatchTuner::getAverageOutputLength() const +{ + return mOutputLengthStats.empty() ? 0 : static_cast<double>(mOutputLengthSum) / mOutputLengthStats.size(); +} + +SizeType32 DynamicBatchTuner::getRuntimeBatchSize(SizeType32 maxCapacityBatchSize) const +{ + for (auto const& [batchSizeLimit, batchSize] : mBatchSizeTable) + { + if (maxCapacityBatchSize < batchSizeLimit) + { + return batchSize; + } + } + SizeType32 threshold = maxCapacityBatchSize / kBatchSizeFallbackGranularity * kBatchSizeFallbackGranularity; + if (maxCapacityBatchSize < (threshold + kBatchSizeFallbackThreshold)) + { + return threshold; + } + return maxCapacityBatchSize; +} + +SizeType32 DynamicBatchTuner::getRuntimeMaxNumTokens(SizeType32 maxRuntimeBatchSize) const +{ + // calculate max num token in fully overlapped case + SizeType32 adjustedNumTokens + = 1.0 * (maxRuntimeBatchSize * getAverageInputLength() / getAverageOutputLength() + maxRuntimeBatchSize); + SizeType32 tokenThreshold; + // context heavy (avg ISL/OSL > kMaxNumTokensRatioContextHeavy) + if (getAverageInputLength() / getAverageOutputLength() > kMaxNumTokensRatioContextHeavy) + { + tokenThreshold = kMaxNumTokensThresholdContextHeavy; + } + // balanced case (kMaxNumTokensRatioBalanced < avg ISL/OSL < kMaxNumTokensRatioContextHeavy) + else if (getAverageInputLength() / getAverageOutputLength() > kMaxNumTokensRatioBalanced) + { + tokenThreshold = kMaxNumTokensThresholdBalanced; + } + // gen heavy (avg ISL/OSL < kMaxNumTokensRatioBalanced) + else + { + tokenThreshold = kMaxNumTokensThresholdGenHeavy; + } + // pad it to pow of 2 and max of this value and threshold. + return (std::max(1 << int(ceil(log2(adjustedNumTokens))), tokenThreshold)); +} + +} // namespace tensorrt_llm::executor diff --git a/cpp/tensorrt_llm/executor/dynamicBatchTuner.h b/cpp/tensorrt_llm/executor/dynamicBatchTuner.h new file mode 100644 index 000000000000..df38cd157fde --- /dev/null +++ b/cpp/tensorrt_llm/executor/dynamicBatchTuner.h @@ -0,0 +1,87 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" + +#include <deque> + +namespace tensorrt_llm::executor +{ + +/// @brief A class that maintains runtime input and output length statistics and computes runtime dynamic batch size. +class DynamicBatchTuner +{ +public: + explicit DynamicBatchTuner(DynamicBatchConfig const& config); + + /// @brief Check if dynamic batch size tuning is enabled. + [[nodiscard]] bool isBatchSizeTuningEnabled() const + { + return mEnableBatchSizeTuning; + } + + /// @brief Check if max num tokens tuning is enabled. + [[nodiscard]] bool isMaxNumTokensTuningEnabled() const + { + return mEnableMaxNumTokensTuning; + } + + /// @brief Update current stats given the input and output length from a single request. + void updateStats(SizeType32 inputLen, SizeType32 outputLen); + + /// @brief Get average input length. + [[nodiscard]] double getAverageInputLength() const; + + /// @brief Get average output length. + [[nodiscard]] double getAverageOutputLength() const; + + /// @brief Get the dynamic batch size based on the current statistics. + [[nodiscard]] SizeType32 getRuntimeBatchSize(SizeType32 maxCapacityBatchSize) const; + + /// @brief Get the dynamic max num tokens based on the current statistics. + [[nodiscard]] SizeType32 getRuntimeMaxNumTokens(SizeType32 runtimeBatchSize) const; + +private: + bool mEnableBatchSizeTuning = false; + + bool mEnableMaxNumTokensTuning = false; + + SizeType32 mDynamicBatchMovingAverageWindow = 0; + + std::vector<std::pair<SizeType32, SizeType32>> mBatchSizeTable; + + int64_t mInputLengthSum = 0; + std::deque<SizeType32> mInputLengthStats; + + int64_t mOutputLengthSum = 0; + std::deque<SizeType32> mOutputLengthStats; + + static SizeType32 const kBatchSizeFallbackGranularity = 512; + static SizeType32 const kBatchSizeFallbackThreshold = 128; + + static double constexpr kMaxNumTokensRatioContextHeavy = 2.0; + static double constexpr kMaxNumTokensRatioBalanced = 0.5; + + static SizeType32 const kMaxNumTokensThresholdContextHeavy = 8192; + static SizeType32 const kMaxNumTokensThresholdBalanced = 4096; + static SizeType32 const kMaxNumTokensThresholdGenHeavy = 2048; +}; + +} // namespace tensorrt_llm::executor diff --git a/cpp/tensorrt_llm/executor/executor.cpp b/cpp/tensorrt_llm/executor/executor.cpp new file mode 100644 index 000000000000..091bb5128230 --- /dev/null +++ b/cpp/tensorrt_llm/executor/executor.cpp @@ -0,0 +1,144 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include <tensorrt_llm/executor/executor.h> +#include <tensorrt_llm/executor/executorImpl.h> + +namespace tensorrt_llm::executor +{ + +Executor::Executor(std::filesystem::path const& modelPath, ModelType modelType, ExecutorConfig const& executorConfig) + : mImpl(std::make_unique<Executor::Impl>(modelPath, std::nullopt, modelType, executorConfig)) +{ +} + +Executor::Executor(std::filesystem::path const& encoderModelPath, std::filesystem::path const& decoderModelPath, + ModelType modelType, ExecutorConfig const& executorConfig) + : mImpl(std::make_unique<Executor::Impl>(decoderModelPath, encoderModelPath, modelType, executorConfig)) +{ +} + +Executor::Executor(BufferView const& engineBuffer, std::string const& jsonConfigStr, ModelType modelType, + ExecutorConfig const& executorConfig, std::optional<std::map<std::string, Tensor>> const& managedWeights) + : mImpl(std::make_unique<Executor::Impl>( + engineBuffer, jsonConfigStr, std::nullopt, std::nullopt, modelType, executorConfig, managedWeights)) +{ +} + +Executor::Executor(BufferView const& encoderEngineBuffer, std::string const& encoderJsonConfigStr, + BufferView const& decoderEngineBuffer, std::string const& decoderJsonConfigStr, ModelType modelType, + ExecutorConfig const& executorConfig) + : mImpl(std::make_unique<Executor::Impl>(decoderEngineBuffer, decoderJsonConfigStr, encoderEngineBuffer, + encoderJsonConfigStr, modelType, executorConfig, std::nullopt)) +{ +} + +Executor::Executor(std::shared_ptr<Model> model, ExecutorConfig const& executorConfig) + : mImpl(std::make_unique<Executor::Impl>(std::move(model), std::nullopt, executorConfig)) +{ +} + +Executor::Executor( + std::shared_ptr<Model> encoderModel, std::shared_ptr<Model> decoderModel, ExecutorConfig const& executorConfig) + : mImpl(std::make_unique<Executor::Impl>(std::move(decoderModel), std::move(encoderModel), executorConfig)) +{ +} + +Executor::~Executor() = default; + +IdType Executor::enqueueRequest(Request const& llmRequest) +{ + return mImpl->enqueueRequest(llmRequest); +} + +std::vector<IdType> Executor::enqueueRequests(std::vector<Request> const& llmRequests) +{ + return mImpl->enqueueRequests(llmRequests); +} + +std::vector<Response> Executor::awaitResponses(std::optional<std::chrono::milliseconds> const& timeout) +{ + return mImpl->awaitResponses(timeout); +} + +std::vector<Response> Executor::awaitResponses( + IdType const& requestId, std::optional<std::chrono::milliseconds> const& timeout) +{ + return mImpl->awaitResponses(requestId, timeout); +} + +std::vector<std::vector<Response>> Executor::awaitResponses( + std::vector<IdType> const& requestIds, std::optional<std::chrono::milliseconds> const& timeout) +{ + return mImpl->awaitResponses(requestIds, timeout); +} + +SizeType32 Executor::getNumResponsesReady(std::optional<IdType> const& requestId) const +{ + return mImpl->getNumResponsesReady(requestId); +} + +void Executor::cancelRequest(IdType requestId) +{ + return mImpl->cancelRequest(requestId); +} + +void Executor::shutdown() +{ + return mImpl->shutdown(); +} + +std::deque<IterationStats> Executor::getLatestIterationStats() +{ + return mImpl->getLatestIterationStats(); +} + +std::deque<RequestStatsPerIteration> Executor::getLatestRequestStats() +{ + return mImpl->getLatestRequestStats(); +} + +std::deque<DebugTensorsPerIteration> Executor::getLatestDebugTensors() +{ + return mImpl->getLatestDebugTensors(); +} + +bool Executor::canEnqueueRequests() const +{ + return mImpl->canEnqueueRequests(); +} + +bool Executor::isParticipant() const +{ + return mImpl->isParticipant(); +} + +std::optional<std::shared_ptr<KVCacheEventManager>> Executor::getKVCacheEventManager() const +{ + return mImpl->getKVCacheEventManager(); +} + +KVCacheEvent::KVCacheEvent( + size_t eventId, KVCacheEventData data, SizeType32 windowSize, std::optional<SizeType32> attentionDpRank) + : eventId{eventId} + , data{std::move(data)} + , windowSize{windowSize} + , attentionDpRank{attentionDpRank} +{ +} + +} // namespace tensorrt_llm::executor diff --git a/cpp/tensorrt_llm/executor/executorImpl.cpp b/cpp/tensorrt_llm/executor/executorImpl.cpp new file mode 100644 index 000000000000..9f7fb654a2d5 --- /dev/null +++ b/cpp/tensorrt_llm/executor/executorImpl.cpp @@ -0,0 +1,2791 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/executor/executorImpl.h" +#include "tensorrt_llm/batch_manager/trtEncoderModel.h" +#include "tensorrt_llm/batch_manager/trtGptModelFactory.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaProfilerUtils.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/common/timestampUtils.h" +#include "tensorrt_llm/common/utils.h" +#include "tensorrt_llm/executor/dataTransceiverState.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/orchestratorUtils.h" +#include "tensorrt_llm/executor/requestUtils.h" +#include "tensorrt_llm/executor/serialization.h" +#include "tensorrt_llm/executor/serializeUtils.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/executor/version.h" +#include "tensorrt_llm/runtime/loraCache.h" +#include "tensorrt_llm/runtime/memoryCounters.h" +#include "tensorrt_llm/runtime/utils/mpiTags.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include <algorithm> +#include <cstddef> +#include <cstdint> +#include <cuda_profiler_api.h> +#include <iterator> +#include <memory> +#include <optional> +#include <utility> + +namespace tensorrt_llm::executor +{ + +namespace +{ + +[[nodiscard]] bool executorConfigIsValid( + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig, runtime::ModelConfig const& modelConfig) +{ + // Make sure logic in this function matches fixExecutorConfig + if (executorConfig.getEnableChunkedContext()) + { + if (modelConfig.isRnnBased() || !modelConfig.isKVCacheEnabled() || !modelConfig.getPagedContextFMHA()) + { + return false; + } + } + return true; +} + +[[nodiscard]] ::tensorrt_llm::executor::ExecutorConfig fixExecutorConfig( + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig, runtime::ModelConfig const& modelConfig) +{ + // Make sure logic in this function matches executorConfigIsValid + auto fixedExecutorConfig = executorConfig; + // Disable chunked context when not supported + if (executorConfig.getEnableChunkedContext()) + { + if (modelConfig.isRnnBased() || !modelConfig.isKVCacheEnabled() || !modelConfig.getPagedContextFMHA()) + { + fixedExecutorConfig.setEnableChunkedContext(false); + TLLM_LOG_WARNING( + "Chunked context is not supported for this configuration and will be disabled. " + "Related configs: RNNBased: %d, KVCacheEnabled: %d, PagedContextFMHA: %d", + modelConfig.isRnnBased(), modelConfig.isKVCacheEnabled(), modelConfig.getPagedContextFMHA()); + } + } + return fixedExecutorConfig; +} + +[[nodiscard]] bool statsBufferIsEnabled(SizeType32 maxIterations) +{ + return maxIterations != 0; +} + +[[nodiscard]] bool statsBufferIsBounded(SizeType32 maxIterations) +{ + return maxIterations > 0; +} + +SizeType32 getNumChildRequests(Request const& request) +{ + auto samplingConfig = request.getSamplingConfig(); + return samplingConfig.getBeamWidth() > 1 ? 0 : samplingConfig.getNumReturnSequences().value_or(1) - 1; +} + +} // namespace + +/// @brief Version of TRT-LLM as defined in tensorrt_llm/version.py +char const* version() noexcept +{ + return kTensorRtLlmVersion; +} + +class CancelledRequestsAsyncSend +{ +public: + CancelledRequestsAsyncSend(std::shared_ptr<tensorrt_llm::mpi::MpiComm> const& commSession, + std::unordered_set<IdType> const& cancelledReqIds, int peer) + { + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + mNumReq = static_cast<int64_t>(cancelledReqIds.size()); + TLLM_LOG_DEBUG("start send %ld cancelled requests to rank %d", mNumReq, peer); + mRequest1 + = commSession->sendAsync(&mNumReq, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kCancelledRequestsNumReq); + if (mNumReq > 0) + { + mIds.assign(cancelledReqIds.begin(), cancelledReqIds.end()); + mRequest2 = commSession->sendAsync( + mIds.data(), mIds.size(), mpi::MpiType::kUINT64, peer, mpi::MpiTag::kCancelledRequestsIds); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + } + + ~CancelledRequestsAsyncSend() + { + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + mRequest1->wait(); + if (mRequest2) + { + mRequest2->wait(); + } + TLLM_LOG_DEBUG("end send cancelled requests"); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + } + + CancelledRequestsAsyncSend(CancelledRequestsAsyncSend const& executor) = delete; + CancelledRequestsAsyncSend& operator=(CancelledRequestsAsyncSend const& executor) = delete; + CancelledRequestsAsyncSend(CancelledRequestsAsyncSend&&) = delete; + CancelledRequestsAsyncSend& operator=(CancelledRequestsAsyncSend&&) = delete; + + static std::unordered_set<IdType> cancelledRequestsRecv( + std::shared_ptr<tensorrt_llm::mpi::MpiComm> const& commSession, int peer) + { + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + TLLM_LOG_DEBUG("start recv cancelled requests from rank %d", peer); + std::unordered_set<IdType> cancelledReqIds; + int64_t numReq{0}; + commSession->recv(&numReq, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kCancelledRequestsNumReq); + TLLM_LOG_DEBUG("recv %ld cancelled requests", numReq); + if (numReq > 0) + { + std::vector<IdType> buffer(numReq); + commSession->recv( + buffer.data(), buffer.size(), mpi::MpiType::kUINT64, peer, mpi::MpiTag::kCancelledRequestsIds); + cancelledReqIds = std::unordered_set<IdType>(buffer.begin(), buffer.end()); + } + TLLM_LOG_DEBUG("end recv cancelled requests from rank %d", peer); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return cancelledReqIds; + } + +private: + int64_t mNumReq; + std::vector<IdType> mIds; + std::shared_ptr<tensorrt_llm::mpi::MpiRequest> mRequest1; + std::shared_ptr<tensorrt_llm::mpi::MpiRequest> mRequest2; +}; + +class RequestWithIdAsyncSend +{ +public: + RequestWithIdAsyncSend(std::shared_ptr<tensorrt_llm::mpi::MpiComm> const& commSession, + std::vector<RequestWithId> const& reqWithIds, int peer) + { + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + TLLM_LOG_DEBUG("start send requests to rank %d", peer); + mNumReq = static_cast<int64_t>(reqWithIds.size()); + mRequest1 = commSession->sendAsync(&mNumReq, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kRequestWithIdNumReq); + if (mNumReq > 0) + { + mPacked = RequestWithId::serializeReqWithIds(reqWithIds); + mVecSize = static_cast<int64_t>(mPacked.size()); + mRequest2 + = commSession->sendAsync(&mVecSize, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kRequestWithIdVecSize); + mRequest3 = commSession->sendAsync( + mPacked.data(), mPacked.size(), mpi::MpiType::kCHAR, peer, mpi::MpiTag::kRequestWithIdPacked); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + } + + ~RequestWithIdAsyncSend() + { + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + mRequest1->wait(); + if (mRequest2) + { + mRequest2->wait(); + } + if (mRequest3) + { + mRequest3->wait(); + } + TLLM_LOG_DEBUG("end send requests"); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + } + + RequestWithIdAsyncSend(RequestWithIdAsyncSend const& executor) = delete; + RequestWithIdAsyncSend& operator=(RequestWithIdAsyncSend const& executor) = delete; + RequestWithIdAsyncSend(RequestWithIdAsyncSend&&) = delete; + RequestWithIdAsyncSend& operator=(RequestWithIdAsyncSend&&) = delete; + + static std::vector<RequestWithId> requestWithIdRecv( + std::shared_ptr<tensorrt_llm::mpi::MpiComm> const& commSession, int peer) + { + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + TLLM_LOG_DEBUG("start recv requests from rank %d", peer); + std::vector<RequestWithId> reqWithIds; + int64_t numReq{0}; + commSession->recv(&numReq, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kRequestWithIdNumReq); + if (numReq > 0) + { + std::vector<char> buffer; + int64_t vecSize = 0; + commSession->recv(&vecSize, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kRequestWithIdVecSize); + buffer.resize(vecSize); + commSession->recv( + buffer.data(), buffer.size(), mpi::MpiType::kCHAR, peer, mpi::MpiTag::kRequestWithIdPacked); + reqWithIds = RequestWithId::deserializeReqWithIds(buffer); + } + TLLM_LOG_DEBUG("end recv requests from rank %d", peer); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return reqWithIds; + } + +private: + int64_t mNumReq; + int64_t mVecSize; + std::vector<char> mPacked; + std::shared_ptr<tensorrt_llm::mpi::MpiRequest> mRequest1; + std::shared_ptr<tensorrt_llm::mpi::MpiRequest> mRequest2; + std::shared_ptr<tensorrt_llm::mpi::MpiRequest> mRequest3; +}; + +void Executor::Impl::loadModel(std::optional<std::filesystem::path> const& modelPathOpt, + std::optional<BufferView> const& engineBufferOpt, runtime::GptJsonConfig const& jsonConfig, + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig, bool isEncoder, + std::optional<std::map<std::string, Tensor>> const& managedWeightsOpt) +{ + auto const gpusPerNode = jsonConfig.getGpusPerNode(); + auto const tp = jsonConfig.getTensorParallelism(); + auto const pp = jsonConfig.getPipelineParallelism(); + auto const cp = jsonConfig.getContextParallelism(); + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + auto worldConfig = runtime::WorldConfig::mpi(gpusPerNode, tp, pp, cp, parallelConfig.getDeviceIds()); + + TLLM_CHECK_WITH_INFO(modelPathOpt.has_value() || engineBufferOpt.has_value(), + "Either engine path or deserialized engine buffer should be given to load the model properly."); + auto rawEngine = engineBufferOpt.has_value() + ? runtime::RawEngine(engineBufferOpt.value().data(), engineBufferOpt.value().size()) + : runtime::RawEngine(modelPathOpt.value() / jsonConfig.engineFilename(worldConfig)); + + if (rawEngine.getType() != tensorrt_llm::runtime::RawEngine::FilePath) + { + if (modelPathOpt.has_value()) + { + rawEngine.setPath(modelPathOpt.value() / jsonConfig.engineFilename(worldConfig)); + if (managedWeightsOpt.has_value()) + { + TLLM_LOG_WARNING( + "Executor::Impl::loadModel: managedWeightsOpt argument is ignored when loading engine from file."); + } + } + else if (managedWeightsOpt.has_value()) + { + rawEngine.setManagedWeightsMap(managedWeightsOpt.value()); + } + } + + auto const& modelConfig = jsonConfig.getModelConfig(); + + if (isEncoder) + { + mEncoderModel = createEncoderModel(rawEngine, modelConfig, worldConfig, executorConfig); + } + else + { + mModel = createModel(rawEngine, modelConfig, worldConfig, executorConfig); + } +}; + +Executor::Impl::Impl(std::filesystem::path const& modelPath, + std::optional<std::filesystem::path> const& encoderModelPath, ModelType const modelType, + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig) +{ + auto decoderJsonConfig = runtime::GptJsonConfig::parse(modelPath / "config.json"); + + // for now, assume encoder & decoder models share the same MPI config + auto const tp = decoderJsonConfig.getTensorParallelism(); + auto const pp = decoderJsonConfig.getPipelineParallelism(); + auto const cp = decoderJsonConfig.getContextParallelism(); + initializeCommAndWorkers(tp, pp, cp, executorConfig, modelType, modelPath, std::nullopt, decoderJsonConfig); + + if (mIsWorker) + { + if (modelType == ModelType::kENCODER_DECODER) + { + if (encoderModelPath.has_value()) + { + auto const encoderJsonConfig = runtime::GptJsonConfig::parse(encoderModelPath.value() / "config.json"); + + auto const encoderMaxInputLen = encoderJsonConfig.getModelConfig().getMaxInputLen(); + auto const encoderHiddenSize = encoderJsonConfig.getModelConfig().getHiddenSize() + * encoderJsonConfig.getTensorParallelism(); // recover full hidden size + // add encoder info to decoder for encoder-decoder models + // note: GptJsonConfig can no longer have modelConfig as const member since it must be mutable here + decoderJsonConfig.getModelConfigMutable().setMaxEncoderLen(encoderMaxInputLen); + decoderJsonConfig.getModelConfigMutable().setEncoderHiddenSize(encoderHiddenSize); + + loadModel( + encoderModelPath.value(), std::nullopt, encoderJsonConfig, executorConfig, true, std::nullopt); + } + else + { + TLLM_LOG_WARNING("Encoder model path not provided. Skipping Encoder Run."); + } + } + loadModel(modelPath, std::nullopt, decoderJsonConfig, executorConfig, false, std::nullopt); + } + initialize(executorConfig); +} + +Executor::Impl::Impl(BufferView const& engineBufferView, std::string const& jsonConfigStr, + std::optional<BufferView> const& encoderEngineBufferView, std::optional<std::string> const& encoderJsonConfigStr, + ModelType const modelType, ::tensorrt_llm::executor::ExecutorConfig const& executorConfig, + std::optional<std::map<std::string, Tensor>> const& managedWeightsOpt) +{ + auto decoderJsonConfig = runtime::GptJsonConfig::parse(jsonConfigStr); + + // for now, assume encoder & decoder models share the same MPI config + auto const tp = decoderJsonConfig.getTensorParallelism(); + auto const pp = decoderJsonConfig.getPipelineParallelism(); + auto const cp = decoderJsonConfig.getContextParallelism(); + initializeCommAndWorkers(tp, pp, cp, executorConfig, modelType, std::nullopt, std::nullopt, decoderJsonConfig); + + if (mIsWorker) + { + if (modelType == ModelType::kENCODER_DECODER) + { + TLLM_CHECK(encoderEngineBufferView.has_value() && encoderJsonConfigStr.has_value()); + TLLM_CHECK_WITH_INFO( + !managedWeightsOpt.has_value(), "Managed weights are not supported for enc-dec models"); + + auto const encoderJsonConfig = runtime::GptJsonConfig::parse(encoderJsonConfigStr.value()); + + auto const encoderMaxInputLen = encoderJsonConfig.getModelConfig().getMaxInputLen(); + auto const encoderHiddenSize = encoderJsonConfig.getModelConfig().getHiddenSize() + * encoderJsonConfig.getTensorParallelism(); // recover full hidden size + // add encoder info to decoder for encoder-decoder models + // note: GptJsonConfig can no longer have modelConfig as const member since it must be mutable here + decoderJsonConfig.getModelConfigMutable().setMaxEncoderLen(encoderMaxInputLen); + decoderJsonConfig.getModelConfigMutable().setEncoderHiddenSize(encoderHiddenSize); + + loadModel( + std::nullopt, encoderEngineBufferView.value(), encoderJsonConfig, executorConfig, true, std::nullopt); + } + loadModel(std::nullopt, engineBufferView, decoderJsonConfig, executorConfig, false, managedWeightsOpt); + } + initialize(executorConfig); +} + +Executor::Impl::Impl(std::shared_ptr<Model> model, std::optional<std::shared_ptr<Model>> encoderModel, + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig) +{ + auto const& worldConfig = model->getWorldConfig(); + auto const tp = worldConfig.getTensorParallelism(); + auto const pp = worldConfig.getPipelineParallelism(); + auto const cp = worldConfig.getContextParallelism(); + auto const modelType = encoderModel.has_value() ? ModelType::kENCODER_DECODER : ModelType::kDECODER_ONLY; + initializeCommAndWorkers(tp, pp, cp, executorConfig, modelType, std::nullopt, worldConfig); + if (modelType == ModelType::kENCODER_DECODER) + { + mEncoderModel = encoderModel.value(); + } + mModel = std::move(model); + initialize(executorConfig); +} + +Executor::Impl::~Impl() +{ + shutdown(); +} + +void Executor::Impl::initialize(::tensorrt_llm::executor::ExecutorConfig const& executorConfig) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + mShutdown = false; + mShutdownCalled = false; + mIterStatsMaxIterations = executorConfig.getIterStatsMaxIterations(); + mRequestStatsMaxIterations = executorConfig.getRequestStatsMaxIterations(); + mDebugTensorsMaxIterations + = executorConfig.getDebugConfig() ? executorConfig.getDebugConfig()->getDebugTensorsMaxIterations() : 0; + TLLM_CHECK_WITH_INFO(mDebugTensorsMaxIterations == 0 || mCommMode == CommunicationMode::kLEADER, + "debugTensorsMaxIterations > 0 is only allowed in leader mode."); + mBatchingType = executorConfig.getBatchingType(); + mIsSchedulerMaxUtilization = (executorConfig.getSchedulerConfig().getCapacitySchedulerPolicy() + == CapacitySchedulerPolicy::kMAX_UTILIZATION); + mIsSchedulerGuaranteedNoEvict = (executorConfig.getSchedulerConfig().getCapacitySchedulerPolicy() + == CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT); + mIsChunkedContext = executorConfig.getEnableChunkedContext(); + mPromptTableOffloading = executorConfig.getPromptTableOffloading(); + mMaxQueueSize = executorConfig.getMaxQueueSize(); + + mLastReqId = 1; + + auto const& logitsProcConfig = executorConfig.getLogitsPostProcessorConfig(); + if (logitsProcConfig.has_value()) + { + mLogitsPostProcessorMap = logitsProcConfig.value().getProcessorMap().value_or(LogitsPostProcessorMap{}); + initializeLogitsPostProcessorBatched(logitsProcConfig.value()); + if (!logitsProcConfig.value().getReplicate()) + { + mModel->setReplicateLogitsPostProcessor(false); + } + } + + auto const& commComm = COMM_SESSION; + int32_t const commSize = commComm.getSize(); + if (mIsWorker) + { + if (commSize > 1) + { + auto const& worldConfig = mModel->getWorldConfig(); + auto const& commSession = COMM_SESSION; + auto const& rank = commSession.getRank(); + auto const& tp = worldConfig.getTensorParallelism(); + auto const& cp = worldConfig.getContextParallelism(); + + mCommTensorParallel = std::make_shared<tensorrt_llm::mpi::MpiComm>( + commSession.split(rank / tp, worldConfig.getTensorParallelRank())); + mCommContextParallel = std::make_shared<tensorrt_llm::mpi::MpiComm>( + commSession.split(rank / (tp * cp) * tp + rank % tp, worldConfig.getContextParallelRank())); + mCommPipelineParallel = std::make_shared<tensorrt_llm::mpi::MpiComm>( + commSession.split(rank % (tp * cp), worldConfig.getPipelineParallelRank())); + + if (worldConfig.isPipelineParallel()) + { + mRequestWithIdWaitThread = std::make_unique<tensorrt_llm::mpi::MpiWaitThread>( + "requestWithIdWaitThread", [this]() { mRequestWithIdAsyncSndHdl.reset(nullptr); }); + mCancelledRequestsWaitThread = std::make_unique<tensorrt_llm::mpi::MpiWaitThread>( + "cancelledRequestsWaitThread", [this]() { mCancelledRequestsAsyncSndHdl.reset(nullptr); }); + if (mIsLeader) + { + mRequestWithIdLeaderThread + = std::make_unique<std::thread>(&Executor::Impl::requestWithIdLeaderThread, this); + mCancelledRequestsLeaderThread + = std::make_unique<std::thread>(&Executor::Impl::cancelledRequestsLeaderThread, this); + } + } + } + // Launch the execution thread + mMaxNumActiveRequests = mModel->getMaxNumSequences(); + mExecutionThread = std::thread(&Impl::executionLoop, this); + } + + mEnableBlockReuse = executorConfig.getKvCacheConfig().getEnableBlockReuse(); + + auto const& dynamicBatchConfig = executorConfig.getSchedulerConfig().getDynamicBatchConfig(); + if (dynamicBatchConfig) + { + if (mIsWorker) + { + if (mModel->getModelConfig().isTransformerBased() && mModel->getModelConfig().isKVCacheEnabled()) + { + mDynamicBatchTuner = std::make_shared<DynamicBatchTuner>(dynamicBatchConfig.value()); + } + else + { + TLLM_LOG_WARNING("Dynamic batch tuner can only support transformer models that use KV cache."); + } + } + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +std::shared_ptr<Model> Executor::Impl::createModel(runtime::RawEngine const& rawEngine, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig) +{ + auto const gptModelType = [&executorConfig, &modelConfig]() + { + switch (executorConfig.getBatchingType()) + { + case BatchingType::kSTATIC: + TLLM_THROW( + "Static batching type is deprecated. Please use in-flight batching with " + "CapacitySchedulerPolicy::kSTATIC_BATCH instead."); + case BatchingType::kINFLIGHT: + return modelConfig.isRnnBased() ? batch_manager::TrtGptModelType::InflightBatching + : batch_manager::TrtGptModelType::InflightFusedBatching; + default: TLLM_THROW("Invalid batching strategy"); + } + }(); + + bool const isLeaderInOrchMode = (mCommMode == CommunicationMode::kORCHESTRATOR) && mIsLeader; + auto const& fixedExecutorConfig = executorConfigIsValid(executorConfig, modelConfig) + ? executorConfig + : fixExecutorConfig(executorConfig, modelConfig); + + return batch_manager::TrtGptModelFactory::create( + rawEngine, modelConfig, worldConfig, gptModelType, fixedExecutorConfig, isLeaderInOrchMode); +} + +std::shared_ptr<Model> Executor::Impl::createEncoderModel(runtime::RawEngine const& rawEngine, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig) +{ + auto fixedExecutorConfig = ExecutorConfig{}; + fixedExecutorConfig.setSchedulerConfig(executorConfig.getSchedulerConfig()); + return std::make_shared<batch_manager::TrtEncoderModel>( + modelConfig, worldConfig, rawEngine, std::make_shared<runtime::TllmLogger>(), fixedExecutorConfig); +} + +void Executor::Impl::setOrchLeaderComm( + SizeType32 tp, SizeType32 pp, SizeType32 cp, ParallelConfig const& parallelConfig) +{ +#if ENABLE_MULTI_DEVICE + auto optOrchestratorConfig = parallelConfig.getOrchestratorConfig(); + if (optOrchestratorConfig.value().getIsOrchestrator()) + { + TLLM_CHECK_WITH_INFO(mWorldRank == 0, "Rank 0 must be orchestrator"); + } + + TLLM_CHECK_WITH_INFO(parallelConfig.getParticipantIds(), + "When not spawning processes in orchestrator mode, participant IDs must be provided"); + auto participantIds = parallelConfig.getParticipantIds().value(); + + TLLM_CHECK_WITH_INFO(static_cast<SizeType32>(participantIds.size()) == tp * pp * cp, + "When specifying participantIds, participantIds size must be equal to tp*pp*cp"); + + bool isLeader = (mWorldRank == participantIds.front()); + bool isOrchestrator = (mWorldRank == 0); + + // OrchLeaderComm rank 0 is orchestrator, rank 1 is leader + mOrchRank = 0; + mLeaderRank = 1; + + // Create a leaderOrch comm + std::vector<int32_t> leaderOrchRanks{0, participantIds.front()}; + + MPI_Group worldGroup = nullptr; + MPICHECK(MPI_Comm_group(MPI_COMM_WORLD, &worldGroup)); // NOLINT + int worldGroupRank = 0; + MPI_Group_rank(worldGroup, &worldGroupRank); + + int worldSize = 0; + MPICHECK(MPI_Group_size(worldGroup, &worldSize)); // NOLINT + TLLM_CHECK_WITH_INFO(participantIds.front() < worldSize, "Not enough ranks in world"); + + MPI_Group leaderOrchCommGroup = nullptr; + MPICHECK( + MPI_Group_incl(worldGroup, leaderOrchRanks.size(), leaderOrchRanks.data(), &leaderOrchCommGroup)); // NOLINT + int leaderOrchGroupRank = 0; + int leaderOrchGroupSize = 0; + MPI_Group_rank(leaderOrchCommGroup, &leaderOrchGroupRank); + MPI_Group_size(leaderOrchCommGroup, &leaderOrchGroupSize); + + if (isOrchestrator || isLeader) + { + MPI_Comm leaderOrchComm = nullptr; + MPICHECK(MPI_Comm_create_group( + MPI_COMM_WORLD, leaderOrchCommGroup, participantIds.front(), &leaderOrchComm)); // NOLINT + mOrchLeaderComm = std::make_shared<tensorrt_llm::mpi::MpiComm>(leaderOrchComm, false); + } + else + { + mOrchLeaderComm = nullptr; + } +#endif // ENABLE_MULTI_DEVICE +} + +void Executor::Impl::initializeCommAndWorkers(SizeType32 tp, SizeType32 pp, SizeType32 cp, + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig, std::optional<ModelType> modelType, + std::optional<std::filesystem::path> const& modelPath, std::optional<runtime::WorldConfig> const& worldConfig, + std::optional<runtime::GptJsonConfig> const& decoderGptJsonConfig) +{ + if (modelType.has_value() && modelType.value() == ModelType::kENCODER_DECODER) + { + TLLM_CHECK_WITH_INFO(pp == 1, + "Encoder-Decoder C++ runtime doesn't support Pipeline Parallelism currently. Please switch to Python " + "runtime for PP mode, if necessary."); + } + + tensorrt_llm::mpi::initialize(tensorrt_llm::mpi::MpiThreadSupport::THREAD_MULTIPLE); + mWorldRank = tensorrt_llm::mpi::MpiComm::world().getRank(); + mUsePipelineParallel = pp > 1; + + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + validateParallelConfig(parallelConfig, modelType, modelPath); + + mCommMode = parallelConfig.getCommunicationMode(); + auto optOrchestratorConfig = parallelConfig.getOrchestratorConfig(); + + mRecvPollPeriodMs = executorConfig.getRecvPollPeriodMs(); + + // Need to create communicator between orchestrator and leader if not spawning processes in orchestrator mode + if (mCommMode == CommunicationMode::kORCHESTRATOR && !optOrchestratorConfig.value().getSpawnProcesses()) + { + setOrchLeaderComm(tp, pp, cp, parallelConfig); + } + + if (mCommMode == CommunicationMode::kORCHESTRATOR && optOrchestratorConfig.value().getIsOrchestrator()) + { + initializeOrchestrator(tp, pp, cp, executorConfig, parallelConfig, modelType.value(), modelPath.value()); + } + else + { + initializeWorkers(tp, pp, cp, parallelConfig, worldConfig, decoderGptJsonConfig); + } +} + +void Executor::Impl::validateParallelConfig(ParallelConfig const& parallelConfig, std::optional<ModelType> modelType, + std::optional<std::filesystem::path> const& modelPath) +{ + TLLM_CHECK_WITH_INFO(parallelConfig.getCommunicationType() == CommunicationType::kMPI, + "Only CommunicationType kMPI is supported for now."); + + auto optOrchestratorConfig = parallelConfig.getOrchestratorConfig(); + + if (parallelConfig.getCommunicationMode() == CommunicationMode::kORCHESTRATOR) + { + TLLM_CHECK_WITH_INFO( + optOrchestratorConfig, "OrchestratorConfig must be set when using ORCHESTRATOR communication mode."); + + TLLM_CHECK_WITH_INFO(modelPath, "OrchestratorMode only supports reading model weight from disk currently."); + + TLLM_CHECK_WITH_INFO(modelType, "OrchestratorMode requires modelType to be specified."); + } +} + +void Executor::Impl::initializeOrchestrator(SizeType32 tp, SizeType32 pp, SizeType32 cp, + ::tensorrt_llm::executor::ExecutorConfig const& executorConfig, ParallelConfig parallelConfig, ModelType modelType, + std::filesystem::path const& modelPath) +{ +#if ENABLE_MULTI_DEVICE + namespace su = tensorrt_llm::executor::serialize_utils; + + auto const& worldComm = tensorrt_llm::mpi::MpiComm::world(); + int32_t const worldSize = worldComm.getSize(); + + auto orchestratorConfig = parallelConfig.getOrchestratorConfig().value(); + + mIsWorker = false; + mIsLeader = false; + mIsPipelineLeader = false; + mIsOrchestrator = true; + + // Verify that worldSize is 1 + if (orchestratorConfig.getSpawnProcesses()) + { + TLLM_CHECK_WITH_INFO(worldSize == 1, + "When using the orchestrator mode and isOrchestrator is true, expect MPI worldSize to be 1."); + + // Spawn the worker threads + auto workerExecPath = orchestratorConfig.getWorkerExecutablePath(); + MPI_Comm intercomm = nullptr; + MPI_Info mpiInfo = nullptr; + MPICHECK(MPI_Info_create(&mpiInfo)); + MPICHECK(MPI_Info_set(mpiInfo, "env", "FORCE_NCCL_ALL_REDUCE_STRATEGY")); + + // Binding policy is not inherited for dynamically spawned jobs, resulting in the worker being bound + // to a single core. Override the setting to avoid perf issue - see https://nvbugs/4574329 + MPICHECK(MPI_Info_set(mpiInfo, "bind_to", "none")); + + MPICHECK(MPI_Comm_spawn(workerExecPath.c_str(), MPI_ARGV_NULL, tp * pp * cp, mpiInfo, 0, MPI_COMM_SELF, + &intercomm, MPI_ERRCODES_IGNORE)); + + mOrchLeaderComm = std::make_shared<tensorrt_llm::mpi::MpiComm>(intercomm, true); + // With intercomm, leader is rank 0 in the local group + mLeaderRank = 0; + mOrchRank = 0; + + // Copy the executor config, but set the orchestrator flag to false + auto newOrchConfig = OrchestratorConfig(false, orchestratorConfig.getWorkerExecutablePath()); + parallelConfig.setOrchestratorConfig(newOrchConfig); + auto execConfig = executorConfig; + execConfig.setParallelConfig(parallelConfig); + + // Serialize and send the executorConfig, the modelType and the modelPath + std::ostringstream oStream; + su::serialize(modelPath.string(), oStream); + su::serialize(modelType, oStream); + su::serialize(execConfig, oStream); + + auto str = oStream.str(); + std::vector<char> buffer(str.begin(), str.end()); + auto bufferSize = static_cast<int64_t>(buffer.size()); + mOrchLeaderComm->bcast(&bufferSize, 1, mpi::MpiType::kINT64, MPI_ROOT); + mOrchLeaderComm->bcast(buffer.data(), buffer.size(), mpi::MpiType::kCHAR, MPI_ROOT); + + // Wait for workers to have created their executor instance + MPICHECK(MPI_Barrier(intercomm)); + } + + // Spawn the thread responsible for sending new requests to the leader of the model + mOrchSendReqThread = std::thread(&Impl::orchSendReqThread, this); + + // Spawn the thread responsible for receiving new responses from the leader of the model + mOrchRecvThread + = std::thread([&]() { this->orchRecvThread(mpi::MpiTag::kOrchestratorId, mpi::MpiTag::kOrchestratorData); }); + +#endif // ENABLE_MULTI_DEVICE +} + +void Executor::Impl::initializeWorkers(SizeType32 tp, SizeType32 pp, SizeType32 cp, ParallelConfig& parallelConfig, + std::optional<runtime::WorldConfig> const& worldConfig, + std::optional<runtime::GptJsonConfig> const& decoderGptJsonConfig) +{ + auto const& worldComm = tensorrt_llm::mpi::MpiComm::world(); + int32_t const worldSize = worldComm.getSize(); + + auto const& orchestratorConfig = parallelConfig.getOrchestratorConfig(); + mIsOrchestrator = mCommMode == CommunicationMode::kORCHESTRATOR && orchestratorConfig.value().getIsOrchestrator(); + + TLLM_CHECK_WITH_INFO(mCommMode != CommunicationMode::kORCHESTRATOR || orchestratorConfig.has_value(), + "When using ORCHESTRATOR mode, orchestrator config must be set"); + + if (mCommMode == CommunicationMode::kORCHESTRATOR && !orchestratorConfig.value().getSpawnProcesses()) + { + TLLM_CHECK_WITH_INFO(parallelConfig.getParticipantIds(), + "When not spawning processes in orchestrator mode, participant IDs must be provided"); + + // Check that rank 0 is reserved for the orchestrator + auto const participantIds = parallelConfig.getParticipantIds().value(); + for (auto const& participantId : participantIds) + { + TLLM_CHECK_WITH_INFO(participantId != 0, "Rank 0 is reserved for the orchestrator"); + } + } + + // Participant ids + std::vector<SizeType32> participantIds; + if (!parallelConfig.getParticipantIds()) + { + TLLM_CHECK_WITH_INFO(worldSize == tp * pp * cp, + "With communicationMode kLEADER, MPI worldSize is expected to be equal to tp*pp*cp when " + "participantIds are not specified"); + + participantIds.resize(tp * pp * cp); + std::iota(participantIds.begin(), participantIds.end(), 0); + } + else + { + if (mCommMode == CommunicationMode::kORCHESTRATOR && orchestratorConfig.value().getSpawnProcesses()) + { + TLLM_THROW( + "Participant ids should not be set when using CommunicationMode::kORCHESTRATOR with " + "spawnProcesses=true"); + } + participantIds = parallelConfig.getParticipantIds().value(); + TLLM_CHECK_WITH_INFO(static_cast<SizeType32>(participantIds.size()) == tp * pp * cp, + tensorrt_llm::common::fmtstr("When specifying participantIds, participantIds size (%lu) must be equal to " + "tp*pp*cp (tp is %u, pp is %u, cp is %u)", + participantIds.size(), tp, pp, cp)); + } + + // If deviceIds are specified, check that they match tp*pp*cp + if (parallelConfig.getDeviceIds()) + { + auto deviceIds = parallelConfig.getDeviceIds().value(); + auto const hasNumNodes = parallelConfig.getNumNodes().has_value(); + if (hasNumNodes || static_cast<SizeType32>(deviceIds.size()) != tp * pp * cp) + { + auto const numNodes = hasNumNodes ? parallelConfig.getNumNodes().value() : tensorrt_llm::mpi::getNumNodes(); + TLLM_CHECK_WITH_INFO(static_cast<SizeType32>(deviceIds.size() * numNodes) == tp * pp * cp, + tensorrt_llm::common::fmtstr("When specifying deviceIds, deviceIds (%lu) * numNodes (%u) must be equal " + "to tp*pp*cp (tp is %u, pp is %u, cp is %u)", + deviceIds.size(), numNodes, tp, pp, cp)); + } + } + + // Bool that indicates if current process is worker for this model or not + auto participantIt = std::find(participantIds.begin(), participantIds.end(), mWorldRank); + mIsWorker = participantIt != participantIds.end(); + // Bool that indicates if current ranks is leader for this model + mIsLeader = (mWorldRank == participantIds.front()); + mIsPipelineLeader = (mWorldRank == participantIds[tp * (pp - 1)]); + +#if ENABLE_MULTI_DEVICE + if (mIsWorker) + { + // Create a session, but only assign to COMM_SESSION for ranks participating in this model + MPI_Group worldGroup = MPI_GROUP_NULL; + MPICHECK(MPI_Comm_group(MPI_COMM_WORLD, &worldGroup)); // NOLINT + MPI_Group sessionGroup = MPI_GROUP_NULL; + if (pp > 1) + { + // reverse participantIds to move leader to last pp rank. retain order in each tp group + std::reverse(participantIds.begin(), participantIds.end()); + if (tp > 1) + { + for (SizeType32 ppRank = 0; ppRank < pp; ppRank++) + { + std::reverse(participantIds.begin() + ppRank * tp, participantIds.begin() + (ppRank + 1) * tp); + } + } + } + MPICHECK(MPI_Group_incl(worldGroup, participantIds.size(), participantIds.data(), &sessionGroup)); // NOLINT + MPI_Comm sessionComm = MPI_COMM_NULL; + MPICHECK( + MPI_Comm_create_group(MPI_COMM_WORLD, sessionGroup, 1000 + participantIds.front(), &sessionComm)); // NOLINT + + tensorrt_llm::mpi::MpiComm::setSession(tensorrt_llm::mpi::MpiComm(sessionComm, false)); + } + + if (mIsLeader && mCommMode == CommunicationMode::kORCHESTRATOR) + { + auto optOrchestratorConfig = parallelConfig.getOrchestratorConfig(); + if (orchestratorConfig.has_value() && orchestratorConfig.value().getSpawnProcesses()) + { + mOrchLeaderComm = optOrchestratorConfig.value().getOrchLeaderComm(); + } + else + { + // mOrchLeaderComm has already been created + } + TLLM_CHECK(mOrchLeaderComm.get() != nullptr); + + TLLM_CHECK(worldConfig.has_value() || decoderGptJsonConfig.has_value()); + if (worldConfig.has_value()) + { + mDeviceId = worldConfig->getDevice(); + } + else + { + auto gpusPerNode = decoderGptJsonConfig->getGpusPerNode(); + auto worldConfig = runtime::WorldConfig::mpi(gpusPerNode, tp, pp, cp, parallelConfig.getDeviceIds()); + mDeviceId = worldConfig.getDevice(); + } + // Spawn the thread responsible for receiving new requests from the orchestrator + mLeaderRecvReqThread = std::thread(&Impl::leaderRecvReqThread, this); + + // Spawn the thread responsible for sending new responses to the orchestrator + mLeaderSendThread = std::thread([&]() + { this->leaderSendThread(mSendQueue, mpi::MpiTag::kOrchestratorId, mpi::MpiTag::kOrchestratorData); }); + } +#endif // ENABLE_MULTI_DEVICE +} + +void Executor::Impl::initializeLogitsPostProcessorBatched(LogitsPostProcessorConfig const& logitsProcConfig) +{ + if (logitsProcConfig.getProcessorBatched().has_value()) + { + mLogitsPostProcessorBatched + = [cb = logitsProcConfig.getProcessorBatched().value()]( + std::vector<batch_manager::LlmRequest::RequestIdType> const& reqIdsVec, + std::vector<batch_manager::LlmRequest::TensorPtr>& logitsVec, + std::vector<std::reference_wrapper<batch_manager::LlmRequest::BeamTokens const>> const& beamTokensVec, + CudaStreamPtr const& cudaStreamPtr, + std::vector<std::optional<batch_manager::LlmRequest::RequestIdType>> const& clientIdsVec) + { + std::vector<Tensor> cbLogitsVec; + cbLogitsVec.reserve(logitsVec.size()); + for (auto& logits : logitsVec) + { + cbLogitsVec.emplace_back(executor::detail::ofITensor(logits)); + } + + cb(reqIdsVec, cbLogitsVec, beamTokensVec, cudaStreamPtr, clientIdsVec); + }; + + mModel->setLogitsPostProcessorBatched(mLogitsPostProcessorBatched); + } +} + +IdType Executor::Impl::enqueueRequest(Request const& request) +{ + return enqueueRequests({&request, 1}).at(0); +} + +std::vector<IdType> Executor::Impl::enqueueRequests(std::vector<Request> const& requests) +{ + return enqueueRequests({requests.data(), requests.size()}); +} + +std::vector<IdType> Executor::Impl::enqueueRequests(common::ArrayView<Request const> const& requests) +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called, cannot enqueue requests"); + checkParallelApiUsage(__func__); + + TLLM_LOG_DEBUG("Enqueuing %lu requests", requests.size()); + std::vector<RequestWithId> requestWithIds; + requestWithIds.reserve(requests.size()); + + // First check valid of request in enqueue thread, so Exceptions can be thrown to user. + for (auto const& req : requests) + { + auto logitsPostProcessorName = req.getLogitsPostProcessorName(); + if (logitsPostProcessorName && logitsPostProcessorName.value() != Request::kBatchedPostProcessorName) + { + getLogitsPostProcessor(*logitsPostProcessorName); + } + } + + std::vector<IdType> ids; + { + auto now = std::chrono::steady_clock::now(); + for (auto const& req : requests) + { + ids.emplace_back(generateReqId(req)); + TLLM_LOG_DEBUG("Enqueue new request with id %d", ids.back()); + + std::vector<IdType> childReqIds; + auto numChildRequests = getNumChildRequests(req); + if (numChildRequests > 0) + { + childReqIds.reserve(numChildRequests); + for (int childId = 0; childId < numChildRequests; childId++) + { + childReqIds.emplace_back(generateLocalReqId()); + TLLM_LOG_DEBUG("Add new child request with id %d", childReqIds.back()); + } + } + requestWithIds.emplace_back(RequestWithId{req, ids.back(), std::move(childReqIds), now}); + } + } + + if (mCommMode == CommunicationMode::kLEADER) + { + { + std::scoped_lock<std::mutex> const lck(mQueuedReqMtx); + if (mMaxQueueSize) + { + auto const maxQueueSize = mMaxQueueSize.value(); + + auto totalRequestSize = 0; + for (auto&& reqWithId : requestWithIds) + { + totalRequestSize += (getNumChildRequests(reqWithId.req) + 1); + } + + if (maxQueueSize > 0 && mQueuedRequests.size() + totalRequestSize > static_cast<size_t>(maxQueueSize)) + { + TLLM_THROW("Maximum queue size of %d has been reached, please try again later", maxQueueSize); + } + } + + for (auto&& req : requestWithIds) + { + insertRequestInOrder(mQueuedRequests, std::move(req)); + } + } + mQueuedReqCv.notify_one(); + } + else if (mCommMode == CommunicationMode::kORCHESTRATOR) + { + MpiMessage message(MpiId::PENDING_REQUEST); + message.data = PendingRequestData{std::move(requestWithIds)}; + mSendQueue.push(std::move(message)); + } + return ids; +} + +std::vector<Response> Executor::Impl::awaitResponses(std::optional<std::chrono::milliseconds> const& timeout) +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called"); + checkParallelApiUsage(__func__); + std::unique_lock<std::mutex> lck(mResponsesMtx); + auto pred = [this]() -> bool { return !mResponses.empty() || mShutdown; }; + auto storeResponses = [this]() + { + std::vector<Response> responses; + for (auto it = mResponses.begin(); it != mResponses.end();) + { + responses.insert(responses.end(), it->second.begin(), it->second.end()); + addTerminatedReqId(it->second, it->first); + it = mResponses.erase(it); + } + return responses; + }; + + std::vector<Response> responses; + if (timeout) + { + if (mResponsesCv.wait_for(lck, timeout.value(), pred)) + { + responses = storeResponses(); + } + } + else + { + mResponsesCv.wait(lck, pred); + responses = storeResponses(); + } + return responses; +} + +std::vector<Response> Executor::Impl::awaitResponses( + IdType const& reqId, std::optional<std::chrono::milliseconds> const& timeout) +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called"); + checkParallelApiUsage(__func__); + std::unique_lock<std::mutex> lck(mResponsesMtx); + auto pred = [this, reqId]() -> bool + { return (mResponses.find(reqId) != mResponses.end() && !mResponses.at(reqId).empty()) || mShutdown; }; + auto storeIdResponse = [this, reqId]() + { + std::vector<Response> responses; + responses.swap(mResponses.at(reqId)); + mResponses.erase(reqId); + addTerminatedReqId(responses, reqId); + return responses; + }; + + // We don't process a terminated request again. Terminated request is defined as a response + // with isFinal = true for a given requestId. + if (mTerminatedReqIds.contains(reqId)) + { + if (mResponses.find(reqId) != mResponses.end()) + { + TLLM_THROW("ReqId should already be removed from responses!"); + } + std::string const err = "ReqId " + std::to_string(reqId) + " has already been processed and was terminated."; + TLLM_LOG_ERROR("%s", err.c_str()); + + return {Response(reqId, err)}; + } + + std::vector<Response> responses; + if (timeout) + { + if (mResponsesCv.wait_for(lck, timeout.value(), pred)) + { + responses = storeIdResponse(); + } + } + else + { + mResponsesCv.wait(lck, pred); + responses = storeIdResponse(); + } + return responses; +} + +std::vector<std::vector<Response>> Executor::Impl::awaitResponses( + std::vector<IdType> const& requestIds, std::optional<std::chrono::milliseconds> const& timeout) +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called"); + checkParallelApiUsage(__func__); + std::vector<std::vector<Response>> responses; + responses.reserve(requestIds.size()); + if (timeout) + { + auto const start_time = std::chrono::high_resolution_clock::now(); + for (auto const requestId : requestIds) + { + auto const elapsed_ms = std::chrono::duration_cast<std::chrono::milliseconds>( + std::chrono::high_resolution_clock::now() - start_time); + responses.emplace_back(awaitResponses( + requestId, timeout.value() > elapsed_ms ? timeout.value() - elapsed_ms : std::chrono::milliseconds{0})); + } + } + else + { + for (auto const requestId : requestIds) + { + responses.emplace_back(awaitResponses(requestId)); + } + } + return responses; +} + +SizeType32 Executor::Impl::getNumResponsesReady(std::optional<IdType> const& optId) const +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called"); + checkParallelApiUsage(__func__); + std::scoped_lock<std::mutex> lck(mResponsesMtx); + SizeType32 numResponsesReady = 0; + if (optId) + { + auto const reqId = optId.value(); + auto const respIt = mResponses.find(reqId); + if (respIt != mResponses.end()) + { + numResponsesReady = static_cast<SizeType32>(respIt->second.size()); + } + } + else + { + for (auto const& [id, responses] : mResponses) + { + numResponsesReady += static_cast<SizeType32>(responses.size()); + } + } + return numResponsesReady; +} + +void Executor::Impl::shutdown() +{ + // Cannot call shutdown multiple times + if (mShutdownCalled) + { + return; + } + mShutdownCalled = true; + + if (!mShutdown) + { + if (mCommMode == CommunicationMode::kLEADER && mIsLeader) + { + // Enqueue a request to indicate to other ranks to terminate + enqueueTerminateRequest(); + } + else if (mCommMode == CommunicationMode::kORCHESTRATOR) + { + if (mIsOrchestrator) + { + // Send to the leader the termination signal + mShutdown = true; + mResponsesCv.notify_all(); + + mSendQueue.push(MpiMessage(MpiId::TERMINATION)); + + // Wait for sender thread to exit + if (mOrchSendReqThread.joinable()) + { + mOrchSendReqThread.join(); + } + // Wait for recv response thread to exit + if (mOrchRecvThread.joinable()) + { + mOrchRecvThread.join(); + } + } + else if (mIsLeader) + { + // Wait for sender thread to exit + if (mLeaderRecvReqThread.joinable()) + { + mLeaderRecvReqThread.join(); + } + // Wait for send response thread to exit + if (mLeaderSendThread.joinable()) + { + mLeaderSendThread.join(); + } + } + } + } + + // Wait for execution thread to terminate + if (mExecutionThread.joinable()) + { + mExecutionThread.join(); + } + + // If we overwrote COMM_SESSION with split, free it now. Otherwise, since + // COMM_SESSION is a global static object, it will be destroyed in an + // undefined order and can cause crashes on program exit. + if (mIsWorker) + { + tensorrt_llm::mpi::MpiComm::setSession(tensorrt_llm::mpi::MpiComm(MPI_COMM_WORLD, false)); + } +} + +void Executor::Impl::cancelRequest(IdType requestId) +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called"); + checkParallelApiUsage(__func__); + + // Check if the request is terminated already. If so, return + { + std::scoped_lock<std::mutex> lckResp(mResponsesMtx); + if (mTerminatedReqIds.contains(requestId)) + { + TLLM_LOG_INFO("Ignoring already terminated request %lu", requestId); + return; + } + } + + if (mCommMode == CommunicationMode::kLEADER) + { + std::scoped_lock<std::mutex> lck(mCancelReqMtx); + auto& selCancelledReqIds = mUsePipelineParallel ? mPipelineCancelledReqIds : mCancelledReqIds; + selCancelledReqIds.insert(requestId); + } + else if (mCommMode == CommunicationMode::kORCHESTRATOR) + { + MpiMessage message(MpiId::CANCEL_REQUEST); + std::vector<IdType> cancelledReqIds{requestId}; + message.data = RequestIdsData{std::move(cancelledReqIds)}; + mSendQueue.push(std::move(message)); + } +} + +std::deque<IterationStats> Executor::Impl::getLatestIterationStats() +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called"); + checkParallelApiUsage(__func__); + std::scoped_lock<std::mutex> lck(mIterStatsMtx); + return std::exchange(mIterationStats, {}); +} + +std::deque<RequestStatsPerIteration> Executor::Impl::getLatestRequestStats() +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called"); + checkParallelApiUsage(__func__); + + std::scoped_lock<std::mutex> lck(mRequestStatsMtx); + return std::exchange(mRequestStats, {}); +} + +std::deque<DebugTensorsPerIteration> Executor::Impl::getLatestDebugTensors() +{ + TLLM_CHECK_WITH_INFO(!mShutdownCalled, "Shutdown called"); + if (mCommMode == CommunicationMode::kORCHESTRATOR) + { + TLLM_LOG_WARNING("getLatestDebugTensors is not supported in ORCHESTRATOR mode yet"); + return {}; + } + if (mEncoderModel) + { + TLLM_LOG_WARNING("getLatestDebugTensors is not supported for encoder model yet"); + } + std::scoped_lock<std::mutex> lck(mDebugTensorsMtx); + return std::exchange(mDebugTensors, {}); +} + +bool Executor::Impl::canEnqueueRequests() const +{ + return !mShutdownCalled + && ((mCommMode == CommunicationMode::kLEADER && mIsLeader) + || (mCommMode == CommunicationMode::kORCHESTRATOR && mIsOrchestrator)); +} + +bool Executor::Impl::isParticipant() const +{ + return mIsWorker; +} + +std::optional<std::shared_ptr<KVCacheEventManager>> Executor::Impl::getKVCacheEventManager() const +{ + if (!mModel) + { + return std::nullopt; + } + auto cacheEventManager = mModel->getKVCacheManager(); + return cacheEventManager ? std::optional(std::make_shared<KVCacheEventManager>(cacheEventManager)) : std::nullopt; +} + +void Executor::Impl::requestWithIdLeaderThread() +{ + TLLM_CUDA_CHECK(cudaSetDevice(mModel->getWorldConfig().getDevice())); + auto constexpr peer = 0; + while (true) + { + int64_t numActiveRequests; + mCommPipelineParallel->recv( + &numActiveRequests, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kExecutorNumActiveRequests); + if (numActiveRequests < 0) + { + break; + } + + bool lowestPriorityActiveHasValue; + std::optional<PriorityType> lowestPriorityActive; + mCommPipelineParallel->recv(&lowestPriorityActiveHasValue, 1, mpi::MpiType::kBOOL, peer, + mpi::MpiTag::kExecutorLowestPriorityActiveHasValue); + if (lowestPriorityActiveHasValue) + { + PriorityType lowestPriorityActiveValue; + mCommPipelineParallel->recv( + &lowestPriorityActiveValue, 1, mpi::MpiType::kFLOAT, peer, mpi::MpiTag::kExecutorLowestPriorityActive); + lowestPriorityActive = lowestPriorityActiveValue; + } + + auto reqWithIds = getLeaderNewReqWithIds(numActiveRequests, lowestPriorityActive); + setupDynamicLogitsPostProcessors(reqWithIds); + auto requestWithIdAsyncSndHdl + = std::make_unique<RequestWithIdAsyncSend>(mCommPipelineParallel, reqWithIds, peer); + requestWithIdAsyncSndHdl.reset(nullptr); + } +} + +void Executor::Impl::cancelledRequestsLeaderThread() +{ + TLLM_CUDA_CHECK(cudaSetDevice(mModel->getWorldConfig().getDevice())); + auto constexpr peer = 0; + while (true) + { + bool shouldExit; + mCommPipelineParallel->recv(&shouldExit, 1, mpi::MpiType::kBOOL, peer, mpi::MpiTag::kExecutorShouldExit); + if (shouldExit) + { + break; + } + + std::unique_ptr<CancelledRequestsAsyncSend> cancelledRequestsAsyncSndHdl; + { + std::scoped_lock<std::mutex> lck(mCancelReqMtx); + cancelledRequestsAsyncSndHdl + = std::make_unique<CancelledRequestsAsyncSend>(mCommPipelineParallel, mPipelineCancelledReqIds, peer); + mPipelineCancelledReqIds.clear(); + } + cancelledRequestsAsyncSndHdl.reset(nullptr); + } +} + +std::vector<RequestWithId> Executor::Impl::getLeaderNewReqWithIds( + SizeType32 numActiveRequests, std::optional<PriorityType> lowestPriorityActive) +{ + std::unique_lock<std::mutex> lck(mQueuedReqMtx); + mQueuedReqCv.wait(lck, [&]() { return (!mQueuedRequests.empty() || numActiveRequests > 0 || mShutdown); }); + + std::vector<RequestWithId> reqWithIds; + + if (mQueuedRequests.empty() || mShutdown) + { + return reqWithIds; + } + + if (mQueuedRequests.front().id == kTerminateReqId) + { + reqWithIds.emplace_back(std::move(mQueuedRequests.front())); + mQueuedRequests.pop_front(); + return reqWithIds; + } + + auto const& firstRequest = mQueuedRequests.front(); + auto const firstBeamWidth = firstRequest.req.getSamplingConfig().getBeamWidth(); + auto const operatingBeamWidth = numActiveRequests > 0 ? mModel->getOperatingBeamWidth() : firstBeamWidth; + + auto const tryInsertQueuedRequestIntoReqWithIds = [this, &reqWithIds, operatingBeamWidth]() -> bool + { + auto& nextRequest = mQueuedRequests.front(); + auto const beamWidth = nextRequest.req.getSamplingConfig().getBeamWidth(); + if (beamWidth != operatingBeamWidth) + { + TLLM_LOG_INFO( + "Can't dequeue request with ID %ld because beam width %d differs from operating beam width %d.", + nextRequest.id, beamWidth, operatingBeamWidth); + return false; + } + + TLLM_LOG_DEBUG("Dequeue request with ID %ld", nextRequest.id); + reqWithIds.emplace_back(std::move(nextRequest)); + mQueuedRequests.pop_front(); + return true; + }; + + auto const maxNewRequests = static_cast<size_t>(std::max(mMaxNumActiveRequests - numActiveRequests, 0)); + for (size_t req = 0; !mQueuedRequests.empty() && req < maxNewRequests;) + { + req += (getNumChildRequests(mQueuedRequests.front().req) + 1); + if (req > maxNewRequests) + { + break; + } + if (!tryInsertQueuedRequestIntoReqWithIds()) + { + break; + } + } + + if (lowestPriorityActive) + { + while (!mQueuedRequests.empty() && mQueuedRequests.front().req.getPriority() > (*lowestPriorityActive)) + { + if (!tryInsertQueuedRequestIntoReqWithIds()) + { + break; + } + } + } + return reqWithIds; +} + +std::vector<RequestWithId> Executor::Impl::getNewReqWithIds( + SizeType32 numActiveRequests, std::optional<PriorityType> lowestPriorityActive) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + auto const& worldConfig = mModel->getWorldConfig(); + + if (worldConfig.isPipelineParallel()) + { + mRequestWithIdWaitThread->waitStop(); + } + + TLLM_CUDA_CHECK(cudaSetDevice(mModel->getWorldConfig().getDevice())); + std::vector<RequestWithId> reqWithIds; + if (mIsPipelineLeader) + { + if (!worldConfig.isPipelineParallel()) + { + reqWithIds = getLeaderNewReqWithIds(numActiveRequests, lowestPriorityActive); + setupDynamicLogitsPostProcessors(reqWithIds); + } + else + { + auto const peer = worldConfig.getPipelineParallelism() - 1; + auto numActiveRequestsValue = static_cast<int64_t>(numActiveRequests); + auto request1 = mCommPipelineParallel->sendAsync( + &numActiveRequestsValue, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kExecutorNumActiveRequests); + bool lowestPriorityActiveHasValue = lowestPriorityActive.has_value(); + auto request2 = mCommPipelineParallel->sendAsync(&lowestPriorityActiveHasValue, 1, mpi::MpiType::kBOOL, + peer, mpi::MpiTag::kExecutorLowestPriorityActiveHasValue); + auto request3 = lowestPriorityActiveHasValue + ? mCommPipelineParallel->sendAsync(&lowestPriorityActive.value(), 1, mpi::MpiType::kFLOAT, peer, + mpi::MpiTag::kExecutorLowestPriorityActive) + : nullptr; + request1->wait(); + request2->wait(); + if (request3) + { + request3->wait(); + } + reqWithIds = RequestWithIdAsyncSend::requestWithIdRecv(mCommPipelineParallel, peer); + } + if (worldConfig.isTensorParallel() || worldConfig.isContextParallel()) + { + auto packed = RequestWithId::serializeReqWithIds(reqWithIds); + if (worldConfig.isTensorParallel()) + { + mCommTensorParallel->bcast(packed, 0); + } + if (worldConfig.isContextParallel()) + { + mCommContextParallel->bcast(packed, 0); + } + } + } + else + { + if (worldConfig.isFirstPipelineParallelRank()) + { + std::vector<char> buffer; + mCommTensorParallel->bcast(buffer, 0); + mCommContextParallel->bcast(buffer, 0); + reqWithIds = RequestWithId::deserializeReqWithIds(buffer); + } + else + { + auto const peer = worldConfig.getPipelineParallelRank() - 1; + reqWithIds = RequestWithIdAsyncSend::requestWithIdRecv(mCommPipelineParallel, peer); + } + } + if (!worldConfig.isLastPipelineParallelRank()) + { + auto const peer = worldConfig.getPipelineParallelRank() + 1; + mRequestWithIdAsyncSndHdl = std::make_unique<RequestWithIdAsyncSend>(mCommPipelineParallel, reqWithIds, peer); + mRequestWithIdWaitThread->notifyStart(); + } + TLLM_CUDA_CHECK(cudaSetDevice(mModel->getWorldConfig().getDevice())); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return reqWithIds; +} + +std::tuple<Executor::Impl::RequestList, double> Executor::Impl::fetchNewRequests( + SizeType32 numActiveRequests, std::optional<PriorityType> lowestPriorityActive) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_SCOPED_RANGE(fetchNewRequests); + + // If grab requests from queue, do exchange between ranks + auto reqWithIds = getNewReqWithIds(numActiveRequests, lowestPriorityActive); + RequestList newRequests; + double newActiveRequestsQueueLatencyMS{0.}; + for (auto& reqWithId : reqWithIds) + { + if (reqWithId.id == kTerminateReqId) + { + mShutdown = true; + mResponsesCv.notify_all(); + return {}; + } + + try + { + std::optional<LlmRequestLogitsPostProcessor> llmRequestLogitsPostProcessor; + bool applyLogitsPostProcessorBatched{false}; + if (mModel->getWorldConfig().isLastPipelineParallelRank()) + { + auto logitsPostProcessorName = reqWithId.req.getLogitsPostProcessorName(); + if (logitsPostProcessorName) + { + if (logitsPostProcessorName.value() == Request::kBatchedPostProcessorName) + { + TLLM_CHECK_WITH_INFO( + mLogitsPostProcessorBatched, "Batched logits post processor is not defined."); + applyLogitsPostProcessorBatched = true; + } + else + { + if (logitsPostProcessorName->compare(0, + std::char_traits<char>::length(Request::kDynamicPostProcessorNamePrefix), + Request::kDynamicPostProcessorNamePrefix) + == 0) + { + TLLM_CHECK_WITH_INFO(!mModel->getReplicateLogitsPostProcessor() + || mModel->getWorldConfig().getTensorParallelism() == 1, + "Dynamic logits postprocessor must be used with replicate=false or no tensor " + "parallelism."); + } + if (mModel->getWorldConfig().isFirstTensorParallelRank() + || mModel->getReplicateLogitsPostProcessor()) + { + llmRequestLogitsPostProcessor = getLogitsPostProcessor(logitsPostProcessorName.value()); + } + else + { + llmRequestLogitsPostProcessor + = [](IdType reqId, RtTensorPtr& logits, BeamTokens const& beamTokens, + CudaStreamPtr const& cudaStreamPtr, std::optional<IdType> clientId) {}; + } + } + } + } + auto newLlmReq = std::make_shared<batch_manager::LlmRequest>( + reqWithId.id, reqWithId.req, llmRequestLogitsPostProcessor, applyLogitsPostProcessorBatched); + + auto numReturnSequences = newLlmReq->getNumSubRequests(); + if (numReturnSequences > 1) + { + TLLM_CHECK(reqWithId.childReqIds.size() == static_cast<size_t>(numReturnSequences - 1)); + mChildReqIdsMap[reqWithId.id] = reqWithId.childReqIds; + } + + for (auto seqIdx = 0; seqIdx < numReturnSequences; seqIdx++) + { + auto newReq + = seqIdx == 0 ? newLlmReq : newLlmReq->createChildRequest(reqWithId.childReqIds.at(seqIdx - 1)); + + // If static batching and streaming, disable streaming and exclude input + if (mBatchingType == BatchingType::kSTATIC && newReq->isStreaming()) + { + newReq->setStreaming(false); + newReq->setExcludeInputFromOutput(true); + } + + // Validate the request parameters + newReq->validate(mModel->getMaxInputLen(), mModel->getMaxSequenceLen(), mModel->getMaxDraftLen(), + mModel->getVocabSizePadded(), + mEncoderModel ? std::optional<SizeType32>(mEncoderModel->getMaxInputLen()) : std::nullopt, + mEnableBlockReuse); + + TLLM_CHECK_WITH_INFO(!mEncoderModel || !mIsSchedulerMaxUtilization, + "Encoder or Encoder-Decoder model don't support max utilization scheduler yet. Only max requests " + "or guaranteed no evict."); + + // When streaming is enabled and scheduling policy permits evict/restart, need to guard against the case + // where the sequence is truncated on eviction (to respect maxInputLen limits), resulting in loss of + // some tokens that have been streamed out. In this case, resuming generation may result in different + // completion for locations whose tokens have already been returned. There is no way to protect against + // this, so disallowing. + if (newReq->isStreaming() && !mIsSchedulerGuaranteedNoEvict && !mIsChunkedContext) + { + auto const maxReqSeqLen = newReq->mPromptLen + newReq->mMaxNewTokens; + auto const maxRestartLen = maxReqSeqLen - 1; + TLLM_CHECK_WITH_INFO(maxRestartLen <= mModel->getMaxInputLen(), + "Request sequence length is potentially greater than max input length. This cannot be run " + "unless streaming is disabled, context chunking is enabled or the GUARANTEED_NO_EVICT " + "scheduling policy is used"); + } + + // Create the encoder output tensor + if (mEncoderModel) + { + TLLM_CHECK_WITH_INFO(mModel || (!mModel && newReq->getReturnEncoderOutput()), + "Encoder-Decoder models allow optionally returning encoder output. But if it is Encoder-only " + "models, please make sure returnEncoderOutput is always true."); + + // gpu buffers for passing to the next phase + newReq->allocEncoderOutput(mEncoderModel->getBufferManager(), mEncoderModel->getLogitDataType()); + newReq->allocEncoderHiddenStates( + mEncoderModel->getBufferManager(), mEncoderModel->getLogitDataType()); + // pinned buffers for returning results to host + if (newReq->getReturnEncoderOutput()) + { + newReq->allocEncoderOutputHost( + mEncoderModel->getHiddenSize() * mEncoderModel->getWorldConfig().getTensorParallelism(), + mEncoderModel->getLogitDataType()); + } + } + + if (!mEncoderModel && newReq->getEncoderInputFeatures()) + { + TLLM_LOG_INFO("Allocating buffers for encoder output"); + // gpu buffers for passing to the next phase + newReq->allocEncoderOutput(mModel->getBufferManager(), mModel->getLogitDataType()); + newReq->allocEncoderHiddenStates(mModel->getBufferManager(), mModel->getLogitDataType()); + } + + // Create the context logits tensor + if (newReq->getReturnContextLogits()) + { + TLLM_CHECK_WITH_INFO(mModel->getModelConfig().computeContextLogits(), + "Return context logit need to build engine with gather_context_logits"); + newReq->allocContextLogitsHost(mModel->getVocabSizePadded(), mModel->getLogitDataType()); + } + + // Create the generation logits tensor + if (newReq->getReturnGenerationLogits()) + { + TLLM_CHECK_WITH_INFO(mModel->getGatherGenerationLogits(), + "To return generation logits, gather_generation_logits must be enabled in ExecutorConfig"); + + if (mModel->getModelConfig().getSpeculativeDecodingMode().isDraftTokensExternal() + && newReq->hasDraftTokens()) + { + newReq->allocTargetModelAcceptedTokenLogitsHost( + mModel->getVocabSizePadded(), mModel->getLogitDataType()); + } + else + { + newReq->allocGenerationLogitsHost(mModel->getVocabSizePadded(), mModel->getLogitDataType()); + } + } + + if (mModel->getWorldConfig().isLastPipelineParallelRank() && newReq->getGuidedDecodingParams()) + { + TLLM_CHECK_WITH_INFO(mModel->hasGuidedDecoder(), + "Request is specified with GuidedDecodingParams, but GuidedDecoder is not setup. Please " + "provide a valid GuidedDecodingConfig to setup GuidedDecoder."); + } + + if (mModel->getWorldConfig().isLastPipelineParallelRank() && newReq->hasAdditionalOutputs()) + { + newReq->allocAdditionalOutputs([this](std::string const& name) + { return mModel->getTensorDataType(name); }, + [this](std::string const& name) { return mModel->getTensorShape(name); }); + } + + mModel->updatePeftCache(newReq); + + newRequests.emplace_back(std::move(newReq)); + } + + auto queuedEnd = std::chrono::steady_clock::now(); + auto reqQueueLatencyMS + = std::chrono::duration<double, std::milli>(queuedEnd - reqWithId.queuedStart).count(); + newActiveRequestsQueueLatencyMS += reqQueueLatencyMS; + } + catch (runtime::LoraExpectedException const& e) + { + if (mIsLeader) + { + // In case of an expected LoRA exception (e.g. cache full, cache miss), log a warning and enqueue + // response + TLLM_LOG_WARNING("%s", e.what()); + enqueueNewResponses({{reqWithId.id, e.what(), reqWithId.req.getClientId()}}); + } + } + catch (std::exception const& e) + { + if (mIsLeader) + { + // In case of error, create a response with error for this request + auto err = std::string("Encountered an error when fetching new request: ") + e.what(); + TLLM_LOG_ERROR("%s", err.c_str()); + enqueueNewResponses({{reqWithId.id, err, reqWithId.req.getClientId()}}); + } + } + } + TLLM_LOG_DEBUG("[RANK %d] num new requests fetched from queue: %d", COMM_SESSION.getRank(), newRequests.size()); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return {newRequests, newActiveRequestsQueueLatencyMS}; +} + +void Executor::Impl::terminateActiveRequests(RequestList& activeRequests, std::string const& err) +{ + TLLM_LOG_ERROR("%s", err.c_str()); + + // Create a response for all requests and add to queue + for (auto it = activeRequests.cbegin(); it != activeRequests.cend();) + { + auto llmReq = (*it); + + llmReq->setState(batch_manager::LlmRequestState::kGENERATION_COMPLETE); + mModel->terminateRequest(llmReq); + + if (mIsLeader) + { + enqueueNewResponses({{llmReq->mRequestId, err, llmReq->mClientId}}); + } + + // Remove from the requestList + it = activeRequests.erase(it); + } +} + +void Executor::Impl::forwardSync(RequestList& activeRequests) +{ + TLLM_LOG_TRACE("[RANK %d] %s start", COMM_SESSION.getRank(), __PRETTY_FUNCTION__); + try + { + if (mEncoderModel) + { + mEncoderModel->forwardSync(); + } + mModel->forwardSync(); + } + catch (std::exception const& e) + { + std::string const err = std::string("Encountered an error in forwardSync function: ") + e.what(); + terminateActiveRequests(activeRequests, err); + } + TLLM_LOG_TRACE("[RANK %d] %s stop", COMM_SESSION.getRank(), __PRETTY_FUNCTION__); +} + +// The function is used to change the state of a request to context_init from encoder_init for enc-dec model whose +// encoder is skipped. The encoder output is populated accordingly with input features given through model executor of +// decoder. +void Executor::Impl::prepRequestsForEncoderSkip(RequestList& activeRequests) +{ + + for (auto& req : activeRequests) + { + + if (req->isEncoderInitState() && req->getEncoderInputFeatures()) + { + TLLM_LOG_INFO("Changing state of request and setting encoder output to skip encoder run"); + req->setState(batch_manager::LlmRequestState::kCONTEXT_INIT); + req->setEncoderOutput(req->getEncoderInputFeatures()); + } + } +} + +void Executor::Impl::finishTimedOutRequests(RequestList const& activeRequests) +{ + if (mIsLeader) + { + for (auto const& request : activeRequests) + { + if (request->isTimedOut() && !request->isFinished()) + { + // workaround to cancelRequest since it throws an error if + // mCommMode == CommunicationMode::kORCHESTRATOR && !mIsOrchestrator + { + std::scoped_lock<std::mutex> lck(mCancelReqMtx); + auto& selCancelledReqIds = mUsePipelineParallel ? mPipelineCancelledReqIds : mCancelledReqIds; + selCancelledReqIds.insert(request->mRequestId); + } + } + } + } +} + +void Executor::Impl::forwardAsync(RequestList& activeRequests) +{ + try + { + TLLM_LOG_DEBUG("num active requests in scope: %d", activeRequests.size()); + + if (mDynamicBatchTuner) + { + auto const averageInputLength = static_cast<SizeType32>(mDynamicBatchTuner->getAverageInputLength()); + auto const averageOutputLength = static_cast<SizeType32>(mDynamicBatchTuner->getAverageOutputLength()); + auto const maxCapacityBatchSize = mModel->getMaxCapacityBatchSize(averageInputLength, averageOutputLength); + + if (mDynamicBatchTuner->isBatchSizeTuningEnabled()) + { + auto runtimeBatchSize = mDynamicBatchTuner->getRuntimeBatchSize(maxCapacityBatchSize); + mModel->setRuntimeBatchSize(runtimeBatchSize); + } + + if (mDynamicBatchTuner->isMaxNumTokensTuningEnabled()) + { + auto runtimeBatchSize = mModel->getRuntimeBatchSize(); + auto runtimeMaxNumTokens = mDynamicBatchTuner->getRuntimeMaxNumTokens(runtimeBatchSize); + mModel->setRuntimeMaxNumTokens(runtimeMaxNumTokens); + } + } + + if (mEncoderModel) + { + mEncoderModel->forwardAsync(activeRequests); + auto const& encoderStream = *(mEncoderModel->getRuntimeStreamPtr()); + auto const& decoderStream = *(mModel->getRuntimeStreamPtr()); + runtime::CudaEvent encoderFinished; + encoderStream.record(encoderFinished); + decoderStream.wait(encoderFinished); + } + + if (!mEncoderModel) + { + prepRequestsForEncoderSkip(activeRequests); + } + + mModel->forwardAsync(activeRequests); + } + catch (std::exception const& e) + { + std::string err = std::string("Encountered an error in forwardAsync function: ") + e.what(); + terminateActiveRequests(activeRequests, err); + } +} + +IterationStats Executor::Impl::getCurrentIterationStats(RequestList const& activeRequests, double iterLatencyMS, + SizeType32 numNewActiveRequests, double newActiveRequestsQueueLatencyMS, SizeType32 numCompletedRequests) +{ + IterationStats stats; + // Timestamp + stats.timestamp = tensorrt_llm::common::getCurrentTimestamp(); + stats.numNewActiveRequests = numNewActiveRequests; + stats.iterLatencyMS = iterLatencyMS; + stats.newActiveRequestsQueueLatencyMS = newActiveRequestsQueueLatencyMS; + // Active request count + stats.numActiveRequests = static_cast<SizeType32>(activeRequests.size()); + // Queued request count + { + std::scoped_lock<std::mutex> lck(mQueuedReqMtx); + stats.numQueuedRequests = static_cast<SizeType32>(mQueuedRequests.size()); + } + stats.numCompletedRequests = numCompletedRequests; + // Max number of requests + stats.maxNumActiveRequests = mMaxNumActiveRequests; + // Runtime memory allocation statistics + auto const& memoryCounters = runtime::MemoryCounters::getInstance(); + stats.gpuMemUsage = memoryCounters.getGpu(); + stats.cpuMemUsage = memoryCounters.getCpu(); + stats.pinnedMemUsage = memoryCounters.getPinned(); + + // Model specific stats + mModel->getCurrentIterationStats(stats); + return stats; +} + +RequestStatsPerIteration Executor::Impl::getCurrentRequestStats( + RequestList const& activeRequests, RequestList const& finishedRequests) +{ + std::vector<RequestStats> requestStatsVec; + + auto includeDisServingStats = [](LlmRequestPtr const& request, tensorrt_llm::executor::RequestStats& requestStats) + { + auto requestType = request->getLlmRequestType(); + if (requestType == batch_manager::LlmRequestType::LLMREQUEST_TYPE_CONTEXT_ONLY + || requestType == batch_manager::LlmRequestType::LLMREQUEST_TYPE_GENERATION_ONLY) + { + requestStats.disServingStats + = executor::DisServingRequestStats{request->getKvCacheTransferTimeMS(), request->getKvCacheSize()}; + } + }; + + for (auto const& request : activeRequests) + { + RequestStats requestStats; + requestStats.id = request->mRequestId; + requestStats.stage = request->getRequestStage(); + requestStats.contextPrefillPosition = request->getContextCurrentPosition(); + requestStats.numGeneratedTokens = request->getMaxBeamNumTokens() - request->getOrigPromptLen(); + requestStats.avgNumDecodedTokensPerIter = request->getAvgDecodedTokensPerIter(); + includeDisServingStats(request, requestStats); + requestStats.allocTotalBlocksPerRequest = request->getAllocTotalBlocksPerRequest(); + requestStats.allocNewBlocksPerRequest = request->getAllocNewBlocksPerRequest(); + requestStats.reusedBlocksPerRequest = request->getReusedBlocksPerRequest(); + requestStats.missedBlocksPerRequest = request->getMissedBlocksPerRequest(); + requestStats.kvCacheHitRatePerRequest = request->getKVCacheHitRatePerRequest(); + requestStatsVec.emplace_back(requestStats); + } + + { + std::unique_lock<std::mutex> lck(mQueuedReqMtx); + for (auto const& request : mQueuedRequests) + { + // Still waiting for the first scheduling + RequestStats requestStats; + requestStats.id = static_cast<executor::IdType>(request.id); + requestStats.stage = executor::RequestStage::kQUEUED; + requestStats.contextPrefillPosition = 0; + requestStats.numGeneratedTokens = 0; + requestStats.avgNumDecodedTokensPerIter = 0; + requestStats.allocTotalBlocksPerRequest = 0; + requestStats.allocNewBlocksPerRequest = 0; + requestStats.reusedBlocksPerRequest = 0; + requestStats.missedBlocksPerRequest = 0; + requestStats.kvCacheHitRatePerRequest = 0; + requestStatsVec.emplace_back(requestStats); + } + } + + for (auto const& request : finishedRequests) + { + // Still waiting for the first scheduling + RequestStats requestStats; + requestStats.id = static_cast<executor::IdType>(request->mRequestId); + requestStats.stage = executor::RequestStage::kGENERATION_COMPLETE; + requestStats.contextPrefillPosition = request->getContextCurrentPosition(); + requestStats.numGeneratedTokens = request->getMaxBeamNumTokens() - request->getOrigPromptLen(); + requestStats.avgNumDecodedTokensPerIter = request->getAvgDecodedTokensPerIter(); + includeDisServingStats(request, requestStats); + requestStats.allocTotalBlocksPerRequest = request->getAllocTotalBlocksPerRequest(); + requestStats.allocNewBlocksPerRequest = request->getAllocNewBlocksPerRequest(); + requestStats.reusedBlocksPerRequest = request->getReusedBlocksPerRequest(); + requestStats.missedBlocksPerRequest = request->getMissedBlocksPerRequest(); + requestStats.kvCacheHitRatePerRequest = request->getKVCacheHitRatePerRequest(); + requestStatsVec.emplace_back(requestStats); + } + + RequestStatsPerIteration stats{0, std::move(requestStatsVec)}; + + // Model specific stats + mModel->getCurrentRequestStats(stats); + return stats; +} + +void Executor::Impl::appendCurrentIterStats(IterationStats&& currentIterStats) +{ + std::scoped_lock<std::mutex> lck(mIterStatsMtx); + if (statsBufferIsBounded(mIterStatsMaxIterations)) + { + auto const maxIterStats = static_cast<std::size_t>(mIterStatsMaxIterations); + if (mIterationStats.size() >= maxIterStats) + { + mIterationStats.pop_front(); + } + } + mIterationStats.emplace_back(std::move(currentIterStats)); +} + +void Executor::Impl::appendMultipleIterStats(std::vector<IterationStats>&& currentIterStatsVec) +{ + std::scoped_lock<std::mutex> lck(mIterStatsMtx); + mIterationStats.insert(mIterationStats.end(), std::make_move_iterator(currentIterStatsVec.begin()), + std::make_move_iterator(currentIterStatsVec.end())); + if (statsBufferIsBounded(mIterStatsMaxIterations)) + { + auto const maxIterStats = static_cast<std::size_t>(mIterStatsMaxIterations); + while (mIterationStats.size() > maxIterStats) + { + mIterationStats.pop_front(); + } + } +} + +void Executor::Impl::updateIterationStats(RequestList const& activeRequests, double iterLatencyMS, + SizeType32 numNewActiveRequests, double newActiveRequestsQueueLatencyMS, SizeType32 numCompletedRequests, + bool flushToOrchestrator) +{ + NVTX3_SCOPED_RANGE(updateIterationStats); + if (statsBufferIsEnabled(mIterStatsMaxIterations) && mIsLeader) + { + auto currentIterStats = getCurrentIterationStats( + activeRequests, iterLatencyMS, numNewActiveRequests, newActiveRequestsQueueLatencyMS, numCompletedRequests); + // Send the stats to the orchestrator + if (mCommMode == CommunicationMode::kORCHESTRATOR) + { + bool hasSchedThisIter = (currentIterStats.inflightBatchingStats + && currentIterStats.inflightBatchingStats->numScheduledRequests > 0) + || (currentIterStats.staticBatchingStats + && currentIterStats.staticBatchingStats->numScheduledRequests > 0); + appendCurrentIterStats(std::move(currentIterStats)); + if (hasSchedThisIter || flushToOrchestrator) + { + std::deque<IterationStats> iterStatsQueue; + { + std::scoped_lock<std::mutex> lck(mIterStatsMtx); + iterStatsQueue = std::exchange(mIterationStats, {}); + } + MpiMessage message(MpiId::ITER_STATS); + std::vector<IterationStats> iterStates( + std::make_move_iterator(iterStatsQueue.begin()), std::make_move_iterator(iterStatsQueue.end())); + message.data = IterStatsData{std::move(iterStates)}; + mSendQueue.push(std::move(message)); + } + } + else + { + // Add current iteration stats + appendCurrentIterStats(std::move(currentIterStats)); + } + } +} + +void Executor::Impl::appendCurrentRequestStats(RequestStatsPerIteration&& currentRequestStats) +{ + std::scoped_lock<std::mutex> lck(mRequestStatsMtx); + if (statsBufferIsBounded(mRequestStatsMaxIterations)) + { + auto const maxRequestStats = static_cast<std::size_t>(mRequestStatsMaxIterations); + if (mRequestStats.size() >= maxRequestStats) + { + mRequestStats.pop_front(); + } + } + mRequestStats.emplace_back(std::move(currentRequestStats)); +} + +void Executor::Impl::appendMultipleRequestStats(std::vector<RequestStatsPerIteration>&& currentRequestStatsVec) +{ + std::scoped_lock<std::mutex> lck(mRequestStatsMtx); + mRequestStats.insert(mRequestStats.end(), std::make_move_iterator(currentRequestStatsVec.begin()), + std::make_move_iterator(currentRequestStatsVec.end())); + if (statsBufferIsBounded(mRequestStatsMaxIterations)) + { + auto const maxRequestStats = static_cast<std::size_t>(mRequestStatsMaxIterations); + while (mRequestStats.size() > maxRequestStats) + { + mRequestStats.pop_front(); + } + } +} + +void Executor::Impl::updateRequestStats( + RequestList const& activeRequests, RequestList const& finishedRequests, bool flushToOrchestrator) +{ + NVTX3_SCOPED_RANGE(updateRequestStats); + if (statsBufferIsEnabled(mRequestStatsMaxIterations) && mIsLeader) + { + // Add current iteration request stats + auto currentRequestStats = getCurrentRequestStats(activeRequests, finishedRequests); + // Send the stats to the orchestrator + if (mCommMode == CommunicationMode::kORCHESTRATOR) + { + bool hasScheduledReqs = false; + if (!flushToOrchestrator) + { + size_t activeSize = activeRequests.size(); + TLLM_CHECK_WITH_INFO(currentRequestStats.requestStats.size() >= activeSize, + "currentRequestStats num is %ld should >= activeRequest num:%zu", + currentRequestStats.requestStats.size(), activeSize); + hasScheduledReqs = std::any_of(currentRequestStats.requestStats.begin(), + currentRequestStats.requestStats.begin() + static_cast<int64_t>(activeSize), + [](RequestStats const& requestStat) { return requestStat.scheduled; }); + } + appendCurrentRequestStats(std::move(currentRequestStats)); + if (hasScheduledReqs || flushToOrchestrator) + { + std::deque<RequestStatsPerIteration> requestStatsQueue; + { + std::scoped_lock<std::mutex> lck(mRequestStatsMtx); + requestStatsQueue = std::exchange(mRequestStats, {}); + } + std::vector<RequestStatsPerIteration> requestIterStates( + std::make_move_iterator(requestStatsQueue.begin()), + std::make_move_iterator(requestStatsQueue.end())); + MpiMessage message(MpiId::REQUEST_ITER_STATS); + message.data = RequestStatsPerIterationData{std::move(requestIterStates)}; + mSendQueue.push(std::move(message)); + } + } + else + { + // Add current iteration stats + appendCurrentRequestStats(std::move(currentRequestStats)); + } + } +} + +void Executor::Impl::appendCurrentDebugTensors() +{ + if (mDebugTensorsMaxIterations > 0) + { + std::scoped_lock<std::mutex> lck(mDebugTensorsMtx); + if (mDebugTensors.size() >= mDebugTensorsMaxIterations) + { + mDebugTensors.pop_front(); + } + mDebugTensors.emplace_back(mModel->getCurrentDebugTensors()); + } +} + +void Executor::Impl::terminateCancelledRequests(RequestList& activeRequests) +{ + NVTX3_SCOPED_RANGE(terminateCancelledRequests); + auto const& worldConfig = mModel->getWorldConfig(); + auto const broadcastCancelledRequests = [this, &activeRequests, &worldConfig] + { + auto const& commSession = COMM_SESSION; + + if (worldConfig.isPipelineParallel()) + { + mCancelledRequestsWaitThread->waitStop(); + } + + if (commSession.getSize() > 1 && !activeRequests.empty()) + { + if (mIsPipelineLeader) + { + if (worldConfig.isPipelineParallel()) + { + auto const peer = worldConfig.getPipelineParallelism() - 1; + bool shouldExit = false; + mCommPipelineParallel->send( + &shouldExit, 1, mpi::MpiType::kBOOL, peer, mpi::MpiTag::kExecutorShouldExit); + auto pipelineCancelledReqIds + = CancelledRequestsAsyncSend::cancelledRequestsRecv(mCommPipelineParallel, peer); + mCancelledReqIds.insert(pipelineCancelledReqIds.begin(), pipelineCancelledReqIds.end()); + } + + auto numCancelledRequests = static_cast<int64_t>(mCancelledReqIds.size()); + if (worldConfig.isTensorParallel()) + { + mCommTensorParallel->bcastValue(numCancelledRequests, 0); + if (numCancelledRequests > 0) + { + std::vector<IdType> cancelledReqIdsVec(mCancelledReqIds.begin(), mCancelledReqIds.end()); + mCommTensorParallel->bcast( + cancelledReqIdsVec.data(), cancelledReqIdsVec.size(), mpi::MpiType::kUINT64, 0); + } + } + if (worldConfig.isContextParallel()) + { + mCommContextParallel->bcastValue(numCancelledRequests, 0); + if (numCancelledRequests > 0) + { + std::vector<IdType> cancelledReqIdsVec(mCancelledReqIds.begin(), mCancelledReqIds.end()); + mCommContextParallel->bcast( + cancelledReqIdsVec.data(), cancelledReqIdsVec.size(), mpi::MpiType::kUINT64, 0); + } + } + } + // If not leader + else + { + if (worldConfig.isFirstPipelineParallelRank()) + { + int64_t numCancelledRequests = 0; + mCommTensorParallel->bcastValue(numCancelledRequests, 0); + mCommContextParallel->bcastValue(numCancelledRequests, 0); + if (numCancelledRequests > 0) + { + std::vector<IdType> cancelledReqIdsVec(numCancelledRequests); + mCommTensorParallel->bcast( + cancelledReqIdsVec.data(), cancelledReqIdsVec.size(), mpi::MpiType::kUINT64, 0); + mCommContextParallel->bcast( + cancelledReqIdsVec.data(), cancelledReqIdsVec.size(), mpi::MpiType::kUINT64, 0); + mCancelledReqIds + = std::unordered_set<IdType>(cancelledReqIdsVec.begin(), cancelledReqIdsVec.end()); + } + } + else + { + auto const peer = worldConfig.getPipelineParallelRank() - 1; + mCancelledReqIds = CancelledRequestsAsyncSend::cancelledRequestsRecv(mCommPipelineParallel, peer); + } + } + if (!worldConfig.isLastPipelineParallelRank()) + { + auto const peer = worldConfig.getPipelineParallelRank() + 1; + mCancelledRequestsAsyncSndHdl + = std::make_unique<CancelledRequestsAsyncSend>(mCommPipelineParallel, mCancelledReqIds, peer); + mCancelledRequestsWaitThread->notifyStart(); + } + } + }; + + std::unique_lock<std::mutex> lck{mCancelReqMtx, std::defer_lock}; + if (!worldConfig.isPipelineParallel()) + { + lck.lock(); + } + + broadcastCancelledRequests(); + + if (!mCancelledReqIds.empty()) + { + // Loop over active requests and terminate those that have been cancelled + std::unordered_set<IdType> terminatedReqIds; + for (auto& req : activeRequests) + { + auto reqId = req->isChild() ? req->getParentRequestId() : req->mRequestId; + if (mCancelledReqIds.find(reqId) != mCancelledReqIds.end()) + { + auto finishReason = req->isTimedOut() ? FinishReason::kTIMED_OUT : FinishReason::kCANCELLED; + mModel->terminateRequestSync(req, finishReason); + // Parent and child requests share the same request id. + // Mark it terminated first and remove from the set later. + terminatedReqIds.insert(reqId); + } + } + + for (auto const& reqId : terminatedReqIds) + { + mCancelledReqIds.erase(reqId); + } + } +} + +void Executor::Impl::terminateContextFinishedRequests(InTransList& inTransmissionRequests) +{ + NVTX3_SCOPED_RANGE(terminateContextFinishedRequests); + for (auto it = inTransmissionRequests.begin(); it != inTransmissionRequests.end();) + { + auto& item = *it; + auto req = item.request; + if (req->isDisaggContextCompleteState()) + { + // If pinnedBlockIds were tracked, unpin them. Otherwise, just terminate. + auto kvMgr = mModel->getKVCacheManager(); + if (kvMgr && !item.pinnedBlockIds.empty()) + { + kvMgr->unpinBlocksById(item.pinnedBlockIds); + } + else + { + mModel->terminateRequest(req); + } + it = inTransmissionRequests.erase(it); + } + else + { + ++it; + } + } +} + +void Executor::Impl::appendNewResponses(std::vector<Response>&& newResponses) +{ + { + std::scoped_lock<std::mutex> lck(mResponsesMtx); + for (auto& response : newResponses) + { + mResponses[response.getRequestId()].emplace_back(std::move(response)); + } + } + mResponsesCv.notify_all(); +} + +Executor::Impl::RequestList Executor::Impl::populateNewResponses( + RequestList& activeRequests, InTransList& inTransmissionRequests, std::vector<Response>& newResponses) +{ + NVTX3_SCOPED_RANGE(populateNewResponses); + RequestList finishedRequests; + for (auto it = activeRequests.begin(); it != activeRequests.end();) + { + auto const& llmReq = (*it); + bool const requestDone = llmReq->isFinished(); + // Only leader should store responses + if (mIsLeader) + { + auto response = llmReq->createResponse(mModel->hasSpeculativeDecodingFastLogits(), mWorldRank); + if (response) + { + newResponses.emplace_back(std::move(response.value())); + } + } + // Remove from active requests if last response has been generated + if (requestDone) + { + // move the in transmission requests to another tracker + if (llmReq->isDisaggContextTransmissionState()) + { + std::vector<SizeType32> pinnedBlockIds{}; + auto kvMgr = mModel->getKVCacheManager(); + if (kvMgr && kvMgr->isEnableBlockReuse() && !kvMgr->getBlockManager().isVariableWindow()) + { + pinnedBlockIds = kvMgr->storeBlocksForReuse(llmReq->mRequestId, llmReq, /*pinBlocks=*/true); + mModel->terminateRequest(llmReq); + } + inTransmissionRequests.push_back(InTransmissionItem{*it, pinnedBlockIds}); + } + finishedRequests.push_back(*it); + it = activeRequests.erase(it); + } + else + { + ++it; + } + } + return finishedRequests; +} + +void Executor::Impl::executionLoop() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + tensorrt_llm::common::setThreadName("executionLoop"); + + auto const& worldConfig = mModel->getWorldConfig(); + TLLM_CUDA_CHECK(cudaSetDevice(worldConfig.getDevice())); + + auto const [profileIterIdxs, stopIterIdxs] = tensorrt_llm::common::populateIterationIndexes( + kPROFILE_START_STOP_ENV_VAR_NAME, kLEGACY_PROFILE_START_STOP_ENV_VAR_NAME); + + SizeType32 numNewActiveRequests{0}; + std::chrono::time_point<std::chrono::steady_clock> iterStart; + std::chrono::time_point<std::chrono::steady_clock> iterEnd; + bool firstIteration{true}; + RequestList activeRequests; + InTransList inTransmissionRequests; + std::vector<Response> newResponses; + while (!mShutdown || !activeRequests.empty()) + { + double iterLatencyMS{0.0}; + double newActiveRequestsQueueLatencyMS{0.0}; + bool reportFinishedRequests = true; + RequestList finishedRequests; + if (!activeRequests.empty()) + { + finishTimedOutRequests(activeRequests); + terminateCancelledRequests(activeRequests); + forwardSync(activeRequests); + finishedRequests = populateNewResponses(activeRequests, inTransmissionRequests, newResponses); + cleanupDynamicLogitsPostProcessors(finishedRequests); + auto const iterCounter = mModel->getIterCounter(); + auto const stopIter = !stopIterIdxs.empty() && (stopIterIdxs.count(iterCounter - 1) > 0); + if (stopIter) + { + cudaProfilerStop(); + } + + // When there are no active or inflight requests, we need to update the stats before calling + // fetchNewRequests to make sure that the stats are reported accurately. + if (activeRequests.empty() && (!firstIteration)) + { + mModel->resetIterationStats(); + updateIterationStats(activeRequests, iterLatencyMS, numNewActiveRequests, + newActiveRequestsQueueLatencyMS, static_cast<SizeType32>(finishedRequests.size()), true); + updateRequestStats(activeRequests, finishedRequests, true); + reportFinishedRequests = false; + } + if (!newResponses.empty()) + { + enqueueNewResponses(std::move(newResponses)); + newResponses.clear(); + } + iterEnd = std::chrono::steady_clock::now(); + iterLatencyMS = std::chrono::duration<double, std::milli>(iterEnd - iterStart).count(); + } + + if (!inTransmissionRequests.empty()) + { + terminateContextFinishedRequests(inTransmissionRequests); + } + + if (!mShutdown) + { + auto const iterCounter = mModel->getIterCounter(); + auto const profileIter = !profileIterIdxs.empty() && (profileIterIdxs.count(iterCounter) > 0); + if (profileIter) + { + cudaProfilerStart(); + } + iterStart = std::chrono::steady_clock::now(); + std::optional<PriorityType> lowestPriority = std::nullopt; + if (!activeRequests.empty()) + { + lowestPriority = activeRequests.back()->priority(); + } + + auto [newRequests, newActiveRequestsQueueLatency] + = fetchNewRequests(static_cast<SizeType32>(activeRequests.size()), lowestPriority); + newActiveRequestsQueueLatencyMS = newActiveRequestsQueueLatency; + numNewActiveRequests = newRequests.size(); + + if (firstIteration) + { + firstIteration = false; + } + + for (auto const& newRequest : newRequests) + { + insertRequestInOrder(activeRequests, newRequest); + } + + // Update dynamic tuning stats + if (mDynamicBatchTuner) + { + for (auto const& req : activeRequests) + { + auto const inputLength = req->mPromptLen; + auto const outputLength = req->mMaxNewTokens; + mDynamicBatchTuner->updateStats(inputLength, outputLength); + } + } + } + if (!activeRequests.empty()) + { + forwardAsync(activeRequests); + updateIterationStats(activeRequests, iterLatencyMS, numNewActiveRequests, newActiveRequestsQueueLatencyMS, + static_cast<SizeType32>(finishedRequests.size()), false); + // Finished requests were reported once. Avoid reporting it twice. + if (reportFinishedRequests) + { + updateRequestStats(activeRequests, finishedRequests, false); + } + else + { + updateRequestStats(activeRequests, {}, false); + } + appendCurrentDebugTensors(); + } + } + + if (mCancelledRequestsWaitThread) + { + mCancelledRequestsWaitThread.reset(nullptr); + } + if (mRequestWithIdWaitThread) + { + mRequestWithIdWaitThread.reset(nullptr); + } + if (worldConfig.isPipelineParallel() && mIsPipelineLeader) + { + auto const peer = worldConfig.getPipelineParallelism() - 1; + int64_t numActiveRequests = -1; + mCommPipelineParallel->send( + &numActiveRequests, 1, mpi::MpiType::kINT64, peer, mpi::MpiTag::kExecutorNumActiveRequests); + bool shouldExit = true; + mCommPipelineParallel->send(&shouldExit, 1, mpi::MpiType::kBOOL, peer, mpi::MpiTag::kExecutorShouldExit); + } + if (mRequestWithIdLeaderThread) + { + mRequestWithIdLeaderThread->join(); + mRequestWithIdLeaderThread.reset(nullptr); + } + if (mCancelledRequestsLeaderThread) + { + mCancelledRequestsLeaderThread->join(); + mCancelledRequestsLeaderThread.reset(nullptr); + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void Executor::Impl::enqueueTerminateRequest() +{ + { + std::scoped_lock<std::mutex> lck(mQueuedReqMtx); + Request dummyReq({1}, 1); + RequestWithId reqWithId{std::move(dummyReq), kTerminateReqId}; + mQueuedRequests.emplace_back(reqWithId); + } + mQueuedReqCv.notify_one(); +} + +void Executor::Impl::enqueueNewResponses(std::vector<Response>&& newResponses) +{ + TLLM_CHECK_WITH_INFO(mIsLeader, "Only leader should store responses"); + + if (mCommMode == CommunicationMode::kLEADER) + { + appendNewResponses(std::move(newResponses)); + } + else if (mCommMode == CommunicationMode::kORCHESTRATOR) + { + MpiMessage message(MpiId::RESPONSE); + message.data = ResponseData{std::move(newResponses)}; + mSendQueue.push(std::move(message)); + } +} + +// Orchestrator thread sending new requests to leader of the model +void Executor::Impl::orchSendReqThread() +{ + tensorrt_llm::common::setThreadName("orchSendReq"); + + while (true) + { + auto message = mSendQueue.pop(); + + if (message.id == MpiId::TERMINATION) + { + mOrchLeaderComm->send(&message.id, 1, mpi::MpiType::kUINT64, mLeaderRank, mpi::MpiTag::kOrchestratorId); + TLLM_LOG_INFO("Orchestrator sendReq thread exiting"); + break; + } + if (message.id == MpiId::PENDING_REQUEST) + { + auto& reqWithIds = std::get<PendingRequestData>(message.data); + auto packed = RequestWithId::serializeReqWithIds(reqWithIds.requests); + + TLLM_LOG_DEBUG("Orchestrator sendReq thread sending %d pending requests", reqWithIds.requests.size()); + // Temporary WAR to indicate to client that we cannot send the serialized request + // because it exceeds int32_t size limit. + // TODO: Should fix as part of https://jirasw.nvidia.com/browse/TRTLLM-708 + if (packed.size() > std::numeric_limits<int32_t>::max()) + { + for (auto const& reqWithId : reqWithIds.requests) + { + { + std::scoped_lock<std::mutex> lck(mResponsesMtx); + mResponses[reqWithId.id].emplace_back(reqWithId.id, + "Request is too large, or you are enqueuing too many requests at once " + "to be sent via MPI_Send, please try to enqueue the request(s) again. " + "This issue will be resolved in a future version of TRT-LLM."); + } + mResponsesCv.notify_all(); + } + } + else + { + mOrchLeaderComm->send(&message.id, 1, mpi::MpiType::kUINT64, mLeaderRank, mpi::MpiTag::kOrchestratorId); + mOrchLeaderComm->send( + packed.data(), packed.size(), mpi::MpiType::kCHAR, mLeaderRank, mpi::MpiTag::kOrchestratorData); + } + } + else if (message.id == MpiId::CANCEL_REQUEST) + { + auto& data = std::get<RequestIdsData>(message.data); + + mOrchLeaderComm->send(&message.id, 1, mpi::MpiType::kUINT64, mLeaderRank, mpi::MpiTag::kOrchestratorId); + mOrchLeaderComm->send( + data.ids.data(), data.ids.size(), mpi::MpiType::kUINT64, mLeaderRank, mpi::MpiTag::kOrchestratorData); + } + else + { + TLLM_THROW("Invalid message id"); + } + } +} + +// Leader thread receiving new requests from orchestrator +void Executor::Impl::leaderRecvReqThread() +{ + tensorrt_llm::common::setThreadName("leaderRecvReq"); + TLLM_CUDA_CHECK(cudaSetDevice(mDeviceId)); +#if ENABLE_MULTI_DEVICE + auto& selCancelledReqIds = mUsePipelineParallel ? mPipelineCancelledReqIds : mCancelledReqIds; + while (true) + { + if (mRecvPollPeriodMs > 0) + { + mOrchLeaderComm->recvPoll(mOrchRank, mpi::MpiTag::kOrchestratorId, mRecvPollPeriodMs); + } + + // Blocking is okay: terminate message is expected to arrive here + MPI_Message msg = nullptr; + MPI_Status status; + mOrchLeaderComm->mprobe(mOrchRank, mpi::MpiTag::kOrchestratorId, &msg, &status); + + int32_t count = 0; + MPICHECK(MPI_Get_count(&status, MPI_UINT64_T, &count)); // NOLINT + TLLM_CHECK(count == 1); + + MpiId mpiId{}; + MPICHECK(MPI_Mrecv(&mpiId, count, MPI_UINT64_T, &msg, &status)); // NOLINT + + // EXIT condition from receiving TERMINATE msg + if (mpiId == MpiId::TERMINATION) + { + // Enqueue a request to indicate to other ranks to terminate + enqueueTerminateRequest(); + + // Send message to orchestrator to indicate to terminate orch recv thread + mSendQueue.push(MpiMessage(mpiId)); + TLLM_LOG_INFO("Leader recvReq thread exiting"); + break; + } + if (mpiId == MpiId::PENDING_REQUEST) + { + mOrchLeaderComm->mprobe(mOrchRank, mpi::MpiTag::kOrchestratorData, &msg, &status); + MPICHECK(MPI_Get_count(&status, MPI_CHAR, &count)); // NOLINT + std::vector<char> buffer(count); + MPICHECK(MPI_Mrecv(buffer.data(), count, MPI_CHAR, &msg, &status)); // NOLINT + + auto requestWithIds = RequestWithId::deserializeReqWithIds(buffer); + TLLM_LOG_DEBUG("Leader recvReq thread receiving %d pending requests", requestWithIds.size()); + { + std::scoped_lock<std::mutex> lck(mQueuedReqMtx); + if (mMaxQueueSize) + { + auto const maxQueueSize = mMaxQueueSize.value(); + if (maxQueueSize > 0 && mQueuedRequests.size() >= static_cast<size_t>(maxQueueSize)) + { + auto err = tensorrt_llm::common::fmtstr( + "Maximum queue size of %d has been reached, please try again later", maxQueueSize); + TLLM_LOG_ERROR("%s", err.c_str()); + std::vector<Response> responses; + responses.reserve(requestWithIds.size()); + for (auto const& reqWithId : requestWithIds) + { + responses.emplace_back(reqWithId.id, err); + } + enqueueNewResponses(std::move(responses)); + continue; + } + } + for (auto&& req : requestWithIds) + { + req.queuedStart = std::chrono::steady_clock::now(); + insertRequestInOrder(mQueuedRequests, std::move(req)); + } + } + mQueuedReqCv.notify_one(); + } + else if (mpiId == MpiId::CANCEL_REQUEST) + { + // Prepare receiving data + mOrchLeaderComm->mprobe(mOrchRank, mpi::MpiTag::kOrchestratorData, &msg, &status); + MPICHECK(MPI_Get_count(&status, MPI_UINT64_T, &count)); // NOLINT + std::vector<uint64_t> cancelledReqIds(count); + MPICHECK(MPI_Mrecv(cancelledReqIds.data(), count, MPI_UINT64_T, &msg, &status)); // NOLINT + + std::scoped_lock<std::mutex> lck(mCancelReqMtx); + selCancelledReqIds.insert(cancelledReqIds.begin(), cancelledReqIds.end()); + } + else + { + TLLM_THROW("Invalid message id"); + } + } +#endif // ENABLE_MULTI_DEVICE +} + +// Leader thread sending responses to orchestrator +void Executor::Impl::leaderSendThread(MpiMessageQueue& sendQueue, mpi::MpiTag idTag, mpi::MpiTag dataTag) +{ + tensorrt_llm::common::setThreadName("leaderSend"); + TLLM_CUDA_CHECK(cudaSetDevice(mDeviceId)); + +#if ENABLE_MULTI_DEVICE + while (true) + { + auto message = sendQueue.pop(); + + if (message.id == MpiId::TERMINATION) + { + mOrchLeaderComm->send(&message.id, 1, mpi::MpiType::kUINT64, mOrchRank, idTag); + TLLM_LOG_INFO("Leader sendThread exiting"); + break; + } + if (message.id == MpiId::RESPONSE || message.id == MpiId::ITER_STATS + || message.id == MpiId ::REQUEST_ITER_STATS) + { + std::vector<char> buffer; + if (message.id == MpiId::RESPONSE) + { + auto& responseData = std::get<ResponseData>(message.data); + TLLM_LOG_DEBUG("Leader sendResp thread sending %d responses", responseData.responses.size()); + buffer = Serialization::serialize(responseData.responses); + } + else if (message.id == MpiId::ITER_STATS) + { + auto& iterStatsData = std::get<IterStatsData>(message.data); + TLLM_LOG_DEBUG("Leader sendResp thread sending iter stats"); + buffer = Serialization::serialize(iterStatsData.iterStatsVec); + } + else if (message.id == MpiId::REQUEST_ITER_STATS) + { + auto& requestIterStatsData = std::get<RequestStatsPerIterationData>(message.data); + TLLM_LOG_DEBUG("Leader sendResp thread sending iter request stats"); + buffer = Serialization::serialize(requestIterStatsData.requestStatsPerIterationVec); + } + mOrchLeaderComm->send(&message.id, 1, mpi::MpiType::kUINT64, mOrchRank, idTag); + mOrchLeaderComm->send(buffer.data(), buffer.size(), mpi::MpiType::kCHAR, mOrchRank, dataTag); + } + else + { + TLLM_THROW("Invalid message id"); + } + } +#endif // ENABLE_MULTI_DEVICE +} + +void Executor::Impl::orchRecvThread(mpi::MpiTag idTag, mpi::MpiTag dataTag) +{ + tensorrt_llm::common::setThreadName("orchRecv"); + +#if ENABLE_MULTI_DEVICE + while (true) + { + if (mRecvPollPeriodMs > 0) + { + mOrchLeaderComm->recvPoll(mOrchRank, mpi::MpiTag::kOrchestratorId, mRecvPollPeriodMs); + } + + MPI_Message msg = nullptr; + MPI_Status status; + mOrchLeaderComm->mprobe(mLeaderRank, idTag, &msg, &status); + + int32_t count = 0; + MPICHECK(MPI_Get_count(&status, MPI_UINT64_T, &count)); // NOLINT + TLLM_CHECK(count == 1); + + MpiId mpiId{}; + MPICHECK(MPI_Mrecv(&mpiId, count, MPI_UINT64_T, &msg, &status)); // NOLINT + + if (mpiId == MpiId::TERMINATION) + { + TLLM_LOG_INFO("Orchestrator recv thread exiting"); + break; + } + if (mpiId == MpiId::RESPONSE || mpiId == MpiId::ITER_STATS || mpiId == MpiId::REQUEST_ITER_STATS) + { + mOrchLeaderComm->mprobe(mLeaderRank, dataTag, &msg, &status); + MPICHECK(MPI_Get_count(&status, MPI_CHAR, &count)); // NOLINT + + std::vector<char> buffer(count); + MPICHECK(MPI_Mrecv(buffer.data(), count, MPI_CHAR, &msg, &status)); // NOLINT + + if (mpiId == MpiId::RESPONSE) + { + auto newResponses = Serialization::deserializeResponses(buffer); + TLLM_LOG_DEBUG("Orchestrator recv thread receiving %d responses", newResponses.size()); + appendNewResponses(std::move(newResponses)); + } + else if (mpiId == MpiId::ITER_STATS) + { + appendMultipleIterStats(Serialization::deserializeIterationStatsVec(buffer)); + } + else if (mpiId == MpiId::REQUEST_ITER_STATS) + { + appendMultipleRequestStats(Serialization::deserializeRequestStatsPerIterationVec(buffer)); + } + } + else + { + TLLM_THROW("Invalid message id"); + } + } +#endif // ENABLE_MULTI_DEVICE +} + +Executor::Impl::LlmRequestLogitsPostProcessor Executor::Impl::getLogitsPostProcessor(std::string const& name) +{ + auto const postProcIt = mLogitsPostProcessorMap.find(name); + TLLM_CHECK_WITH_INFO( + postProcIt != mLogitsPostProcessorMap.end(), "LogitsPostProcessor %s not found.", name.c_str()); + auto executorLogitsPostProcessor = postProcIt->second; + return [executorLogitsPostProcessor](IdType reqId, RtTensorPtr& logits, BeamTokens const& beamTokens, + CudaStreamPtr const& cudaStreamPtr, std::optional<IdType> clientId) + { + auto logitsTensor = executor::detail::ofITensor(logits); + executorLogitsPostProcessor(reqId, logitsTensor, beamTokens, cudaStreamPtr, clientId); + }; +} + +void Executor::Impl::setupDynamicLogitsPostProcessors(std::vector<RequestWithId>& newReqWithIds) +{ + for (auto& reqWithId : newReqWithIds) + { + auto logitsPostProcessor = reqWithId.req.getLogitsPostProcessor(); + if (logitsPostProcessor) + { + std::string const name = Request::kDynamicPostProcessorNamePrefix + std::to_string(reqWithId.id); + mLogitsPostProcessorMap[name] = logitsPostProcessor.value(); + reqWithId.req.setLogitsPostProcessor(std::nullopt); + reqWithId.req.setLogitsPostProcessorName(name); + } + } +} + +void Executor::Impl::cleanupDynamicLogitsPostProcessors(RequestList const& finishedRequests) +{ + for (auto& req : finishedRequests) + { + std::string const name = Request::kDynamicPostProcessorNamePrefix + std::to_string(req->mRequestId); + auto const postProcIt = mLogitsPostProcessorMap.find(name); + if (postProcIt != mLogitsPostProcessorMap.end()) + { + mLogitsPostProcessorMap.erase(name); + } + } +} + +void Executor::Impl::addTerminatedReqId(std::vector<Response> const& responses, IdType const& reqId) +{ + for (auto const& response : responses) + { + if (response.hasError() || (!response.hasError() && response.getResult().isFinal)) + { + mTerminatedReqIds.insert(reqId); + if (mChildReqIdsMap.find(reqId) != mChildReqIdsMap.end()) + { + for (auto childReqId : mChildReqIdsMap.at(reqId)) + { + mTerminatedReqIds.insert(childReqId); + } + mChildReqIdsMap.erase(reqId); + } + } + } +} + +void Executor::Impl::checkParallelApiUsage(std::string const& methodName) const +{ + // If leader mode, and not leader, throw error + if (mCommMode == CommunicationMode::kLEADER && !mIsLeader) + { + // Non-leader are not expected to call cancelRequest + TLLM_THROW("With LEADER communication mode, only leader rank is expected to call %s", methodName.c_str()); + } + if (mCommMode == CommunicationMode::kORCHESTRATOR && !mIsOrchestrator) + { + TLLM_THROW( + "With ORCHESTRATOR communication mode, only orchestrator rank is expected to call %s", methodName.c_str()); + } +} + +} // namespace tensorrt_llm::executor diff --git a/cpp/tensorrt_llm/executor/executorImpl.h b/cpp/tensorrt_llm/executor/executorImpl.h new file mode 100644 index 000000000000..f812b55a3fa0 --- /dev/null +++ b/cpp/tensorrt_llm/executor/executorImpl.h @@ -0,0 +1,385 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/common/arrayView.h" +#include "tensorrt_llm/executor/dynamicBatchTuner.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/intervalSet.h" +#include "tensorrt_llm/executor/model.h" +#include "tensorrt_llm/executor/orchestratorUtils.h" +#include "tensorrt_llm/executor/requestWithId.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/gptJsonConfig.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +#include <atomic> +#include <condition_variable> +#include <list> +#include <mutex> +#include <optional> +#include <queue> +#include <thread> +#include <unordered_map> +#include <unordered_set> + +namespace tensorrt_llm::executor +{ + +class RequestWithIdAsyncSend; +class CancelledRequestsAsyncSend; + +class MpiMessageQueue +{ +public: + void push(MpiMessage&& message) + { + std::lock_guard<std::mutex> const lock(mMutex); + mQueue.push(std::move(message)); + mCv.notify_one(); + } + + MpiMessage pop() + { + std::unique_lock<std::mutex> lock(mMutex); + mCv.wait(lock, [this] { return !mQueue.empty(); }); + MpiMessage message = std::move(mQueue.front()); + mQueue.pop(); + return message; + } + +private: + std::queue<MpiMessage> mQueue; + std::mutex mMutex; + std::condition_variable mCv; +}; + +class Executor::Impl + +{ + using LlmRequestPtr = std::shared_ptr<batch_manager::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + // When block reuse is enabled for context worker for disaggregated serving, + // we need to store the pinned block ids so that we can unpin them when + // the request is finished. + struct InTransmissionItem + { + LlmRequestPtr request; + std::vector<SizeType32> pinnedBlockIds; + }; + + using InTransList = std::list<InTransmissionItem>; + +public: + Impl(std::filesystem::path const& modelPath, std::optional<std::filesystem::path> const& encoderModelPath, + [[maybe_unused]] ModelType modelType, ExecutorConfig const& executorConfig); + + Impl(BufferView const& engineBufferView, std::string const& jsonConfigStr, + std::optional<BufferView> const& encoderEngineBufferView, + std::optional<std::string> const& encoderJsonConfigStr, [[maybe_unused]] ModelType modelType, + ExecutorConfig const& executorConfig, std::optional<std::map<std::string, Tensor>> const& managedWeightsOpt); + + Impl(std::shared_ptr<Model> model, std::optional<std::shared_ptr<Model>> encoderModel, + ExecutorConfig const& executorConfig); + + ~Impl(); + + Impl(Impl const& executor) = delete; + Impl& operator=(Impl const& executor) = delete; + Impl(Impl&&) = delete; + Impl& operator=(Impl&&) = delete; + + IdType enqueueRequest(Request const& request); + + std::vector<IdType> enqueueRequests(std::vector<Request> const& requests); + + std::vector<IdType> enqueueRequests(common::ArrayView<Request const> const& requests); + + std::vector<Response> awaitResponses(std::optional<std::chrono::milliseconds> const& timeout = std::nullopt); + + std::vector<Response> awaitResponses( + IdType const& reqId, std::optional<std::chrono::milliseconds> const& optTimeout = std::nullopt); + + std::vector<std::vector<Response>> awaitResponses( + std::vector<IdType> const& requestIds, std::optional<std::chrono::milliseconds> const& timeout); + + SizeType32 getNumResponsesReady(std::optional<IdType> const& optId = std::nullopt) const; + + void cancelRequest(IdType requestId); + + void shutdown(); + + std::deque<IterationStats> getLatestIterationStats(); + std::deque<RequestStatsPerIteration> getLatestRequestStats(); + std::deque<DebugTensorsPerIteration> getLatestDebugTensors(); + + bool canEnqueueRequests() const; + + bool isParticipant() const; + + std::optional<std::shared_ptr<KVCacheEventManager>> getKVCacheEventManager() const; + +private: + using RtTensorPtr = runtime::ITensor::SharedPtr; + using CudaStreamPtr = runtime::BufferManager::CudaStreamPtr; + using LlmRequestLogitsPostProcessor + = std::function<void(IdType, RtTensorPtr&, BeamTokens const&, CudaStreamPtr, std::optional<IdType>)>; + + void initialize(ExecutorConfig const& executorConfig); + + void loadModel(std::optional<std::filesystem::path> const& modelPath, std::optional<BufferView> const& engineBuffer, + runtime::GptJsonConfig const& jsonConfig, ExecutorConfig const& executorConfig, bool isEncoder, + std::optional<std::map<std::string, Tensor>> const& managedWeightsOpt); + + std::shared_ptr<Model> createModel(runtime::RawEngine const& rawEngine, runtime::ModelConfig const& modelConfig, + runtime::WorldConfig const& worldConfig, ExecutorConfig const& executorConfig); + + std::shared_ptr<Model> createEncoderModel(runtime::RawEngine const& rawEngine, + runtime::ModelConfig const& modelConfig, runtime::WorldConfig const& worldConfig, + ExecutorConfig const& executorConfig); + + void setOrchLeaderComm(SizeType32 tp, SizeType32 pp, SizeType32 cp, ParallelConfig const& parallelConfig); + + void initializeCommAndWorkers(SizeType32 tp, SizeType32 pp, SizeType32 cp, ExecutorConfig const& executorConfig, + std::optional<ModelType> modelType = std::nullopt, + std::optional<std::filesystem::path> const& modelPath = std::nullopt, + std::optional<runtime::WorldConfig> const& worldConfig = std::nullopt, + std::optional<runtime::GptJsonConfig> const& decoderGptJsonConfig = std::nullopt); + + static void validateParallelConfig(ParallelConfig const& parallelConfig, std::optional<ModelType> modelType, + std::optional<std::filesystem::path> const& modelPath); + + void initializeOrchestrator(SizeType32 tp, SizeType32 pp, SizeType32 cp, ExecutorConfig const& executorConfig, + ParallelConfig parallelConfig, ModelType modelType, std::filesystem::path const& modelPath); + + void initializeWorkers(SizeType32 tp, SizeType32 pp, SizeType32 cp, ParallelConfig& parallelConfig, + std::optional<runtime::WorldConfig> const& worldConfig = std::nullopt, + std::optional<runtime::GptJsonConfig> const& decoderGptJsonConfig = std::nullopt); + + void initializeLogitsPostProcessorBatched(LogitsPostProcessorConfig const& logitsProcConfig); + + IdType generateReqId(Request const& request) + { + // If the request has a disaggregated request id, prefer it. + if (request.getDisaggRequestId().has_value() && request.getDisaggRequestId().value() > kMaxLocalReqId) + { + return request.getDisaggRequestId().value(); + } + // Otherwise, generate a local request id in range [1, kMaxLocalReqId). + return generateLocalReqId(); + } + + IdType generateLocalReqId() + { + return (mLastReqId++ % kMaxLocalReqId); + } + + std::vector<RequestWithId> getLeaderNewReqWithIds( + SizeType32 numActiveRequests, std::optional<PriorityType> lowestPriorityActive); + std::vector<RequestWithId> getNewReqWithIds( + SizeType32 numActiveRequests, std::optional<PriorityType> lowestPriorityActive); + + std::tuple<Executor::Impl::RequestList, double> fetchNewRequests( + SizeType32 numActiveRequests, std::optional<PriorityType> lowestPriorityActive); + + void forwardSync(RequestList& activeRequests); + + void forwardAsync(RequestList& activeRequests); + + void prepRequestsForEncoderSkip(RequestList& activeRequests); + + void terminateActiveRequests(RequestList& activeRequests, std::string const& err); + + IterationStats getCurrentIterationStats(RequestList const& activeRequests, double iterLatencyMS, + SizeType32 numNewActiveRequests, double newActiveRequestsQueueLatencyMS, SizeType32 numCompletedRequests); + + void appendCurrentIterStats(IterationStats&& currentIterStats); + void appendMultipleIterStats(std::vector<IterationStats>&& currentIterStatsVec); + void updateIterationStats(RequestList const& activeRequests, double iterLatencyMS, SizeType32 numNewActiveRequests, + double newActiveRequestsQueueLatencyMS, SizeType32 numCompletedRequests, bool flushToOrchestrator); + void appendCurrentRequestStats(RequestStatsPerIteration&& currentRequestStats); + void appendMultipleRequestStats(std::vector<RequestStatsPerIteration>&& currentRequestStatsVec); + RequestStatsPerIteration getCurrentRequestStats( + RequestList const& activeRequests, RequestList const& finishedRequests); + void updateRequestStats( + RequestList const& activeRequests, RequestList const& finishedRequests, bool flushToOrchestrator); + + void appendCurrentDebugTensors(); + + void terminateCancelledRequests(RequestList& activeRequests); + + void terminateContextFinishedRequests(InTransList& inTransmissionRequests); + + void appendNewResponses(std::vector<Response>&& newResponses); + + /// @brief Populates new responses from active requests. + /// Active requests that have completed are erased from activeRequests + /// and returned for bookkeeping. + /// @return A list of requests that have completed. + RequestList populateNewResponses( + RequestList& activeRequests, InTransList& inTransmissionRequests, std::vector<Response>& newResponses); + + void executionLoop(); + + void enqueueTerminateRequest(); + void enqueueNewResponses(std::vector<Response>&& newResponses); + + LlmRequestLogitsPostProcessor getLogitsPostProcessor(std::string const& name); + void setupDynamicLogitsPostProcessors(std::vector<RequestWithId>& newReqWithIds); + void cleanupDynamicLogitsPostProcessors(RequestList const& finishedRequests); + + void orchSendReqThread(); + void orchRecvThread(mpi::MpiTag idTag, mpi::MpiTag dataTag); + void leaderRecvReqThread(); + void leaderSendThread(MpiMessageQueue& sendQueue, mpi::MpiTag idTag, mpi::MpiTag dataTag); + + void addTerminatedReqId(std::vector<Response> const& responses, IdType const& reqId); + + // Check that the current process is the leader or orchestrator + void checkParallelApiUsage(std::string const& methodName) const; + + // These functions wait for MPI async sends on separate threads + void requestWithIdWaitThread(); + void cancelledRequestsWaitThread(); + // These functions send data from leader to pipeline leader on separate threads + void requestWithIdLeaderThread(); + void cancelledRequestsLeaderThread(); + + /// @brief mark requests that have timed out before ever being executed as finished. + /// uses cancellation based on communication mode. + /// + /// @param activeRequests [in] List of active requests to check for timeouts + void finishTimedOutRequests(RequestList const& activeRequests); + + // The model to execute + std::shared_ptr<Model> mModel = nullptr; + std::shared_ptr<Model> mEncoderModel = nullptr; + + // The maximum number of activeRequests + SizeType32 mMaxNumActiveRequests; + + // Thread the executes the main loop + std::thread mExecutionThread; + + // Atomic that indicates threads should shutdown + std::atomic<bool> mShutdown; + + // Atomic that indicates if shutdown method has been called + std::atomic<bool> mShutdownCalled = false; + + // Queued requests + std::mutex mQueuedReqMtx; + std::condition_variable mQueuedReqCv; + std::deque<RequestWithId> mQueuedRequests; + std::optional<SizeType32> mMaxQueueSize; + + // Cancelled requests + std::mutex mCancelReqMtx; + std::unordered_set<IdType> mCancelledReqIds; + std::unordered_set<IdType> mPipelineCancelledReqIds; + + // Ready responses + std::unordered_map<IdType, std::vector<Response>> mResponses; + mutable std::mutex mResponsesMtx; + std::condition_variable mResponsesCv; + + // Since the request IDs are generated sequentially, IntervalSet is preferred over unordered_set for its efficient + // memory usage to stores request ID intervals rather than individual request ID numbers. + IntervalSet<IdType> mTerminatedReqIds; + + std::unordered_map<IdType, std::vector<IdType>> mChildReqIdsMap; + + // Iteration stats + SizeType32 mIterStatsMaxIterations; + std::mutex mIterStatsMtx; + std::deque<IterationStats> mIterationStats; + + // Request stats + SizeType32 mRequestStatsMaxIterations; + std::mutex mRequestStatsMtx; + std::deque<RequestStatsPerIteration> mRequestStats; + + // Debug + IterationType mDebugTensorsMaxIterations; + std::mutex mDebugTensorsMtx; + std::deque<DebugTensorsPerIteration> mDebugTensors; + + IdType mLastReqId = 1; + + static constexpr IdType kTerminateReqId = 0; + // Request id > kMaxLocalReqId is reserved for disaggregated requests. + // This max ID is also in Python side. + static constexpr IdType kMaxLocalReqId = 1ULL << 42U; + + BatchingType mBatchingType; + bool mIsSchedulerMaxUtilization; + bool mIsSchedulerGuaranteedNoEvict; + bool mIsChunkedContext; + bool mPromptTableOffloading; + + CommunicationMode mCommMode; + bool mIsWorker = false; + bool mIsLeader = false; + bool mIsPipelineLeader = false; + bool mUsePipelineParallel = false; + + std::unordered_map<std::string, LogitsPostProcessor> mLogitsPostProcessorMap; + std::optional<Model::LogitsPostProcessorBatched> mLogitsPostProcessorBatched; + + bool mIsOrchestrator = false; + std::shared_ptr<tensorrt_llm::mpi::MpiComm> mOrchLeaderComm; + + std::thread mOrchSendReqThread; + std::thread mOrchRecvThread; + std::thread mLeaderRecvReqThread; + std::thread mLeaderSendThread; + + int32_t mRecvPollPeriodMs = 0; + + int32_t mLeaderRank = -1; + int32_t mOrchRank = 0; + int32_t mWorldRank = -1; + int32_t mDeviceId = 0; + + MpiMessageQueue mSendQueue; + + std::shared_ptr<tensorrt_llm::mpi::MpiComm> mCommTensorParallel; + std::shared_ptr<tensorrt_llm::mpi::MpiComm> mCommPipelineParallel; + std::shared_ptr<tensorrt_llm::mpi::MpiComm> mCommContextParallel; + std::unique_ptr<RequestWithIdAsyncSend> mRequestWithIdAsyncSndHdl; + std::unique_ptr<CancelledRequestsAsyncSend> mCancelledRequestsAsyncSndHdl; + std::unique_ptr<std::thread> mRequestWithIdLeaderThread; + std::unique_ptr<std::thread> mCancelledRequestsLeaderThread; + std::unique_ptr<tensorrt_llm::mpi::MpiWaitThread> mRequestWithIdWaitThread; + std::unique_ptr<tensorrt_llm::mpi::MpiWaitThread> mCancelledRequestsWaitThread; + + // for validating requests + bool mEnableBlockReuse; + + inline static std::string const kPROFILE_START_STOP_ENV_VAR_NAME = "TLLM_PROFILE_START_STOP"; + inline static std::string const kLEGACY_PROFILE_START_STOP_ENV_VAR_NAME = "TLLM_GPTM_PROFILE_START_STOP"; + + std::shared_ptr<DynamicBatchTuner> mDynamicBatchTuner; +}; + +} // namespace tensorrt_llm::executor diff --git a/cpp/tensorrt_llm/executor/intervalSet.h b/cpp/tensorrt_llm/executor/intervalSet.h new file mode 100644 index 000000000000..01f40a685f46 --- /dev/null +++ b/cpp/tensorrt_llm/executor/intervalSet.h @@ -0,0 +1,141 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/executor/types.h" + +namespace tensorrt_llm::executor +{ + +/// @brief An interval inclusive on both ends. +/// A single number interval is represented as [num, num]. +template <typename T> +struct Interval +{ + T lowerEnd; + T upperEnd; +}; + +template <typename T> +bool operator<(Interval<T> const& a, Interval<T> const& b) +{ + return a.lowerEnd < b.lowerEnd; +} + +/// @brief A container to store unique numbers, represented as a vector of ordered and disjoint intervals. +template <typename NumType> +class IntervalSet +{ +public: + /// @brief Check if the given number is in set. + bool contains(NumType num) const + { + // Binary search + SizeType32 left = 0; + SizeType32 right = static_cast<SizeType32>(mIntervals.size()) - 1; + while (left <= right) + { + SizeType32 mid = left + (right - left) / 2; + if (mIntervals[mid].lowerEnd <= num && num <= mIntervals[mid].upperEnd) + { + return true; + } + else if (num < mIntervals[mid].lowerEnd) + { + right = mid - 1; + } + else + { + left = mid + 1; + } + } + return false; + } + + /// @brief Insert a number into set. Do nothing if the number is already in the set. + void insert(NumType num) + { + auto intervalToAdd = Interval<NumType>{num, num}; + + if (mIntervals.size() == 0) + { + mIntervals.insert(mIntervals.begin(), intervalToAdd); + mNumElements++; + return; + } + + // Iter is the first place in mIntervals such that num <= it.lowerEnd + auto iter = std::lower_bound(mIntervals.begin(), mIntervals.end(), intervalToAdd); + + bool iterAtBegin = iter == mIntervals.begin(); + bool iterAtEnd = iter == mIntervals.end(); + + if ((!iterAtEnd && iter->lowerEnd == num) || (!iterAtBegin && num <= (iter - 1)->upperEnd)) + { + // Number falls within the current interval or previous interval. No need to add again. + return; + } + + if (!iterAtBegin && !iterAtEnd && (iter - 1)->upperEnd + 1 == num && iter->lowerEnd - 1 == num) + { + // Merge two adjacent intervals + (iter - 1)->upperEnd = iter->upperEnd; + mIntervals.erase(iter); + } + else if (!iterAtBegin && (iter - 1)->upperEnd + 1 == num) + { + // Number is adjacent to the upper end of the previous interval. Merge left. + (iter - 1)->upperEnd = num; + } + else if (!iterAtEnd && iter->lowerEnd - 1 == num) + { + // Number is adjacent to the lower end of the current interval. Merge right. + iter->lowerEnd = num; + } + else + { + mIntervals.insert(iter, intervalToAdd); + } + mNumElements++; + } + + /// @brief Clear interval set and reset numElements to 0. + void clear() + { + mIntervals.clear(); + mNumElements = 0; + } + + /// @brief Return the size of the set. + SizeType32 getNumElements() const + { + return mNumElements; + } + + /// @brief Return the underlying mIntervals. + std::vector<Interval<NumType>> const& getIntervals() const + { + return mIntervals; + } + +private: + std::vector<Interval<NumType>> mIntervals; + SizeType32 mNumElements{0}; +}; + +} // namespace tensorrt_llm::executor diff --git a/cpp/tensorrt_llm/executor/model.h b/cpp/tensorrt_llm/executor/model.h new file mode 100644 index 000000000000..52fedf1d1113 --- /dev/null +++ b/cpp/tensorrt_llm/executor/model.h @@ -0,0 +1,131 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/batch_manager/logitsPostProcessor.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/worldConfig.h" + +#include <nlohmann/json.hpp> + +namespace tensorrt_llm::executor +{ + +class Model +{ + using LlmRequestPtr = std::shared_ptr<batch_manager::LlmRequest>; + +public: + Model() = default; + + virtual ~Model() = default; + + /// @brief Function that marks a request Id as complete and cleans up associated state + virtual void terminateRequest(LlmRequestPtr const& llmRequest, bool pause) = 0; + + void terminateRequest(LlmRequestPtr const& llmRequest) + { + terminateRequest(llmRequest, false); + } + + /// @brief Terminate request in the next forwardSync call that includes the request. + virtual void terminateRequestSync(LlmRequestPtr const& llmRequest, FinishReason finishReason) = 0; + + /// @brief Function that synchronizes the decoder + virtual void forwardSync() = 0; + + /// @brief Function that tries to advance the active requests + /// Depending on resources available, it's possible that not all requests will get advanced + /// @param activeRequests The list of request to try to advance + virtual void forwardAsync(batch_manager::RequestList const& activeRequests) = 0; + + /// @brief Override the runtime batch size for the model + virtual void setRuntimeBatchSize(SizeType32 runtimeBatchSize) + { + // By default, we ignore the runtimeBatchSize unless the model actively supports it + } + + /// @brief Get the runtime batch size for the model + [[nodiscard]] virtual SizeType32 getRuntimeBatchSize() const + { + TLLM_CHECK_WITH_INFO(false, "getRuntimeBatchSize is not implemented"); + } + + /// @brieft Override the runtime max num tokens for the model + virtual void setRuntimeMaxNumTokens(SizeType32 runtimeMaxNumTokens) + { + // By default, we ignore the runtimeMaxNumTokens unless the model actively supports it + } + + virtual void updatePeftCache(LlmRequestPtr const& llmRequest) = 0; + + /// @brief Reset the iteration stats when there are no inflight requests + virtual void resetIterationStats() = 0; + + [[nodiscard]] virtual SizeType32 getMaxNumSequences() const = 0; + [[nodiscard]] virtual SizeType32 getMaxInputLen() const = 0; + [[nodiscard]] virtual SizeType32 getHiddenSize() const = 0; + [[nodiscard]] virtual SizeType32 getMaxSequenceLen() const = 0; + [[nodiscard]] virtual SizeType32 getVocabSizePadded() const = 0; + [[nodiscard]] virtual SizeType32 getMaxDraftLen() const = 0; + [[nodiscard]] virtual SizeType32 getNumMicroBatches() const = 0; + [[nodiscard]] virtual SizeType32 getOperatingBeamWidth() const = 0; + [[nodiscard]] virtual nvinfer1::DataType getLogitDataType() const = 0; + [[nodiscard]] virtual runtime::WorldConfig const& getWorldConfig() const = 0; + [[nodiscard]] virtual runtime::ModelConfig const& getModelConfig() const = 0; + [[nodiscard]] virtual runtime::BufferManager const& getBufferManager() const = 0; + [[nodiscard]] virtual runtime::BufferManager::CudaStreamPtr getRuntimeStreamPtr() const = 0; + [[nodiscard]] virtual IterationType getIterCounter() const noexcept = 0; + [[nodiscard]] virtual bool hasSpeculativeDecodingFastLogits() const noexcept = 0; + [[nodiscard]] virtual bool getGatherGenerationLogits() const = 0; + [[nodiscard]] virtual nvinfer1::DataType getTensorDataType(std::string const& name) const = 0; + [[nodiscard]] virtual nvinfer1::Dims getTensorShape(std::string const& name) const = 0; + + /// @brief Function that provides per iteration stats specific to a certain model + /// @param stats The json object to write stats to + virtual void getCurrentIterationStats(IterationStats& stats) const = 0; + + /// @brief Function that provides per request stats specific to a certain model + /// @param stats The request stats to be updated + virtual void getCurrentRequestStats(RequestStatsPerIteration& stats) const = 0; + + [[nodiscard]] virtual DebugTensorsPerIteration getCurrentDebugTensors() const = 0; + + using LogitsPostProcessorBatched = tensorrt_llm::batch_manager::LogitsPostProcessor::LogitsPostProcessorBatched; + + virtual void setLogitsPostProcessorBatched(std::optional<LogitsPostProcessorBatched> logitsPostProcessorBatched) + = 0; + virtual void setReplicateLogitsPostProcessor(bool replicateLogitsPostProcessor) = 0; + [[nodiscard]] virtual bool getReplicateLogitsPostProcessor() const = 0; + + [[nodiscard]] virtual bool hasGuidedDecoder() const noexcept = 0; + + [[nodiscard]] virtual std::shared_ptr<tensorrt_llm::batch_manager::kv_cache_manager::BaseKVCacheManager> + getKVCacheManager() = 0; + [[nodiscard]] virtual std::shared_ptr<tensorrt_llm::batch_manager::kv_cache_manager::BaseKVCacheManager const> + getKVCacheManager() const = 0; + + //! \brief Get the batch size that can fill the kv cache to the maximum capacity give the sequence length + //! \param seqLen The sequence length + //! \return The batch size that can fill the kv cache to the maximum capacity. If unsuporrted, return 0. + [[nodiscard]] virtual SizeType32 getMaxCapacityBatchSize(SizeType32 inputLength, SizeType32 outputLength) const = 0; +}; + +} // namespace tensorrt_llm::executor diff --git a/cpp/tensorrt_llm/executor/requestImpl.h b/cpp/tensorrt_llm/executor/requestImpl.h index 145336f77f9a..cd0a03cbb349 100644 --- a/cpp/tensorrt_llm/executor/requestImpl.h +++ b/cpp/tensorrt_llm/executor/requestImpl.h @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -515,7 +515,12 @@ class Request::Impl private: void validate() { - TLLM_CHECK(!mInputTokenIds.empty()); + if (mInputTokenIds.empty()) + { + TLLM_LOG_WARNING( + "Request created with empty inputTokenIds; expected only on empty Helix CP ranks when num_total_blocks " + "< cp_size."); + } TLLM_CHECK(mMaxNewTokens > 0); // Show warning message unless mNumReturnSequences is the default value. diff --git a/cpp/tensorrt_llm/executor/serialization.cpp b/cpp/tensorrt_llm/executor/serialization.cpp index 02077b6857ad..020306e03e56 100644 --- a/cpp/tensorrt_llm/executor/serialization.cpp +++ b/cpp/tensorrt_llm/executor/serialization.cpp @@ -17,7 +17,6 @@ #include "tensorrt_llm/executor/serialization.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/executor/requestImpl.h" @@ -627,8 +626,8 @@ kv_cache::CacheState Serialization::deserializeCacheState(std::istream& is) auto hasRnnConfig = su::deserialize<bool>(is); std::optional<CacheState::RnnModelConfig> rnnModelConfig; std::vector<SizeType32> rnnLayerNumPerPP; - tensorrt_llm::DataType convStateDataType{tensorrt_llm::DataType::kFLOAT}; - tensorrt_llm::DataType ssmStateDataType{tensorrt_llm::DataType::kFLOAT}; + nvinfer1::DataType convStateDataType{nvinfer1::DataType::kFLOAT}; + nvinfer1::DataType ssmStateDataType{nvinfer1::DataType::kFLOAT}; if (hasRnnConfig) { CacheState::RnnModelConfig rnnCfg; @@ -642,8 +641,8 @@ kv_cache::CacheState Serialization::deserializeCacheState(std::istream& is) rnnCfg.mNumHeads = su::deserialize<decltype(CacheState::RnnModelConfig::mNumHeads)>(is); rnnCfg.mConvSectionLayout = static_cast<CacheState::RnnModelConfig::ConvSectionLayout>(su::deserialize<SizeType32>(is)); - convStateDataType = su::deserialize<tensorrt_llm::DataType>(is); - ssmStateDataType = su::deserialize<tensorrt_llm::DataType>(is); + convStateDataType = su::deserialize<nvinfer1::DataType>(is); + ssmStateDataType = su::deserialize<nvinfer1::DataType>(is); rnnLayerNumPerPP = su::deserialize<std::vector<SizeType32>>(is); rnnModelConfig = std::move(rnnCfg); } @@ -765,8 +764,6 @@ DataTransceiverState Serialization::deserializeDataTransceiverState(std::istream { state.setCacheState(std::move(cacheState).value()); } - auto isArbitraryTransferState = su::deserialize<decltype(DataTransceiverState::mIsArbitraryTransferState)>(is); - state.setIsArbitraryTransferState(isArbitraryTransferState); return state; } @@ -774,7 +771,6 @@ void Serialization::serialize(DataTransceiverState const& state, std::ostream& o { su::serialize(state.mCommState, os); su::serialize(state.mCacheState, os); - su::serialize(state.mIsArbitraryTransferState, os); } std::vector<char> Serialization::serialize(DataTransceiverState const& state) @@ -793,7 +789,6 @@ size_t Serialization::serializedSize(DataTransceiverState const& state) size_t totalSize = 0; totalSize += su::serializedSize(state.mCommState); totalSize += su::serializedSize(state.mCacheState); - totalSize += su::serializedSize(state.mIsArbitraryTransferState); return totalSize; } diff --git a/cpp/tensorrt_llm/executor/tensor.cpp b/cpp/tensorrt_llm/executor/tensor.cpp index 9c508c0ec5c3..c38feb0e34b8 100644 --- a/cpp/tensorrt_llm/executor/tensor.cpp +++ b/cpp/tensorrt_llm/executor/tensor.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/executor/tensor.h" #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -54,17 +53,17 @@ DataType Tensor::getDataType() const } switch (mTensor->getDataType()) { - case tensorrt_llm::DataType::kBOOL: return DataType::kBOOL; - case tensorrt_llm::DataType::kINT8: return DataType::kINT8; - case tensorrt_llm::DataType::kINT32: return DataType::kINT32; - case tensorrt_llm::DataType::kUINT8: return DataType::kUINT8; - case tensorrt_llm::DataType::kFP8: return DataType::kFP8; - case tensorrt_llm::DataType::kHALF: return DataType::kFP16; - case tensorrt_llm::DataType::kFLOAT: return DataType::kFP32; - case tensorrt_llm::DataType::kBF16: return DataType::kBF16; - case tensorrt_llm::DataType::kINT64: return DataType::kINT64; - case tensorrt_llm::DataType::kINT4: [[fallthrough]] /* do nothing */; - case tensorrt_llm::DataType::kFP4: [[fallthrough]] /* do nothing */; + case nvinfer1::DataType::kBOOL: return DataType::kBOOL; + case nvinfer1::DataType::kINT8: return DataType::kINT8; + case nvinfer1::DataType::kINT32: return DataType::kINT32; + case nvinfer1::DataType::kUINT8: return DataType::kUINT8; + case nvinfer1::DataType::kFP8: return DataType::kFP8; + case nvinfer1::DataType::kHALF: return DataType::kFP16; + case nvinfer1::DataType::kFLOAT: return DataType::kFP32; + case nvinfer1::DataType::kBF16: return DataType::kBF16; + case nvinfer1::DataType::kINT64: return DataType::kINT64; + case nvinfer1::DataType::kINT4: [[fallthrough]] /* do nothing */; + case nvinfer1::DataType::kFP4: [[fallthrough]] /* do nothing */; default: TLLM_THROW("Unsupported data type"); } } @@ -136,19 +135,19 @@ tr::ITensor::Shape toDims(Shape const& shape) return dims; } -tensorrt_llm::DataType toDataType(DataType dataType) +nvinfer1::DataType toDataType(DataType dataType) { switch (dataType) { - case DataType::kBOOL: return tensorrt_llm::DataType::kBOOL; - case DataType::kUINT8: return tensorrt_llm::DataType::kUINT8; - case DataType::kINT8: return tensorrt_llm::DataType::kINT8; - case DataType::kINT32: return tensorrt_llm::DataType::kINT32; - case DataType::kINT64: return tensorrt_llm::DataType::kINT64; - case DataType::kBF16: return tensorrt_llm::DataType::kBF16; - case DataType::kFP8: return tensorrt_llm::DataType::kFP8; - case DataType::kFP16: return tensorrt_llm::DataType::kHALF; - case DataType::kFP32: return tensorrt_llm::DataType::kFLOAT; + case DataType::kBOOL: return nvinfer1::DataType::kBOOL; + case DataType::kUINT8: return nvinfer1::DataType::kUINT8; + case DataType::kINT8: return nvinfer1::DataType::kINT8; + case DataType::kINT32: return nvinfer1::DataType::kINT32; + case DataType::kINT64: return nvinfer1::DataType::kINT64; + case DataType::kBF16: return nvinfer1::DataType::kBF16; + case DataType::kFP8: return nvinfer1::DataType::kFP8; + case DataType::kFP16: return nvinfer1::DataType::kHALF; + case DataType::kFP32: return nvinfer1::DataType::kFLOAT; case DataType::kUNKNOWN: TLLM_THROW("Unsupported data type"); } diff --git a/cpp/tensorrt_llm/executor_worker/CMakeLists.txt b/cpp/tensorrt_llm/executor_worker/CMakeLists.txt new file mode 100644 index 000000000000..2feb6dfe5790 --- /dev/null +++ b/cpp/tensorrt_llm/executor_worker/CMakeLists.txt @@ -0,0 +1,26 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +set(SRCS executorWorker.cpp) + +include_directories(${PROJECT_SOURCE_DIR}/include) + +set(EXECUTOR_WORKER_TARGET executorWorker) + +add_executable(${EXECUTOR_WORKER_TARGET} ${SRCS}) + +target_link_libraries(${EXECUTOR_WORKER_TARGET} + PUBLIC ${SHARED_TARGET} nvinfer_plugin_tensorrt_llm) + +target_compile_features(${EXECUTOR_WORKER_TARGET} PRIVATE cxx_std_17) diff --git a/cpp/tensorrt_llm/executor_worker/executorWorker.cpp b/cpp/tensorrt_llm/executor_worker/executorWorker.cpp new file mode 100644 index 000000000000..aa1b06c2cb74 --- /dev/null +++ b/cpp/tensorrt_llm/executor_worker/executorWorker.cpp @@ -0,0 +1,92 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/serialization.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include <csignal> + +namespace tle = tensorrt_llm::executor; + +int main(int argc, char* argv[]) +{ +#if ENABLE_MULTI_DEVICE + + if (std::getenv("FORCE_NCCL_ALL_REDUCE_STRATEGY") != nullptr) + { + TLLM_LOG_INFO("FORCE_NCCL_ALL_REDUCE_STRATEGY env variable detected in worker"); + } + + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + tensorrt_llm::mpi::initialize(tensorrt_llm::mpi::MpiThreadSupport::THREAD_MULTIPLE, true); + + MPI_Comm parentComm; + MPI_Comm_get_parent(&parentComm); + if (parentComm == MPI_COMM_NULL) + { + TLLM_LOG_ERROR("TRT-LLM worker has no parent!"); + return -1; + } + + int size; + MPI_Comm_remote_size(parentComm, &size); + if (size != 1) + { + TLLM_LOG_ERROR("Parent size is %d, must be 1", size); + return -1; + } + + // Since parentComm is an intercommunicator, input root + // is the rank of the parent process in his group + // (always 0 as the parent size is checked before) + + // Receive from the parent the executor configuration + int64_t bufferSize; + MPICHECK(MPI_Bcast(&bufferSize, 1, MPI_INT64_T, 0, parentComm)); + std::vector<char> buffer(bufferSize); + MPICHECK(MPI_Bcast(buffer.data(), bufferSize, MPI_CHAR, 0, parentComm)); + std::istringstream is(std::string(buffer.begin(), buffer.end())); + auto modelPath = tle::Serialization::deserializeString(is); + auto modelType = tle::Serialization::deserializeModelType(is); + auto executorConfig = tle::Serialization::deserializeExecutorConfig(is); + + // Create the orchestrator config for workers + auto orchLeaderComm = std::make_shared<tensorrt_llm::mpi::MpiComm>(parentComm, true); + auto parallelConfig = executorConfig.getParallelConfig(); + TLLM_CHECK_WITH_INFO(parallelConfig.has_value(), "Parallel config should have a value."); + TLLM_CHECK_WITH_INFO( + parallelConfig.value().getOrchestratorConfig().has_value(), "Orchestrator config should have a value."); + auto orchConfig = parallelConfig.value().getOrchestratorConfig().value(); + TLLM_CHECK_WITH_INFO(parallelConfig.has_value(), "Parallel config should have a value."); + auto newOrchConfig = tle::OrchestratorConfig(false, orchConfig.getWorkerExecutablePath(), orchLeaderComm); + parallelConfig.value().setOrchestratorConfig(newOrchConfig); + executorConfig.setParallelConfig(parallelConfig.value()); + // In orchestrator mode, the spawned threads will wait for termination signal from orchestrator + auto executor = tle::Executor(modelPath, modelType, executorConfig); + + // Wait for all workers to have created their instances + MPI_Barrier(parentComm); + TLLM_LOG_INFO("Executor instance created by worker"); + +#endif // ENABLE_MULTI_DEVICE + + return 0; +} diff --git a/cpp/tensorrt_llm/kernels/beamSearchKernels.cu b/cpp/tensorrt_llm/kernels/beamSearchKernels.cu index 2dadc9d5b682..005a1539168e 100644 --- a/cpp/tensorrt_llm/kernels/beamSearchKernels.cu +++ b/cpp/tensorrt_llm/kernels/beamSearchKernels.cu @@ -16,7 +16,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/reduceKernelUtils.cuh" #include "tensorrt_llm/kernels/beamSearchKernels.h" using namespace tensorrt_llm::common; @@ -136,26 +135,21 @@ void invokeUpdateCacheIndirection(int* tgtCI, int const* srcCI, BeamHypotheses& sync_check_cuda_error(stream); } -__global__ void addCumLogProbs(float* __restrict pStage1LogProbs, int const* __restrict pStage1Ids, - float const* __restrict cumLogProbs, FinishedState const* finished, int const* endIds, float const* diversityRates, +__global__ void addCumLogProbs(float* __restrict pStage1LogProbs, float const* __restrict cumLogProbs, + FinishedState const* finished, int const* endIds, float const* diversityRates, runtime::SizeType32 const* batchSlots, size_t const nBS, size_t const nBMIn, size_t const nBMOut, size_t const nBM) { int const bid = blockIdx.x; // Index of request in batch runtime::SizeType32 const slot = batchSlots[bid]; float const diversityRate{diversityRates[slot]}; float* pLocalLogProbs = pStage1LogProbs + bid * nBMIn * nBMOut * 2; - int const* pLocalIds = pStage1Ids + bid * nBMIn * nBMOut * 2; for (int i = threadIdx.x; i < nBMIn * nBMOut * 2; i += blockDim.x) { int const iBMIn = i / (nBMOut * 2); - if (finished[slot * nBM + iBMIn].isFinished()) + if (finished[slot * nBMIn + iBMIn].isFinished()) { - // In V2 path, i is a candidate-slot index (0..nBMIn*nBMOut*2-1), NOT a vocab token id. - // Use pStage1Ids to look up the actual token id for the EOS comparison. - bool const isEOS = (pLocalIds[i] == endIds[slot]); - // Keep only the EOS candidate with its proper cumulative score; suppress all others. - pLocalLogProbs[i] = isEOS ? (pLocalLogProbs[i] + cumLogProbs[slot * nBM + iBMIn]) : -FLT_MAX; + pLocalLogProbs[i] += (i == endIds[slot]) ? 1.0f : 0.0f; } else { @@ -166,27 +160,21 @@ __global__ void addCumLogProbs(float* __restrict pStage1LogProbs, int const* __r return; } -__global__ void addCumLogProbs(half* __restrict pStage1LogProbs, int const* __restrict pStage1Ids, - float const* __restrict cumLogProbs, FinishedState const* finished, int const* endIds, float const* diversityRates, +__global__ void addCumLogProbs(half* __restrict pStage1LogProbs, float const* __restrict cumLogProbs, + FinishedState const* finished, int const* endIds, float const* diversityRates, runtime::SizeType32 const* batchSlots, size_t const nBS, size_t const nBMIn, size_t const nBMOut, size_t const nBM) { int const bid = blockIdx.x; // Index of request in batch runtime::SizeType32 const slot = batchSlots[bid]; float const diversityRate{diversityRates[slot]}; half* pLocalLogProbs = pStage1LogProbs + bid * nBMIn * nBMOut * 2; - int const* pLocalIds = pStage1Ids + bid * nBMIn * nBMOut * 2; for (int i = threadIdx.x; i < nBMIn * nBMOut * 2; i += blockDim.x) { int const iBMIn = i / (nBMOut * 2); - if (finished[slot * nBM + iBMIn].isFinished()) + if (finished[slot * nBMIn + iBMIn].isFinished()) { - // In V2 path, i is a candidate-slot index (0..nBMIn*nBMOut*2-1), NOT a vocab token id. - // Use pStage1Ids to look up the actual token id for the EOS comparison. - bool const isEOS = (pLocalIds[i] == endIds[slot]); - // Keep only the EOS candidate with its proper cumulative score; suppress all others. - pLocalLogProbs[i] - = isEOS ? (half) (float(pLocalLogProbs[i]) + cumLogProbs[slot * nBM + iBMIn]) : (half) -HALF_FLT_MAX; + pLocalLogProbs[i] += (i == endIds[slot]) ? 1.0f : 0.0f; } else { diff --git a/cpp/tensorrt_llm/kernels/beamSearchKernels.h b/cpp/tensorrt_llm/kernels/beamSearchKernels.h index 345e4659c941..d8a9266e9406 100644 --- a/cpp/tensorrt_llm/kernels/beamSearchKernels.h +++ b/cpp/tensorrt_llm/kernels/beamSearchKernels.h @@ -131,15 +131,13 @@ void invokeTopkBeamSearch(T const* logProbs, T const* bias, void* workspace, Bea void invokeUpdateCacheIndirection(int* tgtCI, int const* srcCI, BeamHypotheses& bh, runtime::SizeType32 const maxAttentionWindow, runtime::SizeType32 sinkTokenLength, cudaStream_t stream); -__global__ void addCumLogProbs(float* __restrict pStage1LogProbs, int const* __restrict pStage1Ids, - float const* __restrict cumLogProbs, ::tensorrt_llm::kernels::FinishedState const* finished, int const* endIds, - float const* diversityRates, runtime::SizeType32 const* batchSlots, size_t const nBS, size_t const nBMIn, - size_t const nBMOut, size_t const nBM); - -__global__ void addCumLogProbs(half* __restrict pStage1LogProbs, int const* __restrict pStage1Ids, - float const* __restrict cumLogProbs, ::tensorrt_llm::kernels::FinishedState const* finished, int const* endIds, - float const* diversityRates, runtime::SizeType32 const* batchSlots, size_t const nBS, size_t const nBMIn, - size_t const nBMOut, size_t const nBM); +__global__ void addCumLogProbs(float* __restrict pStage1LogProbs, float const* __restrict cumLogProbs, + ::tensorrt_llm::kernels::FinishedState const* finished, int const* endIds, float const* diversityRates, + runtime::SizeType32 const* batchSlots, size_t const nBS, size_t const nBMIn, size_t const nBMOut, size_t const nBM); + +__global__ void addCumLogProbs(half* __restrict pStage1LogProbs, float const* __restrict cumLogProbs, + ::tensorrt_llm::kernels::FinishedState const* finished, int const* endIds, float const* diversityRates, + runtime::SizeType32 const* batchSlots, size_t const nBS, size_t const nBMIn, size_t const nBMOut, size_t const nBM); __global__ void gatherId(int const* __restrict pStage1Id, int* __restrict pStage2Id, size_t const nBS, size_t const nBMIn, size_t const nBMOut, size_t const nV); diff --git a/cpp/tensorrt_llm/kernels/beamSearchKernels/beamSearchKernelsTemplate.h b/cpp/tensorrt_llm/kernels/beamSearchKernels/beamSearchKernelsTemplate.h index 09b114ef9752..eb0d9e072997 100644 --- a/cpp/tensorrt_llm/kernels/beamSearchKernels/beamSearchKernelsTemplate.h +++ b/cpp/tensorrt_llm/kernels/beamSearchKernels/beamSearchKernelsTemplate.h @@ -208,7 +208,6 @@ __launch_bounds__(BLOCK_SIZE) __global__ void beamStage3Kernel( __shared__ float smemCumLogProbs[PBM]; __shared__ int smemSeqLen[PBM]; __shared__ KVPair smemTopKV[(IS_V2) ? 1 : PBM * 2]; // Just a placeholder in V2 workflow - __shared__ int smemNBeamForNextStep; if (bh.numBeamsCBA != nullptr) { @@ -218,11 +217,14 @@ __launch_bounds__(BLOCK_SIZE) __global__ void beamStage3Kernel( // Initialize worst score in the first call bh.minNormedScoresCBA[slot] = 0.0f; // logProbs is in range (-inf, 0] } - else if (earlyStopping == 1 && bh.numBeamsCBA[slot] >= nBM || earlyStopping != 1 && bh.batchDones[slot]) + else if (earlyStopping == 1 && bh.numBeamsCBA[slot] == nBM + || earlyStopping != 1 && bh.finished[slot * nBM].isFinished()) { // Condition of early return: // 1. In EarlyStopping mode, and we have got enough beams // 2. In NonEarlyStopping mode, and this batch has been marked as done + // TODO: improve the condition like below + // earlyStopping == 1 && bh.numBeamsCBA[slot] == nBM || earlyStopping != 1 && bh.batchDones[slot] return; } } @@ -294,11 +296,6 @@ __launch_bounds__(BLOCK_SIZE) __global__ void beamStage3Kernel( } __syncthreads(); - // Timestep at which each next-step destination beam's token is stored in the work tree. - // This is the parent beam's sequence length (the true generation step), which may differ - // from the destination slot's own (possibly stale) length when a finished slot is reused. - __shared__ int smemWriteStep[PBM]; - if (tid == 0) { int nBeamForNextStep{0}; @@ -327,14 +324,10 @@ __launch_bounds__(BLOCK_SIZE) __global__ void beamStage3Kernel( { // Condition of this branch: // This token is end-token and belongs to top nBM range in Beam search mode - // Use the actual parent beam index (topId / nV) % nBM, not the candidate rank i, - // to look up the correct sequenceLength and inputLength for length-penalty scoring. - int const parentBeam = (topId / nV) % nBM; - int const nSeqLen - = bh.sequenceLengths[slot * nBM + parentBeam] + 1 - bh.inputLengths[slot * nBM + parentBeam]; + int const nSeqLen = bh.sequenceLengths[slot * nBM + i] + 1 - bh.inputLengths[slot * nBM + i]; float const score = applyLengthPenalty(topLogProb, nSeqLen, lengthPenalty); int nCBA = bh.numBeamsCBA[slot]; - if (nCBA >= nBM) + if (nCBA == nBM) { // There are already nBM beams if (score < bh.minNormedScoresCBA[slot]) @@ -414,18 +407,13 @@ __launch_bounds__(BLOCK_SIZE) __global__ void beamStage3Kernel( // 1. bh.numBeamsCBA == nullptr && i < nBM, i.e., beam search is disable // 2. bh.numBeamsCBA != nullptr && i < nBM && isEndToken == false, i.e., add token at the end // 3. bh.numBeamsCBA != nullptr && i >= nBM && isEndToken == false, i.e., add token at the end - // Write at the parent beam's sequence length (the actual generation step), - // not the destination slot's length, which can be stale if the slot was - // previously finished and is now being reused for a new continuation. - int const parentBeam = topId / nV % nBM; - int const step = bh.sequenceLengths[slot * nBM + parentBeam]; - smemWriteStep[nBeamForNextStep] = step; + int const step = bh.sequenceLengths[slot * nBM + nBeamForNextStep]; // Copy the selected token to work tree bh.outputIdsPtr[slot][nBeamForNextStep * nMSL + step] = topId; if (bh.logProbsTiled != nullptr) { int const index = step * nMBS * nBM + slot * nBM + nBeamForNextStep; - int const indexBeam = parentBeam; + int const indexBeam = topId / nV % nBM; bh.logProbsTiled[index] = (float) topLogProb - smemCumLogProbs[indexBeam]; } bh.cumLogProbs[slot * nBM + nBeamForNextStep] = (float) topLogProb; @@ -449,7 +437,6 @@ __launch_bounds__(BLOCK_SIZE) __global__ void beamStage3Kernel( break; } } - smemNBeamForNextStep = nBeamForNextStep; } // Update bh.batchDones @@ -494,40 +481,23 @@ __launch_bounds__(BLOCK_SIZE) __global__ void beamStage3Kernel( if (tid < nBMOut) { int const indexBatchBeam = slot * nBM + tid; - if (tid < smemNBeamForNextStep) + int const step = smemSeqLen[tid]; + if (!bh.finished[indexBatchBeam].isFinished()) { - // This slot received a valid next-step token from the selection phase. - // Use the timestep recorded by the selection phase (the parent beam's length), - // which matches the position where the encoded token was stored. - int const step = smemWriteStep[tid]; - int const newId = bh.outputIdsPtr[slot][tid * nMSL + step]; - int const newBeamId = (newId / nV) % nBM; - int const newTokenId = newId % nV; - int const indexParentBeam = slot * nBM + newBeamId; - int const parentSeqLen = smemSeqLen[newBeamId]; - bh.sequenceLengths[indexBatchBeam] = parentSeqLen + (!bh.finished[indexParentBeam].isFinished() ? 1 : 0); - if (newTokenId == bh.endIds[slot]) - { - bh.finished[indexBatchBeam].setFinishedEOS(); - } - else - { - // Reset any stale finished state: this slot may have been marked finished in a - // previous step and is now reused for a valid non-EOS beam; otherwise it would be - // wrongly skipped downstream. - bh.finished[indexBatchBeam] = FinishedState::empty(); - } - bh.parentIdsPtr[slot][tid * nMSL + step] = newBeamId; - bh.outputIdsPtr[slot][tid * nMSL + step] = newTokenId; + smemSeqLen[tid]++; } - else + int const newId = bh.outputIdsPtr[slot][tid * nMSL + step]; + int const newBeamId = (newId / nV) % nBM; + int const newTokenId = newId % nV; + bh.sequenceLengths[indexBatchBeam] = smemSeqLen[newBeamId]; + if (newTokenId == bh.endIds[slot]) { - // No valid next-step token for this slot: all top candidates went to CBA. - // Mark as finished so downstream stages (cache indirection, next decode) skip it. - bh.finished[indexBatchBeam].setFinished(); + bh.finished[indexBatchBeam].setFinishedEOS(); } + bh.parentIdsPtr[slot][tid * nMSL + step] = newBeamId; + bh.outputIdsPtr[slot][tid * nMSL + step] = newTokenId; - if ((earlyStopping == 1) && (bh.numBeamsCBA != nullptr && bh.numBeamsCBA[slot] >= nBM) + if ((earlyStopping == 1) && (bh.numBeamsCBA != nullptr && bh.numBeamsCBA[slot] == nBM) || (earlyStopping != 1) && bh.batchDones[slot]) { bh.batchDones[slot] = true; @@ -661,7 +631,7 @@ void beamSearchKernelLauncher( sync_check_cuda_error(stream); int nThread = min(roundUp(nBMIn * nBMOut * 2, 32), 1024); - addCumLogProbs<<<nBS, nThread, 0, stream>>>(pStage1LogProbs, pStage1Ids, bh.cumLogProbs, bh.finished, bh.endIds, + addCumLogProbs<<<nBS, nThread, 0, stream>>>(pStage1LogProbs, bh.cumLogProbs, bh.finished, bh.endIds, bh.diversityRates, bh.batchSlots, nBS, nBMIn, nBMOut, nBM); sync_check_cuda_error(stream); diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.cu b/cpp/tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.cu index cf0ad7040f59..5be8b1c2ff78 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.cu +++ b/cpp/tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.cu @@ -16,7 +16,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/reduceKernelUtils.cuh" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.h" #include "tensorrt_llm/kernels/quantization.cuh" #include <cooperative_groups.h> @@ -817,7 +816,7 @@ void dispatch_dtype(MiniMaxReduceRMSParams const& params) bool use_float4 = (params.allreduce_in_k != nullptr) && (params.hidden_dim * params.nranks == 6144) && (params.hidden_dim_k * params.nranks == 1024); - if (params.dtype == tensorrt_llm::DataType::kHALF) + if (params.dtype == nvinfer1::DataType::kHALF) { if (use_float4) { @@ -828,7 +827,7 @@ void dispatch_dtype(MiniMaxReduceRMSParams const& params) minimax_reduce_rms_kernel_launcher<half, NRanks>(params); } } - else if (params.dtype == tensorrt_llm::DataType::kBF16) + else if (params.dtype == nvinfer1::DataType::kBF16) { if (use_float4) { @@ -839,7 +838,7 @@ void dispatch_dtype(MiniMaxReduceRMSParams const& params) minimax_reduce_rms_kernel_launcher<__nv_bfloat16, NRanks>(params); } } - else if (params.dtype == tensorrt_llm::DataType::kFLOAT) + else if (params.dtype == nvinfer1::DataType::kFLOAT) { if (use_float4) { diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.h b/cpp/tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.h index bf5775f96de9..b0cfd0ca074c 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.h +++ b/cpp/tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.h @@ -15,7 +15,7 @@ */ #pragma once #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_bf16.h> #include <cuda_fp16.h> @@ -59,7 +59,7 @@ struct MiniMaxReduceRMSParams { int nranks{}; int rank{}; - tensorrt_llm::DataType dtype; + nvinfer1::DataType dtype; int size_q{}; // numel of Q (num_token * head_dim_q) int hidden_dim{}; // head_dim_q int size_k{}; // numel of K (num_token * head_dim_k) diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu b/cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu index d9fbc9da0a0c..5a3edda04a70 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu +++ b/cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.cu @@ -16,7 +16,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/reduceKernelUtils.cuh" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.h" #include "tensorrt_llm/kernels/quantization.cuh" #include <cooperative_groups.h> @@ -796,15 +795,15 @@ void allreduce_fusion_op(AllReduceFusionParams const& params) } #define DISPATCH_DTYPE(NRanks) \ - if (params.dtype == tensorrt_llm::DataType::kHALF) \ + if (params.dtype == nvinfer1::DataType::kHALF) \ { \ DISPATCH_PATTERN(half, NRanks); \ } \ - else if (params.dtype == tensorrt_llm::DataType::kBF16) \ + else if (params.dtype == nvinfer1::DataType::kBF16) \ { \ DISPATCH_PATTERN(__nv_bfloat16, NRanks); \ } \ - else if (params.dtype == tensorrt_llm::DataType::kFLOAT) \ + else if (params.dtype == nvinfer1::DataType::kFLOAT) \ { \ DISPATCH_PATTERN(float, NRanks); \ } \ diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.h b/cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.h index 769776273ef6..6d2074a6589e 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.h +++ b/cpp/tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.h @@ -16,7 +16,7 @@ #pragma once #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_bf16.h> #include <cuda_fp16.h> @@ -124,7 +124,7 @@ struct AllReduceFusionParams { int nranks; int rank; - tensorrt_llm::DataType dtype; + nvinfer1::DataType dtype; int size; int hidden_dim; void** workspace; diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.cu b/cpp/tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.cu index 09a742494a9a..f1d5c08bda6b 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.cu +++ b/cpp/tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.cu @@ -21,7 +21,6 @@ #include "tensorrt_llm/common/customAllReduceUtils.h" #include "tensorrt_llm/common/dataType.h" #include "tensorrt_llm/common/envUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.h" #include <cooperative_groups.h> #include <tuple> @@ -1358,7 +1357,7 @@ std::vector<size_t> splitNumber(size_t number) } LowPrecisionAllReduceParams LowPrecisionAllReduceParams::deserialize( - size_t tpSize, size_t tpRank, tensorrt_llm::DataType dataType, int token_num, int hidden_size) + size_t tpSize, size_t tpRank, nvinfer1::DataType dataType, int token_num, int hidden_size) { // Get appropriate static buffer @@ -1402,7 +1401,7 @@ LowPrecisionAllReduceParams LowPrecisionAllReduceParams::deserialize( } LowPrecisionAllReduceParams LowPrecisionAllReduceParams::deserialize_hier( - size_t tpSize, size_t tpRank, tensorrt_llm::DataType dataType, int token_num, int hidden_size) + size_t tpSize, size_t tpRank, nvinfer1::DataType dataType, int token_num, int hidden_size) { // Get appropriate static buffer @@ -1617,7 +1616,7 @@ int32_t max_workspace_size_lowprecision(int32_t tp_size) } void customLowPrecisionAllReduce( - kernels::LowPrecisionAllReduceParams& params, tensorrt_llm::DataType dataType, cudaStream_t stream) + kernels::LowPrecisionAllReduceParams& params, nvinfer1::DataType dataType, cudaStream_t stream) { TLLM_CHECK_WITH_INFO(lowPrecisionConfigurationSupported(params.ranks_per_node, params.elts_total), "Low Precision Custom all-reduce configuration unsupported"); @@ -1626,10 +1625,10 @@ void customLowPrecisionAllReduce( switch (dataType) { - case tensorrt_llm::DataType::kFLOAT: lowPrecisionAllReduceDispatchType<float>(params, stream); break; - case tensorrt_llm::DataType::kHALF: lowPrecisionAllReduceDispatchType<half>(params, stream); break; + case nvinfer1::DataType::kFLOAT: lowPrecisionAllReduceDispatchType<float>(params, stream); break; + case nvinfer1::DataType::kHALF: lowPrecisionAllReduceDispatchType<half>(params, stream); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: lowPrecisionAllReduceDispatchType<__nv_bfloat16>(params, stream); break; + case nvinfer1::DataType::kBF16: lowPrecisionAllReduceDispatchType<__nv_bfloat16>(params, stream); break; #endif default: TLLM_THROW("Unsupported dataType for customAllReduce"); } diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.h b/cpp/tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.h index 62d19039cc4e..5fc87ef1a523 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.h +++ b/cpp/tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.h @@ -19,8 +19,8 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/customAllReduceKernels.h" +#include <NvInferRuntime.h> #include <cuda_bf16.h> #include <cuda_fp16.h> #include <vector> @@ -111,15 +111,15 @@ struct LowPrecisionAllReduceParams uint64_t* ag_notify_peer_inside_numa_flags[LP_ALLREDUCE_MAX_BLOCKS * 4]; // 3*flags , 3 is other rank inside numa static LowPrecisionAllReduceParams deserialize( - size_t tpSize, size_t tpRank, tensorrt_llm::DataType dataType, int token_num, int hidden_size); + size_t tpSize, size_t tpRank, nvinfer1::DataType dataType, int token_num, int hidden_size); static LowPrecisionAllReduceParams deserialize_hier( - size_t tpSize, size_t tpRank, tensorrt_llm::DataType dataType, int token_num, int hidden_size); + size_t tpSize, size_t tpRank, nvinfer1::DataType dataType, int token_num, int hidden_size); }; bool lowPrecisionConfigurationSupported(size_t msg_size, size_t n_ranks); void customLowPrecisionAllReduce( - kernels::LowPrecisionAllReduceParams& params, tensorrt_llm::DataType dataType, cudaStream_t stream); + kernels::LowPrecisionAllReduceParams& params, nvinfer1::DataType dataType, cudaStream_t stream); int32_t max_workspace_size_lowprecision(int32_t tp_size); } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/mnnvlAllreduceKernels.cu b/cpp/tensorrt_llm/kernels/communicationKernels/mnnvlAllreduceKernels.cu index a8ff3dd431a8..eb44f1638a19 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/mnnvlAllreduceKernels.cu +++ b/cpp/tensorrt_llm/kernels/communicationKernels/mnnvlAllreduceKernels.cu @@ -33,7 +33,6 @@ #include "tensorrt_llm/common/lamportUtils.cuh" #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/reduceKernelUtils.cuh" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/quantization.cuh" TRTLLM_NAMESPACE_BEGIN @@ -857,9 +856,9 @@ void oneshotAllreduceFusionOp(AllReduceFusionParams const& params) }; #undef LAUNCH_ALLREDUCE_KERNEL #undef DISPATCH_ALLREDUCE_PATTERN - bool launched = (params.dType == tensorrt_llm::DataType::kBF16 && dispatchImpl((__nv_bfloat16*) nullptr)) - || (params.dType == tensorrt_llm::DataType::kFLOAT && dispatchImpl((float*) nullptr)) - || (params.dType == tensorrt_llm::DataType::kHALF && dispatchImpl((__nv_half*) nullptr)); + bool launched = (params.dType == nvinfer1::DataType::kBF16 && dispatchImpl((__nv_bfloat16*) nullptr)) + || (params.dType == nvinfer1::DataType::kFLOAT && dispatchImpl((float*) nullptr)) + || (params.dType == nvinfer1::DataType::kHALF && dispatchImpl((__nv_half*) nullptr)); if (!launched) { TLLM_CHECK_WITH_INFO(false, "Failed to dispatch MNNVL AllReduceOneShot kernel."); @@ -1247,9 +1246,9 @@ void twoshotAllreduceFusionOp(AllReduceFusionParams const& params) #undef LAUNCH_ALLREDUCE_KERNEL - bool launched = (params.dType == tensorrt_llm::DataType::kFLOAT && dispatchAR((float*) nullptr)) - || (params.dType == tensorrt_llm::DataType::kBF16 && dispatchAR((__nv_bfloat16*) nullptr)) - || (params.dType == tensorrt_llm::DataType::kHALF && dispatchAR((__nv_half*) nullptr)); + bool launched = (params.dType == nvinfer1::DataType::kFLOAT && dispatchAR((float*) nullptr)) + || (params.dType == nvinfer1::DataType::kBF16 && dispatchAR((__nv_bfloat16*) nullptr)) + || (params.dType == nvinfer1::DataType::kHALF && dispatchAR((__nv_half*) nullptr)); if (!launched) { TLLM_CHECK_WITH_INFO(false, "[MNNVL AllReduceTwoShot] Failed to dispatch twoshotAllreduce kernel."); @@ -1389,9 +1388,9 @@ void twoshotAllreduceFusionOp(AllReduceFusionParams const& params) return true; }; - launched = (params.dType == tensorrt_llm::DataType::kFLOAT && dispatchRN((float*) nullptr)) - || (params.dType == tensorrt_llm::DataType::kBF16 && dispatchRN((__nv_bfloat16*) nullptr)) - || (params.dType == tensorrt_llm::DataType::kHALF && dispatchRN((__nv_half*) nullptr)); + launched = (params.dType == nvinfer1::DataType::kFLOAT && dispatchRN((float*) nullptr)) + || (params.dType == nvinfer1::DataType::kBF16 && dispatchRN((__nv_bfloat16*) nullptr)) + || (params.dType == nvinfer1::DataType::kHALF && dispatchRN((__nv_half*) nullptr)); if (!launched) { TLLM_CHECK_WITH_INFO(false, "[MNNVL AllReduceTwoShot] Failed to dispatch rmsnorm lamport kernel."); diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/mnnvlAllreduceKernels.h b/cpp/tensorrt_llm/kernels/communicationKernels/mnnvlAllreduceKernels.h index f2006f52240c..2a228e815b8d 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/mnnvlAllreduceKernels.h +++ b/cpp/tensorrt_llm/kernels/communicationKernels/mnnvlAllreduceKernels.h @@ -17,8 +17,8 @@ #define TRTLLM_MNNVL_ALLREDUCE_KERNELS_H #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.h" +#include <NvInferRuntime.h> #include <cstdint> TRTLLM_NAMESPACE_BEGIN @@ -39,16 +39,16 @@ struct AllReduceFusionParams //! \name Environmental and Auxiliary Data //! @{ - int nRanks; //!< Total number of participating ranks in the AllReduce operation - int rank; //!< Current rank ID - tensorrt_llm::DataType dType; //!< Data type of the tensors (e.g., FP16, BF16, FP32) - int numTokens; //!< Number of tokens in the input tensor - int tokenDim; //!< Hidden Dimension - void** bufferPtrsDev; //!< Unicast Device pointers to communication buffers for each rank - void* bufferPtrLocal; //!< Local buffer pointer for temporary storage (i.e., bufferPtrsDev[rank]) - void* multicastPtr; //!< Multicast buffer pointer. - uint32_t* bufferFlags; //!< Synchronization flags for coordinating communication phases - bool rmsNormFusion; //!< Whether to fuse RMS normalization with the AllReduce operation + int nRanks; //!< Total number of participating ranks in the AllReduce operation + int rank; //!< Current rank ID + nvinfer1::DataType dType; //!< Data type of the tensors (e.g., FP16, BF16, FP32) + int numTokens; //!< Number of tokens in the input tensor + int tokenDim; //!< Hidden Dimension + void** bufferPtrsDev; //!< Unicast Device pointers to communication buffers for each rank + void* bufferPtrLocal; //!< Local buffer pointer for temporary storage (i.e., bufferPtrsDev[rank]) + void* multicastPtr; //!< Multicast buffer pointer. + uint32_t* bufferFlags; //!< Synchronization flags for coordinating communication phases + bool rmsNormFusion; //!< Whether to fuse RMS normalization with the AllReduce operation ar_fusion::AllReduceFusionPattern pattern = ar_fusion::AllReduceFusionPattern::kAllReduce; //!< Fused epilogue pattern diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.cu b/cpp/tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.cu index 5b0da0d0e80c..306d42677e2f 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.cu +++ b/cpp/tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.cu @@ -16,7 +16,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/reduceKernelUtils.cuh" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.h" #include "tensorrt_llm/kernels/quantization.cuh" #include <cooperative_groups.h> @@ -443,11 +442,11 @@ void moereduction_allreduce_fusion_op(MoeReductionAllReduceFusionParams const& p #define MOE_DISPATCH1(DTYPE, NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT) \ return moereduction_allreduce_fusion_kernel_launcher<DTYPE, NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT>(params); #define MOE_DISPATCH0(NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT) \ - if (params.nranks == NRANKS && params.dtype == tensorrt_llm::DataType::kHALF) \ + if (params.nranks == NRANKS && params.dtype == nvinfer1::DataType::kHALF) \ { \ MOE_DISPATCH1(half, NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT); \ } \ - else if (params.nranks == NRANKS && params.dtype == tensorrt_llm::DataType::kBF16) \ + else if (params.nranks == NRANKS && params.dtype == nvinfer1::DataType::kBF16) \ { \ MOE_DISPATCH1(__nv_bfloat16, NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT); \ } @@ -557,20 +556,11 @@ __global__ void moefinalize_allreduce_fusion_kernel_oneshot_lamport(MoeFinalizeA } // * MoE finalize - // Accumulate the top-k weighted expert sum and the shared-expert add in - // fp32 (local `facc`), rounding to DType (bf16/fp16) only once when packing - // into `accumulator` for the 128-bit Lamport all-reduce store below. - // Accumulating directly in DType here rounds after every one of the top_k - // terms; across the many routed MoE layers this rounding bias is large - // enough to visibly degrade the routed output, and with attention-DP - // disabled + MTP speculative decoding it drifts the target hidden states - // enough to lower the acceptance length. The non-deferred in-kernel - // finalize (do_finalize=true) already accumulates in fp32; match it here. - float facc[kElemsPerAccess]; + ACC_TYPE accumulator; #pragma unroll for (int i = 0; i < kElemsPerAccess; ++i) { - facc[i] = 0.f; + accumulator.unpacked[i] = static_cast<DType>(0); } for (int k = 0; k < top_k; k++) @@ -592,15 +582,17 @@ __global__ void moefinalize_allreduce_fusion_kernel_oneshot_lamport(MoeFinalizeA permuted_data.packed = reinterpret_cast<float4 const*>(params.allreduce_in)[thread_offset_across_token / kElemsPerAccess]; - // * acc += scale(data) (fp32 accumulation) + // * acc += scale(data) #pragma unroll for (int i = 0; i < kElemsPerAccess; ++i) { - facc[i] += static_cast<float>(permuted_data.unpacked[i]) * block_scale; + // assume computation is done in ScaleType + accumulator.unpacked[i] + += static_cast<DType>((static_cast<float>(permuted_data.unpacked[i]) * block_scale)); } } - // * Add shared expert output (fp32 accumulation) + // * Add shared expert output if (params.shared_expert_output) { // * Load shared expert output @@ -611,18 +603,10 @@ __global__ void moefinalize_allreduce_fusion_kernel_oneshot_lamport(MoeFinalizeA #pragma unroll for (int i = 0; i < kElemsPerAccess; ++i) { - facc[i] += static_cast<float>(shared_expert_output.unpacked[i]); + accumulator.unpacked[i] += shared_expert_output.unpacked[i]; } } - // Round the fp32 accumulator to DType once, packed for the Lamport AR store. - ACC_TYPE accumulator; -#pragma unroll - for (int i = 0; i < kElemsPerAccess; ++i) - { - accumulator.unpacked[i] = static_cast<DType>(facc[i]); - } - // * AR Store int access_id = token_id * params.hidden_dim / kElemsPerAccess + access_id_in_token; int idx = access_id; @@ -743,13 +727,13 @@ void moefinalize_allreduce_fusion_op(MoeFinalizeAllReduceFusionParams const& par #define MOE_FINALIZE_DISPATCH1(DTYPE, NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT) \ return moefinalize_allreduce_fusion_kernel_launcher<DTYPE, NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT>(params); #define MOE_FINALIZE_DISPATCH0(NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT) \ - if (params.nranks == NRANKS && params.dtype == tensorrt_llm::DataType::kHALF \ - && params.scale_dtype == tensorrt_llm::DataType::kHALF) \ + if (params.nranks == NRANKS && params.dtype == nvinfer1::DataType::kHALF \ + && params.scale_dtype == nvinfer1::DataType::kHALF) \ { \ MOE_FINALIZE_DISPATCH1(half, NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT); \ } \ - else if (params.nranks == NRANKS && params.dtype == tensorrt_llm::DataType::kBF16 \ - && params.scale_dtype == tensorrt_llm::DataType::kBF16) \ + else if (params.nranks == NRANKS && params.dtype == nvinfer1::DataType::kBF16 \ + && params.scale_dtype == nvinfer1::DataType::kBF16) \ { \ MOE_FINALIZE_DISPATCH1(__nv_bfloat16, NRANKS, RESIDUAL_OUT, NORM_OUT, QUANT_OUT); \ } diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.h b/cpp/tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.h index e526a70268b3..556dd4e5cd24 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.h +++ b/cpp/tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.h @@ -16,7 +16,7 @@ #pragma once #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_bf16.h> #include <cuda_fp16.h> @@ -44,7 +44,7 @@ struct AllReduceFusionParams { int nranks; int rank; - tensorrt_llm::DataType dtype; + nvinfer1::DataType dtype; // size = token_num * hidden_dim int size; int hidden_dim; @@ -94,7 +94,7 @@ struct MoeFinalizeAllReduceFusionParams : public AllReduceFusionParams // Refer to kernel implementation on layout of those params // number of active experts on current device int top_k; - tensorrt_llm::DataType scale_dtype; + nvinfer1::DataType scale_dtype; // [num_tokens, top_k] void* expert_scale_factor = nullptr; void* shared_expert_output = nullptr; diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.cu b/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.cu index 74e40dbb2b81..0f2d453c363a 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.cu +++ b/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.cu @@ -17,7 +17,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/envUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/common/vec_dtypes.cuh" #include "tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h" #include "tensorrt_llm/kernels/quantization.cuh" @@ -130,25 +129,25 @@ using tensorrt_llm::common::launchWithPdlWhenEnabled; #define SWITCH_DTYPE(dtype, TYPE, ...) \ switch (dtype) \ { \ - case tensorrt_llm::DataType::kHALF: \ + case nvinfer1::DataType::kHALF: \ { \ using TYPE = half; \ __VA_ARGS__; \ break; \ } \ - case tensorrt_llm::DataType::kBF16: \ + case nvinfer1::DataType::kBF16: \ { \ using TYPE = __nv_bfloat16; \ __VA_ARGS__; \ break; \ } \ - case tensorrt_llm::DataType::kFLOAT: \ + case nvinfer1::DataType::kFLOAT: \ { \ using TYPE = float; \ __VA_ARGS__; \ break; \ } \ - case tensorrt_llm::DataType::kFP8: \ + case nvinfer1::DataType::kFP8: \ { \ using TYPE = __nv_fp8_e4m3; \ __VA_ARGS__; \ @@ -393,7 +392,7 @@ __global__ void moeA2APrepareDispatchKernel( // Dispatch Kernels // ============================================================================ -template <typename ThreadingPolicy, int TOP_K, bool ENABLE_EPLB, bool ENABLE_RANK_MASK> +template <typename ThreadingPolicy, int TOP_K, bool ENABLE_EPLB> __global__ void moeA2ADispatchKernel(int32_t const* token_selected_experts, // [local_num_tokens, TOP_K] const DispatchKernelPointers ptrs, // Struct containing all kernel pointers int num_payloads, // Number of payloads @@ -441,17 +440,15 @@ __global__ void moeA2ADispatchKernel(int32_t const* token_selected_experts, // [ // Supports the non-divisible case where num_experts % ep_size != 0. int target_rank = compute_target_rank_id(expert_id, ep_base, ep_remainder); + // Skip duplicates AND dead ranks: both produce the same -1 sentinel that combine + // checks via topk_send_indices[k] < 0. A token whose only target is dead is dropped + // from this collective; higher-layer logic (EPLB redistribution) is responsible + // for re-routing such tokens on subsequent iterations. int const mask_word = target_rank >> 6; uint64_t const mask_bit = 1ULL << (target_rank & 63); - bool const target_already_copied = (already_copied[mask_word] & mask_bit) != 0; - bool skip_target = target_already_copied; - if constexpr (ENABLE_RANK_MASK) - { - // This is a fail-closed safety guard until post-commit routing is enforced end to end. - // A masked route is not valid model output; the failed execution epoch must be discarded. - skip_target = skip_target || !is_rank_active(ptrs.active_rank_mask, target_rank); - } - if (skip_target) + bool const target_already_copied = already_copied[mask_word] & mask_bit; + bool const target_dead = !is_rank_active(ptrs.active_rank_mask, target_rank); + if (target_already_copied || target_dead) { if (thread_idx == 0) { @@ -535,11 +532,8 @@ __global__ void moeA2ADispatchKernel(int32_t const* token_selected_experts, // [ #pragma unroll 1 // No unroll as one iter is typically enough for (int target_rank = lane_id; target_rank < ep_size; target_rank += warpSize) { - if constexpr (ENABLE_RANK_MASK) - { - if (!is_rank_active(ptrs.active_rank_mask, target_rank)) - continue; - } + if (!is_rank_active(ptrs.active_rank_mask, target_rank)) + continue; int send_count = ptrs.send_counters[target_rank]; ptrs.recv_counters[target_rank][rank_id] = send_count; } @@ -551,11 +545,8 @@ __global__ void moeA2ADispatchKernel(int32_t const* token_selected_experts, // [ #pragma unroll 1 for (int target_rank = 0; target_rank < ep_size; ++target_rank) { - if constexpr (ENABLE_RANK_MASK) - { - if (!is_rank_active(ptrs.active_rank_mask, target_rank)) - continue; - } + if (!is_rank_active(ptrs.active_rank_mask, target_rank)) + continue; int* target_stats = ptrs.eplb_gathered_stats[target_rank]; for (int expert_id = lane_id; expert_id < eplb_stats_num_experts; expert_id += warpSize) { @@ -579,11 +570,8 @@ __global__ void moeA2ADispatchKernel(int32_t const* token_selected_experts, // [ #pragma unroll 1 // No unroll as one iter is typically enough for (int target_rank = lane_id; target_rank < ep_size; target_rank += warpSize) { - if constexpr (ENABLE_RANK_MASK) - { - if (!is_rank_active(ptrs.active_rank_mask, target_rank)) - continue; - } + if (!is_rank_active(ptrs.active_rank_mask, target_rank)) + continue; uint32_t* flag_addr = &ptrs.completion_flags[target_rank][rank_id]; asm volatile("st.relaxed.sys.u32 [%0], %1;" ::"l"(flag_addr), "r"(expected_value)); @@ -598,11 +586,8 @@ __global__ void moeA2ADispatchKernel(int32_t const* token_selected_experts, // [ #pragma unroll 1 // No unroll for (int peer_rank = lane_id; peer_rank < ep_size; peer_rank += warpSize) { - if constexpr (ENABLE_RANK_MASK) - { - if (!is_rank_active(ptrs.active_rank_mask, peer_rank)) - continue; - } + if (!is_rank_active(ptrs.active_rank_mask, peer_rank)) + continue; bool flag_set = false; auto s = clock64(); do @@ -651,11 +636,10 @@ void moe_a2a_dispatch_launch(MoeA2ADispatchParams const& params) TLLM_CHECK(params.ep_rank >= 0 && params.ep_rank < params.ep_size); TLLM_CHECK(params.local_num_tokens >= 0); TLLM_CHECK(params.num_payloads > 0 && params.num_payloads <= kMaxPayloads); - if (params.enable_rank_mask) - { - TLLM_CHECK_WITH_INFO((params.active_rank_mask[params.ep_rank >> 6] >> (params.ep_rank & 63)) & 1ULL, - "active_rank_mask must mark the local ep_rank (%d) as active", params.ep_rank); - } + // The local rank must always be marked active in its own view of the mask; + // otherwise the kernel itself would be running on a "dead" rank. + TLLM_CHECK_WITH_INFO((params.active_rank_mask[params.ep_rank >> 6] >> (params.ep_rank & 63)) & 1ULL, + "active_rank_mask must mark the local ep_rank (%d) as active", params.ep_rank); // Prepare kernel pointers struct DispatchKernelPointers kernel_ptrs = {}; @@ -709,15 +693,12 @@ void moe_a2a_dispatch_launch(MoeA2ADispatchParams const& params) grid_size = 1; } int shared_bytes = 2 * params.top_k * (int) sizeof(int); - SWITCH_BOOL(params.enable_rank_mask, ENABLE_RANK_MASK, {SWITCH_BOOL(params.enable_eplb, EPLB_STATS, { - SWITCH_TOP_K(params.top_k, TOP_K, { - auto kernel_fn = moeA2ADispatchKernel<BlockPolicy, TOP_K, EPLB_STATS, ENABLE_RANK_MASK>; - launchWithPdlWhenEnabled("moeA2ADispatchKernel", kernel_fn, grid_size, kBlockSize, shared_bytes, - params.stream, params.token_selected_experts, kernel_ptrs, params.num_payloads, - params.max_tokens_per_rank, params.local_num_tokens, params.ep_rank, params.ep_size, params.num_experts, - params.eplb_stats_num_experts); - }); - })}) + SWITCH_BOOL(params.enable_eplb, EPLB_STATS, SWITCH_TOP_K(params.top_k, TOP_K, { + auto kernel_fn = moeA2ADispatchKernel<BlockPolicy, TOP_K, EPLB_STATS>; + launchWithPdlWhenEnabled("moeA2ADispatchKernel", kernel_fn, grid_size, kBlockSize, shared_bytes, params.stream, + params.token_selected_experts, kernel_ptrs, params.num_payloads, params.max_tokens_per_rank, + params.local_num_tokens, params.ep_rank, params.ep_size, params.num_experts, params.eplb_stats_num_experts); + })) } // ============================================================================ @@ -760,7 +741,7 @@ __device__ void vectorized_combine_impl(T* dst_typed_base, int size_per_token, i { int target_rank = ptrs.topk_target_ranks[local_token_idx * TOP_K + k]; int dst_idx = ptrs.topk_send_indices[local_token_idx * TOP_K + k]; - if (dst_idx < 0) + if (dst_idx < 0 || !is_rank_active(ptrs.active_rank_mask, target_rank)) { acc[k].fill(0.0f); continue; @@ -785,8 +766,12 @@ __device__ void vectorized_combine_impl(T* dst_typed_base, int size_per_token, i #pragma unroll for (int k = 0; k < TOP_K; ++k) { - if (ptrs.topk_send_indices[local_token_idx * TOP_K + k] < 0) + int target_rank = ptrs.topk_target_ranks[local_token_idx * TOP_K + k]; + int dst_idx = ptrs.topk_send_indices[local_token_idx * TOP_K + k]; + if (dst_idx < 0 || !is_rank_active(ptrs.active_rank_mask, target_rank)) + { continue; // acc[k] already holds 0.0f from fill() above + } #pragma unroll for (int j = elems_per_vec - 1; j >= 0; --j) acc[k][j] = static_cast<float>(reinterpret_cast<InT const*>(&acc[k])[j]); @@ -1173,7 +1158,7 @@ __global__ void moeA2APrepareCombineKernel(uint8_t* recv_buffer_bytes, void cons // Generic Combine Kernel Implementation (Templated by data type) // ============================================================================ -template <typename T, typename ThreadingPolicy, int TOP_K, bool ENABLE_RANK_MASK> +template <typename T, typename ThreadingPolicy, int TOP_K> __global__ void moeA2ACombineKernel( const CombineKernelPointers ptrs, // Combine-specific struct, src_data_ptrs[0] is output int max_tokens_per_rank, int elements_per_token, int local_num_tokens, int rank_id, int ep_size, @@ -1218,11 +1203,8 @@ __global__ void moeA2ACombineKernel( #pragma unroll 1 // No unroll for (int peer_rank = lane_id; peer_rank < ep_size; peer_rank += warpSize) { - if constexpr (ENABLE_RANK_MASK) - { - if (!is_rank_active(ptrs.active_rank_mask, peer_rank)) - continue; - } + if (!is_rank_active(ptrs.active_rank_mask, peer_rank)) + continue; uint32_t* flag_addr = &ptrs.completion_flags[peer_rank][rank_id]; asm volatile("st.relaxed.sys.u32 [%0], %1;" ::"l"(flag_addr), "r"(expected_value)); #if ENABLE_DEBUG_PRINT @@ -1237,11 +1219,8 @@ __global__ void moeA2ACombineKernel( #pragma unroll 1 // No unroll for (int peer_rank = lane_id; peer_rank < ep_size; peer_rank += warpSize) { - if constexpr (ENABLE_RANK_MASK) - { - if (!is_rank_active(ptrs.active_rank_mask, peer_rank)) - continue; - } + if (!is_rank_active(ptrs.active_rank_mask, peer_rank)) + continue; bool flag_set = false; auto s = clock64(); do @@ -1348,11 +1327,10 @@ void moe_a2a_combine_launch(MoeA2ACombineParams const& params) TLLM_CHECK(params.ep_rank >= 0 && params.ep_rank < params.ep_size); TLLM_CHECK(params.local_num_tokens >= 0); TLLM_CHECK(params.elements_per_token > 0); - if (params.enable_rank_mask) - { - TLLM_CHECK_WITH_INFO((params.active_rank_mask[params.ep_rank >> 6] >> (params.ep_rank & 63)) & 1ULL, - "active_rank_mask must mark the local ep_rank (%d) as active", params.ep_rank); - } + // The local rank must always be marked active in its own view of the mask; + // otherwise the kernel itself would be running on a "dead" rank. + TLLM_CHECK_WITH_INFO((params.active_rank_mask[params.ep_rank >> 6] >> (params.ep_rank & 63)) & 1ULL, + "active_rank_mask must mark the local ep_rank (%d) as active", params.ep_rank); // Configure kernel launch (one block per token). int const kBlockSize = tensorrt_llm::common::getEnvMoeA2ACombineBlockSize(); @@ -1404,19 +1382,17 @@ void moe_a2a_combine_launch(MoeA2ACombineParams const& params) // When use_low_precision is set the recv buffers contain FP8 data regardless of params.dtype, // so dispatch the FP8 accumulation kernel in that case. - auto const effective_dtype = params.use_low_precision ? tensorrt_llm::DataType::kFP8 : params.dtype; + auto const effective_dtype = params.use_low_precision ? nvinfer1::DataType::kFP8 : params.dtype; // Launch appropriate kernel with compact macros - SWITCH_BOOL(params.enable_rank_mask, ENABLE_RANK_MASK, { - SWITCH_DTYPE(effective_dtype, TKernelType, { - SWITCH_TOP_K(params.top_k, TOP_K, { - auto kernel_fn = moeA2ACombineKernel<TKernelType, BlockPolicy, TOP_K, ENABLE_RANK_MASK>; - launchWithPdlWhenEnabled("moeA2ACombineKernel", kernel_fn, grid, kBlockSize, 0, params.stream, - kernel_ptrs, params.max_tokens_per_rank, params.elements_per_token, params.local_num_tokens, - params.ep_rank, params.ep_size, stride_per_token); - }); + SWITCH_DTYPE(effective_dtype, TKernelType, { + SWITCH_TOP_K(params.top_k, TOP_K, { + auto kernel_fn = moeA2ACombineKernel<TKernelType, BlockPolicy, TOP_K>; + launchWithPdlWhenEnabled("moeA2ACombineKernel", kernel_fn, grid, kBlockSize, 0, params.stream, kernel_ptrs, + params.max_tokens_per_rank, params.elements_per_token, params.local_num_tokens, params.ep_rank, + params.ep_size, stride_per_token); }); - }) + }); } // Kernel to sanitize expert ids for invalid tokens diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h b/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h index 5184878ffc51..138ca92e71a8 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h +++ b/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h @@ -16,7 +16,7 @@ #pragma once #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_bf16.h> #include <cuda_fp16.h> @@ -62,16 +62,17 @@ struct DispatchKernelPointers int* local_token_counter; // Atomic counter for completed tokens // Top-K compact routing info per local token (size: [local_num_tokens, top_k]) - int* topk_target_ranks; // target rank per k, -1 for invalid or duplicate routes - int* topk_send_indices; // dst index per k, -1 for invalid or duplicate routes + int* topk_target_ranks; // target rank per k, -1 for duplicates + int* topk_send_indices; // dst index per k, -1 for duplicates // Optional: Statistics for EPLB int const* eplb_local_stats; // [eplb_stats_num_experts] int* eplb_gathered_stats[kMaxRanks]; // [ep_size, eplb_stats_num_experts] per rank - // Active-rank bitmask: bit i set => rank i participates in this collective. - // Word 0 covers ranks 0..63; word 1 covers ranks 64..127. The masked kernel - // rejects inactive route targets and skips their peer counters, stats, and flags. + // Active-rank bitmask: bit i set => rank i is alive and participates in this collective. + // Word 0 covers ranks 0..63; word 1 covers ranks 64..127. Tokens routed to a masked + // rank are dropped (topk_*[k] = -1); flag writes/waits to/from masked peers are skipped. + // The local rank's own bit must always be set; this is checked at launch time. uint64_t active_rank_mask[kRankMaskWords]; }; @@ -88,11 +89,12 @@ struct CombineKernelPointers uint32_t* flag_val; // The value of the flag for this round (stored on the local rank) // Top-K compact routing info per local token (size: [local_num_tokens, top_k]) - int const* topk_target_ranks; // target rank per k, -1 for invalid or duplicate routes - int const* topk_send_indices; // dst index per k, -1 for invalid or duplicate routes + int const* topk_target_ranks; // target rank per k, -1 for duplicates + int const* topk_send_indices; // dst index per k, -1 for duplicates - // Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Combine skips - // completion flag writes/waits to/from inactive peers. + // Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Combine skips flag + // writes/waits to/from masked peers and also skips per-token accumulation for ranks that + // become inactive between dispatch and combine. uint64_t active_rank_mask[kRankMaskWords]; }; @@ -137,14 +139,9 @@ struct MoeA2ADispatchParams int const* eplb_local_stats; // [eplb_stats_num_experts] int* eplb_gathered_stats[kMaxRanks]; // [ep_size, eplb_stats_num_experts] per rank - // Whether to instantiate a kernel with active-rank checks. - // This is a launch-lifetime mode, independent of future execution-abort handling. - bool enable_rank_mask{false}; - - // Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Used only when - // enable_rank_mask is true; defaults to all-ones for backwards-compatible behavior. - // The mask is copied by value into kernel arguments. Rank-mask mode must reject - // CUDA graph replay until generation-scoped invalidation and recapture are available. + // Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. The launch function + // copies these words into the kernel pointers struct. Defaults to all-ones for + // backwards-compatible "no masking" behavior. uint64_t active_rank_mask[kRankMaskWords] = {~uint64_t{0}, ~uint64_t{0}}; // CUDA stream @@ -174,8 +171,8 @@ struct MoeA2ACombineParams // Output tensor void* output_data; // Output buffer [local_num_tokens, elements_per_token] // Payload information - int elements_per_token; // Number of elements per token - tensorrt_llm::DataType dtype; // Data type of the payload (used for combine kernel dispatch) + int elements_per_token; // Number of elements per token + nvinfer1::DataType dtype; // Data type of the payload (used for combine kernel dispatch) bool use_low_precision; // If true, prepare kernel quantizes payload→FP8; combine kernel accumulates FP8→output dtype @@ -192,14 +189,9 @@ struct MoeA2ACombineParams // rank has signaled the target rank void const* recv_buffers[kMaxRanks]; // Per-rank receive buffers (only for single payload) - // Whether to instantiate a kernel with active-rank checks in peer synchronization. - // This is a launch-lifetime mode, independent of future execution-abort handling. - bool enable_rank_mask{false}; - - // Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Used only when - // enable_rank_mask is true; defaults to all-ones for backwards-compatible behavior. - // The mask is copied by value into kernel arguments. Rank-mask mode must reject - // CUDA graph replay until generation-scoped invalidation and recapture are available. + // Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. The launch function + // copies these words into the kernel pointers struct. Defaults to all-ones for + // backwards-compatible "no masking" behavior. uint64_t active_rank_mask[kRankMaskWords] = {~uint64_t{0}, ~uint64_t{0}}; // CUDA stream diff --git a/cpp/tensorrt_llm/kernels/compressorKernels/compressorKernels.cu b/cpp/tensorrt_llm/kernels/compressorKernels/compressorKernels.cu index 1457efeb3138..66eabae9394c 100644 --- a/cpp/tensorrt_llm/kernels/compressorKernels/compressorKernels.cu +++ b/cpp/tensorrt_llm/kernels/compressorKernels/compressorKernels.cu @@ -64,6 +64,7 @@ #include "tensorrt_llm/kernels/compressorKernels/compressorKernels.h" #include "tensorrt_llm/common/assert.h" +#include <algorithm> #include <cmath> #include <cstdint> #include <cuda_bf16.h> @@ -212,8 +213,8 @@ enum class CacheScaleType // ============================================================================ // Decode Kernel: pagedKvCompressKernel // -// Template: <HEAD_DIM, KV_SCORE_ELEM_BYTES, STATE_ELEM_BYTES, COMPRESS_RATIO, NEXT_N, NUM_RED_WARPS> -// NEXT_N: number of new tokens per sequence in this decode step (1-8) +// Template: <HEAD_DIM, KV_SCORE_ELEM_BYTES, STATE_ELEM_BYTES, NEXT_N> +// NEXT_N: number of new tokens per sequence in this decode step (1-4) // // Grid: (batch_size) — one block per batch element // Block: (NTHRD) where NTHRD = HEAD_DIM / VEC (>= 32 threads) @@ -622,10 +623,9 @@ __global__ void pagedKvCompressKernel(void const* __restrict__ kv_score_raw, flo // KV_EB — kv_score element bytes in {2 (bf16), 4 (fp32)} // STATE_EB — paged state element bytes in {2 (bf16), 4 (fp32)} // CR — COMPRESS_RATIO in {4, 128} -// NN — NEXT_N (new tokens / decode step) in {1..8} -// NRW — NUM_RED_WARPS — 4 when CR=128 and NN<=4 (multi-warp Phase 3 -// reduction hides DRAM latency for the heavier R=128 chunk); -// 1 when CR=4 or when CR=128 and NN>=5. +// NN — NEXT_N (new tokens / decode step) in {1..4} +// NRW — NUM_RED_WARPS — 4 only when CR=128 (multi-warp Phase 3 reduce +// hides DRAM latency for the heavier R=128 chunk); 1 otherwise. // // Multi-warp SMEM budget (per block): 3 * NRW * ELEM_PER_BLOCK * sizeof(float). // HD=128: ELEM_PER_BLOCK=128 → 6 KB @@ -634,30 +634,22 @@ __global__ void pagedKvCompressKernel(void const* __restrict__ kv_score_raw, flo // ============================================================================ // Per-axis fan-outs (used to keep the master list compact). -#define FOREACH_DECODE_NN_1_4(F, HD, KV, ST, CR, NRW) \ +#define FOREACH_DECODE_NN(F, HD, KV, ST, CR, NRW) \ F(HD, KV, ST, CR, 1, NRW) F(HD, KV, ST, CR, 2, NRW) F(HD, KV, ST, CR, 3, NRW) F(HD, KV, ST, CR, 4, NRW) -#define FOREACH_DECODE_NN_5_8(F, HD, KV, ST, CR, NRW) \ - F(HD, KV, ST, CR, 5, NRW) F(HD, KV, ST, CR, 6, NRW) F(HD, KV, ST, CR, 7, NRW) F(HD, KV, ST, CR, 8, NRW) -#define FOREACH_DECODE_DTYPE_1_4(F, HD, CR, NRW) \ - FOREACH_DECODE_NN_1_4(F, HD, 2, 2, CR, NRW) \ - FOREACH_DECODE_NN_1_4(F, HD, 2, 4, CR, NRW) \ - FOREACH_DECODE_NN_1_4(F, HD, 4, 2, CR, NRW) FOREACH_DECODE_NN_1_4(F, HD, 4, 4, CR, NRW) -#define FOREACH_DECODE_DTYPE_5_8(F, HD, CR, NRW) \ - FOREACH_DECODE_NN_5_8(F, HD, 2, 2, CR, NRW) \ - FOREACH_DECODE_NN_5_8(F, HD, 2, 4, CR, NRW) \ - FOREACH_DECODE_NN_5_8(F, HD, 4, 2, CR, NRW) FOREACH_DECODE_NN_5_8(F, HD, 4, 4, CR, NRW) -#define FOREACH_DECODE_DTYPE_1_8(F, HD, CR, NRW) \ - FOREACH_DECODE_DTYPE_1_4(F, HD, CR, NRW) FOREACH_DECODE_DTYPE_5_8(F, HD, CR, NRW) +#define FOREACH_DECODE_DTYPE(F, HD, CR, NRW) \ + FOREACH_DECODE_NN(F, HD, 2, 2, CR, NRW) \ + FOREACH_DECODE_NN(F, HD, 2, 4, CR, NRW) \ + FOREACH_DECODE_NN(F, HD, 4, 2, CR, NRW) FOREACH_DECODE_NN(F, HD, 4, 4, CR, NRW) // Master list. Order does not matter; the dispatcher walks linearly. // clang-format off #define FOREACH_DECODE_CONFIG(F) \ - /* CR=4: single-warp for next_n 1..8 (small reduction; multi-warp would over-subscribe). */ \ - FOREACH_DECODE_DTYPE_1_8(F, 128, 4, 1) FOREACH_DECODE_DTYPE_1_8(F, 512, 4, 1) \ - /* CR=128: single-warp fallback for next_n 5..8. */ \ - FOREACH_DECODE_DTYPE_5_8(F, 128, 128, 1) FOREACH_DECODE_DTYPE_5_8(F, 512, 128, 1) \ - /* CR=128: multi-warp fast path for next_n 1..4. */ \ - FOREACH_DECODE_DTYPE_1_4(F, 128, 128, 4) FOREACH_DECODE_DTYPE_1_4(F, 512, 128, 4) + /* CR=4: single-warp only (small reduction; multi-warp would over-subscribe). */ \ + FOREACH_DECODE_DTYPE(F, 128, 4, 1) FOREACH_DECODE_DTYPE(F, 512, 4, 1) \ + /* CR=128: single-warp fallback (covers next_n>4 path which currently isn't reached). */ \ + FOREACH_DECODE_DTYPE(F, 128, 128, 1) FOREACH_DECODE_DTYPE(F, 512, 128, 1) \ + /* CR=128: multi-warp fast path. Used whenever next_n <= 4 (i.e. MTP-3 and below). */ \ + FOREACH_DECODE_DTYPE(F, 128, 128, 4) FOREACH_DECODE_DTYPE(F, 512, 128, 4) // clang-format on // Generate explicit template instantiations. @@ -671,7 +663,7 @@ FOREACH_DECODE_CONFIG(INST_DECODE) // Decode Launch Wrapper // // Dispatches to the correct template instantiation based on head_dim, elem_bytes, -// and next_n (number of new tokens per decode step, in the range 1..8). +// and next_n (number of new tokens per decode step, capped at 4). // Grid is 2D: (batch_size, head_blocks) where head_blocks = NTHRD_BASE / 32. // For HD=512 bf16: head_blocks=2; for HD=128 bf16: head_blocks=1. // ============================================================================ @@ -687,10 +679,6 @@ void pagedKvCompressLaunch(void const* kv_score, float const* ape, void* paged_k TLLM_CHECK_WITH_INFO( (kv_score_elem_bytes == 2 || kv_score_elem_bytes == 4) && (state_elem_bytes == 2 || state_elem_bytes == 4), "pagedKvCompressLaunch only supports bf16/fp32 kv_score and paged state"); - constexpr int kMinNextN = 1; - constexpr int kMaxNextN = 8; - TLLM_CHECK_WITH_INFO(next_n >= kMinNextN && next_n <= kMaxNextN, - "pagedKvCompressLaunch only supports next_n in [1, 8], got %d", next_n); // Compute HEAD_BLOCKS: mirrors the compile-time constant in the kernel. // VEC = max_vec if HEAD_DIM/max_vec >= 32, else HEAD_DIM/32. @@ -705,9 +693,10 @@ void pagedKvCompressLaunch(void const* kv_score, float const* ape, void* paged_k // For large compress_ratio, use 4-warp parallel reduction to cut the serial // softmax loop from COMPRESS_RATIO iterations to COMPRESS_RATIO/4 per warp. - // The multi-warp path supports CR=128, (HD=128 or HD=512), and NEXT_N in - // 1..4. Larger NEXT_N values use a single reduction warp to limit block - // size while still processing every new token exactly. + // Supported configs: CR=128, (HD=128 or HD=512), NEXT_N in 1..4. NEXT_N>2 + // is required for MTP-3 decode (each step accepts up to 4 tokens per request); + // without multi-warp the slow path is a single warp doing 128 serial paged + // loads, which is DRAM-latency-bound (no other warps to hide it). // // smem per block = 3 * MULTI_WARP * ELEM_PER_BLOCK * sizeof(float) // where ELEM_PER_BLOCK = nthreads_inner * vec = HEAD_DIM / HEAD_BLOCKS. @@ -723,11 +712,15 @@ void pagedKvCompressLaunch(void const* kv_score, float const* ape, void* paged_k dim3 grid(batch_size, head_blocks); + // Clamp the runtime next_n into the supported range; configs above 4 fall + // back to the NN=4 instantiation (matches the prior `default:` arm). + int const next_n_dispatch = std::min(next_n, 4); + // Walk FOREACH_DECODE_CONFIG until we find a matching (HD, KV, ST, CR, NN, NRW) // tuple, then launch that instantiation. Any unsupported tuple bails via TLLM_THROW. #define TRY_LAUNCH(HD, KV_EB, STATE_EB, CR, NN, NRW) \ if (head_dim == HD && kv_score_elem_bytes == KV_EB && state_elem_bytes == STATE_EB && compress_ratio == CR \ - && next_n == NN && num_red_warps == NRW) \ + && next_n_dispatch == NN && num_red_warps == NRW) \ { \ pagedKvCompressKernel<HD, KV_EB, STATE_EB, CR, NN, NRW><<<grid, nthreads, smem_bytes, stream>>>(kv_score, ape, \ paged_kv, paged_score, block_table_kv, block_table_score, output, kv_lens, cu_seq_lens, cu_kv_comp, \ @@ -739,15 +732,12 @@ void pagedKvCompressLaunch(void const* kv_score, float const* ape, void* paged_k TLLM_THROW( "pagedKvCompressLaunch: no matching instantiation for HD=%d, kv_eb=%d, state_eb=%d, CR=%d, NN=%d, NRW=%d", - head_dim, kv_score_elem_bytes, state_elem_bytes, compress_ratio, next_n, num_red_warps); + head_dim, kv_score_elem_bytes, state_elem_bytes, compress_ratio, next_n_dispatch, num_red_warps); } #undef FOREACH_DECODE_CONFIG -#undef FOREACH_DECODE_DTYPE_1_8 -#undef FOREACH_DECODE_DTYPE_5_8 -#undef FOREACH_DECODE_DTYPE_1_4 -#undef FOREACH_DECODE_NN_5_8 -#undef FOREACH_DECODE_NN_1_4 +#undef FOREACH_DECODE_DTYPE +#undef FOREACH_DECODE_NN // ============================================================================ // Prefill Kernel: prefillReductionKernel diff --git a/cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h b/cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h index f4ccb43ce96e..68c567105665 100644 --- a/cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h +++ b/cpp/tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h @@ -147,8 +147,6 @@ struct MHARunnerFixedParams bool useSparseMLA = false; // Use sparse attention in trtllm-gen ? bool useTllmGenSparseAttention = false; - // Fuse DSv4 inverse RoPE and FP8 output quantization in trtllm-gen. - bool fusesDsv4InvRopeFp8Quant = false; // Convert to string for debug. std::string convertToStrOutput() @@ -201,7 +199,6 @@ struct MHARunnerFixedParams output += ", sageBlockSizeV = " + std::to_string(sageBlockSizeV); output += ", useSparseMLA = " + std::string(useSparseMLA ? "true" : "false"); output += ", useTllmGenSparseAttention = " + std::string(useTllmGenSparseAttention ? "true" : "false"); - output += ", fusesDsv4InvRopeFp8Quant = " + std::string(fusesDsv4InvRopeFp8Quant ? "true" : "false"); return output; } @@ -292,21 +289,8 @@ struct MHARunnerParams KVBlockArray pagedKvSfCache; // The output buffer ptr. void* outputPtr; - // The output scaling factor buffer ptr. Used by FP4 output and DSv4 fused epilogue. + // The output scaling factor buffer ptr. (only used for FP4 output) void* outputSfPtr; - - struct Dsv4EpilogueFusionParams - { - // Enable DSv4 inverse-RoPE + FP8 quant epilogue fusion. - bool enabled = false; - // The cos/sin cache used by the fused inverse-RoPE epilogue. - float const* cosSinCache = nullptr; - // The physical token stride of the FP32 output scale tensor. - int32_t scaleBufM = 0; - }; - - // DSv4 fused inverse-RoPE + FP8 quant epilogue parameters. - Dsv4EpilogueFusionParams dsv4EpilogueFusion; // The softmax_status ptr for RingAttention. void* softmaxStatsPtr; // The attention sinks ptr. diff --git a/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu b/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu index 8af43b2b4914..81e947977797 100644 --- a/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu +++ b/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu @@ -23,7 +23,6 @@ #include "cutlass/cutlass.h" #include "cutlass/gemm/device/gemm_grouped.h" #include "cutlass/gemm/kernel/default_gemm_grouped.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/device/splitk_gemm_grouped.h" #include "tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/kernel/default_splitk_gemm_grouped.h" @@ -118,16 +117,16 @@ void cudaGraphGroupedGemmTemplate(cutlass::gemm::GemmCoord* problemSizesPtr, int template <int M1, int N1, int K1, int M2, int N2, int K2, int kAlignmentAB, int kAlignmentC, int kStages> void cudaGraphGroupedGemmType(cutlass::gemm::GemmCoord* problemSizesPtr, int problemCount, void** ptrAGpu, void** ptrBGpu, void** ptrCGpu, void** ptrDGpu, int64_t* ldaGpu, int64_t* ldbGpu, int64_t* ldcGpu, int64_t* lddGpu, - tensorrt_llm::DataType dataType, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, cudaStream_t stream) + nvinfer1::DataType dataType, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, cudaStream_t stream) { - if (dataType == tensorrt_llm::DataType::kHALF) + if (dataType == nvinfer1::DataType::kHALF) { cudaGraphGroupedGemmTemplate<M1, N1, K1, M2, N2, K2, cutlass::half_t, kAlignmentAB, kAlignmentC, kStages>( problemSizesPtr, problemCount, ptrAGpu, ptrBGpu, ptrCGpu, ptrDGpu, ldaGpu, ldbGpu, ldcGpu, lddGpu, hostMaxProblemSizesPtr, stream); } #ifdef ENABLE_BF16 - else if (dataType == tensorrt_llm::DataType::kBF16) + else if (dataType == nvinfer1::DataType::kBF16) { cudaGraphGroupedGemmTemplate<M1, N1, K1, M2, N2, K2, cutlass::bfloat16_t, kAlignmentAB, kAlignmentC, kStages>( problemSizesPtr, problemCount, ptrAGpu, ptrBGpu, ptrCGpu, ptrDGpu, ldaGpu, ldbGpu, ldcGpu, lddGpu, @@ -142,7 +141,7 @@ void cudaGraphGroupedGemmType(cutlass::gemm::GemmCoord* problemSizesPtr, int pro void cudaGraphGroupedGemm(cutlass::gemm::GemmCoord* problemSizesPtr, int problemCount, void** ptrAGpu, void** ptrBGpu, void** ptrCGpu, void** ptrDGpu, int64_t* ldaGpu, int64_t* ldbGpu, int64_t* ldcGpu, int64_t* lddGpu, bool isLoraIn, - tensorrt_llm::DataType dataType, int minKN, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, cudaStream_t stream) + nvinfer1::DataType dataType, int minKN, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, cudaStream_t stream) { if (isLoraIn) { @@ -284,17 +283,17 @@ void cudaGraphSplitKGroupedGemmTemplate(cutlass::gemm::GemmCoord* problemSizesPt template <int M1, int N1, int K1, int M2, int N2, int K2, int kAlignmentAB, int kAlignmentC, int kStages> void cudaGraphSplitKGroupedGemmType(cutlass::gemm::GemmCoord* problemSizesPtr, int problemCount, void** ptrAGpu, void** ptrBGpu, void** ptrCGpu, void** ptrDGpu, int64_t* ldaGpu, int64_t* ldbGpu, int64_t* ldcGpu, int64_t* lddGpu, - tensorrt_llm::DataType dataType, int splitKSlices, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, + nvinfer1::DataType dataType, int splitKSlices, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, int64_t* splitKOffsetsGpu, cudaStream_t stream) { - if (dataType == tensorrt_llm::DataType::kHALF) + if (dataType == nvinfer1::DataType::kHALF) { cudaGraphSplitKGroupedGemmTemplate<M1, N1, K1, M2, N2, K2, cutlass::half_t, kAlignmentAB, kAlignmentC, kStages>( problemSizesPtr, problemCount, ptrAGpu, ptrBGpu, ptrCGpu, ptrDGpu, ldaGpu, ldbGpu, ldcGpu, lddGpu, splitKSlices, hostMaxProblemSizesPtr, splitKOffsetsGpu, stream); } #ifdef ENABLE_BF16 - else if (dataType == tensorrt_llm::DataType::kBF16) + else if (dataType == nvinfer1::DataType::kBF16) { cudaGraphSplitKGroupedGemmTemplate<M1, N1, K1, M2, N2, K2, cutlass::bfloat16_t, kAlignmentAB, kAlignmentC, kStages>(problemSizesPtr, problemCount, ptrAGpu, ptrBGpu, ptrCGpu, ptrDGpu, ldaGpu, ldbGpu, ldcGpu, lddGpu, @@ -309,7 +308,7 @@ void cudaGraphSplitKGroupedGemmType(cutlass::gemm::GemmCoord* problemSizesPtr, i void cudaGraphSplitKGroupedGemm(cutlass::gemm::GemmCoord* problemSizesPtr, int problemCount, void** ptrAGpu, void** ptrBGpu, void** ptrCGpu, void** ptrDGpu, int64_t* ldaGpu, int64_t* ldbGpu, int64_t* ldcGpu, int64_t* lddGpu, - bool isLoraIn, tensorrt_llm::DataType dataType, int splitKSlices, int minKN, + bool isLoraIn, nvinfer1::DataType dataType, int splitKSlices, int minKN, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, int64_t* splitKOffsetsGpu, cudaStream_t stream) { if (isLoraIn) diff --git a/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.h b/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.h index b447bba3a785..0eecccb78852 100644 --- a/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.h +++ b/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.h @@ -18,7 +18,7 @@ #include "cutlass/gemm_coord.h" #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_runtime.h> TRTLLM_NAMESPACE_BEGIN @@ -45,7 +45,7 @@ namespace kernels */ void cudaGraphGroupedGemm(cutlass::gemm::GemmCoord* problemSizesPtr, int problemCount, void** ptrAGpu, void** ptrBGpu, void** ptrCGpu, void** ptrDGpu, int64_t* ldaGpu, int64_t* ldbGpu, int64_t* ldcGpu, int64_t* lddGpu, bool isLoraIn, - tensorrt_llm::DataType dataType, int minKN, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, cudaStream_t stream); + nvinfer1::DataType dataType, int minKN, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, cudaStream_t stream); /** * @brief CUDA Graph compatible wrapper for split-K grouped GEMM operations. @@ -55,7 +55,7 @@ void cudaGraphGroupedGemm(cutlass::gemm::GemmCoord* problemSizesPtr, int problem */ void cudaGraphSplitKGroupedGemm(cutlass::gemm::GemmCoord* problemSizesPtr, int problemCount, void** ptrAGpu, void** ptrBGpu, void** ptrCGpu, void** ptrDGpu, int64_t* ldaGpu, int64_t* ldbGpu, int64_t* ldcGpu, int64_t* lddGpu, - bool isLoraIn, tensorrt_llm::DataType dataType, int splitKSlices, int minKN, + bool isLoraIn, nvinfer1::DataType dataType, int splitKSlices, int minKN, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, int64_t* splitKOffsetsGpu, cudaStream_t stream); } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/customAllReduceKernels.cu b/cpp/tensorrt_llm/kernels/customAllReduceKernels.cu index ea217d465be1..9cf2b51eb583 100644 --- a/cpp/tensorrt_llm/kernels/customAllReduceKernels.cu +++ b/cpp/tensorrt_llm/kernels/customAllReduceKernels.cu @@ -22,7 +22,6 @@ #include "tensorrt_llm/common/customAllReduceUtils.h" #include "tensorrt_llm/common/dataType.h" #include "tensorrt_llm/common/envUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include <cooperative_groups.h> #include <cstdint> #include <tuple> @@ -1188,14 +1187,14 @@ bool is_lamport_supported(int token_num, int hidden_size) return true; } -bool is_lamport_supported(tensorrt_llm::DataType dataType, int token_num, int hidden_size) +bool is_lamport_supported(nvinfer1::DataType dataType, int token_num, int hidden_size) { switch (dataType) { - case tensorrt_llm::DataType::kFLOAT: return is_lamport_supported<float>(token_num, hidden_size); - case tensorrt_llm::DataType::kHALF: return is_lamport_supported<half>(token_num, hidden_size); + case nvinfer1::DataType::kFLOAT: return is_lamport_supported<float>(token_num, hidden_size); + case nvinfer1::DataType::kHALF: return is_lamport_supported<half>(token_num, hidden_size); #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: return is_lamport_supported<__nv_bfloat16>(token_num, hidden_size); + case nvinfer1::DataType::kBF16: return is_lamport_supported<__nv_bfloat16>(token_num, hidden_size); #endif default: return false; } @@ -1659,7 +1658,7 @@ static __global__ void __launch_bounds__(512, 1) twoShotAllReduceKernel(AllReduc update_barrier_flag(params.barrier_flag_ptr, params.barrier_flag_counter_ptr); } -bool configurationSupported(AllReduceStrategyType algo, size_t msg_size, size_t n_ranks, tensorrt_llm::DataType type) +bool configurationSupported(AllReduceStrategyType algo, size_t msg_size, size_t n_ranks, nvinfer1::DataType type) { size_t elts_per_thread = 16 / common::getDTypeSize(type); int const msg_align = (algo == AllReduceStrategyType::TWOSHOT) ? n_ranks * elts_per_thread : elts_per_thread; @@ -1895,8 +1894,8 @@ void AllReduceDispatchType(AllReduceParams& params, AllReduceStrategyType strat, } } -AllReduceParams AllReduceParams::deserialize(int64_t* buffer, size_t tpSize, size_t tpRank, - tensorrt_llm::DataType dataType, int token_num, int hidden_size, AllReduceFusionOp op) +AllReduceParams AllReduceParams::deserialize(int64_t* buffer, size_t tpSize, size_t tpRank, nvinfer1::DataType dataType, + int token_num, int hidden_size, AllReduceFusionOp op) { void* const* buffer_ptrs = reinterpret_cast<void* const*>(buffer); int flag_offset; @@ -1934,7 +1933,7 @@ AllReduceParams AllReduceParams::deserialize(int64_t* buffer, size_t tpSize, siz return params; } -void customAllReduce(kernels::AllReduceParams& params, tensorrt_llm::DataType dataType, AllReduceStrategyType strat, +void customAllReduce(kernels::AllReduceParams& params, nvinfer1::DataType dataType, AllReduceStrategyType strat, AllReduceStrategyConfig config, AllReduceFusionOp fusionOp, cudaStream_t stream) { TLLM_CHECK_WITH_INFO(configurationSupported(strat, params.elts_total, params.ranks_per_node, dataType), @@ -1944,10 +1943,10 @@ void customAllReduce(kernels::AllReduceParams& params, tensorrt_llm::DataType da switch (dataType) { - case tensorrt_llm::DataType::kFLOAT: AllReduceDispatchType<float>(params, strat, config, fusionOp, stream); break; - case tensorrt_llm::DataType::kHALF: AllReduceDispatchType<half>(params, strat, config, fusionOp, stream); break; + case nvinfer1::DataType::kFLOAT: AllReduceDispatchType<float>(params, strat, config, fusionOp, stream); break; + case nvinfer1::DataType::kHALF: AllReduceDispatchType<half>(params, strat, config, fusionOp, stream); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: AllReduceDispatchType<__nv_bfloat16>(params, strat, config, fusionOp, stream); break; #endif @@ -1992,22 +1991,22 @@ void launchResidualRmsNormKernel(kernels::AllReduceParams& params, cudaStream_t } void residualRmsNorm( - kernels::AllReduceParams& params, tensorrt_llm::DataType dataType, cudaStream_t stream, AllReduceFusionOp fusionOp) + kernels::AllReduceParams& params, nvinfer1::DataType dataType, cudaStream_t stream, AllReduceFusionOp fusionOp) { sync_check_cuda_error(stream); switch (dataType) { - case tensorrt_llm::DataType::kFLOAT: launchResidualRmsNormKernel<float>(params, stream, fusionOp); break; - case tensorrt_llm::DataType::kHALF: launchResidualRmsNormKernel<half>(params, stream, fusionOp); break; + case nvinfer1::DataType::kFLOAT: launchResidualRmsNormKernel<float>(params, stream, fusionOp); break; + case nvinfer1::DataType::kHALF: launchResidualRmsNormKernel<half>(params, stream, fusionOp); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: launchResidualRmsNormKernel<__nv_bfloat16>(params, stream, fusionOp); break; + case nvinfer1::DataType::kBF16: launchResidualRmsNormKernel<__nv_bfloat16>(params, stream, fusionOp); break; #endif default: TLLM_THROW("Unsupported dataType for customAllReduce"); } sync_check_cuda_error(stream); } -void lamportInitialize(void* buffer, size_t size, tensorrt_llm::DataType dataType, cudaStream_t stream) +void lamportInitialize(void* buffer, size_t size, nvinfer1::DataType dataType, cudaStream_t stream) { sync_check_cuda_error(stream); if (size == 0) @@ -2016,14 +2015,14 @@ void lamportInitialize(void* buffer, size_t size, tensorrt_llm::DataType dataTyp } switch (dataType) { - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: reduce_fusion::lamport_initialize_kernel_launcher<float>(buffer, size, stream); break; - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: reduce_fusion::lamport_initialize_kernel_launcher<half>(buffer, size, stream); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: reduce_fusion::lamport_initialize_kernel_launcher<__nv_bfloat16>(buffer, size, stream); break; #endif diff --git a/cpp/tensorrt_llm/kernels/customAllReduceKernels.h b/cpp/tensorrt_llm/kernels/customAllReduceKernels.h index 93f67ffdd911..f7151f1cd0ab 100644 --- a/cpp/tensorrt_llm/kernels/customAllReduceKernels.h +++ b/cpp/tensorrt_llm/kernels/customAllReduceKernels.h @@ -17,7 +17,7 @@ #pragma once #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_bf16.h> #include <cuda_fp16.h> #include <limits> @@ -178,23 +178,23 @@ struct AllReduceParams AllReduceFusionParams fusion_params; - static AllReduceParams deserialize(int64_t* buffer, size_t tpSize, size_t tpRank, tensorrt_llm::DataType dataType, + static AllReduceParams deserialize(int64_t* buffer, size_t tpSize, size_t tpRank, nvinfer1::DataType dataType, int token_num, int hidden_size, AllReduceFusionOp op); }; -bool configurationSupported(AllReduceStrategyType algo, size_t msg_size, size_t n_ranks, tensorrt_llm::DataType type); +bool configurationSupported(AllReduceStrategyType algo, size_t msg_size, size_t n_ranks, nvinfer1::DataType type); -void customAllReduce(kernels::AllReduceParams& params, tensorrt_llm::DataType dataType, AllReduceStrategyType strat, +void customAllReduce(kernels::AllReduceParams& params, nvinfer1::DataType dataType, AllReduceStrategyType strat, AllReduceStrategyConfig config, AllReduceFusionOp fusionOp, cudaStream_t stream); void residualRmsNorm( - kernels::AllReduceParams& params, tensorrt_llm::DataType dataType, cudaStream_t stream, AllReduceFusionOp fusionOp); + kernels::AllReduceParams& params, nvinfer1::DataType dataType, cudaStream_t stream, AllReduceFusionOp fusionOp); -void lamportInitialize(void* buffer, size_t size, tensorrt_llm::DataType dataType, cudaStream_t stream); +void lamportInitialize(void* buffer, size_t size, nvinfer1::DataType dataType, cudaStream_t stream); namespace reduce_fusion { -bool is_lamport_supported(tensorrt_llm::DataType dataType, int token_num, int hidden_size); +bool is_lamport_supported(nvinfer1::DataType dataType, int token_num, int hidden_size); } } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp b/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp index bcc0eb1165ac..7bba57a03d5e 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2026, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -17,7 +17,6 @@ #include "tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.h" #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cudaBf16Wrapper.h" -#include "tensorrt_llm/common/cudaUtils.h" #ifdef __GNUC__ // Check if the compiler is GCC or Clang #pragma GCC diagnostic push @@ -32,7 +31,6 @@ #pragma GCC diagnostic pop #endif // __GNUC -#include <algorithm> #include <cuda_runtime_api.h> #include <set> #include <vector> @@ -575,7 +573,7 @@ std::vector<CutlassGemmConfig> get_candidate_configs_sm120(CutlassGemmConfig::Ca std::vector<CutlassGemmConfig> candidate_configs; if (config & CutlassGemmConfig::FP8FP4_MIXED) { - // Mixed FP8 x FP4 only supports the 128x128x128B tile. + // Mixed FP8 x FP4: restrict to 128x128x128B only candidate_configs.push_back(CutlassGemmConfig{CutlassTileConfigSM120::CtaShape128x128x128B, MainloopScheduleType::AUTO, EpilogueScheduleType::AUTO, ClusterShape::ClusterShape_1x1x1}); return candidate_configs; @@ -591,34 +589,9 @@ std::vector<CutlassGemmConfig> get_candidate_configs_sm120(CutlassGemmConfig::Ca MainloopScheduleType::AUTO, EpilogueScheduleType::AUTO, ClusterShape::ClusterShape_1x1x1}); candidate_configs.push_back(CutlassGemmConfig{CutlassTileConfigSM120::CtaShape256x128x64B, MainloopScheduleType::AUTO, EpilogueScheduleType::AUTO, ClusterShape::ClusterShape_1x1x1}); + return candidate_configs; } - else - { - TLLM_THROW("Not Implemented: SM120 group GEMM only supports mxfp8-mxfp4 mixed or nvfp4."); - } - // Filter configs by device shared memory. SM100 (B200) has 228 KiB, but - // consumer Blackwell (SM120 RTX PRO 6000, SM121 GB10 / DGX Spark) has only - // 99 KiB. On these constrained devices, keep only CtaShape128x128x64B which - // fits within 99 KiB including FINALIZE epilogue (~80 KiB total). - // CtaShape128x256x64B/256x128x64B overflow with FINALIZE (~100 KiB). - // CtaShape128x128x128B also exceeds 99 KiB at typical stage counts. - { - constexpr int kMinSmemForFullTileSet = 120 * 1024; - int device = 0; - tensorrt_llm::common::check_cuda_error(cudaGetDevice(&device)); - int maxSmem = 0; - tensorrt_llm::common::check_cuda_error( - cudaDeviceGetAttribute(&maxSmem, cudaDevAttrMaxSharedMemoryPerBlockOptin, device)); - - if (maxSmem < kMinSmemForFullTileSet) - { - auto const it = std::remove_if(candidate_configs.begin(), candidate_configs.end(), - [](CutlassGemmConfig const& config) - { return config.tile_config_sm120 != CutlassTileConfigSM120::CtaShape128x128x64B; }); - candidate_configs.erase(it, candidate_configs.end()); - } - } - return candidate_configs; + TLLM_THROW("Not Implemented: SM120 group GEMM only supports mxfp8-mxfp4 mixed or nvfp4."); } else { diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h b/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h index 6632f273cc35..dbbed4e08c97 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h @@ -17,7 +17,7 @@ #pragma once #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include "cutlass/half.h" #include <cuda_fp16.h> @@ -38,34 +38,34 @@ namespace kernels namespace cutlass_kernels { /////////////////////////////////////////////////////////////////////////////////////////////////// -// tensorrt_llm::DataType to Cutlass +// nvinfer1::DataType to Cutlass /////////////////////////////////////////////////////////////////////////////////////////////////// -template <tensorrt_llm::DataType> +template <nvinfer1::DataType> struct CutlassType { using type = void; }; template <> -struct CutlassType<tensorrt_llm::DataType::kHALF> +struct CutlassType<nvinfer1::DataType::kHALF> { using type = cutlass::half_t; }; template <> -struct CutlassType<tensorrt_llm::DataType::kBF16> +struct CutlassType<nvinfer1::DataType::kBF16> { using type = cutlass::bfloat16_t; }; template <> -struct CutlassType<tensorrt_llm::DataType::kFP8> +struct CutlassType<nvinfer1::DataType::kFP8> { using type = cutlass::float_e4m3_t; }; template <> -struct CutlassType<tensorrt_llm::DataType::kFP4> +struct CutlassType<nvinfer1::DataType::kFP4> { using type = cutlass::float_e2m1_t; }; diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_quant_packed.cu b/cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_quant_packed.cu index adaf60ccc3a8..a4923fdbd072 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_quant_packed.cu +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_quant_packed.cu @@ -52,7 +52,7 @@ __device__ __forceinline__ float reciprocal_approximate_ftz_local(float a) // (8 lanes × 16 BF16 elems = 128 elems). After per-block amax, lanes // 0/8/16/24 each hold one UE8M0 scale byte; lane 0 packs them into a uint32 // and stores in the deep_gemm-expected MN-major layout. -template <int WarpsPerBlock, bool OutputCuteDslSf> +template <int WarpsPerBlock> __global__ void fp8_quantize_1x128_packed_kernel_impl(__nv_fp8_e4m3* __restrict__ fp8_output, int32_t* __restrict__ packed_scale_output, __nv_bfloat16 const* __restrict__ input, int const m, int const k, int const scale_leading_dim_uint32) @@ -72,7 +72,6 @@ __global__ void fp8_quantize_1x128_packed_kernel_impl(__nv_fp8_e4m3* __restrict_ bool const row_in_range = (m_idx < m); uint32_t packed = 0u; - uint32_t scale_byte = 0u; if (row_in_range) { int const k_base = packed_sf_k_idx * 512 + lane_id * 16; @@ -121,7 +120,6 @@ __global__ void fp8_quantize_1x128_packed_kernel_impl(__nv_fp8_e4m3* __restrict_ float const dequant_scale_raw = amax * reciprocal_approximate_ftz_local(448.0f); __nv_fp8_e8m0 ue8m0_scale; ue8m0_scale.__x = __nv_cvt_float_to_e8m0(dequant_scale_raw, __NV_SATFINITE, cudaRoundPosInf); - scale_byte = static_cast<uint32_t>(ue8m0_scale.__x); // Recover quant_scale = 1 / 2^(exp - 127) for fp8 conversion. constexpr uint32_t FP32_EXPONENT_BIAS = 127u; @@ -164,58 +162,32 @@ __global__ void fp8_quantize_1x128_packed_kernel_impl(__nv_fp8_e4m3* __restrict_ } // ---- 5. Pack 4 UE8M0 scales (lanes 0/8/16/24). ---- - if constexpr (!OutputCuteDslSf) + uint32_t const s0 = __shfl_sync(0xFFFFFFFFu, static_cast<uint32_t>(ue8m0_scale.__x), 0); + uint32_t const s1 = __shfl_sync(0xFFFFFFFFu, static_cast<uint32_t>(ue8m0_scale.__x), 8); + uint32_t const s2 = __shfl_sync(0xFFFFFFFFu, static_cast<uint32_t>(ue8m0_scale.__x), 16); + uint32_t const s3 = __shfl_sync(0xFFFFFFFFu, static_cast<uint32_t>(ue8m0_scale.__x), 24); + if (lane_id == 0) { - uint32_t const s0 = __shfl_sync(0xFFFFFFFFu, scale_byte, 0); - uint32_t const s1 = __shfl_sync(0xFFFFFFFFu, scale_byte, 8); - uint32_t const s2 = __shfl_sync(0xFFFFFFFFu, scale_byte, 16); - uint32_t const s3 = __shfl_sync(0xFFFFFFFFu, scale_byte, 24); - if (lane_id == 0) - { - // Mask off scale bytes whose sf_k is past the actual K. - int const num_sf_k = (k + 127) / 128; - int const sf_k_base = packed_sf_k_idx * 4; - if (sf_k_base + 0 < num_sf_k) - packed |= s0; - if (sf_k_base + 1 < num_sf_k) - packed |= (s1 << 8); - if (sf_k_base + 2 < num_sf_k) - packed |= (s2 << 16); - if (sf_k_base + 3 < num_sf_k) - packed |= (s3 << 24); - } + // Mask off scale bytes whose sf_k is past the actual K. + int const num_sf_k = (k + 127) / 128; + int const sf_k_base = packed_sf_k_idx * 4; + if (sf_k_base + 0 < num_sf_k) + packed |= s0; + if (sf_k_base + 1 < num_sf_k) + packed |= (s1 << 8); + if (sf_k_base + 2 < num_sf_k) + packed |= (s2 << 16); + if (sf_k_base + 3 < num_sf_k) + packed |= (s3 << 24); } } - if constexpr (OutputCuteDslSf) - { - // Native MXF8 MMA consumes one UE8M0 scale per 32 K values. Preserve - // the production 1x128 quantization contract by replicating each scale - // four times directly into CUTLASS/CuTe's 128x4 swizzled layout. - if (lane_id % 8 == 0 && m_idx < scale_leading_dim_uint32) - { - int const sf128_idx = packed_sf_k_idx * 4 + lane_id / 8; - int const num_sf128 = (k + 127) / 128; - if (sf128_idx < num_sf128) - { - int const num_sf32 = (k + 31) / 32; - int const num_k_tiles = (num_sf32 + 3) / 4; - int64_t const dst_offset = static_cast<int64_t>(m_idx / 128) * num_k_tiles * 512 - + static_cast<int64_t>(sf128_idx) * 512 + (m_idx % 32) * 16 + ((m_idx % 128) / 32) * 4; - uint32_t const replicated = row_in_range ? scale_byte * 0x01010101u : 0u; - *reinterpret_cast<uint32_t*>(reinterpret_cast<uint8_t*>(packed_scale_output) + dst_offset) = replicated; - } - } - } - else + // Always write the packed scale — `packed` is 0 for padded rows. The grid + // covers the full [0, scale_leading_dim_uint32) leading dim (rounded up to + // WarpsPerBlock), and the m_idx guard drops the few rows past the buffer end. + if (lane_id == 0 && m_idx < scale_leading_dim_uint32) { - // Always write the packed scale — `packed` is 0 for padded rows. The grid - // covers the full [0, scale_leading_dim_uint32) leading dim (rounded up to - // WarpsPerBlock), and the m_idx guard drops the few rows past the buffer end. - if (lane_id == 0 && m_idx < scale_leading_dim_uint32) - { - packed_scale_output[static_cast<int64_t>(packed_sf_k_idx) * scale_leading_dim_uint32 + m_idx] = packed; - } + packed_scale_output[static_cast<int64_t>(packed_sf_k_idx) * scale_leading_dim_uint32 + m_idx] = packed; } #if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) @@ -243,27 +215,8 @@ void launch_fp8_quantize_1x128_packed_bf16_e4m3(__nv_fp8_e4m3* fp8_output, int32 dim3 const block(kWarpsPerBlock * 32, 1, 1); tensorrt_llm::common::launchWithPdlWhenEnabled("fp8_quantize_1x128_packed_kernel_impl", - fp8_quantize_1x128_packed_kernel_impl<kWarpsPerBlock, false>, grid, block, 0, stream, fp8_output, - packed_scale_output, input, m, k, scale_leading_dim_uint32); -} - -void launch_fp8_quantize_1x128_cutedsl_bf16_e4m3(__nv_fp8_e4m3* fp8_output, uint8_t* swizzled_scale_output, - __nv_bfloat16 const* input, int m, int k, int padded_m, cudaStream_t stream) -{ - if (m <= 0 || k <= 0) - { - return; - } - - constexpr int kWarpsPerBlock = 4; - int const num_packed_sf_k = (((k + 127) / 128) + 3) / 4; - int const m_blocks = (padded_m + kWarpsPerBlock - 1) / kWarpsPerBlock; - dim3 const grid(num_packed_sf_k, m_blocks, 1); - dim3 const block(kWarpsPerBlock * 32, 1, 1); - - tensorrt_llm::common::launchWithPdlWhenEnabled("fp8_quantize_1x128_cutedsl_kernel_impl", - fp8_quantize_1x128_packed_kernel_impl<kWarpsPerBlock, true>, grid, block, 0, stream, fp8_output, - reinterpret_cast<int32_t*>(swizzled_scale_output), input, m, k, padded_m); + fp8_quantize_1x128_packed_kernel_impl<kWarpsPerBlock>, grid, block, 0, stream, fp8_output, packed_scale_output, + input, m, k, scale_leading_dim_uint32); } } // namespace kernels::fp8_blockscale_gemm diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_quant_packed.h b/cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_quant_packed.h index 7f8c4f07d33d..a4079cd9b054 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_quant_packed.h +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_quant_packed.h @@ -47,12 +47,6 @@ namespace kernels::fp8_blockscale_gemm void launch_fp8_quantize_1x128_packed_bf16_e4m3(__nv_fp8_e4m3* fp8_output, int32_t* packed_scale_output, __nv_bfloat16 const* input, int m, int k, int scale_leading_dim_uint32, cudaStream_t stream); -// Quantizes with the same 1x128 scale as above, but replicates each UE8M0 -// scale over four 32-wide groups and writes the native SM100 CuTe/CUTLASS -// 128x4 swizzled scale layout. -void launch_fp8_quantize_1x128_cutedsl_bf16_e4m3(__nv_fp8_e4m3* fp8_output, uint8_t* swizzled_scale_output, - __nv_bfloat16 const* input, int m, int k, int padded_m, cudaStream_t stream); - } // namespace kernels::fp8_blockscale_gemm TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h b/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h index ab7ed876257d..24781bec76e7 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h @@ -27,7 +27,7 @@ #include <cuda_fp4.h> #endif #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <array> #include <cuda_runtime_api.h> #include <map> @@ -1032,8 +1032,8 @@ struct GemmProfilerBackend using Config = cutlass_extensions::CutlassGemmConfig; using GemmToProfile = MoeGemmId; - void init(CutlassMoeFCRunnerInterface& runner, GemmToProfile gemm_to_profile, tensorrt_llm::DataType dtype, - tensorrt_llm::DataType wtype, tensorrt_llm::DataType otype, int num_experts, int k, int64_t hidden_size, + void init(CutlassMoeFCRunnerInterface& runner, GemmToProfile gemm_to_profile, nvinfer1::DataType dtype, + nvinfer1::DataType wtype, nvinfer1::DataType otype, int num_experts, int k, int64_t hidden_size, int64_t unpadded_hidden_size, int64_t inter_size, int64_t group_size, ActivationType activation_type, bool bias, bool use_lora, bool min_latency_mode, bool need_weights, MOEParallelismConfig parallelism_config, bool const enable_alltoall, bool use_mxfp8_weight_scaling = false) @@ -1061,21 +1061,20 @@ struct GemmProfilerBackend mSM = common::getSMVersion(); mScalingType = TmaWarpSpecializedGroupedGemmInput::FpXBlockScalingType::NONE; - if (dtype == tensorrt_llm::DataType::kFP8 - && (wtype == tensorrt_llm::DataType::kFP4 || wtype == tensorrt_llm::DataType::kINT64)) + if (dtype == nvinfer1::DataType::kFP8 + && (wtype == nvinfer1::DataType::kFP4 || wtype == nvinfer1::DataType::kINT64)) { mScalingType = TmaWarpSpecializedGroupedGemmInput::FpXBlockScalingType::MXFPX; } - else if (dtype == tensorrt_llm::DataType::kFP8 && wtype == tensorrt_llm::DataType::kFP8 - && use_mxfp8_weight_scaling) + else if (dtype == nvinfer1::DataType::kFP8 && wtype == nvinfer1::DataType::kFP8 && use_mxfp8_weight_scaling) { // MXFP8 W8A8: e4m3 acts × e4m3 weights with UE8M0 1x32 block scales on both sides. // Profiler must produce MXFPX block-scaled inputs (otherwise the per-expert SF // pointer arrays stay uninitialized and the kernel reads garbage SF addresses). mScalingType = TmaWarpSpecializedGroupedGemmInput::FpXBlockScalingType::MXFPX; } - else if ((dtype == tensorrt_llm::DataType::kFP4 || dtype == tensorrt_llm::DataType::kINT64) - && (wtype == tensorrt_llm::DataType::kFP4 || wtype == tensorrt_llm::DataType::kINT64)) + else if ((dtype == nvinfer1::DataType::kFP4 || dtype == nvinfer1::DataType::kINT64) + && (wtype == nvinfer1::DataType::kFP4 || wtype == nvinfer1::DataType::kINT64)) { mScalingType = TmaWarpSpecializedGroupedGemmInput::FpXBlockScalingType::NVFP4; } @@ -1107,9 +1106,9 @@ struct GemmProfilerBackend int mSampleIndex = 0; - tensorrt_llm::DataType mDType{}; - tensorrt_llm::DataType mWType{}; - tensorrt_llm::DataType mOType{}; + nvinfer1::DataType mDType{}; + nvinfer1::DataType mWType{}; + nvinfer1::DataType mOType{}; // This will be a unique value for every iteration of warmup and actual bench constexpr static int64_t NUM_ROUTING_SAMPLES = 16; diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_lora_grouped_gemm.h b/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_lora_grouped_gemm.h index 15af49fc1839..55ab4e40a3ae 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_lora_grouped_gemm.h +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_lora_grouped_gemm.h @@ -18,7 +18,7 @@ #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cstdint> #include <cuda_runtime.h> @@ -56,7 +56,7 @@ struct MoeLoraGroupedGemmModule; // stream: CUDA stream to launch onto. using MoeLoraGroupedGemmRunFn = void (*)(MoeLoraGroupedGemmModule const& mod, int64_t num_permuted_tokens, int64_t in_hidden_size, int64_t max_lora_rank, int64_t dtype_bytes, int64_t splitk_slices, void const* input_base, - void* output_base, tensorrt_llm::DataType data_type, cudaStream_t stream); + void* output_base, nvinfer1::DataType data_type, cudaStream_t stream); // Per-module device-resident scratch for the MoE LoRA capture-safe path. // Pointers refer to device memory unless noted. diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_util_kernels.h b/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_util_kernels.h index f3e8940b0c28..e902e2c9d6d3 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_util_kernels.h +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_util_kernels.h @@ -25,6 +25,7 @@ #ifdef ENABLE_FP4 #include <cuda_fp4.h> #endif +#include <NvInferRuntime.h> #include <array> #include <cuda_runtime_api.h> #include <map> diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.inl b/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.inl index 9c5ebbdaa19d..0044528b4dff 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.inl +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_launcher.inl @@ -15,8 +15,6 @@ */ #pragma once -#include "tensorrt_llm/common/cudaUtils.h" - #include "cutlass/array.h" #include "cutlass/numeric_conversion.h" @@ -679,19 +677,6 @@ using namespace cutlass::epilogue; "Workspace is size %zu but only %zu were allocated", calculated_ws_size, \ tma_ws_input.gemm_workspace_size); \ \ - /* Check if kernel SMEM fits on the active device before launch. */ \ - { \ - using GemmKernel_ = typename GemmGrouped::GemmKernel; \ - int smem_size = static_cast<int>(sizeof(typename GemmKernel_::SharedStorage)); \ - int device_ = 0; \ - tensorrt_llm::common::check_cuda_error(cudaGetDevice(&device_)); \ - int maxSmem_ = 0; \ - tensorrt_llm::common::check_cuda_error( \ - cudaDeviceGetAttribute(&maxSmem_, cudaDevAttrMaxSharedMemoryPerBlockOptin, device_)); \ - TLLM_CHECK_WITH_INFO(smem_size <= maxSmem_, \ - "MoE grouped GEMM requires %d bytes shared memory but device supports %d", smem_size, maxSmem_); \ - } \ - \ auto can_implement = gemm.can_implement(args); \ TLLM_CHECK_WITH_INFO(can_implement == cutlass::Status::kSuccess, \ "Grouped GEMM kernel will fail for params. Error: " \ diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.inl b/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.inl index cdf5ea8dc3c9..f37920dcf73c 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.inl +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/launchers/moe_gemm_tma_ws_mixed_input_launcher.inl @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2026, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -19,8 +19,6 @@ #pragma GCC diagnostic ignored "-Wstrict-aliasing" #endif // __GNUC__ -#include "tensorrt_llm/common/cudaUtils.h" - #include "cutlass/epilogue/collective/default_epilogue.hpp" #include "cutlass/epilogue/thread/linear_combination.h" #include "cutlass/gemm/collective/collective_builder.hpp" @@ -275,17 +273,6 @@ void sm90_generic_mixed_moe_gemm_kernelLauncher(GroupedGemmInput<T, WeightType, // This is not initialized during workspace size calculation so check after TLLM_CHECK_WITH_INFO(hopper_inputs.swap_ab, "swap_ab must be true for mixed dtype WS grouped GEMM"); - { - int smem_size = static_cast<int>(sizeof(typename GemmKernel::SharedStorage)); - int device = 0; - tensorrt_llm::common::check_cuda_error(cudaGetDevice(&device)); - int maxSmem = 0; - tensorrt_llm::common::check_cuda_error( - cudaDeviceGetAttribute(&maxSmem, cudaDevAttrMaxSharedMemoryPerBlockOptin, device)); - TLLM_CHECK_WITH_INFO(smem_size <= maxSmem, - "Mixed dtype WS grouped GEMM requires %d bytes shared memory but device supports %d", smem_size, maxSmem); - } - auto can_implement = gemm.can_implement(arguments); if (can_implement != cutlass::Status::kSuccess) { diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu b/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu index 8bed9c16b58e..b7a32be2e285 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu @@ -58,7 +58,6 @@ #include "tensorrt_llm/kernels/preQuantScaleKernel.h" #include "tensorrt_llm/kernels/quantization.cuh" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/cutlass_kernels/include/moe_lora_pointer_expand.h" #include "tensorrt_llm/kernels/cutlass_kernels/include/moe_util_kernels.h" // NOTE: the grouped-GEMM dispatch (cudaGraph(SplitK)GroupedGemm, @@ -3687,7 +3686,7 @@ void CutlassMoeFCRunner<T, WeightType, OutputType, InputType, BackBoneType, Enab inline void runMoeLoraGroupedGemmModule(::tensorrt_llm::kernels::cutlass_kernels::MoeLoraGroupedGemmModule const& mod, int64_t num_permuted_tokens, int64_t in_hidden_size, int64_t max_lora_rank, int64_t dtype_bytes, int64_t splitk_slices, void const* input_base, void* output_base, - ::tensorrt_llm::kernels::cutlass_kernels::MoeLoraGroupedGemmRunFn run, tensorrt_llm::DataType data_type, + ::tensorrt_llm::kernels::cutlass_kernels::MoeLoraGroupedGemmRunFn run, nvinfer1::DataType data_type, cudaStream_t stream) { TLLM_CHECK_WITH_INFO(mod.permuted_ranks_dev != nullptr, @@ -3699,25 +3698,25 @@ inline void runMoeLoraGroupedGemmModule(::tensorrt_llm::kernels::cutlass_kernels data_type, stream); } -// Map the activation/back-bone type to the DataType enum the +// Map the activation/back-bone type to the nvinfer1 enum the // cuda_graph_grouped_gemm wrappers expect. Only fp16/bf16/fp32 are handled; // anything else is a compile-time error rather than a silent fall-through. template <class ScaleBiasType> -constexpr tensorrt_llm::DataType moeLoraDataType() +constexpr nvinfer1::DataType moeLoraNvInferType() { if constexpr (std::is_same_v<ScaleBiasType, half>) { - return tensorrt_llm::DataType::kHALF; + return nvinfer1::DataType::kHALF; } #if defined(ENABLE_BF16) else if constexpr (std::is_same_v<ScaleBiasType, __nv_bfloat16>) { - return tensorrt_llm::DataType::kBF16; + return nvinfer1::DataType::kBF16; } #endif else if constexpr (std::is_same_v<ScaleBiasType, float>) { - return tensorrt_llm::DataType::kFLOAT; + return nvinfer1::DataType::kFLOAT; } else { @@ -3926,7 +3925,7 @@ auto CutlassMoeFCRunner<T, WeightType, OutputType, InputType, BackBoneType, Enab if (lora_params.grouped_gemm.enabled) { auto const& grouped_gemm = lora_params.grouped_gemm; - tensorrt_llm::DataType const data_type = moeLoraDataType<ScaleBiasType>(); + nvinfer1::DataType const data_type = moeLoraNvInferType<ScaleBiasType>(); // The grouped-GEMM GEMM skips rank-0 rows, but the bias/reorder paths // read lora_fc1_result_ for every valid row. Zero the buffer first so @@ -4021,7 +4020,7 @@ void CutlassMoeFCRunner<T, WeightType, OutputType, InputType, BackBoneType, Enab if (lora_params.grouped_gemm.enabled) { auto const& grouped_gemm = lora_params.grouped_gemm; - tensorrt_llm::DataType const data_type = moeLoraDataType<ScaleBiasType>(); + nvinfer1::DataType const data_type = moeLoraNvInferType<ScaleBiasType>(); // As in loraFC1, zero the output so rank-0 rows the GEMM skips do not // feed stale data into the downstream add. @@ -4751,18 +4750,18 @@ std::map<std::string, std::pair<size_t, size_t>> GemmProfilerBackend::getProfile size_t k = mK; size_t num_expanded_tokens = mMinLatencyMode ? maxM * mNumExpertsPerNode : maxM * k; - TLLM_CHECK(mDType != tensorrt_llm::DataType::kINT4); + TLLM_CHECK(mDType != nvinfer1::DataType::kINT4); // nvllm still uses int64 because torch doesn't have fp4 yet. - bool is_4bit_act = mDType == tensorrt_llm::DataType::kFP4 || mDType == tensorrt_llm::DataType::kINT64; - bool is_4bit_weight = mWType == tensorrt_llm::DataType::kINT4 || mWType == tensorrt_llm::DataType::kFP4 - || mWType == tensorrt_llm::DataType::kINT64; + bool is_4bit_act = mDType == nvinfer1::DataType::kFP4 || mDType == nvinfer1::DataType::kINT64; + bool is_4bit_weight = mWType == nvinfer1::DataType::kINT4 || mWType == nvinfer1::DataType::kFP4 + || mWType == nvinfer1::DataType::kINT64; TLLM_CHECK_WITH_INFO(!is_4bit_act || is_4bit_weight, "Cannot have 4-bit activation with non-4-bit weight"); float dtype_bytes = is_4bit_act ? 0.5f - : static_cast<float>(mWType == tensorrt_llm::DataType::kINT4 ? getDTypeSize(mOType) : getDTypeSize(mDType)); + : static_cast<float>(mWType == nvinfer1::DataType::kINT4 ? getDTypeSize(mOType) : getDTypeSize(mDType)); float weight_bytes = is_4bit_weight ? 0.5f : static_cast<float>(getDTypeSize(mWType)); size_t output_bytes = getDTypeSize(mOType); - size_t gemm_output_bytes = (mOType == tensorrt_llm::DataType::kFP8) + size_t gemm_output_bytes = (mOType == nvinfer1::DataType::kFP8) ? sizeof(TmaWarpSpecializedGroupedGemmInput::OutputTypeAdaptor_t<__nv_fp8_e4m3>) : output_bytes; @@ -4804,18 +4803,18 @@ std::map<std::string, std::pair<size_t, size_t>> GemmProfilerBackend::getProfile // TODO Make quant 2 & 4 bigger for FP8 if we ever change to scaling per expert bool is_int_w_quant - = (mWType == tensorrt_llm::DataType::kINT8 || mWType == tensorrt_llm::DataType::kINT4) && mGroupSize <= 0; + = (mWType == nvinfer1::DataType::kINT8 || mWType == nvinfer1::DataType::kINT4) && mGroupSize <= 0; bool is_int_groupwise_w_quant - = (mWType == tensorrt_llm::DataType::kINT8 || mWType == tensorrt_llm::DataType::kINT4) && mGroupSize > 0; - bool is_fp8_act_quant = mDType == tensorrt_llm::DataType::kFP8; - bool is_fp8_w_quant = mWType == tensorrt_llm::DataType::kFP8; + = (mWType == nvinfer1::DataType::kINT8 || mWType == nvinfer1::DataType::kINT4) && mGroupSize > 0; + bool is_fp8_act_quant = mDType == nvinfer1::DataType::kFP8; + bool is_fp8_w_quant = mWType == nvinfer1::DataType::kFP8; // nvllm still uses int64 because torch doesn't have fp4 yet. - // bool is_fp4_act_quant = mDType == tensorrt_llm::DataType::kFP4 || mDType == tensorrt_llm::DataType::kINT64; - bool is_fp4_w_quant = mWType == tensorrt_llm::DataType::kFP4 || mWType == tensorrt_llm::DataType::kINT64; + // bool is_fp4_act_quant = mDType == nvinfer1::DataType::kFP4 || mDType == nvinfer1::DataType::kINT64; + bool is_fp4_w_quant = mWType == nvinfer1::DataType::kFP4 || mWType == nvinfer1::DataType::kINT64; bool is_w4afp8_quant = is_int_groupwise_w_quant && is_fp8_act_quant; // bool is_wfp4afp8_quant = is_fp4_w_quant && is_fp8_act_quant; - bool is_wfp4a16_quant = (mDType == tensorrt_llm::DataType::kHALF || mDType == tensorrt_llm::DataType::kBF16) - && mWType == tensorrt_llm::DataType::kUINT8; + bool is_wfp4a16_quant = (mDType == nvinfer1::DataType::kHALF || mDType == nvinfer1::DataType::kBF16) + && mWType == nvinfer1::DataType::kUINT8; // Int sizes size_t quant_1_size = is_int_w_quant ? fc1_out_size * num_experts_per_node * dtype_bytes : 0; @@ -5048,19 +5047,19 @@ void GemmProfilerBackend::prepareQuantParams(int num_tokens, char* workspace_ptr GET_WS_PTR(float const*, w4a8_alpha); #undef GET_WS_PTR - if ((mWType == tensorrt_llm::DataType::kINT8 || mWType == tensorrt_llm::DataType::kINT4 - || mWType == tensorrt_llm::DataType::kUINT8) + if ((mWType == nvinfer1::DataType::kINT8 || mWType == nvinfer1::DataType::kINT4 + || mWType == nvinfer1::DataType::kUINT8) && mGroupSize < 0) { TLLM_CHECK(quant_1 && quant_2); mQuantParams = QuantParams::Int(quant_1, quant_2); } - else if (mWType == tensorrt_llm::DataType::kINT4 || mWType == tensorrt_llm::DataType::kUINT8) + else if (mWType == nvinfer1::DataType::kINT4 || mWType == nvinfer1::DataType::kUINT8) { TLLM_CHECK(quant_1 && quant_2); - if (mDType == tensorrt_llm::DataType::kFP8 - || (mWType == tensorrt_llm::DataType::kUINT8 - && (mDType == tensorrt_llm::DataType::kHALF || mDType == tensorrt_llm::DataType::kBF16))) + if (mDType == nvinfer1::DataType::kFP8 + || (mWType == nvinfer1::DataType::kUINT8 + && (mDType == nvinfer1::DataType::kHALF || mDType == nvinfer1::DataType::kBF16))) { TLLM_CHECK(w4a8_alpha); mQuantParams = QuantParams::GroupWise( @@ -5071,7 +5070,7 @@ void GemmProfilerBackend::prepareQuantParams(int num_tokens, char* workspace_ptr mQuantParams = QuantParams::GroupWise(mGroupSize, quant_1, quant_2, nullptr, nullptr, quant_3, quant_4); } } - else if (mWType == tensorrt_llm::DataType::kFP8) + else if (mWType == nvinfer1::DataType::kFP8) { if (mUseMxfp8WeightScaling) { @@ -5090,8 +5089,8 @@ void GemmProfilerBackend::prepareQuantParams(int num_tokens, char* workspace_ptr static_cast<float const*>(quant_3), static_cast<float const*>(quant_4)); } } - else if (mDType == tensorrt_llm::DataType::kFP8 - && (mWType == tensorrt_llm::DataType::kFP4 || mWType == tensorrt_llm::DataType::kINT64)) + else if (mDType == nvinfer1::DataType::kFP8 + && (mWType == nvinfer1::DataType::kFP4 || mWType == nvinfer1::DataType::kINT64)) { TLLM_CHECK(quant_1 && quant_2 && quant_3 && quant_4 && quant_5 && quant_6); mQuantParams = QuantParams::FP8MXFP4(static_cast<float const*>(quant_1), @@ -5100,8 +5099,8 @@ void GemmProfilerBackend::prepareQuantParams(int num_tokens, char* workspace_ptr static_cast<TmaWarpSpecializedGroupedGemmInput::MXFPXElementSF const*>(quant_5), static_cast<float const*>(quant_6)); } - else if ((mDType == tensorrt_llm::DataType::kFP4 || mDType == tensorrt_llm::DataType::kINT64) - && (mWType == tensorrt_llm::DataType::kFP4 || mWType == tensorrt_llm::DataType::kINT64)) + else if ((mDType == nvinfer1::DataType::kFP4 || mDType == nvinfer1::DataType::kINT64) + && (mWType == nvinfer1::DataType::kFP4 || mWType == nvinfer1::DataType::kINT64)) { // nvllm still uses int64 because torch doesn't have fp4 yet. TLLM_CHECK(quant_1 && quant_2 && quant_3 && quant_4 && quant_5 && quant_6); @@ -5121,9 +5120,9 @@ void GemmProfilerBackend::prepareTmaWsInputs(int num_tokens, char* workspace_ptr return; } - bool use_w4afp8 = (mDType == tensorrt_llm::DataType::kFP8 && mWType == tensorrt_llm::DataType::kINT4); - bool use_wfp4a16 = ((mDType == tensorrt_llm::DataType::kHALF || mDType == tensorrt_llm::DataType::kBF16) - && mWType == tensorrt_llm::DataType::kUINT8); + bool use_w4afp8 = (mDType == nvinfer1::DataType::kFP8 && mWType == nvinfer1::DataType::kINT4); + bool use_wfp4a16 = ((mDType == nvinfer1::DataType::kHALF || mDType == nvinfer1::DataType::kBF16) + && mWType == nvinfer1::DataType::kUINT8); bool const use_finalize_fusion = fusion == TmaWarpSpecializedGroupedGemmInput::EpilogueFusion::FINALIZE; bool const finalize_fusion_not_supported = !mInterface->use_fused_finalize_ || mMinLatencyMode || use_wfp4a16 || mGemmToProfile != GemmToProfile::GEMM_2; diff --git a/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQAImplJIT/compileEngine.cpp b/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQAImplJIT/compileEngine.cpp index 0438ed1d6b99..b9c58643664d 100644 --- a/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQAImplJIT/compileEngine.cpp +++ b/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQAImplJIT/compileEngine.cpp @@ -55,13 +55,7 @@ namespace jit CubinObj CompileEngine::compile() const { tllmXqaJitProgram program; - // QGMMA runs attention fully in native FP8 format, including the output of the tiled - // Q @ K^T. For the multi-query spec-dec verify, this introduces undue errors on the verify - // path, which causes the rejection of otherwise legitimate tokens. In these cases, we defer - // to the HMMA kernel, which avoids that particular issue. - bool const isLinearSpecDec = mXqaParams.multi_query_tokens && !mXqaParams.is_spec_dec_tree; - bool const useQGMMAKernel = supportConfigQGMMA(mXqaParams, mSM, true) && !isLinearSpecDec; - + bool const useQGMMAKernel = supportConfigQGMMA(mXqaParams, mSM, true); tllmXqaJitRopeStyle ropeStyle = tllmXqaJitRopeStyle::TLLM_XQA_JIT_ROPE_NONE; bool const applyRoPEInXqaKernel = appliesRoPEInXqaKernel(mXqaParams, useQGMMAKernel); if (applyRoPEInXqaKernel) diff --git a/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/xqaParams.h b/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/xqaParams.h index dc7794752e47..e421be0a6bd7 100644 --- a/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/xqaParams.h +++ b/cpp/tensorrt_llm/kernels/decoderMaskedMultiheadAttention/xqaParams.h @@ -63,7 +63,6 @@ struct XQAParams int64_t* spec_decoding_bl_tree_mask_offset; // for blackwell spec-dec tree mask offset uint32_t* spec_decoding_bl_tree_mask; // for blackwell spec-dec tree mask int32_t* spec_bl_tree_first_sparse_mask_offset_kv; // for blackwell spec-dec tree first sparse mask offset kv - bool force_prepare_spec_dec_tree_mask = false; int32_t const* mrope_position_deltas = nullptr; // Helix parallelism params. int32_t const* helix_position_offsets = nullptr; diff --git a/cpp/tensorrt_llm/kernels/decodingKernels.cu b/cpp/tensorrt_llm/kernels/decodingKernels.cu index 7c72091abad2..967b1b60411c 100644 --- a/cpp/tensorrt_llm/kernels/decodingKernels.cu +++ b/cpp/tensorrt_llm/kernels/decodingKernels.cu @@ -122,7 +122,7 @@ __global__ void gatherTree(gatherTreeParam param) { int const levelBeamIx = batch * param.beamWidth * param.maxSeqLen + beam * param.maxSeqLen + level; int const levelParentIx = batch * param.beamWidth * param.maxSeqLen + parent * param.maxSeqLen + level; - if (parent < 0 || parent >= param.beamWidth) + if (parent < 0 || parent > param.beamWidth) { param.outputIds[levelBeamIx] = param.endTokens[batch]; parent = -1; @@ -702,11 +702,10 @@ __global__ void transposeLogProbs(float* outputLogProbs, float* outputLogProbsTi } auto const batchSlot = batchSlots[batchIdx]; - auto const batchBeamIdx = batchSlot * beamWidth + beamIdx; - if (pos < sequenceLengths[batchBeamIdx]) + if (pos < sequenceLengths[batchSlot]) { - auto const outputIndex = batchSlot * beamWidth * maxSeqLen + beamIdx * maxSeqLen + pos; - outputLogProbs[outputIndex] + auto const batchBeamIdx = batchSlot * beamWidth * maxSeqLen + beamIdx * maxSeqLen + pos; + outputLogProbs[batchBeamIdx] = outputLogProbsTiled[pos * maxBatchSize * beamWidth + batchSlot * beamWidth + beamIdx]; } } diff --git a/cpp/tensorrt_llm/kernels/deepseekV4BlockTable.cu b/cpp/tensorrt_llm/kernels/deepseekV4BlockTable.cu deleted file mode 100644 index 7b6188c74eca..000000000000 --- a/cpp/tensorrt_llm/kernels/deepseekV4BlockTable.cu +++ /dev/null @@ -1,372 +0,0 @@ -/* - * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include "tensorrt_llm/kernels/deepseekV4BlockTable.h" - -#include <cstdint> - -TRTLLM_NAMESPACE_BEGIN - -namespace kernels -{ -namespace -{ - -constexpr int32_t kBadPageIndex = -1; -constexpr int32_t kThreadsPerBlock = 256; -constexpr int32_t kVecThreadsPerBlock = 128; -constexpr int32_t kRowKernelMinBlocks = 256; -constexpr int32_t kVecRowsPerBlock = 8; - -__device__ __forceinline__ int32_t computeBasePageIndex(int32_t const* __restrict__ blockOffsets, - int32_t const* __restrict__ copyIdx, int64_t const* __restrict__ poolIds, bool const* __restrict__ validPool, - int32_t const* __restrict__ scales, int32_t const* __restrict__ layerOffsets, int32_t numPools, - int32_t copyIdxCapacity, int32_t numAttnTypes, int32_t maxBlocksPerSeq, int32_t layerId, int32_t attnTypeId, - int32_t tableId, int32_t blockId) -{ - int32_t const layerAttnOffset = layerId * numAttnTypes + attnTypeId; - int64_t const poolId64 = poolIds[layerAttnOffset]; - bool const isValidPool = validPool[layerAttnOffset] && poolId64 >= 0 && poolId64 < numPools; - if (!isValidPool) - { - return kBadPageIndex; - } - - int32_t const mappedTableId = copyIdx[tableId]; - if (mappedTableId < 0 || mappedTableId >= copyIdxCapacity) - { - return kBadPageIndex; - } - - auto const poolId = static_cast<int32_t>(poolId64); - int64_t const blockOffsetsIndex - = (((static_cast<int64_t>(poolId) * copyIdxCapacity + mappedTableId) * 2) * maxBlocksPerSeq) + blockId; - int32_t const base = blockOffsets[blockOffsetsIndex]; - if (base == kBadPageIndex) - { - return kBadPageIndex; - } - - return base * scales[layerAttnOffset] + layerOffsets[layerAttnOffset]; -} - -__device__ __forceinline__ int32_t applyScaleAndOffset(int32_t base, int32_t scale, int32_t layerOffset) -{ - return base == kBadPageIndex ? kBadPageIndex : base * scale + layerOffset; -} - -__device__ __forceinline__ void fillBadSlidingBlockTableRow(int32_t* outputRow, int32_t maxBlocksPerSeq, bool useVec4) -{ - if (useVec4) - { - int4 const bad = {kBadPageIndex, kBadPageIndex, kBadPageIndex, kBadPageIndex}; - auto* outputVec = reinterpret_cast<int4*>(outputRow); - int32_t const vecsPerRow = maxBlocksPerSeq / 4; - for (int32_t vecId = threadIdx.x; vecId < vecsPerRow; vecId += blockDim.x) - { - outputVec[vecId] = bad; - } - return; - } - - for (int32_t blockId = threadIdx.x; blockId < maxBlocksPerSeq; blockId += blockDim.x) - { - outputRow[blockId] = kBadPageIndex; - } -} - -__global__ void computeSlidingBlockTablesRowsTiledKernel(int32_t const* __restrict__ blockOffsets, - int32_t const* __restrict__ copyIdx, int64_t const* __restrict__ poolIds, bool const* __restrict__ validPool, - int32_t const* __restrict__ scales, int32_t const* __restrict__ layerOffsets, int32_t* __restrict__ output, - int32_t numPools, int32_t copyIdxCapacity, int32_t numLayerAttn, int32_t numTables, int32_t maxBlocksPerSeq) -{ - bool const useVec4 = maxBlocksPerSeq % 4 == 0; - int32_t const vecsPerRow = maxBlocksPerSeq / 4; - int32_t const firstTableId = static_cast<int32_t>(blockIdx.x) * kVecRowsPerBlock; - int32_t const layerAttnOffset = static_cast<int32_t>(blockIdx.y); - if (layerAttnOffset >= numLayerAttn) - { - return; - } - - int64_t const poolId64 = poolIds[layerAttnOffset]; - bool const isValidPool = validPool[layerAttnOffset] && poolId64 >= 0 && poolId64 < numPools; - if (!isValidPool) - { -#pragma unroll - for (int32_t localRow = 0; localRow < kVecRowsPerBlock; ++localRow) - { - int32_t const tableId = firstTableId + localRow; - if (tableId >= numTables) - { - continue; - } - - int64_t const outputOffset - = (static_cast<int64_t>(layerAttnOffset) * numTables + tableId) * maxBlocksPerSeq; - fillBadSlidingBlockTableRow(output + outputOffset, maxBlocksPerSeq, useVec4); - } - return; - } - - auto const poolId = static_cast<int32_t>(poolId64); - int32_t const scale = scales[layerAttnOffset]; - int32_t const layerOffset = layerOffsets[layerAttnOffset]; - -#pragma unroll - for (int32_t localRow = 0; localRow < kVecRowsPerBlock; ++localRow) - { - int32_t const tableId = firstTableId + localRow; - if (tableId >= numTables) - { - continue; - } - - int64_t const outputOffset = (static_cast<int64_t>(layerAttnOffset) * numTables + tableId) * maxBlocksPerSeq; - auto* outputRow = output + outputOffset; - int32_t const mappedTableId = copyIdx[tableId]; - bool const isValidTable = mappedTableId >= 0 && mappedTableId < copyIdxCapacity; - if (!isValidTable) - { - fillBadSlidingBlockTableRow(outputRow, maxBlocksPerSeq, useVec4); - continue; - } - - int64_t const blockOffsetsOffset - = ((static_cast<int64_t>(poolId) * copyIdxCapacity + mappedTableId) * 2) * maxBlocksPerSeq; - auto const* blockOffsetsRow = blockOffsets + blockOffsetsOffset; - if (useVec4) - { - auto const* blockOffsetsVec = reinterpret_cast<int4 const*>(blockOffsetsRow); - auto* outputVec = reinterpret_cast<int4*>(outputRow); - for (int32_t vecId = threadIdx.x; vecId < vecsPerRow; vecId += blockDim.x) - { - int4 const base = blockOffsetsVec[vecId]; - int4 const value = {applyScaleAndOffset(base.x, scale, layerOffset), - applyScaleAndOffset(base.y, scale, layerOffset), applyScaleAndOffset(base.z, scale, layerOffset), - applyScaleAndOffset(base.w, scale, layerOffset)}; - outputVec[vecId] = value; - } - continue; - } - - for (int32_t blockId = threadIdx.x; blockId < maxBlocksPerSeq; blockId += blockDim.x) - { - int32_t const base = blockOffsetsRow[blockId]; - outputRow[blockId] = applyScaleAndOffset(base, scale, layerOffset); - } - } -} - -__global__ void computeSlidingBlockTablesWithScratchKernel(int32_t const* __restrict__ blockOffsets, - int32_t const* __restrict__ copyIdx, int64_t const* __restrict__ poolIds, bool const* __restrict__ validPool, - int32_t const* __restrict__ scales, int32_t const* __restrict__ layerOffsets, - int32_t const* __restrict__ scratchPages, int32_t const* __restrict__ scratchBegs, - int32_t const* __restrict__ scratchEnds, int32_t const* __restrict__ scratchSlots, - int32_t const* __restrict__ numContexts, int32_t* __restrict__ output, int64_t totalElements, int32_t numPools, - int32_t copyIdxCapacity, int32_t numAttnTypes, int32_t numTables, int32_t maxBlocksPerSeq, int32_t scratchCapacity, - int32_t maxScratchSlots) -{ - int64_t const linearIdx = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x; - if (linearIdx >= totalElements) - { - return; - } - - int64_t remaining = linearIdx; - int32_t const blockId = static_cast<int32_t>(remaining % maxBlocksPerSeq); - remaining /= maxBlocksPerSeq; - int32_t const tableId = static_cast<int32_t>(remaining % numTables); - remaining /= numTables; - int32_t const attnTypeId = static_cast<int32_t>(remaining % numAttnTypes); - int32_t const layerId = static_cast<int32_t>(remaining / numAttnTypes); - - int32_t const layerAttnOffset = layerId * numAttnTypes + attnTypeId; - int32_t const basePageIndex = computeBasePageIndex(blockOffsets, copyIdx, poolIds, validPool, scales, layerOffsets, - numPools, copyIdxCapacity, numAttnTypes, maxBlocksPerSeq, layerId, attnTypeId, tableId, blockId); - - int64_t const poolId64 = poolIds[layerAttnOffset]; - bool const isValidPool = validPool[layerAttnOffset] && poolId64 >= 0 && poolId64 < numPools; - int32_t const activeContexts = numContexts[0]; - bool const canUseScratch = isValidPool && tableId < scratchCapacity && tableId < activeContexts; - - if (!canUseScratch) - { - output[linearIdx] = basePageIndex; - return; - } - - auto const poolId = static_cast<int32_t>(poolId64); - int64_t const scratchRangeOffset = static_cast<int64_t>(poolId) * scratchCapacity + tableId; - int32_t const scratchBeg = scratchBegs[scratchRangeOffset]; - int32_t const scratchEnd = scratchEnds[scratchRangeOffset]; - bool const inScratchRange = blockId >= scratchBeg && blockId < scratchEnd; - if (!inScratchRange) - { - output[linearIdx] = basePageIndex; - return; - } - - int32_t const scale = scales[layerAttnOffset]; - int32_t const rangeIndex = blockId - scratchBeg; - int32_t const totalOffset = rangeIndex * scratchPages[layerAttnOffset]; - int32_t slotIdx = totalOffset / scale; - if (slotIdx >= maxScratchSlots) - { - slotIdx = maxScratchSlots - 1; - } - - int64_t const slotOffset = scratchRangeOffset * maxScratchSlots + slotIdx; - int32_t const slotId = scratchSlots[slotOffset]; - int32_t const offset = totalOffset % scale; - output[linearIdx] = slotId * scale + ((offset + layerOffsets[layerAttnOffset]) % scale); -} - -__global__ void computeSlidingBlockTablesWithScratchRowsKernel(int32_t const* __restrict__ blockOffsets, - int32_t const* __restrict__ copyIdx, int64_t const* __restrict__ poolIds, bool const* __restrict__ validPool, - int32_t const* __restrict__ scales, int32_t const* __restrict__ layerOffsets, - int32_t const* __restrict__ scratchPages, int32_t const* __restrict__ scratchBegs, - int32_t const* __restrict__ scratchEnds, int32_t const* __restrict__ scratchSlots, - int32_t const* __restrict__ numContexts, int32_t* __restrict__ output, int32_t numPools, int32_t copyIdxCapacity, - int32_t numAttnTypes, int32_t numTables, int32_t maxBlocksPerSeq, int32_t scratchCapacity, int32_t maxScratchSlots) -{ - int32_t const rowIdx = static_cast<int32_t>(blockIdx.x); - int32_t const tableId = rowIdx % numTables; - int32_t const layerAttnIdx = rowIdx / numTables; - int32_t const attnTypeId = layerAttnIdx % numAttnTypes; - int32_t const layerId = layerAttnIdx / numAttnTypes; - int32_t const layerAttnOffset = layerId * numAttnTypes + attnTypeId; - int32_t const outputOffset = rowIdx * maxBlocksPerSeq; - - int64_t const poolId64 = poolIds[layerAttnOffset]; - bool const isValidPool = validPool[layerAttnOffset] && poolId64 >= 0 && poolId64 < numPools; - if (!isValidPool) - { - for (int32_t blockId = threadIdx.x; blockId < maxBlocksPerSeq; blockId += blockDim.x) - { - output[outputOffset + blockId] = kBadPageIndex; - } - return; - } - - auto const poolId = static_cast<int32_t>(poolId64); - int32_t const scale = scales[layerAttnOffset]; - int32_t const layerOffset = layerOffsets[layerAttnOffset]; - int32_t const activeContexts = numContexts[0]; - bool const canUseScratch = tableId < scratchCapacity && tableId < activeContexts; - int64_t const scratchRangeOffset = static_cast<int64_t>(poolId) * scratchCapacity + tableId; - int32_t const scratchBeg = canUseScratch ? scratchBegs[scratchRangeOffset] : 0; - int32_t const scratchEnd = canUseScratch ? scratchEnds[scratchRangeOffset] : 0; - int32_t const scratchPageCount = scratchPages[layerAttnOffset]; - - int32_t const mappedTableId = copyIdx[tableId]; - bool const isValidTable = mappedTableId >= 0 && mappedTableId < copyIdxCapacity; - int64_t const blockOffsetsOffset - = ((static_cast<int64_t>(poolId) * copyIdxCapacity + mappedTableId) * 2) * maxBlocksPerSeq; - - for (int32_t blockId = threadIdx.x; blockId < maxBlocksPerSeq; blockId += blockDim.x) - { - bool const inScratchRange = canUseScratch && blockId >= scratchBeg && blockId < scratchEnd; - if (inScratchRange) - { - int32_t const rangeIndex = blockId - scratchBeg; - int32_t const totalOffset = rangeIndex * scratchPageCount; - int32_t slotIdx = totalOffset / scale; - if (slotIdx >= maxScratchSlots) - { - slotIdx = maxScratchSlots - 1; - } - - int64_t const slotOffset = scratchRangeOffset * maxScratchSlots + slotIdx; - int32_t const slotId = scratchSlots[slotOffset]; - int32_t const offset = totalOffset % scale; - output[outputOffset + blockId] = slotId * scale + ((offset + layerOffset) % scale); - continue; - } - - if (!isValidTable) - { - output[outputOffset + blockId] = kBadPageIndex; - continue; - } - - int32_t const base = blockOffsets[blockOffsetsOffset + blockId]; - output[outputOffset + blockId] = base == kBadPageIndex ? kBadPageIndex : base * scale + layerOffset; - } -} - -} // namespace - -void invokeDeepseekV4ComputeSlidingBlockTables(int32_t const* blockOffsets, int32_t const* copyIdx, - int64_t const* poolIds, bool const* validPool, int32_t const* scales, int32_t const* layerOffsets, int32_t* output, - int32_t numPools, int32_t copyIdxCapacity, int32_t numLayers, int32_t numAttnTypes, int32_t numTables, - int32_t maxBlocksPerSeq, cudaStream_t stream) -{ - int64_t const totalElements = static_cast<int64_t>(numLayers) * numAttnTypes * numTables * maxBlocksPerSeq; - if (totalElements == 0) - { - return; - } - - int32_t const numLayerAttn = numLayers * numAttnTypes; - int32_t const itemsPerRow = maxBlocksPerSeq % 4 == 0 ? maxBlocksPerSeq / 4 : maxBlocksPerSeq; - int32_t threadsPerBlock = itemsPerRow >= kVecThreadsPerBlock ? kVecThreadsPerBlock : itemsPerRow; - if (threadsPerBlock < 64) - { - threadsPerBlock = 64; - } - - dim3 const block(static_cast<uint32_t>(threadsPerBlock)); - dim3 const grid(static_cast<uint32_t>((numTables + kVecRowsPerBlock - 1) / kVecRowsPerBlock), - static_cast<uint32_t>(numLayerAttn)); - computeSlidingBlockTablesRowsTiledKernel<<<grid, block, 0, stream>>>(blockOffsets, copyIdx, poolIds, validPool, - scales, layerOffsets, output, numPools, copyIdxCapacity, numLayerAttn, numTables, maxBlocksPerSeq); -} - -void invokeDeepseekV4ComputeSlidingBlockTablesWithScratch(int32_t const* blockOffsets, int32_t const* copyIdx, - int64_t const* poolIds, bool const* validPool, int32_t const* scales, int32_t const* layerOffsets, - int32_t const* scratchPages, int32_t const* scratchBegs, int32_t const* scratchEnds, int32_t const* scratchSlots, - int32_t const* numContexts, int32_t* output, int32_t numPools, int32_t copyIdxCapacity, int32_t numLayers, - int32_t numAttnTypes, int32_t numTables, int32_t maxBlocksPerSeq, int32_t scratchCapacity, int32_t maxScratchSlots, - cudaStream_t stream) -{ - int64_t const totalElements = static_cast<int64_t>(numLayers) * numAttnTypes * numTables * maxBlocksPerSeq; - if (totalElements == 0) - { - return; - } - - if (maxBlocksPerSeq >= kRowKernelMinBlocks) - { - int32_t const numRows = numLayers * numAttnTypes * numTables; - dim3 const block(kThreadsPerBlock); - dim3 const grid(static_cast<uint32_t>(numRows)); - computeSlidingBlockTablesWithScratchRowsKernel<<<grid, block, 0, stream>>>(blockOffsets, copyIdx, poolIds, - validPool, scales, layerOffsets, scratchPages, scratchBegs, scratchEnds, scratchSlots, numContexts, output, - numPools, copyIdxCapacity, numAttnTypes, numTables, maxBlocksPerSeq, scratchCapacity, maxScratchSlots); - return; - } - - dim3 const block(kThreadsPerBlock); - dim3 const grid(static_cast<uint32_t>((totalElements + kThreadsPerBlock - 1) / kThreadsPerBlock)); - computeSlidingBlockTablesWithScratchKernel<<<grid, block, 0, stream>>>(blockOffsets, copyIdx, poolIds, validPool, - scales, layerOffsets, scratchPages, scratchBegs, scratchEnds, scratchSlots, numContexts, output, totalElements, - numPools, copyIdxCapacity, numAttnTypes, numTables, maxBlocksPerSeq, scratchCapacity, maxScratchSlots); -} - -} // namespace kernels - -TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/deepseekV4BlockTable.h b/cpp/tensorrt_llm/kernels/deepseekV4BlockTable.h deleted file mode 100644 index f57b757faaf3..000000000000 --- a/cpp/tensorrt_llm/kernels/deepseekV4BlockTable.h +++ /dev/null @@ -1,43 +0,0 @@ -/* - * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#pragma once - -#include "tensorrt_llm/common/config.h" - -#include <cstdint> -#include <cuda_runtime.h> - -TRTLLM_NAMESPACE_BEGIN - -namespace kernels -{ - -void invokeDeepseekV4ComputeSlidingBlockTables(int32_t const* blockOffsets, int32_t const* copyIdx, - int64_t const* poolIds, bool const* validPool, int32_t const* scales, int32_t const* layerOffsets, int32_t* output, - int32_t numPools, int32_t copyIdxCapacity, int32_t numLayers, int32_t numAttnTypes, int32_t numTables, - int32_t maxBlocksPerSeq, cudaStream_t stream); - -void invokeDeepseekV4ComputeSlidingBlockTablesWithScratch(int32_t const* blockOffsets, int32_t const* copyIdx, - int64_t const* poolIds, bool const* validPool, int32_t const* scales, int32_t const* layerOffsets, - int32_t const* scratchPages, int32_t const* scratchBegs, int32_t const* scratchEnds, int32_t const* scratchSlots, - int32_t const* numContexts, int32_t* output, int32_t numPools, int32_t copyIdxCapacity, int32_t numLayers, - int32_t numAttnTypes, int32_t numTables, int32_t maxBlocksPerSeq, int32_t scratchCapacity, int32_t maxScratchSlots, - cudaStream_t stream); - -} // namespace kernels - -TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/fmhaDispatcher.cpp b/cpp/tensorrt_llm/kernels/fmhaDispatcher.cpp index eb8f258d0f1d..2a914e16d41a 100644 --- a/cpp/tensorrt_llm/kernels/fmhaDispatcher.cpp +++ b/cpp/tensorrt_llm/kernels/fmhaDispatcher.cpp @@ -58,9 +58,9 @@ FmhaDispatcher::FmhaDispatcher(MHARunnerFixedParams fixedParams) if (mUseTllmGen) { auto [dataTypeK, dataTypeV] = unpack_kv_data_type(mFixedParams.dataTypeKv); - mTllmGenFMHARunner.reset(new TllmGenFmhaRunner(mFixedParams.dataType, dataTypeK, dataTypeV, - mFixedParams.dataTypeOut, mFixedParams.sageBlockSizeQ, mFixedParams.sageBlockSizeK, 0, - mFixedParams.sageBlockSizeV, mFixedParams.fusesDsv4InvRopeFp8Quant)); + mTllmGenFMHARunner.reset( + new TllmGenFmhaRunner(mFixedParams.dataType, dataTypeK, dataTypeV, mFixedParams.dataTypeOut, + mFixedParams.sageBlockSizeQ, mFixedParams.sageBlockSizeK, 0, mFixedParams.sageBlockSizeV)); if (!isSupported()) { TLLM_LOG_WARNING("TRTLLM-GEN does not support the requested kernels."); @@ -141,7 +141,6 @@ bool FmhaDispatcher::isSupported() = (mFixedParams.useSparseMLA && mFixedParams.headSizeV == mFixedParams.headSize) ? SparseType::DynamicTokenSparse : SparseType::StaticTokenSparse; - tllmRunnerParams.mDsv4EpilogueFusion.enabled = mFixedParams.fusesDsv4InvRopeFp8Quant; tllmRunnerParams.mKernelType = FmhaKernelType::Generation; tllmRunnerParams.mMaskType = TrtllmGenAttentionMaskType::Causal; // Generation-style kernels on long KV can pick MultiCtasKv cubins @@ -222,12 +221,6 @@ void FmhaDispatcher::run(MHARunnerParams runnerParams) tllmRunnerParams.oSfScalePtr = runnerParams.oSfScalePtr; tllmRunnerParams.oPtr = runnerParams.outputPtr; tllmRunnerParams.oSfPtr = runnerParams.outputSfPtr; - if (runnerParams.dsv4EpilogueFusion.enabled) - { - tllmRunnerParams.mDsv4EpilogueFusion.enabled = true; - tllmRunnerParams.mDsv4EpilogueFusion.cosSinCache = runnerParams.dsv4EpilogueFusion.cosSinCache; - tllmRunnerParams.mDsv4EpilogueFusion.scaleBufM = runnerParams.dsv4EpilogueFusion.scaleBufM; - } // The sequence lengths for K/V. tllmRunnerParams.seqLensKvPtr = reinterpret_cast<int const*>(runnerParams.kvSeqLenPtr); // Assume same headDim for Qk and V here. diff --git a/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu b/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu index 6e26dfce65bf..e06b0f200e4b 100644 --- a/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu +++ b/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2025-2026, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -174,17 +174,6 @@ __global__ void fusedQKNormRopeKernel( // pos_id is selected per rotary half-dim (interleaved mRoPE); for plain RoPE // selectMRopePosId always returns position_ids[tokenIdx]. - // Hoist log2(base) and the loop-invariant constants out of the per-thread - // per-elem loop. powf(base, -2*hd/rd) == exp2f(-2*hd/rd * log2(base)); base - // and rotary_dim are kernel-uniform, so one MUFU.LG2 per warp instead of - // one per (thread, iter). Uses the fast __log2f intrinsic (a few ULPs of - // error, absorbed by bf16 downcast at store time). - float const neg2_log2base_over_rd = -2.0f * __log2f(base) / static_cast<float>(rotary_dim); - // rotary_dim is even by contract; when it's also a power of 2 (always in - // practice — 64/128/256) '% rotary_dim' becomes '& (rotary_dim - 1)'. - // The bool is warp-uniform → predicated select, no branch divergence. - int const rd_mask = rotary_dim - 1; - bool const rd_is_pow2 = ((rotary_dim & rd_mask) == 0); // TODO: cos sin calculation could be halved. if constexpr (interleave) { @@ -202,7 +191,7 @@ __global__ void fusedQKNormRopeKernel( int dim_idx = laneId * numElemsPerThread + i; int half_dim = dim_idx / 2; - float freq = exp2f(static_cast<float>(half_dim) * neg2_log2base_over_rd); + float freq = powf(base, -2.0f * half_dim / static_cast<float>(rotary_dim)); if (factor != 1.0f) { @@ -243,9 +232,9 @@ __global__ void fusedQKNormRopeKernel( } int dim_idx = laneId * numElemsPerThread + i; - dim_idx = rd_is_pow2 ? ((dim_idx * 2) & rd_mask) : ((dim_idx * 2) % rotary_dim); + dim_idx = (dim_idx * 2) % rotary_dim; int half_dim = dim_idx / 2; - float freq = exp2f(static_cast<float>(half_dim) * neg2_log2base_over_rd); + float freq = powf(base, -2.0f * half_dim / static_cast<float>(rotary_dim)); if (factor != 1.0f) { diff --git a/cpp/tensorrt_llm/kernels/gptKernels.h b/cpp/tensorrt_llm/kernels/gptKernels.h index d855aade79c2..e13e9bca4d6a 100644 --- a/cpp/tensorrt_llm/kernels/gptKernels.h +++ b/cpp/tensorrt_llm/kernels/gptKernels.h @@ -16,7 +16,6 @@ #pragma once #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h" #include "tensorrt_llm/runtime/iTensor.h" #include <cstdint> @@ -228,11 +227,11 @@ struct BuildDecoderInfoParams std::string toString() const { std::stringstream ss; - auto printTensor = [&ss](char const* name, void* ptr, tensorrt_llm::Dims shape) + auto printTensor = [&ss](char const* name, void* ptr, nvinfer1::Dims shape) { ss << name << ": "; if (ptr) - ss << *(runtime::ITensor::wrap((void*) ptr, tensorrt_llm::DataType::kINT32, shape)); + ss << *(runtime::ITensor::wrap((void*) ptr, nvinfer1::DataType::kINT32, shape)); else ss << "nullptr"; ss << std::endl; diff --git a/cpp/tensorrt_llm/kernels/groupGemm.cu b/cpp/tensorrt_llm/kernels/groupGemm.cu index b41021ffe6f7..5b8c0d929150 100644 --- a/cpp/tensorrt_llm/kernels/groupGemm.cu +++ b/cpp/tensorrt_llm/kernels/groupGemm.cu @@ -28,7 +28,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" TRTLLM_NAMESPACE_BEGIN @@ -64,7 +63,7 @@ template <int M1, int N1, int K1, int M2, int N2, int K2, typename cutlassType, void groupedGemm_(std::vector<cutlass::gemm::GemmCoord> problem_sizes, std::vector<void*> const& ptrA, std::vector<void*> const& ptrB, std::vector<void*> const& ptrC, std::vector<void*> const& ptrD, void* gemmParamsWorkSpace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkspaceSize, - tensorrt_llm::DataType dataType, cudaStream_t stream) + nvinfer1::DataType dataType, cudaStream_t stream) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); using ElementA = cutlassType; @@ -179,20 +178,20 @@ template <int M1, int N1, int K1, int M2, int N2, int K2, int kAlignmentAB, int void groupedGemmType_(std::vector<cutlass::gemm::GemmCoord> problem_sizes, std::vector<void*> const& ptrA, std::vector<void*> const& ptrB, std::vector<void*> const& ptrC, std::vector<void*> const& ptrD, void* gemmParamsWorkSpace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkspaceSize, - tensorrt_llm::DataType dataType, cudaStream_t stream) + nvinfer1::DataType dataType, cudaStream_t stream) { - if (dataType == tensorrt_llm::DataType::kHALF) + if (dataType == nvinfer1::DataType::kHALF) { groupedGemm_<M1, N1, K1, M2, N2, K2, cutlass::half_t, kAlignmentAB, kAlignmentC, kStages>(problem_sizes, ptrA, ptrB, ptrC, ptrD, gemmParamsWorkSpace, gemmParamsWorkSpaceSize, gemmWorkSpace, gemmWorkspaceSize, dataType, stream); } - else if (dataType == tensorrt_llm::DataType::kFLOAT) + else if (dataType == nvinfer1::DataType::kFLOAT) { TLLM_CHECK_WITH_INFO(false, "not support float input/output"); } #ifdef ENABLE_BF16 - else if (dataType == tensorrt_llm::DataType::kBF16) + else if (dataType == nvinfer1::DataType::kBF16) { groupedGemm_<M1, N1, K1, M2, N2, K2, cutlass::bfloat16_t, kAlignmentAB, kAlignmentC, kStages>(problem_sizes, ptrA, ptrB, ptrC, ptrD, gemmParamsWorkSpace, gemmParamsWorkSpaceSize, gemmWorkSpace, gemmWorkspaceSize, @@ -204,7 +203,7 @@ void groupedGemmType_(std::vector<cutlass::gemm::GemmCoord> problem_sizes, std:: void groupedGemm(std::vector<cutlass::gemm::GemmCoord> problem_sizes, std::vector<void*> const& ptrA, std::vector<void*> const& ptrB, std::vector<void*> const& ptrC, std::vector<void*> const& ptrD, void* gemmParamsWorkSpace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkspaceSize, - bool isLoraIn, tensorrt_llm::DataType dataType, int minKN, cudaStream_t stream) + bool isLoraIn, nvinfer1::DataType dataType, int minKN, cudaStream_t stream) { TLLM_LOG_TRACE("%s start, isLoraIn: %d, minKN = %d", __PRETTY_FUNCTION__, static_cast<int>(isLoraIn), minKN); if (isLoraIn) diff --git a/cpp/tensorrt_llm/kernels/groupGemm.h b/cpp/tensorrt_llm/kernels/groupGemm.h index c526e08e986c..dbc1e498b7b2 100644 --- a/cpp/tensorrt_llm/kernels/groupGemm.h +++ b/cpp/tensorrt_llm/kernels/groupGemm.h @@ -17,7 +17,7 @@ #include "cutlass/gemm_coord.h" #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> TRTLLM_NAMESPACE_BEGIN @@ -29,7 +29,7 @@ int64_t getGroupedGemmParamsWorkSpaceSize(int64_t problem_count); void groupedGemm(std::vector<cutlass::gemm::GemmCoord> problem_sizes, std::vector<void*> const& ptrA, std::vector<void*> const& ptrB, std::vector<void*> const& ptrC, std::vector<void*> const& ptrD, void* gemmParamsWorkspace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkspaceSize, - bool isLoraIn, tensorrt_llm::DataType dataType, int minKN, cudaStream_t stream); + bool isLoraIn, nvinfer1::DataType dataType, int minKN, cudaStream_t stream); } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.cu b/cpp/tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.cu index 3332c2918a3f..409968bb510d 100644 --- a/cpp/tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.cu +++ b/cpp/tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.cu @@ -22,7 +22,6 @@ #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/memoryUtils.h" #include "tensorrt_llm/common/reduceKernelUtils.cuh" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.h" TRTLLM_NAMESPACE_BEGIN @@ -656,9 +655,9 @@ void GroupRMSNormBaseKernelLauncher(GroupRMSParams<n>& params) switch (params.dtype) { - case tensorrt_llm::DataType::kHALF: GROUP_RMS_NORM_DISPATCH(half); break; - case tensorrt_llm::DataType::kBF16: GROUP_RMS_NORM_DISPATCH(__nv_bfloat16); break; - case tensorrt_llm::DataType::kFLOAT: GROUP_RMS_NORM_DISPATCH(float); break; + case nvinfer1::DataType::kHALF: GROUP_RMS_NORM_DISPATCH(half); break; + case nvinfer1::DataType::kBF16: GROUP_RMS_NORM_DISPATCH(__nv_bfloat16); break; + case nvinfer1::DataType::kFLOAT: GROUP_RMS_NORM_DISPATCH(float); break; default: TLLM_CHECK_WITH_INFO(false, "Unsupported data type for GroupRMSNorm"); } @@ -751,9 +750,9 @@ void GroupRMSNormKernelLargeBatchLauncher(GroupRMSParams<n>& params) switch (params.dtype) { - case tensorrt_llm::DataType::kHALF: GROUP_RMS_NORM_LARGE_BATCH_DISPATCH(half); break; - case tensorrt_llm::DataType::kBF16: GROUP_RMS_NORM_LARGE_BATCH_DISPATCH(__nv_bfloat16); break; - case tensorrt_llm::DataType::kFLOAT: GROUP_RMS_NORM_LARGE_BATCH_DISPATCH(float); break; + case nvinfer1::DataType::kHALF: GROUP_RMS_NORM_LARGE_BATCH_DISPATCH(half); break; + case nvinfer1::DataType::kBF16: GROUP_RMS_NORM_LARGE_BATCH_DISPATCH(__nv_bfloat16); break; + case nvinfer1::DataType::kFLOAT: GROUP_RMS_NORM_LARGE_BATCH_DISPATCH(float); break; default: TLLM_CHECK_WITH_INFO(false, "Unsupported data type for GroupRMSNormV2"); } @@ -814,15 +813,15 @@ void GroupRMSNormKernelLauncherWithHeuristic(GroupRMSParams<n>& params) // Choose the appropriate DType switch (params.dtype) { - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: base_warps = calculateNumWarpsBase<half, n>(params); large_batch_warps = calculateNumWarpsLargeBatch<half, n>(params).num_warps_to_launch; break; - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: base_warps = calculateNumWarpsBase<__nv_bfloat16, n>(params); large_batch_warps = calculateNumWarpsLargeBatch<__nv_bfloat16, n>(params).num_warps_to_launch; break; - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: base_warps = calculateNumWarpsBase<float, n>(params); large_batch_warps = calculateNumWarpsLargeBatch<float, n>(params).num_warps_to_launch; break; diff --git a/cpp/tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.h b/cpp/tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.h index 70425f924217..335adf44ed67 100644 --- a/cpp/tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.h +++ b/cpp/tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.h @@ -15,7 +15,7 @@ */ #pragma once #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_bf16.h> #include <cuda_fp16.h> #include <map> @@ -44,7 +44,7 @@ struct GroupRMSParams float eps; float weight_bias; bool enable_weights; - tensorrt_llm::DataType dtype; + nvinfer1::DataType dtype; cudaStream_t stream; }; diff --git a/cpp/tensorrt_llm/kernels/internal_cutlass_kernels/include/moe_kernels.h b/cpp/tensorrt_llm/kernels/internal_cutlass_kernels/include/moe_kernels.h index 0b02686f5e53..132990603db9 100644 --- a/cpp/tensorrt_llm/kernels/internal_cutlass_kernels/include/moe_kernels.h +++ b/cpp/tensorrt_llm/kernels/internal_cutlass_kernels/include/moe_kernels.h @@ -26,7 +26,7 @@ #ifdef ENABLE_FP4 #include <cuda_fp4.h> #endif -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <array> #include <cuda_runtime_api.h> #include <map> @@ -869,8 +869,8 @@ struct GemmProfilerBackend GEMM_2 }; - void init(CutlassMoeFCRunnerInterface& runner, GemmToProfile gemm_to_profile, tensorrt_llm::DataType dtype, - tensorrt_llm::DataType wtype, tensorrt_llm::DataType otype, int num_experts, int k, int64_t hidden_size, + void init(CutlassMoeFCRunnerInterface& runner, GemmToProfile gemm_to_profile, nvinfer1::DataType dtype, + nvinfer1::DataType wtype, nvinfer1::DataType otype, int num_experts, int k, int64_t hidden_size, int64_t inter_size, int64_t group_size, ActivationType activation_type, bool bias, bool use_lora, bool min_latency_mode, bool need_weights, MOEParallelismConfig parallelism_config) { @@ -895,13 +895,13 @@ struct GemmProfilerBackend mSorter.updateNumExperts(mNumExpertsPerNode); mScalingType = TmaWarpSpecializedGroupedGemmInput::FpXBlockScalingType::NONE; - if (dtype == tensorrt_llm::DataType::kFP8 - && (wtype == tensorrt_llm::DataType::kFP4 || wtype == tensorrt_llm::DataType::kINT64)) + if (dtype == nvinfer1::DataType::kFP8 + && (wtype == nvinfer1::DataType::kFP4 || wtype == nvinfer1::DataType::kINT64)) { mScalingType = TmaWarpSpecializedGroupedGemmInput::FpXBlockScalingType::MXFPX; } - else if ((dtype == tensorrt_llm::DataType::kFP4 || dtype == tensorrt_llm::DataType::kINT64) - && (wtype == tensorrt_llm::DataType::kFP4 || wtype == tensorrt_llm::DataType::kINT64)) + else if ((dtype == nvinfer1::DataType::kFP4 || dtype == nvinfer1::DataType::kINT64) + && (wtype == nvinfer1::DataType::kFP4 || wtype == nvinfer1::DataType::kINT64)) { mScalingType = TmaWarpSpecializedGroupedGemmInput::FpXBlockScalingType::NVFP4; } @@ -932,9 +932,9 @@ struct GemmProfilerBackend int mSampleIndex = 0; - tensorrt_llm::DataType mDType{}; - tensorrt_llm::DataType mWType{}; - tensorrt_llm::DataType mOType{}; + nvinfer1::DataType mDType{}; + nvinfer1::DataType mWType{}; + nvinfer1::DataType mOType{}; // This will be a unique value for every iteration of warmup and actual bench constexpr static int64_t NUM_ROUTING_SAMPLES = 16; diff --git a/cpp/tensorrt_llm/kernels/kvCachePartialCopy.cu b/cpp/tensorrt_llm/kernels/kvCachePartialCopy.cu index 04105721dfca..3b91cf3f1776 100644 --- a/cpp/tensorrt_llm/kernels/kvCachePartialCopy.cu +++ b/cpp/tensorrt_llm/kernels/kvCachePartialCopy.cu @@ -15,7 +15,6 @@ */ #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/kvCachePartialCopy.h" #include <cstdint> #include <cuda_runtime_api.h> @@ -90,42 +89,42 @@ void kvCacheBlockPartialCopy(IBuffer& dst, IBuffer const& src, unsigned int numL TLLM_CHECK_WITH_INFO(dataType == dst.getDataType(), "src and dst dataType does not match"); switch (dataType) { - case tensorrt_llm::DataType::kINT64: + case nvinfer1::DataType::kINT64: hostKVCacheBlockPartialCopy<SizeType64>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; - case tensorrt_llm::DataType::kINT32: + case nvinfer1::DataType::kINT32: hostKVCacheBlockPartialCopy<std::int32_t>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: hostKVCacheBlockPartialCopy<float>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: hostKVCacheBlockPartialCopy<__nv_bfloat16>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; #endif - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: hostKVCacheBlockPartialCopy<half>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; - case tensorrt_llm::DataType::kBOOL: + case nvinfer1::DataType::kBOOL: hostKVCacheBlockPartialCopy<bool>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; - case tensorrt_llm::DataType::kUINT8: + case nvinfer1::DataType::kUINT8: hostKVCacheBlockPartialCopy<std::uint8_t>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; - case tensorrt_llm::DataType::kINT8: + case nvinfer1::DataType::kINT8: hostKVCacheBlockPartialCopy<std::int8_t>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; #ifdef ENABLE_FP8 - case tensorrt_llm::DataType::kFP8: + case nvinfer1::DataType::kFP8: hostKVCacheBlockPartialCopy<__nv_fp8_e4m3>( dst, src, numLayers, numHeads, tokensPerBlock, numHidden, numTokensToCopy, kvFactor, stream); break; diff --git a/cpp/tensorrt_llm/kernels/lora/dora.cpp b/cpp/tensorrt_llm/kernels/lora/dora.cpp index 883d02df9291..43dbf4fdccb0 100644 --- a/cpp/tensorrt_llm/kernels/lora/dora.cpp +++ b/cpp/tensorrt_llm/kernels/lora/dora.cpp @@ -20,13 +20,13 @@ #include "tensorrt_llm/common/memoryUtils.h" #include "tensorrt_llm/kernels/doraScaling.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <numeric> #include <vector> using tensorrt_llm::kernels::DoraImpl; -DoraImpl::DoraImpl(std::vector<int> const& outHiddenSizes, tensorrt_llm::DataType type) +DoraImpl::DoraImpl(std::vector<int> const& outHiddenSizes, nvinfer1::DataType type) : mType(type) { mCumModuleSizes.resize(outHiddenSizes.size()); @@ -73,14 +73,14 @@ int DoraImpl::run(int64_t numTokens, void const* input, void const* const* loraW auto const* deviceCumModuleSizes = reinterpret_cast<int64_t const*>(workspace); auto const* deviceScalePtrs = reinterpret_cast<void const* const*>((&deviceCumModuleSizes[numModules])); - if (mType == tensorrt_llm::DataType::kHALF) + if (mType == nvinfer1::DataType::kHALF) { tokenPerChannelScale<half>(numel, numModules, numTokens, deviceCumModuleSizes, reinterpret_cast<half const*>(input), reinterpret_cast<half const* const*>(deviceScalePtrs), reinterpret_cast<half*>(outputs[0]), stream); } #ifdef ENABLE_BF16 - else if (mType == tensorrt_llm::DataType::kBF16) + else if (mType == nvinfer1::DataType::kBF16) { tokenPerChannelScale<nv_bfloat16>(numel, numModules, numTokens, deviceCumModuleSizes, reinterpret_cast<nv_bfloat16 const*>(input), reinterpret_cast<nv_bfloat16 const* const*>(deviceScalePtrs), diff --git a/cpp/tensorrt_llm/kernels/lora/dora.h b/cpp/tensorrt_llm/kernels/lora/dora.h index 02cd68e7c1f0..fc21fe669366 100644 --- a/cpp/tensorrt_llm/kernels/lora/dora.h +++ b/cpp/tensorrt_llm/kernels/lora/dora.h @@ -17,7 +17,7 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> TRTLLM_NAMESPACE_BEGIN @@ -28,7 +28,7 @@ class DoraImpl public: DoraImpl() = delete; - DoraImpl(std::vector<int> const& outHiddenSizes, tensorrt_llm::DataType type); + DoraImpl(std::vector<int> const& outHiddenSizes, nvinfer1::DataType type); ~DoraImpl() = default; @@ -41,7 +41,7 @@ class DoraImpl private: std::vector<int64_t> mCumModuleSizes; std::vector<int64_t> mHostBuf; - tensorrt_llm::DataType mType; + nvinfer1::DataType mType; }; } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/lora/lora.cpp b/cpp/tensorrt_llm/kernels/lora/lora.cpp index 7a2b7d330afc..61f6af00fedc 100644 --- a/cpp/tensorrt_llm/kernels/lora/lora.cpp +++ b/cpp/tensorrt_llm/kernels/lora/lora.cpp @@ -19,7 +19,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/groupGemm.h" #include "tensorrt_llm/kernels/lora/lora.h" #include "tensorrt_llm/kernels/splitkGroupGemm.h" @@ -49,9 +48,8 @@ void _getProblemParams(cublasOperation_t& transa, cublasOperation_t& transb, int // TODO should reuse the function in gemmPlugin void _runGemm(int const M, int const N, int const K, bool const transA, bool const transB, - tensorrt_llm::DataType const type, CublasGemmWrapperPtr const& cublasWrapperPtr, void const* act, - void const* weight, void* output, std::optional<cublasLtMatmulHeuristicResult_t> const& heuristic, void* workspace, - cudaStream_t stream) + nvinfer1::DataType const type, CublasGemmWrapperPtr const& cublasWrapperPtr, void const* act, void const* weight, + void* output, std::optional<cublasLtMatmulHeuristicResult_t> const& heuristic, void* workspace, cudaStream_t stream) { cublasWrapperPtr->setStream(stream); cublasWrapperPtr->setWorkspace(workspace); @@ -67,8 +65,7 @@ void _runGemm(int const M, int const N, int const K, bool const transA, bool con } LoraImpl::LoraImpl(int in_hidden_size, std::vector<int> out_hidden_sizes, bool transA, bool transB, - int num_lora_modules, tensorrt_llm::DataType type, int max_low_rank, - std::shared_ptr<CublasGemmWrapper> cublasWrapper) + int num_lora_modules, nvinfer1::DataType type, int max_low_rank, std::shared_ptr<CublasGemmWrapper> cublasWrapper) : mInHiddenSize(in_hidden_size) , mTransA(transA) , mTransB(transB) @@ -85,16 +82,16 @@ LoraImpl::LoraImpl(int in_hidden_size, std::vector<int> out_hidden_sizes, bool t void LoraImpl::setGemmConfig() { TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); - if (mType == tensorrt_llm::DataType::kHALF) + if (mType == nvinfer1::DataType::kHALF) { mCublasWrapper->setFP16GemmConfig(); } - else if (mType == tensorrt_llm::DataType::kFLOAT) + else if (mType == nvinfer1::DataType::kFLOAT) { mCublasWrapper->setFP32GemmConfig(); } #ifdef ENABLE_BF16 - else if (mType == tensorrt_llm::DataType::kBF16) + else if (mType == nvinfer1::DataType::kBF16) { mCublasWrapper->setBF16GemmConfig(); } @@ -124,7 +121,7 @@ int64_t getGemmWorkSpaceSize(int64_t numTokens, int64_t maxLoraModuleNum, int64_ } size_t LoraImpl::getWorkspaceSize( - int64_t const numTokens, int64_t const numReqs, tensorrt_llm::DataType const type) const noexcept + int64_t const numTokens, int64_t const numReqs, nvinfer1::DataType const type) const noexcept { TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); auto const typeSize = tensorrt_llm::common::getDTypeSize(type); diff --git a/cpp/tensorrt_llm/kernels/lora/lora.h b/cpp/tensorrt_llm/kernels/lora/lora.h index 73a2cbe330be..7215a7af74d4 100644 --- a/cpp/tensorrt_llm/kernels/lora/lora.h +++ b/cpp/tensorrt_llm/kernels/lora/lora.h @@ -19,7 +19,7 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cublasMMWrapper.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cassert> #include <vector> @@ -37,10 +37,9 @@ class LoraImpl { public: LoraImpl(int in_hidden_size, std::vector<int> out_hidden_sizes, bool transA, bool transB, int num_lora_modules, - tensorrt_llm::DataType type, int max_low_rank, std::shared_ptr<CublasGemmWrapper> cublasWrapper); + nvinfer1::DataType type, int max_low_rank, std::shared_ptr<CublasGemmWrapper> cublasWrapper); - [[nodiscard]] size_t getWorkspaceSize( - int64_t numTokens, int64_t numReqs, tensorrt_llm::DataType type) const noexcept; + [[nodiscard]] size_t getWorkspaceSize(int64_t numTokens, int64_t numReqs, nvinfer1::DataType type) const noexcept; void setBestTactic(std::optional<Config> config); int run(int64_t numTokens, int64_t numReqs, void const* input, int32_t const* loraRanks, void const* const* loraWeightsPtr, int weightIndex, void* const* outputs, void* workspace, cudaStream_t stream); @@ -55,7 +54,7 @@ class LoraImpl private: bool mTransA; bool mTransB; - tensorrt_llm::DataType mType; + nvinfer1::DataType mType; int mNumLoraModules; // @fixme: seems this is shared across multiple clones. diff --git a/cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.cu b/cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.cu index 1165f39b3e5d..c3276ea487bc 100644 --- a/cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.cu +++ b/cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.cu @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include <cub/block/block_load.cuh> #include <cub/block/block_scan.cuh> @@ -359,7 +358,7 @@ void launchLoraGroupGEMMParamFillRowReorderFusion(int32_t* in_sizes, int32_t* ou int64_t a_base, int64_t d_base, int64_t d_prime_base, int32_t const* slot_counts, int32_t const* slot_ranks, int64_t const* slot_offsets, int32_t const* module_out_sizes, int64_t const* module_out_prefix, int64_t const* b_ptrs, int64_t const* b_prime_ptrs, void const* input, int64_t const* sorted_ids, - int32_t module_count, tensorrt_llm::DataType dtype, cudaStream_t stream) + int32_t module_count, nvinfer1::DataType dtype, cudaStream_t stream) { // Determine block dimensions (1D) // Requirements: 1) >= max_lora_count * module_count 2) >= 256 3) divisible by 32 diff --git a/cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.h b/cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.h index 835b9f8bed96..3043054ca4b5 100644 --- a/cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.h +++ b/cpp/tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.h @@ -17,7 +17,7 @@ #pragma once #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cstdint> #include <cuda_runtime.h> @@ -70,7 +70,7 @@ void launchLoraGroupGEMMParamFillRowReorderFusion(int32_t* in_sizes, int32_t* ou int64_t a_base, int64_t d_base, int64_t d_prime_base, int32_t const* slot_counts, int32_t const* slot_ranks, int64_t const* slot_offsets, int32_t const* module_out_sizes, int64_t const* module_out_prefix, int64_t const* b_ptrs, int64_t const* b_prime_ptrs, void const* input, int64_t const* sorted_ids, - int32_t module_count, tensorrt_llm::DataType dtype, cudaStream_t stream); + int32_t module_count, nvinfer1::DataType dtype, cudaStream_t stream); } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/mhcKernels/mhcFusedHcKernel.cu b/cpp/tensorrt_llm/kernels/mhcKernels/mhcFusedHcKernel.cu index 234318b769f8..3be14448766b 100644 --- a/cpp/tensorrt_llm/kernels/mhcKernels/mhcFusedHcKernel.cu +++ b/cpp/tensorrt_llm/kernels/mhcKernels/mhcFusedHcKernel.cu @@ -187,13 +187,13 @@ static CUtensorMap makeTma2D(void* base, CUtensorMapDataType dtype, uint64_t gme // CUDA-graph capture: cuTensorMapEncodeTiled is a pure host function that does // not record any stream operation, so cache miss inside capture is safe. The // descriptor is passed by value as __grid_constant__; the recorded graph node -// holds those bytes and replays correctly while captured tensor addresses -// remain stable for the lifetime of the graph. +// holds those bytes and replays correctly under workspace-stable replay +// (already enforced by _FusedHcWorkspaceCache in mhc_cuda.py). // // Eviction: LRU bounded to kTmaDescCacheCap entries per thread. Eager mode // without CUDA-graph capture sees the PyTorch caching allocator hand out -// fresh `base` pointers as public outputs are allocated, so the unbounded -// version would grow across shape transitions. 128 entries × ~256 B = ~32 KB +// fresh `base` pointers when the workspace cache misses, so the unbounded +// version would grow on every shape transition. 128 entries × ~256 B = ~32 KB // per host thread — fits in L1, sized to cover the working set of any single // model (~4-8 distinct shapes × 4 descriptors each = O(20) live, with // headroom for shape transitions). diff --git a/cpp/tensorrt_llm/kernels/mlaKernels.h b/cpp/tensorrt_llm/kernels/mlaKernels.h index 1d081e0965a6..f4c0dbbadcd0 100644 --- a/cpp/tensorrt_llm/kernels/mlaKernels.h +++ b/cpp/tensorrt_llm/kernels/mlaKernels.h @@ -53,16 +53,6 @@ struct MlaMetaParams template <typename T> struct MlaParams { - struct Dsv4EpilogueFusionParams - { - // Enable DSv4 inverse-RoPE + FP8 quant epilogue fusion. - bool enabled = false; - // The cos/sin cache used by the fused inverse-RoPE epilogue. - float const* cos_sin_cache = nullptr; - // The physical token stride of the FP32 output scale tensor. - int32_t scale_buf_m = 0; - }; - T const* latent_cache; // cKV + k_pe // Tensor Q for both context and generation MLA, contiguous. Pre-process kernel will apply RoPE and modify it // in-place. For context MLA, shape: [total_q_len, h * (d_nope + d_rope)], stride: [h * (d_nope + d_rope), 1] @@ -126,9 +116,6 @@ struct MlaParams // skipped. Nope segment must be pre-filled (see deepseek_v4_q_norm_fused_fp8). bool fuse_q_fp8_in_rope = false; - // DSv4 fused inverse-RoPE + FP8 quant epilogue parameters. - Dsv4EpilogueFusionParams dsv4_epilogue_fusion; - // for Helix parallelism: the rotary position offsets [b] int32_t const* helix_position_offsets{nullptr}; diff --git a/cpp/tensorrt_llm/kernels/quantization.cuh b/cpp/tensorrt_llm/kernels/quantization.cuh index e4a5edab9b1f..89b96b288b46 100644 --- a/cpp/tensorrt_llm/kernels/quantization.cuh +++ b/cpp/tensorrt_llm/kernels/quantization.cuh @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2026, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2019-2023, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -279,34 +279,6 @@ constexpr int CVT_ELTS_PER_THREAD = 8; constexpr int CVT_FP4_THREADS_PER_WARP = 32; constexpr int CVT_FP8_TO_FP4_ELTS_PER_THREAD = 16; -// Membermask for the __shfl_xor_sync butterfly among the NUM_THREADS_PER_SF -// lanes that share one scale factor. The xor-1/xor-2 exchange never crosses -// this aligned lane group, so only the group has to converge on the shuffle. -// -// Do not widen the mask to the full warp. A sync shuffle waits until every -// lane named in the mask reaches the same call site, but here not every lane -// of a warp gets there: in quantize_with_block_size, lanes that drew padding -// columns (or ran out of columns) skip the cvt call and go wait at the CTA -// barrier at the end of the kernel. If the data/padding boundary cuts through -// a warp, a full-warp shuffle deadlocks against that barrier. A full mask can -// also name lanes that were never launched, because blockDim is not always a -// multiple of 32 (e.g. 200 threads leave the last warp with only 8 lanes); -// that is undefined behavior. -// -// The group mask is always safe: blockDim and every column boundary (data, -// padded, SF-padded) are multiples of the group size, so the lanes of one -// group always reach the same set of shuffle calls together. -template <int NUM_THREADS_PER_SF> -inline __device__ uint32_t cvt_sf_group_shfl_mask() -{ - static_assert(NUM_THREADS_PER_SF == 2 || NUM_THREADS_PER_SF == 4, "Unsupported SF group size."); - constexpr uint32_t groupSize = static_cast<uint32_t>(NUM_THREADS_PER_SF); - constexpr uint32_t groupMask = (1U << groupSize) - 1U; - uint32_t laneId = 0; - asm("mov.u32 %0, %%laneid;" : "=r"(laneId)); - return groupMask << (laneId & ~(groupSize - 1U)); -} - // Convert 8 float32 values into 8 e2m1 values (represented as one uint32_t). inline __device__ uint32_t fp32_vec_to_e2m1(float (&array)[8]) { @@ -467,11 +439,10 @@ __device__ uint32_t cvt_warp_fp16_to_fp4(PackedVec<Type>& vec, float SFScaleVal, constexpr int CVT_NUM_THREADS_PER_SF = SF_VEC_SIZE / CVT_ELTS_PER_THREAD; // Get the absolute maximum among all 16 values (two threads for 16, four threads for 32). - uint32_t const sfGroupMask = cvt_sf_group_shfl_mask<CVT_NUM_THREADS_PER_SF>(); - localMax = cuda_max(__shfl_xor_sync(sfGroupMask, localMax, 1), localMax); + localMax = cuda_max(__shfl_xor_sync(uint32_t(-1), localMax, 1), localMax); if constexpr (CVT_NUM_THREADS_PER_SF == 4) { - localMax = cuda_max(__shfl_xor_sync(sfGroupMask, localMax, 2), localMax); + localMax = cuda_max(__shfl_xor_sync(uint32_t(-1), localMax, 2), localMax); } // Get the final absolute maximum values. float vecMax = float(cuda_max(localMax.x, localMax.y)); @@ -569,7 +540,7 @@ __device__ uint64_t cvt_warp_fp8_to_fp4(PackedVec<Type>& vec, float SFScaleVal, if constexpr (CVT_NUM_THREADS_PER_SF == 2) { // For block 32, we need to reduce the local max across two threads. - localMax = __hmax2(__shfl_xor_sync(cvt_sf_group_shfl_mask<CVT_NUM_THREADS_PER_SF>(), localMax, 1), localMax); + localMax = __hmax2(__shfl_xor_sync(uint32_t(-1), localMax, 1), localMax); } // Get the final absolute maximum values. @@ -645,11 +616,10 @@ __device__ uint64_t cvt_warp_fp16_to_mxfp8(PackedVec<Type>& vec, uint8_t* SFout) constexpr int CVT_NUM_THREADS_PER_SF = SF_VEC_SIZE / CVT_ELTS_PER_THREAD; // Get the absolute maximum among all 16 values (two threads for 16, four threads for 32). - uint32_t const sfGroupMask = cvt_sf_group_shfl_mask<CVT_NUM_THREADS_PER_SF>(); - localMax = cuda_max(__shfl_xor_sync(sfGroupMask, localMax, 1), localMax); + localMax = cuda_max(__shfl_xor_sync(uint32_t(-1), localMax, 1), localMax); if constexpr (CVT_NUM_THREADS_PER_SF == 4) { - localMax = cuda_max(__shfl_xor_sync(sfGroupMask, localMax, 2), localMax); + localMax = cuda_max(__shfl_xor_sync(uint32_t(-1), localMax, 2), localMax); } // Get the final absolute maximum values. float vecMax = float(cuda_max(localMax.x, localMax.y)); @@ -927,16 +897,6 @@ quantize_with_block_size( } } } - // PDL completion is reported when every CTA has either exited or called - // this function at least once (per CUDA Programming Guide). Without a - // CTA-wide barrier, an early-finishing warp can trigger completion while - // other warps in the same CTA are still writing sf_out / out, allowing the - // downstream NVF4 GEMM consumer to read partial data once - // wait_on_dependent_grids returns. Each thread first makes its own stores - // device-visible; the barrier then guarantees every thread has done so - // before any thread can reach the trigger. - __threadfence(); - __syncthreads(); cudaTriggerProgrammaticLaunchCompletion(); #endif } diff --git a/cpp/tensorrt_llm/kernels/rmsNormFp4QuantKernels.cu b/cpp/tensorrt_llm/kernels/rmsNormFp4QuantKernels.cu deleted file mode 100644 index 45b1a38266b1..000000000000 --- a/cpp/tensorrt_llm/kernels/rmsNormFp4QuantKernels.cu +++ /dev/null @@ -1,443 +0,0 @@ -/* - * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include "rmsNormFp4QuantKernels.h" -#include "tensorrt_llm/common/cudaBf16Fallbacks.cuh" -#include "tensorrt_llm/common/cudaTypeUtils.cuh" -#include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/envUtils.h" -#include "tensorrt_llm/kernels/quantization.cuh" -#include <cstdint> - -TRTLLM_NAMESPACE_BEGIN - -namespace kernels -{ - -// Self-contained device helpers for this kernel. These mirror the small -// reduce_fusion utilities in customAllReduceKernels.cu but are copied here (in a -// private namespace) so this translation unit does not depend on the AllReduce -// files at all. Kept byte-identical to the originals. -namespace rms_norm_fp4_quant -{ - -static constexpr int kBytesPerAccess = 16; -static constexpr int kWarpSize = 32; -static constexpr int kMaxCtaSize = 1024; - -// Type converter that packs data format to 128-bit data type. -using PackedFloat = union -{ - int4 packed; - float unpacked[4]; -}; - -using PackedHalf = union -{ - int4 packed; - half2 unpacked[4]; -}; - -template <typename T> -struct PackedOn16Bytes -{ -}; - -template <> -struct PackedOn16Bytes<float> -{ - using Type = PackedFloat; -}; - -template <> -struct PackedOn16Bytes<half> -{ - using Type = PackedHalf; -}; - -#ifdef ENABLE_BF16 -using PackedBFloat16 = union -{ - int4 packed; - __nv_bfloat162 unpacked[4]; -}; - -template <> -struct PackedOn16Bytes<__nv_bfloat16> -{ - using Type = PackedBFloat16; -}; -#endif - -// add two 128b data -template <typename T> -inline __device__ int4 add128b(T& a, T& b) -{ - T c; - c.unpacked[0] = a.unpacked[0] + b.unpacked[0]; - c.unpacked[1] = a.unpacked[1] + b.unpacked[1]; - c.unpacked[2] = a.unpacked[2] + b.unpacked[2]; - c.unpacked[3] = a.unpacked[3] + b.unpacked[3]; - return c.packed; -} - -inline __device__ float warp_reduce_sum(float val) -{ - val += __shfl_xor_sync(~0, val, 16); - val += __shfl_xor_sync(~0, val, 8); - val += __shfl_xor_sync(~0, val, 4); - val += __shfl_xor_sync(~0, val, 2); - val += __shfl_xor_sync(~0, val, 1); - return val; -} - -inline __device__ float block_reduce_sum(float val) -{ - __shared__ float smem[kWarpSize]; - int lane_id = threadIdx.x % kWarpSize, warp_id = threadIdx.x / kWarpSize, warp_num = blockDim.x / kWarpSize; - val = warp_reduce_sum(val); - if (lane_id == 0) - { - smem[warp_id] = val; - } - __syncthreads(); - val = lane_id < warp_num ? smem[lane_id] : 0.f; - val = warp_reduce_sum(val); - return val; -} - -template <typename T, typename PackedStruct> -inline __device__ float accumulate(float acc, PackedStruct& vec) -{ - static constexpr int kLoopNum = sizeof(PackedStruct) / sizeof(T); -#pragma unroll - for (int i = 0; i < kLoopNum; ++i) - { - float v = static_cast<float>(reinterpret_cast<T*>(vec.unpacked)[i]); - acc += v * v; - } - return acc; -} - -template <typename T, bool Affine, typename PackedStruct> -inline __device__ int4 rms_norm(float denom, PackedStruct& vec, PackedStruct& weight) -{ - static constexpr int kLoopNum = sizeof(PackedStruct) / sizeof(T); - PackedStruct ret; -#pragma unroll - for (int i = 0; i < kLoopNum; ++i) - { - float v1 = static_cast<float>(reinterpret_cast<T*>(vec.unpacked)[i]); - if constexpr (Affine) - { - float v2 = static_cast<float>(reinterpret_cast<T*>(weight.unpacked)[i]); - reinterpret_cast<T*>(ret.unpacked)[i] = static_cast<T>(v1 * denom * v2); - } - else - { - reinterpret_cast<T*>(ret.unpacked)[i] = static_cast<T>(v1 * denom); - } - } - return ret.packed; -} - -// Fused (optional residual-add +) RMSNorm + NVFP4 input-quantize. Invoked by -// the standalone thop ops fused_add_rmsnorm_fp4_quantize (Residual=true) and -// fused_rmsnorm_fp4_quantize (Residual=false) on the attention-DP path. -// Performs the residual-add and reduction, then emits a per-block -// (SF_VEC_SIZE=16) NVFP4 representation to (quant_out, scale_out). When -// OutNorm=true, BF16 norm_out is also written (so a downstream consumer can read -// the un-quantized value). -// -// Layout assumptions (match cvt_warp_fp16_to_fp4): -// - Each thread accesses kPackedSize=8 BF16/half elements (one int4 = 16 B). -// - Two adjacent threads cover one SF_VEC_SIZE=16 block; SF is computed -// warp-cooperatively via __shfl_xor_sync inside cvt_warp_fp16_to_fp4. -// - Caller guarantees hidden_size % SF_VEC_SIZE == 0. -template <typename T, bool Bias = false, bool Residual = false, bool Affine = false, bool UseSmem = false, - bool OutNorm = false> -__global__ void rmsNormFp4QuantKernel(RmsNormFp4QuantParams params) -{ - static constexpr int kPackedSize = kBytesPerAccess / sizeof(T); - static constexpr int kSfVecSize = 16; - using PackedStruct = typename PackedOn16Bytes<T>::Type; - - extern __shared__ uint8_t smem_ptr[]; - T* smem = reinterpret_cast<T*>(smem_ptr); - - int const bid = blockIdx.x; - int const tid = threadIdx.x; - - T const* bias_buffer = reinterpret_cast<T const*>(params.bias_buffer); - T const* residual_buffer = reinterpret_cast<T const*>(params.residual_buffer); - T const* weight_buffer = reinterpret_cast<T const*>(params.weight_buffer); - T const* intermediate_buffer = reinterpret_cast<T const*>(params.intermediate_buffer); - T* residual_out_buffer = reinterpret_cast<T*>(params.residual_out_buffer); - T* norm_out = reinterpret_cast<T*>(params.norm_out); - - int const block_offset = bid * params.hidden_size; - // Input rows may be strided (e.g. a column slice of a wider projection, - // such as the leading q_lora_rank columns of kv_a_proj_with_mqa). When - // input_row_stride <= 0 it defaults to hidden_size (packed rows), making - // this byte-identical to all existing callers. Only the INPUT read offset - // uses the stride; every output (residual/norm/quant/scale) stays packed. - int const input_block_offset = bid * (params.input_row_stride > 0 ? params.input_row_stride : params.hidden_size); - int const thread_offset = tid * kPackedSize; - - if constexpr (Residual) - { - residual_buffer += block_offset; - // residual_out is packed [m, hidden_size], so it uses the dense block - // offset (not the possibly-strided input offset). - residual_out_buffer += block_offset; - } - intermediate_buffer += input_block_offset; - if constexpr (OutNorm) - { - norm_out += block_offset; - } - -#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) && (__CUDA_ARCH__ < 1200)) - cudaGridDependencySynchronize(); -#endif - - PackedStruct inter_vec, weight_vec; - float acc = 0.f; - for (int offset = thread_offset; offset < params.hidden_size; offset += blockDim.x * kPackedSize) - { - inter_vec.packed = *reinterpret_cast<int4 const*>(intermediate_buffer + offset); - if constexpr (Bias) - { - PackedStruct bias_vec; - bias_vec.packed = *reinterpret_cast<int4 const*>(bias_buffer + offset); - inter_vec.packed = add128b(inter_vec, bias_vec); - } - if constexpr (Residual) - { - PackedStruct residual_vec; - residual_vec.packed = *reinterpret_cast<int4 const*>(residual_buffer + offset); - inter_vec.packed = add128b(inter_vec, residual_vec); - // Write the residual sum to a distinct output buffer (packed offset), - // leaving the input intermediate_buffer untouched. - *reinterpret_cast<int4*>(residual_out_buffer + offset) = inter_vec.packed; - } - acc = accumulate<T>(acc, inter_vec); - if constexpr (UseSmem) - { - *reinterpret_cast<int4*>(&smem[offset]) = inter_vec.packed; - } - } - acc = block_reduce_sum(acc); - float const denom = rsqrtf(acc / params.hidden_size + params.eps); - - float const sf_scale = params.scale_factor_ptr ? *params.scale_factor_ptr : 1.f; - int const hidden_dim_packed = params.hidden_size / kSfVecSize; - - // cvt_warp_fp16_to_fp4 performs full-mask __shfl_xor_sync exchanges, which - // require every lane of the warp to execute the call. When hidden_size / - // kPackedSize is not a multiple of kWarpSize (e.g. hidden_size = 32, 128, - // 8208), the tail warp would otherwise be only partially active in this - // loop -- undefined behavior. Iterate with a warp-uniform bound (the warp's - // lane-0 offset) so all 32 lanes stay converged through the shuffle, and - // mask the per-lane loads/stores instead. Out-of-range lanes feed zeros to - // the shuffle; this is safe because hidden_size % kSfVecSize == 0 makes the - // active region end on an SF-pair boundary, so a padding lane's xor-1 - // partner is always another padding lane. - int const lane_id = tid % kWarpSize; - for (int offset = thread_offset; offset - lane_id * kPackedSize < params.hidden_size; - offset += blockDim.x * kPackedSize) - { - bool const valid = offset < params.hidden_size; - if (valid) - { - if constexpr (UseSmem) - { - inter_vec.packed = *reinterpret_cast<int4 const*>(&smem[offset]); - } - if constexpr (Affine) - { - weight_vec.packed = *reinterpret_cast<int4 const*>(weight_buffer + offset); - } - inter_vec.packed = rms_norm<T, Affine>(denom, inter_vec, weight_vec); - if constexpr (OutNorm) - { - *reinterpret_cast<int4*>(norm_out + offset) = inter_vec.packed; - } - } - else - { - // Benign values for the warp-cooperative SF exchange below (a - // padding lane's first-loop inter_vec may be uninitialized). - inter_vec.packed = make_int4(0, 0, 0, 0); - } - - // FP4 quantize this 8-element packed vec; warp-cooperate with the - // neighbour thread (offset ^ kPackedSize) to compute a single SF for - // their joint 16-element block. All lanes (incl. padding) must execute - // this call; padding lanes pass a null SF pointer and drop the result. - ::tensorrt_llm::kernels::PackedVec<T> pv - = *reinterpret_cast<::tensorrt_llm::kernels::PackedVec<T>*>(&inter_vec); - uint8_t* sf_out_ptr = nullptr; - if (valid) - { - int const access_id_in_token = offset / kPackedSize; - sf_out_ptr = ::tensorrt_llm::kernels::cvt_quant_get_sf_out_offset<uint32_t, 2>(std::nullopt, bid, - access_id_in_token, std::nullopt, hidden_dim_packed, reinterpret_cast<uint32_t*>(params.scale_out), - params.sf_layout); - } - uint32_t const quant_val = ::tensorrt_llm::kernels::cvt_warp_fp16_to_fp4<T, kSfVecSize, /*UE8M0_SF=*/false>( - pv, sf_scale, sf_out_ptr); - if (valid) - { - int const access_id = bid * (params.hidden_size / kPackedSize) + (offset / kPackedSize); - reinterpret_cast<uint32_t*>(params.quant_out)[access_id] = quant_val; - } - } -#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900) && (__CUDA_ARCH__ < 1200)) - cudaTriggerProgrammaticLaunchCompletion(); -#endif -} - -template <typename T, bool OutNorm = false> -void launchRmsNormFp4QuantKernel(RmsNormFp4QuantParams const& params, cudaStream_t stream) -{ - static constexpr int kPackedSize = kBytesPerAccess / sizeof(T); - TLLM_CHECK(params.hidden_size % kPackedSize == 0); - TLLM_CHECK(params.hidden_size % 16 == 0); // SF_VEC_SIZE - int need_threads = params.hidden_size / kPackedSize; - int cta_size = need_threads <= kMaxCtaSize ? (need_threads + kWarpSize - 1) / kWarpSize * kWarpSize : kMaxCtaSize; - int cta_num = params.elts_total / params.hidden_size; - bool const need_smem = (cta_size * kBytesPerAccess / sizeof(T) < params.hidden_size); - int smem_size = need_smem ? params.hidden_size * sizeof(T) : 0; - bool const use_smem = need_smem; - - bool const has_bias = params.bias_buffer != nullptr; - bool const has_residual = params.residual_buffer != nullptr; - bool const has_weight = params.weight_buffer != nullptr; - - // Macro-dispatch over Bias/Residual/Affine/UseSmem and the OutNorm template arg. - // Launch through launchWithPdlWhenEnabled so the kernel's PDL primitives - // (cudaGridDependencySynchronize / cudaTriggerProgrammaticLaunchCompletion) - // are actually enabled when TLLM_ENABLE_PDL is set. -#define DISPATCH_FP4_QUANT(BIAS, RESIDUAL, AFFINE, SMEM) \ - if (use_smem == SMEM) \ - { \ - tensorrt_llm::common::launchWithPdlWhenEnabled("rmsNormFp4Quant", \ - rmsNormFp4QuantKernel<T, BIAS, RESIDUAL, AFFINE, SMEM, OutNorm>, dim3(cta_num), dim3(cta_size), \ - static_cast<size_t>(smem_size), stream, params); \ - } - - auto launch = [&]() - { - if (has_bias && has_residual && has_weight) - { - DISPATCH_FP4_QUANT(true, true, true, true) - else DISPATCH_FP4_QUANT(true, true, true, false) - } - else if (!has_bias && has_residual && has_weight) - { - DISPATCH_FP4_QUANT(false, true, true, true) - else DISPATCH_FP4_QUANT(false, true, true, false) - } - else if (has_bias && !has_residual && has_weight) - { - DISPATCH_FP4_QUANT(true, false, true, true) - else DISPATCH_FP4_QUANT(true, false, true, false) - } - else if (!has_bias && !has_residual && has_weight) - { - DISPATCH_FP4_QUANT(false, false, true, true) - else DISPATCH_FP4_QUANT(false, false, true, false) - } - else if (has_bias && has_residual && !has_weight) - { - DISPATCH_FP4_QUANT(true, true, false, true) - else DISPATCH_FP4_QUANT(true, true, false, false) - } - else if (!has_bias && has_residual && !has_weight) - { - DISPATCH_FP4_QUANT(false, true, false, true) - else DISPATCH_FP4_QUANT(false, true, false, false) - } - else if (has_bias && !has_residual && !has_weight) - { - DISPATCH_FP4_QUANT(true, false, false, true) - else DISPATCH_FP4_QUANT(true, false, false, false) - } - else - { - DISPATCH_FP4_QUANT(false, false, false, true) - else DISPATCH_FP4_QUANT(false, false, false, false) - } - }; - launch(); -#undef DISPATCH_FP4_QUANT -} - -} // namespace rms_norm_fp4_quant - -void residualRmsNormFp4Quant(RmsNormFp4QuantParams const& params, tensorrt_llm::DataType dataType, cudaStream_t stream) -{ - // The NVFP4 epilogue (cvt_warp_fp16_to_fp4) is compiled only for - // __CUDA_ARCH__ >= 1000 and emits zeros otherwise, so this kernel is correct - // only on SM 10.x (Blackwell). Fail fast on unsupported archs rather than - // silently producing wrong FP4 (the Python dispatch in rms_norm.py already - // routes those to the unfused path). - int const sm = tensorrt_llm::common::getSMVersion(); - TLLM_CHECK_WITH_INFO(sm >= 100 && sm < 120, - "residualRmsNormFp4Quant requires SM 10.x (Blackwell); got SM %d. The fused NVFP4 epilogue is unsupported on " - "this arch.", - sm); - TLLM_CHECK_WITH_INFO(params.quant_out != nullptr && params.scale_out != nullptr, - "residualRmsNormFp4Quant requires quant_out and scale_out output buffers."); - sync_check_cuda_error(stream); - bool const out_norm = (params.norm_out != nullptr); - if (out_norm) - { - switch (dataType) - { -#ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: - rms_norm_fp4_quant::launchRmsNormFp4QuantKernel<__nv_bfloat16, /*OutNorm=*/true>(params, stream); - break; -#endif - case tensorrt_llm::DataType::kHALF: - rms_norm_fp4_quant::launchRmsNormFp4QuantKernel<half, /*OutNorm=*/true>(params, stream); - break; - default: TLLM_THROW("Unsupported dataType for residualRmsNormFp4Quant"); - } - } - else - { - switch (dataType) - { -#ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: - rms_norm_fp4_quant::launchRmsNormFp4QuantKernel<__nv_bfloat16, /*OutNorm=*/false>(params, stream); - break; -#endif - case tensorrt_llm::DataType::kHALF: - rms_norm_fp4_quant::launchRmsNormFp4QuantKernel<half, /*OutNorm=*/false>(params, stream); - break; - default: TLLM_THROW("Unsupported dataType for residualRmsNormFp4Quant"); - } - } - sync_check_cuda_error(stream); -} - -} // namespace kernels - -TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/rmsNormFp4QuantKernels.h b/cpp/tensorrt_llm/kernels/rmsNormFp4QuantKernels.h deleted file mode 100644 index 0ecba538d567..000000000000 --- a/cpp/tensorrt_llm/kernels/rmsNormFp4QuantKernels.h +++ /dev/null @@ -1,87 +0,0 @@ -/* - * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#pragma once - -#include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" -#include "tensorrt_llm/kernels/quantization.h" -#include <cstdint> - -TRTLLM_NAMESPACE_BEGIN - -namespace kernels -{ - -// All parameters for the fused (optional residual-add +) RMSNorm + NVFP4 -// input-quantize kernel: inputs, outputs, and layout config alike, so the -// launcher takes a single struct. Self-contained (does not borrow -// AllReduceParams) since this kernel performs no allreduce — it backs the -// standalone thop ops fused_add_rmsnorm_fp4_quantize / -// fused_rmsnorm_fp4_quantize on the attention-DP path. -struct RmsNormFp4QuantParams -{ - // --- inputs --- - // Input values [m, hidden_size] (read with input_row_stride). Read-only: the - // residual sum is written to residual_out_buffer, never back into this. - void const* intermediate_buffer{nullptr}; - // residual_in [m, hidden_size]; nullptr disables the residual add. - void const* residual_buffer{nullptr}; - // optional bias add [hidden_size]; nullptr disables it. - void const* bias_buffer{nullptr}; - // RMSNorm gamma [hidden_size]; nullptr selects the non-affine path. - void const* weight_buffer{nullptr}; - // Device pointer to the global per-tensor scale (= 448*6 / amax for - // static-quant Linear); nullptr means 1.0. - float const* scale_factor_ptr{nullptr}; - - // --- outputs --- - // Packed FP4 (E2M1) values, 2 per byte. - void* quant_out{nullptr}; - // E4M3 scaling factors (one per SF_VEC_SIZE=16 block), laid out per sf_layout. - void* scale_out{nullptr}; - // Optional BF16/FP16 post-RMSNorm value (packed rows); nullptr to skip. - void* norm_out{nullptr}; - // residual_out [m, hidden_size] (packed): receives input + residual when - // residual_buffer != nullptr. A distinct buffer from intermediate_buffer so - // the input is never mutated (keeps the thop op functionalizable under - // torch.compile) and no pre-kernel copy is needed. nullptr when no residual. - void* residual_out_buffer{nullptr}; - - // --- config --- - int hidden_size{0}; - float eps{0.f}; - // Total element count (= m * hidden_size); used to derive the row count. - int64_t elts_total{0}; - // Scaling-factor layout (typically SWIZZLED). - ::tensorrt_llm::QuantizationSFLayout sf_layout{::tensorrt_llm::QuantizationSFLayout::SWIZZLED}; - // Element stride between input rows in intermediate_buffer. 0 means - // "== hidden_size" (packed rows). Set >0 to read a strided slice (e.g. a - // column-slice of a wider projection) without a preceding contiguous copy. - // Outputs are always written packed. - int input_row_stride{0}; -}; - -// Fused (optional residual-add +) RMSNorm + NVFP4 input-quantize. Folds RMSNorm -// and the next op's NVFP4 input-quant so the (flashinfer RMSNorm + standalone -// fp4_quantize) pair becomes one launch on the attention-DP path. All inputs, -// outputs, and layout configuration are carried in params (see the struct -// field docs above); dataType selects the fp16/bf16 instantiation. -void residualRmsNormFp4Quant(RmsNormFp4QuantParams const& params, tensorrt_llm::DataType dataType, cudaStream_t stream); - -} // namespace kernels - -TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/speculativeDecoding/dynamicTreeKernels.cu b/cpp/tensorrt_llm/kernels/speculativeDecoding/dynamicTreeKernels.cu index caa090f0e25c..3d167660e521 100644 --- a/cpp/tensorrt_llm/kernels/speculativeDecoding/dynamicTreeKernels.cu +++ b/cpp/tensorrt_llm/kernels/speculativeDecoding/dynamicTreeKernels.cu @@ -29,10 +29,12 @@ #include "tensorrt_llm/common/reduceKernelUtils.cuh" #include "tensorrt_llm/common/vec_dtypes.cuh" #include "tensorrt_llm/kernels/decodingCommon.h" +#include <ATen/cuda/CUDAContext.h> #include <algorithm> #include <cfloat> #include <cstdint> #include <limits> +#include <torch/extension.h> TRTLLM_NAMESPACE_BEGIN using namespace tensorrt_llm::common; @@ -41,6 +43,480 @@ using namespace tensorrt_llm::runtime; namespace kernels::speculative_decoding { +// --------------------------------------------------------------------------- +// Two-stage top-k / top-p masking kernels +// Mirrors the approach in invokeBatchTopKSampling (samplingTopKKernels.cu), +// but outputs a masked logits tensor instead of sampling a token. +// --------------------------------------------------------------------------- + +// Stage 1: Parallel top-k reduction across BLOCKS_PER_BEAM_ blocks per row. +// Each block handles (vocabSize / BLOCKS_PER_BEAM_) elements and finds its +// local top-k, writing (global_index, logit_value) pairs into the tmp buffers. +template <typename T, int32_t BLOCK_SIZE_, int32_t BLOCKS_PER_BEAM_> +__global__ void topKProbStage1(T const* __restrict__ logits, T* tmpLogProbs, int32_t* topKTmpIdBuf, T* topKTmpValBuf, + int32_t maxTopK, int32_t const* topKs, int32_t vocabSize) +{ + typedef cub::BlockReduce<TopK_2<T>, BLOCK_SIZE_> BlockReduce; + __shared__ typename BlockReduce::TempStorage tempStorage; + + auto const tid = static_cast<int32_t>(threadIdx.x); + auto const bid = static_cast<int32_t>(blockIdx.x); + auto const rowId = bid / BLOCKS_PER_BEAM_; + auto const blockLane = bid % BLOCKS_PER_BEAM_; // chunk index within the row + + auto const k = (topKs != nullptr) ? topKs[rowId] : maxTopK; + + bool const IS_FP16 = std::is_same<T, half>::value; + T const MAX_T_VAL = IS_FP16 ? HALF_FLT_MAX : FLT_MAX; + + // Base offset into the flat (nRows * vocabSize) logits array for this row. + auto const rowOffset = rowId * vocabSize; + // Base offset into the tmp buffers for this (row, blockLane). + auto const tmpIdxBase = rowId * BLOCKS_PER_BEAM_ * maxTopK + blockLane * k; + + // Copy this block's chunk of logits into tmpLogProbs scratch space. + for (auto elemId = tid + blockLane * BLOCK_SIZE_; elemId < vocabSize; elemId += BLOCK_SIZE_ * BLOCKS_PER_BEAM_) + { + tmpLogProbs[rowOffset + elemId] = logits[rowOffset + elemId]; + } + __syncthreads(); + + // Iteratively find the top-k values via max-reduction, zeroing each found max. + TopK_2<T> partial; + for (int32_t ite = 0; ite < k; ite++) + { + partial.init(); + for (auto elemId = tid + blockLane * BLOCK_SIZE_; elemId < vocabSize; elemId += BLOCK_SIZE_ * BLOCKS_PER_BEAM_) + { + partial.insert(tmpLogProbs[rowOffset + elemId], rowOffset + elemId); + } + + TopK_2<T> total = BlockReduce(tempStorage).Reduce(partial, reduce_topk_op_2<T>); + + if (tid == 0) + { + topKTmpIdBuf[tmpIdxBase + ite] = total.p; // global index (rowOffset + vocabIdx) + topKTmpValBuf[tmpIdxBase + ite] = total.u; // logit value + if (total.p >= 0) + { + tmpLogProbs[total.p] = -MAX_T_VAL; // zero out so next iteration finds next-best + } + } + __syncthreads(); + } +} + +// Stage 2: Merge BLOCKS_PER_BEAM_ * k candidates per row, apply optional top-p, +// then scatter selected logit values back to an output logits tensor (all other +// positions are set to -inf so that a subsequent softmax produces 0 probability). +template <typename T, int32_t BLOCK_SIZE_, int32_t BLOCKS_PER_BEAM_> +__global__ void topKProbStage2ForLogits(int32_t const* __restrict__ topKTmpIdBuf, T* topKTmpValBuf, float* outputLogits, + int32_t maxTopK, int32_t const* topKs, float const* topPs, int32_t vocabSize) +{ + bool const IS_FP16 = std::is_same<T, half>::value; + T const MAX_T_VAL = IS_FP16 ? HALF_FLT_MAX : FLT_MAX; + + auto const tid = static_cast<int32_t>(threadIdx.x); + auto const rowId = static_cast<int32_t>(blockIdx.x); + + auto const k = (topKs != nullptr) ? topKs[rowId] : maxTopK; + // size: number of valid candidates written by Stage 1 for this row. + // stride: row pitch in the tmp buffers (same as in invokeBatchTopKSampling). + auto const size = k * BLOCKS_PER_BEAM_; + auto const stride = maxTopK * BLOCKS_PER_BEAM_; + + typedef cub::BlockReduce<TopK_2<float>, BLOCK_SIZE_> BlockReduce; + __shared__ typename BlockReduce::TempStorage tempStorage; + extern __shared__ char sharedArray[]; + // Shared layout: sId[maxTopK] | sVal2[maxTopK] + auto* sId = reinterpret_cast<int32_t*>(sharedArray); + auto* sVal2 = reinterpret_cast<float*>(sId + maxTopK); + + // Pointer to this row's candidates in the tmp value buffer (modified in-place during reduction). + T* sVal = topKTmpValBuf + rowId * stride; + + // Step 1: Initialize output row to -inf (all threads cooperate for bandwidth). + float* outRow = outputLogits + rowId * vocabSize; + float const negInf = -std::numeric_limits<float>::infinity(); + for (int32_t i = tid; i < vocabSize; i += BLOCK_SIZE_) + { + outRow[i] = negInf; + } + __syncthreads(); + + // Step 2: k-round block-reduction over the k * BLOCKS_PER_BEAM_ valid candidates. + // (Only the first 'size' entries of the row's tmp buffer were written by Stage 1.) + TopK_2<float> partial; + __shared__ float sMaxLogit; + for (int32_t ite = 0; ite < k; ite++) + { + partial.init(); + for (int32_t i = tid; i < size; i += BLOCK_SIZE_) + { + partial.insert(static_cast<float>(sVal[i]), i); + } + + TopK_2<float> total = BlockReduce(tempStorage).Reduce(partial, reduce_topk_op_2<float>); + + if (tid == 0) + { + if (ite == 0) + { + sMaxLogit = total.u; + } + sId[ite] = total.p; + sVal[total.p] = -MAX_T_VAL; // zero out so next iteration finds next-best + sVal2[ite] = total.u; // store raw logit value (not exponentiated) + } + __syncthreads(); + } + + // Step 3: Determine top-p cutoff (tid=0 only). + // sVal2 contains logit values in descending order; we exponentiate to get unnormalized probs. + if (tid == 0) + { + int32_t cutoff = k; + if (topPs != nullptr) + { + float const topP = topPs[rowId]; + if (topP < 1.0f) + { + // Compute unnormalized probabilities and their sum. + float sSum = 0.0f; + for (int32_t ki = 0; ki < k; ki++) + { + sVal2[ki] = __expf(sVal2[ki] - sMaxLogit); // reuse sVal2 to hold exp probs + sSum += sVal2[ki]; + } + // Walk in descending-probability order; stop as soon as cumulative prob >= topP. + float cumProb = 0.0f; + for (int32_t ki = 0; ki < k; ki++) + { + cumProb += sVal2[ki] / sSum; + if (cumProb >= topP) + { + cutoff = ki + 1; // always keep at least this token + break; + } + } + } + } + + // Step 4: Scatter selected logit values back to output. + // topKTmpIdBuf stores (rowOffset + vocabIdx); recover vocabIdx with % vocabSize. + auto const rowStride = rowId * stride; + for (int32_t ki = 0; ki < cutoff; ki++) + { + auto const candidateIdx = sId[ki]; + auto const globalIdx = topKTmpIdBuf[rowStride + candidateIdx]; + if (globalIdx >= 0) + { + auto const vocabIdx = globalIdx % vocabSize; + // sVal2 was overwritten with exp probs when topP < 1; we need the original logit. + // Re-read from the original tmp buffer — the stored value IS the logit (set in Stage 1). + // However sVal[candidateIdx] was zeroed during Stage 2 reduction; but + // topKTmpValBuf still holds the original value at that index (sVal points there). + // We stored the logit as sVal2[ite] = total.u BEFORE any exp, so if topP was not + // applied we can use sVal2[ki] directly. If topP was applied, sVal2[ki] now holds + // the exp prob — we cannot recover the logit. To handle both cases cleanly we use + // log(sVal2[ki]) + sMaxLogit when topPs was applied, otherwise sVal2[ki] directly. + float logitVal; + if (topPs != nullptr && topPs[rowId] < 1.0f) + { + // sVal2[ki] = exp(logit - sMaxLogit), so logit = log(sVal2[ki]) + sMaxLogit + logitVal = __logf(sVal2[ki]) + sMaxLogit; + } + else + { + logitVal = sVal2[ki]; // still the raw logit + } + outRow[vocabIdx] = logitVal; + } + } + } +} + +#define CASE_K_PROB(K_MAX, BLOCK_SIZE_1_, BLOCK_SIZE_2_, BLOCKS_PER_BEAM_) \ + do \ + { \ + topKProbStage1<T, BLOCK_SIZE_1_, BLOCKS_PER_BEAM_> \ + <<<dim3(nRows* BLOCKS_PER_BEAM_, 1), BLOCK_SIZE_1_, 0, stream>>>( \ + logits, tmpLogProbs, topKTmpIdBuf, topKTmpValBuf, maxTopK, topKs, vocabSize); \ + topKProbStage2ForLogits<T, BLOCK_SIZE_2_, BLOCKS_PER_BEAM_> \ + <<<dim3(nRows, 1), BLOCK_SIZE_2_, K_MAX * (sizeof(int32_t) + sizeof(float)), stream>>>( \ + topKTmpIdBuf, topKTmpValBuf, outputLogits, maxTopK, topKs, topPs, vocabSize); \ + } while (0) + +// Host launcher: allocates workspace tensors internally and dispatches the two-stage kernels. +// logits [nRows, vocabSize] – temperature-scaled input (float or half) +// outputLogits [nRows, vocabSize] – output: -inf everywhere except selected top-k-p positions +// topKs [nRows] – per-row k values (int32, on device) +// topPs [nRows] or nullptr – per-row p values (float, on device) +// maxTopK – maximum k across all rows (CPU scalar, 1–1024) +template <typename T> +void invokeTopKTopPMaskingForProbs(T const* logits, float* outputLogits, int32_t const* topKs, float const* topPs, + int32_t maxTopK, int32_t nRows, int32_t vocabSize, cudaStream_t stream) +{ + constexpr int32_t BLOCKS_PER_BEAM = 8; + + // Workspace buffers (allocated as CUDA device tensors via ATen). + auto opts = at::TensorOptions().dtype(torch::kFloat32).device(at::kCUDA); + auto tmpLogProbsTensor = torch::empty({nRows * vocabSize}, opts); + auto topKTmpIdBufTensor + = torch::empty({nRows * BLOCKS_PER_BEAM * maxTopK}, at::TensorOptions().dtype(torch::kInt32).device(at::kCUDA)); + // topKTmpValBuf uses the same dtype as T; we allocate as float and reinterpret for half if needed. + auto topKTmpValBufTensor = torch::empty({nRows * BLOCKS_PER_BEAM * maxTopK}, opts); + + T* tmpLogProbs = reinterpret_cast<T*>(tmpLogProbsTensor.data_ptr<float>()); + int32_t* topKTmpIdBuf = topKTmpIdBufTensor.data_ptr<int32_t>(); + T* topKTmpValBuf = reinterpret_cast<T*>(topKTmpValBufTensor.data_ptr<float>()); + + int32_t logMaxTopK = 0; + int32_t recursor = maxTopK - 1; + while (recursor >>= 1) + { + ++logMaxTopK; + } + + switch (logMaxTopK) + { + case 0: + case 1: + case 2: + case 3: // 0 < maxTopK <= 16 + CASE_K_PROB(16, 128, 128, 8); + break; + case 4: // 16 < maxTopK <= 32 + CASE_K_PROB(32, 256, 128, 8); + break; + case 5: // 32 < maxTopK <= 64 + CASE_K_PROB(64, 256, 256, 8); + break; + case 6: + case 7: + case 8: + case 9: // 64 < maxTopK <= 1024 + CASE_K_PROB(1024, 256, 256, 8); + break; + default: TLLM_CHECK_WITH_INFO(false, "topKProbMasking supports 1 <= k <= 1024 but got k=%d", maxTopK); + } +} + +#undef CASE_K_PROB + +namespace +{ +constexpr double kGreedyTempThreshold = 1e-4; + +torch::Tensor computeSoftmaxForProbOp(torch::Tensor logits) +{ + TORCH_CHECK(logits.is_cuda(), "logits must be a CUDA tensor"); + TORCH_CHECK(logits.dim() == 2, "logits must be a 2D tensor"); + + auto probs = logits.contiguous().to(torch::kFloat32); + auto stream = at::cuda::getCurrentCUDAStream(probs.device().index()); + + BiasSoftmaxParams<float> biasSoftmaxParams; + biasSoftmaxParams.logits = probs.data_ptr<float>(); + biasSoftmaxParams.probs = probs.data_ptr<float>(); + biasSoftmaxParams.batchSize = static_cast<SizeType32>(probs.size(0)); + biasSoftmaxParams.maxBatchSize = static_cast<SizeType32>(probs.size(0)); + biasSoftmaxParams.maxBeamWidth = 1; + biasSoftmaxParams.vocabSize = static_cast<SizeType32>(probs.size(1)); + biasSoftmaxParams.vocabSizePadded = static_cast<SizeType32>(probs.size(1)); + biasSoftmaxParams.skipSoftMax = false; + biasSoftmaxParams.batchSlotsLogits = false; + biasSoftmaxParams.checkParams(); + + invokeAddBiasSoftMax(biasSoftmaxParams, stream); + return probs; +} + +// Fast path for top-K (and optional top-P) filtering using torch::topk instead of a +// full vocab-size sort. kMax must be provided as a CPU integer (the caller computes it +// via topK.max().item() on the Python side). When kMax == 0 or kMax >= vocabSize the +// function falls back to the original sort-based path. +// +// Key advantages over the full-sort path: +// 1. torch::topk with small kMax is O(V * log kMax) vs O(V * log V) for full sort. +// 2. The topk index tensor is [nRows, kMax] instead of [nRows, V] — much smaller. +// 3. No scatter-back of sorted indices needed; masking is done directly on logits. +// 4. For combined top-K + top-P, softmax/cumsum are computed on kMax values (not V). +torch::Tensor applyTopKTopPForProbOp(torch::Tensor logits, torch::optional<torch::Tensor> const& topK, + torch::optional<torch::Tensor> const& topP, int32_t kMax) +{ + int64_t const vocabSize = logits.size(1); + // Host-only checks: the caller is expected to pass nullopt when filtering is fully + // disabled (see SpecMetadata.skip_top_k / skip_top_p). Probing the tensor contents + // via `.item<bool>()` here would force a host-device sync and break CUDA graph + // capture; the per-row `effectiveTopK` formula below already handles disabled rows. + bool const hasTopK = topK.has_value() && topK->defined(); + bool const hasTopP = topP.has_value() && topP->defined(); + + if (!hasTopK && !hasTopP) + { + return logits; + } + + torch::Tensor effectiveTopK; + if (hasTopK) + { + auto topKLong = topK->to(torch::kLong); + effectiveTopK + = torch::where(topKLong > 0, topKLong, torch::full_like(topKLong, vocabSize)).clamp_max(vocabSize); + } + + // Fast path uses `topk(kMax)` which is unsafe when any row has effective top-k > kMax + // (i.e. disabled rows expand to the full vocab). Detecting this requires a tensor + // reduction + `.item<bool>()`, which is incompatible with CUDA graph capture. Only + // probe when the caller explicitly opted into the fast path via kMax > 0 (today only + // the dynamic-tree caller, which is not graph-captured). + bool hasDisabledTopKRows = false; + if (hasTopK && kMax > 0 && kMax < vocabSize) + { + auto topKLong = topK->to(torch::kLong); + hasDisabledTopKRows = topKLong.le(0).any().item<bool>(); + } + + if (hasTopK && !hasDisabledTopKRows && kMax > 0 && kMax < vocabSize) + { + // Fast topk path ───────────────────────────────────────────────────────────── + // topKValues/topKIdx: [nRows, kMax], values in descending order + auto [topKValues, topKIdx] = logits.topk(kMax, /*dim=*/-1, /*largest=*/true, /*sorted=*/true); + + // validTopK[i, j]: True when position j falls within top-K[i] for row i + auto kArange = torch::arange(kMax, torch::TensorOptions().dtype(torch::kInt64).device(logits.device())) + .unsqueeze(0); // [1, kMax] + auto kVals = effectiveTopK.to(torch::kInt64).unsqueeze(1); // [nRows, 1] + auto validTopK = kArange < kVals; // [nRows, kMax] + + // Start with everything masked; scatter will unmark the kept positions. + auto mask = torch::ones( + {logits.size(0), vocabSize}, torch::TensorOptions().dtype(torch::kBool).device(logits.device())); + + if (hasTopP) + { + // Compute top-P on the kMax descending-sorted values only (much cheaper). + // Positions beyond K[i] are treated as -inf so their probability ≈ 0. + auto validTopKValues = topKValues.masked_fill(~validTopK, -std::numeric_limits<float>::infinity()); + auto sortedProbs = validTopKValues.softmax(/*dim=*/-1); // [nRows, kMax] + auto cumsum = sortedProbs.cumsum(/*dim=*/-1); // [nRows, kMax] + // Mask positions where the cumulative probability *before* this token + // already reaches topP — i.e. we have enough probability mass already. + auto topPMask = (cumsum - sortedProbs) >= topP->unsqueeze(1); // [nRows, kMax] + topPMask.select(/*dim=*/1, /*index=*/0).fill_(false); // always keep the top-1 token + // combinedMask: True → mask this vocab position + // False → keep this vocab position + auto combinedMask = topPMask | (~validTopK); // [nRows, kMax] + mask.scatter_(/*dim=*/1, /*index=*/topKIdx, /*src=*/combinedMask); + } + else + { + // Top-K only: unmark the first K[i] positions (those within validTopK). + // ~validTopK is True for positions j >= K[i] → they should stay masked. + mask.scatter_(/*dim=*/1, /*index=*/topKIdx, /*src=*/(~validTopK)); + } + + return logits.masked_fill(mask, -std::numeric_limits<float>::infinity()); + } + + // Fallback: full-sort path (used for top-P only, or when kMax == 0) ──────────── + auto sortResult = logits.sort(/*dim=*/-1, /*descending=*/false); + auto logitsSort = std::get<0>(sortResult); + auto logitsIdx = std::get<1>(sortResult); + + if (hasTopK) + { + auto topKMask = logitsSort.size(1) - effectiveTopK; + topKMask = topKMask.clamp_min(0); + auto topKThreshold = logitsSort.gather(1, topKMask.unsqueeze(1)); + auto mask = logitsSort < topKThreshold; + logitsSort.masked_fill_(mask, -std::numeric_limits<float>::infinity()); + } + + if (hasTopP) + { + auto probsSort = logitsSort.softmax(/*dim=*/-1); + auto probsSum = probsSort.cumsum(/*dim=*/-1, /*dtype=*/probsSort.scalar_type()); + auto topPMask = probsSum <= (1.0 - topP->unsqueeze(1)); + topPMask.select(/*dim=*/1, /*index=*/logitsSort.size(1) - 1).fill_(false); + logitsSort.masked_fill_(topPMask, -std::numeric_limits<float>::infinity()); + } + + return logitsSort.scatter(/*dim=*/-1, /*index=*/logitsIdx, /*src=*/logitsSort); +} + +} // namespace + +torch::Tensor computeProbsFromLogits(torch::Tensor const& logits, torch::Tensor const& temperatures, + torch::optional<torch::Tensor> const& topK, torch::optional<torch::Tensor> const& topP, bool skipTemperature, + int32_t kMax) +{ + TORCH_CHECK(logits.is_cuda(), "logits must be a CUDA tensor"); + TORCH_CHECK(temperatures.is_cuda(), "temperatures must be a CUDA tensor"); + TORCH_CHECK(logits.dim() == 2, "logits must be a 2D tensor"); + TORCH_CHECK(temperatures.dim() == 1, "temperatures must be a 1D tensor"); + TORCH_CHECK(logits.size(0) == temperatures.size(0), "logits and temperatures size mismatch"); + if (topK.has_value() && topK->defined()) + { + TORCH_CHECK(topK->is_cuda(), "top_k must be a CUDA tensor"); + TORCH_CHECK(topK->dim() == 1, "top_k must be a 1D tensor"); + TORCH_CHECK(topK->size(0) == logits.size(0), "top_k and logits size mismatch"); + } + if (topP.has_value() && topP->defined()) + { + TORCH_CHECK(topP->is_cuda(), "top_p must be a CUDA tensor"); + TORCH_CHECK(topP->dim() == 1, "top_p must be a 1D tensor"); + TORCH_CHECK(topP->size(0) == logits.size(0), "top_p and logits size mismatch"); + } + + auto const isGreedy = temperatures <= kGreedyTempThreshold; + auto const safeTemperatures = torch::where(isGreedy, torch::ones_like(temperatures), temperatures); + auto scaledLogits + = (skipTemperature ? logits : logits.div(safeTemperatures.unsqueeze(1))).contiguous().to(torch::kFloat32); + + int64_t const vocabSize = scaledLogits.size(1); + int64_t const nRows = scaledLogits.size(0); + // Host-only presence checks; see comment in applyTopKTopPForProbOp() for why we + // avoid probing tensor contents (would sync and break CUDA graph capture). + bool const hasTopKPresence = topK.has_value() && topK->defined(); + bool const hasTopPPresence = topP.has_value() && topP->defined(); + + // The kernel path produces -inf for rows whose top_k value is 0, so it is only + // safe when every row has an active top_k filter. Determining that requires a + // host-device sync, so only probe when the caller has opted into the kernel + // path (kMax > 0). The kMax > 0 callers (dynamic-tree) are not graph-captured. + bool useKernelPath = false; + if (hasTopKPresence && kMax > 0 && kMax < vocabSize) + { + useKernelPath = torch::logical_and(topK->gt(0), topK->lt(vocabSize)).any().item<bool>(); + } + + torch::Tensor maskedLogits; + if (useKernelPath) + { + // Two-stage CUDA top-k/top-p masking (mirrors invokeBatchTopKSampling). + maskedLogits = torch::empty_like(scaledLogits); + auto topKForKernel = topK->to(torch::kInt32).contiguous(); + auto topPForKernel = hasTopPPresence ? topP->to(torch::kFloat32).contiguous() : torch::Tensor(); + auto stream = at::cuda::getCurrentCUDAStream(scaledLogits.device().index()); + invokeTopKTopPMaskingForProbs<float>(scaledLogits.data_ptr<float>(), maskedLogits.data_ptr<float>(), + topKForKernel.data_ptr<int32_t>(), hasTopPPresence ? topPForKernel.data_ptr<float>() : nullptr, kMax, + static_cast<int32_t>(nRows), static_cast<int32_t>(vocabSize), stream); + } + else + { + // Fallback: PyTorch-based sort path (top-P only or kMax == 0). + maskedLogits = applyTopKTopPForProbOp(scaledLogits, topK, topP, kMax); + } + + auto probs = computeSoftmaxForProbOp(maskedLogits); + + auto argmaxIds = maskedLogits.argmax(/*dim=*/-1, /*keepdim=*/true); + auto oneHot = torch::zeros_like(probs).scatter_(1, argmaxIds, 1.0); + return torch::where(isGreedy.unsqueeze(1), oneHot, probs); +} + //! \param parentList [in] layer-wise parent indices [bs, topK*(depth-1)+1] //! \param selectedIndex [in] resampled history buffer indices [bs, draftTokenNum-1] //! \param treeMask [out] attention mask (which nodes each node can see) diff --git a/cpp/tensorrt_llm/kernels/splitkGroupGemm.cu b/cpp/tensorrt_llm/kernels/splitkGroupGemm.cu index 1f63189ec657..6397396ea6f6 100644 --- a/cpp/tensorrt_llm/kernels/splitkGroupGemm.cu +++ b/cpp/tensorrt_llm/kernels/splitkGroupGemm.cu @@ -26,7 +26,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/device/splitk_gemm_grouped.h" #include "tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/kernel/default_splitk_gemm_grouped.h" #include "tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/kernel/splitk_gemm_grouped.h" @@ -204,20 +203,20 @@ template <int M1, int N1, int K1, int M2, int N2, int K2, int kAlignmentAB, int void splitkGroupedGemmType_(std::vector<cutlass::gemm::GemmCoord> const& problemSizes, std::vector<void*> const& ptrA, std::vector<void*> const& ptrB, std::vector<void*> const& ptrC, std::vector<void*> const& ptrD, void* gemmParamsWorkSpace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkSpaceSize, - tensorrt_llm::DataType dataType, int splitKSlices, cudaStream_t stream) + nvinfer1::DataType dataType, int splitKSlices, cudaStream_t stream) { - if (dataType == tensorrt_llm::DataType::kHALF) + if (dataType == nvinfer1::DataType::kHALF) { splitkGroupedGemm_<M1, N1, K1, M2, N2, K2, cutlass::half_t, kAlignmentAB, kAlignmentC, kStages>(problemSizes, ptrA, ptrB, ptrC, ptrD, gemmParamsWorkSpace, gemmParamsWorkSpaceSize, gemmWorkSpace, gemmWorkSpaceSize, splitKSlices, stream); } - else if (dataType == tensorrt_llm::DataType::kFLOAT) + else if (dataType == nvinfer1::DataType::kFLOAT) { TLLM_CHECK_WITH_INFO(false, "not support float input/output"); } #ifdef ENABLE_BF16 - else if (dataType == tensorrt_llm::DataType::kBF16) + else if (dataType == nvinfer1::DataType::kBF16) { splitkGroupedGemm_<M1, N1, K1, M2, N2, K2, cutlass::bfloat16_t, kAlignmentAB, kAlignmentC, kStages>( problemSizes, ptrA, ptrB, ptrC, ptrD, gemmParamsWorkSpace, gemmParamsWorkSpaceSize, gemmWorkSpace, @@ -229,7 +228,7 @@ void splitkGroupedGemmType_(std::vector<cutlass::gemm::GemmCoord> const& problem void splitkGroupedGemm(std::vector<cutlass::gemm::GemmCoord> const& problemSizes, std::vector<void*> const& ptrA, std::vector<void*> const& ptrB, std::vector<void*> const& ptrC, std::vector<void*> const& ptrD, void* gemmParamsWorkSpace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkSpaceSize, - bool isLoraIn, tensorrt_llm::DataType dataType, int splitKSlices, int minKN, cudaStream_t stream) + bool isLoraIn, nvinfer1::DataType dataType, int splitKSlices, int minKN, cudaStream_t stream) { TLLM_LOG_TRACE("%s start, isLoraIn: %d, minKN = %d", __PRETTY_FUNCTION__, static_cast<int>(isLoraIn), minKN); if (isLoraIn) diff --git a/cpp/tensorrt_llm/kernels/splitkGroupGemm.h b/cpp/tensorrt_llm/kernels/splitkGroupGemm.h index bcde457db7d2..6ada8255292e 100644 --- a/cpp/tensorrt_llm/kernels/splitkGroupGemm.h +++ b/cpp/tensorrt_llm/kernels/splitkGroupGemm.h @@ -17,7 +17,7 @@ #include "cutlass/gemm_coord.h" #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <vector> TRTLLM_NAMESPACE_BEGIN @@ -30,7 +30,7 @@ int64_t getSplitkGroupedGemmParamsWorkSpaceSize(int64_t problem_count); void splitkGroupedGemm(std::vector<cutlass::gemm::GemmCoord> const& problem_sizes, std::vector<void*> const& ptrA, std::vector<void*> const& ptrB, std::vector<void*> const& ptrC, std::vector<void*> const& ptrD, void* gemmParamsWorkspace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkspaceSize, - bool isLoraIn, tensorrt_llm::DataType dataType, int splitKSlices, int minKN, cudaStream_t stream); + bool isLoraIn, nvinfer1::DataType dataType, int splitKSlices, int minKN, cudaStream_t stream); } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 4a458ca2eb81..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a503a33663a82dac6b0a041ea80029ee856ea720c1669947a00a266442138803 -size 24785 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index b8c1a64db2dd..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9a031403496fe7ec90e01a1a737e0ee928b61b9e88f4658acc4c2059e4f8b49e -size 23016 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c5ade09c3d69..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:1e8b1dde4c2c55dfc9ba0dba716981f825414812ecc0c37b7fd13bff6bf9b839 -size 23952 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 97f6221f22d2..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:cb24c063c1657f9e90ab64bf8dc6f9df4d8078fc16b6c594ee0fbb0e2fcf842f -size 25899 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 9c70231c5abd..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b466b76aea35df0bc7b7da19176da249a305f9ef16891f4c359d6ef942f8ca0c -size 22051 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 805d03795c57..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8b7955054cc91001d7c9d16a3f1645b6319d8cd69c8264aff43e5bff8cf0bc34 -size 23388 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 40559b4c178c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:17924331bd2a809d439f3cd2a042ea173b552f149df0b4e6ef678af2c75501f2 -size 25446 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 41d691409ccc..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:030c8a638e48794c5d84bd1d1586efd2b0dcddaf49395b715bb0c1b457fdac9b -size 21948 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 7e069fbab936..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a008086a28baaaa4fb70895b4e266600c8516f85a00dd4727aba24d0386082b3 -size 21116 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 421a11d9e6a0..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:641b36b4de6672687e8aef56c298909794708898ae916e9fb4dc126e51cc4b87 -size 18529 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 4be2fab2b20c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:815e6770f402fd156696c2184ad369f2b110f56240a96aa4e343258f303bce55 -size 23115 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 5c34d46bc526..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9cc36929649f6fccebf889c7424d3173695ab46c0e2a8a94c4faa18142ee4147 -size 20713 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c77accfdc892..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:13dc2ad438de03b826552dc5c1e1d41e7c20e60bc70780f840b9872bbc985fb2 -size 19772 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index ed7aa37f2507..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:733632e5236b93e5b866645d653a5d6ada087547bb2d353383c379b07c8a6ab3 -size 17441 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 6db3f921271f..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:076e7f1d4008613bc754f9440753b4183796f0939e34c02b87820e05749b940a -size 23508 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 5922757c3f01..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ebf843eedb570cbc5bc0ab926a9089e2adedaf95d012c31911b5ff5c4c2b57ce -size 21680 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index d89b31f6bcdf..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d5548ad24a30b3438a8eba833777f5b6e0fcff11c102551eebc575a1836c577a -size 22962 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index dd99a47690fa..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:1f527fe03f1dee3dcb66fe948b58ebd2d25cf8cab222a43245b1f28a548dcb8b -size 21659 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 690414211bbf..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:952265e1c6ec8dd349d3bc7ab9881ebda45b6dd9bc8785511edfa6814040de04 -size 24714 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 535f56ba1b40..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ca36c48b48053af3f50777e064e0c838d2c2edb3bab7855d94220aae52e73193 -size 22865 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f531acda7375..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:efb5305aedde3fa90fd2b4fc56a658f9f36754acd9a8c22618a42e42ca747433 -size 20912 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index ea02a5e4595e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b3a3f4471767a84bf43743d10c98e4f87dc662e04b9e54cbea31d203360beab3 -size 18493 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index e0e98ebd2dfa..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b1106a11165971ff57c9b5f568b9bbd7164809356ea341178e003f9e7ef48598 -size 19716 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 32658b7cb6ad..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:427ac27fb198d3ac4861da35bd7b064cc9822706b68bccce9c97da6a3a583552 -size 17445 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index a7dba5ca5efe..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:93068ef747b6901a2cace751ce303d49f5d9cf06d34976ffc72221cf5077fea8 -size 24564 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index df4160685735..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:23bba9e1d1a7593fad468da4499372dcbfde5a24e56470124524608151ecb54d -size 22238 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c0e5110b8755..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:35a37c601c9482254172a0a19bccee938bc79643cfffb00a2be00c5c471b0c4c -size 24072 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c511431c5bb4..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:842e622d1577dcc4e648a783510d298f34fc18ca5c7e54d2cf4620c0b3a29e02 -size 22804 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 2360c942468d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3aec5938ca870813f6d4d8e231144f6774d41b128a1e084007da231faaa8c2ea -size 26558 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 9cb37b24e46d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:985c11dc2f10cfca3a712b928834059f2af0ae9a0c88c5e6e1c2d45c5858dc36 -size 24597 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 2fe1a252b15c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f865e67427e8366459f7b00fc99e3259333734e8edcc0c20bf425e96c4d670f1 -size 21910 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 973d591ad1f7..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3225136cffe79437b3542e480510599223ae84e3b3b9513aed04e47145f43994 -size 18986 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index e3669afc928c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:7347f2f7054d36f08be4dd50351466ccc9e2d8bc0c45ae4efa34323667ef0e86 -size 20655 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 41264ec94990..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6794b45c8109ebcc73509981494eab0de71fc0ad1af46f3f5bf2212f28ed91f0 -size 17837 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 4fd1c267232e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:805ef3981ecd54462815e132081a2d75e12ccd909d54f08ebfb6e3dc4f114322 -size 30195 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 456eea49949d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:dddd5e66e76ba82b23bdf01c8bbf5f1fe33551d9fb6ef97c3e02efb7d55c84b4 -size 24637 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 2f988b45e346..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:1fffe042bc4e3ff9d76d0c5900063a6895bb206985c6c59b3769b1f1c82b6e45 -size 26848 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index b6f9d0fc12f8..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:783d16a0cec0542a267e3ad23dcecf0184963dc37a71faa7f8556f015333f5bd -size 23116 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 32a86a57dd6f..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b94c9a85598325e9a292429c32fccf31d32540bb8b6930f2fb5ae706cc03b4b2 -size 25575 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 60e2fb254da7..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b5fc98c54085364c4a15593fa660f46e542d566e9c1dadc2591f914a559fc56f -size 22871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 5206c9bc72d6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:959652701a99ebd284eb6d5e67cef59d86e3462d1716e74458ed40da3c89c8d1 -size 28364 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 96b5b933a892..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:cca3c15d37260e1b2ec617894d400fc08352589556e52eab4a9532ec6b012182 -size 27002 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f6826798c4d1..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8c5c587f737d6724e6d8355d0392e25794f8bb752df3e935ac11fbbc9e4f3ecd -size 27614 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index cc98ea911379..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ceca895537b08367f0e1993d4f51b84085cb90a58fa284bb079b81d5d4396d0f -size 26795 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 1733bae7a643..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:04e35880630ea83f0a4c57208e75e5ac20d1ed721c092311102bd4997f0465dc -size 25853 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 31734d1f9fa3..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:587d387bd32e9bfc28929d5aa049222fd0732fbbd320deff039fb7cae2d248c1 -size 24828 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index b11e45a7a886..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c6422d0276876388c539981a8406dd7c02ff8d8efb9be006f3fd92deb9500538 -size 25825 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index d1d5844865da..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b6913501e7c354ef801e4ba60280f28cb08dd8c2f22f9aabda5de102cbe1decd -size 23035 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index e324db6514de..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ff5c6585dcea540686a9ec8b2979f77b526af910fa16d5f043d1e886c7e92baa -size 24549 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 3de4d5d971e4..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d44bb4edfffd0db2486095531682667574b11a9fba40915e58144334d58db34f -size 21823 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 4cdf2cddfa8a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:fdeae075c34815df64964fac63f9cbfefe4d479e0781f12f7f88f547805c36a7 -size 29576 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 105f427e04ac..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d77a4b5a50368e3093080ad28d38fc2ce4885f93c1bac7b7a1db1652c6400687 -size 28157 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c6eb33e15cfa..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b2b6c9b1c2ccec0bc8cf136c02519c192a0c2b89146f9024d16720162eb8dd9c -size 28810 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 27287a99d1e7..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:dcea9b01ef1c8002c93f377c49e00886c4a5273ff2af4907bd4837632248c59d -size 27664 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 6952cc08f94d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c4698cebb22a5240db39b9309cbe0e2de2cdf71311e53fbcd4857d82ee90bd46 -size 27353 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 125b36ec0df8..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:62a9d7f1099d19e5a0a83f61ca347f0f3cf3b104953a37dd9707fa47157b5bcd -size 24686 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index d908c5c02f64..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:390171d148fe71134299b477eabc03a9fec8905423561202b23c8d9a3a10e407 -size 26874 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index dba03948e37a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:19dd7968d1864a57df724c4850462a6d3874300abf53da40833d5bbdc2a83b2c -size 23552 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 3e90e14e7a7e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5bbf7c88b8739fec5f9056beba4f9ac17e3e95836ea8f09b1412d5e3ae7991d4 -size 25701 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index d33c891480d2..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:2857bccb64e642e087225e4c4bddd4bead2a3214d2254e3c959511355b4051dc -size 22265 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 4023c5bebb03..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:45ffdaaee88cd89fb1879b34f13ac020e908f79d5605799cda09c9f9fc6d25f8 -size 22558 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index d16e5441124d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ed35a333d99c28479257f0fd4ac588f064a180753363d2a489d265d8c1955f6e -size 20932 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 56ea96835a3b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e2b8ecfb8489ed668cd40d6603ae2d1b329b04ef8da6ab0588f4ddf0ac924081 -size 21149 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f883be7fd862..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:28ccb15269bfb8a68cec3c2d9f341733c4eff53ef18c1d72cbdb082378226476 -size 24261 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 2231ff3bfd52..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:aa3b1a7d84e5a9ee0fc9da1e74c07ccd97f6bdbd74408f1954372061bd273730 -size 20242 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index b2a2c41b1bd6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:35dfdcaf9a453cf9b1c4a9745c4d2740a79de1daaa60d344a36d09892b26a0b6 -size 21446 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index ec14a68d2915..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8dfc357fae018d344b47d8d5ea837d23260b396c88b8efbce0887709340dec75 -size 23685 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index b75d6bfd91cb..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:86f84075f70e995faf31cac64aa1a3d70bd49c879182c424397c0536a548d0fc -size 20666 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 124911f3d421..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:125d806f16928b8db551573ef5326d6766805c7543f2c7cb1438dda9e5fcb386 -size 19348 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index bcf1d0d41990..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:89adf69b2182b810290a8b48e894934231ce35fa742b3974bb385dd965cda0de -size 17184 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 72dee4f79913..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:99ceec5275aeab6b1dceef41f06785fc323fe89eda16196196f5e0db29492223 -size 21341 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index d84bea559820..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b0fe22202f2ce0923ae0f274bc5f7d00e34d3405b683b024039c97bf697e360a -size 19092 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index efd823138b08..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4473cb570d1f1077b57a5a5bb9d3a877eabc06344f064552f409c7e7c33f7888 -size 18619 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index babe12716035..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:2a4ad8a08d0ce440df105c175030f4b232b9487f62fd683c003cfa42310450ef -size 16319 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index b198ae1eb9bf..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c4f3905573c910adbc184fae87de257689969f739f4440ed8f33ec845f456ce7 -size 20648 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f52ac54c1ed0..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:99f03c7ab332abd5a89a992973c7c8b8a526079e4de6cc09833a6a0af9d8c696 -size 19796 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 75394bfec4ea..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:78e79741391f12d469c03e3b2cbd4dff2402d3a5a247e73fe1b495860d3bff6a -size 21001 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 2166e1dc6f68..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:af4cab7963ce853e98c4c81ed809ff0086c0cd8bdea93dd5fb2a0f6b76c8eb83 -size 20261 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 5663f4574b07..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:73ad850cf882202b2300e6bd33c65aae1c4c021de718e07eaa31ceec496268b8 -size 22794 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 2687ae2ecca7..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9d306b38c4670dc6fceccdaafce815dcd82ad0b853b098dc80f5b5517d34bc0a -size 20711 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index fae8bf94cf3f..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8bac49862ac5874d15a1804e22abc34fb3a8c53e252f20bebc8eb84b1f002adf -size 19294 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 34d67133833d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a90422473729f1ce11972981a3d98267bb926bc48b4c12d40560acc1e9171ea6 -size 17078 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 8b1cc3eee6ea..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:50e18196cea8b515eebedc65339cd60a493008f51eaf805bfae2ee590365f7c6 -size 18504 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 921a9bde5714..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:572a6896ba8bbb113b55333bb11d4fbc8e29efcaee5a054a05bd3ac33037f61e -size 16191 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 05d30dae8a04..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:abeded5ad3db67cf0cb1e085e5d8c5846d7e9ce676595f8288e7520af3398c3e -size 21798 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 5ecf05803741..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3f281dd64fda86ebc02c0aac7627000d4c8b73dee3b0fa34725ea13a37d62699 -size 20977 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 5a583b374c1b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e5f98709486a5be226ebfdbc535f8cee3088a671fc9656411222849037801860 -size 22189 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index bf432d6aba2a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:170f74a174b08a134b4efb93eaca40e30f85c5bd2c891dcd96b0aef69d78d78e -size 21459 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 331645552888..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:524bf2e4f090c2cfc16dfdee9abcd02cd67f0982dc80b0ec6a68369a41a373e6 -size 24624 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 1c4e2b976191..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8408e22b12b495114c1bacd25d40cf2fa1b916ad4c8d349478eff6f4448979c3 -size 22872 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 563a2327e06b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:00a1170831ca48b70a064d204348301e51aef171d42868f74deba9e9447459a0 -size 20151 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 4554bca03d79..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c4d9cb004618b37fbc5b279b9fb5b8fe3ae0837063245e144db527d3d990ae9d -size 17689 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 225783439945..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:295c29b49454cc8426a2fe1c1e339c7314c4b2f23b841987ba60adcbac5f185d -size 19058 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index d67b71f4d822..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:991efba2e28d45732b5027c6f9eb3547b1cd280679429a2e7a48407136c3d8b2 -size 16885 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 13b156904d26..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:950fd67f582e65887e62cfcf944e0f2adf41a6b5d1df5a40d22b713bc536e57a -size 28618 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 5c840180a9e1..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f0e798edaa3d90e761c2e53dc717224b30a994dbf0c7f9645a7a2272578ae848 -size 22560 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 11940086383a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:86353c18f3d16d61f397816fc7fb2170c561bbcd04809b234eed9fa3ff181adf -size 25035 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 15f5f974d495..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d3e75d148cc3917884ff09d5c9eaa8ce0ea477033241225f264627516066a29c -size 21735 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ae20c121f347..4f1bff53c47d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5eadcefda04c666b18c0ec0b65509588929fc3f4c0429904cb63ea1f105baedb -size 25558 +oid sha256:cdef265186f95dbcb569806d727cf8edcf383e760ae50798fdc02a5560fd22bf +size 25565 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6ce3cd43f2b1..1fcde91db9c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2fbf47299a128801a4119974bf2cc6250d1ed24ba119ce1c6556a46a3f573f06 +oid sha256:a645606058f3f2644e75dc520a12555e38ab603ab415fafa5cef644b7a3d4bb8 size 23424 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b79c20de498d..13c1061b92cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:09196accf629bfcfdcc1a37b0cac233baeb968292a966c8abbdef7a28906140d -size 24645 +oid sha256:d09f7f986c337467df56398621c2a65eb232ef0acff38b02fc7867c030cb4b63 +size 24863 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 80e023f3e305..dbe9d490a12e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:568db976482763a3eeeaa11466ec251ffc3e2be540bb35ce114b6ee2aeb4e5a2 -size 23815 +oid sha256:485a3378913014c07d61211e2707544eadbc088144f5ec8ac1935d8d1b892be8 +size 23665 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 30bc52f55cbc..a9f26035cd20 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8ba7a60f9128980f4f567f1fea542a2ecb84dc4b0678ee19df85ba2cfc08d7c -size 25911 +oid sha256:1731b78fe04d9f3ad13645e39583d770ae37f47ff1ecafe876207858414f58bf +size 25877 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 93ad5f0ffe9c..c232af7b6cf9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:07e55d0d27bb0361cfcb6c205de4a1a0597f359a11752f8981b47b56cd3f2401 -size 22065 +oid sha256:0ec9470055420cd53411f772cdcc096d2420e0557d5a8852820a997e51b00272 +size 22024 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 220695bc7385..afbae8241256 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1c75da03f7a205e99daabb0ed96bf61839163bd963f865ce0be8e8fb455c7c2 -size 25503 +oid sha256:52e39edc1d84c95f8ed52bfc224c1fe7ea53d10a7709e45787657fd29ac824e2 +size 25679 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f5fb95609619..a5ab42a6c562 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0acbb08d603d0f056513b1d9056a6bb01146f8c121e42ebaa32d778b944a356b -size 24497 +oid sha256:5cd20065e9d97b01c141b021f5b3b552019702b01b46426f5a4fd88a78e6f6e7 +size 24546 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 6670c6d2f033..65f63ea07845 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:442d049998f294fab507beb447108d6eb0502c1beeed42853926a393f424127f -size 23328 +oid sha256:b7489fc8a29d0360bb5bd9ac45a6fe23b3ff38c067286c889a5d932bf0d6cab1 +size 23206 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index fedb21bce05b..ba261f445134 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:752b99211946d6914ae37e4d0fe93710e869426a9e5d49f55603b4340746d3fd -size 25194 +oid sha256:dfe962442fb6370fa3600227d7b8facdc7fd45ce7e74a149abd19f8651b43d47 +size 25167 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 93b211b6198f..04eea82c9cd3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2534c1b521c9b60987b0d5362bda3544321600d27fc9ef9ce79b1c16356730c4 -size 21924 +oid sha256:25a6cc6b02b8f58872406508135b6c8eb31ad9d97e8dc607210d5dceec276f4e +size 21825 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f7723b9c2ef1..731f0591889c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:07e00c4dc8a89d07397758d9bf46b7795910e42a77e2961b64c3a8ad4f7e3760 -size 25367 +oid sha256:d59c743fef57a5097e751f282679cfed25119aed77070e4bdac32f8424570c85 +size 25360 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index f2daf4c77425..35773d2f1076 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:29924109e579b9abdff168b36bc649d7522a3fc44b5bcb43b7e55379f2315381 -size 22438 +oid sha256:891aa1d6f1e8f251adf04e5fea7705a193d0417a4de5c03ae92bfa304ae28fe7 +size 22448 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fa74326bed97..b06492452042 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac9236090cd6e941d0132789fb7116222dd8f79b22adefe3ef0f023a2ad88e4e -size 19973 +oid sha256:4b575ea8d769e11072c3a784d4e243fbb57abca6349a3549a4f1ca672fc14259 +size 19947 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 9a4f057395ef..6f8add0aad9a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d1fb01e2be7a8b5955858070a1da94974bceb01bd6ccc7ab1ec7857dcdf9c9f5 -size 20645 +oid sha256:c6d5420790e704b1ec0f0f498c5fa63fc7b9945243476c0693c2994ce33f71bc +size 20595 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 2b36388262ea..886a38e8080d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7f3e105a05faa08aeef88517ec5a225912d84b19e0f5f2a0a9776f0a77e525f0 -size 18207 +oid sha256:efbb026edd260db083f9c0d87b30d8f261c406b59d7c43c4dc240d9c2d2d4ab9 +size 18294 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index a8c1f837bd59..8e8a6ecc3dab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:444f79bcd39967a4e3cbfd1a9602e3e4247503bb43c5c24ecc7b9c3751563324 -size 22661 +oid sha256:3fe8f9536a3c2b2a08e3277efad794127c445bb00722a8a19c076f1bbbc85679 +size 22279 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index cc82129711f2..478e82d4b1da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5e303c31eff0b62f0b6d3b5ee5e621f3f52f45a0888abad3cdb52c9270bf4c6f -size 20144 +oid sha256:e77eadf72c114ad6564d33178920e50981959799f6c1fb1b7af13374322d843c +size 20251 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 2e24b6e1edb6..b87815565e71 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:088989f2082dd2d3b166da23de3ae2d77e7e6976874a183b67dbc3c81c7fa7ba -size 19698 +oid sha256:0ae165aba28028c339b134e99ac69dd099edae95257df81381c3accd3bf6d351 +size 19683 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 6931c9e38230..dabe084a2079 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e099b5a4169f39436e3dc5670525325b364e90c0b057b8cd0f6ed17e036f234 -size 17385 +oid sha256:4461d90714fceb94ad46ea4202a2aa2d80242e0953208c7b78cc9fc37a19860f +size 17443 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2ba562c95db2..d5fd95c64251 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99ef8616b7b61fe159edb1688c5ee0743d4803eb3bc420d15e9065e397148df5 -size 23379 +oid sha256:70b7821d054d0ba20428298dc53b5c0380f78c5a37eb5095b80645131f1ee0bc +size 23390 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 432de125f434..4bc3d79fd466 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ecb30d763696d8c946a468c0a78d797d1f6af7f653643ad804cbfef9375342c -size 20939 +oid sha256:219e6437b270d6d13bdac3fe6f477ed627f4097d8f4bcb1eb65056a03e9d87f9 +size 20951 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 0010d6f8d725..b01054945bc7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eaa6644a25a3b64535023e53198a02ee171a98ba4c84c94ae69c61480bae6b40 -size 23360 +oid sha256:452b883fe892e5627d4c9ccac02cb62f4bf03b393f54005a4c8440a243cf4f08 +size 23328 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index a0cc790a2ea2..6a82a9770693 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b78d7924bafdba53317a688dbe45558a513e8c4360e2c197c49bf6e7fa3c45d4 -size 21692 +oid sha256:602b29849bf2615b165251dc7df6920a1acbedc114406c1bc4a42f1d2f8dbfa8 +size 21656 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 50b911a8edfb..ecd79f35bfac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce1ed6433f18bd85d3d593921b01789c55e7e658328a2ef3c0f652ae51bf2dc5 -size 22984 +oid sha256:3648ca28c5711741fc006405160773c417f95c587ce161c2a3738c57f8e3ab66 +size 22910 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 3a64f35e9f19..e796390209f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35c586366f8ed892a81c8b2e798c01fd52cb5735983f4a4a214c56b743ab6eb1 -size 21602 +oid sha256:dac179dca19a7956a5bd23bd37587706001d5f2d03f99bd07f2069de78937d45 +size 21569 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 90da356b27dc..061c97301ad2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:311fbc593bea5a079f2384ec45fa3b3837138500963296381ae389ecaff71f37 -size 25223 +oid sha256:1eb170bce1d943650a5f846cd33e393fa88d228f7bf76e40b67db987e1b50525 +size 25212 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 8a09c61496f3..56355cd6948a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:46fa4112addc58440da2cce958e02cd362c63162f51f582cc76059fe2be5e9dc -size 23128 +oid sha256:32039ff898174d703bbbc0f2777e2e9ea7ebf0c7f328abcea62720297f7307e7 +size 23158 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 15ff731fa870..baae83fb4da7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac522098038b3962430d0f818c4450bf58b379ac7dcf38caeda04559158555a1 -size 20514 +oid sha256:53a32567c483de3077ab6864ea456069f8c70f2a585775a506243143d53886ab +size 20495 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index dabcd9334edc..c281b4f89b60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:152cdd5da64a809a75e9dd3ca840006c087489313cd1a45cfd2f8a0760867d87 -size 18061 +oid sha256:0c3dbfc61593c9d8e98ff97b3faa5b5cbbde66fa103dd6ce2952db06b7bde941 +size 18057 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index e2a7364378f0..8cb561cd77bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ac88d138cbc5e33de58fe654b0ccc3a335b997de90a896fc2987201d41edd6a -size 19629 +oid sha256:a00befaa48764b589923e95d849df27d20c9ec037d6c3ec58233c6e287d00a45 +size 19597 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index c4ce2c9ce1f1..ce64e08b44d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9a2f9937646e3e517aa0217f9ae4f83b0b8c7f21aeec112cbc778889753ad9b -size 17307 +oid sha256:7637d9df00c54c674c4c5aa974bfb0abd7389c346e0fb8a18851f47f65ab46a1 +size 17298 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index db2f8dc21103..57e32b9b0352 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:422f573a261eed99fd33ecab01fa357aa26799f80bf5f1712f6e3fcfbba5d5da -size 24518 +oid sha256:2b5f21d3913cd472af63efed20d4495df6fc76fb6db5d592e83d5602899f9a7b +size 24535 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 29fafe3ec111..48dc336cb977 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d504bdaafe0f0890385907e33e45c42c9c321ccc32443a129351587df990578a -size 22245 +oid sha256:fb2214afdd812a1908c1a830bec556d7145d327a14672a2bf912e79f117fe6e7 +size 22237 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 82d373998439..13f69950f8bd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9177908af02b5daf695c9992da4d015e38cf99f93bce98ac8e144f0618f50715 -size 24067 +oid sha256:e08663bc10272fe330521cd98799a30dab6c586255d81102388c57c841c564d8 +size 24007 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index fc989bf4a6a4..554e5a2d03ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa16db59641fa24aae954dbbfee7155830d6d7164e7a7c704d732f0cdc11b9c1 -size 22660 +oid sha256:ca2fcbae1f513194896f64ad55402c1e76a7f3b34e35a9606618f07e95a9599b +size 22633 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7128c926c191..ef3deac8052a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:213717140ed543aeac6d28000e49e56d27da59d4bacc0edf5bc897a6088087a1 -size 27245 +oid sha256:6803d1af12d736055f8f87e00eeb846f68fb9494eb10e9603e112d427a056354 +size 27221 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 115bf9599c87..39a494aa7e8b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98501e9cd32d2499b567c5b225037790f905ba57c299874d9e96f359d28f8005 -size 25350 +oid sha256:9f1175980937bd27369168a85c771e1b68d3db51ff37ad038883e1f956588961 +size 25322 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index dbd3816708be..338b26d1005c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca68153ed90e3ffc580dd4624b91c0b0c666a0c542aabfadf47418e54fc77835 -size 21452 +oid sha256:c012b677e13d269a5baf483d4cce39e4e75976bed6111661ad1a8dfed57c6ceb +size 21419 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index b219b4c4202a..10c12ae294bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6d7d02fd8d9207070b76a5d6d77a8e1b5b42e10ebc07a8b7775a5295bcd2ec06 -size 18661 +oid sha256:998d14de1caa3e19f9a968c5e8b4041f566e061ff1da3edd7cbf1f4559bef5c3 +size 18650 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 1ff4c73bac53..e2154a257b4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6597fdab926412de6a82b7ade1c23b518774d5d2a2ca6747473b23104fb03733 -size 20474 +oid sha256:52009c36395a3c073de4ed2bf3e139154725341a77f7034fa67063a2f86d2b30 +size 20499 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index bd4d5f75b664..b57ccf7db1e3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:642abc45a0bd19678ba98243040355d0f4b9cd5dbdccaf1d0b27abd582802745 -size 17784 +oid sha256:2437a3d23ea546c76e4e5782c7189a7f5b7ed29841e20a8c854444759a648aa3 +size 17702 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index d2c4fb02f34d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4a98d34f7abebf3b8ac896441f06f445d9a55b72660eb761e7569b778bdab169 -size 30774 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index ef16d0fb95b6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:2ba58a05e9b0299dd88b2c36661852de2c101e1897af23cc9d3f64014fe8375b -size 24501 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 79c1434869e4..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:0b0b52d4c0519af8d80cc0e4926bfc8438751714e16d060163cee729cc764014 -size 26735 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index beb9a9d6264f..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e9791d0d4dfe0eb39d3c71df246eee0c41f2506fe853c53c51987b71e066d504 -size 23027 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index fdd6bdbedade..856b616b1a88 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ab3abbb28234f7aa139bbdc5b29a598a57422c0f710dbc3d861acc9f6a216c49 -size 26471 +oid sha256:b6e79099d7bc9fcb1ff18803b3a288f7cb0655e307dd0c50886ff42dd819b5ce +size 26443 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index b8b835feca08..60db7f5305e0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ae3e0e49fa1fd809f1c1a7a09166cb5d3d84503ffcf1d20ff33b6a8e00e75a7 -size 23513 +oid sha256:297de5aa7c0899981248e8e7092215247d962ebe1c049c6160d9a5ab5c16c6a1 +size 23534 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 8f1c2687ea6e..ea2b72f0d559 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cfd28ba0959047c9583b9d7f1410b1a41ee591986ad9ac391f6201acb2c03317 -size 28313 +oid sha256:2e3abda8ec424b0242101b5454aad5096529bf0d0f15443d99a4cce639891583 +size 28248 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 6563cf3fb10a..42d4f00f87b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:124355bc5164897a34cb508877346a7e659887d12620dd394c50b0c82ce94294 -size 26985 +oid sha256:71ce78b8ab1411c1df99485f8653dc92d30a1020583d5bda2c738af85fa6b793 +size 26938 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index d20ee91b04e6..2f3a7e7b4b6f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:03465ca9ee3254dc6310d760014e03b86eb0308d830bb3a25af67722413b60fa -size 27624 +oid sha256:a3117eb14facb52fc90b6b6597d1024a8bdb5ebfd59d866a0caded971bc54727 +size 27538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 37b0cadf8cc9..0cba32e3dd34 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:41126203a7dff3e1dd1c449140c6bc0ba96b2d86c025215c640952139684cbe5 -size 26712 +oid sha256:163f2a1d75327cad6e418d661227ca8d1892ce2a64730ef053286a1e638034df +size 26623 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0649162818ab..9751144ad29d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50fd95d7d142d883470ea6bc6c8b164844ace8f144e2bdb5ad2579a82bc74471 -size 26626 +oid sha256:1ddd9869df50f327bd4341e66cdcd1a3bbd76f5a6b88f646e0135196ec5146b1 +size 26597 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d2a83365646c..041ad58ceaff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:73e33d91604057685599ac58ee04ee5bad9f8ad87045518c6114e96b623a867d -size 25572 +oid sha256:5d4dd1b594eba621d33955fe0815538320379dbf714d9521c242e849efa34850 +size 25538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index e489204626aa..5216afee5b4d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3332fb97fc963a90b474a8a573c8aa98b594faea36cfde432e36e678d969c332 -size 25324 +oid sha256:9fae68322a31d2bcf6d71ac1b3cec46e1a4ff4ee118ff1f85549a2bb0fa3be96 +size 25276 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index a3a0f78f2432..4bf7ea194d5a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1cb77395a529f1ed9febda7eece02f62471da2d2c72c48908bb9dad2ae92eb5 -size 22556 +oid sha256:43cde9f8d0e81b269d6a6d379326f2c42857307c462381c7a6a5e1539b537b0e +size 22488 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 81f7fcc72335..4bcab508269f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa4bcd47e9022d4d9fe8230bbbb2733f0f37583d4806ee467bae5a70ebf60c9d -size 24312 +oid sha256:2c1b2b7b1c4ae426ee9ef7875f7d582bd3a0c28982b4777bfd44a71db1b5f8ce +size 24360 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 0ca92e8456c5..980d2d066c33 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c706748f7316f8668a359ef4cef278ab8ba1c1e06cb50ecea412b9790a6abe74 -size 21710 +oid sha256:e17bcc96be14b9ab929e78b447c19c12b30da5bb27dedb2648e0c00e521f6e6a +size 21632 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index f7786aee5b99..6a27fc9db0ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:082a15119e5cd87f5ae0945fe8d253dd9823cabf319805daace5f636245f96af -size 29371 +oid sha256:ee1e59fc2ba30791b173a6378b1e6360e6782450855b550a62f377e569d2abd4 +size 29222 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index eb33118bc888..bae4c39418b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:20e958395cc4c1a8f2efa908fab1e535db5f55556d2cc8732fdb027d492a7f65 -size 28128 +oid sha256:4e9551b32d2eb579140808831393fffc3f8ca6f24068dbcb2506388f716dc471 +size 28030 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 5b8f23c891f1..5188b226d631 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d1c7f276141444b4b43f60ad31978b284f9b8cc15db72006b7413bec50889a7 -size 28754 +oid sha256:e3e3851f2d93043d5f8b2b38b1b253b322aed198634174f0e57cf9e4ab9235c4 +size 28665 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 33248b37addd..9cdf1d601f60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7bf32189afb4be1da5a82d88d1a128acafd2976da2ec918a0375626c5cbea8bc -size 27782 +oid sha256:bfd88666de95506efddc9c90e8b88314e4150cd4e86ec695db1c9e3eaf20ca17 +size 27607 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b6db28f72d79..70701599fa59 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:066eca143149fbcdbd8e33cbf8dd1610c42572cef46714cef1b2dbea7919e894 -size 28000 +oid sha256:4e1a0e36f2019cb57cbb1483e6f3f7b6c9f2bebf6d9d0446f1854f340d3e4166 +size 27994 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 5f1ce5ec202a..2f8ec7d57b44 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85e2d928ac97845d3d9fa43c46d7871fc1a32292475e01116ce73a4c8aee1ae5 -size 25473 +oid sha256:c5c3ebb46978873e1b1edcca897c9ec189d379d6ccdbc730672a0a6b0b097da4 +size 25485 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 6c1b6e803e9b..cfbd9885962d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:903ca7cd9e30ea28296315f4cd617a24023c7c62352730676f069854f4b416dd -size 26378 +oid sha256:ac9ed746150a8bbeef1ec67b056469c8b2bde4cd74346634e64869b426abbcbd +size 26407 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 08e9b0f5edfe..7e74d12b435c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c287f812f4b64afac3a14013598650e9405fa6e3877ace52852de1f6d27a8e27 -size 23113 +oid sha256:4d2d6615d71ee4b9808a19f6b50d0d129c6aed2a8db4584a9d5a5fe57d29cce5 +size 23101 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 155bc457d5a7..27dd033148ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c3af8ccd9a861a854abe246ad1003fe4899fa2f2f56c3816dacad51c2b4d7404 -size 25513 +oid sha256:4d544390aef676969ad1bcdb0230ef58fd942ad69a96c109372400ff08de861a +size 25538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 5dea83f3ca43..5c0ecc822565 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8327e5934fa809b4533bfbda799ac4185758b007233dc17899a006a24798ed4c -size 22171 +oid sha256:3a251cd17d91da5f6c9398038f95cfad9ef6989560f8a23877463dd57cc188bd +size 22125 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 077f9c0c2f29..aa3b865d3619 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ad38ce666d8102d82734275abd1b0d654faf313eeea4869a4c3f807b72e38b7c -size 22341 +oid sha256:79c3fc709cd7d7c8d75a4d038585574ca078439c29594514fc8cd9435f5455d0 +size 22284 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index fddc54eb61ef..33056217e802 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb99f2ca6c6826c4940cbd03399b7b5d6b4480e483a06265befb815baf5106c3 -size 20719 +oid sha256:92261083977b4383c4ecc85aa773932d109a42b5f2d6d3ede2ecb6540dbcf209 +size 20683 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 5df8cc5001aa..a8f0399661ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3469a633a42ab756f55ac1051376b1bcef4ed0a89d8fe014809d59d869b6c43e -size 23477 +oid sha256:d13b175233cd0783e37cb8fabc7c324db2496758727926b9433c9cafa12b2f7f +size 23675 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 91b3e9b3c821..d04f8a693c06 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b63fa0b95e5eef6af3b67a54b6485c172f985086e693bc91da89d12bfd3e947 -size 21699 +oid sha256:082a6add9d3187a0cc71f56379b2035ba71ff3aa97ef13b8ee4e2642149808ff +size 21578 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 5ed03715ce00..d7d84b051fcc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a295257e573ab0a1cd18c16e43cc9b97123ac3b5775ac17a98027f13c6abae4b -size 24163 +oid sha256:dc80b7526ef160cd0e01e94d8323163379948b82f657763c14f79ba8ba4d9706 +size 24179 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index f479e342bcdd..83b37f0083c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8b48e51e9544eb1a97d668fcb7aad8d314bcc1c900f20a6ad7866b918fc4bad -size 20846 +oid sha256:bc3b0875ff087c1104965045c242a783391ddbdf2c084f2e043f1d11466dfe3e +size 20790 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 216ff903a5a0..d9dd51f9c613 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:123fe0dde088d33e71418f4baaff6fafe47c35dbc013dd9a909b8a2dbf1c5a03 -size 24383 +oid sha256:e8d1ccb75190814394b0de69253ef915e10f6bbda6265d6170b93f5741f249f3 +size 24616 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6be41e994078..49dc31271cfb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8de9d874d1d1f130ceb5481388eb33eeefa2ae1f3fc20fd97858e7120d0e414f -size 22631 +oid sha256:661d42b1af5b46b16548b648c69083698d7ab97b6e7b84808edbce8a0ec72d05 +size 22614 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index d48f8d8470eb..905f9e0110aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:079e41031dbc810e9a811687a5422727914d7844f83f822e70f47d5b4dfaa843 -size 21380 +oid sha256:d2f3d58eca3a89648448d4756f82174dbc570561bd0f2e4482af2fbf5975460d +size 21248 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 848fda33eca0..b5b559fe2dd6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e443e2dbad3e3641eaffe6f8b92ed9c808b9cedefd5444616104b367102c0aa8 -size 23571 +oid sha256:8ef580e5e0cb5df5423a43e2ad0e2157ca7a2afc55686bfa7529caf4eaf7c354 +size 23465 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 1291d3799d41..cabfeaa75b29 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c067178ecf56c13f456bbc7069d1b717ecd08afa1d11fcedcac35fef225d38d1 -size 20608 +oid sha256:d81d60f846461241ab35f018d50086cb17098dc3159498c0ac7d6470f474ffc2 +size 20405 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2d75af1f45cd..eea6465ed0b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c3df11e32f114d5dc812df1c703a44c728c13a7f5db3d68bc4fdfee40e7dd69 -size 23610 +oid sha256:76c2a77b2a462c60b9393d6bed3f30148076b2d52911cc145a13973f344a10ad +size 23548 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1a61c90fe1da..22b5892d5acb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08b184c8fba0dcd0737f90d48bdfe1256907c62cdd63ba8a4ae8f5065aec1b5e +oid sha256:2c41d0fbee885635206d338283088e193f83bd732d6137f48013b23f68c2897d size 20534 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 83b1e5dd5415..f8fd67c30328 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ee82002e1786e2b196e011c3fa0a75ef757dc840a8ab2cf66c3e009dc0437db -size 18543 +oid sha256:2f0dc69cbf932829ea86cc69415a3bfc9ba319f3739d308d2adc98123c251e27 +size 18547 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index d5990efd097d..781bffe1417f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28c22c12b39fac8c52a2edaebd14a5913763be3c5df18918dbc40f1a84bae264 -size 19293 +oid sha256:a9ecf7466df16f3568b79187014382b0491c692d031f82d20239e6f209a944e9 +size 19372 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index ff318ba067a8..22a793e9a465 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:52b6a245bd28714e8eae5f4bbbdbc5fb96dc5d7092a92aad78d7845b6b67b5e5 -size 17103 +oid sha256:6ef33a368df5239a99d2da633a0c60f7aebf35ccf9473f8c5386c70f276ddcbd +size 17173 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index fe141904f4d3..6a011c3110e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9540f90e3949d28a8dcda7664ec83f4b388d05c6af3418ca9b3f3e63675682a5 -size 21119 +oid sha256:4a74a63fdc44a383bbe625059f7ab5072e4fd23b0a1570dd5c128fabae44f14c +size 21217 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index f071d4be07d0..8e925826f367 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:054c666965399610ff1a1bd122ae2642d5f6ac2c67667ceaf8546ffc097412be -size 18813 +oid sha256:05eadf96ef5bf46c45cacaa5c7027f27ffa34b667e34f073030a455e0cdb096a +size 18978 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index d2d6866a2e98..c9107b0503aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5736a224365b5a8d8635d20ab98135fdf57c4b9c3ec778a18570b73e05f81bad -size 18580 +oid sha256:c89f9b935c435d62325701c809e324a643602ac98555a2c9ead791c36b068b6c +size 18582 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index c83a74760fed..1daa7ed01d82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:566156d63772a58200bb31f116fb5e5abbf07f68de83523ac734a51dffe87ba4 -size 16354 +oid sha256:15c3bd00179ef09257019c0b1e4cb88636eb820587711e3d97aba63d6f256fbe +size 16430 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 53ad803dae19..802e1afe465a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:820ed7040fc4efcb47d463f05a686260488d05248b3e588a7c082121a03f771c -size 21468 +oid sha256:62f7d8409a17766f45c9b860c258c730926f684cb06851087f63da1bcd7c7bfb +size 21449 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 777204b0dfe9..42c6e89142c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d956a85d0c2249a5dee9e66b47c97d8ae52acd21e9d9247e05da6031865cb9f -size 19548 +oid sha256:27ebd58885e731901b88b00e2b1a553b90507ff51724528862bc0e14f70be165 +size 19522 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index b397ed36ec97..e63536bd7de8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b821a0a15f646b0895057ae41a905659b5a17b2d1e1af6f45ee4827e9d14c66 -size 21256 +oid sha256:ce297b2bc793b8615a9902a7c213f7b6b82412c412ea648f170d031d56dff4ab +size 21211 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index b10d5c295bac..fd0c7caef0c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9b7ab88045711a2e88bb4265f66991e1685008b8770a05ef300b2abf41f371a -size 20472 +oid sha256:4546611a1b2ac9201bfa3f8bb401140813a3c9229f556d3d78e82ae65d66d9c8 +size 20466 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 5a42dda5a809..9d6333e5a056 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ebbc04315d531a069b993e23669953699d5a23a4701b9d3997e8786ba1e511ac -size 20955 +oid sha256:775d072e7180a9aaa54d08a628febcecfb4c7b86fb1579c2c70b01da0bdb6d94 +size 20920 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 7e958b11279c..26ca9a690d0e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:408d80898a0793e4675c4e1c2fc54cec1ffe17f75b344df77738b4611baffb68 -size 20214 +oid sha256:03c8fa3f7c108f07f4bee9cc635e3c8bdcf86b91b64e16709b4d5c8b09e1e423 +size 20164 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 42cd1c869eff..9152b9096dd8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d94744d3989b478e04f49e2d141a7f78f1b1bfbc639083087b07a98fe6c04ec -size 22551 +oid sha256:35168e2ef111d819941f3df8a2ee09d60a2bd21faee2960d5a15d7f4fbb393d3 +size 22563 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 0e06dcd4e3cc..16e125b47472 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b73dffac3143cca775ef7cccf77ddc572b99a89a1cc31a380ad1ff4e989415a3 -size 20431 +oid sha256:6b9db82d3d76cf07794c130d3e2af1147616e6b17f36f1fa22b9459bb03d87dc +size 20429 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 78c300dae191..c2fc89aa2b96 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8da759f5e18414cc956c5788b32b9b32394f81b901fe5ec47d97d4bd5bfe33fe -size 19270 +oid sha256:2f875f30eb6d6119ad0411fd4b0bf23bed6d016d3fbcbf4a5b02301710051c03 +size 19258 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 6e2ddca5f399..1dd5913ecbfe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b7244a8931f884060bccef0d04ba61013b882836a9a6aff8540b719c8cd0d0e -size 17024 +oid sha256:b5c3aa7158a1c908f374dd6beb36a853849766bd7f35901f4fba8bc4dd9908db +size 17002 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index e47979cd52c2..cb36e278df83 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b422575f5078e386c87b5b2dfc53de965a8b20a4aa7f33a440e5e70a5c77a99d -size 18465 +oid sha256:3eeb651c26a385f547c40cfe6d04046577cea6add130bd024bce23b0f9517cef +size 18480 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 5f51732e238c..f7ece265bfdf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:905c4c4f3344048d4766c2833f96eaa0bd81094718f8cf283a1afe81b76f7245 -size 16292 +oid sha256:52c71e9129682df532ae98eb5e2c495510ea034ecb722db6d42406275c929f1b +size 16220 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index a440a4047a34..d8791ceed8a5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:249973c8ba5af0086e38a8c8d7454e76ad741b5d066ba13467514c4a4ac17975 -size 21717 +oid sha256:c4589199299cdaf388f0f850ed7859b1929175a5ef69ca6e884f6fb057a7fd92 +size 21706 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 8fe01c72d798..04ea162a62c4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:068685c16db0292ba626bf8c0144b5098ca669e1b20a78ba3da719811dc4c96d -size 20913 +oid sha256:c77f7e8caf8cc5d1f670755e6786c971e058c1ee232cd561e76e1526f75181ae +size 20877 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index bb9e5f9426cb..f18f9c13b851 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9fea1b3c6728f96498d7dd652d976b775f786896a516d68e67a337b1e1d81e90 -size 22161 +oid sha256:516c90ac5d01409917c55a3524b68a1c358e289b6dd17b8f9bbd8cbabdf3723e +size 22065 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index a8951c9d9b44..5f151319ad78 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:11f0fa1aa1fc8d591635bd1a342b74f7a970e6b25762fea6acf589e5d0956827 -size 21416 +oid sha256:e50ffb9c7166db4b8c760aeb0ab3080183ca8caf59a96840ea2f51979c9de11e +size 21342 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e88d612d28e7..6c967a20cdaa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8758a04991c8e57ea9e6d1de546099043a88f05b88016874673c600cf636ba17 -size 24411 +oid sha256:8aa423e1ba4e2687d2c8d8b03c9c3aa51abafdbc4b28e425479ba4de8e181cf6 +size 24365 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 5f5dd7a84c4e..656528f7f2ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f0d5c6202b813ddd51fe71201cb9cdc645fd3f880304bf8302cd214779808b68 -size 22620 +oid sha256:38b263e27842060fd84e96100349a9df115207854803d5cf3b72f26764cfafff +size 22572 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 931944e69ae9..dd018f0fed3a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d75b02a5622905aee5aa39e2db66f157afdf27d55097c09fd78c9c21a47c0c65 -size 20122 +oid sha256:36c7f169e4da46decc11b2d0ababbe557fda3120d090b104bf6ca79393d63e3a +size 20085 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 2c74d7553a2e..b81aa3fdaaef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c653664663a63715ac03d6346122408c5f05e5ab037693cb0a3e934d4f2d01f2 -size 17589 +oid sha256:0a6fdf147feb285a0dfe954f707f8fc00e91cde01b7f9c71856e9e3cb7677f41 +size 17570 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index ecc33e177c7b..f38db4e70d30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f0d51b85c52e5adb41f042919ea82ffd7fb00acf09a5002e3f987a12c515fb70 -size 18963 +oid sha256:28c611d9c33a22d27f1f3b49a79cf88e92b859a80199f803d9c7356f4f74c019 +size 18935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 0b7fea458017..bdfd346e5a33 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:96523fbf4a67b58a519d59211d517713ce1d00cabc727335635f472c8837e760 -size 16797 +oid sha256:6ee23da1940bbe8e8d6586a115bddf8e5b9d7a9cd46cda3f71a667fb9954a13a +size 16752 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 1156d5cc1d67..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ec8bc9bf3eb5ec99602b006d01a9e0c1a724c8528482e8880de666da80bb9718 -size 28586 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index b63f62f50b0a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4dc65fdb74f03c9cd71c3dc66805be46b2aee2609d0b5a81b3de4e7d9a73131c -size 22516 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 97d8e97e31da..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e24f5118da580be65774d893d8ba0f83659eb32eedc59971791c2ec6e5e0226f -size 25046 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 3ef340bfa85c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9b8862ab0fae9b214e28376adcd0b26ac3194890020110d3592ce00e41a6ee6f -size 21718 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index cc6c30dec895..78f295ae1e9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:52364e50919442ac53893ecc2189061e0c94e1b158d3988aedb8bdb3f016a8b8 -size 29239 +oid sha256:4d53717e3d120a47947f434391220d19ab0a51ac6977a79a0d8031c54f06a400 +size 29969 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 2b3d23fb5c80..8134f5c012c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:620f48d46d29476f606f34a0bcbabed12d2c230b9689dd1d72b39c1b975f8486 -size 22038 +oid sha256:1fb5bb6ca655d28a59f342bad308b618a908ff944ce76af928664a6d13c9ed6c +size 22159 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 6e40e1fc99b8..6eed5ced5b76 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c10d397ba0d530d42e382ae66f629eb384cd6619b6e0e9efe5fdd669f4f684f -size 39330 +oid sha256:7beaa5118b6866d39c58b3897137aa1882c4b931efc352c939e7e9d079e66058 +size 31278 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index b7e4ba10b79c..682abbac4ef8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9345dcfbe859b2af10d7cb4d014ca27b18cd29f718711410db75e9231e7e58ff -size 30969 +oid sha256:cb3de648bacfd3423f05cb53c6a7f38883c48a3f128ab48343f31c59fb98cff1 +size 23874 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index b8217086fa83..b7de38c068d1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e42e0dfeff127e9ab1a5ce7fe4fecbc5668d888cfedb574f5a34b43741d1f7a -size 30646 +oid sha256:760c920392920e2f6d64726ae0042e2056fd697ca4e8d26f5f51eed679913e5a +size 30676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 2ee54e541603..eadeeb3269e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c2ba81a7d4499b99f94dbd878fa4f63a8c9f6940d33407127d4f2cdc52a949bc -size 23163 +oid sha256:4679706c350d07718e89ea9f5923969310953dff114dfc6e346efc94c0c4add6 +size 22795 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 8963ad8899e2..8499d1fba183 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f08c7197ee4658d68795a3cd3c4a2e014eeac0846ffad621d908e648332de49 -size 30162 +oid sha256:fbb70324bfb484079145a122f43ba476573551cd2ae82d1bf08833480cdd76bf +size 30365 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 4e7e3fc58b05..894b863658b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:907f93d4a45afb0a80aada78d28bb2e0936347314269c5a55dc6fec566a09424 -size 23057 +oid sha256:1b5b930cfda769e0808cf3f3ec49b93375eaf46f4ca1f20730903ed75aa6a211 +size 22881 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 828e3e6e8b11..711197ebc6b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ab1c2b9cca63b8c0e14aa8fc193562f139cdda7ed6776e9dc2f335d297063ad -size 39771 +oid sha256:ef8c1090ce4c2d5ea4570635521db22cf095075891ef6c575dbb99c660f6065a +size 31973 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 405207b19e54..2cf5aba6e5df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:057b6f6d64b3e4b19e5fad29129e056906eef660da60ed533f7a213ec8b7208a -size 31926 +oid sha256:ce4282a47c0ee7225176b930e1ca287ebea0aea9c40690da0e43ff842555a901 +size 24360 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 6037c5b8fd64..9b158df3b559 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5cc1d32216f8d2470c5d2d5ca4eebc55ce8fbc2a3e88b8b70169dbdd8bf74058 -size 30966 +oid sha256:68bd6aa99e47215e37c6e9087bdfac9e2998fb02e9ec7c29dba70202b23f3163 +size 31442 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 31c4e905c121..3815602103df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:683dd4cf60dd70cefce07a6fb6258da2ec81beff8eccf32ccdc0940419b13945 -size 23965 +oid sha256:09a3330c1af996ddbbe2147a636920738dd7c34d143cd9b997ec8e966efee4ec +size 23784 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 68d47c65eb42..561da638e436 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ebde92a31f3d32429f76ad6ad411a1ac131ae1a4d8f80a0dbf176b0dd18bacd5 -size 29254 +oid sha256:f820591dba3b9de9fee7fd4b355cc0a0f89e7e510241c6d97c5bddf2f4d50bc9 +size 29959 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 7528851eed53..5e38bb6a0d98 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd99a5a4d8e6cb2a51aa72331b89826931e0f212a5e9c45a2714355edab78944 -size 22062 +oid sha256:907ca168e689e0702b71d8c03336ef4eeb242f676aa119535162e644d538a7b1 +size 22160 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 1ce9de873262..e774e4ec21c6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:387d2ba343187af9fb6bab9e36ce20160eb9756ea3dedb6c808a08adfbc14f00 -size 39337 +oid sha256:e710796b5c631b522454c337a364c945df075c1e56a4e4d2da992d3ba542d8c1 +size 31293 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8380d88999e9..54d317932fb2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:470e7b12622b6dff087f6964998d8d67c99d67028898e371bec6af708259cc3d -size 30946 +oid sha256:09d467de2dab5d2e4f31b039a8c9da71fb6d50776eda8ac63c6c7698e249b598 +size 23874 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 7d8469511ab0..d2c0f4dae47e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:649b2f96ee8aecf965784221b016f7aa52111481dbaa684a37235dfbb4c462bd -size 30630 +oid sha256:7f5bf8bdeaba613e9d22d2fab2cfd5690a731c7f9a965d1d6e4d3e59f81d935c +size 30708 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 42d5c326dfb7..bcd567d994eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db3cb2a1f7aaa89f0459bb0f69c8de8dfc2b66bca99b4624b88aef19a997ee59 -size 23147 +oid sha256:1a9880918c32adb39bf48c8b0ef1749b07b9b476845a8bcf6fd101e18a03c425 +size 22785 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index bd82e049ad80..f595fa6ece77 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d2e94f186b0295b5099a8ef29f175f5e22e20b7d7e318c33546ab737e249def9 -size 30171 +oid sha256:6c9bdae1c1ad8fb9d09024968859431cc1cf722ee74c9a5e0bb88f11fa78dd6c +size 30385 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index d55a02f77d0d..f71f4354723b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aade1506df066f43e4a25b98f22aa082ee943ebf6c9e5026f69c105f7ec1a4cf -size 23054 +oid sha256:613014ea48c81770d1359d63d80fd4442ef97538813047c068345a99e9a57d36 +size 22889 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 742b5b689887..0f7746a4a937 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9f08479943b4f2fcea729281a84462f6ccda8a8d99d953efd6477b34af5e66b -size 39798 +oid sha256:ef2abe9220c6bc3acb50ce91da737e7bc5f4e9ff892873644d1ec0cffd6f38a9 +size 31961 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index b95902b8f527..129ef727d359 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d5fbf661469c485ccd56c626b6bb0119d02e2682bcc5870af7fa6403ef6bef11 -size 31927 +oid sha256:29a609c27988c1841630cfcd5194a482211bb0feb9408004efc1980f02c4499c +size 24367 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 5e6804038711..d4a23f0ada82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2621ae6651526f302aeb1e6be345b9df346ff96d8dd6ced155977d24b2abf7bf -size 30968 +oid sha256:80969c0640b017cda8799b1366058d6e10e89919c65a5a03cb80e2f0397aec55 +size 31421 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index f48eaa0334a4..1a05ee455308 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9dd9e88e7203b6e0905161b8e2a5c7730bc890e3091527fb57e8fd2d1696582 -size 23979 +oid sha256:be5935d940bccc3ed5894ae0ae921071b760e712422a32e5a457cdb2e630a603 +size 23785 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index b0e5be689c32..ed11f327ad82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4095cae9e6e95880abc754edf4d368af70d68177453a743d51d8d768f296d7e1 -size 28628 +oid sha256:4da426b6ff38cb503128d0e6b195000bdd0364ffa9664c000809177fc63dd7ec +size 29724 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 69e0be504d03..c2ff75180350 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3b1b39192bdb5481e41ff9cf69b27b34253905b82574ec238c3501048ca74dc7 -size 21447 +oid sha256:c924165f6b41370be4588395bd5e2de68769540c578b15c38dfd768cc05fe13e +size 21968 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 5ac50512916f..bcba4e39f9a5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ea19c69ce23018db0af3fd18c8fb0c8684626e7b54fcea044873b24a92da9605 -size 38103 +oid sha256:6433c2ebfe2d2d935e8486278d8ded82d6bb98c01386560bafbe5edb3c33a751 +size 30953 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index f8d2355ba9a6..11754fde0e20 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c7e2638cfa8c115363f65341f9c8d5b48a62b97d02c52436eb963b0c46a7b6a8 -size 30123 +oid sha256:805f8e1d61abbba5cbea6ca1954d02f8835e409d07c4868154b8d8680586a2a6 +size 23698 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 26c08da485e4..f3ac1b22eac5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:78b0316ef236c5b6590e73091189484a5795421abd2d40fc6630b23595d8da6a -size 29278 +oid sha256:79efc7f8b6e06fd6c7d5b0866c41516be18e44f640b11108e437eb720db78328 +size 30433 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index c9f85a93dafa..a36c03a7f8e0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:70391257b3b7b1a9c3e6cd0e6ed9f61ba4e1fad90eb8e9133f9ba2ac297f4691 -size 22902 +oid sha256:695c202288d004be1994e11c769b7bd375663ff13eaec32d1e7a25762561864d +size 22567 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index d2b2e8e1ab65..0533cb6c5647 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee9c30dd5a846c53949a3b8e3819d235e3acfe7b2c0a7214bb5ca856238d98d9 -size 28863 +oid sha256:684a94c1b12c21077f0de8676665eccf1147e15f83bb5bc675a9d74d3eacd48b +size 30184 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 3d4998d59675..8cdcdbbdd5e3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:054a4dc9e0b4b8b66cc67c84ae8ae22c1b1ab4007d7ccd329ab75fd3679bfdfc -size 22727 +oid sha256:4b7027f4babefe88f367798c803970916830f3d9538653b2f556a1e3ca7d550a +size 22610 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index f08d2c800297..689c32eb8733 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:588b3265a81d22b42b46a913d8279c6b2f305721e39939e9c29c45e4ad9ddb0b -size 38726 +oid sha256:3ed40ff8d25875e7710b67195fde79a2115bb0c1508b4cee9d17af80af7f181f +size 31646 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 4b9e118dda4d..250d0143c0bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd8ca6825d6e73478f91760f0fc552a9efa57eaba09539e55084cf1282ce6d50 -size 30992 +oid sha256:1619dcef3df91df21b1f32f073c5a0be8400e976b60fb85009bbb8ce01b4ee24 +size 24189 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 687b6db30104..b52a3367291d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87ef2e846714608e15c258d5991f4ab5ca8eb88b973c5044edece68fd684b016 -size 30328 +oid sha256:48d56f240b8cb5f6dbcdd4df00df7cd32162b6221c94dd8b1c45dc9b4a1a2580 +size 31235 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 37fd3f931875..9da3bc5bdd11 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c16143a6f7a9993a4bb905fbb61387db6859f867605565df12e0cfa53df1e362 -size 23730 +oid sha256:ca7b1d4ec63e01a7a978f136ac448d2b56194d35e74b263212a61f8c9892f6c4 +size 23553 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index a590c6d43b78..8277ea8ace2d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:52d1e5d321410257baabe504477ae9c5f9d7492045729b3ed0697d99cba3515a -size 28621 +oid sha256:d0e52f14b1a4c9713addca8a7aa5cc980e093995a777f4c87d1ea9cb11ecd82b +size 29746 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index c885c7644125..af880ff45b00 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1483fefe7f55aae793b1adda5e7f6bdc19e923e85ab29ab5bee21517808d1c53 -size 21442 +oid sha256:80f78fc882f6f3fb3097fed4a320801280e93b8c6414d8c12f9a8b8356d88511 +size 21979 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 28cfc2abf971..96b8f5fd2be3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5cea262d87e20f1229777fed7a00248c864363571f0ca631b4b4392053db2803 -size 38119 +oid sha256:43081982c69423f454ce72df1f71756add74ae593bde884a3e8d465ae34028b4 +size 30971 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8c6b128b0886..466a3ddd7ac6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb9ef01d72e2db3243ecfd100f5587acc467153244c7ae74a394d3930c21b556 -size 30125 +oid sha256:ec1b5efd5230bf98e49705ab25358e8a29229060171b9176cda6b2ccdcf8a85d +size 23679 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index f381b853379e..5758905a5153 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:063a34276d4857ce088b4f7db110dbc7b7ebebc6acc2c9fb42b093efdddc8e2c -size 29291 +oid sha256:fb32ed2fff8f7a53141a5cda9dee523f192e811e5e020e74a2669fa1b7038fef +size 30437 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 4f157f981a64..df73dd7b5f5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a597fd665be00eaf914b9f3f981f573c629790326484b7bc13612294dde0ad48 -size 22899 +oid sha256:ff9933b570e897850268e1665db5649059a30c532d2c05c57a603577a07e2989 +size 22566 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 6948374cbcc2..e52d0a5f9bb9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ddcca9169d1b464d3bd78e860ac8558baad827e1257f3f9f3b9f9a0b2f0d683 -size 28847 +oid sha256:6d521c63b66ef9e37d5cadbbab1d5dbca79a86775fda5efde9ae04621f897498 +size 30175 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 9cad320aa573..1413de47a6e6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:544ca7be375c08a9fcc58cc14af2a7fbfcfec524919d8885fb6a3abf74bdefa1 -size 22745 +oid sha256:2bba90fe42b297222f3ddda52c636988c03151b3f4fb971d7bb1eac03fac0c85 +size 22608 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 75be5b279505..24fd0df23a59 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:31668f7976a19d88f20c7ae18f937b84e207d95cad8da4ca4f716b259da60546 -size 38731 +oid sha256:032ae83851d87db6289e47e5638259bc95734abf2161dfb330e5f1bcfe06ca99 +size 31635 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index c58eec915df0..9dd09008229d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0498275c6722ff48e3220a940e6d7e3945c95040ee3a02ccb39c4c34135471ff -size 30999 +oid sha256:3dc3fb7573287d6b92d7954f3ac221262d20d18c82a76367909601e5a5b54487 +size 24183 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index e4f3389dd2de..72afda904254 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:787b0555e0d235a0af9e507bce7eb916cbc053d673ced1bb85b2065ed7acaa4b -size 30316 +oid sha256:72e8b488c4356d429765d53d8ea2af2941aa0493526bd775ad24ffbcfa688df0 +size 31222 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index ab734e733c94..e0523dcbf186 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d48fc590172f5260e971c7ab3ffb86260621cf53aeb19227063869a5729fe9b -size 23738 +oid sha256:0b7ec473282a1204b5cfdb5bd8a8fa7f98f5cc689c677c57c44578c2583ad2fb +size 23562 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index e35a7f81107a..4219ae965fa4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7ed62793513cd7b4ac02287a9391667684b14a203f067eedf928cc15bf9f4856 -size 22625 +oid sha256:da0113bd445776dd7b9139231a4642d53dadd4c90f3b9d6278625cb3b2cd5b10 +size 22517 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 4407d14ba7a7..8ecb82343276 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8fd36250cff002f191fcb61c1d101eaaf5c8aa35f3b69b0040e49dbb1f2220ce -size 19489 +oid sha256:dfda4bf6fd7f22b82ecc3157ad942c73adfce79e1ad285f59a551b987935280a +size 19335 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index dbe452bffc48..79fb2d9797a3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a3a586b3975f9c5b79d5d3037f0e0cfbacb937c3391e7d2828e02559d8e2bf74 -size 23642 +oid sha256:18b279c0e206ae756cc06ce53137c7de1f46dc80d1bb25257381b86c2bf1e7f0 +size 23542 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c3995e97a733..473ca16dcfdd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ae483ea96655106f32ddef9c79abc2a3adb3f79353e1b6967b33a1e2fc4e957 -size 20554 +oid sha256:9837b2e6b781eccb349f2441af9f0e514d8d125835c4333f926b5cca952fcaae +size 20496 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index feace7954a3e..567ef4dee938 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a24ea2d8102c4f60f70cae750cc11c594bd1eeb20385306d405e093dce1f6efb -size 24224 +oid sha256:93700e9dac63cb32d9f0c1b8affca06e56695722eae624e0c5922f46173f21e7 +size 24155 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c1d08d93d93b..07fc344fe442 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4476925ce0245ab4fa328b5b3c07fe01ad25edb6aa4ae7032f09f585614873a -size 20894 +oid sha256:4ef8eb82965416ba465272275493c6873a735dd400c01c456f4093b6096d51c5 +size 20767 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f088ab7bce9b..24f750c96377 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48297d8a57353bf1ac660b18ba7370b64dab9b79b22286d3d763efb9957a7e99 -size 24434 +oid sha256:35cba586b9a9ec7a5bc968995883b0ebc45370a58ad844fffd8728668418a05b +size 24371 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7e17cc801763..e3ec4586425c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7790ec64c903ada4b9d171693ec6a987f00fbcc8b1e81bf5f20eef774dcc0a59 -size 21387 +oid sha256:17682cdec5c199205bbd998ee6b579868f664a9548c13ad4daff3fd34e6d620e +size 21262 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 40245f0d12c3..f0c36b146c70 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0434259be8a8c1eb4c4d98b08e484e4786142dc5835c5d5d16ef6f3c98637eb1 -size 23625 +oid sha256:df402b8fcbf1e5de4e47e7f7082e66feb57de15d03a8975a42a77cc9e397e003 +size 23464 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 75ba9f381f12..bbb7d02d2b6a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:528769e03b3aeb94ae19d9ea1e10b81c3a973db576801e04e7916784104683c9 -size 20744 +oid sha256:b753965cf210f471b3656d40bafd9d919558e294cf5246cf5af58c77c7fb6b3b +size 20604 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9461c6303638..a1725a4bd1fa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6970a013f55822cb106c9d3657c43bf90a19f19e1d8e27166b4f5740767156e2 -size 23876 +oid sha256:c06cdb87bb42273d58be69cf4e0ba4056e9d45e91f6cf941a8e03eed129934d4 +size 23702 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c18b4d0eca05..29dfc58cd86e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dfe8747005525e916a05a7ab4d96b7feb47dd66637f2eef7f5fe071fdadd5e58 -size 21199 +oid sha256:918651bb3c8df2cf46e14492b2e7cafbbb61cbd4812a242487eefb878651e5b4 +size 21067 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 689aef5ac7c0..abf71223d460 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bf7ec317fd8c457f673488c2f6e27df0de1a317dd56e7993349f126cf8d1625b -size 25752 +oid sha256:4c371b1f3cb54f9f406d6dc8f114ebb2d8f3011c0c884c3d7e839f45ac3049f8 +size 25556 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 86cd72b95c51..0c787bfc28ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54e726c95868372a2376f969c8328ed7b87720db8247b102fc9d09f73dd41950 -size 23044 +oid sha256:0498524c8789d8b762154c2da21bd53b63aba1484c77578730437b7a9531cdb3 +size 22799 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 616a5a30925b..4a0444316779 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:09044faa367db3f0b0f3993c8d79ac5c5d6211a4563e0965bc87e30412fa46ac -size 26029 +oid sha256:9c926527e85cc1c4dd304612e6b28e246d467c3b5c745bb0d035f212c384efe2 +size 25804 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c1707cc86b70..7e1b28c5f5ff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4433320d2e198cac7a8f3f39f6e94f69634db54caf92be1e8353764e79a93cb -size 23656 +oid sha256:e47eb2638fbcbb4f510a69a1960a9d6150c62812a0cb46ad8918b30a9b9f2280 +size 23475 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 38207ea88004..e40dd16f127c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a6af033a75096790a924c27e3186dfec4665eeec4a28a86a8debfaa80b5bc9de -size 29621 +oid sha256:5ec86eb4776d6b394459936e54ebedae9a9e7e1f92b9c761b84abdf07e373b3c +size 29620 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c94cc90636de..8b3335455300 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b204dcf5d8174d9b22452527c47a092d290989e9c9ce93429a2505c67db1622b -size 28272 +oid sha256:17ef734c5d44a22c76eeeed3de1838646879b260df0fc9e66c62dd9f19261b2d +size 28219 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7e0e81e6fee5..71b6d3c7d62b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:997c7863a9d091e4fbfd5c319bcd798e5c793c23e6f8e9c6968a79a0fe6c486a -size 31810 +oid sha256:6185c3a5817b8eadd4c7a4a215071eebaf873c003aad86a3e2ab4e8360dcf979 +size 31737 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index bbec21b0ed11..c98f9928916e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4ddd06fdb0b0b6bcb5fe5b3680cf860178ed65515e60ac5b2ce5617549210c2 -size 29159 +oid sha256:05b1cc51bd737e9f3241c47627a0d6705688d7c485d56ef224c31c48cb4ed985 +size 29034 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 875dabf73694..fc003211333d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb77dd22ad72ee7d741b5dd49d27ca4e2c9aac25efd39b2a2a0b386288813fea -size 30259 +oid sha256:ab03397f7a26e0105f57fcf3c81cc8907b7abb3e916e0df0cc786494dada2d7a +size 30134 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7020dd75ddcc..a7c7197e0860 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7f03bb0ba17f43b0eabad16aafac5901a703e899bc75b9bf6f113c7b6f5b4e13 -size 26948 +oid sha256:722ca36afd998af9810dbc54684ce5d5c2efaf45c0f05af0dd7800ce9d4af219 +size 26841 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 565db60db2e9..4a7107d2ff76 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f4b4b9676c0f62870eb0611139c26ddc04969df681a005888acae6085a836bc -size 32486 +oid sha256:efb8006828fc04f84ba58d3b8a6ea79f2b34bc8ed3481289e109d8ad6488a817 +size 32296 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 57efc731eab3..84516173df2b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b28b1f7d2a1a63229b1eb9761c2ead85732b3d52bb9b0d74fad7cae8ef190b56 -size 27812 +oid sha256:2e7bfc08534a3a461f8bf8b1082141ff9602f74fcc1899c26f926f2fa2b001e9 +size 27738 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 32585b04f469..e55a1685c51b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc5abedaa56eb03ae927e842c0ddcf78d5b35cc4c215e9daf50bb3435210938b -size 27380 +oid sha256:eeb3442cf2efa40093912bc08da85e7e00e8114ef9c02aed27da838ba12a1f4d +size 27318 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 90f76cdcd0ba..140523a80777 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cc8f217e5dc8b357c5948bc469cf589e54d30ba55588a03ebc4d89bb1e1b3089 -size 28673 +oid sha256:41f5086139353ce3f11dd37d168ca17a2e7a3d00142343b65108fe7d1ba2b13b +size 28473 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 5c67a1515800..06e4e9d7ee6c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35468c8c04faa10a8e63dd5d146413d6287731a97f5ea9af8d63db83bd570bc7 -size 24818 +oid sha256:33779ed322ee07abc6cb5923dc15351c39303bfaa1ec7ec124cb8c6aa65b1c0c +size 24668 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d956f304a630..66608bd0ca7c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:51f5b43bc0960c24a700ce34317f0d6d98fab8426d1580c014be0e49fffcf51f -size 25210 +oid sha256:eaae8b08addcec916f2faa1535e32c7f55051ccc102b1da4b02b5cb249b6499f +size 25157 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 0345fe03c271..e73b491f188b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c5582acbf17460da2a12ceeb35ddb499e989f186ac9a3e8bac038a7a3089efc -size 24209 +oid sha256:0be7939b425cd83e5ef2efe32763bb5384a68b338e781d998331fef73f18142c +size 24201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5c2f4f36fac0..15cf6614009d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af931c0afcc94c5dd684a69998b09379c5103d7acacd505de1596cd38560c8cd -size 21060 +oid sha256:8d2bc9523e11cb8cb1e7d8479a1cd4dd41cbe489d1b06f7be718fba4775c23ae +size 21162 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 76f36c5da603..f935955b87f4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38b88a59beabc03da3383c06761a258815efd788a523fe70f4bde8f5058d4f69 -size 27524 +oid sha256:c864cd4dbc994e72fc73c79d9b6c244e312676a267a60efde3e4ab437f139ca9 +size 27437 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9abfd176df71..d0085d33fc3f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98327eabfd603e2656d2048bdc1ec080dbb63356657277ecfde5774a5f795d17 -size 30502 +oid sha256:c4fc17495e733c791999c84089d7f06312173af154176a41a542ede4324f27b6 +size 30487 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4f3000ae3e86..822d456ef8c3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49b2888db28368dba3b1f6c8f432f394f9b5e514cc24c1401f17fe8a7f7c1396 -size 25266 +oid sha256:1458d3ff037c599959b218ef465d44a481a203256948c63607d3084e6c9e886d +size 25103 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f44edf1f69d9..1abe61805727 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1d3f9498c1209ddb6585462020389336e3f1e9a44eeef8fbcffa50b47e08abbe -size 27322 +oid sha256:f9314350cb4ebf82a316d0ce11369a064e6aa08d8eb206c0964679b1e53697ba +size 27300 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 47c1936e3df1..dca0da31a234 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ebcdea2a2e1ad7e867f57f9bacac4daca6877bae5e83d2ba824ae024d9ec99e -size 25128 +oid sha256:0fb774ecac19fe0e79c468655d3437f3242ee7808761cff176f13f10143733cb +size 25019 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ab788b05370d..2e700c719b5a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:56aaab9ecbb39d0f9d5704a4a1cc8fcc751423c5e46e9b7c806f0557c8617d62 -size 22362 +oid sha256:1380e9c50aacbd40828e00c0957e0cdf29876b072e0acd9791b499f7fc19018b +size 22454 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ed19b7d5c289..bb0aa7bbd6e5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c6bcdc59adb5a42b6f1075932c6af4bb79cd194c04e7078b5ac14358020936ea -size 29692 +oid sha256:64379eba41c3c98860cda24371038ace76e78e34316da749535dec3baa0c9288 +size 29620 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b168ce734cb3..016d0d405011 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e5122fb57ec2882ef0b8cd0fe8dddd6c74a074aceaa04ce7662be035dbe145f2 -size 31255 +oid sha256:bbe9518a2b657a3b0d2ebd298b1b77bc8916c69ee8d656c1db2f9b57d79135cd +size 31142 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a777910a84f8..18c6b4d955ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b860c8be170b97edf54db991adfec5b349316c9d231a78776de50fa426658a21 -size 30241 +oid sha256:1fe48dc714678a6454836bac0e5845afd33a445747c4b4020802767834bc8d5e +size 30252 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 38f9e51fbadf..9c2a9a5b2201 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5bfc90d265a9f4b75390d445f27b436232aff17adce4c6c7c7cbe50a1419029d -size 31855 +oid sha256:934005c1c30d192f3a9c9b85a21a17e5b7f12644e11dbf24f5d597045158ccb2 +size 31809 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 90c3ce6d7c02..3a7d4a0f5b8e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:34e28c379ac8a5107e748d16e588e2965a64ffabe2251306bef1f21053210160 -size 28487 +oid sha256:bc20d4c0a4ad7117d831a9f4cd94b09438b6363d275dd94d91e5fcc1f6274b7c +size 28417 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index af8b8a798ef6..e2fb6b7d142a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:64e5e0a67d4c8c1e51bb9e76cf83d2ac6ba084a8f1e17a9ef08b42ceb8e1d74f -size 25331 +oid sha256:b621a61add0875e7919553a8120954ba54212342852cbbd64eb85847f4124493 +size 25202 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c8e216ac0538..7d21b223dc7c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cae47c3d45d792611d131c2ca4e87ad2bd2f6b891b70fcebeb43efec60cd4d8c -size 29958 +oid sha256:30e73a6401ff1d98ea4b2ad6edd459fa099606554a77ffe6054f7e70f3bd888e +size 29909 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8174b53857a8..d3bac791cc6b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee863a0ba963da01929ffbe4d3c10a9dc4ee567043cfaa57caadee13596e8200 -size 26869 +oid sha256:ce17d0087d628af6359e66965637c18665e82b428106465b24bf1080944a6197 +size 26863 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 9f9c1b3c7ea1..1d2e3857ecd0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c9f5449a74d76b77cd15c89d4f893ae7d21c7df719f86262b3465ac54d7c463 -size 27354 +oid sha256:7cd4add7e42546ab11807954c5d79a4a0d1945393d560a14939ca29cc9ee2294 +size 27185 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 00f196777852..47f3ba572871 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a2f86cb1c0681df88607f511c650ad43a32bc7a54ea82b1419eb83db1fed667b -size 24575 +oid sha256:e8041e79eca6d36cc5de7fc633269b30b1b01bf88ee358bafdc40ffa06743ef8 +size 24442 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d882391bff41..484afd8f54a0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36193e248e02c8f224ebd1a51a885fe4bb8c95fd303ec168c35562231732b5bf -size 27413 +oid sha256:a47c7cc30d65e91107cc3ef010dc30fc139416ac01360ebe5dbef2cbad5efcc6 +size 27370 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d21324b699b0..08f215d8a98a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d0df82d638c5053ec6f82d669e4b5c6f3ce4be7c46f698d6f0bef6edcae1760 -size 24978 +oid sha256:f3da4f85f496b1ed2a7fe65e8e9e2d2724f17ad59e278303fb60c23d3831e91e +size 24914 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c1ff5d9f88bd..5c565b73d6b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cb290e9ddf2367ad77b016690934b4c28061cc7c2c8d156e7f27a5c5d4f08e91 -size 32375 +oid sha256:dc9a65d427ccbed9eeae4ebe3a6f481b867e091c96ce0acd0579d656460b0bae +size 32240 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 51ce3f9de629..c285fd0e67ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a83b3cc54cd3fb4dc6cf4d21c7ad338b1be56ed0d4780a78f270e86641eb9752 -size 28990 +oid sha256:34bb349ba5a5aa95dc29e0a71ba62ff1b7af6889331ba6e8045cc62faeac3ffe +size 29004 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d1108a6ed9f4..c8625898a49c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a1ddae9560e2f06bc5daeaac7af5eff2a4ceee27ba570995a6e3449a2ab75b2 -size 34056 +oid sha256:34bb1122d73f356d0ba9748e73844fd195b889d9baba036622489c41cdc31666 +size 33935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 65c7a731a5b2..c63d4baffbfb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ab406977167a2cd0d4ce1c9ca37a9c01ac54e84572bb37df17feb7ecec11e24 -size 30000 +oid sha256:19510d98f15a2e6dbaf060704ba667d595cf0e5702610b5e869cf94361fc318a +size 29926 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 530313113acf..47965adafe70 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:599f250d8b494b5f5d7adef93617e303d031141e9c32aba449280345de5d8c4c -size 33019 +oid sha256:3d4705967f4b32baf2d70d9db322cf1ac7966494b0582a7dcd48627d1ede83b4 +size 32845 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c1635bc1e3a8..9ab02183ae4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b24added7291dc9e2354387e5f924ab132c9eceb00c1a3c0b85d23ade99c4eb8 -size 27812 +oid sha256:54d107722e943bc46355461cb144a91690b6336a29adcd6ca55d5a4e77bc1a81 +size 27714 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 74dda3eaa8e7..00883bd26d99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5814da625aec3c778b3ff53790b98cf1a3088cf9626151324918262f99f0d8ea -size 34436 +oid sha256:f307c439abe7fb176f9cb56f24a23440b05a4373dae7db3ec873d9d4481b8afe +size 34268 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 49aeb9864e36..d1dffcbf73f9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c7f2295b173fb3b361964c0a33123f0850343dace6fb4514fbb0cb640cea160e -size 28642 +oid sha256:3bdc48feec1b0294a8ee68c45a6a2c8a4eaf08127373bbf20a997753b16eb2b7 +size 28522 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 238ee345b1c6..bde493a1a1c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84cfa1e8ebaafa7108c02df135147b59b457514f51941506d87cdd1a89c31f08 -size 29614 +oid sha256:0555697b04da2049c5d2427bff341532445283f137d76204ce0a7669eacb5cca +size 29416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 20b6bd2f6da7..500e53e880fd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d85108bfd23b2ac65863224035c62817ff99be9afa529f71be175d2350a10a3f -size 30802 +oid sha256:8bd4368d1330a5225e4400865e35ee93c74ab87dd131125f533e736a4e567eda +size 30717 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 3aad66205aff..fa5ac56bab57 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f44533f494a51817cd42d0450c55f9d12695c267bee1765a92f0ebe7ea6264de -size 26694 +oid sha256:481cf4921563300268d746a9ed279a8362ad99cc59c6443b1786033fa6c385cd +size 26483 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ca6c49818a99..95d5d26101da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:44483fdf3b71edc8cc959973cd79707babbb447606cb874ab46cd3103fdf2341 -size 27223 +oid sha256:ed51314e969746acbb801e6d23c91721db9baf2bb9a149a02b49e7b75a4b4703 +size 27183 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 6ad05eab3363..d752e13e4453 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2b5cb2e626e3925ab733f25a067d1937e03b940def7d1a62d53b4ba61ab241cf -size 25546 +oid sha256:0f3d120657c2e4377db40d760f6dfcd69a306dfa6b06e6ce88d7c4072489607f +size 25369 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e313b024be90..69e73d25414b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:723b44e1170b621fe30fe207e8167cfdab70d2dabb7e72d48352043addf9715d -size 22601 +oid sha256:3207e505bd24b418fe2ef87e9d287b07074fc00b4c25602536ba04331728f97f +size 22670 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 85ee7609f54e..1d55cc9ed357 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec88af5a6ba46a228bffc4b09a3c0597a919b6fce887f8574d1627773d4ad7ff -size 29249 +oid sha256:cf037b5a1b7bfc8e318d242599ac5eca5f437246a547722cd6166836dc24b710 +size 29044 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1132ab8ef4bc..1bcba35c426f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38428d36aa6ee4b4a641afd7db4e1753a4de92a51430210211027a7e61062c16 -size 32329 +oid sha256:4dd89ea7330e33a2327c4f85129c398d1aac6cc85ec94b81cbdd463a2f466983 +size 32240 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c89e2b9fc7e4..a323841de0da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37b9fd3d3aecc8d8ee9c1959bc3f66b96031959dcbaa67e6c4d85e644c649990 -size 27198 +oid sha256:bccd1125603eaac70fb5286a4309ad36862c5597afe10ce163fd39016af58410 +size 26980 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0b98a0098dad..485eb756ec7a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c63d5fb4596fc166dcd381a76179931bd4d7d151cef88287d22062c53e27fb7f -size 28665 +oid sha256:470274abc6906f80f331ba2eb1358d00797bb77daf40ef6ca308bdee9d31e2c2 +size 28523 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 3ed335f1fba9..be8e3eff60d5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:77f90f300f03106e3d8d1f3a4152a860fe38c4f6ac4c974c403e1a1804d31a5e -size 26441 +oid sha256:8b6d5477b89fe723f6f214899b9f9456bd17b684d4a41d1a599f6b3148215d4c +size 26339 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c364590b50fe..0f551ab74d6d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50b4bc45a09edbce8c8c68adfe9cf5798346a616cf7f4d27478401baea7e024d -size 23644 +oid sha256:49002a0d2d7f409993e1bb8df909b4c0009da1e66c756c2633e8931c5c6d30ea +size 23771 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index e51276e9ed7a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d0cd0672f84140d5eba6b654b2763b2de6051edc3011f4da858a98457d62d678 -size 37617 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index d414d29aa4ce..4a64e08fd711 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c54289edae9dd2851dd0ddfb5fa62dd6cf60db36c92522e8e758d768c712b76 -size 24039 +oid sha256:3e592f92dbc193caf5e5a80200b7aca114a8fb49acc949a271e1be5750d52df3 +size 23993 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index baf4f7ec166d..b8c4751b9a19 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:903de1defa39745e672be575db1cad7b2bf6143b4911648e7b1f2f9ab30b799f -size 20892 +oid sha256:225e356e07d0cf6db8160c04a4c1693a68678f9aa5585438580b2d1026cc1016 +size 20765 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d7df9beb1cf2..5affbac1d340 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:86b25e3f2dd0d5b3739c2baf9c8a850982d60a64a2f4b5b43690ab589dcc3f9e -size 24318 +oid sha256:526b7373de85d6fbe938fc5c95e974728da43453cda69227e69e7c73ca45b67b +size 24302 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 94b7125695ab..14936ff2fd5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:996bf54af8be851c9fc1a37d9e74484569a1325cb641f0f0ce655d7eea80af47 -size 21361 +oid sha256:cf7daee6758edf10410ebb0ff63a24c6b17ef37dc9b226f23795471987b10d80 +size 21256 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index a905cc07b7ad..80c67d21388d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:afdf798dfea52b29cd5dce2d17a0a4b75cef936f7e10f8d1531922323dde3cfd -size 23612 +oid sha256:8ff84d6b5aedef29d475242a493689d83e579936dd59c7a2ca7b997106555419 +size 23455 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 4b4a468baa6c..7c2c3a30f37f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:24981d21621e1f7f576f4a65cf6e3973de1482f74e9f7b573e8b4e1f13b9d76b -size 20733 +oid sha256:250efc3a42a797381aa6f26b46e8b777bd866023e8a51d2ff3cbeac894a207ee +size 20595 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index fcc0e5b02f40..8986ba42a8b5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7a44101395dbe0fd1196a6e35a3dd943b46b0474012df087b95b71dcfc886611 -size 23858 +oid sha256:e85521dc050a22dfeece0aeb5da63aec659732a9d0165d8569f7f3d105a3fdca +size 23683 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 9fb5e91961b3..89c3ad36a2bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2294d2f7cffe59435b8782480d7ae0f99e09ca2cefedc356f218b59bc73dc701 -size 21219 +oid sha256:cb7db00ecf86471d0bbee6084b49302526ec9dde2b06d628954dc6f9b407f723 +size 21074 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1047aaf1faef..e7ece83ec38a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9cf20bf2962f2406c44ef3796c5e9a53b382035ae4a858af98ce64b6e3c25621 -size 21524 +oid sha256:09a8136f56a46b63e179ce65255bae213c5789e9710c8cd66b677e666cfb3efb +size 21406 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c184890df21e..1c9f048b5fc7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1df2389a226e32726920cde75334dffcf78675df2aa4f9c6a386e67503fb0a7 -size 18602 +oid sha256:391d733170176b65bedcd12ee71481bf9cd28f02ed9ea6711fdde57d135d7903 +size 18507 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 67e0cf71710d..d95f48dbd16f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59e1013e864c2d14b2c3167d8bf1887e4dcca724ee9d1235dd572f3fde5edb7d -size 22333 +oid sha256:a2ccf01ac93800df0ac131220805036d0c00a1e8b656b42a27674dbb5903f9ef +size 22275 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 29f031e37253..9b970b50d97e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ab9422393328336f8eda729c31713d1adadfe2b3a0396b9e27055b85b4564ed -size 19332 +oid sha256:3a6e5cd27e59d18ee41add03277a96dfbb1d7e735eabcd685be34056880c484f +size 19285 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ed131b4d1047..8b71d8949ed7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff60a9f53ea7431d6ef4ac193da5970134cb8c1d5d305186c1c27b30476ca784 -size 21802 +oid sha256:3c441296af690cad9c6b73bb9ca89db9eccf33bf7b3ca76d9d44324a6af2f3d9 +size 21660 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index bc5d29c494a9..d9f98d1cc4ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ed72d08bdc577709e40602c81ef4f841e036d765e68ba8ebfc25f80515736b9 -size 19212 +oid sha256:edf3f04527693d307f999cf48f82f53022a5f34751c77d4c95c5ee0a74581ff7 +size 19097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 797dcdfb74a8..e9fb71331bd0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c2d755580cc12b29c27f804c183826faa0996f5c67a0c9dd1556392470b0ed5d -size 22292 +oid sha256:12aaabe2486f156d567b32ccbfe75c3e85154dfb0971220f595757640b4ed182 +size 22157 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d5730d4ebe02..5e432bde4880 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13fffa59ee81b3f2d7995e474425e215a52db7d9f4300a4dec539d58db39df9a -size 19747 +oid sha256:da15514a78e1b221e4bcc0b42b61625f92664baf82067355e4bde34b50d89668 +size 19621 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 2d13582ada2a..b124d8223ada 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9acde9e326d2cb6c3df3cf88fa514e6caac99d435a6d87397797f1ca031c7e3d -size 23766 +oid sha256:bbc6bf62a9dc123f3e4e400209e5447b243e15534c397832e1b3ecae7c2367d5 +size 23557 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index d63a14281cac..5ef1bcc54fb1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13e9464d5de4087e733b88128ba2f6c54447c1d4a2324d4277bfbb4f25fa72f4 -size 20905 +oid sha256:ad4aeeda5578f20cd8405984cb30a32392c2acd603fbf3877aecc4c9b320bd4b +size 20711 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 50c81c92b0ca..a476f555f9bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b189adbaac0b5c0d0e57dd8ec072f2f3e449dcda98f33cbe84d7669673a665c8 -size 24274 +oid sha256:fe4319e14630b39b468c5b31443b57572642e5dd54122cf6d3f15aaf0539f92d +size 24241 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a8472c7d1a48..72a224b38839 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:621c5e1eb6823ed344bf450c26a44290698eeb71abaabe9eaa2364cb00baf40d -size 21253 +oid sha256:4f86a7c40d9449e19eaa59b5bd98c4c6e2f03a1e8888f8b6c63c372bb91e270b +size 21248 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index bfd0a0d43d13..dc23afbd1d67 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:414f17ba3c03cda8021d8002b6b6efa252f30ead074663a8677de644be7f6d16 -size 29544 +oid sha256:c96867d5ef66d75bc631e6ff4b7a9238ae1604a9a66624a50883d9888b0f917f +size 29381 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 53cf21b7a4df..414550781237 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:374d8fdab9d1af809283971df92387212dc6b3ab8950004642c9563777e33bbc -size 26598 +oid sha256:59265a425e24b617452f28b49ceeb94976cb13cfa08890ada1751ac54001a1b9 +size 26550 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 550b01338f3e..7413f82e1d98 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:380916560e00cea2ac362bf5219d9aea483331e82bec32067912a9fdf4f1ef52 -size 31211 +oid sha256:a67f12a66f9acf00828838650cc15e18a5103f1231d7cdb19767facc73cfbf74 +size 31139 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9aa12ee31730..878d07ea9f97 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dff1c8602057eb7d71b0caa18dbd5eb86f5dfba69e2645b0149942a8c507327d -size 27574 +oid sha256:03d943e769d00769c70152409d2d00fcfbfffe46aa9d14f3246394d5b81cd654 +size 27485 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e4c3246925e9..7e541b2cb4c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9a0491b42a9cc7f23f573915d00df8d9e4bb6a64e1cb93c93358af136ebbeef -size 27528 +oid sha256:239bfef67376f9064b31fba919f3a044af3728c1b629620d23b8580d0aef2c19 +size 27391 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9ed72777cfba..3ca6cf43088e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:531b1995f4b565e4c135b11855ff77c67a2665201ed81132177ab5744d0f1d1a -size 25787 +oid sha256:46b6945254ecb61db3430bbf8f3296c932e962b6920ad50f36d14f57f1e97a37 +size 25781 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 09b098940559..22eea3b82f90 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:40031d5ae0e4a7199bb8836a15441f4ec0a5f46230734cee194e351f86b6bdf5 -size 29425 +oid sha256:b4bd26686059dbb3cf7de1704047a7a91679dc3cb9513a418a4c0879d5abf09d +size 29334 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cc6dd6117ad8..0c4ba75cecc4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35d285299319da28ca640271d1255218dced333b535a1e214dc24aa916f72a50 -size 26622 +oid sha256:b92760af4a836c84675f16b8723ee188c8f2eb40287958cd5fb8f9a94a1f4177 +size 26531 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ce6407664944..60a44729fd0f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:742ff3e252577562d20d082043f94b62132210a67ca8e25165c904f8a50353e8 -size 25555 +oid sha256:d3fc6aa3776341a410e18a42216d5ade47c1018f09766f49dfba75cc22dd91a4 +size 25425 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9768e7b7f431..2851da3f8b3b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e52d2e191ace4745ded75c427432769cd31f2c81b241f6298865bf967b4cba70 -size 25285 +oid sha256:e3fa07768823041648f86b174d493a9377aaf26ef45657d8877c861ad3e27cdf +size 24964 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6810000e7356..ee37cae7afc6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d30982c43e150b83fb078cdc328817b3988a5edef95ccdef8f2f6e60a564c86c -size 22699 +oid sha256:1a6cbd57eb62a37e7a0157591b64bbec8fb76be1f44555fe7aacaec5f96cf76f +size 22603 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4d3753f42992..e66ad73d3d0f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3501d71a09c33b8900d086d1d1047da1ad9424e15df781f83e50eb75863d2cbf -size 21968 +oid sha256:ccbc6a01839d1725e64d8edfcb94aaeb4d18e973e9b197b9404bb0a6dd90ba7b +size 21932 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index a961002b1df2..59a370d62b33 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6cb20ee99c10edfe559953446ae98097cd94afcf3c2ddf9a4548fced8a2025bd -size 23770 +oid sha256:115e3584b639486fd115ff73d1a485d5aba155f8c098444d4d40dcaef9f2fccf +size 23715 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e0b8d04e5ad5..e39ba329dbd0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e27d736be7d7f52ed8bb7ecce6d768b9f7a5e59db9abf7dfca5a4a826764cb1 -size 20446 +oid sha256:61754c5654c3f9edfe4094ac50ef0b22e44e25e127d972db90cdd130abebfea4 +size 20426 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index a8ce3cb0c5ef..4d5904a3bc28 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b455568c7ee09b3a41e0ed3ba6fb7ccdad6559e2cdcb3028457444488572d85e -size 26320 +oid sha256:8669c067f880a130fd21534d8ff6c2e88fec16af75ffb4411ee79675f4db0164 +size 26174 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index bdd21037852d..1c25ef5d8539 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f5ef525b6f88c236790fe75d1e52b440376e2477532cad3eecfbff4eadcadbc6 -size 27013 +oid sha256:307c334d31c5baf2c6291ebfaf0a01c03c7c87773602fb7b217f5214b4491fb3 +size 26890 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c4a78c09bf8c..2f2b0d8b03e3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:420e09b414ba7c28f57fcc2d20ce8526acf57d4c94f017aab978a7e0eca86255 -size 23459 +oid sha256:f7554653a55ddc6dd3fe3eb17ccd644b11cfb3542ca20e7d379c0ab90b20d772 +size 23445 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6717b055c18d..3576e3279973 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:df94106e998fc3b49955ed5075511ea812181abf73734abb04cee5ad251af3e7 -size 24451 +oid sha256:eccdda2cfd594949348423171a41379fb541d4281c0daa9b424ce60d9b69d3b9 +size 24331 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index cdabd9b3a8e1..f6528e35ef4e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0b790b3bca89d55b354f2853c2b8446b2f51870b5cbc620e498eec19f113d4e -size 24866 +oid sha256:6db242c2a884010ef4a4b8cf51c758ba77fd202a79b7ef426fd0383721d46e69 +size 24755 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index dad432e287a1..2b5a165c5127 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7932bd3b0cd64f8124f488903e5f78888307a04e8675016a4aba5f4bd9688e15 -size 22069 +oid sha256:655798be9144884c552f8ea1ce09ed5acb453b1795458b51b5d51b8d5f53ac86 +size 22024 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e005c80db6ca..abed4bd6437c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a6223cc6c3bc60ebb587842336e1f9a163d72e719337ba821cb70f6f2a5a166d -size 25427 +oid sha256:1b9ef41de4256439727b3cb4f559ed9bff57e27e1b3170883d015fe4f91b8469 +size 25366 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 96ce9aff5363..3d9c5c439590 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e223335fb08fcbbd7a05e33a9a32135e47c2832f9f3e90ac1d0274926d7537df -size 23095 +oid sha256:4c8aa3ba3612c35a8f0750e8b518865341cf04d267b16ca14db8f17994a8ff89 +size 22987 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e750eaa74835..c595fd51d4ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ecd6caa5e429c465b931940337fe6567e58f7def975f29faa3a66a6eec850b61 -size 26190 +oid sha256:2c58f3441068e71b1a441ef16cc324dcff8d0890cf63e281cf11aaaac0a1d37e +size 26039 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a9f5b0a24a7f..958949facce4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7463ea9d388861cf1191151f53a2e28e14ef388fb117a26af1d78d479189e4ec -size 23709 +oid sha256:a3711373c574020e68735ebc9ac54143369b7398c962cc2393802d5c9da4b3f3 +size 23579 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 221a9722a879..b918cc645fd4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b039fa5c29919e639e344f8f6816e7deedb078b3b8b67cd87f8df9f37affdf46 -size 32094 +oid sha256:7aa0e86e1b931cefec9724d09ff924f7b6ab9f30a3e799b75a722c2d56457609 +size 31851 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a93f69ae22ab..6d59c0d312f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6537ff460fc1bc20339823ca2f8742610ff5c53a88b433d60783bf80d1470a6f -size 28193 +oid sha256:4283b9cdcc1f0b81832c4c1f74df1e700f2d851020d34bd9db470f6c8d73cbfa +size 28104 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 15174806f9f4..0e05d9221c19 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28e2d3b22874e9ea5202f401c571deca075fd44431938e509389c304f4f3e3d5 -size 32923 +oid sha256:2bbf7b2dbbece3ca6dd29eabac1f5389ea8f87b50d5f73d7c50b75992fa9ea75 +size 32885 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index daf6b4996304..d9c470173119 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:74fe8ae283ea5c21ee835a477c1878f65215ab66b94ee9ab567c4e9f5bb00df2 -size 29059 +oid sha256:2418a7c3bb2a71655ccfa544b64ac070faebb81a7f528a426c65cf6961813822 +size 29044 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f76520caff0c..bd141938c85b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:56b89027c9129b7844b8436602d1b922fa2f5758fc7b2c2402d1aa45b1dc333c -size 30063 +oid sha256:ab45f7dff9fab7ad779f7574ab0a81653d964d8b518e39dab049893876aaca75 +size 29928 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9022c0ff4700..e10e5b5a22f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:baa48b3a3d5cb1c952e68aaf993197448c243fdc89fe4322adf41b564175b485 -size 27135 +oid sha256:7248956e6e7b921bf7b91e044b6da69cbaf25a25872645ae5e3ea902a820de33 +size 27093 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index aaa123cb740f..d5b7aeee340a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef782c99b6f044fb3984a2852737e07f916864c07acf9fcf49d59b9c8842e31d -size 31166 +oid sha256:86e747e0d0bd2601f207a158c300cdd89b0396a6023206d9110b86449a4f7e1d +size 31051 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b6e3c0f8c4b2..9b2a821c2cbe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1783c80a925bb2f2afd51f4459dd6f38cf5eaaa00610d826adeae0538f16bed -size 28106 +oid sha256:044050147952996ec9c2bb119d08811574d1d259721fba933520e9375dfc45f5 +size 28061 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 862f60579139..b4c3adb84419 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cd3419b04e2557c339064e527b74efc8222b5ad3cdfcc873fddc86af2fb55471 -size 28000 +oid sha256:edf7181c2a49c8f0794f548b27d3aaf5c0d0584d6a58593a299a71b47815ed9d +size 28011 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 6e694075de4c..19943b7bce8b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a480bae4b4e7a1c0baa8581c119fb265c3cf628085e5bf31bd57500dada27b42 -size 27812 +oid sha256:53cb88d12594ff1e4ac80cc8aa9d9385ac39fc4582af34a6fbd9e3d43898f4db +size 27827 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index ba15b37cee9f..89a24e3e666c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a92c0c1c3989f75ea4d740808bfbceef3d8a095296ebdf4722134d79ff69a1bd -size 25131 +oid sha256:3faeb32afdba47cbe18f83882ba460753414aedd2a25494704fcc208847f2a86 +size 24926 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 17871beb0745..b2d38ee3d6f7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9976060187fe8936171735bd929d6278e4f1d141a3322781d3a6d1b76b5f111d -size 24368 +oid sha256:883f7827593bdc41a60d48d45464157f2af1ec408c5431703c7277bf73058e8d +size 24359 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 43a81902e830..bcf141fd1cd6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:822148838754fe6148a20b57162946955177d8317405c957ad925c3fc369541b -size 25118 +oid sha256:b3e9e8f22f86ccebae08e1e082b670a5b965cd23944314368dfd9800ba267e8b +size 25073 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 86e7a27967f5..3099045fd4cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4a4e0a324349877984458572e4873cdba0bce5ab583beb3a5eb805329f197934 -size 21816 +oid sha256:200aa891da402439cc988316534ec0f622885f453691a03d1ff17ad82d7050d0 +size 21795 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 12b41b6aa5c0..eb78059818c9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fae0e0011f6b1b3ae1891793f02563d0e3d9c83cf1b7ef5e0a01e178e0c86e07 -size 28770 +oid sha256:1e4db2882cab0d1514f20e48e181c872443c3eafa17a1c9d17453ed914ce7b12 +size 28609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 57bfcbccc522..294e89b4a4cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:093ec8dbfe6b3d7d825871839aa40b120850a16e70b1fc484e7dc019e12e4cb5 -size 28530 +oid sha256:42c748976e8654ca341207b9a90b8fdf2e673e3e274efda8a41a4445d89daebd +size 28390 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index bb9088a87aa4..e76ee6e5728e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:635b1618cdfe8c1522555dedf5f8a82dcb71701707834e642cf78e1f012b6d31 -size 25688 +oid sha256:1bf2067e050ea45015966413e789aee9b1028f50a5fa399627c83fb613031394 +size 25690 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3e045a23c26d..f320fa04a4cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08345ee2b1d4a037081f62b6f95ced60e64a7bef4b720b0f395d0d6d31a45174 -size 25964 +oid sha256:a7b656c4c03a97299c97f1a5041ee8c73cdf9c2a04fc13ecfa954352a82b0e56 +size 25805 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 537b038dc38e..c0f341cd6d1b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4c88392a3dedd8b4742386289760cf927aeb67d10d288e1aa90fd398c273590 -size 25891 +oid sha256:15376d1792f896199f2a1ca2fc1eb2dffa525c2e842577ff5d8045296ab3bb01 +size 25742 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9752f6d613c3..bb4096fb0068 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54db5414e5ac73da36aba41faf852a65ae43838ba748f7e3f632f9c097a1ff0e -size 23235 +oid sha256:2fb93b08f16cbe8c681a4706386d860197e0dfd370a5ec70d2d0f3e82c7dd4f7 +size 23166 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index e61e810a75a6..5734585ceaf9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d11b325569cee4d71a251349c1e166aad93e03e959b72c0864c69e33438842a6 -size 21396 +oid sha256:9e9a203db006b895752364b3f98c61ec0a5e6b49fe064846d1d47299a8a5dcf1 +size 21280 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index fac5de702d66..8d3c25547dfa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8922b71783c31ab558e3f4f696c4e507d1b20ebdf367e6be5a2d2ef89f6e77a -size 18629 +oid sha256:9ef6e14026b092697b87d2b8a1ec6a189b48363d8bf42986198cffa1ac0397d3 +size 18520 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 5659059fdf17..e32640e50e80 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:52421fb6351620426c621936ea2003a03b910763b13464c0fd2e1f20b8485584 -size 22526 +oid sha256:0cff7ccb2f536749554381c2a88e25bd56ef0ad47d36d7b2f7f45d5243cdf419 +size 22413 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 59cef489bbc4..af3d2f53df48 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e43007ff296fb37b7327ccf217cb21e7348a90461cc2ee71f126119b0c965957 -size 19376 +oid sha256:d0bf7e04d177f6dc74e04110e7f09700fe56ddf1adde89e84e9d9c02877546af +size 19353 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 883b5ebae593..b77adaa811a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a3fc261e729ec1d088a25c68f6532287aa1e7acf1caf9244ca0507cd3e866402 -size 21791 +oid sha256:fb51e0eeb0ce34b2ed06af86f513cab9be45382a56669477008c06b07ff9c599 +size 21656 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 5534e891238e..7e533fbd10d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c38240601c7bf4d48995e6f06c64532aedf128a39986f2e0266ee1c03202ba6e -size 19211 +oid sha256:08c28e39a5c0467bf08ef77ff26b2bf7f71ac34449daeafb40280bef8f0302cc +size 19085 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 15711c977d0e..bb27fb64f374 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:699c04c88a591a2deba3f59a79097dd4cdfddcf47a38a73a804fce24737fe58b -size 22295 +oid sha256:4d015b7bdab1ed64e7b72ec706a397f2abdf4201613c1c7a33a4ad66a859cd68 +size 22160 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c3a0c370c3d4..86be0731776f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:46acb50842fe7bc62f3630e5f2a459be37dd5510de515103e50dbf173d2af304 -size 19745 +oid sha256:4606f35a39903e550f1ffed7261e552065a9347f50ddd80dae265030f3a0fa75 +size 19631 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index cd8d416a5326..d24ee6b64f99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:143ac7fa8c630d3f61c5807b5d4dfd1318aec9acb15e3921de3c25d48bdb04de -size 33434 +oid sha256:83289a44eacae0cc2c1010b021e1c0040b643a379134528d75c2acafbbc74953 +size 32078 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index a4bb2708c0bb..89b841295737 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37eb4b7d5700595ffa07160f96b3f70f23397d559dbd8b0a6d9c45e10e4c82b5 -size 36140 +oid sha256:3a45d8c821790898587259539171f0c879495e02b510ad6279fe0bc614f7864c +size 34017 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 9440499f71fa..ba5cb6f616ac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac7c284da01cb154d85962327d51234ad848ee19ef544a02dc7e7247e448245e -size 31462 +oid sha256:5337cf3f89faaebc603a67d1730ee61f152bb4268b7adbdd3b5a747797f7ae9b +size 29770 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3fa4fa6a2ee4..6162a8d35ad4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3f8065c8768e1045d6f5242fec1efa6eedab3ecda780e66c77865d486502d3c6 -size 29711 +oid sha256:01006286892c982fa212fd5d32e06f7664e9d60cfb6efdffc2638b1e5b1d5437 +size 27627 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index a10aad29b700..a979bc6f6f38 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6bb91965517690d4071aed823ebfdc1b3d4fd903804dc9a36911a1c576cc6828 -size 32740 +oid sha256:1060e8993011a10484867741f5245a32973031048b2935f612be97aa4d66b5ac +size 31387 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index fc56405c6cb1..6e292d2a5f35 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eadb5f9ad875c31fe01138eb5d8b1b1cb2d5883786cc1c2a5da0439f9f12f138 -size 35211 +oid sha256:8a6227291ccde6ef5bfde480e81fcd1939bce895cb4a641c65d5f5d2b1c49671 +size 33117 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index b96134762eb8..13068b5ffc97 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b1c0a22c5141ec04286b65fa793780b36b21a89c08fd80bd2f5dfebdf913a73 -size 31007 +oid sha256:64df55c47c2cb79d8f36aaa9203bf78ce0dceccb0116b601a2b97d5a44d17391 +size 29216 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 2a697f4d30e7..24b2ad3745f9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:51a3ab2aa5c215293e2bfaa6fecc80dbac68b539ab3f96dc743243dc3af9cd15 -size 29112 +oid sha256:5ba77d556d488813163fdaeff0506a922f178cbcae1f2bd68f785a2163fd9dfa +size 28920 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index d17ed4ccb2a7..cbe10655c984 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:287913cf05026ee3768a298dc226539978d3a2641790865188322e93c8c6a6e9 -size 26730 +oid sha256:b3ab0f1d8f1bf2d7cf0dcea525c50705721696a0e061b5c8cac50e50c8caac5b +size 26003 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index 15cbd0ded7c3..1e849af78097 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d7e79169138230803f33b8b783be60a808d9566d840c4bcab4e3874a3a3cd3b -size 31291 +oid sha256:c7d51ebdfec5de5238314a29b28a66f044cf9df646521cee04ada66634a85495 +size 31476 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index bdb7f1094948..5f4e5588cef3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b2b29f55f2a23bb98dc42313d88c853999fe1041667b01bace841ee058dd2569 -size 29288 +oid sha256:18ca4b1436ab240c7b00dda88276703c49c8bccb3f05c0324c16e6dcd6de6f24 +size 28297 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 63d7f680f5e3..5c6fa25f3d32 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e7571513c5ede1f5502aea3b9401d4a63dd50b793fe1c044d53a46c19b192f4 -size 28302 +oid sha256:17e53a120832638382456eca5c70e4a94790ea03f21a27bd635d5c8aa8f40ec9 +size 28485 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5b143a78f99c..d9c5f2010d89 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:30dfbfd0af6d09a6342a6afdd0a3290fd8ebee09a405818d77655237dd5c7a9c -size 25917 +oid sha256:c82f0e4b9bec9cad1b99d3aa93a6511e38fa471f5cd0f68caf2c7fefd26d684d +size 25021 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index cbc5a6b05c65..c172c72a8727 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a67a3d69883d026967975a7faa1c5b82f76c47d3e5e9949e42f65880c0c7651b -size 27612 +oid sha256:c14e7e78c2a762a8b558c04c549df1d1457ba5135a04774b5b29edc3d5f83299 +size 27643 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 0fe6d89e5a45..a6185f782103 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1fe803721bbc73c89d969ac732c75e6f6125c1ced326ea995f6b5d00b12c142 -size 25499 +oid sha256:c7ac22f9df89268178e5e6ec9642ddc0fc0a67d7192ee9e63eb6086dc3bd2366 +size 24670 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6b7e3ddb092b..ad15ca90faf6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:63c7864c8a9f5e0bfa56ddbe846123f439b2978969b9a6b7f9aae4c1c1b96677 -size 22772 +oid sha256:44a8490a7fa95446ee33b68030d3bfe442970271ee09daa98b4f22106490214b +size 22635 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 711d7efe0b1c..d89a4426c629 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c32653e4fe7e9726581e2c9605a1f6a72899b6d02535476afd72d98dbce0ec7f -size 20198 +oid sha256:8d81ef3f5e5d639f402cf6ac49f98a5934d55179f0ed5988821a8c11620ec8c4 +size 20075 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 210fe5ff31fb..83a45419c0e0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e4b7947f578b2f4435f1e4a2146b89506a90baf0c06673b6ab47f77038d85a16 -size 23300 +oid sha256:b9742fafb3594013ea848cffd838667acc6a381aca4ba58b07ab01ef9f07bdd8 +size 23293 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b35e40370db0..c1b7e05a4a86 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8e7a6fa169565869893de9730933c4272bb8645c076a0a556909784f1699c0ec -size 20856 +oid sha256:a96c575bb18ccdc6a33c9100404e808721dc4c5318d42e10d448a0f1d6f5f598 +size 20849 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ed3b26f67bd8..6f78e30c09bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:565d53bcf2045e8e06aede1cdf6a1bd94d9c4efcf931d3b371a8e568dd5d0912 -size 22716 +oid sha256:964cb2830ac8fd2f7fe7ee0b7f4a7f6b574c3a7f0914c2cb182c9af1b650f11e +size 22566 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 0c755895c5b7..fdd93224dce6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dfc0a1f08fc81cdef04e2168358850306aa5a35eb8a71af582b2198357a53b21 -size 20542 +oid sha256:183f4995890c44720498382bbbc6e56dc0cf88416e161ba724d3c6d33a9e08e2 +size 20421 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3cf2168cc2ec..0937a52a4e31 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:697d81adc1bae76148fa6dff9fe5db417003ff86f599bfdcf9d3a7b2c712089a -size 23432 +oid sha256:b64578e8d50e7d90efa9654576a17fc29d94f567eba0355b8425d04d1128d3a7 +size 23256 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 297aa591a03b..a2afa3c23f84 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48edd0cba48d817b4c7c2973e2613208833f06efa2b587ec623f857e4548bcae -size 20993 +oid sha256:c6cc67a55a4ad11f20c06f400a482ddfe854fc8c3214553b734923f60755b300 +size 20899 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fdf83638588c..1b874c5120ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8970e39b1f3bc6831c3c9908c81917503e06a4e9b31f2028f254fb0703a78695 -size 26228 +oid sha256:cc68dc4effba6fad03c9f2b1df4b2110e760c94ad3b0e9a498754a566aab5863 +size 26238 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 788614437f9e..1b76fbb35e62 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ec967591eee0dd29c1a1da900e197722e0c3568a9e948732e0309a198600963 -size 27639 +oid sha256:0712c4ba31dd0beb979ef607e4426e95157fe83c663041cddbfb7e31b34dff1d +size 27549 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index edf5ca17bdbf..7ce57fe3cb6e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:407ff6f914105b99b8255c90cc6eb222361fddbcfee47a550c0aa700be1757a7 -size 28227 +oid sha256:a82a284c8c9ea3764a01630db3c52bdddcbc668ecf33ef53c238890e160d5857 +size 28218 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index d7853bd3b35e..a23eb66ebc10 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f7cd2ca7dee24bd5f71836ca752cbe5c1cb1c1b04159556c42ab3ec8e46fdd5 -size 28155 +oid sha256:18aca2dc4edf5fd0dd803e475cc957f8367e75421610721824fda1ad12cbabc7 +size 28091 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index bbf6ab59566a..773e04a8fa10 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1e80596cab1b2a2d39215d537a79f4078703e2dfd2b3da8d91e955da11225a8 -size 27545 +oid sha256:46712982726056f22c9d5f9e503fe89a16b3bace2c32f3a1b005cc9ebfaeda99 +size 27425 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index cc2ad92324d6..375a6d77a2f4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b53302bef9672352cc53ef4a684699a583bd6929c30a4885bbce95138ae74ca7 -size 27251 +oid sha256:3da40977e26e1ce7ba892f1988bc6bc06341a71db618366861af6eada83756f7 +size 27076 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 54b7531eb561..0eae82ce38c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21c25855793adce1356525fbb7d1c7ad2cd66c445d7e9ebe7416f7a5d67b0b66 -size 24786 +oid sha256:c3d158bab663f0577cf9939273588c79c9062c4c6b0c230a21521a1fb63075ad +size 24601 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 902d16dfcac9..6ef1410ae0a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:70daf352131c2bf815fbc10c61d452e911c17c4469e15f7e7c0ee7beae282814 -size 24061 +oid sha256:dcc36f74666c54c244b263889bf51d0b1a0701bf09e3a3f9ea62cb60b412842a +size 23986 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index e97c04777c83..639ea9bcae05 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b99d15e7488c76db2b3ba3e1a24d475e32c8460a6b79ea0fc6cb9160ffe0795f -size 26771 +oid sha256:0993d0833339a9d5847daa555b291a138b9cd55dcb786f6152de88705a5a07f8 +size 26792 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 96f2e27ec318..09ccb7750366 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9cb7346fea9e6b17c9f3e2e9854015f7b9429d03069acfbd9a09a16098a26b77 -size 24458 +oid sha256:f23dcdfd7aebe9dedc580de230d384b9ff0272edb51d1bb987fbc887cb087a36 +size 24350 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7f6febd3b566..e9c944ec7481 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:829382c6f42cbf5509ea3c8a723b9d9c101dd4ca446dad995032d29171f26849 -size 28249 +oid sha256:b1fb8b9b9c627d73514fdb6a818849a5e158a7eb92253f42a41da8cc71f16b52 +size 28191 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index cc4f016541c2..fde7cd5cea9d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:44ae1efb926dc23285b024c44ced1e911735a6f758e6f35aad2adb1b0887005b -size 29046 +oid sha256:8c3245bc7995c68ef7bb2c980fea5e243174e5d6ed9cdd37bcd822ca90bac5bb +size 28913 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d23b9d3050a1..edccb7f3ba99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6a12cc86c5fed70a2955e1f2e523f153bf18f3f0a35e0dd21a65c3ee65399ce8 -size 25612 +oid sha256:233ea051990f6ca36a59e003f31b2c279883033f7429d80081f99021a3d9acdd +size 25561 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index de851f0435ca..e7b53b9c252e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8de4c4b2221b512584966246bbe1454af030fd69d75df18c9d9ebfd10259ba9 -size 26498 +oid sha256:1381e8ac27501661cdf3bd600baf7e029fc8ee3bd8d6b57c418d9e2f89a82184 +size 26418 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index fb63700b9075..19f3755c0b65 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:edeb5ba6ab95008f19c1b3088d99c5b3cee67ed721330195c83ace9f99d4abfe -size 27835 +oid sha256:f104ee63fc58145472ff12ac2777d60bc69e21271d8e1b2a8c0f9209a1b4a1c7 +size 27847 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 516907270053..80bc32c5706a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6227e0fdc21d3bace82f8bea608f99f2a2c3e428d1731adfcacfe90a773b20fb -size 24572 +oid sha256:116c2cf060f93391ff4e77bf8af20cc0fed15384420a0d96b9e472576eebbdc2 +size 24507 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 721adb747a28..c44ea267a318 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0870ab407a46001a1e40ce876ab434fa5a9223b63bcdfc685aab33aff79c1d1d -size 37577 +oid sha256:6eb450a4d9b01c08a4d331e5234daa21d59ff23c54e9ae88eab2143db3865082 +size 37381 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index f38047cb1368..5389ec3cd3e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b490e89bd9f44fe11b455c600909ba18da50ec74476c5e7c0029a71167705d72 -size 39762 +oid sha256:7b08fd30274f59bbe04fb8bfbfe3153874bc2730a871e19d225a4ad2c029c5f3 +size 39217 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 0bc7d7d56999..c178fd0568b5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26120e1145f5c717f380d73784b551a2d147900221cbea799d9ef17f43d6329b -size 36666 +oid sha256:f0e455edd3a48cfd359829deb3c7a957b7a962232ac46e7c492944385bd6a146 +size 35963 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index badfa553ddbf..64c58073c4a8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a7d00e6e7b37a4990d9357dbae0257d3cef169252e55f0d1547b993fe886a91f -size 36954 +oid sha256:5fd8a69343595635828f9b0816791dbe20f585feb24dc263b6ebffbbf313fde9 +size 37962 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 6ccc05c8c662..02f3e1f32aa3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:22e9e2ce0f4e62b3b1edad2b2d84762a82271a3ef957c7cbdd72650ccb491b30 -size 34297 +oid sha256:ffadd4769d3a9abcc4f4935552fad23e08f0513b886e0f36a9f6afb39f35b6fa +size 33675 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 10310a13b46a..9f5c8adc3d1f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e9c6347f803921a60bdbb50349c63527d39ee2634729eb70e332a85ce49c81c -size 36688 +oid sha256:39092f7a725d1cad2a5b7c1167e2b84e9c36a2ba2b0197fa13733db882d2fc95 +size 36614 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 5a6950b53cc9..141bca8a40a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f543fbdd1d07282c6d777f5a9eb9cf6145c113b48cc977179b14b3d2e98044a4 -size 38695 +oid sha256:5cf297a1fbf553e50e01c9ccc2e4a33474737c201d7665724918fb575ca2bbae +size 38204 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 7524c6f61a03..19515a807f91 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:caf5f271d985c70e3294a98e9ea534709b1c8c40e30c88a41f77bcb67486fd69 -size 36215 +oid sha256:b32428a083e6ab5c771cf131e4e002037515126cfc6db0cf67029a72065faad0 +size 35135 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 30b71c7e2a9d..8bfb4e52c91b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33a1d8fa6ec14595e88bf10a0b7716b8cae74e303b67f50c4b1cbae24e93e24c -size 32672 +oid sha256:6e5991da6822c322890675ba5378b698b98e77d99f38fd32a98b124bea2832cb +size 33187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 047decc7e87e..b450e094cc58 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5a9bc5d47a6ff04107e9f4a9d66afb29e5b289ebdb6affb4b767cd05fafc9ee1 -size 31109 +oid sha256:6de43bea2a30064946a995af9e358603e24c4b53f5ac223b2b55e684428a9839 +size 30666 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index 94d0d3894655..41c829b70e1a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8890db3403230aec112a74f9e3a4fbf2c06d658d5bbaf22e0794012d5546084 -size 35132 +oid sha256:5f3b97cb32d96dac84f6e2848a4930199d14e5b154714dbca748e53c6cf32239 +size 35319 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 2636985f1e9d..e9143baaaa13 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:671a790f5bfd8da4ff9826e2657ea11fa607b965f2fb7813a102c1ee4c72238a -size 33486 +oid sha256:4b63b302d300c8e3db39a51a33c13c94d052665c933a3757e0bc4d3b4caf0bfd +size 32848 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst index eaebd023e1cd..7e789986e381 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:30cc1d750839bdc33da3ab5269248ff5d243d66c5e54e5bfb1697a90fcb864b4 -size 39321 +oid sha256:1b9cf30fe7191575732a1036d0797ca8d7bf262e04bb2879f630f4992534fd83 +size 40601 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8abbc7a286a3..7f5299964cb4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b68ddba2ed9f225de9fed0514cd85ec18a0864090f5088b8fbb1c15f9550b23 -size 32686 +oid sha256:63c2cdc012cc777a21613db3bf3d98d0afdcfefbf7d7612f691a371f8f0cdfa2 +size 33086 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index a18dea1e3e28..a18782543d67 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0af333f0eaf3ceedfefeb68025a83a95669f111e5fec8a112a09040a7fa4ccb3 -size 34285 +oid sha256:d0bb8f83163d72a8bf44df8dc6cb1c8b9cdd600748c456f2417e9d1c53f00c76 +size 34616 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 66009e3ab761..99bd7eecaf7b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9dad09f851fe62f64d07b7d82516704f4f90da3473fe3ea54caeaaf184bc6934 -size 30880 +oid sha256:142813a1a2140f9080546ffa4608218a00df3532cc305f70cb121e8a4f3194f9 +size 30372 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index eb1939481c16..6fbd7072cdeb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6a19baa61d2f2826502be293aac899946395303b0a799452de2d23610fc4ca2b -size 31240 +oid sha256:7a9fd287c00e4c520e11f5835e6fe02959c6fe530be2a72e1443ee952035e70b +size 31593 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index f3d2d1ce0b65..cea49bf19481 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6de7923a3607ef7993b930deb1bb00df845c23bf8db4cbc7ecb64f7cfa398dfd -size 29687 +oid sha256:d5bc5371e271b3c7e321a2ddb65f689bf8f13422bac5c83550880583f6e868ac +size 29025 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 28ad8f4119ef..d5d87efafaac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bda9076ea854687a9be3245bd0edbcabfd14a15813e8fe0eb8097d14e066b76f -size 27448 +oid sha256:b3285e9b659f3f0f2579f709a2d8cf558fac324853bd43df6417fe4b7565396d +size 27335 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 8bf06640652b..99996bf228dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:82536affecf39addbe56cbb9c3b9c8ac185e991ed6ca47ed31f28f19bd611663 -size 24949 +oid sha256:0b42e00aa198487fd17d5d69f1055af5e0e612b332a6404eaa6c9ce2638be6bf +size 24753 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b230e2ab762d..0d3b2af319ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:950af6cce5fea1f501b5e0b6c7cbac67f4e7d9765dabf80fb3ddaf77f466dd92 -size 28167 +oid sha256:4b8d248dcf380c248a653d1161ffcea8e4603012f5340af745f5dae222a37e12 +size 28092 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 12bdd52ce982..5751d15d709b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:52fe84355f743a8cdcefc772a21bc878248f629d0ed314125c4f4eadb37a8ce6 -size 25832 +oid sha256:e3c139acdff0989e0549c3a94d44e451ccf0cea34780e2dc313f4334ce485add +size 25658 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fa034a074eed..985ce7f1de20 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1457b86206cc79a42e3e2dadfce6cc7357d1ac2057275a4ff19a4f9b20e4d61 -size 28767 +oid sha256:4f502ffc51a8cfce15cd3c7ab1c35ce559c83c4eb2572ee53c53a16b3348f313 +size 28601 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c5bf44dc9541..af2cb058fc2b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b404435a58992c958605dc16588327ab46e2c4f095b397c97d79dacc768d531 -size 29141 +oid sha256:c7c3387f1e47906210aaf372079a62e59c4a541e11f8746adfa178024cf045c8 +size 29062 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0c1c8ca8d5f3..1c3871298b3a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8cae0aa15a24264e507c96f793c5f07dad9d5c4c162148705dbaef446b125a2b -size 29825 +oid sha256:e01f914beb3aeaa96ad6e95aaa6856291b4846c1c104c7a30b81c1575499abd0 +size 29838 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index fa1332dcbfc6..3edc890de08b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f193ef3a2cef35a5634d1d83bdd7210c2b8356da1f10bdc59923bb3616633a37 -size 29626 +oid sha256:ffb7fc488237780d9e69bdebf7c30ddfef278d162671e0fa39b69834180d80b6 +size 29535 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index ac4e8d50de9d..665c37fbc0a9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d8d9f841f446b380544ab14ed13fe3211ad7e3c6d9e35074e811f3c10269ad44 -size 29647 +oid sha256:cbdb49dc7809e779204b45f4806224567af8b1d3a9b2e308f8d98385a56afae6 +size 29531 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 377e438fadbd..a888a92bebdc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:abfd395db19a4a0bf3964eae3387d7de9469d7a66535625d594c9a2b93b7a23f -size 26378 +oid sha256:46bd81fff37ba23cfdbb5139baed070ca46ae9dfee7f1e7048db4d3c3bf28095 +size 26306 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 26bdf06b01fe..91d7193a413f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:884919fbe5f72aff4a59d197c25756250531967602bb673a4c303a3aa377a8c0 -size 27920 +oid sha256:f5c997be75e57dce39cbc23e3732e972f473e822881ab0d84d8e77f935c0adfc +size 27824 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a10bca1a9546..dd19108233fc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:405d392c513fafcf5f7652130f61f149bf284b10ea330b4d5fc9fe1aac6f5085 -size 25739 +oid sha256:1549efe20ee995fdbca9099ebfe1593e744c3f0a6446bc931e51eb632ca8363e +size 25659 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9d06a2f3a99f..8e18a51f10e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb4e079206abd935d987801336bd961c3722260e5c84828fd2ad10fcec171ade -size 30362 +oid sha256:bcdd651f61af0575cb0f4c3a8db6d37194e026e9c44826c3ce5e8476a97d79ee +size 30191 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8df4ecd9dc90..3efa81b40761 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d9710a35e92b54dc5f711c841aefb1ad6c156df6aa294de6c18df2fcc151a7d -size 27864 +oid sha256:4ce7f09204581e765fbe485978087a28747ecb491f92e5da5cc40d1665e4f61d +size 27751 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 507b08d9f6a1..00d6a4cbb55e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f228badfbe060f3a3250e199bfb72d4ee1a01693f2f9c903d5aa28b591187cf9 -size 28540 +oid sha256:eb921b72ac59d7904cc1d67c095182123899ad7411514e2af822cbc8ea586702 +size 28504 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4c34d01fd20f..831111a078c3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ed7b6f54a8ea50cf8277d88cce5ec49301b69603897cf50c0a816f0b47cb2699 -size 25921 +oid sha256:bee6532f68699768b12a850078a665f7127b2bb793c8f971680d15d98082937b +size 25866 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 6157eb55dc1e..f686e1800819 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a7dfaa753842847783ec5e761f81bce1eba6fa39fe5113990ed2326ae2462dd2 -size 38197 +oid sha256:9b71938789eb20556d9d8e4478842b1bcc359b8a82cadd37ad28be0c3ed179a8 +size 35468 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 6c70c3fd7a3f..c72faf9fe22e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e0ad56856300c6935eaf1abb661a5e5f887689012916be55d7145bae2830ae9 -size 36854 +oid sha256:665b1b3f94ac63f191543d2c55f6ba267ca3760297b558332e16ed809c7c1996 +size 34433 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2f63a2713312..64e4dd4bff7b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c4d100e7eb94e0f17fc1d9c8cb48a5a4decee2ff310fd4e42afad24aa92b03d4 -size 37756 +oid sha256:a70fd28feb57540149dab98af6deecdf026c14b3d385473ae71bd32fd1b361d6 +size 37920 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 34bc2144c8be..f1f7d7507e1e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f98d7bffd7e247fde7257d52d90a69c3103ca0d93bac0ce885b05d9e8b4611f -size 37185 +oid sha256:78795a10b847a0a73692e609ca701a4b462f524d15d45986dbc85784f9ce30d1 +size 34450 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index b5472ade3f5c..dfd0b240a9ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d6dd2fd1119db088eae80300be4b9e4453e597149cec205c6cafec406284664b -size 39829 +oid sha256:55074cd924f4b9cb40797374e232ddc55bcd764d342872c1b096771e7dd396f7 +size 36996 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 3cff7058a8de..82d8952e9df8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:638849b700c2dbfecacf85131782975bb8d360cc29b27dd4b933e2f2f21ea4a4 -size 35648 +oid sha256:dbdb2fa429c39f909940697042954935d3cbb2512dd9e22641010a05b761886e +size 33434 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 1d02f40d720c..3c42d62fffb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ab42aecd096f1bb5204fc4a82a9d5c3820f20982d08e9af49de22c89c45e82ef -size 32569 +oid sha256:a2d4e25a0f38331201ac56cd331621a0e63134bf6670ba5ea95160aa18452279 +size 32721 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 7dbb048eb90d..65261421bbeb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:25658f784a366cf32d10a024a633f4810bae4616856de6c8449c985c69b72e77 -size 32430 +oid sha256:948fef0c9f244c1f330a60200650fa4b02e186cf1385d801cf01e28c229db69f +size 30959 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index 8ee10385d672..8262f6796942 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3b4b76bd6112df56fdfa36d1beadd7ab6c38a79a31ae7cb8491aaf62e4271979 -size 34770 +oid sha256:eba8e795f77f0b648cad639293f253317dd32655857117e9ca87b289c5fc840e +size 34733 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index c2cbeffcb238..ad523a45ae87 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3b4359324f2ec0da82512b7013f98927eac792fdcf3b7e64157c6f020b33df83 -size 35140 +oid sha256:f17feaada9fa18cb6f7f78cd920ef0129c40f3b3341327df9228f3cc40e627db +size 33534 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 6406e065046a..6e2744159046 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08ec25c37d1ceb2f92cda43d17f214a69287c269b0c4b90bb9f083b1c46d3f2d -size 31848 +oid sha256:b86be8a3b57a11054e66b7073fd6b8152dd8b7479552246b0731f6de727639cb +size 31981 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cfda57c46837..e78aa6a44d4a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:139eeb286ddf65379382bbcbf80c16befc19c5463c1f0f7b1c808a645dcfad68 -size 34372 +oid sha256:ef92fa3072155e4994e49260a1f2ec1110c48d1ce767424e83491a567ac23c3a +size 34226 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 6d8555073ba9..970485e1cc0f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff65ff6cc9730b108bae45e1a4873b57b0488292a31116f2c975915d5cb797dd -size 29572 +oid sha256:f5c045f67fc3bdf7c4ef296eb608ad18980f693d27f0e28b4d5e180e5c88b14a +size 29548 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 5162ed76b0b2..45d201e0e049 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a9d2b7c9f11b4243d1a7e6abbc77f6bed672d73702c7ff9fda8b010725e4a499 -size 31016 +oid sha256:e205772c240cd8d4b93bbf0708382294f72f11c308d139234e8cc38147383073 +size 29330 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ecf759d915f8..71edef9b1ab8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:efdd08066b1b79a0ea77954aac338c8ba28d2481f6c5ce0a09c0db4c1161a9f4 -size 22387 +oid sha256:c4e1635c7d2b92d4349989c27d5e7ce599f6dabac678a7ea42e1c1ed3b371188 +size 22363 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index bd646efc267f..4d5b032588d1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e3c2dbe24832c6b1ee802f4e03413a2599bef91fd9dbd60b900e6274b40bfcb -size 19557 +oid sha256:965ad69918ff2c9365c0734a19b10b8009a0eea8cc0c903e35b43aac6d04e943 +size 19544 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 044e5544958f..6f8c65404765 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:64e3ebea56f91786b40fc4de760a9cc7365ef84237fe3bba71d97ed753da65ca -size 23382 +oid sha256:674c30e9b8f877a455c307381dd0379c523ff1dc4e5502053e08c31943f13605 +size 23367 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4dfb2b8435f9..a794001fbf46 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af1d4b333722dd27b7b65c675ca028c906de6c6fdf324b55dbb4a8b0c163c220 -size 20621 +oid sha256:15f0fccfe9df16db4ed03e4f813e6b81053a745ea4d1b66242ae5a51ad382163 +size 20559 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4e3ea2daf715..04dd77e3876e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be35e50f8a1c3a0293450de0f5a6b5f26383647bc8c3d672e9267d21731dc431 -size 22269 +oid sha256:4a9b0e21bde8bc6ca5e5cee591bdf6028f655700b9452193684aade668f16e23 +size 22270 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index d9571282141a..0bbfd73ada7f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ffb5c523b1cd9d37604c0fb92e3bf9fc8437af7436adf8593d0ae87ac7b3271a -size 19303 +oid sha256:248ce02e539b209f14bfeb669350d103276c99bc53bc07bb21620903d43c23de +size 19264 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4e3c98ed4b35..60301d7b1b1a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9dec67c9a4d10dbe7172728e717d4072e129cda51357582375377c40d7d22ebe -size 23445 +oid sha256:67dcf706f996f35b9a24cdeb787684391e8d7242991645f58419f30e6f1660ee +size 23400 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 128b068a861f..8e6ef7837122 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a571559459e9844716613371f851d4215711c4b925ddb38538e4830e85e211af -size 20161 +oid sha256:8223be293a79383d692b90f7cbc9db380e06bc902c2e729c36fccf1216e71662 +size 20150 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index f61c9868974b..ab0d6ef0b582 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9366620b9a1dbae76e178d72ee096b8cfc78b9578446ebd7a20014f8c87bee2f -size 24495 +oid sha256:09b117152408679954a5529df9f9d5f9afd2b8101eb4116bceaf2f1085c9b09a +size 24502 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 6ad69fdfb099..162fd5bb5742 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a9c48048ab62510fee21f6fee5faa282a7f5ee8266b73dcda1617833132093f -size 21454 +oid sha256:5e356b6ed7659247dd4f228718bfc850a95d9dfaafc3b5415a74f89a527fe17c +size 21430 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 67c55e353922..63403c58bf0c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6661b836b463d0376df78e8eb1728634d8ee9eeadd63cf20629d036d1b1a8f73 -size 25297 +oid sha256:556f4bf735ff0e55b5e926dbcd773566c386d1601858cbb918e8703fbed89873 +size 25268 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 6bb18fc22577..aca0d2a4e954 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2d5a936540ad59e45f7c8ab2e02a00cedbd2661bd67c0e5ac26202e6d2e3a720 -size 22721 +oid sha256:c774beefd081d0a823a23a0d2bf763ce35491cc264da8607b11c046e1710c323 +size 22679 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 53f036b211db..c4ae7c9127e5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10525a861c642c936601859d77d6cb13e9225b9df6d43139369b611ea19a7491 -size 28273 +oid sha256:46791fc57b914fff47502015404e82a40048bd0291dddb230957b06196e90fc8 +size 28123 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 635bdf2c4db8..02bd24029fe1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f8ba45a03da7f50cde34ff15bfc086fb6c83bdd501c2458215027450f399d67 -size 26908 +oid sha256:fbfd81edb26c539b58147cc1f65a3b09851ad0cdd06ddd7d3139587fa2e93bfb +size 26788 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 75cabce871ee..ecd153284397 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:132eeb16cbd2b1c6ab7e1bfc36ba83a286b748f90911cff428355b785b994970 -size 30155 +oid sha256:f593a45c0e5bc2955e51669c4d68ec9f6cc03dc8d3da3dbf43c350b75b684de5 +size 29974 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 0d858d87b69c..f0f162d5ef4e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:607b99a5e75720660622bf1452650270ba5461d32100f234ba0da52c7be599b6 -size 27786 +oid sha256:6eecabc60f488c47fcceeda7a60f2777d3ae98bb049ab8b8cb77148918dccda6 +size 27640 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c670d1a0933a..ceb47434b342 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ef085a1058de172e31d1ba1258b1f0ab19aa0f99267968a6d9a032fa8cbb9e2 -size 27351 +oid sha256:83831f2f770f0a9146eb81e2c863b6f3c9d8bca92425d11b417b717eec630fcd +size 27255 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index d3dd1f6462be..c01dc1ecedd1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:65c88233d58a868257966cd95ee8bb1fc1bfa4bb2374df9a4a9f046bb22444c6 -size 25525 +oid sha256:2a3cfd199369c0fe75acd0e67a13986161e7e650c34bddb3434301b71a811025 +size 25420 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a930c6e77f4c..1d2006f9ace2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:399703c940f029a547106c7518ff48cddf3c936664a2a0e184489d619595ee37 -size 29239 +oid sha256:b9d8e0ba8dd7750e4dcfba03709ec2504be699eb25eddfea317d6b82022dec2d +size 29107 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3f4a6fd0b0b1..9500953db2a9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4bef7bcb81887981fc5509367c0a66e0beb6914c576db8b5f3fe15fead2266ee -size 26264 +oid sha256:a3ccf86bf658f24c80034b5c789290243d74b1db93d88f24fd10ec4837d4dee4 +size 26242 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 75bdff126bd6..e66dbb9be5c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac253c2c9f71c85f00948cd291b5d17e919d77a955730ecee04f3f0f1f9310a3 -size 25735 +oid sha256:37374eff4ca12ca5d7c1c81ca3a0a5bac1b089646b3bb18ab2a839bce39ea95e +size 25685 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 93df1c3d1c5e..57a70c87e345 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bdcb61e2b4a0241b1343a7d97c896ff4ecac8c82199d0d4dbf1bd6fd3add8b5f -size 26028 +oid sha256:c8ae4b2fcf3f2ac1086472edad479d13e4342402dad738c76587e203559821f9 +size 25985 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d0cb4b0617fb..f9b68685115e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ecc554dc9972f84dafe0de691286ce13b9c69096ede9659fbe8e1f7d65c52f69 -size 23225 +oid sha256:4d1542c4897d585cd89c9319fcba47c840697f025690900c60f1bd8465119db0 +size 23203 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 12b4b1ebdd49..3968b0fd16f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b1ec548750303e96dcd1f62fccfe781f620a89edab153d8240916ac8e5ed270 -size 22773 +oid sha256:9a72e196edb5d7513b0090e8d4955ce393c89378702232513206e8e5fc926eb1 +size 22675 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index bbeb28fe47b2..53c887f983da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ec0130054f011e5bac10839d4993627e2ba62aff8224c62b21d4fcff115d22e -size 22440 +oid sha256:a7659ef247526a8c8c0febb6d7aa3d873848cd4402cfd8ffdf936c5f7b195d61 +size 22381 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index d12548a8db59..d40aa0a1579e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c6d8ba16fba668108b232d46230453deaff94b04e3f79387053b306388c9ed7b -size 19877 +oid sha256:bc562358f06c8e0dbbca464b1faa9e9f8785017b50777f1677b6cdf8a7746d70 +size 19852 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 066a367273e2..208f6b3fd393 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7823aa3566c90e72a866fde141cf5825243d5f46eeba3286fc0e99c0b2c2b8ac -size 27011 +oid sha256:00d9a04c4b3ba4c699b94a09b3c486e16be027c06c5cf4d2154c8c9cfeafebbf +size 26979 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 02894a2a6821..87c617edd0e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a415ab9e66e68e3ae8ac9dedc25dad069e6dc3b6970827665ca86b2fde2a8074 -size 27354 +oid sha256:41d50fd1d895eaa53f32eea0da22a1fc7d08d514530f9a5301c80b689b104620 +size 27769 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a5db72afb29e..830cc4451d10 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c71563b314f72492f25f22c5560ebd0e3459d885c658810da8741bddffffab79 -size 24244 +oid sha256:071b4570e254dbe13ac30e7d1ffd8fb9b92e7ecbe097c9ea7376fb9772221563 +size 24153 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2544b24f00cb..8cd3e2ae7a87 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08d9958d90e0fb9acefee04ab499cf032450911e4c4e93e198c2d6ee31290656 -size 24710 +oid sha256:0792c911b8bb004ee09aab5a4d160477a43026192a8ead43598730fac075dbf0 +size 24645 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index c76722565592..4daf0859e5aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5189cefd47a6c800cf1be7baab1335023e45ef14ffdc0886e1e9ea7617892679 -size 23509 +oid sha256:206033d591b888ec7c804c459ca54b4b33258f73f2d56df9be75d7560c4c4e9e +size 23381 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1f45b00573a9..3b2c610688fc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7804a54a767f717c5a380ae390fa45808335e52dc238169b6355594b1c8d9a8a -size 20891 +oid sha256:db3f44d7f735e83f7589a0b553934a1b2ef3d39a845f1b15537de3ea3aa6df45 +size 20904 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 727f579e4972..156e63ac0409 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:692812ff082cbd29ce4ee5989c1c12df24ca4cfa13acf7e5262f2854275601fd -size 27946 +oid sha256:a316de5a5704469e5de52cdbb65b8e9aa8c2289232d9bf9995a0fd3c5d6295d6 +size 27799 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f7549d4d8445..4ce75aeb2037 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8c9e31d26331f85635d75a8bb6b400c4d3ed49a8a7c6218b5b0e771f0bd474d -size 29542 +oid sha256:3d690d8aa0a6933d30742622f00dc0000a397f4cc6beacdcb003b458dc381031 +size 29437 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 05d45460345e..9dc5953d068c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:441f3a843007c800e9bcc8c06437e4a8a77fc4d7b9ee5ac955d63e04a524db15 -size 27201 +oid sha256:65e5dc96769222f67e1cb1cd96ae0c818e75beffd3256bad089fa98f4db59bf2 +size 27157 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 536f51e65278..9512eeeb6aaf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec48ff84aa6935376abca0fdb2f5f34dcc58eadde5e2378ec543f2486803b7d7 -size 28798 +oid sha256:4fa9b3f22dd92ea9fde50620b62595b73ffbe22cd6c9bd03a04a328eaf2162d4 +size 28739 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 07f58baf875e..3b29780efc14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:507b9f31abf7d5a18303932be39844786743d22bc56cdec25c9ff3e06a170450 -size 25879 +oid sha256:1e148cc0b61483a6163adfe6bee11ff246e73ab796c571c21b2323c5d7d90a01 +size 25810 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 61926dc3d3a7..4f9f47a4e700 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9990d99f6894b14fff3043def181c83b36e7733056d1a3ba7098cbbb62ad832 -size 22297 +oid sha256:9046e47032547d8819952f5a513a5b25c55136a5c2d3a6f76bec94c9cde04b25 +size 22248 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 3d2325134d9d..b5561e0a203f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:041cfc64ac587b63ff8cc473a30f1d3af839e9bb001eb1cf9f2fd153a6466ea1 -size 27176 +oid sha256:5cbb479c692dcda4304448ae9ffa68defb990c738d97f864d9420117578a038e +size 27077 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fa1c11c65855..8dee99347c03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4dbed7cbae9f5cee40a67a4075f00bc0eb85a466c60964200b4d1ffadbe6b316 -size 23836 +oid sha256:f11f4dd282cbdf016f654b83dfc4e68160fcb00a76a12a588528a5514414de70 +size 23802 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 49a59538efc2..c4b4d2a2a101 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a90bd686fd294a90a513165c1f557288978d8108670ac38181ae86db52c124db -size 25656 +oid sha256:f146337ea81136432cad2dbef7cffc6952b552a448f0b006a116e9bd5cdacd6f +size 25543 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 202c344c39cb..cc2fa00f0e04 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1bb9e13eee3ea64024a15140f6841ebcdc171f575d7220defadf79d3010bd492 -size 23050 +oid sha256:8116cfef1472c92e233154fba7cc654d137e1af0c2ce1786f3d7e321c227f82e +size 23032 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index abf9940214f2..d86816808f87 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c6dd42d149e892b9d27cb6c2e26285c4d9f7f63a22141903e2abbae90ad7ec4e -size 26897 +oid sha256:4a9bf79020bd68997bfbccd8a3d51a999732fff60613a6744bb2b4db7424cf91 +size 26845 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index fbd9d836c886..70e4a1602208 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa19e0df687a669524cf3fc67bcb7f7cc3c66b16a648eda773645b23d071ec74 -size 24001 +oid sha256:93ec25326380d5eea5ce118da6c7965529c1ff0d07447eb2e66280b17d9fb7a2 +size 23967 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1a92d6a7434f..1ccea77f8704 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd6519314ecbcf8d26f3d7f065a2f9ef7e8ab42f0959f92c3e825c2cb73a6afd -size 30932 +oid sha256:c1da755cbb8e7e3462998cd5bce5d1f46e272ca6b33c7bc39b8a7b14728c1d9d +size 30799 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 486b066a7c66..45e53d8cbe10 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4fa5b67e34fcec0842cadeb83379872f01a562f69a3d65c4b46c4d68375f22c3 -size 27631 +oid sha256:b23a503bdb8421e35e20facaf90953b95a28a6db5e1182aa0aaa8fd76b42953c +size 27611 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fa4e49d63e21..13d2f7bc21ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd58c2eb3de5c7dc284fd3cff252decb6920893e4932aafe784a93e52ceb4207 -size 32307 +oid sha256:bac7964436ed023af912b17cef9387e9e0f262abc354304768de7241939fdc3f +size 32175 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index af800fa6fee5..261498e24669 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33c61383918b39f4115ad5d72646b71b83836e11566110991374d2020b4bd5d3 -size 28482 +oid sha256:1a60e067e4f036803d53a89ec365ad0c8eebc579a78a740b431c4549ccb9eb7b +size 28425 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4956bc3eb80d..844927518253 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:171839d99088e62aa958cc1283c1030ee70081b68758514f8ac6b08b2d3f2bca -size 30161 +oid sha256:0d60801d1754380e21fe970f89b189627da47c9aa835b2eff8c20a7ded7192f5 +size 30108 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index adbc1e60cdec..030d55b2563f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13e43afc55aae15a76c7ec433f76090958e9f00e5e821a9b9515cd9e98ccce84 -size 26413 +oid sha256:e9ad8997b89ab29f06e3ba1aed8e9d4b06baa497a3d5f6708ee835f2ff444a6c +size 26391 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 32ee408f98c0..ddc63f8ea711 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a59a11e72d0946260440d8c689290ed82f7ffa1d56c922913e0f3b305a149ee1 -size 31537 +oid sha256:8f6b3142c258bcfdcf1353944d0e0d23da3f801f02c72c457d5c7351213c935a +size 31290 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cbdc5e93bfc0..b0102a80e8dc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb15c67cf6cae2f618ac1b574c3c45c21f5d4f31d6a61cdcebb19b5b2d6bf972 -size 27266 +oid sha256:58646cb69a5d5de10b00e2414ce65e1f2cdf712a35a5a698abe9d2c8d7ac9b8f +size 27125 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ed7a33973ac9..b9fbfd691da4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b5c4ecda67459e03c32840b4b3742f2864bafcb21531315c2f001687ab6dafc3 -size 27871 +oid sha256:2c604498bd4f34d084a8c9bb71080ac7c69dced5fa5e597ddbf75d8870e44025 +size 27865 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2214b5b58302..56e8a1bde340 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f3d16953db8b16333b26772c4189223295bc886ddd8d8135bccc4a06c56f9352 -size 27311 +oid sha256:28f7f13b8849aceeb23adf4cb2ad6ed367ba6dbaefd9e88f0fc0a62636344100 +size 27263 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index a316f5837ce1..16cde9e356f0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:069bf5f5e84404233ac502608563f2725982d8d0a0a65082facfb2922446f6a1 -size 25105 +oid sha256:7c5f27726d25a8c174aef525994ab57eafafaa0ab1fce758394e3cd200196ad2 +size 25076 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c6896d3bd90a..0873dc74c301 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c3badaf1d4f41ccaf07a0c6ebd61954f5679e1fa99b46c73bee863b1717899ba -size 25170 +oid sha256:d46d82b85079fb1569b068aeb335b1b8c7091cb9978bdc8c782ffad352b6579b +size 25046 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1ec88e8e4666..463b4840fa95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6903981f3040ad7fed2f783e143525f875856e9396c13f794ed6742e55885da8 -size 23843 +oid sha256:4c0d19b6b26e5c5b5e3af2e647c5fd4cb7bdbb9321942d6a81be50f84d660c99 +size 23767 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ba10f3848354..bacb830f4629 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f95f23c1b76d464008808ce7995d0a975f026ef67f9824e7d5250bdcc0a0bc7c -size 21054 +oid sha256:1c08f68fe97fa8682fc3dd941fb991b693c5391529807a42dce7807c9d73348b +size 21097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1df50228b999..3f2bee133978 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7720afc776b8afd619b7add2304bd24fdc53bba9e34d2d60476cd49ffd351af0 -size 28785 +oid sha256:e64fc96781edfd9664d5aa9e91c17d43f02f7bfee3605504562aaa66b0984641 +size 28770 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index fc25ce443d3d..e7b38ee0dfb3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28d9c0f55890f9ffdf727dc6f6e2758a136b50274e4b9eed447bb4910c215faf -size 28541 +oid sha256:2a706fa0b2f42da263e474c175184e6f0384d7f006afd0c39c01bd1ea50a1ca7 +size 28477 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 3fa0bdc180c9..88a7ff93ff8a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:63bcb68d26db2dd20b8852c63696794af59ef5d6f0c7af4309f652cdb8ee18dd -size 26429 +oid sha256:a8aeeec71eb9e1f8d4643a36f2532d9617685cdf9dcfd2848951e9a6096a23e6 +size 26350 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1cf894095394..84814d5098aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a2e8869f7592709ac171ee511b7d548a7d5593d09283cc28550be6b31302a0e1 -size 26244 +oid sha256:1166dd994fa6c95b2953e6b3b7387c667cbfdc164f7f164d9675ef583aaf04f5 +size 26250 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 220af838c23d..2a18a09b456c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ae2a5466c93e88d16ba6151e8950db8b807bf4522066598469d6ef92302bbb2 -size 24841 +oid sha256:b96089d07add0de07d8a322cc6ed3d0adbef5cdfa81d3fb4bf29df24b9a9041e +size 24701 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 0fdc69c146b1..8c0a4737efca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:66d1e519e042d90af60ce4e2c3bda5f65b439b15308840a3d92fc0e9fe7ff4b7 -size 22444 +oid sha256:31d6b053058cd16d1b89744455cbf204d6b8abd525db27aa8956c45b2cd9ebb6 +size 22511 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 8d83cfb822a2..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:fbb79ffd294370f22ee774252c3930b74d0e0ffb2b32627989c35d9e920b789c -size 34800 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index af81ddb1867a..f1a3cc135bef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a244aa62518a447611b7f1937a15c59de9ec90c5d2ba843b33b2c34924e0647b -size 23143 +oid sha256:10bc7e9a349c9764b3f6bd03c3b13256965d8f11ab547f259aa156fa6e9035e1 +size 23012 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c2e5f3f39d7e..f08e6a3b3b77 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e16ede12ba4e19fea38884a430008cb1139bc061bbb47e3cd50dbfa308bf5887 -size 19998 +oid sha256:5dd80425a49f48792e06dee658936ab0ebd7234caff45c52312a34a65da72939 +size 19989 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index eec41be4f3c0..2ea4ffa1e08f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8e121d67c7b2f8159a10115a9bd4e2e48f3611c94027948893796b6bc2e99060 -size 23951 +oid sha256:a6d9f615cfb93cf9a5d19868d4a149956e50a716e5810f1c0fe6fcf6b8f74d20 +size 23942 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 16a0b3df0964..845b1e66e374 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8b7d7060df98d64d8d4ea63b6bdaebda9da2927f7dfff79bd328d32423db7f9 -size 21145 +oid sha256:8795a3b0afe3b2de58827d95f15401f9c169b3561fac0c3059c0490e5b908e9f +size 21129 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index a0c60ba68d5f..a26f7cc3f804 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b05434220d9bf136621bb0539910d39aabd51f854678b7ff079e3b81f993c45 -size 23051 +oid sha256:b051e51f92558bbcad938e9daa03997cd9b44f23f55b16bbd3538d775f25d301 +size 23000 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index e506ea2d5d29..525aa6baf9e4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1f476f1b8bd48111a348325b143ee3e4102620c09e80f10e990fa31d14863ff6 -size 19891 +oid sha256:0f7f602b4c30645aa636351c34e376eba133d2d0b5f7a31abf20d95390e81661 +size 19887 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f3b69767ca4c..25623e0dde18 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee1791aa3aa613a01557a17d1563eff71bd277609939efeb7e5f6b09558b5eed -size 23979 +oid sha256:f6527aef5ddb6ebd2724a9340e0285632c0da95d4c1fe70ee42a39e641d69b79 +size 23933 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 054e83a74870..d20c6977755e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54b8d6eddbd9b5845c6ca5f9f88a9f3a1c279c248126ca1d0d2973a1dd2658c1 -size 20800 +oid sha256:f0334c4fcdb3b7a5590ff1fecd9beaf473f86e020b1ab6cb2864a7ac4a0192b8 +size 20809 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 860856c55a8d..c3f18011d9a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cec8b1b06f78762b13efee8e8e002361a9e806d19d604b4702d22e35b89a05e4 -size 25367 +oid sha256:490cbb73ec9c9d21494038204de4c627676c89cbff581dddaea4bd9ab4113eb2 +size 25239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 51e80354f066..9e9313e423bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8986b74344d4673b36bfa90c0b5a80862cec1ca381c60b5a6438eac96d8c574 -size 21968 +oid sha256:c282a9b1ffb08215e28b3a10ca18e483087553cacc40f8410ad39d93f33d6f1b +size 21943 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 8aed3ac174c5..9e7decfc0171 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b0e8579896e8742d69227a44235aac73aec8b70e951a55109a9eff8f20916d3c -size 26090 +oid sha256:7a52da5d153402d317797726f254886c5c84eb9b2ac4a42b8590936f8fde0b8d +size 26104 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 51eb1346ad34..5bc299dd2763 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e07bae91bd5a6e6e8d9a09ee8cdb52c05ab8beca59977e40d8b3f8ed47948104 -size 23113 +oid sha256:93d500e9613c27c7605e798702f030a598a7dae42a90c390280c028d7b5eba3e +size 23068 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e18cf5ac5955..6dbaf39ff272 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ee6fcdd643ffe21b1a7ebb55216577abb70e1b03998ce6b786e14731ab48b74 -size 26614 +oid sha256:416a8b931d1933fa2152f7ff3cd8bd844f8d905d408490ab7893aa77a7e1ee92 +size 26558 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 7a9629f43bbe..3bb352e5429d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2491176bd2d04402cdf1eea78aef2c8be3cd851fc65da7bb1dd409595683d9ab -size 23976 +oid sha256:12dc827231508127e7e184429f8753f619896267b6fff4b320703742e0f73b69 +size 23953 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e38d3468f3d3..2d23ffd9f7f7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:976c702bc78680687aa5a490c7acf374961dcbdd5f33b66d9eebc7466d7b4b1c -size 27581 +oid sha256:662281367947ea0bd0e95aa5428b1ae4c8cd2f700583a12a089fbaa513ee7955 +size 27561 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7da7592fa699..66e9bf0d2695 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1c09cf55fff69745e2e4bdb8cfdc154a3fa3683929fba1a9dbff77ea949ed491 -size 25093 +oid sha256:b99faae246e267cdd669e357e65445513d6508780b9e38c3441532c041e33e8f +size 25025 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 395e285b59c4..1cf1e19a1ad6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8ece5e6570a4680607ee075b6b7a14f4c4e5231f4b5bb82e8abd4642f3016ad -size 26384 +oid sha256:400b682549736b541f929993a80268623e508889e9b42b653dfca058c6160f57 +size 26364 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 4e8fca1aeefa..f9505707be99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:44746391a111c6b9d60d797cd3347155f767712f605bce99061536335fc65051 -size 23783 +oid sha256:78572202a5b3888b383a55eba4855f453d3148f8ccdc4c60fe6c5bb7c853a6c9 +size 23768 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 106c4cd654c0..83755d7ff26f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:857c5a50d69b097eb06fbc147e3178ab27d24ed1860ee52de9d80f34a635313c -size 27554 +oid sha256:df7589f0ab800820dc09d868b24dd1eab27d2612e57528e4b8570c9569166744 +size 27491 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b371373c39dd..c3a668e26288 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:31433153e0fd51da4be01a6df5c1c5304a198016cfe66a0daab197fb5f217ece -size 24799 +oid sha256:8925a11976072a64334386462de7934ea50c868e97c4426f0bd463c25884c8b1 +size 24751 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index d74c308070cb..3bda2f59457d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:addbed645cc5c3661ec106afeca896133a292f72c23dc51a86282b19c176163b -size 28548 +oid sha256:f70e5d8dd8e25c2d436ee37773e103c61bc0af38e8ddef45f5d529e014a4e49e +size 28450 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 54a00043b584..ff243a9b2a61 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98cfc8a4f8baf3bed046ad618e1b26b5b8f8f78b8d4fcb034ecf3f44c5d1b10b -size 25939 +oid sha256:c7f74450a3c8acdebc8f65941c0fbfeeeff4e46eccc03577115a1a6757408bcd +size 25904 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3c67ab362c46..fc8d4192ae4c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e06f4cc762386d225825cbcd2bc6bc0cee4c56953bdcd688b964efd04eaed536 -size 29667 +oid sha256:c6249824bea646ebb2fc759ad7b97a270eb05723a9513100ee5af8b04e13c699 +size 29700 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 6a0a854ddb7b..4a5c25cdaf49 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84002a5f7d2a44fe1a8c1da608bf9d695b341f79cea919d0cfed0162c78a5dce -size 27340 +oid sha256:113d8293905dfaa2b2748e6da872abbe26b280300c8057e12ac894d0c6b8197f +size 27293 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 54159697ddee..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4e0bb72e7c073a29c9479d88ff667603e26231aa499a47073ddb5442a79a2545 -size 24329 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 63ed97b1d8cd..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c95956b62f181d88631c0489283cec6752a3a2d7eccdc23f131fc262e4a905af -size 21057 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index c2d60f3108e8..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8111319d3ad801654bb3d58fad67100cd0667fb67412f5bea70a24641830f31c -size 24570 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 6f221bf868d0..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3ed23ef35df2ec456c221938b9bdbab385858496c2b90418f4e74731e86637b8 -size 22201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 7ff053513011..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:67ae5fad87d8591bdad57ef300d8b83c2f501374c1a80a50622429b387f63f0a -size 24106 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index cc10897e9afe..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c53029e8e6094c09cdc07ce4119e9a8040785bffd65a04bc192b0087b7168c22 -size 20694 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 51370538a731..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ded79fbdd2080c504ed8d3fca38c6d366322a3563bfb9eaefeab941d6ecfcc29 -size 24984 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 73ae40b37d60..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4e1328bdfc1462a938e17bfb305ee461f238d60d0a8fb72a584a0fde6a43d972 -size 21981 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4209330fe0e8..34aa21f11484 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d86f58f7c464f6fe882fd48c1b88bb5e0362ee8cbafc55c94268ae1cd9aa8598 -size 24705 +oid sha256:fa0f69dd7d4629e29f26e17d0cc7ec9fca3ac3019f69f2757543878914262d47 +size 24569 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1e3124b23e6c..ed2119c0830e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd3257750e6a7014c16ff25d0b2a8c27dc9a3c25ee8aef153ea08d551c6a27f3 -size 21339 +oid sha256:482108ff286cfba9e73b62423da152f45abf9cf60fdc61cec56f1026c1a017df +size 21190 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 575e2b109c75..4f530213aceb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:69432b5cf78cd265e8b7f036988dfaf5bbf119db5d35b033c24402c088f07677 -size 25021 +oid sha256:6ded6e3a0617a2a5a0d0dd31130c91d34388d208242c5b742693209a08f17886 +size 24897 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 90c34c7c10b3..3192c0f7887b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:77d14fccc312b429ce955e583b7e0e8501444105e14606c55e0cfbd72f7801f3 -size 21707 +oid sha256:525a599a48faf083e9c6df74fa9c9981021e3932988713ac05ec5c23901b7706 +size 21624 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ea3de8c43b8a..711f470cc86b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:724eebb55e109904b12476d9b9f8f6731ac719f7423a4e2f83d6db3a99478e1f -size 24396 +oid sha256:05671e56932fb47586ba8d155e1aaca8c4a91d49b5fd8bbc65682e9139c2192d +size 24239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index fef16250aa40..b7837e2d6e5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e6799f565db2a311e6b8afa7adc3f87bf9f37f28634ec2c2ee82c4621658c64 -size 21306 +oid sha256:eaeae781520b02ba67fcff0fbd646d1aaea1a29bfd7ea51d3a6113ac07c0689b +size 21124 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0a9167efe743..dca47197f5fe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa3bdcc120ee1614f7c5763a11744609cc528978a243330389d0fcd6f920375f -size 25067 +oid sha256:3a79c6f14348fe3de5e66e770c733382e3d19d3600ddf4f7e44ee8592065c214 +size 24942 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 132f41158c6b..f3c49dec9119 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:134b248da25f45b64e205a5bdf434e2e1cc2caa11a7659b3e88f9607b05bbd69 -size 21761 +oid sha256:4d264ef77016dc1162b0ddc6eaeb62f97dfc860ace1ca6b3e928becd29e078fa +size 21636 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index bf2007c765eb..b060e417980e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6144a4fe3c222a22ab607bb7fa41e8a6f06f7db03d833cb7bfcb345ee6942113 -size 29618 +oid sha256:20ecc0575fd47d4d61252ac11c6979844a99380756bf3cb9e9a93472c64de70c +size 29500 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 6cd09e3d7ca1..d38dd212b22f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c0143f15e4f868f02e98555a3c5973a662588fa14e8be931db06bdd2c201cf75 -size 26741 +oid sha256:6147ffa156e6c8484fea099509ba5120204be3e2daa2a5c1b9945818af455a6d +size 26730 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 770ca593d06b..77aff2323357 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2316c8945f0d412ebc8d8861dddf6b70d6ab8a6e3c0beaaf4738db55348af17c -size 30577 +oid sha256:b495e9c36f86a6f97ff4d5fbf1fdf28034c16ddf29fdc7b28fb4d424c2558aea +size 30458 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5115b9fc2697..7ea7a410b36d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9110b26a8b265e49d62c3c757f3ffbd7406b0b04a75b32d14f001a9d6281f03 -size 27836 +oid sha256:fa70e63cf4685dc49ac67ce96fc6685b91e49063a9d84a8a732eebbfe2ead455 +size 27771 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ac9b67ed6616..e446f221d273 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:222af48a5f5b05bb8a1e04181bb72fb642ec5b8f0d67d77f02a4ecddbb3175f3 -size 27593 +oid sha256:3c22c7877385a5cb1b34bfb651d7788a9186c7d4e5466e2f7ebe88cc508a002d +size 27524 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ea78c71ea489..e1eefa561ee0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3a42adead53dc678dbcf222680d3508cce536667d7dcc875a7d69958bcf9a2bd -size 25957 +oid sha256:bf69bbc16e28834271985193d46bcdb7d4efdbd0091ed6d3f3e95e23716237c0 +size 25834 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6cde8f0fc206..4f9330959a13 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd02703483e5f85e3e0811d4d77632404d7b65d8edcabb896f265872080c36d0 -size 28494 +oid sha256:6179e04d8cd2a99460e9b6bb0714ce50b500b55a3500db0721c7ab67d9c7da37 +size 28411 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a2d9384ee985..2fd0387abd07 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e550a2b747d89c2e115cbe9747639eb09fa9452a76064c3b91900051853622c -size 26937 +oid sha256:d66c9cc4717c5f87a0d69cb43758e107804f19b6b7b0be0078b3972e3abcd6d5 +size 26778 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8e08f6a1a538..a87a854404e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ae90b6a554e68ed07add1a6e0aaf1373ce221a940166de6d39f21231bbf4257 -size 25148 +oid sha256:e4da6589d86f2e5c63c1df310425c7c68398e54e7b2eccacfb24460d8a85994c +size 24952 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d0ace8b15394..d304c0df1abd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac2bfdd08a7f49ad418d6b493960f8e8bd39e4ca914a9464d8a49cd9e25b3692 -size 23045 +oid sha256:5c55bb5da202f9c22f3d43dac4c0d417883e8e3f9d542e95217de77de2d974cf +size 22964 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index c24a426423f4..a90b4084b63d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a9b7effd33bfa6bd26f8a0fe6c6be505e6a91cee358937973593f7b1b8f2415 -size 24016 +oid sha256:f95c1ef744ff734cc48f16b2c19ae3e5abcfd0dba9950935171c2fe70204e418 +size 23965 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index fdaeb810c309..1c80f0a20958 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb7a3ba442e8843feeabe3fec14f94c85bdc1b49ed376ff692bd22b2c3193453 -size 21361 +oid sha256:5a25270a35c22dcb109188560c0d8c212ab9416359551a481fa9b2c37d133f0a +size 21265 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index dc08d275b5be..701191d11324 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35ffa6244180da1271508c5e4eac7668218eff0916853e2499d67ebbd4937439 -size 25881 +oid sha256:9e6fc8ac2abfd24eeb8eb5523acf6274347e5a2a2a92dadf1c4a487a7607c722 +size 25674 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e7c66d159ecb..c7a8566e5da6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4b9d58a01a39d19bd127ea7c0fd776c62b171e49b2f74900b63706789fd9d1a1 -size 24167 +oid sha256:22e50117a35d3a1052c85f94d14d46ccdd21dace9c36e90cf3f54dfcd8410ad9 +size 24061 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index cb0cedc54d61..91c0a27f2e5e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac22fdaf414689a9c5238d949d6fe4716da63dc7286ebff38da41813a9c0d331 -size 24662 +oid sha256:ab9b73a585d1aa6e2a023f09070898a979767b1fb8e0adcbf74b8979bd326445 +size 24643 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 0bb3b088940c..63879546cefb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:156779a78ec56bd9cb8b03a34cf49f5d85ffc47793523d86cc59b196ced10f15 -size 22376 +oid sha256:51f31d6be619e6ac50d50bc8227118ba89a90666d67d78d2fa36758f531b2c05 +size 22342 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0becd07ba6f3..8e5b94a39e07 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:86164eb89ffacd9d3ddf2e317e7d073b70cf3edef7cfa06942188f57d765fcb6 -size 31691 +oid sha256:d2f80c539a973f535772a842180e2b872aa2d79b2ad9cbbcf1ab75d4158bc3fa +size 31462 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 10637ed867aa..0bc13dd0a0bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8011b536cc0b28511be64595a3eeec67dbb628570ba5b649ed38c5f67fad8979 -size 28593 +oid sha256:7eb85adb3357a6edefa20313d1407322deb372f630bba7ff53c6b058fe46cb0e +size 28580 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c3dc72abdea1..88e597e607b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee21a89bc4e6a0e13d7e5256959686fa9d40fa778a1be2685c7e9f56e467a08d -size 32853 +oid sha256:ed97722e234850a837640831550dc414b6ff15146b283ed2b50ecee7246787e2 +size 32670 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 69b1619c756d..2f1256bc7c28 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0f60e38603fc68283ba3dd1eea73647410ef06259b5c13de1b20879b14121bf7 -size 29572 +oid sha256:c6f16a92a369d9d2621802e91526361be2846d72877129668911ae5da4b5824f +size 29555 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b430d7f383c7..bbd5cad82f5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0acb532d4d2164a9e3eb025d9ec85bc7005e1902812c4d7619c4c0ee423b17d7 -size 29833 +oid sha256:9cdb468fdb6c909bb72047992dd0b9623c852093d58f7231e18195c832d03742 +size 29666 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f043c3bb5a4b..be948843c205 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2924289a33f22732e1d4d0b76c339216f42dee8d1c9a4de6367e6738913e81d2 -size 27778 +oid sha256:b4ce0e8acaab9cbdf7598ac0e78925819f94e736a8692c8801a54ddc58819d31 +size 27606 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 53e597783140..16c7a66d4643 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:32d02d46da52fd636978e08d756ad115a9f4409534ae04f34d671f82216a6b85 -size 30652 +oid sha256:dc33dd894bffa5dd0e3d25af51f4cb82f982c019db946d878bd03a72ccc1e0f5 +size 30539 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b9ece9e78d69..d2bc67cd9268 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dce5aeda9f76e33b40b8110151cd5c80776df48e518ee8d12a0b83d58bc65bee -size 28724 +oid sha256:52b5fab8d387561ddb1d0247931016eb56cfacdbe440446fbc3136aea4883bff +size 28609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 00736d397ab7..141792fc7b1d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7a90c790eabb535a946732e916c7a2e7ee12dd7eb26d9a4e497e582814228ebc -size 27274 +oid sha256:cee188d5ed95cbcd09c326a16cb31bf2b0eb25a54c58e5ff41c45ce1d84086b3 +size 27066 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4d9965521ae6..0ddc569ce7ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98770c661d7f8775a683dd9657b9349edf4599be324f5fe0541d0ac0edd2af75 -size 24989 +oid sha256:e3e14099764150dfa54ec4c5f553bf645e265fa96471747eb3057bcaf9faa1f8 +size 24813 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 7496cfd28bbf..05637fb3cd5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0647a9cd232c110d8e8b763badc6cd837b8eca8bec72e2741b75c1214d54156f -size 25183 +oid sha256:06bff448f6562959ad2c569a744e15357000e03eb1f33b4f91431da16c8c7ffa +size 25221 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8801b1ce813a..024d01ec9b63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f714ce35b25e374f8d71577ba6b868b271e07432080b021198de2455fdc023d2 -size 22881 +oid sha256:0553ebde7f2f80ef872876ee6362f3cbce7bc9f2b9427694d5ad9cfa255afe6f +size 22825 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index a8d7cc0a7d1f..3073169135ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0cb98f9825c9999653947e756e5aa086d38ccd386874fa1a5cf038761166a817 -size 27761 +oid sha256:9d77de1ff81674b2cbe60fb74eed991e891cb7fe4d104f559b99337fde12d45b +size 27493 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ea1c4d0a3adc..4ad9ca2df02b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b39e5990f4b30df66bec093356594f8db7dc61da9af6ef3e05b8ba27faa9907 -size 25636 +oid sha256:f85cb14d76bdfac5ce0e7be9858ff717e58fec2eecde4b4bf91bca469272f979 +size 25447 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 5e4f92130d45..f393e9131cb3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f8de72248910bd1b29b3307ba7958e7d161c191701ee3b1c4a95ca27d3b5089 -size 25883 +oid sha256:8f4cf7706d1ac8a8593fb2da95953c3f8d3319fe36879cd4cd0e3e8bb91322bb +size 25855 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4fafe548e588..58362ada9aa0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5979890716ddae7930613ba5707ff5e7f1ac4ceba9c0255a603271f0e23c65a4 -size 23417 +oid sha256:dabb664d3f283ab1b2a539b71fd87e81597be9e03c2f6b0d97fb3a11f5101551 +size 23351 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ffbc43145edd..2882e2fde8f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a8f05160bf73e85c615257883ecb91a48d18c73b0398e8e213dd08f0ad6d920 -size 26030 +oid sha256:ef973e7583a099e82e45ab0864b93d78d460efe98bd3e06b6ee6e15dbe4afa7f +size 25945 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b4d0f60d36b3..ac1217899f4d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62e5137fbfd7c8427133c31701b4a1390aba27af6e78f69ff1ab087861c5d3bc -size 25839 +oid sha256:e473c00d1491d3eb56d43bce4a863d1f2b1e5ec1f6f04a6fe27b084b6d57df14 +size 25827 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4057b5e8d195..1a4f52522aa7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:709ba0683eb6b37bfbd6252eb6bf67a4871c941a44ee428719f9dc432273b3e0 -size 23569 +oid sha256:695737a5b85d815448017cb17020de22479670f9beff5ab89fafab99e461eb14 +size 23606 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 6e850f78f83c..bcfcc6eb845c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1aebba76f350ff088e8685dda683138086473f2a05daab99f4158314c8a50c2 -size 23984 +oid sha256:92f42ffd5f099e76133483ef0471b670a03e73ccfa2475b2372607f4cf43e7b9 +size 23991 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index d6f7ddf28ba2..14e52a900a81 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b425bbad4f4fca031d75ccda54988e7763b8e56718cd49b5ac12753fabacbb56 -size 24372 +oid sha256:0285305296700303f2218213ad62373cdbdc80fb0c232b011f9397a85806fc25 +size 24395 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8a2a49e67729..7a0cb342f713 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:96147638f2ee4dd22e3010b1d141c2a9ddc7c57bf197e5a74eabbe8682170bf1 -size 23898 +oid sha256:01cb1b70505c6ddf00200ebccbce6305d756aa92fd6c49e3169a3f392eb5e98c +size 23919 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 70e18349d733..602d88956b83 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:213f0b0b4c9c6d3d2041ba6813bcd8df703f37dcff81e147542ea2009562f555 -size 27577 +oid sha256:ebbf493679d14ff313115528c0f20f7ff6e35a01993e136d782e7c38478f799a +size 27501 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index a7a943d64670..cbf80df4ba7b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e326c6839f5bee2212d0f5884af568cd6e3eed192d76b9b1b33ae45227167d1 -size 28626 +oid sha256:fc83f84a7097162e758e287ca4840af82a55751b3a9590f6237b2e082fa6bcb3 +size 28538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7a22b5f74aeb..05d604dc9b4e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ff1355026957f925092022540553da4168b73f81b8b25291aa734a794e9ee13 -size 25672 +oid sha256:4e663a0a1daeac3a3e039b8f9d2336591b3421ce50c17aba643fa8e94a606294 +size 25595 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 178c426e210e..26c1e2898c93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae6f0b82f543ed9793fcbc330c47a2c3ee219b6c7dda19fa236487c342bf508e -size 27408 +oid sha256:6fcd0d70c5503bd06d555891a70fea4ff3ddb6ba2f5373cf92283761f05a7dee +size 27393 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3f6a9ca7fbbb..1ea1b5b87382 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cba25b5c0e441dd7395490eb8bbde5d8278c98af290c35a4ba19b9f9de886a1e -size 28935 +oid sha256:6ba53c068fa6f2bf9b4547a5ab5f51f11aa5c0f80137d5c0a8b706fa23711f34 +size 28847 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 95b1d66bfda1..eb60e2bb9a21 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8016129425493820fb00119bb4f77a03e7e0733b6c7255c69bb6e61e17852e7 -size 27789 +oid sha256:f74473d19d89a99a0a0f0dd21d47aabf4d77b471a611608e0b6bc25351af4a87 +size 27685 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e47fcb65a667..f115e0f6ba77 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c097ff14e4d8b54cd3abfd818feb073d8045d41aa77a959e53b211e0912c1eb4 -size 29344 +oid sha256:682cfb66f654ffcc75c17ce91494d69f97cd004c170e714c7b16c8261fc63a55 +size 29210 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst index 3b87fe5fd24f..344dbd8b97b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bfcddc127287f43a129e5891f8591f29cbbbdae386daeec4c4f2e589926665e3 -size 28317 +oid sha256:9bda90cfe86098d94ae5efeb7a0ab93568a5d4eff09d03efa631237a4cdfeab0 +size 28231 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 6f079ae15dd7..78526397efc4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:70bf6ce9f63de8f23f9635eaa354a9413afd39ea770ad041f7e6de4c8742e701 -size 26536 +oid sha256:10dad76e7e94616c98c020df7d9c08804575be9221c2be4ac2dbbb1feaeceb20 +size 26623 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 86141c32fc37..26e184967fe4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af1a2d0cfd42192a5ea7f42340c7b5c87f195c88823e16b228e445b5a100ca1d -size 25851 +oid sha256:4b5ccd026a33c732068e8a71d46dc0a55e9f582dff6e7f4cc6aac5023f2a82e1 +size 25856 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e61b390c4be6..6c1aaa2586f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3648699dc5ed5e6f404f869026235271426125309f16c957ca5be233a7c27b03 -size 26620 +oid sha256:92c216400298eba624a21723fa9c675860553fb375c12a53a7e2c52d2b8b5d39 +size 26571 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst index 87de2fc62a39..9668c5994886 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4b8937633a3deeec28411c0956b1b995479accaed8f64715629264c0c6fd8eaa -size 29312 +oid sha256:1c3698abc9012dc281723d2b85c16359ba45bf279b4bbcea50beb42bd9b639a6 +size 29235 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index a4573129e362..6d18428f53dc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:70e1849b84420f3cccffdc564fc13f3e2f0749c6662345f8cb2cdaa446b995c9 -size 27029 +oid sha256:8231555b8c346e7af07867abf47b0808d0d2fa0830334f0613f259b4690a7368 +size 27087 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index bb4cf7a1978f..5e5d9a62dec4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:43fd57938c8f508c43aebd97ed4a37b4542bc2d2a4218d990244624768c3e7df -size 26773 +oid sha256:2b7cd1ccdd753699ca70f2f1514ef98079049b9bfc184b8da1aac4468d7b1a56 +size 26721 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7bdd67b9d211..649ab5d5a2ac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53a531b3f8e23fdacfd1c0048d110f7bca61c0069d47b0f36a4f9044a3d130e4 -size 27109 +oid sha256:3428bf683e2ac7afcceff89c9b1203200798e90c35b96be9a005efb40775fd2d +size 27028 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index fa4f622b41fd..6712508af80d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3eed56fdc5e1d57b6b37ff8add7dd9e7a9d586f09da48ddf182478ebeb08fe5a -size 33421 +oid sha256:ac2716a5b8055f20b58fdca00272d30bfd9900a30c178ca3e42a6ee5a8bbcb48 +size 33308 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b2e15421d4c5..66f6305df85e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b95491e4329fd987ae8adf367ce679020657ccd7497da41c81c16df6288198f -size 30696 +oid sha256:bfdcc7649e2b626922b0ae2b0f14012b68438123aba7df32593e2c733efe651e +size 30567 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst index c38d004c28c7..5a4277e1c734 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:47584c322b0fdbf1492e48b8a029a91bcce2c63b80a6e0451cbe8e66bc40e393 -size 39046 +oid sha256:8e34eed59bd63e3b58f9a0df0d20566d2853b49f1d06e5d666d76edd991298ea +size 39192 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2c3e69a3eb75..012d394206f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c71bb7881b636ee19d47a532a24489abce8db6813a155d740c15291382fcb7c7 -size 31873 +oid sha256:c6edb6719b3628d8ad3563c2c0e34a9fdd6fd0112e9a3af95cffecaf938bbc06 +size 32033 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 71773e87a267..c9892b16e8b0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ebb99d7850c5b023e4bd662f9161af4ac2004a1f466ed34f4777f9370ed2499d -size 31191 +oid sha256:a75df16630de7e3fa8a9a22fda200bc8fd55c8fc80808dccbabd2a5060cda502 +size 31136 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c2f73d987bfa..ab02613b9deb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:82e676cac883561f7641f00b415f727b5be4c0ceaa0875baf151d71400cb9f93 -size 28961 +oid sha256:1460de6d662c8094607eb08424144abcccc2ffa574633e71d5066d32f7b3b012 +size 28816 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 45b553a6f829..930f1f0031cf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f2fdd8144652c787f230e3a8b445d65e6ecbf848121cfa47de500e42c89bd5a1 -size 30167 +oid sha256:be2b15a5d6b79ad39e29548798854f56fb94e2df9254524d37e9db1fff2be2b0 +size 30107 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ac302957f01b..8e49923c000f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd80d7c8696fdec576759ade0dba295ba5125f161077e7064058dad712e062dd -size 30713 +oid sha256:2499c21313a9bed65f6bcf10a6037b421ab141fd4a208ae03980cd03ad0f938c +size 30650 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index aba078a76088..33a2ebd37615 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0a66e18d0a35fb6b79c06883b81764deb9beba424fc5ca1a127a3e5a63283acf -size 27421 +oid sha256:a9202523ed471691807baabeed562f2bb8291d56ad0f289fb482b28aba22748f +size 27377 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index d4f702cda207..9ffb34c3e34c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79b415c96674ded53cd8dd598e34ceceb182e513df864073a485292969124b3f -size 27781 +oid sha256:7a875694b3c216a4674d593ccb8a5fd35d5820e3ba1268ceb50d9f66693e591a +size 27744 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index e7ebe9ccc589..d0a6aa0e53d6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9d96c68bf96157ad6dba8134a9739c1ae751fdbf3a1b760a7f71e5a4565f00e -size 27799 +oid sha256:ea8cf3fbce766c4676ce9dfec26c856ad3c18f6b2884142e104703e3634b809e +size 27788 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8d69cdf312ac..707cec2b031d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6948c46a35dc3f1078d9f21f762a18d554c0c9a7b22d7e39a47faae2baf6f7f1 -size 28581 +oid sha256:0c2899320065622d232f04b6dc84f7f326c8ed1947689da19bf5235c97cb3c9b +size 28565 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 287df10c320e..0677cce51d39 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:67f922146b74d94810a6e7de14df6af4bf973ec397f52ec937e5f358afdb99e6 -size 32699 +oid sha256:16ceb2c02f4b957c41e33cee1865c98ecea8b35b1daf2361d6b5a12653bb0c98 +size 32645 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 16c457e12a09..3a53943a247c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94a3907fd6764d803d39a70de04e6a22cf26c575b656707dc64fd2ac4eb95374 -size 33614 +oid sha256:d5b04a297d9d14b1a80b99192049b02c0040456c145a28e609216a2c677cc561 +size 33493 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a095011523e8..dff2bd91ce1e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f158be6c41495f7d7aa123d0959408652c4a5710dc90ef8b1d75dd781dd50560 -size 30399 +oid sha256:9dfcc66f8cddb14e015254ebfca1c5f6527d40e7999e72ebf4954d97a1e2e8d3 +size 30376 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index b1f36f9201b1..f6a389729030 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b282f3f61113da3b6b02047ea73c8895167c49495debf687830a12789a9e81a -size 21469 +oid sha256:775e57c89f245686029d532d9d0266c3284aacbe252e3a47215978484df3e59d +size 21352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c8c638cfdae4..09ad77e611a5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca2d765b984290835abc33ac1476506497218b80e011bd226ff9fa77be837560 -size 18700 +oid sha256:963ab790460cd2bfcea0f6d838a838203d6cb9d69bc3d04c896425ccb5fff20d +size 18597 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b595472d5861..421470e933ff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d3d5663959a7b39ca58387adfdf59bf39e17b272b573d503adaf96c9f975b030 -size 22358 +oid sha256:d9c2584b9b1bf77631ef2f5ba8392e4da2a7c197776699b79cc552bb59f21aed +size 22231 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index cfb22372bccb..9ed631dec1f0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3f8be00d4098b9b592432cbf60b0906f3895c17a596c8d4c1556c830e62fb0ec -size 19808 +oid sha256:3660df18c795e08a318f9ed0005368fe35f55bf11f70ca9eb504f35927b6c6f7 +size 19747 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 9c271bbe002b..5d0bd869abcb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98fdf3787e72a05e73c7d8e61f09f37556fa192b60305fc21c7ca37ed8afc6dc -size 21411 +oid sha256:ff18ad8d950a3fba9dcd274c37c577ef1ff2efacca1114a400c061544d530d5d +size 21310 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index efa4bc3f2845..7a96681e6508 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a705f10fae4f1e64fdca7511299ddc55257e575a6d7c5291691ffbc4f96ca56c -size 28788 +oid sha256:43a47dcc46caed3a5375f26369d9f4ea85f91c6ee5d93bfdfd9b81377dc0bcc3 +size 28621 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 7f682c35b401..6cb27343a1ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9a7c77e0cdf761b5e8ec93aafdb757644c0f32c407ccc489bc228897533d6a3 -size 21643 +oid sha256:40a79ff0406f6ba6640d7ba62414302871c88b29c6248dd7ebca4f5d3d3ff15d +size 21547 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index ddc03d143d1c..96aed46c76a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:194e5be462e34311fd7583faec4f53ac1986d31b693fe49e2e88e5505f5d333c -size 37970 +oid sha256:4a92ac9b73bce292956e1a34fd5b7b1fe1ed50e4b172d53f671161b2159aa31c +size 30726 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 0494df9c478e..71a30a85f973 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91e952ac7b299051b27d9ed3857442edf3d6c205737789cafb0a14ea0c0e8c29 -size 29469 +oid sha256:278810cc022ebd4c55f1d32d8e189442d5b4c89a5de17f4ad1a5e1f9735833be +size 23240 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 797d754f6f39..848db04af02c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3abac7f5dd7fcbac4b354c181adf334eda468fcb0fbd601f829b215717010b7d -size 29937 +oid sha256:c19562782b787e12968c4d9616ff5afc0039f275fc5ec1f6a18d74c6418942fb +size 29927 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 28c81509ef85..fcd48b0bf918 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1c76fbb94508de577a2cf42f9125f774addb1a5a155d45d2ca25bfef97363061 -size 22587 +oid sha256:9c74e3713dcb5d7e508dad6a412fa3c7a0abd75b1ffa0daa65e77ea068a6281e +size 22452 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index f7f0d0bf37da..d93a2e691390 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:396c088f57789b0093d40bea6407b3dc134a80a7f677bd70b675f271d363a9db -size 18661 +oid sha256:427dcd1d5843277dd2c674a3f41f2fb50508c0f2918846ef9ad4f7b9b3d041f6 +size 18600 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index fba40bd6fedf..c071b3759d3c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac4579ffee6bffbec91aa170129c12bcc9980780f3e998e152cc7418fcb3fdd5 -size 22405 +oid sha256:828d576044865d1f8f45c24b1b1d351376c613c8cab5e0de5248ea64a17e8f87 +size 22303 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index fd1939fedee8..442a4d464a91 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e7c881c28f1fd9a14aec54235e2f243feebb8626b6b88cfad5d6668f46acd8a7 -size 29506 +oid sha256:9c3d4a62ac1f60c072fa01f459c1bf1e0cb2db7fb2cfc2a70fcf3d21e6dd6a72 +size 29453 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 5a47b5562b04..31caf6ca4af3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c8368ed75e6947762cf6f22bc605f771d88575ba599311c1b85759cae044d3f -size 22799 +oid sha256:3d375dea61bedf070ebaac95f7e9ad03cbfe0f2022054b718b8b2f6478658e89 +size 22683 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index b14fb345efcc..53cbed8347b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a2dd356aa50920a0ffd8c70670910d70599dee3b30fdbd119894767d7f15cfac -size 38306 +oid sha256:f33ff817e25ec9f3d85a7999799ca12f2bbf1a490f0935e218b942b9af1f282f +size 30860 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8693cb8e3b7e..e686f568d597 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:444de2f4d76121648618fbffd8842a5e6e9f04cd7eb89bae74f1bd3fcf2edc85 -size 30449 +oid sha256:b2d96ef9c967c6f8df8062739f9d0283333def0d1f79ebb92519fc17a830a858 +size 24108 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index cd7ddc7368e3..c6f864fe6470 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1186eb49a1b103886afc2764f6369b3bddb6455352389dbfef3910dccd7a4c84 -size 30383 +oid sha256:4a7e9a7eef6d623fc07f0267a6d266ae4b872af67481103466dab20a43966568 +size 30343 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 3364767b152d..25046abb28b5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:17c70ed5b01a20d8e06050a0f6eb78f70bed3298fbf09515b586eebe6ace4e74 -size 23356 +oid sha256:f071a98ceae34d747e5f793a22e927542a3433c1191e9a0eeeb37c92e0694d1a +size 23202 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 10c756abfcd1..11d6534c52eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8badd887bca8e07d6507e3a34b8b773d338d1dacd4fbb5ae23164247369e9f80 -size 19538 +oid sha256:3f11f9cecb4007b226ecffc3a61c2ecdd2ba5eb5a20b4a1e5489cb299f82acde +size 19461 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index eff61345f2f6..fd0a874c8b96 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:941126a974e26ab46f09b7e3a552624cdb18f42bb580889c36edb383b8ddadaa -size 23654 +oid sha256:b11d582a7e82aad36e4079b5652d873bdd6650b4f4cecc3a2300890a6932f9de +size 23452 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index a03fb0cf6dc6..d133cac4c8c0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84b7908f309482a305b55e1b5504889416daad70c7c7817d01f9f3e5e2285e74 -size 20851 +oid sha256:43c094a9ae266a5aa414389d07defeb93c01ae0c213ec4f52795b37807f7788e +size 20651 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 80907e26faea..b3824ac68d91 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0a4024726e52f5fa0755c7b31a1fbbb06f7f92089146ccc30274ac5f4a40052 -size 24531 +oid sha256:a6de35e8f1492ba7f530d9286d00e4bbe5bd4c06a1d1ab3c7ed938ed9724d9fc +size 24272 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 854dad7a27d7..6a813a79abeb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c24b3e99742b8227a0b44495ef3aa6bd28df766f98586c532de31740377e465d -size 21939 +oid sha256:32d8d74e38648bf64cdc3e1605d54c543dad3f63e4bd626ee04b21a81c59059d +size 21805 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 02719a9ec41f..e47d8fc33077 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:363ab3e75a9c42834825d6ff6ede7a1f806d4080492816368a3ba8bf698282fe -size 28166 +oid sha256:5c5a57de50e72f76973ae817e5d755864b55742ada845dfc2b83e3823dbdbf63 +size 28054 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index d7a0a874dcb3..0772eec8e18d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8b3a2b6fb6c904169fa41ab512e2d87b21e8dd2a805629c05d18c783c1ff106 -size 27146 +oid sha256:8fdde78b1033cbdab190df383757aa9d08c92c1a0cf65e3afa33b07c2ce0e12d +size 27079 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c7e837c21d5e..36f01e8c4901 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:34d5f1ba1018de10599fd585fd1e15ff2d45d4e5421bf3a50d5bcb15cd919f13 -size 29085 +oid sha256:11908cdabba841c01364d6463684e65526c7786a307e03766f430c9cbdd10a6a +size 29039 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1c3c8733de3a..08c79523bf2b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa0563c6693b43f87453d68a667d5b51ef920df9d6087db4b008c5c3f61f849d -size 28000 +oid sha256:c67d6846894f977e6a92b07924ddcbba558f605eef5f533714fcb1dd20e8a149 +size 27911 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b4ebbdf5596b..b83c59c33f44 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08569773632e5bdface3b87bc45c2c562d47b2024c500f1a104aaaad735fc32e -size 28650 +oid sha256:0641dabf11ceccae54de4d58a4cf7557d0758f23af857b349e8099a4bb6dbe97 +size 28549 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e80adc5497ed..1be2c08ba058 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c7c685f83452adc052fc867ca219ffd961b2ead830e3a470121371253f568fd2 -size 25793 +oid sha256:ad9a2a998d49e47d987c9e9a0785c77bf206d102b1df80867025be8e322e8b2e +size 25706 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index be027e9cb950..f6e18899f598 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8cc261587a1df90545ed3962c1511a941c7d7a5c26154072014089ee344ddeb2 -size 29658 +oid sha256:d713810d296f4344d3744d38379c551565cf0d4d785544266e46daebcfc92d17 +size 29612 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index eb8ce60e0bab..4947504f79d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:25b5a62117088c3053eddee859897dcd272554e6e5525fabb7aed2d921cb8d82 -size 26674 +oid sha256:ddb1d06ecb06f7c8687e2c24f283c74d7a3c34f1b59a92c25be3d437adf01239 +size 26612 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6819d3018397..3bf41138dd84 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4fcd16feaf70eb57c90698de64b707ae2b0daada6bb276e7c37c171937145ab2 -size 24905 +oid sha256:1b92917439bbb8e8b0caa23e13ee2ce7c5df88d1cdf26612f6576bcb05d32cb4 +size 24756 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 31e891920d4f..f9565623f57e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:caa7d73e82ad89108d28a62dc52c03d94a1ead3f313480765a22238fbd83102f -size 26805 +oid sha256:33692fd63ef8749d7e79aaddbf39d4bf2d01ddd881cedf025c153f0be44f6ad5 +size 26828 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 8bc9d955657b..62548ecd3e3b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5bcd7b69a9fdde67cf411017146e3a398ef98deae967f26bdedbff9ac7625d18 -size 22321 +oid sha256:9e76a5e7ac89f74d555c0da2f795f0ac87afbe4c17bf091b06b31ab418f0881f +size 22185 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2873b6f6371b..5d305396bade 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54aeb3f05978f69bab2c8b8d566bc6ce48eebc844c4c138ff23704190e1ae0f1 -size 23745 +oid sha256:031cdf7feab9b3e19e56d208c10344f67da9b557782b4e7096ee6923001e42dd +size 23763 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index c1ee8a3492cb..f2aa83cdbd36 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ea510f8ed5f33f638a602d62810fce29b71c49c67dc126a29126126d9c95b2b4 -size 22646 +oid sha256:a2ace7fbdfe3dbcd1c58298276dd2796acc1e1f369230b86725d8daf45e127a7 +size 22684 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 44895fc11b55..736d4a75bd94 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e6fc5a9c93a89f1e75c7ceae5259e98af5710972bef40c762ce9aa301ddb0b1b -size 20266 +oid sha256:492e377992510d1255134a2998e2c1cb558d4d4b2a6789a4870fc8f3062de448 +size 20239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index a320d5ee6e66..ced63befb76b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f4e5ed904e0f69448e220db937feca9308f00823b0d84acd72a2d84fed71a455 -size 25852 +oid sha256:8a594ceef75f2163c35cb8899e721493dd3195f21055e2153fdaaca6594b88ea +size 25776 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9834e3b3a2ed..2030bf4d156c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87e1a581d23c8c9c3ff1c7b23f9bd518356cb885226792f0450360b6141f4672 -size 28066 +oid sha256:ae4567f9122564e68cc6165c8dab472fa6359e05389223ecc12cf9008b3e5ccd +size 27935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 723117516c10..6b7a0d504735 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8eafd1fbeec97ebf8bfb810d4d0ca1a094cbb7c0b449d204f05f82d904aa80cf -size 23364 +oid sha256:69c9ba1ad271c1446296e0065807688eaa24e3f19e4dab36706cdb82c3754d71 +size 23352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index df296cfcabdf..36bcc944c2fc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa134e5dcee2b093e6d5e91b22f01429013e4e4aee3a481a4663521d540cda8d -size 24759 +oid sha256:e36364f05e80b505790205ace738fc76cf2bf599b2b58b86b0ca4c5e6ea62488 +size 24712 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 793b2ed0a62d..d83b6a3f0146 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21cfcde9e2e9fc76f342d6954a15de4f7042aeafd0eee4811e59abcc6c23788d -size 23649 +oid sha256:369329614ca5d58ffff3a08494ee7eaf55c1262c1885db7954af71b18700d46b +size 23544 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9262b6dc9741..809754e1448c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee3d14187ae3689a209d69fa7fed22b3d70d5bf6e7b03f531639779ba923de26 -size 21340 +oid sha256:1c9f65489bb98ac7338342bb26a4f652dcd1cfb13016ec076283f4937e64620e +size 21412 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1808ccbb5a84..b0e97717d1cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a3fef1eafecc19021494500daac8a738f4e00cc9c14f409d064c55cef71743b3 -size 28143 +oid sha256:ff3a9aba0af892c7fe5ac377875ee5875797aa28ac241dd95f67c30aed1964fa +size 28064 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 242bf6f74e44..58f71974dc96 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e2ef562451f819d0e5f780bd4ac184efabd69174683725d26e7c8175be46eba -size 29132 +oid sha256:b2912fe2d71698df1a6089222507e3af6c8c645da9a4d04a836c9acf53695bc8 +size 29045 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2f31b6023bb5..5dfbdde793ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fafc238bab83344d365b3daadc78daafd829597f3285031958affdba288960ee -size 28497 +oid sha256:0f5281fd0c7c81280e54e91f69be4b4eee2b85d2097dd476acaa930b3bd49d03 +size 28480 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b8371694723f..3065e7c2098e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b1e36cd0e84a828dfa38e33bc0ace88d4eb7bab6a074d23827882605e475f856 -size 29579 +oid sha256:76c09adafbce5d6d60a246fe3c1aca189276e72b68631d4ff58ba67654f38044 +size 29572 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c11d7affcebf..1b92b3f5051d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bbf9d27978d3a202a16ea2fa547b545ce401c8ed05c3b4063a572c801b2044b6 -size 26701 +oid sha256:526da3976b3c0df044002501961765ca2a6f073e764373205403c2a9321d163f +size 26621 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 5a679126847a..a8504a862409 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4fddc50bdf1cc5cfd18150ec6fc334d564f50debe0d3f4b5a3c637724ac50150 -size 23552 +oid sha256:315aad88f2d9384aff737e4c6ffa00ab08f0e61b195803339ef50c71d4fc92ba +size 23525 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index d2623013a40c..b3b4e79a4273 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49007b329b75536d6d5c4b9874a5b82205fe0dc3edd7e2b43895a9387ee41b21 -size 27971 +oid sha256:f9453dca87e18a74ffd7c36aad39d782c899545622653a3851f62463cccb43db +size 27929 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 65bd37702b23..bb5fbe7abb87 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cf7e5dc12ff6afa02a0dbf825e38d7e62097d046d9c4efa48d36f1a951db50e6 -size 24764 +oid sha256:a51681448d47ee2c709ea3f046fc7a1a5451522f8542dc52913a8f11c4e11b5a +size 24731 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0ec1bea8e4f5..dafb74d99d5b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e0d3c30191025b2b1702a106c06cb536db372d36ee020ba2c37e98a16344c6da -size 24600 +oid sha256:222cddf820c0e3569b7eafcd4db896e568d8d31aef81f645e0cdc4db8e516d54 +size 24599 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d065f8951d16..1372e89bc80e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1d7747ae442c47c2b536a01f0aac6261e58b2385bb28fc8fe043bcdd58d2e1e2 -size 22309 +oid sha256:924272c6e8862b44c3259ab43c644a49cb3ec35e2e3c863c9496b226cba1f40d +size 22219 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 5c363789a5f7..a35e47b92bca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2733c4f56d9fc35e61afb697bb24432e6db664fc981c53f503d5afef3bdd4dd4 -size 25709 +oid sha256:3e19541d7c3e17927f9be4159bf2056e383ae1c144b37ad4c84d903a4beef73b +size 25685 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 32f16fc755ed..25bc376a9f10 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:321877d02ef46482b8c3e0b9b46b84b46df50fdf41c2fcfd5e5184598600bea4 -size 23268 +oid sha256:421d5b75ca9c5e5c48fd6d9edc5acb2f41e0e7676aa0bbbd9784dee4211cfd36 +size 23162 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ddaa21c17d0c..5a57e88a6db7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e71feeede7eeb06b0b49517c6c127c385e2b0d9f998a9d8c954060bb141eee23 -size 31036 +oid sha256:36e656a35251e634c873f548fb45998462e7cd11689840a7f1a473abc6a6e4f8 +size 30983 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index aef911e76bcb..9f4ca7573ea4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c2a72252633a30072d32f29a9d6357023f3988ae2f7307c5a607828181c4651f -size 27829 +oid sha256:8049113bd5e0d98b0a7b1612eb71b05873957f20cf2540a09e22873f13042629 +size 27708 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 44bde93bbcff..9b52b1aea4f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6a8c7740f715e0e5cab212751dd5aca0600968fad7e65f905eaa882f26ac95f6 -size 31806 +oid sha256:5d87f2076fa04dff2fcb528efddebb58cf5453a2d3bd07a0114ae12ef5d1ac73 +size 31651 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8c22858c0f52..d7b68b9dddae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0a8ece620217371b150673d9daf7120db73f5983e380a1f0513bd342fc8ec97f -size 28656 +oid sha256:ef3c75f3f8d2ff15b587a4bf26f3e812522dc504b687a6d2af56524aa117a0d3 +size 28594 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2bb4b9109541..431a639429c4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6d71229136eb2a923e4924ca02e45b89c7f908579053981f0b79ff936ac09d0 -size 31513 +oid sha256:a30fee41c191e89c0cc56c36ab6ea292e50b045642678f9ab53192abfa40feea +size 31409 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8455c7c889b4..4314fda97065 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1971ff1da5285e2240f2b2a94a82a5bf3e359939fb13328dc15688ec8f83b355 -size 26488 +oid sha256:a44acc855ee633d9e6643a84b24dc4e94ac1fd1844d0b2ba1fab002c8267a2e0 +size 26363 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1337c82a5da5..55f8196ede38 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4bf91c6103186ba6808e9f5ae2c385ef71f18b10faefab02a672e07efeb2bf39 -size 32384 +oid sha256:5437a5d5c853759a4be3b4297c49f8cf33dd72b68056b4014200b6d2a15aa78d +size 32317 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 78ed85eb22a1..88d65589c9d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c52db078c9390f52d817facf0a57401c39c48a38bda2602053d1334498edb5c9 -size 27380 +oid sha256:f4f703365b798a1624f5bc3c78fcccf12172fa825f3478c15905de76d5f6c0fb +size 27227 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 769eb01a869b..8c79d20b6803 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e95950901c8b0f3da87b38e4dfa5d82684158f7f962041b04868eb3736ff1538 -size 26970 +oid sha256:db54b82965ec54cbe85899b072c8028794bfdb19896bafcb0ce4a28d3ec1bd58 +size 26735 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c35f0d50a531..b84ca09a5d39 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:074159085568ba72cccfa426a7db5b04d3ee54478b702212248f753dff335503 -size 29270 +oid sha256:1b96d27a81efc2944078f8f679e954ad124b8eb50a6cb71bd908533256e2e20c +size 29185 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index c23e5b851f84..37072b87310b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21d713d329b51a4461da1a0e43cb7c46eef5308580940def181d5fdcf6b92ca8 -size 24652 +oid sha256:cdf6bba6717953377bb5959026fac97880ed780d86db90294f8c12766afc7ff8 +size 24505 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 19c6bac234b0..4d7a824f1d6b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7c99db4e323e684a26e18fe93f99eb191eb3832b92abd4b55adbf17720ffa11b -size 26078 +oid sha256:d4efef5aa8c547e43d507343a8ede501d280a5aaf7599f2b809133a9bbf26392 +size 26109 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index c26800158971..fdf6f2f1c771 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5031dea5cf9f0974c14fddd25caa34c249cde36fbda127cb31d381ff62e46c92 -size 23928 +oid sha256:e48ccb0b1c51907bef8dc21944f8c9ad6a768a4c8bd2f4eff8aec7625021d202 +size 23895 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1e41490ce9c7..48587eb07c7c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7ea721ac409b59ca2c48e517200a16f1c1560ccbb4f3e54df01d006855058908 -size 21797 +oid sha256:a4453828d4a8389d399fe052c83b76fda543b45be6d1c1f6b8fb96c2d47990f1 +size 21758 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f61700240374..66f265852863 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:843f33c8907d34cffc38a6f9dfe27d32fdc6e79a916d886eec3063a3bbcc34ef -size 27860 +oid sha256:1035e9673b388156f46a1ab82c14fbcff99a8fd76c33851163b36d0f83d3c688 +size 27651 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2c9dd7014273..e26d68c6c815 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:39fe0112df05f26d25a2b278a54d387a42aa862fc5fa1ee42fd6799ccd25a8e7 -size 30139 +oid sha256:349bd399cb19fa5d901b75efa85c2d483fe1773b7d36f15864d7ef9030750c54 +size 30180 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index db7bf9905d89..3464bdfaf063 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:394f99c2f9e0c9368432067ec578c44b5416bf72300a74c26ac3dc1ec80bdd0a -size 25625 +oid sha256:ba65798c39e2ff97f550b9813df2d973298df4c46ffebe37a0c44e6bad6f393b +size 25447 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 9d8bd5b1c67a..9724ecf83b80 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:968b2a4269d4c24f4789e8abcdb33291279a41e63f2bee4ae88f37b8948a22fd -size 26892 +oid sha256:bdf052b6f18ef8e94b7f086d8af3ab8d622accead3fb7d24a7e5d4c3b87a683b +size 26789 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 0b38db752358..f5f14bb4ab82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13aedfaabdb6803dbdd460530e17278c854ef5920cc4ff33c4b5ca8dacb14ad4 -size 24900 +oid sha256:ca0bb7f120a42f68f3d32f1bd4b24561fe59f5e0d1d2adaf50d1d61488d9feef +size 24897 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 022acb014aea..7db58793fd09 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e335488f90c435c9c4f3330bad9f44493ee74811769a695a8146a37c1475dfa3 -size 22732 +oid sha256:6f6c0954a1d9d900be66ffb40846b36bbe905a36c53f5fccf31bbc17ab6ae315 +size 22827 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 2b2ee33cfd47..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5ed12a6743633ff56073d0660a65619551ea4fee20f5eabfed3ccb018f160f89 -size 36323 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4c8877428a6c..b3cb3aa02a68 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9f4aecd012b116f6c6dfafb7d5db38870f2f684bf387e4eac0f66d98bf60f778 -size 21410 +oid sha256:e3693a940af071e359c242da7e45f69dc6bd32c23c8ab7594e01649576db5936 +size 21296 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 188b2b30e478..302dd77a2b79 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8eb76a39b9c7c7f4942e6fa8fcadaf02bb2efc845ed5dba723d2617c9a0a6ce -size 18760 +oid sha256:017bece7b2543f243bffec26538415c629a9c6f142b0b108deb9dd7c9d6a4fab +size 18660 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 109747677019..79d0107e5f0d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9d87dad57a6d88a477f83b6b2b5f3b01781f182d0a0eab06d89599b04434835 -size 22447 +oid sha256:138247c6d0114e6c1b0e9bea948171a498fe5aa65e21cfb51c85584ad9cb84f3 +size 22294 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index bca344539a90..03d38f660497 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2250a85ac79dcb7513e0c1c3c185843a74bb7c3b0e779088879b9cb14b123c0b -size 19727 +oid sha256:3afb12a5743bfa7ea82da7b98af713a688728ade3ac4be4504241bc9262989d0 +size 19656 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8537c4cf9ae6..740b8a61540c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4bad9c759f2d268ce8504deeca5222aa53b958a7cae539ebf7483166c19558ff -size 21407 +oid sha256:5ce73e67ed07cd84e8afb9cf899bd21f3378abccc542bd70116de9dec8a3796a +size 21312 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index b46d4057b6e3..18d5edeede52 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:56a612ae71f219af5dc68c80bb0c0ec7bd5b4ec559a0bf41a06625aa6bb9efe0 -size 28775 +oid sha256:aaaafcf90a258fad7ff5ee9dfeb55d181b08b6408a5ff38238597b8e8419f453 +size 28612 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 96658fedc6cf..2d9c873b7a37 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9e35e23e6fd5558b4a3986a7f4c039d03faf83f8e252eca99d6b09cc54d5af1e -size 21656 +oid sha256:3126a429abe6da90076be861ff48916ca3ff7cff753fce3a5c5a77a8ee868f32 +size 21556 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index a1fb1f9ba648..32b898c13dc2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a17b8ff1246be3eb9e8f064ca99ef4def6ce299892dcb1000973e0c11882b79 -size 37989 +oid sha256:f53779d9ab19966b6bd2094f7749a15b972fdf6ce6b9fb44b969de0dd7757b77 +size 30731 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 70422cd1c73c..c22451388332 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a13097769a698b0580644882fd684f9c2c1decc30eb586e7a26637715195546a -size 29482 +oid sha256:04fcdced4070c4bf6ac42b9bd643092d923b6ed652570616e34292b5a379f840 +size 23249 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 894e20e8eccc..97abef57a98b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:752b8ccc1ca8c94a67cbe382c0cb5f6b6daa3ac1c6c5049b157ab5beaf4f28a4 -size 29926 +oid sha256:0eddc708fcc268071e6740d15519b9412a0ded5708eef8e9ddea9ba21c4286b2 +size 29912 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 15e364b55c20..efd36b0c3893 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c8bbcb399b0383f137d8abbc0db4bf648f76d49d3431815305439a567214a1f0 -size 22583 +oid sha256:cfd7dc6b39d63cf3453009fd8633e1c14648ad0d14bee9839414b92e080cf339 +size 22447 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 5e4affea7078..073a5a59881c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:396f38aa0c296fa0d5c022c112a1e7dac4a86b71520793b9d474686990b0123f -size 18656 +oid sha256:c7668a0bbe5b8ccdfda236b570b8e4ff81475fc4c674ecc55e158ffb660d80fd +size 18597 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 585ae73dab9a..c06372aa9e24 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0996c44c18a0952a8f47e910f42c31553f509cffbfecbebdb8f9fa0c113982f9 -size 22418 +oid sha256:607ed434b6452b3cd3d5347597d08dc0b4521ab621a9f9508cb2078a1e5b928c +size 22300 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index be1b5236546b..599fae08b843 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c21ecd6b5507a24b11bd34bcd77e3c2c85120e42d023178bb0b51904bae3327 -size 29502 +oid sha256:8e02d0c0fae03e233671d336b2f14cbaff21a03a10d922d352a8b30b77e08bde +size 29442 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 988256e385a7..03d94ecaa755 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0fb531be709cfc5d5ad10506ac5bd66a313192ee153271953a38c1076f6da4a7 -size 22781 +oid sha256:9f33a2a6df46491851f741b27cf1943f7b7a6457081e195bc4af98f87bcdaf2b +size 22665 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 20391c8e674c..afd1494f854c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:12063fb69399cd670ea513bdd8081c4672fb4d1280db2dc2cac10091fe2c22bb -size 38313 +oid sha256:c7b85bb209228fe87189c49ffd41aeef8f8589d6979cd81d4f30e9a616d5d2da +size 30884 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8f78ad66c9db..60280f9032cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3940c62648a4c6183b9989fa885675eb50329b06b586a48c4cfae336db69a501 -size 30448 +oid sha256:ab748d02b6286c9712505c084acceeb823a6c013d2c40532c7c57287ae3045da +size 24114 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 18671c2a1e99..7bb41f8f3780 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59e3b43ab449312ab9af687e3a154cc1b28913dc7634cc0672e467187d97d181 -size 30413 +oid sha256:2d5522e69781665a71d442c917b31043fd3a282598d92fc1c1c913174f65c6ea +size 30358 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index c1c999e86872..bd40e151f2a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:407ae91da7c97d95986063acb75bbc116cd5a817fa516b8f19bd976568b62ff8 -size 23361 +oid sha256:34c7f9ec3b5fbbbe1d4ba5ab14ac9e492089f5a14d1b4505a9ec4dbdf30ee53d +size 23200 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ac765a040788..37e59d0f11f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5a82fbcd2f73772ce63358cc0eeddda53d4adb1675fdc387fc2984fdd998b9a7 -size 19561 +oid sha256:95bdd63a5b64a2e66d7bf122ee76697c56ff4b50f80ffa0b17331c45c6351d7d +size 19487 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 596b681cb474..280f38605ce1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aff605e02ce3a86dda1dfa1a6ecb5c6ba2d7ba7c573d39b15e7681e10d3834f2 -size 21000 +oid sha256:f86140f5b875526c4b5543c8e5cb4e727353c8901432ca640182b3bee5dd8689 +size 20853 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index af60748863db..180d4f918640 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ed5b83410af645a60a274912b9d44d1c73ee5dd130b192e5d8ed19f0a140f701 -size 18323 +oid sha256:0969e250b8756f2dbd2da2ada06e1e49ba4536cb90a6d5b05ca49c2d32f2a94f +size 18200 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c5d4e8fa7a7c..978cf05f3c30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c6b71b90281e5e86a723f3a54e254da837067f2127ac21e3872dec4eb2459ce -size 21520 +oid sha256:f3d1703bdfa639bacc72b8f594a97bb37f29f2611176698f8dbc938838b04b3c +size 21484 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 8e3ccf897001..614fc7aa0c3c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48afc5e1ae19b0d99f457de7d136a278e356447e3212c2bc14e1a8abb01aec4d -size 18756 +oid sha256:0a00b069107bfdbdbbfd003e61bcb233f6c87b238cace8290584c67d3e0295c4 +size 18636 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 495bd8176eca..fdb833dab1f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a2ff18b668e02e17aaac1b2b847ee086231de33ad8a10677e76d26f6953f7939 -size 21283 +oid sha256:d04d7159e0c549b8c8318c424ef3573a3e5d5c6e7213cebdf936bf9dd2ecf5c6 +size 21169 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index c0badb754549..47120c2c4955 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5260a4e1425054d33336094e19b0de740c6739591f57406ea096698b239945d6 -size 18226 +oid sha256:08f222d89137d6b72db1f137992f8654f3d9f75da811d29e926009247f0a5b49 +size 18104 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 18f59d3201f4..da394bed382e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4452b6581be7a36ef34373fcb44a6b4c79092964c47cb4bc55996ce59bf36d17 -size 21712 +oid sha256:f4ecc7b4ea0e527d59070ee3062c7a469dd7415d7cf99eac4b676d90a20d2801 +size 21678 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7dade962ecaf..328dbfda5332 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:def4138df002d445b31c1eeb595310e8cf1536b4ea1b79ce37945223bb384822 -size 18538 +oid sha256:d89781ac14e4b27d34bc440cfac7f4b45f3bb9ad0ca09edc7a80591a8d7455e0 +size 18411 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ce96b90b3b1f..9c08aa2805a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:689ee8945ed88916687955c6362d471d8bff3b5eba5ab6b87f71fbeecaf9c6f6 -size 23139 +oid sha256:4b55044638c2b1e16277766a75b28a98712aee42bda4837a382318a0688fedd6 +size 22933 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index d4a5ec166181..f613dc9215f0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ee206051388a4c75850deefd60de38cfecef7222fca4623c0fa48abc68005f4 -size 20588 +oid sha256:389147e25b46e17d2ac33e04cfd614eaa07261dd5b71caffdaa8562de3e245f7 +size 20349 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7f1542af2da6..530d8ea12219 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2b415e8b2d9bc8a18b717bfc9f71dc7e0bf851fffb859f4bd530d5a1261f5fd1 -size 23611 +oid sha256:6e62f5c109468160e71f49a210d7c4a1a43c8663dfda77a6c237a6460fada4e6 +size 23636 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 8df37ede9ac2..5214fb705794 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ccc15fd747d5a9448784f9eb922f14834cb6951f306246bad757150e085847dc -size 21016 +oid sha256:a200ba739b2a15de6d8e90a700c1d1b62852a6fa32c26ceae30b8dac4e18f511 +size 20799 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 290d49e7d1d6..b53f8f459b0f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa1c71f9d0412e2b7b6fd128d38840e4469ab43f5d24818cf8b64ce03fde63b7 -size 28418 +oid sha256:2d08c5d488265fd4969ac8c1bd425b38549405d41e242e49d00b664672bff86e +size 28290 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 704ba09458b5..d9c82908d7f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ed3c2ed2f15d224928059234fbaa057cd5e4b5064941bcbf4088247c1da72d5 -size 25321 +oid sha256:5e4822db21607d0e807562656d0289415033f253912481ecf195b94bc0aaba83 +size 25243 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f8f415092896..75a46dfc179b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be09a5eae0c84b51410b8a7a9148ebf152f34f7a04c1fd3a7c0fd812c5b3bb18 -size 28957 +oid sha256:38624afb3a3cbc5779a457ebf8992cdf6122c856fa948d9212e53d443cf19ce0 +size 28785 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5348f8932cba..e1159c572855 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c2aa6bc2c32ffab125b67913b28a3b3d83a9a8b42cb61df0e131231dbad3996f -size 26105 +oid sha256:594bd54282d59f0c117863ce3d84266aa199395520eb0357c54f6ca940c4eb0b +size 26063 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1a9b94afed46..e84fb57eb068 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91d0368f0667ab5446ccc699eeab3a861f9a3e7b7b9480944081857e43c299b6 -size 26187 +oid sha256:56a898ec1d4c193d98a0db9466033b9cbcac9ea596392cfa0ded490bf82ee52b +size 26108 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7c2d75efc847..d89432f604c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9f54212f680903cc5f27c2444948a340f3c710fec0bb4495ecfcce09784c1da9 -size 24537 +oid sha256:8e4f9edaea92aa6375ff1d167ea49586812ce93bfa89e8c1b919f546214b25a5 +size 24581 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 84af9636e6de..d670db8724e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4dc79d4738f89673e9a13bdfa8ccc31de966968c3ff5b0f0b3faba2746be32a -size 27329 +oid sha256:40a13a207e069ab21c02817241a9297c34a14128b93537a2235a12ce6fe05c91 +size 27202 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4e95ddee3777..da92b1f4e82a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8400a5db12ae9db059caada004326d4c805e38a1de2988c61737ce501effb11 -size 25342 +oid sha256:e2ef972d05c4159f68ae0ad8e66e5569fb88fd5d32e6a50457581ada2df64dac +size 25246 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index aba865919f03..289bee8bb7b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1b1fc624c5b85a379845944e58247ceeb509de95f03c410ca957cdbb9b2d0d41 -size 24727 +oid sha256:239d892a8a6db06dd99d40afd781c693fd99dde3ba56b183696c8031bce32405 +size 24540 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 3c484738317f..728e11bba609 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:47a7006d988be271ed995d02a594110bd36aa8a1ab714ccbfc2bde5c82f6bfbc -size 24568 +oid sha256:dbbd29dea0a459779d31ed0848783bf13708644bef7cd3022bf172b64243910d +size 24412 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index ce93566d8929..676b2d6a2ffa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b3313f3a38338c1bf4e70438969568763b2890dd20bd0e649005e9d0ca0d444e -size 22206 +oid sha256:009221d27766fdb602d0046ae96806d7b777405d9856f0fdb28ceb2cf9d4baaf +size 22066 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d7cd3b8aa48f..41ddfc075b39 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6244009de8ae2c89c4577fe5b652008d8c11d44c2c62f9f962ea024f6b7fcf5 -size 21579 +oid sha256:289c3abe104826efc946e050958b86b8e794ac029b487ae167266fb88e47439d +size 21618 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index b04c6c3bf213..ca23812202ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac1e73b099cec9271806c6eb540b401198b29db527bbb112a21a3bdc2014d2ca -size 23000 +oid sha256:79fb247638c92225268f3ad5a0eb18213971c56ffedbae2ef945c7cdf5b9c89a +size 22877 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ab8298252508..bc0f294f8c8f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e9b1fdf4bdfd8610ddf72433ff514e154f559ac257ef43378a7ac1aaf4a6ad8 -size 19811 +oid sha256:02fc24b11d144431760a651d57e8c9810600b71bb3e606bba52775ea6f977b4d +size 19739 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9e70b11e4cb3..8505e749766c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e98d1cccd14301f685d09df00dd1633513b9b5dd62fc48193197f6733c69c08 -size 25527 +oid sha256:0e2e5c677b7e8c7ce38a755d7c3ebac0ca0cfa28a5c782b078a63c53d08646ad +size 25376 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index b1cdbd88a19e..70e3fb6e4687 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0eb8c5bf31848801807e060e982e9346f3c263cb81ffd78f4a534dfca7171ca4 -size 25094 +oid sha256:1be643d95bc1345df095f1fa3ae8c0317433e67a517a1c63fe7ff576ebea139e +size 24910 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f9c6caa89523..e0ca1545c376 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0eedd5143490ee5821fe70fbfa59aa5f7667ff1016eac8b0c86fd8098252ae39 -size 22540 +oid sha256:bb8045e671b5aa26d0bde66552151e62e8bec54f63da0f4fe6ecade75a04f957 +size 22348 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 171191cb44f3..0fb8370aa14c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:63d506941cf2868f13720fc101b9fbc7522cfcb07a002bddbfa0c16ab5daa78c -size 22314 +oid sha256:faeab270510d234d5f6427cf0ebf6b05b3d094748029ee287b4aeea313086ec5 +size 22200 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index aed3ee792919..f92ed48751fb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c0848e84f522575dc4df1a216407ae2054e03e72dad13234994452d1c0f03884 -size 24021 +oid sha256:2e64757abb99213b0dacb049e59f1be067d8db8b580e96a3de2dc2c60f7bb72e +size 23860 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 55de455bc88a..858da9dfe75e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2b34a2a22cde85970dd425d74d77bd0e1cb2c28497a6756a7fb2e197acf9f6a1 -size 20831 +oid sha256:24e8901f19768af74595288b1103eafe1c7f6902f3b2e1e6b078a1a47ab021ca +size 20705 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index f62fc379c42e..f836cf6bc9d1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:71553c3d22c36d18ec1c51e7d3c898cddd6f4869cf156159ce387672c2be2663 -size 24730 +oid sha256:a8cd19726515e9cc00604a284e5e57e9a2ce5618a3a27528c61ba5696bec84f1 +size 24602 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 7958a2d07227..8a2c4f6ddcfc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:128b3bd301a6577ff30cca7f75e9d228a8e9d9586b3e776c1a2fe3d5ab56f6b6 -size 21958 +oid sha256:287433a0744c42af6124292390e22d07a18ee0f9b4328725cccc6e679d958a40 +size 21806 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f312df14d620..679d74653e46 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9add742fca850d38fea5413ec59edaca099d154dc3db24e65b6774f45a27c50b -size 25470 +oid sha256:de49553ec0f5e1fc9daf08c0cbe59dbf799bc484d59af6e7ee426e6ef8e7fd43 +size 25358 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5aa909d5bb32..0d2def4be947 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8027ac0d0958dc1bc562747c19e8a46fa11f88f625f31af2b644f05fc3a66cf -size 22216 +oid sha256:c6131022b72066f2b2f64f5a470ecedcf6c06ccf48091d1a6a8bbe0c993c6330 +size 22079 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 661ec01fd7e4..b0dd9e519f2e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:507b9f7c56d9ebdaedc2f5b1375434ce841371abf46bccbce5c842e876195bee -size 31026 +oid sha256:bd9ee7bd483d6c3429916577a0fbd7d935a77dbdf2c926356c3c0ea1bdfa41f7 +size 30867 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 82ecd4f37084..7420e1e39c17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a5c9786f0c3230d4af70a3231c3d0c97b2c7278127a3b985183a9580fc19e58 -size 26834 +oid sha256:83b25c72fa0ad1f113cc87be5510c702cb21081fadf26ebc3af9395deffe3776 +size 26750 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6dd2400e7dfd..c494824bfae9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:44b25ae88a6815bd69cc23e87715ea36a5cc81a89be2bc3c427162646ff8260c -size 31540 +oid sha256:3a1ac6d807343c7f649c341b3aeecdcb26cbe7ca8cbe6dfe76acb08f206c6317 +size 31394 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7a59b8ae549a..432cb18aa5d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb2dd9046847a6247a7c0cbc9094df2d64873c894f9bd7f2ed43b5228c507df4 +oid sha256:2844e12a1e53844eb95d951dce8c54f82f5c9ab845b633e7162579d87ec5ae48 size 27551 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b3fbcfb53d06..29dd4b85446a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b9a6018f52faf8c0cc0d40d5cf9b9ac6faf711f884219bf63b4d87b22d92c7c -size 28848 +oid sha256:84ded7974162d16ca3898f79393bd28e353611b140108f2f4394e51ca8d4be1b +size 28711 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 95b833f485d2..864d4eadfb78 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e1dfb9854229779ff3f19a89c1c87fc7653578bd86df1257cff2f59f0cb03ee -size 25886 +oid sha256:112d38c738b56a439344e841c8b2127d934f6b8fcba83d274a9738a568c84f08 +size 25806 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a4425bf85a9a..d05bf7817124 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:58670c2a90589f15df0abc1d1d0a802adda4bb429d6f537268ced67ab5392fcf -size 29812 +oid sha256:8c9d2c173fe167e6e0418f0eebab9440579d645b0861004776b23919d01e7480 +size 29679 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a267e4f957a5..88f4b5dfa50a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:644d869f5cc83c21c5faff9b47ad709edf24392b1509683a16407a7f38cc6d80 -size 26645 +oid sha256:0704a9cdb81a84aee5e744cf7e28a3c85cb3e25c43be9d00f39699b42ee86011 +size 26569 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5c6aaa748f5f..6dde16f3f49c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e68089558019945e9c578088489b848d9b9f6f50aa10084f29fcd228ee04b3c9 -size 26890 +oid sha256:6030a2746ef95610b421da66246e46581906ee063e4f8a767dd68bec08838c9d +size 26683 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8db71f6f87d5..bd8a714e4e9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a4872120773a82af41107d8fd779023bd0d0620ed398310fde89d6f51a15f25 -size 26994 +oid sha256:2ba99ff4d9173e79216ac3620db9c22ad22f10e3796f206a533b4b394ad9e61a +size 26874 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 3062122a06d5..d6c51df72ef2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:25def7caf45eaab75f207a788114b570e6378111fb21b7a4b0dd5c54f941d3e2 -size 24623 +oid sha256:5cd258d81adcfa4d4603ee6ee1c6acbf90cb7de4647be06db0266f5bd3260385 +size 24394 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 41e6dc21fe40..36ddb167d311 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b2ab11c91489077945d954d6b469019219230baa574750d5a54ea887ecbcc53 -size 24080 +oid sha256:6194e6059f9d4359e120704c2b5d09ffbec96e23ace6437312373b80eb4ee918 +size 23920 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index b755b0f781b6..4e55955f5c67 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:19eb7888888296e412af738a0007b40bfa2a8d8f488e88fdb7c866a94a4f44d8 -size 24271 +oid sha256:52aff6a530df83153f261a5bd0e45ed4468e9e4e7787ba665c0e4a566a096fd0 +size 24115 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 20d72e785c81..3c0edb988494 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e4f334beb3d4e73b7b608c27992009387fc9065d3b66520414adaec5c24ef430 -size 21231 +oid sha256:fe29ed22379cc1562c9f6a7281c02528bfd4dee4794c21aeb776c0388989ba17 +size 21202 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4864fe40061c..d7a4ea84888f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4921b589d644c8e23f14c8d4b16caa4d4c38d7b5c64d789adb4a80358b451307 -size 27512 +oid sha256:5754fcb733fdd41d81ad84d859b9689f3f85d97e6fcaf0f80d23b046111c6fcc +size 27415 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index a118470f2a1a..2c59f7687df6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:360ff57f37722f0acb151d7fa8e13b1c7c5323951934080b61b04bbd96f0610f -size 27329 +oid sha256:36236dc0517d55f980cb5818bc1858d5d78954852fe41ba703a29bf44b50cba3 +size 27386 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d9ac626c5a1b..5723375c0c42 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef69107da61e237912ba8a46bff9fd25af9cb94f4e3d481ab94cd2f9f112f974 -size 24962 +oid sha256:1cee31e11eb2a39ec3e8a8a3cbd9fc86f28a2fb6a91b975f497a6a7d4aa0adbd +size 24773 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1d525143d3fc..2aa7b253f85c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cda3fc6a925a455832edfc38b737a0d00af53a4fc8111d2eeebef78c4f89965a -size 24635 +oid sha256:f057d9fa8568bb910ff38c2720ed3c69e86fb1498d46dfe4767399f13aadc6c7 +size 24545 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 6c7837da72f0..f19e05bd3f84 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59db6ba522ac395eb5e9e3d4b6149201fc5f0593ebb5409939f12871b42ac6cb -size 24876 +oid sha256:ce89df8fdcf31b5c3ee071055bacd194c03f1a9fb3a8393a30febfecb48c30cd +size 24758 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 20749fd916aa..5729cee782f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e3c7a8ddb5c14aa01c6cdcf0a228ab1f2f0f3df0bd89b1628d98838a6cfbfd94 -size 22085 +oid sha256:e410767cf29dc3f3dfc80a5110d22b5648b8d62368cb6aa911b4405c75dd726f +size 21984 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index bc49bde36d96..5446c69e00e4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8c938c375e64b7874822a86e860c43f73e46c8ffb48b543d56cf89358da29fcd -size 20971 +oid sha256:4f8de656eb1880e26a812e49f4a66e5cf3d08101a446c90e7bb57be65d3fd25a +size 20818 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 942eec96ea9a..a0e82e718656 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce182f56f9af795d989e20d42907f4d7681a71a1b3971b49a7a16b858fd59c6c -size 18296 +oid sha256:ae41f16b0c4669615d60270305617f9c43aa7a9bd084547a6fbf724266f95df3 +size 18213 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 12d120564eb5..f53bf05463f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c26a436e48bfff7537ef8f88a04413bb57bbe17e62329958b303f00a64e1f15d -size 21452 +oid sha256:76c191decfe3549e8c1aa8ce1e075bcf7ce7862d5a10704678a38b7aa0df46cf +size 21374 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 47fa923cff77..1c6d67055c00 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d725327f33d4154802894b5323f578328e5a47915cd3caeb4f1f0128e1555a60 -size 18836 +oid sha256:a9cba4348038353d8e4eb3afa50036fe80b04969bc30afc8b459b223d9e6c4d0 +size 18679 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index f0b0342750d2..fe9fab458472 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dee9266ed6ad2fcfd646b308b4242e905611ee3263a68a07f6f6d1c1e1c85199 -size 21277 +oid sha256:15242a22beb6cd0a0d8c9674603a46275be91e056edd6f629129aa97ef4d7c97 +size 21155 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index d9c2fbf52a77..7dd30cef67d3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:68f28fe83c2fea920af090a48bc2df9072d40979380c9a13ea86d388eeaa2f6e -size 18234 +oid sha256:1f6df1ae2131089c1530bb6fa84c25b4aedef6be078d2044adf326587bbd9a23 +size 18106 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 53b68f27ef59..71a775c8402d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88fcca7b0502519db6e5ab00017ec987fbd939afb68f081995216180cb7a9148 -size 21711 +oid sha256:6d29128c275725c18753a54bbcb900ae4f464e0cd5c83da48aa63600cbb98ab1 +size 21680 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 6022e0436827..5c2d6acd63f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8aefb6882c81300d22a3c5ae7f38eebc8c279f96347066999406c573ed983ab -size 18545 +oid sha256:c4f1692a97c45a8eeb80cc8cae59ee9b224d62e5a06a78c567d931b63c3fa7e9 +size 18414 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 983d6494371d..e5937fe59c43 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5000476ba97f33319bf8a2d85e43c7fb7c1426e437fabfe16f468a8d1e7a920e -size 28360 +oid sha256:250af56e676f044a37155243cc16454a456603a82cccf21b78901d7cf3318761 +size 27744 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index e92aabc7cbfb..0265ac6b9c1d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee37fe0bbbd2cb014db106d388e88506db014e6f27bdcd6008fd8dc02d914186 -size 31001 +oid sha256:07371b8274433156aa3490b95fbcc26b4b07bfd80b6f11a5962c13d01b545fd0 +size 29457 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 1a4543927c8e..f8f684ae8b7d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c66bf897b0dfe2d86b6f5b99600e20a8c8f5f5358fa8f0e14e36e81a23f6d3e -size 26353 +oid sha256:60b8f8c8e1751702cc80b149d771b8efd379ce4b237ff1473f80347b06a49071 +size 25052 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 59e677674421..8ba5bbafa20e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7d589008b67edf32a7bfb9fa1143270d0c080e7f394b9eb8d5faa3ee973618e2 -size 24580 +oid sha256:10739be35145fd3e70e786a50072248b89af1de990cd1bfac6464c249799b4e3 +size 23026 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 678a8a1af7fd..85fc9ffae423 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93eabe7c3439182b3cc59e19a2bc34b1e452ebf0f6b54f037c90d69bc6c81d67 -size 27741 +oid sha256:0d28bd01b3704d055e24b8daf72e209cf4c8877856801be7b0900f244e4c9429 +size 26975 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 369eed2362b8..aab905a64731 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5536595b7cf4f774e72f515f1555b803692ab55e7b63e2ffcd784deb722c7faa -size 30138 +oid sha256:26055d79f3bd00fd1d540bdc0c9fe46572af5076fe3a7adfb8180a3d48004b93 +size 28645 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 811f63508c0d..28c87e5d61c6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d28a4cbc55e616c653f1d6ace22c75af621dc4a441596c77a8412e936caa0cd5 -size 26340 +oid sha256:839b0520f3f5b762b10f62cffe8522120c03d887357639e1e470ba1ddfe7c7ae +size 24805 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index d5649f95de69..e5cf8a4a3d77 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bf979a4d02c36f4a88270fb111535b788eb61d5106c7e43759a67fe0b6911255 -size 24526 +oid sha256:6c9d6236e2ef07112ab4f0c65ff7b50e8b6c869d02c41d3003389e02af959d70 +size 24943 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 501fe3729ab5..2b62c0625978 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:508c589abbe912f86b68446a23a6a397116f7780bba9084e459ab76c4c625bd5 -size 22396 +oid sha256:5c3187154885362cea382a7e65df84ebdcf4b2edaa97744aa90893652a9c9b1d +size 22238 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index 2530bde3c7f8..3c78252ea665 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce60607f1dd81408f008b03c9d43af4ede1ba717ed3e1b3bc89ae81398add645 -size 26965 +oid sha256:58a1ec19a728d1e028003ca96f17a332aa431e95f58596a8a15cf30957e56edb +size 27330 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 32aff1da76f0..c259a384e0a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:74b108ab90ab6c5c42fe50181d1288855b1fff5dbaa9dd8c7f3a090d21481724 -size 24947 +oid sha256:68c87592a19cf1eb3571d01b01617d1ed9146cb27ba746717a9e299b22e4e64e +size 24498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4a2ebd1643d4..74bb4099659b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f1f10713a479de2aef4486b4d70f3b04c2cf36e25af947dff9691a850c113dd -size 24051 +oid sha256:a3d4d4e0f6151b9701f10a3e89284ec1bec3cc5045be23a02d5d571d6a9d943d +size 24287 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1285247827a5..ab5d7aa0bbd8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62ada189bcf21c1db71a1ae8a715013a901be09e24769b9bff7f4436fcb69683 -size 21609 +oid sha256:b6ffc77d14f2774bdb5b7f07b1991d0e1d67fa77af26a715ab062ffb9463449a +size 21238 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 2884694888d4..12367658cd73 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:929ed1fdf2b5d64fd4f1f3fe82f8d44efdd1da61d6dbc91d072585c081f1f2d9 -size 23065 +oid sha256:d02dcf61eb38801e9feb8e96251e6da61cc85b288b8cdd4869089400b85c40fb +size 23589 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 0fb5995b9cc3..77fc55fd33c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b980e7238db5da86ce4f44f561af77a1b7dabe80924a2944d9a550582c1b3ae3 -size 21191 +oid sha256:ad95f0d8cf80d8e450abb446cbc6767186de0847b69e7b3b10868ac03f1fbdc4 +size 20258 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ab78890ad568..34f5ee030814 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:818b80da2b6485b0a3f12069e94a0daa6345d7d3afbbbd6b8114ba4abf63685e -size 22055 +oid sha256:0ecad019b81d8c28de866b6e8098ac685e1735501d3d5c982295e5a268361bab +size 21916 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index f7eff05ff787..73531b73055a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:283e80821943bbc124cf92cd9253e0bbc2e920c9a0809db7b102e4b78c449371 -size 19153 +oid sha256:540af9da2d9efd8fae5fcaa01499fb15bfbe9bd39eeaea571c4133780402bca9 +size 19050 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 6301337b03d8..60d981836568 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85bad5be8050bb9881272ff68fd1de74621b1457eb4e2745b15add38b39719a2 -size 22636 +oid sha256:5c4ccd9797b9e134c5015f0924ea7dc37ad0dd09d3eee89caf8da2390c1335d2 +size 22528 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ed286f31f523..a16de605d9ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4535d6e2baa2c9588faf1c43c3e3e79981e46aea63193a64db4440811462c32b -size 19434 +oid sha256:34d5ff5711af46d72f01da55e46cff152b34e61369de9c317a2c342e2bcae183 +size 19316 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index feec5f31ef4a..60f6dafb5e6a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:734fc88586f30f686512870bfde9853d1c770b5f6263acf8e43bfbcde5e6fb30 -size 22235 +oid sha256:68b5721ae8c5e249d84276b29db5512998b8d0c67c1f8e04ad2fea676b125120 +size 22063 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 8521ac92d237..127c2a37b0e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d568572ccef0a7329b373420f0c955d43a301f74ca2fc79f6278933be397b207 -size 19634 +oid sha256:85838b7950a06cb41f7596cd7e1b976f80c71a2ecaae3cc03076e4889180921a +size 19495 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d17f26bd905e..600ac841575d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1197f51ebd287b4cbba2362074774f554c3aa238a5e391a9278469f2569d1ad -size 22754 +oid sha256:e44bd26787fef5ae80f88974d04cbaaadee7dc617c53cbe1f2d7efc20d8f1bd0 +size 22649 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1e3a389358e1..61e82ae221cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7fbc073bec824614253210e368c6e21c49898aa2cb94b5081e93929ac296a0b3 -size 20012 +oid sha256:715b4061a70d8d3c7639cdaf3d7efecb2c9f9aa0c9abdeff706c7bf9f68fd9cc +size 19882 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 9f570a22b385..c824311c5175 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94499d2ffa5f1ad7d82b2815aa400ec0a1f4533e18d430a430c359d644955278 -size 25138 +oid sha256:75c9ca7eeca2853c5cfd3f88ae6f406ac52a91c62ad2b0d7b9bc7417976088b8 +size 25095 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 84e1fa146e68..c09e91fc4f91 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e482edacfc5ac77aba17e9e943c7cf60aa796e897bcbd143a82a08cee2df75c -size 23200 +oid sha256:f73b8673357e862c1ba3ff578c5289017c230e43e041f90268cb90811299cf81 +size 23123 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0657ff1ee2f4..bada9550bdc5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f805865cec10845d508fd33eb32e2123653d6cce4a76b7ce668dd054e4108c1f -size 25553 +oid sha256:8c9abc9d4bbb057d7f4f17d70a1b1b82f7c8dbd25fbcfac61f457f4cc4d2e339 +size 25420 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 175d8763a72f..df06e6e9a57f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81ad8310f17ab35172821e43e03baee6c59e69f43fa5c618b355bf2979761c8c -size 23914 +oid sha256:074abd11748bc125209d2c7689c667c1f820b267986e7f7f68e29fb2a0604624 +size 23841 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 31f479905b1d..6bb2d655e569 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:74130eafd4cad38ee61107ebcff895c7d1a5557744a5c511c0b197ef5c3b92e5 -size 26156 +oid sha256:5c6e701fae9b87b227669d7b5aadc2daee29bc5e9a7e032cd931155aceed6097 +size 25987 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c28d40216f4c..fd86c9834d37 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a29b1a49509ef4b860596b44907c37bb765809d440cc9e5c454574b5b210534c -size 25914 +oid sha256:28dc6a7225bd6dcecc4f2488f04c2b1870579f1495b2726d5d36f7e57628f5db +size 25811 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 405777291452..e78d64f8dcef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8425ba01bc992f54439401c1fc91a2dbbea3af4aaf27b01819c6cd7bd60cd3c -size 23550 +oid sha256:7163b36d1a64a85ef827f6851e469894e6ac18c9e33e65bfe93394e4ee1b90fe +size 23446 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 703bfa507f8c..8bd43bdad507 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a10299dd7c8019d8bdd2114331512f2f62cfb0424c13bdb5f68d4f8c94d69ce2 -size 23117 +oid sha256:2cfe80c7d333aff3c3450624522a66e2dc3970435a3e297d84722ac57c87af68 +size 23132 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 27df4cee1de1..9fa7c323c985 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b13331c4d8b1bf83bae8f81e1148e4f369c625784c76fcef33ee21da5688787 -size 24467 +oid sha256:5dba7fb08c2bf1c645ed3179402984c40e627897dd6dafb7318a43c4d9a6a987 +size 24450 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8881eaad0a44..91df30f63813 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7fc3d13e0ef2d6a4d8d2f577bf9654e8eee4a9cdbfa2e106c46d147fd498c278 -size 21435 +oid sha256:76ef1d73029fd4ba3b4d67c24a345063fbf9639f58c59aa7e62cc82fcc6b217e +size 21411 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 8688ca71c807..3a3a5ad42872 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2d9ea9f783488713436f234fc531a348cb385bc0a0fc88726077240dcc25d79c -size 26862 +oid sha256:cf61845d2175f2f2d8f95cf4a94c57d88c842ee2f2ca7296ce5b55f52c854393 +size 26825 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index e08a9439e917..153a1a1bf3e9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c6fb158f4d1b15425529c5338d9780b93dc4f81e61dc7e7bb3ebc3b2527d88c -size 26537 +oid sha256:1be0e367133c33e736f6be841b7112046038f06a40728ccb24968e40b4f03dcf +size 26326 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 8274bbdc5e2e..8e72ac1ef977 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:598ae850a6b232351fd0857872484d435a7180b5bc73b8b36eec6e897ff36da6 -size 24032 +oid sha256:bf8210abd1ed84224110d00e34027a6662ec1029d773df26a718f1e30d157d48 +size 23846 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a767861a6194..030c7864fcfa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1c695b376e2254701f117f5367d3f91461bf37f5a8ccec66de8bfcd41107d165 -size 23670 +oid sha256:e011016e77f99b14219789959c0c08af257f70107a3cb301240f58c8b2fca6c2 +size 23520 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index cde8030d6b5d..690e7c047959 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62afeb8cd7a01ff6d4484a3a7bc5b1e07573d397179c7dd4be60e8db48ef50d0 -size 25307 +oid sha256:09e55bd4d56332bf1cefd7a113608e65cb2a00a8c2323aeb0c1613fa668c2745 +size 25299 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1f15afdb9a29..8fb2f51c5d19 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a34f383e5c8468f73032fcba54aa041f7c2bc02e59a0e430370600e1bf8b87ef -size 22132 +oid sha256:8fbee03f3d5f13ad6cfbc70b0ddb7da9f11b26971aeb208b53042f63e1282d7d +size 22022 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 114526feb79a..30c7a8ac085b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce09aefc38914b0b3825374da7f653d8f82b86d0966dee0274703a3842c53cd6 -size 31244 +oid sha256:4d450a215c3b87fbc2b31905c491182a0da195bae16a03c9ac9cf7c080f4976a +size 30474 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 8edc5537b6dd..647b61cbda78 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:73bb6273782d269082f1e21a1cb01c8703da2fca3ccbf54905017ad2efbb20c1 -size 33978 +oid sha256:d45b8b12f1c4ec204cc86b3807fc2c656b9ae3cc70280f0cc7ceccd4e3049649 +size 32289 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 0bbf13118841..b58d7a83ce75 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc149ae817a058e75366324798b66933d30416e07154d1fc6fd760f086d6a997 -size 30304 +oid sha256:466a1864c2cfc3b415b3083d06305bb1ba015bac432753f5be9b420230432a4b +size 28893 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index b5b5cc19b806..4707fdd02adf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4652bf8e33689d325101a2605170d7ebf704193b0a4eb7b298e2fa1ccc2cb8c -size 27070 +oid sha256:d1530543b2c235e0213a30891adc309e1d69405fd7d6d6819b28091717546fe3 +size 26741 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 810e2edb7154..1cfe5120eca2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49acc8bb55fd68115ee5125dc0a08be2b6273fc85b1952c5c4bf0cb418c1b35f -size 26210 +oid sha256:3c1a2052ed10f39215327e90ce6420f1cd30186f2b71368b8ea992c0ec5bfb3a +size 24894 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 849b88ddb7c4..1b9ae399ed68 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e92ea935544d822218c6d58275439fa0106995081cec2e088da5a7e6fcfa938 -size 30402 +oid sha256:ace6d9d854340bc58025cf032314e6fbd4eb4b59555119bfd208094cd52e9241 +size 29723 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 9e370b7e7cff..323ff204b724 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:de442585be47e4f544a01337798ea657378c783e7184facbf9c0fa03c2002708 -size 32984 +oid sha256:f04e5a0dfbb3aa6af1ecd47fef8c84698e4b15bdc7f438807b0589fee3b98116 +size 31269 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index f833a8a7c9b5..03116547304d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:76d1a0eaa9400e3c53b5264676c1252639525d42867b3891a9c5f16bc8b66fc1 -size 29913 +oid sha256:841cd3faa65346efef87e153644f455567c30cf50ab894f315ccef3c4d587a3e +size 28066 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index f584d321276d..8dd832a061e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:20838d7068a3a1ce690f4b12aa1335d83e6e686ea6edd70542242ec45d1f20a6 -size 27188 +oid sha256:3530ed9c67c840220ca2cde877f0eb55449bea6b58f2a002b5332f4ca09fbd73 +size 27498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index d040acb9bf0a..e8696d763723 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd96c0db866067255f2988657f876bf04bea3113294f972e7f20a6abf57682b5 -size 25029 +oid sha256:7b5260b616540ed6b6acf01c345fd606c21c987e44d16fa2f6167e335adc82e6 +size 25010 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index 803854d78f35..b73e0c817b1c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f8f872fc4ee9b9ad8ce5fb2907431063f9d77b51f83c1621252b8b2f5a872c9 -size 29598 +oid sha256:4553d98735b7957dabbce3c2f059c1339a1613a1bd6f291a5136d625486e6c57 +size 29730 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 8c77e92bb5ae..20c57d57b35b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2d618efe7b2dd0ece85bd7a21a15a0910b8e1ca4e0c4f43e639b9eb7c0aa40fd -size 27538 +oid sha256:61b1750b463f9da90ee4a88f8cf0df998956f88a299f6544890afddc17e3d781 +size 27529 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst index 5c980a1ae1f0..786494550473 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4eedb0a76a4e8a41453f4af3efd691f8cce0d7536da6a1af957cafdf00c90abb -size 27901 +oid sha256:35e49254879b73a2f0d87d45772753e29b9274ea45be55a933d265d6ddee2b29 +size 28447 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index b4a44690948e..28ad45272eae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c5c0007f212c7b44d863505560e285d61ada42ccdbb7170959dfb3afafd25688 -size 26020 +oid sha256:d4e2041802e764e9bbbb9537206ab23b7c06308b0342e4ea72403dc776e9d6c8 +size 25632 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 5426085cf1b4..df825deeb8a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94ac7799107792a57e2bc4a54c63ca889fa39b36cdd3f7c7bf190c74d3430ffc -size 24388 +oid sha256:b2ee9c6daee397024a7364ec162162e4346e9847e1cba8336a93966222233625 +size 24026 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2a190c8b84aa..ebc0aacd4e0f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1551d13eb21100408da766098fe23d3514c3497fa47e4f4037cdbd32a0e5595c -size 23118 +oid sha256:52a49bc8f1d7204fec1a9295e1e511a32233808bdca33c87e6f560badc86babc +size 22196 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index fdcc22033700..7f446453f46a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:51478b6681ff18c62b8e87eefcd9bd49ba1854f9c90a4429bb03f000e0b5a10c -size 25666 +oid sha256:7a9144d79e3157a57b290d05fb1924972ed804b830dcf504f7aae4e5ab9afc84 +size 25779 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 2c0b30458d4b..e5103e11d417 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:44fc7718e4025555af7d2980c43fbbaad8646bc49821950195eeb9173c7862a2 -size 23773 +oid sha256:d7ee758c5bfd508661b61250bb5ce5828981711beecd4f8f65fd980d9a813744 +size 23093 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ee072ebad355..2fa788c6fbe3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a07fa8d740f1509929b9a3f9a399eb7b4fbb784e737355ab5b1eb27f7c88143e -size 26346 +oid sha256:9b9cccae4cdfbbf9af3335b9b7dbb5181d17291329c91504674759bbe020b2eb +size 26152 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 13462e1b9849..2555365a45a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c8e5d1dfb296f0ffc2d417b6f8559f4efbaa4e3316c96ce137516ccfb64932d7 -size 23833 +oid sha256:ab7ab7c9cc323df8bce416f370a1d979a4317657fc90dace9d412d0e77224561 +size 23719 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4c997024ca50..0294aba1cca0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac29a8fc279c60cd9ecf9a89bc672f636297a69379b3d59285a28c6d9865c9e1 -size 26868 +oid sha256:2f741474400014faaddbb7c61a0bc08a1cd35908542404a8cd0378ddb9aa4ef2 +size 26817 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 02db0c3f9f61..c29e3bdccda0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f94b8e691be4555e0cc89fae81fa4db6e52b029802ed713966012e9a93ff30ef -size 24308 +oid sha256:9d6b2a33a4b6f7e4c500b9671ebccb934ac289f8f9166e0aef967e3c435a6c99 +size 24188 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fefdfce4e9bd..a78957fb7821 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53c23ac55a473636b97362b0027632f96b3c46e6582229696f56962eaa9addf4 -size 27782 +oid sha256:b148d8ced2f7d4d2052ad67fa9ab8a31669f7292e33ed168174317d0cb205c00 +size 27627 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 50e0217f75db..d0b866f11400 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c3cbdef1fd59aa6188002e1349f54301b8c0db8293c56c11290febeee0137630 -size 24714 +oid sha256:a874c189c515784c0b40dd8cb2ef13d14dcd4da8657e4dda321b2ee7c36ae6f4 +size 24583 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index aac96ce4f989..9b98452b261d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec0e4d5c6d4448a6b4867c25dd0c7c5efe845d096337085c49ef19a19694a268 -size 28145 +oid sha256:ecb0a25cc00047624967a3719d0a1ad045aebab80d8e03e6325399ec172f3545 +size 27996 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index fd9e3a808013..96bea0ed2a0d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec59ca8a76392039ac498c44dad033dbe3cfe201a610768e08f0642be1ab00cf -size 25434 +oid sha256:ce4761a52ea7c5e02f6193978edb1cf65f28e135215b4f719381e1984741aa4c +size 25325 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8749f5e326ed..115a8bfa0f03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b7ec8b3745636b29260722b3129814ad08e3e9842603fd2d46715b438a75eab -size 28200 +oid sha256:cc9dc5bf4eb526886fc57543b4da934138aee68aff5b3e327950e7ea7c2ca716 +size 28137 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 121799a38e67..e837da1d9c23 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:de55db53247b3f3b245ff21c78eab9b084fc5fe32a76fdf5c5466653aa987f4a -size 25513 +oid sha256:ad863a63282fd3b773af9206f454d17c36904c46bbb471be4a132e2d5e963a17 +size 25338 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 80cd2f172a6d..7da322642cac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b04707bd9d20b64feca8f184861212adcb994fc4d821eb6868ba2176c31c1f46 -size 25552 +oid sha256:bafec1f466eb0e30115fd54376563b180de2f5816d27a5569d2b2e83ae835612 +size 25474 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3a5623368950..795a0208ebb6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:858702d5017081fea022f6ec886c1a0c4ee94e9d26e0b3658e952e8e1d5b20c5 -size 22587 +oid sha256:44effc518083f2a3ba96c25d08be0c412ab40d9b7c863685f5c15b6119e23349 +size 22557 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index ad4014817be2..3ab82c9f0743 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85fb7c43e7a0450eb572ff1b52323e22572faee5fc5bf782945a58f3908e4e39 -size 28655 +oid sha256:083346bdc726918673c6b4300cf63bbb8680c0ed4e9c0b41e00caffda0c99b32 +size 28647 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 93ba98aafcad..7654bfe2d0bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:209688f62d136f47c10428a33a013820f6f3a7effa623e1cef09a1d634d69b2d -size 25875 +oid sha256:5ce2e1b0b0dd0109ce3ba0f116e1e6015467d550324f737d833ffea00b97c40f +size 25765 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 067ee349aee4..38fda784bcb9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:174845112cf73ed4ebe1e3c771b5f868cbcb77ab36bf986882e6536448cb7dd5 -size 26214 +oid sha256:e290b89167b34f07ccfe0b8fd1e61e9d5ed6acd0b5f86378ab018755a74a0d3d +size 26160 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 22683ef83b30..8b91652abde5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a5c72835af64383bedf324e63531d1faa1ab198923b0c25e5db05e1269003e0 -size 23270 +oid sha256:37b8de003d5f7ed2be2183669eca1eed40b5911d2ab83c690a212f28cd1ab714 +size 23199 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 0a7e02a2ddbd..6da472f208cf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98f859ab13e6387e9ac7e97b1a0255d483883332a23fa9cb708c325698d1da34 -size 32741 +oid sha256:8095a21f8b4367518b8b2d7af59d0204bc1d83dcb0d9e7cdef4d956f98fde481 +size 30729 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 17e79c8df68a..3cbc3682821d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28604fdb9388582744eed2013ee1762ece9d9300a1fb928b54e3d3bd35c88451 -size 31818 +oid sha256:18aeda3a529a05f9348ee35e2986c4a1a2c8d3fd4ec5a8d766d3b12cd65df31e +size 29152 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 98b6e902fd40..cc9f7b7dfa3d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:de66b76887ae26bfa4d15f764f90a4c0c3403fc58b7caf26bb1931dca5f17d7b -size 29380 +oid sha256:429625354b3a5501ce44d763dba2e24ee92ebe0ec0940f0960212152d7a88130 +size 29323 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index a570b4101467..610ff5a35954 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:712135befcbaea935bd4af4e39df456de59da80ed7870c506ac33da0d3f8ea2c -size 31781 +oid sha256:74649c7f2a028fb55f7e70b6cf0b1b395210ebf644b09399d470265bc701f588 +size 29896 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 899f5b83327a..91d6730325f4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1ad86903dabe0831610006cab34bbab2373e1901a7c018d58ee825c18630cbb -size 34775 +oid sha256:0b55a60702de0f81b348b3b52c0ab0ae46a5c9b932caeb9b40c332c364cb99d9 +size 31849 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index c478dafb4d66..6571912af804 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:932434277382ea26a34a8d2f6e5c8f1bafb2eebf3544d5e11a8303658adb42ca -size 30353 +oid sha256:8bb85b49e7bf8f7e8bd4c87248c3fbf60139e45ed638a8d296d4d677f9c40a7a +size 27974 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 3d00f9e3e405..68fac1ccd9ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c82a6edcf54835e6b815b8797dac1eca9700bfec4b01cefbf983af103c493b9d -size 29523 +oid sha256:53014ec8ba8b232f958d6f494e9f20ef1bba94aea6f5ee9bf7681845aac8bf46 +size 28634 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 19111eea8eb3..25bbb3e8dfb4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:24d4bb037e7ae571a52d51cee195f4a445d17eb1a2988f5f7113febb04deb3a9 -size 26822 +oid sha256:79a934d8d3b90f1523685dd8ec26dcb27af09b6d30fb6ebcba4deb7bf5ff76a0 +size 25626 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index 16ec67097302..a200eecec530 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81b16f858513ae2e0731e0bb11ae64f1d2dbe01c308f81fd189c418e0fce6682 -size 31847 +oid sha256:46ca7c893199d6f4581ff351e6fe7ec85486c450371d70d751c7801108411a74 +size 31063 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 1444d057cdb4..88474cc8045a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7229d3ced4d7babef503bd94a0c8f9bb198a5d633076e0545064be04d82678db -size 29261 +oid sha256:171581715a6183a677512ea1a054043a5c9081b8de83824c2ab7cc93bb842613 +size 28054 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 59dd71ec40c3..588390490858 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84c82654524b2c78eb2d09255ebb1601e3a6992d7a8e69baca430ef030529c4b -size 27686 +oid sha256:da74ae03e53d96a8cc593aca17526be82a1a4b9b631e940189568b360770bddd +size 28807 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 527913400197..e5f931f4ecc9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b5aaf4d66725797b8cdbcfa625978388751c2fb34879f3ce7c38951dc02ca47 -size 26587 +oid sha256:82d86407b58b4182c023de1ec64b32f9bd0678c4fb538f9e3653db0ee918d772 +size 26569 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 7efac35ff430..66ab2b848773 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be0f3edd0beefe107ae3c2e15fd785c25ef1b5150288f6e4bf6e111e83e9983c -size 26460 +oid sha256:46ae4fe6ba380e4857394339c6c01795db63f7bb267dfab6ebf18e6e1c7c6b6d +size 25607 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 21ce701e1b1f..9fdf5cf25386 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:09a4f3566b908bea73b0635e1bfec03745f0b42405c1ba7ac82ab3a57fde0655 -size 25407 +oid sha256:5d9076da994ca2d426f13f4df296356082d2c35808d6747413e8eb74220f418e +size 23904 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0e08f4dae5a9..acf61d454588 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:47e2a2a4fa23c6d8cacbbbd920a4569c08f1d9335c4d2f7249b5c582e1398fb9 -size 21145 +oid sha256:913232282ab1b684fda765265c56a8ba253f45eb6aeb878634fbeb21d500266f +size 21032 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1bacff0f4657..2ba2a1b394c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4b0a0c9622b957ab463d24b77d2b302c3ba6e355dd362f5b5541c651851f7295 -size 18168 +oid sha256:10535e325b9646920e451395b4308256ec67bd77926282b1bd93c062fd7da30f +size 18067 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9557c0a756de..b6f89eeeee53 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:56edc13b18978c153ee29809a9554d25690b1a7e10cde1ef1f160192a9114ce0 -size 22006 +oid sha256:630127fc9f976d09bb9052345471e24ed5c76e01e154bdca17913f89b9098da0 +size 21992 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a874d4013d5c..cb45360111fc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7ecbf9e6322747411f6c101642126c0d82b79057f744d3b3db5301ec7eb7fc53 -size 19140 +oid sha256:e622a5bdc10ca925c30170e4f600efd32fe4b81b7e7403e7301c46587aec3647 +size 19002 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 09df0537039d..4c0d6d24b2ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d66870ad54e8a737dae2413caa6e320a7793cb74573f6d65b213fb3bdc4716f -size 20955 +oid sha256:c785d2d582a4adb89b9244c09e9ff35567ff494dd07e866a7639fbecb4d6d29e +size 20844 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 8fbb3ea127a6..210494554fc4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1795e7fd6b511eb9a6bd9787be9930dcb6e1534084fdacd618d0aa8bb3565a6d -size 28329 +oid sha256:c3296181e5beaed0e326903cb79df45739b28a339952d510116f52f7343b9f4c +size 28308 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 1c1147373bc4..282ecf0e5123 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aac70e68ea6af122d6566c878c83aad419c2c92983d589a561d0d7099b4e007e -size 20724 +oid sha256:8e104483dea2b7730127229efb08c8533e31785c4567ccb81af4aaccf3cde44f +size 20590 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 3dd62cdccb45..24866e8c6642 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce8751eab6a83d74437daba8e6254b62d871e5d596e9d7e2f42a61fec70215c5 -size 36922 +oid sha256:0e7fdcaf3bfe81b5b282392175a8877818145d35a38a3abf0606d8733a5ec3af +size 31292 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 63ce1964a25f..e74cbe2bc489 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:96ce9c6689975f5d3615dfa0670265f0ca5af918773913360a95657d90df040c -size 28524 +oid sha256:77806755b0f7748dcaf21855cfa300a7c17ca5604467c34a6aec94c77e2f3bc3 +size 22073 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 1dbbd114bcd8..64c890ce2ed9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:146f2a469ac316b5602acfd67ac94d63805eb2ef8d147d43457646a52dc52a98 -size 28716 +oid sha256:4a46f7732dc4585ce070d85a90fcc6f11ff63c9e75e8eed66b4986df48e299d5 +size 28693 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 34c52b8ea9d9..1903dcd02e93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d243f69fb75c5cd0c4b853bf2dd57128bacff098fcf37d7bf1c3a8ff800d7d11 -size 21565 +oid sha256:00431b34157a123d94be87a771e6a6f4e41de81a6d91a3e9dee4a4eefaf27a6f +size 21470 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 6445908f8a60..502d121265ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a0c03a377a0f8e41ac08ed63edf3ad20b2b9f3a25b2e16e37a9f90fed0cdaaf -size 17979 +oid sha256:a8bd25681439fda4f1fe9040e064aa407a2b3ab833d0f9d660071af4498c89ca +size 17832 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 61abe25297ef..6463d1cff994 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0f3c76c838e9bd99ee14a6452df53353cbb45dad6e767678b5b4e40ddb4f0aa2 -size 21961 +oid sha256:c54b3ae1332e845b5f7c7003b5c907334da60264839148f930eb204f5e15dca4 +size 21868 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 309ea71bfcba..b257d55b0de1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:253be1e97c8e887c3a99c048fc2fb0f21787e1a4c6714fc1229bd44aa7d80bff -size 27963 +oid sha256:dd521b1be40e792af7ae45760bc531a51bd8470667f8b7369e1e2178e8d2000d +size 27992 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 740aa20379ba..1e0f7d754e68 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cda9bc462d485b7e3027756c398f3fda79fcdbad033446621d70a477c79a2664 -size 21830 +oid sha256:88a0f1c6671ba3bd27f3fb3fde16c51058879ee1a67646613b7da053c5f2f6c6 +size 21735 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index be5c9e6b3ff2..7966ca8bcfde 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb6c383bac882d5ab46a7cf17d243f3fa823a1c35d1b8fe0a58ee89c7d9d09d0 -size 36820 +oid sha256:0eebdfef3abbae04814e80e36428af52283e713c78c051ac687568ffe4996b7d +size 33852 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 397c7b2a5268..22f6a6f01ed0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5882e1acdcacbc5daba14b42f12096b0c8ff2574c97ecee90717f6d6b433c6c1 -size 29703 +oid sha256:20f74f9fd469f1eced53111b874a0ff2cde7655adb1eebf1b5e7a4d4c0e0c60c +size 22959 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 0d9123a6078e..df560500a592 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:45359c99dcd86ed31bd02d041d1679950b3f359ea089bf80c7adc3f1ab53c78d -size 29616 +oid sha256:cb8351a4dd30de640e7645703c9a331601700073ede3e14aa8bf932b2f43f957 +size 29541 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 006b4a3204dc..4b011c82de61 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:82aa01c994460298c2946a478ac214de7e354f622314ef135d6c2dd359c5e4c0 -size 22540 +oid sha256:62f40270c698d73a11ec5ab6bb9316da8f4185443a52dea557aae6350242a8cd +size 22420 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c81f7d4f2086..f4464648a089 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d6a07d21e21f72d4e7807192cf87f4cb695690faf671169d8fff3d5b228f54cc -size 18963 +oid sha256:e08e9c355469f9cf99e481a04d8f099b7d0bc81fc8a2b3e4816abed673918715 +size 18859 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3bb1e66b45fb..9ced3cd211b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b61bc1495ef531c469b73db3072eed519838fb956144f4ded75a7cedca844852 -size 23245 +oid sha256:104f151906ca1cc2f0caccfed513476b26222300c2c95ccbc2e01d7b08815e0b +size 23069 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1a8eab571b53..bd80b0bd6280 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8054ce79b83a50b0fba4704bb1879a3efe768239ac7149f359f9cc878299678a -size 20233 +oid sha256:dec827255ae889d78c33338cc36fb64c7dd124ef1f4c2e6c89c1a10e018380a9 +size 20043 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 63070b36d7a5..e16d598498bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cb0c13a894c4833671db95fc0de529262d3f789be94c3d01852af54e128bcffc -size 24107 +oid sha256:575a44c024c218e1b43448753e9f77341e093a3b6893981207eefb52d5c98f7b +size 23900 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 595ac6a56ded..68ae1d6bd88a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b1e4a9a4f14887bdfe8dfb8c9ef152ad925089ab909d12f8eb4347733326b2b4 -size 21155 +oid sha256:a7257b91651daf3ffe25c664e36a345f80ada21fe582d9db13b431ea90204889 +size 20943 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 977baa952bdd..de893fd62679 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dfdd97ef117d536e0798e6d3a56dd0f5343d4a827aeb5bbc2e233ae8b475e7f2 -size 27151 +oid sha256:0e4cfbc224b75f207a9e3e550193125403c6a92c4d7c5d943b45dd448bc55f05 +size 26999 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a5fdf58eed97..1b944925d56b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:721dd291a7df6137aae34bfd97a70323c4adb016f500685a40280b38de4954e2 -size 26142 +oid sha256:be9c8570467000569234ddbb72a55b9f998fe73e194f06fef5adb4f3b0b97111 +size 26123 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 21f604f35dca..7d53c7f9f078 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ad6e0f6f2aaf79d25ade3aa33d9247cc8172b48ae328e0d49bc7f6c64c7f69e4 -size 28271 +oid sha256:a4633896564561f83bd0b119fbe72d2d4bcad46a1b5f77f82dfd599dad85a2f9 +size 28236 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b9951c490b82..3074e15aefb2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ee3e295187011e395dba9e447c925801757c2ed253dbaf43d29dd8071c57290 -size 27047 +oid sha256:f4b34e43c2db154f5c235b2f08aae3e81db128373f2d739c3ac49c8c680fa138 +size 26998 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3f3e59255e9b..7726be1a0c69 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b300497372195ae619dddf427c40fb79405d602f1d4c79351b3c9f9f818861b -size 26038 +oid sha256:0c4a76a4a8b6122156a1dec855b19e7f6893e88163042b73305a1effe3b5d083 +size 25962 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 10f9758f9ca7..652f59cdd34e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d2e8cda7390b4b766062933a782ac680e357dff429af3db2aff3d8760a78dd9 -size 24836 +oid sha256:7f2225053b030dee5d6728c07d769059cfe4df8c33b08c16a2d973da02745712 +size 24711 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 22ee1d73eaad..32c6789763f0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0b5774ce3a89a9d8ee297fe4522d2dccf2f461094eee04fa03fd52b6ee40789 -size 27314 +oid sha256:2ee2bbe256c456191cf4d86c18013a012dedfe1666a11dc368697b543d04f12e +size 27242 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2d6b043c0e0a..63e393b3696f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3287e38fd8803ab283d5f6bcdc72294e5c165128a502a9cf5df234ceaab978a0 -size 25760 +oid sha256:c8425ae8b3899033d49de030bc5e5932a45d7d7ad9f04f93906a974152a019e4 +size 25616 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 74f64a3b6554..2144dbadd908 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9e1ef934d84e338d90e3db413a225b656c5cb4de00eb17e8fa5861c5a1b95814 -size 24028 +oid sha256:60aed6b6483c3739675dcfca35cc9bea64d69c8bef2aad0507b39dab2ccc7eca +size 23839 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4f98f3ee3f75..08dbbb002c9c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:526b83ddf5e2e1ab7477f64f2b15f56ee7235bea2cb17396d2333f000bf0b924 -size 24752 +oid sha256:29000f2c171ee978f668a96abc6f95e2b629e83963e2165f709502ed58a97d1f +size 24772 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 99c97e2df883..2018cb2a31d1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:acbfeec7035013af7b72a1c40a1e50b70bd531440b86d3a0bf1c332a113306aa -size 21732 +oid sha256:c0e84c11e09ff61d62b2898b532a24d024d78f000bfc67d04c974340f84d5dff +size 21613 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 939a17bfc65e..b0aa60d0b549 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:201a76196511eb53a0f337de562abec96636cfca59ffd2616922b03a2913dc7a -size 21443 +oid sha256:72ebf479067d0d03ef4a13d100c3f61977a8a3a6a4f2a76ee3ebfa9933be23e2 +size 21377 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8c60f756be86..82c311e42ec0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3fe6fe1c478c2b85a9dfd86894db1d76704074ff87b98ce74caa07c9d34e733f -size 21959 +oid sha256:9162d4ca1e00e2f722b85b7fb7a9c9594bb2017e11dca4e06e9d32d0f30ee0b2 +size 21921 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ea4a079369e6..f17e16878808 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91a6950b370a27834dac997a01d2c621d6f5f813ebbf06c820726edb6369a624 -size 19240 +oid sha256:28139eb601f2c7e1004b55a592de94f067bc91633b8cb90f79856f2fba3298a0 +size 19199 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index eeecddcb997a..50197ce6937a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:893c2f1d71e7ca445ca4f6169737283afe955e7035055522d2aedba5289ebabc -size 25203 +oid sha256:19c144eb30c28cd8e658a949a0007fbaa830fafcf67585330c39daf7dcb9023d +size 25084 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index aa3f46afa310..0284e7f61902 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88056424035ba02a7655c78da0772b946538b9a1d99cbaaf91c8b8a598b38428 -size 25853 +oid sha256:5d7f609d9b28a239704501776d6956a891954bd869b3a3c12a5b41312518f146 +size 25819 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e2ab74902c3e..d37f66b31140 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c9a5c2345ab029c4fa348dd74aa6a4a7f0f9865fc3d14953e4943d1495f15ea -size 22629 +oid sha256:932347abf5f52abea77069c997e7ae5b2964c300848457e083b5084a12028e73 +size 22477 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index bd859bda0e6d..e847941e7e4f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e9dcdc94866fc0772d181c29b80da538ebefa6c02ce4a74c9eb6e3b15b26bde -size 22443 +oid sha256:81c5449a64197e701d26386260c902fb35a2bb1ba3039b58815bfb6ba6f85612 +size 22393 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 50172aceae14..d466af0fccea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5cd3523f39706c704c880179ec9973b250a03a886de49190681976153316f2aa -size 22965 +oid sha256:709de3c9e1e1f40fb81e6d6681d40b0c1ecc5017b451d7859db1aec57f818f10 +size 22871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 0f48428960ac..de7fef4be46a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80934c19642ac4889c6bcdd70470196d76fced4eb42ea11f0bed7e37af773e04 -size 20319 +oid sha256:bae2e1f9258337627b7dca6c2a1a936e32bbb1808a8dd8f8040ce7d5675d7d73 +size 20261 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0297782a889b..87091e207286 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c3d93b763ba3960ce1f12403219391c8fa529c7d76ec614c9c96d48287ea453 -size 26707 +oid sha256:864085de9470540789ac2aedc01e87d075dd7b5be3333af3a3f66de3421a9088 +size 26637 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d07964164b31..32d6084e934d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec994e09e56323b5be6c4004b1485bf0d43ecc718b7af97140569140f2df8226 -size 27999 +oid sha256:a46e0d133d2797c2711f82b53661aa78d773331687d66b6c0662464013c9b016 +size 27832 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c969a64e8b18..ef1ce44ec92d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:234dd410bd42d830717d32fc785cc2f89a2c889291c5e3ef529d66e84e5f0141 -size 25743 +oid sha256:cfd14b0427b80c929c07261aab647e3c7af30ebd44be0a82184063a341d1fc81 +size 25698 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f5280452f64a..50a288e613e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:285ed2a2ae8da0bf5ad6b80dfd34d7728a5e276e9114e7b528e189aae5e5180c -size 27018 +oid sha256:53481681b454f901c029234563aa47c457c57c21364c2eb18fc352f39691ec57 +size 26880 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8fbd4ab6d27d..57ba0d57f623 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b3ac907ebef87f48802a69ff97a4b00342ffdc6bb61c4785cac63360d2e5ab27 -size 24641 +oid sha256:847f0d3689cf75bfac46880f4393d7147852aba9e9896de473ea926eb3e75143 +size 24634 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 97c7f1845e92..24baae87605c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e286ce8c275fe86d2f30227218d6c75eb4ca43391808e74e51b3d6ec5dab6fc1 -size 20966 +oid sha256:e76400807708d4a6ee63ee847898c0c76c01e0b95e25a8855efbff28b2c94483 +size 20902 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index fe0a3d2fda42..36fe952e6693 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8e261263ce464c221d2be75778b3172dccfedaede00e42e5bd7bf3b5264d4733 -size 25836 +oid sha256:15d8210a95b95bd4e0a760af378f1887da9f4db3b9cdbf4b348b751ff16f9270 +size 25796 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f9cdce9de6d3..fd10bacde539 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8d5aa1c1f05ac98168db192e9ed3de2a78bff85b47cf3bc07a4b782c9bdd525 -size 22331 +oid sha256:84149e2b57fbc7e95e8f145a3891a95c7ffa30aef9cb05e11cc1e45773035bc6 +size 22280 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0784f82bc5a4..07352be208b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7cac2efb10d420c0941fc1c0010abd7d1b51079abe79aefeda70b05f19e6f22f -size 24078 +oid sha256:8e9bc6f1136d37ac7af6c5f851683fa4e4537629c19066a664d62fd0155e7741 +size 23920 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 0d21185c27e6..1b1f8bbc097f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f642834e9d9b57f38fa3e3dc14162e6578a01b26840ca7898ff993214f5525e3 -size 21469 +oid sha256:82428b03026436bb0083dc33548ecfc06b6913045345f660a95521f069f6cdd2 +size 21324 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 6f2cad725fa2..749a727b6a77 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b10397839c05992242ec9d1d628618c6f331cef80b01194826fc084ea859604 -size 25184 +oid sha256:4dd27427dc95b263a0365e61c435ffe9fc3fcbb65048b08730bda0904db6aa7e +size 25037 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0f0f55de433d..7b09f6d963a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2803bdd3dddd0bd2bc98839323c0c294a05bc8eb3b41657e120465f0af09197a -size 22579 +oid sha256:051acff683e0707fe2d658790140e71f7810ebf418d44ba25d7837159def54c2 +size 22475 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1984b5d96c70..399f3f968dad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d3dce280ebbbf895c9e97808df0d5ed8c61812b79e234de5523adc331425551a -size 29663 +oid sha256:0386bcac8aa7edf8c9a08eebfa15edc0ae7851d68c9ef51e150ed557f3204cfd +size 29469 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 38307684f77a..23ac88be1dd1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55896b9d29c9369fac2fe56f5ee79156416950ff57c98e7d3daeac9fa2a6d96a -size 26967 +oid sha256:796c1772a3b114441886a172abdebfa2193a2302c8aefa395b80906227e750dd +size 26873 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ab841d26aa0d..4f723ae33b92 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5eb978417a93a821f6ded16957bc73c451783e342a04c8b310b5c08c16328fb5 -size 30845 +oid sha256:dd715e3ccff9b358b9c6652f63696db7e56390e58454d9e9885442d3750addf7 +size 30818 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 398e3918d71b..d17459fcdf37 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:09344f382275e7158e6a55392b0d6c1460168cf1e863b927e9de9247e7392f3d -size 27898 +oid sha256:58b3cfaffd3ff99957af7a5e3fef972bc1db3b4e7092d6d75d2eda51ee643a96 +size 27833 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index bc9d27b3d466..a40c53928ca9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2860ac263ee6f0f672ceb97d211d59211f10c2160ffa550623420ff13ac4da58 -size 28853 +oid sha256:74f373337d7e6e9749ea8efaf4af22153fddddce89841e48873c0a8c4b9b88bb +size 28444 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 00dae2edcdb9..614683de8cdd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b6b738369c295231e7eee49f2c4d7106d32d1b10f7869a0ff636b6970098d4d -size 25703 +oid sha256:8c3f6a85eb636a9884f47c12279f645f183cfff558ff39ac2fa4546a27deda63 +size 25576 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 67a8ac753132..b0026b11cfc0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6250edfb4227c8d0a407d8f3cd0f6e319a7197cd0c56efb35d74630e5035c6b4 -size 30185 +oid sha256:91262b7887664d582dfe120cc1423a1c5a24c05882798c7d6d6f6c765cbac15d +size 29764 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 651229bfb0dc..e8f9f6961bac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c0f3e5e68783ac3b7a79f53ef87bf98268cf5781e654a35a50c753462dd0082 -size 26545 +oid sha256:419e348f9ef571adcafde80706aae02f8beeb131f7228788d4c00af5f42db4fd +size 26365 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 561bacd215ba..d299970be324 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:598e574437613ab8583c785ec7ecc921a4f2cdea432776ce3338e0fd437ec5f9 -size 26767 +oid sha256:bdfb852a06270bf5a1229dae6c4448b5518dee6075e4b00a653540a90cf0c215 +size 26562 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index d48103fb06a2..e8c1ba71892d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc3ca3af4d6f766e5327fb968cc83c58a75b4744e8a4cafdabea81a2eade085d -size 26170 +oid sha256:447f68a7e473328bb41719ea20317374775abc4d2b5f89cfa44b88ecece91e0e +size 26137 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d175f4d97dd1..57aecb514f15 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fedfea2a36bf4900d998afdfa24038247fad366abf71c653c9fc47c8d5387fc7 -size 24281 +oid sha256:402cfdedf9cc39d6dc817a5b7a62cc6af2be62f2e485345986b593c7209cc012 +size 24076 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0c3de16b229d..058ee144a57c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d2960921fd8db3b67d3010c76c348027f0092bb9b0e5cb1c4169836c45493d2 -size 23688 +oid sha256:edd8cdd16b8ffc59e4aa4e4ecbe0c022000c3f43b8a27fbe657d219ddfff1b8f +size 23579 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index d655813abcfa..879be1d7d84e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21066bfcf42734d929c48518f2369b8c099ab4fc77e8d496a1bb67afb4a5d547 -size 23256 +oid sha256:d52d49fcfcdea070e80255b9fe78479e0bf3a60f03f9517dcbb0ba11288e27a6 +size 23245 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 71ead409e58c..ca0c2327729d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:efff992d9ae7105ad4649ec09b7511340d155bd02971f123056ca9f139095b00 -size 20390 +oid sha256:10fc4cfe9f1e9e3e1578ac4b09dd8ba20cce03b3e3f566c72c1a875790067a3e +size 20385 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 2159fe8a257e..3a3879ab4fd7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a7a2a561eaf78cd557f69e6051d7d03af36508adc7b74cb7e196cec1490189cc -size 27357 +oid sha256:d6c4db0b612a50e5e42dcd9855e034eec487109f8e25b622635a8a1231ac2954 +size 27169 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 69fb7ccfdc67..9bf7f529736b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:208e5c58b81a20fb679d9a338d8c72c949e949e0907f9210b057b90bd250a43b -size 27408 +oid sha256:91e9fd4b69f692eace08b360698d0d6fc3609b20067859f48b552dc58fa59d42 +size 27314 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 9377e2b72deb..3ef5decbba42 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b85481d3bab1f2cbf4775f5713ac065b61d29fba8121becde01d0fd5338cd0ac -size 25218 +oid sha256:350ea5b3f8cbfc0a1dbbabad3afdaa69387d36744bd9acbc8caa01fa6d3cbb79 +size 24968 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c9b84874cdc0..4288e4bf5382 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:395fd412f7f121b32ea8a33371c5c608dc8a6512401f8974d964a4e1a436e572 -size 24942 +oid sha256:dab7f2c66474fc3c39c2179a9ba664725c27079d8fae06a67613ea92f110fd03 +size 24870 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index ff11c1e531e6..39d5897eb0cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5616aeecce8a75c0a28b3e5603a43dab44f8d40e7a6b9a815dcb09f3b774133c -size 24209 +oid sha256:9f8c0d5ad62ddc47707bae8f3b05e41b3424bacf94defbf0206515df3309a487 +size 24086 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b6e19adc8637..b523074fc45c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:df9d08541c98f57886b67b88fb08a25298ade11e68b85f0f6bb36275723a6c1e -size 21678 +oid sha256:b8a81cff5cb0ff5ac87b1e2260cd4b7b9923cb89eb51fbd005e90d17c5acbc87 +size 21712 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index e7fceed11645..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6b7649e9896e9783b6a7ad68ee8f8ac89b1e67f2f3dd7cff6186194b868ecefc -size 34201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index f597b90615f3..10aa6dae80f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1efe103885f192aa60c05173f57c9ff2f761b3ee32a01aca7e14d651fa0c63d7 -size 28312 +oid sha256:54a6f5a5618bcbd8e6275eb2901da2ecd56adc54a6cae3e7d403524c1ca70678 +size 28297 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index f7d1dafdb4a3..971c70579b88 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5465f49b38c2cd442003bd7ea61d1193dd577d3476498757a5d75f0db974f996 -size 20744 +oid sha256:5ea640dc085e2bfa794826055f50762bb78d9632b54eb17995dcd39c386ccfd6 +size 20616 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index a7adde75e98f..c3d75bd4f48a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49f3b0605317bc3fb5cae0bf32988e774019d321c559bf98a6d2e170d2286906 -size 36925 +oid sha256:a668b39b81f13074f0920e019a15581c9eaabe74b4d909d1a7b7c20401b886fb +size 31278 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 33dfe9cf513f..8605e7d04e0c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d5cc2c86c3ab5e03970b3b396811c48b3bca6cda4d10aa7fa4209453fab4af4 -size 28545 +oid sha256:cfcaa4e1ba165c82e27225eb509d1a92f580324fa4b6b47c2b852b93e23e1a20 +size 22097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index b3b3780e49c6..9e25ddc7bf6c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c5383c6c3ea264b2a63d08a079866d7130fe35b743b7a15b37a022cb22834c16 -size 28687 +oid sha256:d2b4660e1d971a3bade53c8ea5ed7105c6f4796535f90720a23e992d27e13f1a +size 28676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 7231421b6340..1f19dc3d5534 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6fa5f97ca64887dcf53cbe412054ffab9aa1bca47a9ab86c98e429f5e7beea25 -size 21568 +oid sha256:412f6a6d406ea8abd94fb31a851ec2a0f9affb4e054bfe938ad77c53e488e7a1 +size 21492 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 9ebc8a8b32a8..46d801789d92 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:838ead05cda1b300bcdc4e2ec00c92d61f8ae6b3772a15ddebe4227a0cced1c7 -size 27978 +oid sha256:a0be4d06601b3e40a1b5f817ce9414208258b6d51a61419b808470482d8db365 +size 27993 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index ec0c9208e807..402db4ce61cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:89371733b30750a4cc1838cc98b539089931561c0a03e1bb5e5a88c02aa7ca57 -size 21826 +oid sha256:3c7ac41d6929ccc6dfcc1cbe17167b3cbb47c0f6790b3cacf5f64f166d75df3e +size 21732 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 55bbd6a073f8..b0a89af409c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:458e13e51ee8700165a84ba5927b99ae202df0061bf25673375f7e94d91c1156 -size 36786 +oid sha256:22d52ef4136e0b40939a091203fa1fe765d1b3b18791b7eadc7aae7287db8db5 +size 33845 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8171d8b28b81..d6735869e4d8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bdccbe475d67dfe6188c19fcd2811e3289ea9dbbaf921d5c51ed5af83911a906 -size 29706 +oid sha256:1570383baa452c8db3b04e95e18fc0caee172b8fc3af3a2deda0dab3d1a1f81b +size 22974 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 740c06460c76..955571f3e5d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:735494eb7b73dadb0be35bcd6886c63e0c0793d4db623af846df7c0d0111cfc0 -size 29631 +oid sha256:8d1cfd6fc47ebaa0aedba534ffe0601e2526daa00a7331648eeed734ba721834 +size 29542 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 945822e3ae85..f3b8428e1f9e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b616aca6cd763b455383290488711d9999c8141e8b50a9feaa55cbeb4d624f8c -size 22531 +oid sha256:3c04c4e553ecc1628cfcc45839fc043da34d20857de3d76f1cb70b279cd46e07 +size 22417 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 7f000b29a3ba..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:706baad38b53ff4e1d678472963e87243d6559e5e54e55ba82cd7527e9d4e4a6 -size 21524 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index a6f948b730c2..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9fb762984db57b36ebd6aca7cf153a8a978955308b86bf69f753a909864ca03f -size 18872 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 2fe31d88f82d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f7a49cdbe4a67702e5344d691aa708daa6484cfb5b904e6c9e1e8cd24d44f872 -size 22471 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index b322f964ca43..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:aba2df111a0c31a7631b3fe156de5018fb73deeeb482607f47f185807b85cbe0 -size 19717 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 926a2b7dc35e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:fb0d83c2e02a425f6b3dc677add9b404d0dc28433b9ae368623191e0fedc2657 -size 21518 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index e33eb600b3ba..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:597a8325ef203887c0ccb56c585eb6ce4d18bde2809b0b481b7aa3a3c7e99038 -size 18657 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index bf0e4623cc03..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:37884ccf60eac922591c5fe9821c9cad9d87d40c0aa1b99aa15e0991da9453d2 -size 22577 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 96c2b3e93c50..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4bd5dbc4dd1e32373cab4c3f52da858ea932bdd03713b2d7bea19abfe55c7208 -size 19716 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 33674ca60793..8f03708a0ded 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c266970769e93a44a4656c13df830c14737534cca04a2a5dbbdf43ae86e90653 -size 21971 +oid sha256:3fc2d6d62e36613d16cf2f3dca8dcb63c5a0b5237c3aa54d4f9272f7cbdc598e +size 21879 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 9445ad91947c..f889bc04e809 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57306f45a54a61897e2a69650e1b06fb2b61dd35c1be347663b511159480336b -size 18819 +oid sha256:ee39ec61ff0e14c0f932a74a215fd08e6699268d75caf33d740c135168fb4219 +size 18710 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 66369aeec37c..3779ebb80d68 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a778b09d3811a7fb55cb8ebd4f2148b0d61fef39086627e60ebbefb7815adc9a -size 23014 +oid sha256:abe2465eb65619959e1c2a1302ab405c45f9c3db47a4851a87ed2b746f70fb8e +size 22860 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index bcf589c2910e..52d7a41175dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:473bd57678015dd8ded00932faed07bd95bae9e8c0c46fb16f04e1e9ca5b2353 -size 19805 +oid sha256:e624e43b967cd564c6510c01f8f0e486a7ad5411fe0b1b7ab19822bb3d86803a +size 19675 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 232cd01ea44b..a12af2c16750 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:165689dfef5cbcd8ca44aea0b5b48aaec814c2a27971710549dc579b699c3360 -size 22162 +oid sha256:2902051e4c6f40905529cede733a951c51f31ab4696b2078764e47d1e8313018 +size 22032 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1da43c39f0df..03d7ab62da49 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ba3bf20401b67dc8d2420e641daade23437fcc939b2bb139a0a0237ef9b1a93 -size 18961 +oid sha256:73aa32c527f6607ef74b3edd6311002ef366ab9f3b70d586639b817796691147 +size 18841 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1e2d92418905..976118f489d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f28585ef6de955e330cb1208124bcc42214ae79c1871055329623499cc99d00 -size 23051 +oid sha256:add3e2c1850b71d69bced3d2b550c556fe9b74742b94249e5be2c9f0dcf0e195 +size 22954 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 249c836aa7ea..a85e73d7e9b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:da85162c6c891b234911f2a375a669543d26a224cd713d7b856ec0ce9195461e -size 19895 +oid sha256:5cc9da85fbaba247830c77e49ea6760206fc959080f1d10d84ddcd2eadcef2c4 +size 19782 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0e5a0d23045c..007f523fd44d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:957535bb3ec1cdb9baa39f7aeaf8ea6397ea6097a5af360461b869c7e5ddee3e -size 27383 +oid sha256:278d7fbb51171b68bf5d3dd0f28190e73592c0a8796bdcadabeb119530a298dd +size 27253 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 896de5c90f0e..55ddb9b639b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a2d90b77fb96b86656b1e92d117a17292fef977075f0f4d22d60d1078752811d -size 25414 +oid sha256:ab51a5aaef3ea2bde947fa035bae1e7f06f3088419f4f5597b14da660d9b3527 +size 25398 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3808d91db9e1..a063e9c901c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:939625ca372af45a087ec4afdd0c9c2ed11b520639ed73ccdbb90979ae1645cd -size 28135 +oid sha256:fee5262594782012dfa07aee9579d34c0c4feb36c4abace8eb89f171a9258a8b +size 28047 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5d9989ddbb1a..6ff402bb6374 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d61f34639051982426a1ff227b2d592924ff3b1db1dc8240d48541d3fa958d93 -size 26380 +oid sha256:a5a279a94afd2accae325e6ba4f53e537b28f7ea7d4f9101acdf40a1fcd0adb1 +size 26321 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 258bca021011..2d99904ec86c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:525bfc3314e2ac797cd83cb83e3031bbe2c9f7a4d0670fca99e744621c53bf11 -size 25088 +oid sha256:516315c362d7a7651781bf67c2683b1048a4ac455ced36b5ca4297b5799f4f9e +size 25016 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a54440eff51a..e5b138a42a83 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:edeb5c5304c7c05daf9ae5f79024673b0cc394ac1d99293cabb752adaf6db300 -size 23064 +oid sha256:db95ea578d4b3e5dcac1b1952ca5f3979b2b7d1dd288cf93c097f221f54e95e4 +size 22970 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1b078d0b6ef0..86789f1b2947 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:89ef8c19dfe66da84788504de3b25c738b833520144f53017518fa6cfd0f0146 -size 26129 +oid sha256:7881fb6bd420a3cddbdc1943363d9c044a883a87ec19b1c825ec2fb0e97cfd20 +size 25928 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f16382fab8ec..cc72c2127a8c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b19a41854c797a3a78ef1d7d4038814deccb5336a04edfa04ba0783053de3b55 -size 24063 +oid sha256:45872c0f96936d08f8fd5f81ddfd887d64d5c5fc68d230edf69f08776d4f094c +size 24002 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 137ae04528ab..d27d4e6600dc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:12850f0410eb1aab1185e6fa954ee96b883fa2b9db8864d3eda02a18f50efdd1 -size 23540 +oid sha256:6b049bd6a56ecdf11076f497d7dbb9c2eee2638d8803374520088be6929fe951 +size 23397 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3c0e4e80ae1d..65554cc7237b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e383d5807d186c89ec86d369f33ca99b76a8de6edcf312a03df88fb6ce039c67 -size 20618 +oid sha256:a9c7791c8cc6e23afc53d55051aede78d83c53c908f6e518fa3ca7d09a9f25b0 +size 20529 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 028812f95d04..51110692d663 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8844693b5db7bbcfd357bb99b5f1dc58be01f3da656044948faaab183aed8044 -size 21702 +oid sha256:9f5edb05e03eed6aed9fbd2307765359a30b83677129d670cddb658926e528f5 +size 21711 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 0f8c43bb3f06..70823503a54a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49b7109425987a9249d0793afb117055e224885a8bb8e4b0e3d1e34cbd53d497 -size 18565 +oid sha256:99c7b3cdb9c3d1b21d6cd47a0bd46a834c7b8a73f99e62c79953b68ab2680de8 +size 18536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 6efabe8b490b..c499e334d19e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f0c657cc3a67ad00061ecaa5fe36632d1a86fc9f3c4195480b3a101c8bc49025 -size 24038 +oid sha256:deae0dfab42b076b2733812abd18c61d06e7db68494cea886ef87824080072ae +size 23984 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b46afa4f40d1..4bd6578a5f96 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b789388a55d74dfa7a1e6eec1ca349b4c29233bab4d5cc059450e126c3e6f73 -size 21579 +oid sha256:5bcb244ed42ded4f33eb58c84e38dfd3b46121e7328247676e6be8176dce7d66 +size 21447 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 76f90c9e3979..3ba2ed6604d0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2b2bb5f42c0516df1b80b8bf9869bbae3256b1b31c03091159fb1431cb6e9211 -size 22374 +oid sha256:241a28e66f0ff210ced5820645cede1ebea93e2b52b1a8e676db9a51ece053a5 +size 22419 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4509acb92c10..a16bf5a3599a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3b23303ad2e0c85d94dfc4a3d7d49b244f3b40e88ba51ed2c4a87c4124f2008b -size 19594 +oid sha256:b27aa5b1fd64b5bc458665e6940169c45602dab2565fc797a857ef3cdd4a1ab0 +size 19559 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7af8b27cee99..31d9258d2e78 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:63c52db9fa74060adb441bcefc106a0f4cb6148cadfa3302e4fbb04efecd1630 -size 29730 +oid sha256:467f353feaf8c96c95f1e5f3a7dd82526e5e00d3431a4b8366c4f6114422bd03 +size 29548 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e315ee90f1f2..5edfe38a8b1f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4045b98641ebf23c5af13930dbeb3728f6fa40736666a4a7c8f12d846bb9b184 -size 27271 +oid sha256:ce875f2074de5cd0c14e6a31c2f79fcd85a2e066ef36470e3ab2bc917cad284f +size 27209 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2a6cea739b5b..acd32e1c5d80 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4a5247f2dbcf3892049a9ccc2f3c14d7a69b334bb5d543351000c947fc767ed6 -size 30313 +oid sha256:004beb8597a3959e2bbd8ce05ab61b23e1e2ff8d51565193061145d8d3ebb421 +size 30274 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a97aa9f44eb8..4e579ec25b80 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8da6890b5b9d7021cf42d0dddd78e29aedd49d35c9e4971dada8821d54c674c0 -size 28039 +oid sha256:25bf43fe2a88792f798b901c938071e95cb44dbe11515c41b85b34451bb0ef21 +size 27993 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1e4bfdc104f0..d1ad4871817b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57bd7ead0d7164a3d57cc7ec83a0f8d823adedab37ab9990d2e749f6b6ef4981 -size 27421 +oid sha256:7a975c237705e294f3ffa3f1adae6f572d0eb8e811c7d9460e0d8b6f04830e8a +size 27230 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 285cb5d31213..f113d70e5c53 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4077af1080af6378dfef4b0f5eb01c1270b761f2bc0d5c1e3046279c4296c55 -size 24811 +oid sha256:4ceff31385b31cb841bb30ee15d3d37cde06ff034f9da503e64f7418305db177 +size 24806 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 455a61d04780..dcde0aae331f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10cbb9a8f5c9c78a057c98b5047d14dcfcecd8c3a9f8dfd205b988509e47c368 -size 28479 +oid sha256:f0610fbe68c1686814739f9edd8b5b8ebb953e43cedbe41bdebc6a9111f2608b +size 28315 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4fbb20925ba0..57560afe3bd2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c8683afa4a7f1926ef82c542f59d4552137201924b5f176efb72625fc2e1e52c -size 25590 +oid sha256:200a13d466ddbea831e4e7fa2db6d1ee231c997e2bfa4b5ee22f370d71a74356 +size 25559 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 650bd484b304..b5d951a7db67 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28765b1618c578e4d707c91b254b7b3841c72e3d1c3fa9d340e4c0f332f5a772 -size 25707 +oid sha256:6e4e4b67844958327d6c52f69d3b7eff62f02c87563c062d013bcec778dbec63 +size 25470 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c4a0579767cf..0d074ec317f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd61b1f295d5e8633e2840d91e530330240d4dd27b578db07f9e18568ab8da6c -size 22702 +oid sha256:a5728d585f6d3e415a4c1b3d76766c6469f234183805e9e510eb071d683c6d08 +size 22510 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index b2924f698e1e..df828675dd81 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c4ee81353dc1b53200820441ae6ec7873c1dd8bb3850d713ef252a7d081311b0 -size 22911 +oid sha256:ba4427ffadf82226fea672f1341aac94389c317570ce577b4fab484af02c41a3 +size 22954 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f69c70fa3733..f85afad76369 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b0b8acfd293cae155f91227f27666fed7a2c0c9b6a19853cf275d8ad6129ed8a -size 19774 +oid sha256:c6a6d7ab49336cf61b18ecafc8a9ad777d149cd83829f8860acfa5c0838eeda9 +size 19745 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1ea705488dbb..69fd3fcb4c44 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c90d43176b85e32fcf130967d9b12ce48cb53934d41a7e34f3a2eb6112cd4094 -size 26126 +oid sha256:34622cbdcd378f761ada001f3c027fb547f3dced8d9e6ca21a4f0e3de35ed23d +size 25876 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index da6c55950c21..66612c5956ac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b4b53c82bf4489ca3a500ecfa3c4b010ecb4de4c48c782686c057b658fb68ed -size 23356 +oid sha256:262b6440262ee56d0f6121b6b8b9e3a25508e6dadd21d099a00f4142507b716e +size 23186 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 17d5d9ef7a68..0bb10204c6d8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:efd1f7ab7b82693fe470362999d4ee1db751e6a99f236f6b0e021b107e67c690 -size 23675 +oid sha256:7ab831f82d0813fd0504b8bbc2f4796384c20dadd928fade37ad952981071dc5 +size 23745 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cc9452f865c5..9ae697be1d1e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ecc9301bcefe5429059377628d2e6c50698dc2921cf8d5d9cdd5a45f8b58d85 -size 20677 +oid sha256:9ce0e45c9c8347768523404f781017fa0cb198b8d3e2bcc048bcbd8b5e5b672a +size 20636 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f8ffdc3c2bc8..32f0781158c4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e4cf0949ecdef2d8a0fd5842f9c092a8bbca9686291bb9128dee57ce17d40129 -size 21511 +oid sha256:a0d20738aca00a2a6396a381bf3c2acb28f2677260fd5d9600501a988fe11ab2 +size 21436 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f12c9848aa42..fda8a79dd435 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0f0c69c2ab1c459c3c27a7c52eb67317933347783a55a4a65681b16a183da8b8 -size 22442 +oid sha256:244239431806ca086a07eb4b70ecaabe8d7d97edf39ad976bda8231162a659bf +size 22397 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 06b02fd3c4ef..a2aa4d4a1a7a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d1b9a874a7965e59afc1fd2606639c6c248aa0af8f0adbc2c53905b9f9019c59 -size 21924 +oid sha256:2b1957583cc4b6cc78867411943e7a95ec1fe8ffbeba007fa5c1af13795809d7 +size 22044 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e20059e48c96..cbab48ca89b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee5f5bef17c2cc3187624e7a9c09d0719133adf752a9758cfaf8e7510bab5cb1 -size 19707 +oid sha256:44777204a6c416ee58d1bc20b283f0c61226af323af37ff8d943fc609b1dff8b +size 19704 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2df9a71c06ea..9078ee44e76b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:405567939f19a520b4299088b201f45112f16f399cdccc8a30bb4fa550c06432 -size 22800 +oid sha256:835854b50ce3e4b4e6a34ca5bf7518993c70f18a4367ef1c8c01531e7af7f798 +size 22822 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 66ab8e25a542..016fcb0787f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8c1a7f6adc68b2caa45276787606b308e1e070779b29dd1932aacc78a1185620 -size 20651 +oid sha256:417529282413ff8cc1afbbacf12901fe72ba0a08fc98c1a9f21c40a64f98aee3 +size 20671 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ad9d6d099e19..753948413678 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c2cd744ee562cd6de4169e050182b381f44a3acec9b796ee3585a0d00f0c554 -size 23541 +oid sha256:7575339b176a52af1deba0b4b874b7bf22fc115b5e794aeaf7b64b1351bb312f +size 23414 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8439fd7751c9..0f988bc1289b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9357013f22f177859d1a88eca6497051942aaad776032a8b670684f7742d56b4 -size 24720 +oid sha256:e9e3e6329f43ad1284e2c68c665ce60354b555fee41d14b25446d9cba7de89a5 +size 24684 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c571ba0e378e..92d25f4eb122 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:982f014920bd62f13c3f774c2b02ae823cbf620a07f8560e43768e1f2eb4c90c -size 21468 +oid sha256:b8aec8beeff78fb8cd81cd1f8732a7a9057c833dd09929f0c5caa70ef44ac32e +size 21557 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 842b7d377c63..7c098f1fc7eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13b5729342a422d02231000e5f5354ad0e8383ebad093bdec7fa7349c68b31d6 -size 24158 +oid sha256:2827b6debc116e648831a31ae71813a39770de23c4a748d62a237d4f5912d589 +size 24045 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 00840c9df021..b43cf4120118 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce6b866b953497e6bdaa835ee6c768be272c018ae21fc3e351c43f393e547030 -size 22353 +oid sha256:ae53ddd58e922e375d4a884508ae72f2d5b4b18c4794e3f2a1b18deecc688a13 +size 22331 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index bf915937ff4d..e9ebcf7d8145 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87c7439abd365855837d9f822df40e0cd931f05f825be31ffe396a85c8357490 -size 25072 +oid sha256:20d25fa47f4204321d6dcf86b024cc72077b4994e6ba3b7ca35bf6d04f56efd2 +size 24959 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cd8b94315809..d493e8e25bfc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4b73e7eb8cc3f2f6ffb5621f405dc9bbf39d1af685ff19ffe2868844eaccaeff -size 23154 +oid sha256:82256c53042fac37cc83e351b6a921e68008ff90994c818054a4d8e02a6585fd +size 23085 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst index 0e122ca09920..781f52c0cbc5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:07dc93cffb220e1ff562b0c1f3c29f69995836966cd8949ab5ce44aef5fcdeab -size 25821 +oid sha256:9b453a80df03cd68d6b1d3eea51a2ddf3e342e48a80ced8f1c8988c5309efa5c +size 25781 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 05adcd4bb69f..26fe886afd50 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa7973ba4570ad3bd81dc167b69a51fd6e086030c8ade41348314cd2df253865 -size 22916 +oid sha256:3c170fc1bc1cf7027436e591f45395ac7cff367c17ef26e26f1fcd31183a26a6 +size 23073 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 700d1c169e28..8e099743189c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:74f269421472d2040d41c8931ab833382106c3629558bffa3535d96f206050a1 -size 21872 +oid sha256:3c1b3aa2035d9dd27ff6f6694a3d3c89989254fba6e5c28a3723856b7ba60fe4 +size 21827 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index aba525c7bd64..5819906a9a8e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d761f356c37f98d2e3a3bbac13f4b820f04e8b77200b9e8a4fd094907d983aac -size 20524 +oid sha256:c94b7a18afdabeca3ec96c9401baf3312bbdbee4a047f2958a2286afc385fdfc +size 20478 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst index 768ab372ad6b..7e3915bc8090 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9cd199f39a67211bd73917bdde9c2d001ad5a31386ce4655cd0028653ed9d88b -size 26832 +oid sha256:d0ae49eacffdced6dc6c95b85bc25cd6fc9a405b48eb3f356f4d77bcf1252621 +size 26804 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 804f39729c35..a828bfa89d5f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e7120de76f6007b1f34afb4c8f72072963215a7599a89acd435e0c7d7c368919 -size 23952 +oid sha256:c36ffd18b510cc3154151a16f4d8f90219547ab149137c80cf7665c77883bbc7 +size 24048 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 2c2ac166af05..03e99357ebae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:163f2f3075e44c72e9ec4474bbaca60a26e57c42d38806ab66fcace6821bfde0 -size 23171 +oid sha256:a4e22c2606801582de1839475fa6b8c3fdf2404035b3e04397c781439dc7e479 +size 23084 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index fb64c80f9976..1f6cb060b4ca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:afa37c195dbdc8da2b9af7c7927614fc422f28f3742807359c6e51c38af48e9e -size 21297 +oid sha256:4ee15e8e58011a424ca4260412a5528c1faa08944da2be125dd4b561f56f097f +size 21284 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index a25b32c9a92c..9b0d2c3f38bf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae04ac48259dab3ab16116d22898af514b071a4a1bb471e9422a0bc921a35fde -size 26258 +oid sha256:40e1638024a280277900948c02ca0d3cb30c89089460f2382acd651acaa11f48 +size 26230 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2b5607c9acc1..b99c3a81a9eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e178d48771114508195da14db4c731579cff38f9cbd7f2e67caae5129388a0d -size 25479 +oid sha256:4c83a1060728a9cb83533f024c92bb845c8293028578586ec864f11c99a1c79e +size 25375 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst index 0fbd4197f379..c3a9179adfb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cc3394c4ba62ad360f4eaadf18c66c0bd6445363c7cf357ed376f3193010ec76 -size 28537 +oid sha256:c07aa7019b36f0663c3023e043c86bbc6ee86757c1e308917217c0de2e490e4e +size 28729 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index d490152b969e..8bf0237dffdc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ec0d4c9b0a2db491aaefe1c61216481a58d44f61cfda694c6147376e78a4bbc -size 27774 +oid sha256:e6ec4165107785ba72c60a9309912dc09ca6236bfa3d11c64fc4dbbeddc9efa1 +size 27792 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst index 79a69c11086f..7955fa9ad13e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a86df82c7d1001f3268a38b450a456d5e3ca9e7bdafe445d29fa4a53fdba9f9 -size 23910 +oid sha256:6fe6020412bbe6ab2295d48da64181b544bdb0235142999ad923e291e8f8ee6c +size 23935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 35008a2867ee..1a09813b1075 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08229811ae3809f0f300a20326d39e22bd9dd6d60ab3ed93bb434f53f279c35f -size 23878 +oid sha256:3aa9d17b631d353ba8792fc48310b7714d37be82c9edc7a19d0bb4628ce611be +size 23828 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b0ec2f3fb000..dee3c6f349df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fe8a77cfdfdcdcbad49822ae83f188f0c4db655420349d3986b46b1a63cd0f8c -size 22511 +oid sha256:c9082285297445d16c8b592f1a77fa2e06a1d51346715d02376776d73de0fa73 +size 22425 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c016d6ab0da6..f80ebac6f51f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:247dee35a70ad2850ea8b6a8dea2f0c3c70eab0220d23bdab8e360e501b562d5 -size 23319 +oid sha256:4dff939ff42d5920efe3f9f959de1cdaa0c2e962a590f76fa8ef748f3268fc20 +size 23247 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index fb0d7fa4ec3f..b273b6bcf4ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e7dedb91004c1fd296312a5e839f7cfa704c0cb3ffdf9b62ce5697c8367de74 -size 23727 +oid sha256:5b2fd00f75cff703524bca136c88fa2761a6a04e08fba1570dcef189541a9eff +size 23731 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f0079cc4c625..29d074625d80 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f12849cdf6439203a0ccd0fb0ccc2c3ce162dfd169a609ea64e3947781471e7 -size 20681 +oid sha256:8a44d96e8085e8ff11ba727aee5197179ff0cbb1a5925220a29db510d03066fd +size 20631 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 126caaaccf16..189abbb3ebab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7ec1dac19447bdbcc791caa1d137d51dda00bd3537825fbf5656c96b78e84990 -size 24232 +oid sha256:e40eccda2cb5f167eb45953b10d98063590c38ef9aba7b75c5f6e48489f79835 +size 24212 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4186acde41cc..5ec44f3ef6e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d48095f24cc1b27de2799a3b55ea678e0fef8006701ecbe4470cc061de27e5c5 -size 21561 +oid sha256:42d1788fe99f9c0e0f27baee1bc65344088f65c6998babde3e93b1433d2ffa68 +size 21557 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 87f3e5b91c0b..236245927fa9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a6a9aa828ccad29396128cd3970a5edb11e37176d9f19b1f018adfc5ebbb2209 -size 27931 +oid sha256:44ad4f62fb941ae364b2fe9802e6ed849f542b236698c39277e1571ba50eaa8d +size 27911 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index fac393f6412a..75fab5199bfc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b323e1b876d74d8b870bb3cfd24a2002df6868a990af9f87765d702aa803192d -size 29743 +oid sha256:b9d48a72d251cc70110e86fb38bb5c48d618664e540d1ffd3dc664bad43e7d78 +size 29685 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2a4450f625c0..513f6349bd59 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1dadf8be31a2dab464be266d66ea2f4ca262efec2e16749bab9ca46a0c231ee -size 26294 +oid sha256:bab7b4e2109353c72578c644df8a4240fa424f6998828bf1f4086633b59dfc9d +size 26225 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 44f71f87d3e6..22d8e9f738a9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d3c17aa09b4a83f2252b6c62746ec4aa9ded62c57036907109da4e0d80fc53d -size 22620 +oid sha256:fab34d2cc8a26ea22fa9c892be1ea7944459b5f99bf752bd21b8bf72dc98b28f +size 22536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 04c84ba627ac..b8cbb6b7e074 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4b8b5b88c89b38d7a945639d3c07459a7914f16a9a4be7d4889c35478f7f99b3 -size 19729 +oid sha256:4e6cf45b5b37a87b56be9e07c0ec83453abae5cc8b40343af6c35d7c8966eda9 +size 19615 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d36f355875b6..b0ba6bd34aa1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a45c455ecd7a658ff789965a5aad65bd7a9783628dd188efddd27f9aeb3b591f -size 23424 +oid sha256:a3e85b23020259c68203a76baff4e81e4ad192ef1d1e3e3f5c80b69d06229f20 +size 23297 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2f7e455b7743..c1fbed3356ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:901625f1fa6863e44bf96a97cb0a336cd09cb170d87ef12b4153bf7e7b0aeeff -size 20768 +oid sha256:c4714fdba19f10b46b67847ca8a223bff667aac4480bdb0a7544c2ccd3d1c5fe +size 20693 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6c0a2836d1fb..c333b2b65f81 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a0ab6953f3af7a61be5e5479f12973a4a1510856ed2f32d6f6ecd017a6d831b -size 22509 +oid sha256:1ff1434f7a99efcf09449ff748f763b8be580fc4953ab454c2a9aa5784de81f4 +size 22389 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index fcaf1a0bc834..fae5bb1df806 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:69cf4401adc8afcbbf72f86118bbcb0324787cd6b8b49fb3a07801f8d3db5c24 -size 30126 +oid sha256:ce74a675eb5cdacdd2fe544a8a9c59816a0b2d9be5b17b7afe0e32ce8649dc7c +size 29955 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index e416a5b8141c..5640c3a6d3d1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10a16cb892f00d0463c39b993cbc59501c3dcfecae7421b69f519d83619aba5b -size 22983 +oid sha256:2ecc7421691ecf4b2bc51722e80c8ceb9005804dd0a46b5e445bf7041d042e96 +size 22871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index b071ac918c3b..a3210dae510d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e68d4de7d83e51ad2a594d9e93e399d53b7e37d241af7585e8f9cf7c525cffb8 -size 39425 +oid sha256:1e93182e46bd48a257e14172eeede079ccc5c8a5172ae16e07198d66e528c249 +size 32100 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index efd48ea7435a..65002e349767 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2400ea3030498f357f68fe4e5738038fe63f4ceb590278e3ea5f0726cb9a5be9 -size 30823 +oid sha256:b4ce78f0b854321c98614624c6a8da0ec1360f3d74d01cca54f936adee0bccfd +size 24681 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 7107432d0be5..3bf3e3940090 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ac6ef94213b85550f3f511e9d85552896b150bcf0b41e9befa3e3b000d269fc -size 31138 +oid sha256:66f1469c8342f4daa4138a8fd606410ced4d362c7befe6210003287da4bb8d86 +size 31093 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index f29a1ff01d27..38d7ae996ac6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:02432efa1accf6954fb232ccfbfd3c3f43293b1ac31691feb348bad2929f127b -size 23872 +oid sha256:a50f458946adfe1759371b48c2a1cc9d39f0ce16f0c8c344fdd99b8ad51a66f9 +size 23740 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 4eecfe7056d1..8082d0dfe459 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a58c81ebb865dd23daace39427408771c14e661264a5fc2ff880b1c4199a64e -size 19583 +oid sha256:2643e3b4d0fd0d0daebc8b408a343161bd3571ae33faf869aada2d6f98bea964 +size 19500 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 25ebd1f07512..aee3d3d562dc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:631c3737523cedeb92d669fb0596e3bec323e0188e184d29af025f0b7bfa4ccc -size 23507 +oid sha256:dbd3b83f3053fa9c66359f2197f49270855fde387f3c685b0a83671b4d2c5357 +size 23411 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 84e3a804d985..d5325bb94a30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b37654543633708e2a0f638611542a2185cd180490a41d8e114ec1442d49707b -size 30902 +oid sha256:11f1f050d2f7c1dedbf2b3cb75a4aa3114db5f5ec293163847e137e86edaf147 +size 30799 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index c7ab1802140d..3e728de1c465 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cb0620df2ef3eb218f0ee7e78056177d041c39ff5d42d286dbde0123082262fa -size 24158 +oid sha256:65297ebd10ab2b61e7df5865bfc005ae703205ab45b145eed43a5115d6f9de0c +size 24070 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 38ca42d5f7b3..e3a6f92fa8c0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ceb58e4dbfe1e8efb5c4c98c2fb5a69b74985f41dfce4921cb2d0c8d83dfb37a -size 39731 +oid sha256:88876624488f1ec751c3a09aec9f24e0818c6757412938abfa72d90434560ef8 +size 32234 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 50dc8bc2aa71..786149f06804 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ce2ead0e0e2a6ffc957cb8ea773b8c963325d77f7d1440cbee87c47c83cf0ba -size 31950 +oid sha256:2766c38feeb7e006bb3f7837066ec4d216a73b18b3ce5a76eff2970263dec0a3 +size 25525 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index ca550c2c0a16..23bced66b844 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e6662fc007f69b96f6e044711cc5f2ef1ba997475c162f3e94c537d73442661c -size 31708 +oid sha256:e2235dee0adac6540ae55a402679a819ac61c3cc76b8e0334d7b5eeb312fa970 +size 31636 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 273721036fb2..43904f2e690f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f37b8e9d0dda46c551fc2194563efc1cdf8e41f783c72a916eba3811dcb934cd -size 24766 +oid sha256:23796613467ef20f5ade1444f766b3bcada53e3b96a15a0dd1356eb5fbfb7fc1 +size 24611 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 90f31e995101..902e63efeb8a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e5da73b6bfa69acfb228a750a3c3df005e74cc04a0be760fdfa9bf209493a479 -size 20508 +oid sha256:7e0ffb60a893a774e777ad394613bb4ba8ca796242e7ebc552c7187c4292f862 +size 20416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index fb6d61388acb..eff4c1013c07 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26663057750624b74bdd84115475b9852e13d68fa941e0cafa96604577c941bb -size 24699 +oid sha256:2421b0d4d89ec696b7d62f07ea0a3fb76b30a6c045a623e61659e1eae4d8545e +size 24476 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c4e0ad396f42..4def2ffe5b9d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81cdb1c931de705ef902b036d165afae39c27a3168e4692048ba7ce8367b9d41 -size 21767 +oid sha256:193703776e886379262af219b002041f6e728ad3480fa67e4f4886985477c1ba +size 21564 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 55adbf2052ff..5807818bdda5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:20cd3060fb9db70f895521cef3e3b9a08cf97b00ae967cc046fcc7654330f128 -size 25624 +oid sha256:ddd1154a152e5abdb46329c6f038521e004f16a2ee7538e17cd76f6fc36483f1 +size 25351 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 3777c8c5d58a..c9d1b61920e3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c4f0fe218137845d3981961c83a40b1a6e9109ec1287bd5f558fe3bd460042e8 -size 22879 +oid sha256:a8b7546fe62513c6e31edc5a7204a92cea415b05bccca6d3996ade32aac9e1bb +size 22713 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c4fb594066f1..6b16d1e02e25 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6a40fdc53d9ba68bc78a26789421c77f9f4c32862429040c2be11c5ea109a8e -size 29088 +oid sha256:8c4be3bd397e60c998821784ec5b125dc2424652ab43792f7d6d01f2970f2cdb +size 29019 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b554cd36a699..819044e4825d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3fbb6003743a8496365a0dc75608883592676d147a34167ee8d043bb14650bc4 -size 27635 +oid sha256:18cd44308ab4a88a6f45544f136f24df92a3c226886564d8b78334dad812dc5f +size 27603 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 567507b205aa..870a79c269d7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:22f79581c4998fd612b91eecced703f87b80619167b6e6a77f1c0051f10ffcd9 -size 29872 +oid sha256:9e2676d5cfffef6e4f8d9293124c00df16a24ba693aa7395dc3ca3ad845436e1 +size 29934 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4a8114210dc5..ac4f808bb037 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2729c72bacfed8aba0be2bebff40255a8a4747364867a2d99f52c5106800d873 -size 28595 +oid sha256:71a48e8fb1b8bc073abfa6139a5c45c0523e5eeb32103e29cd573360f13b4503 +size 28601 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index faa2b7075865..db6dab82fa44 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85a92611f0b58019e3ac069535044020e473ffed8622bd853bd0942802f8af9c -size 29785 +oid sha256:e91ebc676ba2f2aa00b3104f97252c6b2f1666b2c53eb2926ad60677822959c0 +size 29551 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3996ed19193d..b76aba4736ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:454bebd15d209e33ddea17e5c4b26325dd641e749b876a1dda1049fc2363837f -size 26540 +oid sha256:7f9be4b2c46e1237e964f6b2b492bdb154fd0adbbdbd94d52987e6eeb6245815 +size 26567 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1058b146eb4f..536df4529fc6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:24e1ccf88b4a7adc0c9fc9291801d024058bd304108e9e2c52c1bfda259d6078 -size 30832 +oid sha256:5f21d672b000b4ff511b4ab19d0b4cd626885e15a2b87ec1f373ce80211c7fcd +size 30724 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 74efe4fd7974..be191b4ab83a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec64ec7edbddf3f642fd363172926cd4caeafbe44bbcffd66538c7f523001269 -size 27575 +oid sha256:b0ea4669163edb1b92ad242074d2851c66fcc5f052c1daf077be1c8a20824b9f +size 27536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e97b36665afd..8cbffdfb012a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:30cf42e488d8f7b66f19937afa34dae8a4e90b3c2e2af97c00ea370cca30b57e -size 25862 +oid sha256:772f5e9c2c53d365848cc671f75ad5574224e20d8fc92237d2b0d316655819cd +size 25858 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 98389654f33c..cb95f0af9400 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:125df2a008a77bad4067d6a9b878b25ad7bc2b1d94d183b9a723e73c8e256bec -size 26624 +oid sha256:89577e2e3402b8ee6ca732295b2ca8b065c413e257c158760e39668f15c20c32 +size 26781 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index c2a93b8a3c6e..aa20df196f57 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f800970f96fe07fa7d66f6e72796a85cc8f62b4a42797ae2ad47efc01f740692 -size 23337 +oid sha256:ecdd089a42f8232353af30457202f279b834448bbb5b49349d146aa3a6a53a13 +size 23239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 88bf8912ad2a..0a23dc10e921 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0aacf062c5e3da5bdd949f98448706ac40e144588caf8951c259a74cb6ba6591 -size 23767 +oid sha256:b650c4d6d057d4cc5b33fffe4ddc0e1d28cbd994afdb0743a5e8af7a76b4dc7a +size 23782 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 880455b9fca5..94c05e7e2438 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:68c4808c41de4ea8b0900275fe534b5025c0f18df72f5364e5ac01cc955dfba5 -size 24339 +oid sha256:a6bbdd8df0ae65c075590a9789df03e9304be353b0dd2d4f164e6ec61d6bbab7 +size 24147 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f67567e9a11c..1d6b7c0b228e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d71aa550e6f54fa6a8d66c9b006b463abf9329d35a57e1c226cf668836c000e6 -size 21481 +oid sha256:8696200dfd6351750f7df49747d88b8639cd5cad2ac2ebd777e9a7fe991482f0 +size 21612 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4b9d012f9bd7..4e67b693f3ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ea1444602ed1b37c0dca84242aa07c24baa7ac47f2d286c7246570bb75486124 -size 26848 +oid sha256:adfef6049f2b3d414bf2a7ac8693d314f2bfbc89f517b5872eda1cacdf7ab989 +size 26790 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index b73ae1823a92..93776b21683e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:22cd1ca15d368753ddc1d191d05e6c296a6238f6f6ce396b32c2cd42cac7833e -size 27805 +oid sha256:ed6bbf72645452a9d8b4bfa5dca5d406c93679434b45523744993ac108638319 +size 27861 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e99454687af6..fdb7bd394d29 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d31868e3640ba4f0246fb33f427f9b5bbff6075cc4388c24f3f5636fddad3eeb -size 24383 +oid sha256:ad87b1926e096bc963c7e3260b513aed04846cd6121cff27a243ff218e90a9f4 +size 24289 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e9758274226b..2a65802cab5a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d4b7c93d6f79c9c7f520c95164891b14c8f175498836f0cc45085e9df5b3476a -size 24797 +oid sha256:b47a7f106675394933ca98352e1ff74ec0ccfecd83d256b223e0ccfd0b30d930 +size 24716 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 41ed52a779e9..814b1da8be99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:86aa19c12d9efb61023303f496acf1998d9b45a6ecd46390a6d5d33730b425e2 -size 25097 +oid sha256:642dbcd2bb7809359458b2fc59ad48dc95ddb2f9cd79b5009f52cd21cd8d9be8 +size 25014 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 776c45367fe3..b1be17724bdf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a03985cbc1a0ec435ba41cf18a45b047617acbae358de4303bd391c7f4ec7266 -size 22433 +oid sha256:5d4213dfb24a0dd21b1f395126a5f4bb8db45d66e04044f0f806045d4c114f70 +size 22638 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 581516cab59d..460a18c0f328 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:78e46dac6699f875d1a491542ea46259b0a9c6d956ed5a0f624bbd92dcf29d5c -size 28806 +oid sha256:a8e66ace01221d859e88e991430a8b41536289225d39525d2085818d746a1b88 +size 28816 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 03bbaffff06d..937a0c30325d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83425a8387a974b9e5ab7c098eb849c1966ba13ae5cc0f85fcdc210afeab8e0d -size 29831 +oid sha256:9e20cf5a6d3b85a550a925c55d502f9cc96386d8d830259f0ed0485d189425ee +size 29786 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8af74bef3dcc..a6829c4c44a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:316d77df14c5c0b6adc2911324b440a5625fb4bc50a85588c0c779722fcdcbdb -size 29741 +oid sha256:4f0e9b4457ef1358e69b89224554877a2f82bdfb4b4bad482ee437258e04add9 +size 29760 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6ccf8622d0d7..352a40ce5cc8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:17572132f4a5822c71f82f183357f3dc7d695ea58ead46f59b6e8343fd421b3a -size 30967 +oid sha256:3a7180bc2b9a0a17a7e1c4e74234ce90603428154623181f0fc3261be451b365 +size 31072 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index b9f12b235ca2..ffb5dc96526f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e58223c885780aa55659afe656eb11a9d373d1b6548ffc15d1d705e5bcad1f71 -size 26857 +oid sha256:7a3729c3ab9669a9eb4d700c6f0971b81a0333e3cca8150783322d9145d8c8e8 +size 26751 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index bd5c51f4e799..f10f2026500e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:19e1aed4fa94da6c08b3828493da1dff2caff6ab281ce005866fa2472175d726 -size 23376 +oid sha256:ff61cb79ecfc997cd327e4e9088166ae195650285ffbc3cd61fe54564565247b +size 23304 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index bed56a2a3bca..5e3b2348049d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1eb72520e8174d0a699ab0c7e84e75db271dc2ed017c2ea40ef0464333f67ae -size 28146 +oid sha256:edbbbb2b844d4601301f0ef0ae57c8f1d7477e51a0fb600ceb223e0831330d4e +size 28112 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8a8d0ab07fdb..f02d9370fb7d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:30a651d0e35a61f147ec3eff68b77aad55b74e0ef9a1a453b37beee2d387c41a -size 24726 +oid sha256:55b83a8be97a60aaaec1cc0ae9a0886489d19b581ccb7b85b5d7cec9519068ff +size 24682 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index cc3a2c297f78..e3dbc9a281fa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0bc634d8394e507311edf6675738162f49a80942f60772c1ccf5c8fc149e8c00 -size 25641 +oid sha256:0f3844d3a8abad012abb5bf56d0ddd5d24c84eb5802c288d3d7449bc520780fa +size 25585 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6db878ee18fe..a123f588f3af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c4a461cbabfae0c568dcaff7859bb432e4ff438d61bbc90d22c8ac7b9f93baa0 -size 23316 +oid sha256:d49ed9d7f10075b654ee9009d827ba37bdcb024736ed8ce7ac0c65a9583e46e9 +size 23204 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index a34a0a32aae2..73e41dccd962 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bf10319c9299a85b95fd55683e9bdb14153040fbaefd7f49a7c92de2760bd45a -size 26699 +oid sha256:adecc304909c8f7ab99e7677d9cc2cfc714d442cf0adcd516c2ae482d8997127 +size 26639 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f4422570e278..7adc86206c9a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ddf44705076871d73338b3d7060d5c9a56479c47f5fec3ef9815d94d75ff1ad1 -size 24180 +oid sha256:5a10f7b0e451e1423ebe1182793d3845aa1d666bf889de16c99ec5fbd8e406d8 +size 24076 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7e15436a9ff2..3a29d5bdfe7c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a175b50364f1aafde1f306327911c28e4795aff24cfda41824983b6e9bd6939c -size 31792 +oid sha256:088ab715038c178cb6b8f89d56a4e68b1e497594bd5bdb83f159e26f384d12e4 +size 31790 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8780b8f4baaa..46ef4184d1ac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:148560f67abc7091ed1aaab38ca4a39d7046c6c3c819fab852a5fb24cd3ddcad -size 28473 +oid sha256:a867a3bc2846431e554be52d0f8c431e15ae44b146aa8d2860a7caae8c2515d6 +size 28352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 607ec31141af..70c5c82c15f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81b88cac5fddefb4d17f0dece1f46bef18c8714be62a7d4be7b33289c805a219 -size 32591 +oid sha256:61ad3c724390d9a753705ba1a676687476d263a4eb0aeea277179fac288936bb +size 32565 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 77da78a04783..965edf851fae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f28ad5084cf25e22f60239f4d259da8e178b5430190d546620b55be9a3d4d7a2 -size 29426 +oid sha256:edaa5c0d4714a6d79c5cca70d117beb8900fa13ed65429963d5f7098ece62ba4 +size 29416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0d74ea09a07d..e6b1421e829d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49eb346b769f2787bfe9bc12d201424996f853656ab5e4a26b26e192ed3da4fc -size 33002 +oid sha256:cba340d5ee728691211599af71fcb580a57d8a23caccc91927d07db641e4f0c1 +size 32694 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 24e853858db3..f6659b53a51a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc7f4af013694605d5eff0ab8ac6f79d7d56118653c7bd5410d8902dce2fe842 -size 27465 +oid sha256:321f766d1c2e1dab34f405b121f875ad7d1ba1e31739a7e8509cf02268314183 +size 27423 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8577955f1c0d..03f7c307c674 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d47e12816ab186f287313f6394795e3cbe07d9bae3d3918a46eab47538261c63 -size 33759 +oid sha256:779065384e94f1d59478bc8720194cfd75548f63fa4e2067862bb6fd9a8c564d +size 33558 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cfb4688c74bb..74a859037281 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d93f481f5cff8f251909adebcfe36a33f08600c0e5377597b93271c5e3bb809 -size 28256 +oid sha256:8d2605ca1210d0783a5a09273113376bde8bb7520c68b8c481fabd3e79c3b9cd +size 28237 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 748909f2b0c5..5fcd0a3e6ac7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28d525d4b510dfdd4a1ce2c3df5698397cdea04cdd525dc944f81687ab019484 -size 28034 +oid sha256:da04865f7f2080063a897923cc68c10e08040e5dcefd5163ff70442fba09796c +size 27892 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index dc88c7254927..b6734031aedc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:17592ac4fe9039a67c8a1921ac0fafd1f4890aba09887dcabd1fa1fdca01efe6 -size 29275 +oid sha256:a9619df33fcfb6757674ace700db784dfac64179ecf714ae3df7ff66501e9367 +size 29171 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index b5d43405f720..e3e228d7629d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e11ef78e8816f48cc7e4eba1f07a461a2f69ab390a2bf5f307a00e59de0e96a3 -size 25683 +oid sha256:8dd2d4e65959ddd00624a9347a87343a2610a8f0b0569911ac4dbb3e50cb2240 +size 25468 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fff6ab414649..0744fc0a64b5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91eebae8dcc36ac75e7f111b89b89f1e320b7d8ac7e7d5aac095d0a885378c6f -size 25986 +oid sha256:06a0c01814eb757e91cea6f444e26cd40116b3f605163ac456fe5a97cb8f54f5 +size 25998 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 034fea254c55..1e4d4a28d7ff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42e858d19f2210c5f11e7e650d83ab54d9c20767825b89b7b4ca68000091e1b6 -size 25266 +oid sha256:1c6fffaeeb66e921f009802286cfcd4e9e6a84a5572b72a2d03ad915fc6b5b93 +size 25111 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9697ecd096b0..9106b36994af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9f5333f726c7e963bfd12ef4c55e25a4b971ad1256d1a1fab56c012791f675c -size 22687 +oid sha256:ea68e23bb38ed9c111b0a7213e9991db9998e854ef9c12092ea3a4b060aa8357 +size 22770 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4dbcf24434c8..231bd89f1cc8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d8770e206d48546c41ec4c9f4c5da1352993009e9635c9e7602ae48e159c1019 -size 29079 +oid sha256:65a3b1a604598cba306899ea7490529741a26edd7a5c514bcdda5851e96a289b +size 28898 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index f4ad9eab9cc9..85153097d6c9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53050ea9ef07ab6c0211aeba9d030fe301ff21f1910573775195bb5af367d7e5 -size 30035 +oid sha256:cda8c409c9ede7fe56813f0e817fba4301b802d12d87908ae832103b7159990f +size 30089 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 9e0436b868f9..40feab409979 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3f483707a861efa79acdc162077bd10e885feb317cf9ad6c9c4bcdd3a12c2eb1 -size 26571 +oid sha256:975ea80ada71b0e0eb47fe2cb7dce07f2a0ca86ccdf10fcf54988e742931089b +size 26388 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b6fee3b71368..63478aec64d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b23b633047206726e9b4cf35a001bd284e03341b12829b07a068ac572278591a -size 26822 +oid sha256:2515ffbde5bf23c9e619ee473ec5ba6b85550c43a30efd31b8cc2766aeab770b +size 26724 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4fd48b122415..b596b23f70da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fcb831223114fe3a8ad912b052db2980ed5621317574a73c96a3687867c4cb88 -size 26038 +oid sha256:5a9ce6341700ed9321675ec7275b9b0f25dba1a1bf63a5dd89018bfea03e1844 +size 25853 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 151289411741..d5d177f846d3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79f5e52542b5420080f32dc1e047cddbf7f20f5fbc5f31cfe41bff43154ea504 -size 23703 +oid sha256:d13c81d5ad810977b4ea6f287fc5268a7a5ab9cb27e5d93248548bc1f5be8200 +size 23854 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 5ca411371fcf..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:788502a43a7cd2ca0e1341743131d636d1b8defb5c16e4843ec360d654a9c7f2 -size 37706 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index b2c6f59b59a0..05a0985bc203 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1985e07064074068d351a49d01f28710f4fb2687d91a1ef1e8bfd0bc54dd4239 -size 22515 +oid sha256:3056b9ab49e1e2d6d05ac648d02ea656b5a448b5ad7cb0cddec873767fd310e1 +size 22392 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index dfaf05064002..5d3345dd819a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cef714ce600ee035fbdb92d7f1ccafcb2bda106b1165b7ed343b01530f70764c -size 30085 +oid sha256:211f31209e604dd25c2db448406ec22f8326b5034193dbb47512546101326f59 +size 29912 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index d2b0fe901e7c..33fffd0b3ec7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d2860b2b67608911ba685a680e47c5c0aa9a82c98e56cd712adfacf8060f6b94 -size 23027 +oid sha256:fe7e5d7c705d5c1b747a866ec0dd80eab9a85970da918f566780ca6612811d9a +size 22909 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 2b02db49d69b..424373326afc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:784fb5a1e827705d64b0e11cfc4e8b6b067dd957726d755817841e96c07fd21b -size 39437 +oid sha256:76b1d1257a3648f0b1c869dd47334cbd1dd95133e315367bf6902dc7bd0c6ede +size 32074 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 64428f0bd43f..3f172dbd2776 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99bd9aca39a79ec6dac2b6666449a3bafd3f34ed1cd46a6f37782886865532b3 -size 30898 +oid sha256:cad5b635b11c89a3818452c82d0129e58c97c41978c9ca50ed89c2f6c5f8928d +size 24728 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index c8630c9d5525..a1679f2fa622 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a426334e2870ceb4a5386747d6fbf805f8fbf4ae78dfbd23e09dd0a73c236d3f -size 31169 +oid sha256:c803f07192efd027c5860314ebdde3830641d021d9dfa00081efe1e07a422b3b +size 31107 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 82c4ef8ee07a..f4819a5a2fb9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:592a3774890fe02c3ddbed0e53ca1c57d70ff2e03911bc9615f2d2d97e153ad7 -size 23921 +oid sha256:e3645697c9504970b4349b782eeaf5c158cd58062a1c3bde3a1f510831cd6820 +size 23796 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index a3f6938fab89..115a44b1363a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f4bd6b2059d6ce3399c6094b6fd0e380222954c767ce8ccbbf47923e254d6be -size 19600 +oid sha256:f74fc010261b36d0ee6c6b80cfac003a512a1ea0ec1fada889bf7ed01c697298 +size 19518 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9432ce0abc10..8206a0a81ab0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3909ec3f2de2fd856ba55d535ea36fb121d582ec97e3633ee893c69c584e81d9 -size 23520 +oid sha256:60e183eda882ae2298018731e4a4da7cb1c5c8302e1d88c6b455cb3ef852e70a +size 23410 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 0ae79ffc05d5..e1ee05c88da1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d87abc17d1960025f07ab3fcdc0b4609b4bcd91b7b06425fdf4f9af22188981 -size 30902 +oid sha256:0e5a0094ba065ccd084cd9bef15a0560112af6185cb660bd388b753467985d64 +size 30800 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 96065604eb27..643dbd2d7fa1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c5699bfce7b88caf999d3fe3beb01e4fc8c7b230680a90f2c7a294f37a7b8a3 -size 24172 +oid sha256:d1b66108264be00392501bd00dca331006430be02abb439ee81a23af1618bed9 +size 24081 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index ba0f999b6ae0..2b7be15814e3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6c0455528006301cfe5ef8a7b27a562e1ed1c9ab4987df4767d8bdcd002fcbf4 -size 39732 +oid sha256:a4c27107e28376f0ed5b6b2ba313b1522ee7b9df8638578e6d48e7952aa120d2 +size 32239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 29bd6b25f114..a7284daf49c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98f89b45403ceb1e3483ec018cb7b284cf655844c9c9e2b977daecec24acc4e1 -size 31961 +oid sha256:178824a17070fd8bde2c66e11897eeb8f04b6a35f48f673610e8d2bdc470937b +size 25508 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index f96a0eb0178d..afefeda57b83 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1b8d23a36c730a2105517dea73195cbe2fe91c855e53715dde5b6fa85b6072e7 -size 31704 +oid sha256:3b98ad64a4fee3e9de0b090915034d82058bd5e4d554ab1fc8be4257dc63dbb4 +size 31641 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 2a47112e32c6..2c3ce22aaaaf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3040190dcb0228eb2977558d2769b626395fe8288603065aade16aa4d6e5d196 -size 24765 +oid sha256:4842c4c640a53f70797df0e85a4ac11928ea8f38ec718dc95bd206b21b1e2a05 +size 24597 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index df78f75c3b57..cf02973e4997 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10d3911d5b1a4f6c1727e90eb4d485e487ee0ece8ebb787d7d1e49b09459ea1d -size 20511 +oid sha256:4249475f4a47b3a2534cceab225016757485e7e82978622f872726b945e4ecc4 +size 20404 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0e103b365235..a4776ab28eca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a9114f9f9fe0b34e116fc54d1e395886a782d485250b8dbf6a0ba63cf2bd9f3d -size 21665 +oid sha256:13bd95dbee16778f29b7159e1f2cfc86fba1af84b26eb3a715dd1b475d221c92 +size 21514 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 32a89a8bb10a..50f61fc9199a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b70c7622ff48008440c477510fa168793e778ce04ec37dc41ca2b155c11f2a1b -size 19116 +oid sha256:48fa129ec6ce96be3946b25941e3e59d17d55f6c1ae17caf6485c60229fd4a3f +size 18985 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ac9b67080d2b..ad2f742be534 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f165c751f838495d822c76981d08c30a72947fcd0748e35a68806a6aaebd7cb3 -size 22162 +oid sha256:c19bc824eb539c82de2fd40e17eda23daa131d60725b1512dc2dc1f4404358fd +size 22122 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index cac3eb4ab5de..561cc005a251 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb3c991454c5542d70fe2aceb08e6327124f36075631a4ba42240ca6f5611da9 -size 19545 +oid sha256:0f5bb44974cf0c21b3faa1cc451d4af82d1c294202a932feace005158a0f7a80 +size 19422 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3a6057be679c..9b9cad061865 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b2623662032b455597d12afeea683fbf1f29247d7070123698052ba71cdd7f36 -size 21989 +oid sha256:e8ae430777a1c35c5f3b0ab384e7af38f8277379069b3461651106c746fa3b24 +size 21861 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index ff23994b14b7..c9d0614d3ac2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cc0197305e82278ec3e487ab3ec3aa7b7eea25d6a0d64df39142af0d4b58be9b -size 19019 +oid sha256:bcbf39643431e785c1bbf6fb34ee6ff1ff1666510839a2678d7cd33b9c16fb98 +size 18895 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d270fe4eeb3b..db6f413683ca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b5452e4b245cd867c81ea56fa9fa8741491cd3fb1c64166bd4f1f1c510177e57 -size 22362 +oid sha256:32a86ecfe920a2f5cd05513f0ef771184d0b20e87e982a1e0536ed1591d330c9 +size 22329 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4d9e4e3186f7..55fd3049c77b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bfa0f9340aa26b610b1f547c1076a1eb96b59f8726ef153f0b8e8557cf323888 -size 19524 +oid sha256:0a772262f2e3d88beec4f04da9d74aec8b3a6f29e65a09d8730eba8d91c5f7db +size 19419 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index e5963f0d61d1..f140395f1bc2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff14852fe3d4fd686dac3bd0a2a029384e781c083bd61b6ebba64e1c2c6fe47f -size 23819 +oid sha256:c628028b53c8fbbb5f4d36d7a3b61683342cff1a9962d6e3be5b77c904b1afef +size 23588 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 18a1937f0cd8..4f8e2108e10e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f53f233add08d0874aab285bb38bdf6c82e45d6bc9c23b5be4ceafde552a1ec -size 21334 +oid sha256:9b753a56dbf7986abe9fec061376837f94dce36c6dc3887fd0fa220d98ff9ee4 +size 21110 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0dd55dcddf0b..a6d770f89fb6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9cfe27c30ea63a8bfc98d3fe74ed7137b86f3f356c3bb2eaab8f25290a4a752 -size 24304 +oid sha256:ee977f626135e5ff6e22f1acbb9bc2e1723898ba40ade089e87e8e5a15904350 +size 24212 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 54dda2275a11..0830dfbf9cb1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8977b3f7220032110b243f7020d897de997c2fd070140901e365441d0fd1cdfa -size 22026 +oid sha256:d4e77f85fc4a39c6ac15fbbbf1b4d8eda4e822c551a85c0ae8fa55716b8607c2 +size 21823 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c228f2465e25..9c94b76500eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33dbd64d13d3163c1b7d3142735c16f1a1dd4112573b3615448d27c58bd152f9 -size 29132 +oid sha256:2ff420746c227518abc39414da692cad6fdfe0b69375696866d9fdefc6b4917e +size 29042 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 64fb42b2dc62..61a70aee5117 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d294e56cbb4ebd57ea899ba513a8a4c494c1b8bd37c10ecdc41034d27b4f127 -size 26098 +oid sha256:e2035ccb70897dc09e24e42c78f8df3d67c2ccfd8b4eeb797ea3d8e9fe4fbf57 +size 26113 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1710eca2ee04..9c0aa8b86443 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ed4450e823d23ef183a03fed74f26d0da35fea68be8276682b8606f217a91e1 -size 29748 +oid sha256:f65d1a1b87f6bc632506ea8034c221bdfb0c59f165b71e5bd3c8d1ffe3d60d07 +size 29666 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 34a3f50d2100..93f996be128c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f63e2a493b2c5c6226eb3a9bbccc478b3257a178741bb99be52800832fa2e872 -size 26963 +oid sha256:1f77c1877b54ec7990d99dffeb81240fbe02f7b2b8e728fa095fb1c101d1da6f +size 26974 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8d9eb0b87f39..98a0a23caa4e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6061802792d42f18ac872ba53351bdee4d73583643bb76c161c3f92d7317236a -size 27033 +oid sha256:cab24748c38076cc0ecf9211297038b0c97313c5a17359674946664d916de6a6 +size 27085 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a9f6295d5681..9d61f649026f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a428d6ba68bf4e57001da50e9acd89cc7cdf7c931d3ee4be746b97fd127f1c44 -size 25462 +oid sha256:8daff2cea1a00e159b52010348e8a99f5dfe98762dda6ce0e3d928695e70c5c7 +size 25584 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 20f42a939233..f418276ed7bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bcfc7ba3f01a1cd255e8d7257f05613b5f4f424b2d070873134c5cf47fdf8846 -size 28233 +oid sha256:ce65a4b4c0f0b18fe137f086f00a80745cc65a870ea9d43914f00644f8d9d97e +size 28261 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 939d0ac8430a..bf196af63c36 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4797089a32a8720eae8df72d8950f5343b1c5de30745bdd0fd3ffe0d61ba0b5a -size 26453 +oid sha256:844c1734f014189822a56a51e9e8e8628724da0cb6b0f79218f90c69e6c01e96 +size 26381 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index d4a081f777a8..f15e28a40239 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10f47e15566f6aab7a0f791698ed3558b8c3bda6ef33ed2003becff667e87dcc -size 25409 +oid sha256:a592914e873a78b61b0f9959410f74e12988a2decd947a671ec628d583fd7807 +size 25252 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 7002aa3e5141..1b3dd3071751 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:677e8af2f226c8cac23fe71f599f997541a8b16e977b290b4bea0b16a83d7a1a -size 25212 +oid sha256:a4aae7ab42c1f87658c3bddaaf53d76db6b5a07299fb176e6c6e9e486c221ffe +size 25135 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index e4142f8bb5ba..76dd5d64bc45 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:25591adf7460bd06a268abc2835b7108ae4a9767d3110997f77e623d38e81f04 -size 22766 +oid sha256:3e84917b4a50db0200e14a5fc5723ebb41663c4b3b73befab8de2745ccace998 +size 22640 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b97f0fdae29b..94986b801339 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a8b6bed98edb793d0a69bfc0b8257a61df1dbb7ef265eab08eace3372eb69ac -size 22214 +oid sha256:795899280db114d340a9aef3ece7efd605e69bd962ee3ba24b6cb9ca3aebbd47 +size 22280 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 6626487d8ec9..7f79f05af863 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c63441b331a98f2a55bc6dfb8a0fe60c0b1ed13359c9d678d27f0417abec8e9 -size 23654 +oid sha256:b72fe4567a24bba629298f08c504b6615516217c683bc55addad5de59664a3ef +size 23700 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 51f88bbabbc2..44a74836e47e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8b8c8e546d7124e334de90db4b5a18183bcfeadd2abda0b7aa9a7371472fdaa -size 19990 +oid sha256:abca15b8eab42afee2344d06fabd242808157c0fce303e3152a49e992b37a720 +size 19931 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 8b5e203a6bbf..70c0eb29ef28 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8817388bd42c3c900f2f118490d207ee0ed0b7499d8b3c05cb7632d85d128e0f -size 25991 +oid sha256:d64237f590987e1be25fa64061a24c6ec925ec7ee30c73073a03ecdb2b375f9d +size 25867 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 35819d32fa41..81cffdd72d18 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:18b4c062816bd4e3438fb2593c4f6e509ec6ad38a01261e3113a9406c26a5d2d -size 25657 +oid sha256:dfdc1ee238170317946cbf8c845d7de1ef4e8f5fc64ae8cff8a51c647bbf87a5 +size 25643 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d843c1f3fee1..ffb5c432cea5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0db29602dd10e4dec10c528a18735478f8465e11b1c23c1d28a526e536fd4012 -size 23083 +oid sha256:ee27feb55dca3208bcccc6146081a50583c34dd74455972381f06e9afefbfb40 +size 22900 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 46698ea03c61..8194ef41854c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:adb574eccab9ca064b463ce5559822adbaab2d03b00464a0d7ea7e265ad3fa24 -size 22917 +oid sha256:dd12ef73eeb08dccc3f6b9e749b22119073a82bf72ec70a8d6d5fc90850600f1 +size 22774 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 096387982e8b..c28ca552aece 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cdd0418ce993d444b83db43c85235da2c4449e64103bf1d3b4a1babd14be765d -size 24870 +oid sha256:3fdfa435b165aa4c4c3efaf9524f8f1871a22906b4ebef896aa0c00b6399f278 +size 24919 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f7a7f2203c68..593428a7632b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80e0f9f038d103e3b6ddeb00a1b5695a3aa2197c4387105e1a767840b7b64aa3 -size 20978 +oid sha256:a9e85b6a2b0410971ea0d23a2a09753699422bbca94905ab13bb215234dd83c4 +size 20843 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index a5abf7f4b958..c967d65c5dc7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:445ea82c4fa66a4889638afe513ae7f77c55121c54537ce8ba8fb766ad5b231d -size 25333 +oid sha256:8bbad4e8c77c346d0dc43d464cc31653e7985af97a426758070ede757d0ebc6f +size 25279 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 668d91b0d48d..e4bfe34a217a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54278a35373e9094719a0a125eb8b16da388c6ca5a76ce194881d6980ed48275 -size 22541 +oid sha256:da1ef5260ad53052475db5d3f943b0d1537fdda13ff5c4b1f8244944e833e136 +size 22416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ef2fdc514dfd..0814ed79ff46 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f15ddf84c82464fc8983ccc0a5e7a42742a88463a12df3f3efeb45e81a21e329 -size 26061 +oid sha256:e5ff9d9cd92fc0e9c2bc5cf29f5a8ff5b69a13ca8069829f01e149ee60b68a69 +size 25849 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0c43ed6cd76e..6a1492463b83 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc6f0f493b8833a963288cedcfb7d1cb21a8ea68f738cd42b418efe95ac11585 -size 22833 +oid sha256:3f400d8ef264466d042c0db8dfa2abfdf341bb523bae675997dac6875cb73208 +size 22682 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 952ff3161ab3..0a9007de4fe4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:68d78696a2e53a623d58b3940d676833ed91d6edb115e333ebdaf9d7dcacc612 -size 31667 +oid sha256:0806264895a63ee4de4728f2c8e6dcd93cccb371db4e02164d4721534351fcf0 +size 31516 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ac243ea244d7..69e8d1b7d68a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:68a411e895dd8103a3686a6d9c8e343324b470845e45fe8e0450c524e86dd870 -size 27655 +oid sha256:2b843e31be79ac0e8789eff7a22de4f77ef3af391757986850b12df142e808aa +size 27580 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0f6d18e546aa..753156e70331 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28a19a5872ca086b212b7e4a75f606cd400f4f6b0cf464625c00d49eee9a8944 -size 32289 +oid sha256:ad1d0ed865179fc96a9023b66fcc98fa8441e7dc2dc228b1658633550613732b +size 32091 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 474cd9e95d26..671a390f456e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c4b5f51247ed4fd5813104b1bcaef405c33cdfaacce580acd9b890dc270e06e3 -size 28430 +oid sha256:4258958ca11cf44e7371e59da726aa696f33bdd738691b48ebafc58e39386d6a +size 28375 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0867f1f11e80..a4dc41efd699 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:483aa2c621ea5874f5a8a11f13a39b8aae1c47f0ddbb02f470d95209ce26d70b -size 29782 +oid sha256:42a8273bd42838a94b5f3c6e8a38e87de2ad0f398c024bef7e1c2d119594b0e6 +size 29742 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9d9e1c96e1ac..77bdb28d62d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb18975c17586ea885b73d178b7b950c32c0575d82933cc905dd3fa93400d9cb -size 27105 +oid sha256:43da495b50e9b91c3650b45424f46e21440d906e27e4fce0c1dcd8d9f8a7d0e4 +size 27199 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e0c3ffb0ddcc..95b650c1dc51 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6afd257d25e544f672ece45a58f0b87e836aab86fee0a0bf583ece1ccac0d3e6 -size 30680 +oid sha256:550b1aa97e3031cb21c0e5f25ba11b24243f1e4e748b86be42951138ceb662b8 +size 30624 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3c93c84f78f1..c5216da87d62 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2fb6c346f3264a16d57178101b2650512b0b5b080a1d7765bfa0a0de0cb2bcc0 -size 28043 +oid sha256:95658024e06424d8e5f379516e5f4de364188f902276d8730db74cfbb26617bf +size 27979 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b26512694e60..4eac81c898f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f97e7ee8854af2715197fc9982df108d6c8f5be30f999d5f9e9b2adb3bec191c -size 27486 +oid sha256:8234173c0334480178f50578b70d193ca3acf75a11bad1c4e11ba047e39893ce +size 27263 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 974ccc3ae2a3..1ef3b727b313 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a58c871cc8ada481a803db94e852240f61187876ab850e0c3d13b604b77f9941 -size 27652 +oid sha256:aed9a849b01523d1b10aaea95907d18f6fab19e8eee46e14c6e754364fea1fba +size 27582 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 593c5ecf1167..f4821065b3f9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2d3063753626228d74d19f4130f1eaec8d8ae1aa42ae1717ac354cfa79a1852e -size 25066 +oid sha256:3eedf4351f790c934ade7af97a91a1453cce84a3a0b1ce519802b747ed90bb73 +size 24873 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4c24db9b600c..536dc2877b9d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62db051ec7e9e701189cf30447fa3e99623f40f18a0d86b514e96af46257c558 -size 24696 +oid sha256:ff4a6de952ac0731dd73c177c5ba1843c10a553e92e52a243b147d11014f6540 +size 24521 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 00499af05bd9..7b846fd3cbed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0df310c8c10e0e563e4f9633b7155f5989a0c31973af1aeacb4478200dfb85d1 -size 24831 +oid sha256:8c2de20faee58f06999588daf258d9f26da709994fd974c2d6fe5f2e6ba5e8eb +size 24847 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 10debf56493c..52496039a383 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:484e84e05774083e532167b72680780177bddc9db41b397637330aea5f3af87e -size 21384 +oid sha256:8e7c80e2c5ba696a86860c4c0c4bb1ab7d399781b2cd18212c9ca807bc1c51ac +size 21314 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e51e6e27252c..c6ece30e089d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b05374cdda07f21b9d68fb1c0971a1d870f1b8c1e6081993c993fbd80052be01 -size 28058 +oid sha256:43a5192c90f389046156347b8a56045efa5f79166f47df2dbb45499c0c099b47 +size 28021 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index efd39e6e2357..64d39f8decf3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:427a94fb1a5e39049baaa8f1cb942a296a9c9a7d385e8f5d36faae2c056661cf -size 27972 +oid sha256:46f72ede8fe2b23193b07046d20a5a1864c959a75ee4cb936789b4566d6ced8d +size 28019 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2cf093f4043c..c6f05e21cbc9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36289835831cc98828edc35229f23b3b3b5eebbbb56a637afddb56acebf2dab8 -size 25531 +oid sha256:1415f378be4920c39df4c771b21aee065a991e040447f093c9b4626b565330bf +size 25352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ba6ec17f75a4..a0f87e5a7d9d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:23c24968f056a8f8594576d13f073f74a07acf093eb6b1a0f749e4da8ba2bff5 -size 25227 +oid sha256:d0d3de55b53dda0a1fdc7d61372df3100c1677189bfd8d4dd30967aa34b0bdab +size 25125 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9213e765c457..38cf09cb59ef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b86ec71829589b39ec8862ef45de385700e1c34420144e091b85d0aeca7725d -size 25910 +oid sha256:9348e10c93ddd300bca9597f40f8073e8a0bbe0b7ec7bbf61d5d325366f390ae +size 25917 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5cb5975baf0c..1dd9334416eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4503aae64832ad0844c8d0a7588fa8096b463d5a25d5632ed3eefbb681e64a69 -size 22149 +oid sha256:803c1a6e91cbe1deb37caca0aa330e65c9f4d0c505da01feaee10b84124e36cc +size 22124 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5061e0e78f74..6b7e7f97301c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1fa426c48404d89a8a184d6e305718667d77d3acf46fbf573f173302e0a52440 -size 22964 +oid sha256:97d64eb2dfa69f0b9b659305202c7f939fa55f4f493a0fd5afe9bee2657244e9 +size 22787 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1b5c96c3d319..abf91c17fec6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0bfea78f9fd3792180c9cd80f8bdd9ad787680d46e2449d572046f18ba956ae6 -size 20038 +oid sha256:6aa73cd334ef5cbd29f27156f8a902054cbeb9912ec07d27e2deb0e0c9613220 +size 19951 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 450f8b723398..161387b4c1a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57ac50e3e75670d0be43acb55a56e4ad812f75e8a0e73dc1e2f7c01944ece475 -size 23356 +oid sha256:eb9172a2e8560b060fdb1d220d156dc3af61fd13210853395cbd0d3a42e7043d +size 23251 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 98efcf3dad36..4c7dad96843c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9aa8a367635e7d5e9ec746a2b6c2403c52e6c5c540f8edd17b4c580129c046ea -size 20260 +oid sha256:eae80b4b365e5f8dd6e971e638ec843b65ccd773946e79cb833aa933ad025c94 +size 20120 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 158843c60a54..481cf3fca867 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b80989370c90922b7c4b96c6f5b63a12bbed080d97ae232282f1385ef3ff298e -size 22906 +oid sha256:00d417f5f06ad02ac2eb8423f26dc8b16a70e6a39ba46ec90c69255aff9eb5c3 +size 22752 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index fd6af3973d1e..ce54aa937343 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:363ced6f450c8fcee4db2f41cc16b75cd0b615b042f43df924fc4a30ef81238e -size 20382 +oid sha256:31f5bb57e6b483eac1e82e84e230e040b08df8824e494ab8dd72614aecacce3c +size 20264 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 5733fc94b033..817caca29ef3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d7c5a7a782eadbfb52bc33474a32eab7bd5cb83b9835c3571d71703a3bdbd5e1 -size 23470 +oid sha256:75e1e5c6f2a0ca68b58cdc3706e79854ea9d832e1852d147ad3e8c46b6f6923f +size 23385 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 512cd17a2b68..e451e4f1efd6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b6282c976e4c128e45b9b915ec66d1905c888052eb4097df946d739ebc89e35e -size 21005 +oid sha256:ef231ff28d8e8585d1e923978b6f8240ca6d75b532c7f08996d784869ef3a2f6 +size 20884 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 04de990bc67d..12c401fc5733 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d667deef3e1c085edd72a18523ef031fcd3113a4e4d5a184e984757d0e6d5efd -size 25223 +oid sha256:b0bc986830e4716184aaeebfc9efe6f8d53daf3adf31101b90d55166abab0131 +size 25175 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cc8f8cf4918f..f329a854127c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b588f4621e31e03aff07541fde9bbf845b1b6e71335b65cc6bdca2423e3dd44 -size 23700 +oid sha256:89619bf5ec553a0175e774a02b3efd9ee9e6e9099f56bd6a4e65d55e87207980 +size 23719 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 24748ba89993..40976db4614c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f351e273969e422927f6746365af0cf6c7052643257bd8224bef286b6a982f3 -size 25629 +oid sha256:52dec2cef45e83f5fd4ef0f2867e7c65f80a5ab659b95a2ce48652d291fbf13b +size 25499 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 505917be42a7..b14570be9d15 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a69faf58f3c9023d5fabb37d75b3eaf134af0af482a8aa86f4f9f5b314fee0a2 -size 24296 +oid sha256:12b6197e0da3419cab6d39504497222b538a8e200eb6cacfb188f69123944dbf +size 24200 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index f6d32e302663..969d644c2ef8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:914211d132b4a7f29aed24fd770f6c703477057289c091a6e61801485940d25e -size 26784 +oid sha256:31513a0a976e7313f0cce7ba99360f9359ba0c1a5361a7c53aefae642e7bc2c4 +size 26724 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 473fbce11eea..901b98d57098 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ecdd20ee2100770029e48cc651bc336a052663969593cdc925d4fa2080cb739a -size 26614 +oid sha256:92cd7d3a9256c4d6c8088f319824d4571192c2cd70cf405068703aeca6cd5184 +size 26550 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 3a65259827b6..c214020fd96e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e339b90465d9477b42cdd4ff5840ad0c20d5dc2ea2207ab0da8f08621330755 -size 24247 +oid sha256:052556235384663d05d8e1ae712b1edef2347a6681d17792b1297330ea9537c3 +size 24122 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index dbe2de273c49..cf702d15d793 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c982dcc4f737d147f6c9e2cd44e0a03145c4a2e59a8cfbd4310d47d61348d0b2 -size 23717 +oid sha256:e06f49cdab42d4e4825b112e79d5206d5257b15aee4e4f3978438d5007b770dc +size 23741 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index c581f3b4ec7b..b0c73cdb19f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5bd1a339c5b47a1863b75d57604eafca97de934f84b0952680ce2062b5044cbe -size 25033 +oid sha256:c47b8aafe7889a0549f94dd398999726a9501532bb992f06bbbde133fccfa814 +size 25079 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index fcb9a6000525..ad51b9651cad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:656aabed46e981cdf687a012c57564f0a6b79fbf6d8e63476d23ca838486b381 -size 21570 +oid sha256:687afb82a1a21b4a42cd194356f50f0162cf077eaa242d850dbb17f697ded8f4 +size 21504 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 179843f322b8..4f165aa0e255 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6a5b835778456764ec43373fba9135833de4fadf8ef38bfd3a16c784fde0c5af -size 27564 +oid sha256:772b6bb78ddd44fc72d10e6d127470bf938a7d62bf9acce1828aae02f6c6efc7 +size 27481 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 198134b9dc5e..c935578907f4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a123559d582f24ed506387140c5bcf66cb2047a5521b71e8a21fd255e685269 -size 27254 +oid sha256:9e60ad453d2e1d5ff4e1c51f1b4e7ca08e2b881453fd6ed3da6656edc5393a09 +size 27078 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0b680fbe2379..953d59c5aa04 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e62df40e7c4f8209465b4ca9f67f653bc8ca6e019b1e15288fcfe28dc17b513e -size 24603 +oid sha256:b5d7f81b9b130643c6806d798ebc72a81500a65838a0510f1d69034f367e22cd +size 24415 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3390b52fdc8c..c9a9a5ebd10b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:616fa0b07d9a05b87dfc4e99bb50b1a0a11b77b4ef769e4543198226c8bb025f -size 24303 +oid sha256:afd5b6ffeb1a12d9b908a1205ca3d26bd4896774c8a653d67670a52ea9f2c6fb +size 24154 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4c47c2b575a7..3e7c4e36dd86 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d1d807e645221e95ef721a086c24bb140094ae501149cfd68f9fbe29788e8aed -size 26109 +oid sha256:9c5ebfa6755c475022bc9ad8cfe5e1333a12685ae257152e1da72e58cf7db07c +size 25971 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7b463eabfeaf..0bc09ed8b18c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:590eb06fb2ebb577c652fce8cbfda19d4036794a3ca8df1cf80261ca6d24cf91 -size 22316 +oid sha256:0973b70dfe836434f7fe8dcaa9b32e752cfb32bf7b2e3011be964d91072cd1a1 +size 22176 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7da12065546e..b3ad0b92f2ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fe983e1a81692d9d45ad35e137379f97cc9ae8c04359cbdc75f18ecb7d88e00b -size 27088 +oid sha256:be3f092f15b5bee27cd6ea8b22312e7824e770801547bf7dfe0c8a48fea26615 +size 27022 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 5170a6103108..397df01bafad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:41289ec5ba934222f9618eb293b49fa360097babde829298d8b8906a27a8df03 -size 24413 +oid sha256:0109751c889d812b1870b7fd88d9764a6622e78697fea3f844ef7966693d1fb7 +size 24310 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index bea718ebf55c..ee26c991bde0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee3629398f0ffbafc8355baabd4063e8c04f320a436c01cf9f25ee124284a200 -size 27528 +oid sha256:23e66ad14748e63055bc547eda5f9f0558b885ef3b159c5ab21be3861a78d98d +size 27325 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f1ed900e17d6..db81549fe6f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d90089653fcad76756bf4b02efec90597e4e5c4054320ba2cbe86e62faff0213 -size 24866 +oid sha256:b83b085add98ec2b9b6fa9e803679993a7f80da97cc3757c92bd654f87cad827 +size 24745 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 04dd99b08a4a..762e46e91a30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a955957f2133f99007c1743feb210f2c997ce553c1dc1dada082d3418444329f -size 27881 +oid sha256:b16d5cdd7f5a6bbe81369f73bddf331275f4379cf5325dba54def6828c13a62a +size 27731 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 257e400144aa..33b40dbb638b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:60df5c2e3ebc8f25ec178d94ea5ba565893df6c593d72002a389e305e0978a1e -size 25025 +oid sha256:85c0d777fd0e888e950a81d834705e7faecd36229e728e682f1b24370b7a8167 +size 25074 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0443efc15bb4..b2c325297ff0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:705a2d96d2bd849354b526afa9a1e20dc2532c6f0f69a0e034adc706ec607792 -size 28231 +oid sha256:7b7e6a6482013a59e5040f20433276397fef140369147c16f024c5b74e71b6ac +size 28056 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c2ca4544ffca..d84b1803b0a9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:935ab39e2e4e52768a4d0bd6a158a4a97dd9be8238a3803e2c67cfed49cc7908 -size 25773 +oid sha256:506b813d59911a70bd5ab708953c8cfb2f34deb9ad2bfdf206f49c8e2f97b0a4 +size 25734 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 279fd73de7f9..d2af59dd258b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9f82986a1ca1f1a71a50f3b2d76981395eb974f81d2f7041a179cd8c8c7b4b5c -size 28924 +oid sha256:dc162fb2dbfe25d8c85fd72d9763c1739166b1f21fa1b66036775081b616ffdd +size 28906 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7ac76e308dc6..c5ce190ff14c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee5de4421323885f63b5e10c8bb7240543bbeef24d3273ab4525a4db6f11d7c6 -size 26166 +oid sha256:632a5e80f77da288993206e97d0c7f5b72c9527bfd38a21bdee11e316fb3bdf3 +size 26023 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index ed397a10e7f3..442978dbd54d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eaa4b9863f7eaf60d9f98c03ee66cda22a820c9db15d9f3c3224de25a6be0bc2 -size 26201 +oid sha256:bce86ddab2d7aef5346f2f490c555ebde79dda55d4ce4bbb85ffac2dec5f7b42 +size 26169 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 784a2acd4223..975199f5ac97 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db09831b55ec11bfbfef9f7cb5460191c7f1888815466402ecdfd42af88b111f -size 22644 +oid sha256:9bd07fcbab36bc96ca54385022798ba2754aa51436ec4ba6536ceb32a8bb6330 +size 22670 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 99ea0a3faaa8..3973bedbba16 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d9ea0df6fdebf09a6aa6761d05cfbcc8ee86b92e4d1a8322e60d5ebc3cbc178 -size 29232 +oid sha256:4515a9f7666fb08cc9f3c0e8992e2e243f4f3123eff55a87edb5a4127a9d0e23 +size 29221 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a0034ef78f12..c21418534a33 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a11bbba6555973268a802cd4e47477228129341685143f6af2e6e46743b14528 -size 26494 +oid sha256:9537c8a26885974cb557e9a3c8163b8f9bc6c4988f6d0558fcbc3a50c4372afb +size 26399 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4aaa07c3eebb..707ae95a3812 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e2e0e8d4ea6f1d1136336393bf92557740eaec30550b6aad731ebaa1312bb906 -size 27421 +oid sha256:03025cb675481fb6bb32258738047f07033fc22a07522b4b9b47bfadd48b825a +size 27395 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cf9524ac3c1c..d0be47669363 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:def704b387a31109c6ffc52492f51cdf399b53bddb07cd2d3642387593432709 -size 23422 +oid sha256:73e0b32298267f1af2c6f3283b88ee1a82e5bf8e232fc5195df965ba6da6a4d9 +size 23349 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 16ca5de9e790..6f6b530b815c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d253999c5e95ba42d02f7783f73f1154819d90468fc8bd6799f02001985f0409 -size 23266 +oid sha256:5d7f6d2bd1456e05da840ae3e19d12db0eb1badd09c03956e1992cfabbfa6336 +size 23161 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 4e75d3027a2f..e8b8db2515bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36a048ffbae68af677b2d81925b35ddf3ed27578828189ee8d28f453b4b09afb -size 18878 +oid sha256:fb1e87c00880950498fc8f54e0b94e8281ef6c2b622d460e85f7c5c413bc685c +size 18766 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 919ecd9e85af..c222b14fac88 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:196dab20e7016accd185d306dad89ee9c5187b6b2a6618b7a55d629e749ede3d -size 23911 +oid sha256:9e569fdcc3188e944545e9556b352b42e9b5c1a0e74518841cf78ba826e3e233 +size 23881 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 89e317f223ed..68e46122a288 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:44de2d4189ef79ac46d86fd4edb6db9f84f840bcb11ae8e3f9313046ff169b05 -size 19818 +oid sha256:f0ec2a6c42f7e1b9080c59a00a788b12955c12b9580b429fd65008973a1462e2 +size 19689 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0c04e99024f7..bb8b449b2fbd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5fc153931a0e6d75a574c247c03d76999d89dbac6c107b952a8371c34ff0a89a -size 23121 +oid sha256:954b83c0ad200528c79e4710fa8521afc4f06f5d77c66f361a429c1cdd0934ac +size 23041 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 6523a0e78de2..aeb3dec7f0b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93b1a7916bfe191932bec2c65fe4d76f0005b1a3dc6b332e46fc5e387655293a -size 28583 +oid sha256:1c6e3632c0087f171a8929781cacbd18f092279fdbde96cfbc3f041b498ebea3 +size 28511 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 6008434c402a..b4fb84c8adc7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e5c31b2ff5d361dd0a390be1eefdc63f0e234167a2dd4fe98e76bfb86b0526d -size 21822 +oid sha256:fa2a44563cc56647f6ef545000a24748d2504c2af008a2033bbeb20328813c8e +size 21721 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 2d5c91ecfcbd..0f8eda4c5b3d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:18ccd0bcf3857f7274188fadf6d7e0d4adcffff3f6953bdc9f5fbcded601f193 -size 37547 +oid sha256:e1a1ce8770afcd2d4a55bf3ed0c31841563b1b6d021020e6a5d769e6da78846e +size 30901 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 17982dfe7131..e447aaed45cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0251b7595a22f8111bb9c864c48fee5a8459d414cc631904531bec039e23534c -size 29568 +oid sha256:3aafb80bcd08b1f924777f2e26d4966c9b48e215f936f6f122d1de177da7b463 +size 23359 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 2caf29d3ef35..e40e13bbab76 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b8d95fce9b6f17999ef3699fed324251905b031b0acce0745619a49176c5123 -size 29971 +oid sha256:092afa64056c580073e3b34c8e62d957b30949879132daaa4ff0c8860b1a5d4c +size 29885 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index bb6bcf5981ab..7d48177e320f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:47139ec3bbc42bcbc8f649eb4dee063948f36c271ea7987788ccabc4a9d2a031 -size 22783 +oid sha256:d275aa14a876332a5fc7b0b42b1b31e102cb1e3cf3106ee8d6eb3d578cc22ba8 +size 22676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index da684a2e3db7..c969620eb922 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ac9299e4362d9518bef5dc40c6eaa1fc7539a948e7048f40692f687f4500242 -size 18688 +oid sha256:a4e5321f51c2fcb5cd9462d5d1b781da7b6dabad6bd8c8cce13bc797cb1b3c39 +size 18563 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c1e58d9e8456..2a4247bb16f7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c42f46f0a9ecffae980dc506520690595804e4bf53e58d76eca70f20097aae21 -size 23858 +oid sha256:4394320592ebf576e0df8d9fe9b64040cdcfa860f4ece606633bc49647044d60 +size 23779 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 66a3e3ad98c7..0099916f0f21 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:df7c63a7329ebe1d317ef7e5317f865112dc1dc9f3448ebe79ba9218fcfa40af -size 29191 +oid sha256:e75803fec759b6f1aa28e2bfbfbaa9611e0e45a19b3adb8f009b1f47e9647623 +size 29149 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 230be639f0e1..5ad7942be017 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:336f22bdb197a21c7f79593c29e3a3460dde2e16edcc3d62b6f03f25ab9dd2c2 -size 23027 +oid sha256:69ddf1de12b5dcc9a200ddbf4a595eb05a719fff217ffecb20f432f634a688d4 +size 22930 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 2545a618703b..e2ff716a9819 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:894bed19d1ca9b7f50f0858a0a912fc4b49bf4cca8a72a3a250961feb18351f1 -size 37916 +oid sha256:dd8cf877f80d4c40320c1783a9bc34086c0b4a8c0fe68dc1a87e28e69c027f87 +size 30817 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index c71056dd1aa2..5b5745cc6cc1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8cf0107ea8069b4fe63bd9bf833ab454ec4492329d5d3dc6f7043266e170a1bb -size 30842 +oid sha256:2ba062be4c59b3afea9739113f30450fdace94f6a72f3ed7e3c9915380268157 +size 24257 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 591bdd891938..5ef1eda41d27 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83a3e21ab16721b6b70017c3166bf571440cebc2393ab88300ca7d3b541e3fd8 -size 30340 +oid sha256:2b5e6aaf18bfa7a9111cada4735a4f0a06d739d295c262b218d6ee1d06a324ba +size 30333 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index c66f731d6406..754c7d688c75 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1cb68b4c3f70da60191ecbbb3970440eb88e3b243677785ef3157658b919b8d1 -size 23822 +oid sha256:caee5b6adcc9175d1fd58bff98cb5a07642f6424c52d5ff5f60793b86734bd11 +size 23678 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4d7930fbf3c1..512f8ace1fb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f74fb809cded6ea10093b2d31445316b72e67a3234585cd1dbd5f11df01e378 -size 19671 +oid sha256:580a990a9d8322cd22f9ab9da883290a094b5e474543747e7aeaca8f4fc109a0 +size 19535 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index e266d8a7491b..5e18f2557bda 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6023444d5dc7ef42b865ba0f886a14ddda3caed01dc87b3231c38f638fb0be9a -size 25704 +oid sha256:e92c7e90e0c8e9839f04d550a269f57fa0d19ec6fc77d5e4c20bb8eed077e98a +size 25495 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 8aca50e40781..7fafeea116f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5eda0bf09aa97616c5b451e3bb6ef77c2f36198db2b49727000d1c9ea27b9adb -size 21005 +oid sha256:e2797d2b31a379e7ce0481e8aad256945989963e4d631fda4fabd8488b5874e0 +size 20784 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e476d5f71fd7..73c32934d39d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:61dfc2697331a31675c4463df1e69aeb9ef21eb6ecc10b9e22b4e3b5efbded4a -size 26031 +oid sha256:32adf7cc5baf75745d3f139a5681245667490386bb3f0bbaf2ce130e58c6aae4 +size 25827 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index db791cfa0ae2..028df66c7ff8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d51e070be20c640b54bb1df8035cdfc8c9f8c2cd98f01b7e89c7a454374d8c36 -size 21870 +oid sha256:43124ff6e5a3494bfec4b83f0e750961cd81a3e356e9d179ae533690cdc67764 +size 21670 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 788470660285..90297fbd6a1c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0565572f2666eeff63e81885d529a3ed817fae8e4238299cc554fe6f855e8447 -size 27950 +oid sha256:73a07e35fac82631c32921725052de7de6040bbf5b72bce7aca5a9002adda6f1 +size 27938 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index aadb4505a9fd..357aa51df428 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:899f3fe0f7043736160796da5e04f3ca991fdea38cb3e1d82b453860fd8e1da6 -size 26812 +oid sha256:67ffcb99e8f1e52d2ff88c3f46baa075f00e8872944d325a97ffcdd8878dffe3 +size 26830 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3f1f4973c7e1..24d3252a3433 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:446e7c5f62f78c55cb534e8d2747590b20390c554ac6e68b2913af963128f702 -size 29103 +oid sha256:2c6a716a1c4cfdf9d03942020847a851cea40670b4ccfcfd7ea122c0031c77ec +size 29197 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 44f959580d50..73b8bdb51440 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79588c0394bb263ed8c5fa43df9aceb51a6ef8e60f96624c5b0e6aaf7ad70afa -size 27790 +oid sha256:d3ca214e8b376f938f1749a054eed1356507b42ca6b3a73685359a61c300da06 +size 27826 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3583c2abb85f..eee88352a610 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d4a46a2e8bd2e8ad12e0f18fb29a9841755783e2ed01cbff373b7aa49bc09f88 -size 27015 +oid sha256:32c4f72e7877bbb50ba0ba811908b9e9c91500f4b06be0b06d8a7bb921a683b2 +size 26836 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 335063e3509b..6ac3b2c01a4f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a9528f141c674e342a5deea8682afe2eff56b707ec53184f17b946ba3583838 -size 25497 +oid sha256:991e609bcacc2e0b417a2ef38248cefa20fb5ac90cc241438d3c480d8e3f3ec6 +size 25465 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b8cfc05228e7..653209f88b65 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d113d7033ec284dfb22f3e9701be466cf2db078889a441e495dd0eb85858d4f -size 28232 +oid sha256:0666d652f1256a89fef2c7e6937f442564129d3b413dd6e3ccddbeb44a9aac4a +size 28096 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1d0ed0c65c09..cd9451012fab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1017a2139d005a0936d152f225da4d2a5f6c78e04b22004660d8ff9a537017b9 -size 26509 +oid sha256:a1ca97fc39dcbd319e06426146b79e43caa40a873c5ec8626f0cad5b194189ee +size 26456 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ed7927730752..c368010638e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:04535a2d4de92e43021664f414ae07355d1e7fbe390ce3036d14f850c00b8b4d -size 26480 +oid sha256:901db368d1082f84b6aae8918713619d0cdeb6874dffc01bc0958e757cf0e1b0 +size 26382 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 23e3040a3997..f8e63fae108e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84b15ee8a0ebd28be2953ac3946157bbea2ff51ea134a1d1149923cb0d1e287b -size 25529 +oid sha256:34c660a118c4b5fdfe93401f8a0600a548fd3330235678c33d2a3aecba36079a +size 25498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 82916db7dc25..ebb7c63384bd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13247f1005ad9e01d90fd5bc85651696ad5ce603e1695028577037f9f5755bf9 -size 22397 +oid sha256:1050e26aead5cd9a4d3b5b859485e25b379d3f9f37a9a1601cdc015f38edbe0b +size 22297 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 236ba0d66f97..271e452629b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99c5de02c38ce40e0e3d2f9f4164c3eb3e4ed7ff3966453295cc4c1f05bf35f2 -size 22536 +oid sha256:dd6c8375db90f7674b5703db1af37be9519e4d6ff4e743b682d617a89a763ea4 +size 22417 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index ac356774c038..a282cd405029 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b621323446a64a77a517308f57a40e91136aebbbaedbae239757337baeb7f14a -size 23191 +oid sha256:acc1496887a242cb6220ab2e7acda2f6eba505c58f2b12ec570828be5942a6bd +size 22981 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3711caf6c0b7..d36fcf615d2f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:156c2e8c20915f2ea02bf7acd9c5bd90db30a5c2c333ceac2eff85740a23c4f8 -size 20464 +oid sha256:a63c6ba1cb33b57ccb0605ade94c6bedfd48cfa9fc75e5028526a6e88e9ae24c +size 20201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f03892cc6aef..409b8e48c199 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6db5d9b4658ab0c9640973cb1c75cc905870255ec5f0cb7a42c0817e07941fbc -size 27393 +oid sha256:7dd70adae0817fc9a40bb4df1c2fafc5ff81a509ce3d5d983ce34d8bb1451ad7 +size 27240 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index f73a9deee606..2405aec18559 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55edf53beaf0c07e59ad1a91d35bca1f7c2e1e7c0da5eb7d408a8f1a0557cd91 -size 26899 +oid sha256:24af5bc2b35c0ade04036cdf58a5290dc9f4d9dbe2f58fe6674e29516c2c87fb +size 26744 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 482b5bbefc3a..6d4237f77112 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd76a3b6a2fa5af91409c5416d1a4f49d724f3090f7ab18962e9fbc1b1382527 -size 23338 +oid sha256:a603dc45253bce7a9427164e54de094fdf5f3971dfe808b7e38608057547d101 +size 23208 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b81ee9e2aa42..9015cc7e287f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f07333b6ac24f0b0fbddc1377d99f0734ee8c0becf38ba92a05281c2bf4e410f -size 23738 +oid sha256:9fd3a35461c1759f3bd60ed61ed5b3bf71272d8d7a260a662173ab638e3d1b10 +size 23663 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index d51f667f18ea..4b628bf9a720 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42fbf26ad1161777745591cbde879e04982fdf3ac008309cbcdc086778d42697 -size 24063 +oid sha256:565b3a0177184331f790bc2906aec435f9819a034f1bccdcc6a7548dc3c8efcb +size 23967 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index aa027a4282c5..492c13473a36 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa9774c0a618ddcdead101e116d55ca3ebce6429d023b8374ba1ffb149bbe044 -size 21315 +oid sha256:797abd42c749dd018d79b32f949db3304054f17c63941bd7f6cc27809af01724 +size 21294 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 02a093f5f03c..def5505e29cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c0fd50e9e4c2b601b4977ac840500210f92aa5d42c43300598e286cb1630414 -size 27487 +oid sha256:4e0494da944dec709aff2ebf50dc36a253716a59808675bc7c96439812232fd9 +size 27454 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7454232d89ce..927bb8f7ddbe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4b07090573a71890dea71213264bd29aafaff0dd45be83e61863bba74a2c0f2 -size 28776 +oid sha256:1b41bc18cb8f973ddcc44fab913d827c2901a11b28f22c4aa9df599f12b24900 +size 28778 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 006510a8c2d0..1e13a984c902 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9cff4d1b6e31cbbf55da680e37607dff574b1e2c66bbbfc45e10c89297d1aa83 -size 26634 +oid sha256:042f947536674ec3b18c14bdb07b4d8db5a7d6a44e52973acb44961ade4f918d +size 26550 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 5af2a4914d1c..0d94175f1bb6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9cc0aa3a63d03fbd9810be927d3a18326f932ec03033ce07fe0894a7eb155b7 -size 28015 +oid sha256:aea04b2c17ce2722a8eb12055116371474f1be4a4aed7b9ae92ac76dbb3d101f +size 27910 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 64a06a82c2e7..6bfdad0847c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8deaa6d42a9bcefbaeb88d2e249977aeffe07ed64f49aa7cdda854b6eb63d317 -size 25444 +oid sha256:35d6e97f76c7b7d2853e993e8fe95700640e22fb43c37e127e980c274e137147 +size 25350 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 35d6ab9a8706..cd10de4dfb52 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc144bd0c7b9980027d31ecbc172c1cbc9ab6122140edde364a228749340e42a -size 22035 +oid sha256:cd337d896a5bc66c1151748fdbd8f0a04c70d81a9ebca539a96e6e37ad38d605 +size 21992 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 3392dd1f3a58..0ca9981fa698 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f2f5845ad02165e28ca805724da233edfafad7f03c71dc1c32fb742625092c8 -size 26644 +oid sha256:fc8a25d03d3b9b5dc54d40fead03eaff09119676b764abdb79bbdf1ac7b16087 +size 26626 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a3fb4f522714..50cfe2b9b998 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:06dd02d5da7a63933035a628c1e9659e86e2197536506366ffe08389fdff9576 -size 23303 +oid sha256:c596324695eb237f60b928ac4f26ef6d32c3f5da73552206b61cd424aec6dd44 +size 23330 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4fb69899361b..eaf08be71c8d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a141cb6571ceb928ec76d1fddaa395f1be8b27f7f84f5e9a607e536acd5cb912 -size 26302 +oid sha256:d4983f7a0fac38f43cca53ae17a7ca3a17ce04d7768194671d0bf91ad526f75b +size 26281 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index a689a938c4df..4c368aabb1bf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b590cc16f842f10577e5e70684da5bc8faf883d66ca59b91bf63cc849de1cd34 -size 22148 +oid sha256:b79a449f16be659b963e349d495f0174aea9ef54e1d2cf6e7fec432362cb8429 +size 22012 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1ba1f32a5960..f824bee56101 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d7d57d4f9d3e23ae67b33b9eb44d699ee8781dee5996d98a9a824b5e7c6af31 -size 27217 +oid sha256:00679b1c39b84c19508b356968e757abc403d6828007b71c7fa83dbc4f82f69b +size 27105 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 45760584bd06..dd39c53656ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:47ada28feebb87b332a7e74c7858f226dd3481341dba628213e55caed3ab2466 -size 23178 +oid sha256:fa4c7337eeb38fdae2585bd3152d76329091ff57701202dba61a0df06c2c699a +size 23105 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0e2045189911..8469a174bb2b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e66722bb4dc5e23d44f73a9fc8c0d0e2d3db539ccac2143af1e016922a7c8a9 -size 30469 +oid sha256:aabab2143d5176deae553cfcf9f6c88e79b4a20cdf2d5f897abe63dacfb55613 +size 30359 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a39f8a665e73..08dabfd2f85d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bcd8098428af0b3e9338ce22c13b28c875d5758d6652c30a12ff5c6c804831c3 -size 27788 +oid sha256:d1a44de5e01a5ccf3a62c2b6a2e94390ddad882961a2e2e64611439945bb0f72 +size 27760 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1ef5d361375f..7cecfe9f1a4f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57f9a1c03a23303ff0eb83ad68f73a2cb0392105b9fbc6a04326ca8ddcc270df -size 31666 +oid sha256:27dbccd3de7d09e8b85ea257edcb245dda5cc0d1557f2f1af66705a44eac340c +size 31706 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b295c3454713..a30e100f3cb6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:69f78cf1a26fc33b2a6332d8a8ccc5f0dbac680e904fe5570a5bac57ce31f344 -size 28691 +oid sha256:440877db9b66659efbd2a87cd081cb811b1747fcf9ffb219fd8f20b3eac80294 +size 28692 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ef819c05fdb0..bae98a4e20f4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8dd585b0827d9d73b3bbbcf52d39028e526ea11e6da60274e36db225e83dcbc1 -size 29711 +oid sha256:666697e72e2b5b07f59b48cc5626cc3f847f84319be6a3ff02e11af582e02bf7 +size 29412 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8dd770d96501..546bfb9827c3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:569416a1369071dedf2eecef9191ae439532c6eb7da296e701e0b87d6ce4f94f -size 26540 +oid sha256:32beba22b597d2cb68053df251af8a9fab38a62bc94da0e3ca12fe2ac2be1834 +size 26468 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 22e905f63d7d..aef93d7156ef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36bc52074f575717815232eb1ac6a74893f2a9e1b4d0ed102374f5e9f709ac82 -size 30945 +oid sha256:0f7d49effb6ec79044ee960129ecd9ac06f2c8a528c671be0fc2d6ae566440c3 +size 30627 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ddcd0b20ecff..0c791876ce7b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c688c91a2f8565598b165b1e7cd2dbad63b50e29540fb6c345ea13e867d7faf0 -size 27382 +oid sha256:d1b5fce1fce5dfe4314b9c75bb24b126d1c9b2fa5fff5e088e5b43845b751bfc +size 27312 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7be11d70cc7e..c727083b98d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a6f1b0911bd19477563c9e53a2b8ac67f0c03d0c561aab0264444877bb625a9e -size 28768 +oid sha256:891963515c6c10d74bcbb81bbf7fd2387c24c31c2057b0e48ffd0aea37fe1f06 +size 28560 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index d136083947cf..65604d5936c4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33f73817f0f81d4ebfea02cb0c343019d604eea76261ab16649fafe7c09346a4 -size 27019 +oid sha256:c1852529daab81bf7a07f90d9f1a14f5b892a5aa208a4b2022394a7961bda28a +size 26949 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index a50ea05bccdb..280c75fd76b8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a37587c684a6d66542a1f97341b4aee63ddb87fe8b1249e5b1b300bd32c9557 -size 24876 +oid sha256:76f8f19c89505006a4130c5d4f34b82dec3c4a06430a7f156dcf1d8a4364efe9 +size 24678 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b8b7aa8fd0f7..a808a7be98c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c3a6604d369af8dfb1c26d57da229d5c0008099795d528389fbf28d6ee3713cf -size 24795 +oid sha256:ba3cffc4d0f215b7f882696852d387723ce840e19ec0a03b13a383e4096d1a75 +size 24742 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 29890badd254..33e12e49fbfa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42c87e5a7c1d8fc893b9900aa2e88507bde8e8dd557939d642d7096993a7f478 -size 24352 +oid sha256:16d46a2f94a304eab4c0ea1694c8fe99bd5ae69b215a34b07c79f78984342819 +size 24272 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f8b91b946f89..52ebb32d31c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b2e24afec11c3748296a1d19aeb9df7fc00e901d7b329c9a91b9ea76b3087ca8 -size 21573 +oid sha256:54decbe3251b6eb8ccb9f0d47f26a6c20927cfc8fa4faad46aa34eadd3dd8ab6 +size 21459 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d865ad57a30b..82d515792659 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26048df02f5e2aa9832b9d4a61e551bcbb8c33c901b133e4a7b9729ad7fc2dc3 -size 29474 +oid sha256:9af9b1c839b32d7db87563f2606855afd014cd807d62025fb0e29da91c380273 +size 29277 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index d438457a28d9..3a37602603c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ee5d564f16d3b30a307d87b5f635652fe8270b7fb3a4ecca5ffebcb9485ec73 -size 28020 +oid sha256:d9eae29cf9fdc294a15ed5985b9633ed400ed80abf476877c0657f95ac93b88b +size 27929 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7a966178caeb..6bf1ab5096d3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d775d247cfc66b494157cd4bba53d7ca2f0d3c6f91666efde3388358190a1fe2 -size 25881 +oid sha256:5f14c9be52ddfeaab8ca48e113e4141f9d059a76a6ebd83f98db6399234b8ec3 +size 25662 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6d0ba0f75472..102c02a6bd60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8d4549f15def7f1ea101da4a12d91a37079460f395fe74f0c9f604aec46ad9d4 -size 26056 +oid sha256:0cbfa36fec0b06f45b6d8f8d87c4cbe1c74d3b7797410482ee390ad52eb0995f +size 26013 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4798c698b055..aee53c1f02c6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00b45da91b25ca1731082907a65b31a1359b29feef0de314b3d558ebcad934a5 -size 25295 +oid sha256:e335dbcc641b67bf72807e9b0e6480bda82a864b4efa1fa7ecec5c08a61567fa +size 25143 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5fec90501e5f..538e75aca940 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0fe08822b84a4fe0a01fc7c36732f4fe14f89043ddd30cda6c0404054ec8bc34 -size 22801 +oid sha256:a3bcc69f6e763fee3598e131cf934ff49bf8e66d8c5d0e1132798a336c2ff4fc +size 22727 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index e3ddc3e5365c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6604ac08308e48b883847ac3408763766c1fc9e07799d1c4b4198ad2624761a2 -size 34844 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 62ee9f8fd4fb..24a94b6cb832 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d4d376b0f225b22b645781ac5bc5694131531e9b1ca5a3e8fcb36a0a73c7ac60 -size 28607 +oid sha256:cfefbaef4f644067c1b5ade48e14a3029ef5fa1e84c4a5875493cf09f895b5f1 +size 28535 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 35ffbffbea52..07fc924b9fd1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98c8dcce26ee4bc5d076e02831b36054465e4b0a90a798611da4c08b7b3390c5 -size 21870 +oid sha256:b1d22c881ecc62167459d7937bff05b5e66d812006002bda944f0638bae019e4 +size 21772 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 3c1969383875..ba3e04315272 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d8e9a89de943c09cba26c6dc3fee900d208ef811b20deb4843a69ea3589571f -size 37761 +oid sha256:d2a95fc04720a2bcf928ab3937038b1484f80bdd4d9060445d9d9d232f1ca945 +size 30981 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8cb8c56ca4d2..995075e790a0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca42ac5b007106ea176355948a2f52efae956dfe2d0f0c4cb89bd4412397425c -size 29571 +oid sha256:487d13db8538a6db30896c8df08f838e26c3792cbb6a615dea5a201bc35f533e +size 23357 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index d0e1587ad55c..f389bcffa4eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b7e33f825c2bf9036c66b60127da1f1508f465457f50ccaded89e618a47a090 -size 30026 +oid sha256:10f3f2ba9ebf68f258404e4d9c4729ee15c08818208fe4ea711f59a9b5dd2b55 +size 29929 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 46c2715a13cd..ea5204d81a90 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:73acb0a179c2332befad43c00009d968cc36d7907bdc9c51a80df6b1ee081f8a -size 22780 +oid sha256:70534e2f4caf42fbcac716a4182d5c81412a3186198a0e994581423022d89f1b +size 22673 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 9f8ba6bab359..fd5bf5dedb76 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0db4c5b422103a8727aa9c268023b602e6477c43952ad852aac01456808f1852 -size 29203 +oid sha256:917635be5e156a43e0f75610d587d7950b676862668fde20c751aafef944415d +size 29155 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index abe241b9a22b..dbb80c6a7686 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0548cb902229c3fad6b12f7fd0581c113336c73315645260c917772170baf97d -size 23016 +oid sha256:682d0b680608ae63e6bcfd9b8af0d805c92e98d57dbc53009052e1b9f66c8220 +size 22916 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 5f533988337a..316d78485244 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f58a1d88640ed9df27286ff610687fde3c15eab1e5722d57ed35e8d5d5d79f3f -size 37929 +oid sha256:b92f0d6857ee2ca4f6fc0b69e1a35ae1f1f91be8b8e33f2263414c8750e68f52 +size 30819 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 07a7d92a7c98..938415386b63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7c0ec2c6c3ecd2fc03bdd68f8aafae05e9e0e468bc7b62e4125de0c9c3ca86f9 -size 30845 +oid sha256:dc7e92ba3136bd0ac6929edb8b253109edbc210d60d65b323660e1145abb2362 +size 24248 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index b9412f7a0bf1..e7f93a48094a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2cc99d9ec7c8eaf42e270b17cadc2696d5f0201b08887a94cc320542bdf61b19 -size 30351 +oid sha256:092c41713287548cf2f64a6cb7d15a76c29837cbcb4723c45832bda4d6af3f55 +size 30345 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 8ab893efc69f..bc8effa8bad0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ab2ecfa792b287b71e9a2602107d97783787ee2361506065408594d7f1918e7 -size 23854 +oid sha256:087b734b946dab092c8dae68f6e21fa57f8332a4ac66884800b7fd4d616821fa +size 23695 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 230081444939..025733aca93a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bce1890236e3c9bf69b3856daf752884d99ac20580919090ee5a002fca572251 -size 22437 +oid sha256:8dc81eaee868b6d71273e1f8e512cde33266fe399df5b24b46dcdced38786628 +size 22327 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index f142357325c0..c5481d863bc9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ca27e53331ca18fc828b5a0409fa6fd97d55c0a3e7238ac43180cba05e863f6 -size 19317 +oid sha256:201f64989978631c57ff1ace5eaaf751366781b2086666089be33ad752da1af4 +size 19208 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index cfcc09f072e0..605e8e661be7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:580ee76ce0508b7901a08e82ada614f1c38a3107393732239badf43b6b5c9c2f -size 23227 +oid sha256:f0bb8b9d62c9de5e713763050a5e97b55f4dc1c4efaa5c0a763d59dd8022d703 +size 23132 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7f2d9036f9fe..bce9405d18f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88a584be2b6f8e95fbf98f325c108dd119f59c75f5fbb2521bcbea5427991fc3 -size 20402 +oid sha256:9f7eb5ad9990f075bb6c576e5b85507d8f851e767dc25f9c0723d700b6d9c66e +size 20310 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index bd539241e189..2800f7be188d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5665c82bcb4c62a77157da4a38061402b0c42cd4097eb90e5b637184b48c7dc6 -size 22403 +oid sha256:b232eb9824c5c4704ea01d36fc624b176743b01d946cb37fdef828697654f1f0 +size 22247 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index e79ebd612866..f55a963878b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88f611a3ccbb107966348d76b3671d1e9a6e938e627e313d11851f2b90307d35 -size 29172 +oid sha256:e10a50de69fa6479f96cf1861403991064d067ab63e6424955e4411508dd0b8e +size 29098 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 9545b38a5b35..25c5b7f01cf0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:950b80a586132587535dd9ee51b467c793edf052970b41a2f01f11afa6937196 -size 22237 +oid sha256:a54caa97b99585abd54f2b9718801fb7318bf659097ed52f1f638789271fa6a3 +size 22153 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 017239b4101e..1330d5d76147 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a13d917cc5c40013312a9bb76b63bbe2774b93174059792139d16adabb98b7d -size 38980 +oid sha256:c4c908635b33ec222a667993098c87451fc0f50d8ce4c88e76724525e7d5a9b7 +size 31254 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 1732a2a521f9..ed1fcfa8052d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cac1b60e2b987b78c4f52d4d1d28e62996ab4361e6b296b0837bac021c82bd5d -size 29465 +oid sha256:bab3cd27735f3330557880667c28e233c649866cdf1695697e878dbb1d3a117a +size 23348 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 1e2db77dcfba..344e59df62a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a615eb566198a4b654db36cbabe60d1d64d943eae3dd40bb27e079f4fce13f3 -size 30405 +oid sha256:ca4e0d7dbb366883abc12fcd6407c87b7087cc43a84b1468d42e942dd423c223 +size 30338 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 5e93c011120b..6cfea9385a67 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f4d229b9ed2f54f44a985eca69869fcab2ab3029f5c5efd4ffa8683dd3c57600 -size 22975 +oid sha256:80d71d18a8288007092c3ea0e48e5d232a1246cfbabe9d98621e0c5a50cdd5e1 +size 22807 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 182fac182c7b..8b10e36a1724 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:52df8cadb2d4d2f35c8a01ee5fa0818022b9866979476190e748870cba6fbb60 -size 19222 +oid sha256:b6a57d93a34f40b272ea8e7a40ef5e42a57c68b1bb88312e66d3963af0b478f5 +size 19120 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1822de768dc7..6de58727b0cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c80974a31b4c56b4a94a928667fd8299bf06f30549864a4ec0e8cf9f7f94297 -size 23337 +oid sha256:9df2d40946dd7f133542fc2fbbeb6fe1ce9e04bd6655a809cff7c92242575e33 +size 23206 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 66a1d3e3811b..cc492e6e3ded 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7104f1872f345639ff6a940f7379a2f686c9c83d97ce37194e79c232dc54fc99 -size 29892 +oid sha256:f605db2558c98b7071ca1607e57512b61b760dfcb834c4b234baa7deeaaa78fd +size 29918 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 6d62f40981d0..1ab90ffc32bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:847e25d964b9a226a5d05752ff6809942e1a753408849666bbb8e976fa4e0432 -size 23436 +oid sha256:5959068afe7ea79d965492f43933bd5dae447e8d4f56e3587909ae9ddc0d3339 +size 23228 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 4ea8c4cbbcfd..d977c4cd3f24 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4cbbad1ff0333a40ce999c87c8a898fcb59e063684eed83f07b57fafbe09d5f9 -size 39342 +oid sha256:0ae21aad7c632a000a177e9f588db42e2540b1bc54883097af660b9cb072bbfb +size 31397 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 45ca7cb204e2..fab9d5cf8a1e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:562210d4d787a3bbe65e04d0ccf9ca6fe29707f689f30cd97f4bd74fa03bec5b -size 30998 +oid sha256:fe60ae5cacdc645d881046ef18a28a838a2e50e2755657c8366ff26d2e3637bf +size 24054 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index eddb37dcf63f..9b8bc1b18458 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:58a08808a989ce8c6414673876ca24feb8e4446e6aa895172d13e556c55e252b -size 30744 +oid sha256:ee597fc23bc020d540c83ae3fa418d24180de324ed25c321cd140e0ccc52af81 +size 30722 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index b80ed23e3c4e..fd0a75b9f46a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21399deb46afd463bc00915861270f5db8a63ebb9e09c86f0cbb79bf7403a361 -size 23961 +oid sha256:955ac75eaf6fffdf40afb8fd4ec41baf93103659680e64bd637a0fde9db2e3e8 +size 23780 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2fdab7e9e687..78734c9b9945 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a5bda058424c64a57973a2903dabe6bba449e9b65a65854dc4c482be6bf382cf -size 20166 +oid sha256:19618fc1d76f98634b36029f30b62cb1d9cd77ed3f302e3d24338e448bb1a3d7 +size 20077 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ee9019fb3f5d..e3678b3b056f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd794f86a31b3d842ee661c567eb382918b73d4dd8c5fa83e4c32a655a6b0cbf -size 24489 +oid sha256:3f2768def673249ab126cd402079fd371085b63ab90b95cf9e1cb157755d75bc +size 24318 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 3754c4387873..1b98f55529a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc72e3e1db61c5cd1f1fb94622750a311c44c0588b7f53b9167789807952fff2 -size 21412 +oid sha256:f7c3963bf782b5b800c3d5b046f4c7c726f91c563d278a7097ac767d3643e79c +size 21245 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 96dc90f1065e..1c1f86e98cd1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b7b1b43eaffd13f3c5fe274198ba3dd1bc456d6c36171d8dfcdde433d318af39 -size 25513 +oid sha256:29d7f1fa8b945f51e8c5b0f7b34a9636ccd5a23d0bea1f84e45539ac94b05f05 +size 25241 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ddccb11b6225..24ed9afb0cc5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f6d7c9a37a91b7d4b6f2b9d935f28cbbcbde6cf308cce20542704150fd7f2a0 -size 22491 +oid sha256:3f68fd9b57ba6e87948886a3f9998550995766a270b690944d5f2d91b814b581 +size 22313 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 626b73e55afc..6bbd2f2eec74 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:274c2e9969b9da4fd78ad9c3c0aa8e0b815de005b9278a5da38c0970cfb9981e -size 28178 +oid sha256:b54886539169628c511789cc71c1bf7076835481c2a35737ae9f5d5a3018e3c5 +size 28097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index adf3347351b0..37c32801d7a9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7d1d8c1588469fed3145f14d956e5401b2147d7bb4127ef25e1dbdca11c4d3f3 -size 27050 +oid sha256:2e8a81854c70f8938245ce0c3b5e5a601815af1693df3f0c7d4278efd86d340a +size 26929 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2a1e7d5ce15b..2e4563a204d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb5a44fa61149c33b9ce48e430c35e7522428e0a2fd4923dea21de5971ea30fe -size 29145 +oid sha256:c9f64dc5caa1e616c8915b868c37527e17ec55d27d1609353eb862ea249dc7e1 +size 29092 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2674ef672d4c..3dab0275b6c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93118fdcab0ba09f550a91767b969ae61c41dabc8702c4f07e8476883bb0f672 -size 27958 +oid sha256:8ae00e64608b1a58f6daa2e089f9fa7fe5b47b3c01ce3b9cd7bb34ff686e8e9c +size 27865 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0a0ed82ef80f..d3786dd4a8ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e708ffbb85b3366dac2b4aaea99159610e35d2c2902e210d1fda3a042941c12 -size 28897 +oid sha256:1005a2b744613b32bf70f432fda389419a5b7df69b4170ab5c7fa8c14be089f4 +size 28764 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e8306891d662..1ebc4a32258f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a9c36fa6aa2f66f0d905809d38571569973e3e32c155b9e34c7eb3b63a8fc2a6 -size 25803 +oid sha256:6b5eb533b33d01b07408b8fc8c2fdaf212af88be008cb8c9534636ce372c2a4c +size 25742 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8777977d3852..dece8b41d8e9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:da6b0e68d7b8c91279676430f874a4ea7d72405df94bfb3ab15b841861ed6246 -size 29858 +oid sha256:784a76afb761af3e975e9ff4f62619c9656fe5c6f6a72fa913f57a82c83268f9 +size 29798 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index eb8839fe5cae..3bb6e72b81b8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54558393c231b2a132f039cff230e10514f248eadb78afc3ced8eef7fa050064 -size 26811 +oid sha256:54ff0ad713913af433b04d025032321402ed791631e56495a3261febf850a6ff +size 26709 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4f62ca7f26db..5fb027f668d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50405041baea5f05b48442383fa4343036b36dfff73ab4a2e1757ae82dc3312d -size 25631 +oid sha256:4a8904f144a31eb490a2bc5497bd96ba22bcb06f3cb6579cafa502e31016e212 +size 25573 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index f06ac3c9bf86..26bde7d8bf60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:46ee4e5c05a32b2ce5040735667b1eea153f267b15d049418871357d4b678b70 -size 26011 +oid sha256:5309ee8f42ac1cb4e5992b195df9062c67cd31250c378d2b4233e7074ea3078d +size 26007 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 9d82c2402383..fb43ff6bec8c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0329adab05a4f15dc402bd508b1298c6a0daad91c91ca9e2b0d55710864b9ae7 -size 22979 +oid sha256:9eea80c01f74c1f9e9fbb4735792dfaa327a8ad1c217e8fcf26f64603ce75ccd +size 22838 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a434002301e4..66845c5e88ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:97a7c9c3a1183c1b862c42afb5ca91d20bd36e4d5c55a4779e24828522d2e72e -size 23122 +oid sha256:3b2c0e136e9aaa0fdd0eb790ff2ba891081c5619a4d63fe7f3b754a9b6538acd +size 23139 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 508ea3c0439e..864d43facfa0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7df7352a7601196fb7ea2fd6c782666f13104a3b545f3a2f927377efc017a836 -size 22414 +oid sha256:7d9c13b57ae709d7a9e767d36b7cd0e2db0f250d426ab6cc27e1478f57be6cd9 +size 22377 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 476dbe809a9e..3902dccc229a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:19d1f61e6bb57287cdbb977989a4d24e8f3d08d64b390e94da8d4b9762bfd524 -size 20219 +oid sha256:d4be3e22b13f84cca1f3ba0becb1acf6abbe68c59606e522a8b1ae7a550a40bd +size 20066 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 92c3ecbad29a..90be80b79f74 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5455a87361a39d0f1978e1710a5ae34ff43bfac21497bc04b66c420e9a0d8238 -size 26552 +oid sha256:a22a26c06432b8dd68d40fc7ccab52f0d77bac4ca3bdad8a5334625d8d27d064 +size 26481 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index e35d2f3b2faf..329ee4ce2a1e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f73484d48aa2adab0383f27333e66cdd2a1ff7c768baee0dfdb4e12c440aac8 -size 27374 +oid sha256:9caed0de44a1ce9aaa935ac6a81aa5816656463a19333bb5fc190e7b18440b5c +size 27225 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f5388a82b4e8..6833f2a6a7ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8d6d2ece3cfa3de16ef3b82b29139a29e0e720fdf56468b133e03eab38768dcb -size 24029 +oid sha256:6579782330300a43db2bd20ccbf6749455a8e050ec644a524867fa6fb1c1c6a5 +size 23917 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6673b8f518b5..30eedfaf1c47 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2688b1cf3a88b618eb4c45d6eac468104ac5b8e65a0791b9850f8aae1819f7f3 -size 24159 +oid sha256:6c51aba8ed922b5c11b6e292c0410521a157168f24fa75e7f9d84a9bd49a60d7 +size 24097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 3a9c4ad4e681..d17a41a6c169 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1c974e2d64af19291fd783d79b0948703d7066379427b7ec296aaa6949e0f3cf -size 23400 +oid sha256:eaca35b8fb0605dc441bf1ed05899a9f3ebdc0b7033bbe0429c9ac01db21dd30 +size 23284 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index d8920e9f0ff3..2795111e519d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99ed772455e0792e32dfcd6efeb354bebc2ed6e0ca6908b326b7b9ac3afd793a -size 21147 +oid sha256:87b94458bef4ded2997e6d39f93876b9f72f3dfb046910e38957e4d622a0c366 +size 21144 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 57c646f47a99..f358f1ef24ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:943a4d83fd2da69f77fb519b3ad1be79708a55997955805bfe473ebbe1e174fc -size 28041 +oid sha256:ac4d81bd92c2c428eb679b1c4818a38dddde0524aac88302c9a6bf673dbf5cc1 +size 27986 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a8408a7178a9..d9282434d035 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c918836890513771ca4608e381bd83719afb8dca1146034dcd8ea7e739b2850 -size 29026 +oid sha256:b276e59163501e5e4122a8e55e0eeaf97f7140ee840ff5f8e3b8adb3f1bc7b7b +size 28937 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index baba3c05f513..03548cdf95b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db7503da4d1f930a597df3dc8547583ca6a434597b01a570f25d76cf5e38c991 -size 29036 +oid sha256:de654eede88010544f5876ce806f0e750f54833dc270903cdf317e32725d2194 +size 28936 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b820f29bf06b..d222e72ea202 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:17d2216f9e2d7b68a58ec023bcbe47309863bc878223b2de5639d4094fbdee2a -size 30090 +oid sha256:ff24123e40b90a8305a034ec4fa74b6a2b9917bd299556463463534eb726f9d6 +size 30007 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index e9e58b782f52..67cb13f5f002 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:76c5970bc2509384796eebcdfdbbd0932319df8018b235561380735d28b4e038 -size 26073 +oid sha256:996bfee7448c3b1024c0b539b2cccff71e0df342cbf3a41273bba8e0c2152a80 +size 26006 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6db6dbd2630b..7f933b11aaa7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2030f36dd5e74ea18d9d2893af7ed058c42be1c82cf9987bd77596f72a917d97 -size 22920 +oid sha256:6b7d0aec501ad6778cca5906043969b2a3921f1071afb5e1fea591ab0db149d9 +size 22870 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c8a7b38a3f8b..391bc1f38834 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b4c89b911274af4b5537c0ca65a228b3174132876c1a27efb0d06ecf705a966 -size 27380 +oid sha256:ccb9dfc280f2ef11b73668934251cd17cb1407ef40cd5885f7f3a574ca0a8765 +size 27318 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 32053ce59554..c96dc79ef59f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6bd7a242bac16f3e50bbfc3a0ebfad96a35e722246cdc054227b7e7661a24360 -size 24118 +oid sha256:bf20c41f32888ab52438903367b2acf42ad7b10745dc9afc18b6a452f06d46ad +size 24094 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4352f1f57901..ff521fe5234c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:092b5bf97ad547680983e9df51babf1104f46279281d34463a1c743ee3032771 -size 25543 +oid sha256:3a8ac401c88ce6142722fa0ee26059f4a2d002d3c2112de0a1f0090ad3c1fdfd +size 25509 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6fbe761927c0..2dbbab149c47 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8de70da13c694887b219c716729c66e95cb642887f0e9cc3d3bed99582d508e9 -size 22987 +oid sha256:1626ae38e5ac7628dee8c107749b00c0edc120f5a7b1f17adcc4e67cbb7fe243 +size 22871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ae315974c953..ea8911d96eb5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7da6e1fc31a1a89cc0ff85d5fb61ae63be1c475c0315f375c8b04794a0bd3f53 -size 26623 +oid sha256:84e92880b96a28ee0df3841684e55d8d12ccc1802822ad78b348bcd1712fd638 +size 26536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2e347286e67d..8fdf59e171ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1fd342d9133b6638481bd9f8b15fc133ebd54e1942a5d105a21bed5c12aa250 -size 23880 +oid sha256:3fa92a2c7b30d7a092a0a1ae1d6fdf74eaf676e6630bca88bef5842793702f6d +size 23806 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 715421f9c324..c61a21648c35 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e110cf53077689c1d206ad8653ac7b179e5f5e9fa340a01b5545b7325d6de898 -size 30998 +oid sha256:049384f46f6843670453f337ca01f0f70ad6cd32939eaadd45bf365ddac4f258 +size 30977 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4d3d53dade32..9485e1da4c07 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef3ed8c3e1008f1f28121b4208b9b5f0c81baff6304835e2cfd45ae7c8d71f63 -size 27804 +oid sha256:c2302f4a04dff744e4a8d02f2dd1cd483d10552b317776cf9854d3b49a67a47b +size 27618 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 9f80fd5ff7c5..1cb2f1a53848 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e24c30ca17726438c6c2b57675ecc6e1c70f8839a0cc3a551116befb02776d21 -size 31927 +oid sha256:5043eb258531c03c77e493505947ca79d7cfe1db195b1feac29bb8745c05c8b4 +size 31740 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 29987120b76d..ba2048b85cfa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b60c776182e8088ff71a519be9a7050283770d64e49c723aa37cd209f2048319 -size 28629 +oid sha256:a69985c800e53ffab415040639eb8ebf83523fd61484aef67dedd5a6164331a9 +size 28536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 5dba6aaac7ef..f7042139e71d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cec88f3826e8e92cf51fcd5be8ecee51d257a2a70d80a97dc151d498ea501204 -size 31865 +oid sha256:d8ba97058f0bec5b57b63d9154c571456ba93ba2134921d4b2388703e0c06376 +size 31846 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 77dd053c304c..c661d36a5e76 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a77db89c8f3634a708216608a1d71077dc6be9413827801b5194b4e4be08febd -size 26727 +oid sha256:960640cf8006467221277c04adec2cd48eacc6e6b0670f2188b30d3ef746d5d1 +size 26584 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ac807a4989b2..3515cd127c4d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b1bf8bdac737a6f459e8069bedc1b126f646f325f7daadec150b479316c39efe -size 32704 +oid sha256:e7beb06b7546e8d59d8a1d216d5b9ee56d3b277aeeed1df4b10abf03a8e637c5 +size 32683 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 0498dcc70424..979366fd7367 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fe81d7672a816a01d95a10511153b3e7e124dd280060322fc55d270458805ee9 -size 27514 +oid sha256:9878042d78960e472920d6b30ad89f389d2ef193630d5a23d5abd2e5971eb9e6 +size 27361 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ffe656370db8..bc57bbea78dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37b2c7694173e8501dbb5fe737a78e48bbc2172d351c312083d40c10b7833494 -size 27910 +oid sha256:75711110e1d9a68a57c200f56b60818aaf5437caa8e13b60eae050760eb90abb +size 27722 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 79d4fbffe187..7cc689b56ffa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1c60bf33c2663e29c412388795ea3dafc235ec01bdd37f4919c2898a8a56ccbe -size 28683 +oid sha256:0edce6318c899d6d3ff913e8f8de166874e8835e824c6a4442d970baa03c5ab1 +size 28605 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6fdcb3f9832a..c16cfcc8e883 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9c07c795127b56b701b1337ec64fda6781f68591dd1ae64032ea2ee036979dd -size 25511 +oid sha256:4b9e973744f09a9941ab00b94d5ce4afbb0c1f441472cc9894bf1698e92b00b9 +size 25302 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8c421afc6c37..e6b61d169b49 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c28abe1cd5ecf92aede98707497060b35715327d95e6f6a7158dce5b74c0f6c9 -size 25916 +oid sha256:0e5dc39e83b1fde3a7f39531adb258990e02183f634c20fc6fbd110adf0a7cc2 +size 25882 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8f2da7d4febf..ffe70705da0d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c71bb706939c434b5ca410c16cb5b83999f2b17ae632e18635651b1f4b08aa7 -size 23720 +oid sha256:3c73efa371829051369fd6297df1e2a260b4d8b8d547daad10a554ddf8adfe6f +size 23637 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c32394bb0fb7..32b8e47a2eb1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d5a554db882b9470b47f029acad9cdd8b07d80d2ae4478c07d0b2db3bad2c2f8 -size 21665 +oid sha256:db8a109cc6ca5420eba831a9e9c58fe0726f0c7a899e65470f096dcd370ac6c6 +size 21592 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f3c7632bc433..9d6e2fb86296 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dbd44f47e9f4f3bb8b71a9bb04f81369deb483f5e6644487186b01f7e02dfeff -size 28742 +oid sha256:e3bf787c9123e999f870c49352c907e0570128a2a7def6c776db4b7d4ebaece1 +size 28584 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c457128091b6..2d0874ffd234 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:419785e49a1b8384c6f7edcd4e796befc64a6f9fcad47d18b78f96ea282a3e4d -size 29512 +oid sha256:4c4a1825b8e79e540de605193e689cfcb4739d6ad8a7b8c7717195bb7ff7057d +size 29574 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1c35d4c7c264..bba02db99dc2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e9aa2e5cced4781c55e24c9ec65d5a061c602ce190fb5b6bfa74fbb6400689f -size 26512 +oid sha256:3b8e32558e294fd5f95cc3899f69605296437d21db905222e765e7449092bb0b +size 26321 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7d98fe20eb2a..fc0e64a78e7f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d4eca7637b0ae95398c40efb214eb0b8e6dc0240c05f20e8c53527aed019365 -size 26709 +oid sha256:28d2415e1319b84a3448cadc25c78228e921dd5948df6e15b8b0aff7fbbd010f +size 26618 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index ec921e2d375c..032c3fc468bd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9ac80c9f261b7a478f8f78c8e1a0177ae985d5ea3700819476d4abb13ea756b -size 24709 +oid sha256:3f2a9b771c922567ed3645adea3af45e162d006ec8fd70551a10a1e76051e88c +size 24645 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b029828b07b7..da7813854156 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0261a8c0afb303bf0e7d91a505dac4b3faacf1e6b8996a10b7b62aebfa7adc6a -size 22627 +oid sha256:432e46ede63dd889dd013c66e8f45bbe4ba9362d4c3f171741f2ad23e9ce9172 +size 22648 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index afe2e9d79748..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ef32cf3de82133c80b4b4ad734be9c1c97a7488bad9a335b4131f69f2983bf46 -size 36715 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8a2e464b76c5..f655bd5b7a24 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1b0a2da914712a2646467a49a5f4a0ff28b0df0c863b1823839732fd9a2d9f25 -size 22423 +oid sha256:834b86378e7f2fafa53d680af81bd62b2ac9b3b56ce0d967ec0b3052af31cfe3 +size 22274 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index b9f6048a611d..aac694f224ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3858a8b007fc8966943d23601e55ddd8371f48053872a295c15ee7072623d333 -size 29145 +oid sha256:9d662b0493c4611146d8eb8b375444890dde14403144d8f7b09becc3dedf9071 +size 29077 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 77d1163b2841..b2339a320146 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9e08ef17ec039eedd5145b2e394883f9f337c4517c2f60ae133233c12c4cf96 -size 22283 +oid sha256:3b470ef20846d72ec556236ff812c3095426e786e9361abff787d49e0e007aae +size 22200 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 56c1e7c01fd9..b1e078e96aed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b9a2a6952511938bec9961d474c8952dc6994f101589da636c4a2a80e99c468 -size 39007 +oid sha256:62c7e688b779b2df916d54f0555876bb4ed56167d54a9499b18eb3134b135437 +size 31260 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 34d69b80070c..8fe118ff728d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98601706ccb29d849ef2586dc7ef97a315cf049b99a9a35cd72db80867b436fb -size 29532 +oid sha256:6214b0db5c8f1597c623b43e00a1e7e09c7883a9b3452ac6dd4f199d650f1f70 +size 23392 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index fef52642d85c..2c6ba6612c9d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb8c764b7fd384d2c7b7f8ec95bb02dfa03c0a0c01cfba21d58919c508dd5294 -size 30437 +oid sha256:06bf424d2e2d0961691883aca3b55b2b919fe500b4a70bb66d6ca81d1c7832be +size 30353 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index cca22fb64aae..17c7ed2ea6a5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c3bd500bc871dda3a94b82fb4b51c752b3b61000c0b375ca43423b70fadb77c4 -size 22985 +oid sha256:d4ff919346c238fc6bce589b2d85a23fa6fbf772839ebf888b99f7a1310d96b0 +size 22828 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index df969ca809eb..e29ed41abd17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d995c883a67c7317be706e2eb6d71c7ec138be5a87d4fafd3d7f7571e5824821 -size 19242 +oid sha256:a6fb19d54316306b07b04e4c3079d3196011da5fc690e760cc765c20c2ca62de +size 19169 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index a4edb8390344..79bf008d2ae7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53bd94aee51340f696b8252d4da0e04efe8f6f0906f98973288c1ba5f7f8b389 -size 23332 +oid sha256:fc2985825fb785b864209357a62c6d6236e10da4b387ea31f40c122ca68ad148 +size 23209 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 596e6214205d..31ab244b2fb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d7e50d1734d95900ddc290107989a3a0213fa139521a16a487e7ff364acde4a9 -size 29883 +oid sha256:ca6fb15a41bc25b56f8aae985e7e1514614ee9dd61b78d671935b6c4e891c44c +size 29932 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 527a72498950..06a4a9ec44b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:270b1e9545af1be896abd708b3e809da42904c639fa71a67b5baba0b1ba23261 -size 23445 +oid sha256:0ab640a556b93da22fd359672f0ab0bd4141809784c3d65aa88efcce10276f34 +size 23234 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 4159f8997c55..bb7197fc0785 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef0f72a9b3688a3593827f3a54f717874378c98f17fca46f017b3589ba60b663 -size 39364 +oid sha256:02b0e01044272c6d498e162f9b02359d371df6fb067dcf9e481ec83a3014d460 +size 31397 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8135f2f063b8..64bc8c1ca2ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55e01da6282c086d22c03a96ecf13474245dfb566dc353f780e66576fc3f8379 -size 30995 +oid sha256:bab1ee561d341626498544f46887314eafb91bf02be8ecfa49abf08ef15b694e +size 24063 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 3aa2d033a595..f26f819e3f82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00a05a5f75c3b92f1f2409197a61aaf64bdc88688bcfa9fef5ce9ac50837cea5 -size 30756 +oid sha256:1f0a52d22e6108ab2a928395680afe8333ad0af7d9465ecafe1c3886fe2b7044 +size 30731 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 6c633afb857e..db018b284db4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48df08f418c018fc5610844cc8abcc4262080eef322c9d980e4afa4f904774a5 -size 23950 +oid sha256:e763dab6913143da5eb20702031b1e1fdb2c4fa919a0945e8761dc5b04bd500c +size 23760 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e4cd9dadbb49..f93aad23c1f9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d8e9f3035e5964a1e0a754256b3ddb3d3295319ad53eecbb79aa1568d77135c2 -size 20163 +oid sha256:cb812a63d768247fa7ef41b722e605e73767a9c4dfcf03112bb6d8d46e185986 +size 20065 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7a56c5b9cc23..5ee808832e1b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d02e714808c9f553e0927e7c8e281ac35d058e4b42ac33cad36bff3c1c39a780 -size 22338 +oid sha256:ca9a1541ed1b82125961b5b031203821c6840a0f94c2c267b41723934b9d7518 +size 22195 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index a5153c27b490..b13d73f7ab1e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42e8a46ba35bff81f1e2eeb16cd972757d03d402b5d1340aab1b062f84997b31 -size 19405 +oid sha256:e58371c7bfce8e1cc0b3ad0661c2bec99b45db76e6bf2afb0e8376e459f01bda +size 19276 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 76407ae58793..736764b9019c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4bfe4be72d38a3e1d1a6b93c4b4d00aaefa95dfedf0c1a70725d3d008192513d -size 22729 +oid sha256:4f872e53c87f0268b770018710632bf4cf82f598005d918b0305b94f33f2e820 +size 22682 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1b8ebe70d323..bc53c58e5f21 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a3c3dcd6eac58ec00a5631660473909ac128c1617e60db1b84589962452d5a76 -size 19752 +oid sha256:c77d20a28d2832b74c986f055faddebc87c6c9417142bf215a65b9c22f35e4a2 +size 19606 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8257f99eceef..293bb1922498 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9306704bf04e7bb3e69c2a1da5adc54529f3d0dc2cbd36926e150e176df81c88 -size 22481 +oid sha256:8eaccfaea7b2ed1607096124f00dab918e135dd22bd74835dd15527c9ab25bdb +size 22343 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 77f24ad72b00..e9e2a97fcc56 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b20fb577d421c752f90155ffa2f2338789bfd3c9f7d4e4c7a2723143b4550c2e -size 19279 +oid sha256:2130fce269fea08cb88e9f67282eb763df5b9ac384f8bbe5b3791291111d7be7 +size 19147 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4c9c17d8d1f1..c1e50234feea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62147640d8dac3880312f243ab6c4f6181b09be5701be50da6c66243a47d7aee -size 22935 +oid sha256:4eec133505531c0436746037444009aa07e01c8d1da5d966061a9160feb95366 +size 22901 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 3fe955ad7189..2cb91bc3e71e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff3b69f6f93324ceea349da7f28ba0a756068f6ada783a57919e729e5e97ac44 -size 19489 +oid sha256:0957ad023f23b9a9ea82a564dc2cc365061d6a71e4603e1d0fcf333882bb8a4c +size 19388 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 2b1228b216fe..d6ddd26fb356 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d9a535185f6c680d18bbe2ea0f33de8f7185a53f8e3495399c4d1319c2ece547 -size 24376 +oid sha256:6f6ea70ee1f34c355482f08f270016f6513566fe14172beb3e6bbc0c1f807444 +size 24160 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 27492f0e7306..8ff7924231ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3c8924ec57cdefc67c738e7fe30a103c89f27574c4af03361cf3ff54fbbf77a2 -size 21665 +oid sha256:257e0317b3a731f7fd85109264a737d885684f50ef48728313a251647707f7bc +size 21418 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e02529318f97..11678d57d13d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8fece40973c2f57f7df4d69f238134544f7b8534a6b9982cb757e6f45ad675c2 -size 24842 +oid sha256:482f4acf2cd618eb976c30b56b2fe71771340e44016095533b274402ede43e72 +size 24756 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 506e0667a62e..0a1dbad7524e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1179b744460634210b1bc06afa0cac1ab427b45d8c6ab7bf9174260fb6857311 -size 21928 +oid sha256:eb6c8dbc62475786b8e8f19cbb211383207872e1a19f6627516ab672402c805b +size 21696 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f79c7b8b12de..7f16fec0771c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e5e30f12bbbfbe15fe727e73be228626913afc12348e8d60f9afe9267d9f5d78 -size 27611 +oid sha256:8e5b09edcd991386cfeabc4852e9fc7d905eda1702d6aa8b1976b6ea6c53b603 +size 27545 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ffc7113e8603..158f095b7c17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f404d53ef70649db4d8ccbce7e4a6a3b84f3c4f351746248a5a7b0ab1783f08a -size 25252 +oid sha256:27242eeffb364b41363a5574e9d810b8ebb185a4e76c6db9e01dd0e86fa272f3 +size 25212 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index cd8eeec9592d..77c56d3ff46c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f268af94403287027b6795995b9c700f1f9e1e913c8517cc84b951fdd031149 -size 28099 +oid sha256:f3e9203f187d6dab97228ec89b33a52f531e25808dd531b0ec7841d03c524840 +size 27983 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5a12dad26eaf..2ee80e7b2355 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c94f4c663a0cecf0bf09bce4be07c03f5913ff07a9a046e15f8254f4f550aac2 -size 26067 +oid sha256:4c5acdc815bf40e785984caa59cde08dcf226f5120bdf6ee6bf39771325410e6 +size 26042 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 78b1befcf050..5a2458d27269 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0bbc304260fd7b7f27010bbc544577bdda4b52c7ead6cfa91a741777372c9407 -size 27907 +oid sha256:69d39a1e440f20f29290f39852d9154410a375e6ba84d62f7d3d32562bd7132d +size 27618 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7f305a8caa4a..b0c9e7b1c6f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aab890762eeb92c1c190dc0c680ea30299610c50c543a94ff922740841f0c085 -size 24700 +oid sha256:84a7dd9eef7a9c220f648efb9725383cd88d0f49481de2374eca4c0ebc827815 +size 24571 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index edae52072018..366db92269bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f277acea54339531e751e669878bbc8178eeedc3634c0f29767634c247ded02b -size 29034 +oid sha256:88627a37786f5b255ddec3609662e464e4886947d1a57ea09f04cc9322ff12ea +size 28677 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index fcdafc08a5f3..c27f62ffa05b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af0281195e1faad6f5194add2fad15d4288c33922cf7d9664c819db66ce306d1 -size 25648 +oid sha256:040d9e7f5425c72dc3f9a1fc1a0a53b1683de450f1322ae52d4447c5541f1493 +size 25498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 48025b48e634..e0ecfaf91d6e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1be6dfa9e9fc2fe3997449f5417b6e1c783247ee3a97091dad38a0fa2af4400c -size 25966 +oid sha256:45f4347b94a94f892eb612e69f9698554fc97885d96a5fea3a9fd525993ab258 +size 25782 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 568e2515162c..096a87cd5f99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eaa03a6340c24d96daf79522c25efb5209c8e6ab0f4bec6620cefd5d07238351 -size 25756 +oid sha256:fd34aca242348cf7ce2d26ad0829b7cf9e3b414ee1760248b7dffbe2bc4cbf4c +size 25427 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 9c684f398c43..d4d8f9f7e10b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:34d024751a029bd3d2faada98fd3422d0e04f23d2e7beb3d1adbfa6f861daf1c -size 23155 +oid sha256:49c00f54fc7eea86d98cda71d44d514cff47b31dd6755c4aa98f78852d922781 +size 22963 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4b9a0ba26421..3d873a657ea3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35db9b097b70d5d7e14d225a768833d0017556307539b7ea51fbaced7bd4156a -size 22565 +oid sha256:a9ffb7b7f84c4321066b9244c0aeb5876d0a771f6aab4b8501d922510dd5e2b8 +size 22358 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index f4a2e2e30463..9ab1f17552a8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91d6017e41c572e767d58ca67763edd80c1098b37606b419699fbb4f69bd5a52 -size 22736 +oid sha256:9bdaca1f8c1041e2584c801e25d8d9aadc41481ae5dc8efc1438e9c61dfee670 +size 22712 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 262480632ce7..cc84e277a2b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a982dd1f33e905e61d6e04dbeb345311d6721bff9d5ed6b797b54d566cd48386 -size 19629 +oid sha256:3c2bac4c411bbee96639aa15cb4d904c07630df58efd7723d33f2f506cd8f478 +size 19573 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d23c65a751d0..8a9f104a512c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:72f290e3321ab625ed48a4e384a06009d7f58f15d6a53f2312030f09455d8b35 -size 26535 +oid sha256:b6fa729a14eb48ee602353257ec6d3cfb4c33dd55afa677333239048d2fbd4eb +size 26429 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 026a6319babb..cae591941dae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a55f9f1c03b1ef27397d7322b986bb1e7ac3a257694a56bd3fb3c46874bdbcba -size 26321 +oid sha256:1e9db0f3c0b885670f13751692266d08878d7d4b8f72fa2abaa9b38b489f356c +size 25942 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 400df6a6850a..03f1446bcfe3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d6ed675c8ee67b2ce19ff4bd34373fbde7bf78a05bcfbbb3cd9b2e6bf16d61aa -size 23347 +oid sha256:779753e72303c05d1f0e367adf89cabe4d0df3e5091992b7ac80fc3e3036be46 +size 23171 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e22206022f89..9ac5c65751f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:386809f83582511adc9d913c8e6fd0f7565015f1db93f56f392a752add70ac86 -size 23286 +oid sha256:2f77bd06837e1a4b3bff1463d22295247aa602d37ecfc66011f9972e905eeb66 +size 22983 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 0645c337ed31..616878012401 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ff8375a6f9cb6afafaf4e3b2231e6164244a68ac105f0d21d1c288c58ddfbb9 -size 23739 +oid sha256:57000b2bd999ba553753a17c9b7181a544ac4b0c6987c0b929e75fd4734c06f2 +size 23650 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7c3a11e88840..4a7e31c21819 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:11eb8334fe06bacdc656a7ce2de5c706dd5637e1aefaaa328f6586372ca48d12 -size 20610 +oid sha256:35c24bbcd8402ea4a18559c0c664114afaa05d47a3ee38addd7ecc803e7c129a +size 20467 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index f37dd7a0065c..55d408472d4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc9518ca0cf44ae8509a6b0fbc2af928692135499f93cf4b261c7eb3544ce791 -size 25948 +oid sha256:9af4bee054526765d6aebdc7af2f44f295494ca3f03e96cbd182b94c878e1395 +size 25858 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 010efa041aeb..11717f7c8acf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:810480538363aae4ffd5f1a86827de35c663c437768fea8e07327ee33deebb84 -size 22992 +oid sha256:7c91cb2fdf4f3bbc77faea9113cc64cde1489cccc45fedfbc76b846bdc37e85b +size 22819 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4c583e0daf81..f437d10e4db5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef7c95db4d7773cdb0b233957393d93eeda0abb0cf6c7bae158cd53c0efa5c6c -size 26639 +oid sha256:4168cb6482e036d56fc7720864087c0a63ae0c9dc7ac7b4ea0866e8a2521f4f1 +size 26531 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 63757c9ae332..806295d2b060 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1d10486d3192663330c53b0076219a26b2e8728058630212b75fa39ba6254995 -size 23213 +oid sha256:4e2f84a5a75ff34e86f3cfc9a0dca7ff814ea88655cfe69f2a10a156b7b55e14 +size 23053 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index cad2c06953e7..88ddc5afa743 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f540dcbd49b0cdb4bdcab2ed0e4f8b1b27dbd5ba31c4832ac9d0d98ba30aeea -size 30148 +oid sha256:a96d4aaed99a576112d94fe141e4300578d52d8a73dba14c7e3f154c6d974429 +size 30017 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 76b1d42a6636..95f6565a71cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:86a36ef4d2b6ccdae6d3004f4cab739957d7f4b85d864780bd13b0703e075a10 -size 26773 +oid sha256:afcde1eae82bf3c9d259a15eaf6da1189c82777c348942a53304426f65234bf1 +size 26739 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ac43d4ee33d3..38f1417b2915 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc9b3144a13043504d37432ec38e3b67e342d7ecb18e450c5d1cd1a15c6b5f8a -size 30756 +oid sha256:cf2eb94b960400ae67e1ad29c27937861553d69b5013a97b90f337236585be45 +size 30530 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8a78ce3946b8..9bd03ada2a7e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c63b4e5bf4732652afb5c834f84349ee2d93dcd31f1a5e782ae83aaad9db296e -size 27535 +oid sha256:698178f2a0e5f454b703886ba3dbb1267a95bbe4c5c3b563ae016833749ffe20 +size 27483 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4d6652c41a10..a8d9d46b2d8f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c941ec48a3c9e00a4546062c50048cbb359291f023a51668ea60bd0512bb2fcf -size 30533 +oid sha256:19ce325288b010e40ca4fd6f4293beb85e58e8b197542915cd031c2c2f109bc5 +size 30314 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1cd2bbb9c4e4..74fe3f1507e9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2bfceea634129a4e8120b6837b8aa6300ed8a5342bdad103996ae96bf8d63932 -size 26323 +oid sha256:83b473ced24e909446c9e9655f3419465695902ee3b878cac001ccbe1c212cf7 +size 26184 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e8f85bf5d675..068726cf0179 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec5d8fb6521a03d2912c139067d8d1157bfe2912a9a6c5168171d1da3fe107d7 -size 31469 +oid sha256:4f512a2e2435336b7e5bcb1d32e86e593bb8a69b17d329eb677079da3276b8d2 +size 31157 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 15a038f202be..98b3952967ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e197e17e7818bc7dbae81773f8240603723cfb47ae4bb531c5757cf3bd75a967 -size 27146 +oid sha256:c72012cac6a8888c2a88868f98d7c385c05a72e4664320d9e29994b321bdd449 +size 27041 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6040cb30e333..f198a909ba26 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85a2af1152c59b709541de80515df6479e459fadb144512d5dc771cb517e399d -size 28100 +oid sha256:20c4b88eff77ee2ea464ac6696408ee091fd88a5a64f97ef1335586ec0458ae7 +size 27871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 32e067be55b8..ed8e20ccf165 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0655bdb7ed76ee2503b727fb7ee182d678193af80ef6408538e2047f119ff7a7 -size 28196 +oid sha256:657b59a85106fefc31d4b65f72399d778b02e8cafc6c09c9a87105bd79958a6c +size 27849 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index abc61a9577ab..cf9d9826a578 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:19496278eafb4fb04c3351084bc64c5e560792923748bace997a4ba1425ddfda -size 25378 +oid sha256:fe61a6e9d3f8729ea7e45a2605516f450750706558d292488f398ee8017f9ec0 +size 25151 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fe64ada1ef61..ae2544d1edcd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e34f2a65c5f328f39ca4596d17b153ec245e27e0cfd716d83ccfe5bfafac1a23 -size 24997 +oid sha256:0111135f00a2c727abedea5e83f757c57a58318173c45f058e4be588802c5045 +size 24669 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9b2f6b83dbff..d685e13bfcc6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:727e953bf27561156a529c640c584eb0780cbbb6076d356c68cf1679e09a3619 -size 24047 +oid sha256:2c0ad6f2e0818388f3798357c6d76d31fece5317d827b14a7714190a45f3fa46 +size 23964 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 63934020abe4..56663a34f521 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b873a48a8809519075fda1616203c8d9f4beaf8a1e7c747c7279b9ef57adf3ca -size 21007 +oid sha256:874485d8245fd17fd29cd83e9ad7146436f1dcbaa3027a9118dde4cacf5ce6c1 +size 20994 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9a3b5018f4b6..df5f76efbefe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d7e2312174f6fd39b054960bc47b1412ecee219d1ea31de665952653bdb1382 -size 28666 +oid sha256:ca684a6fe2ad631e726ebcba07a2ecb396a2296b921b1dd8ab75d49d57485604 +size 28602 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index f08062a3532b..5aea4cabeeeb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:388ef3743db193228e468f6bd7aadd405bac5d2e16a1fb3a61fdb6fefe10097e -size 28607 +oid sha256:d45455ca9fffedc74f4066a7124b8543db28cd7d609535ddfc0768b9f2cd76fa +size 28411 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 91116ef6d676..6cd12ba0e648 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7fa292bdf5aee6536c93cf9b3a9b5cc6f27f380a07abe644b79b604761340bf4 -size 25805 +oid sha256:442452d8fc02018c40370850061b275880834c08ae2377c4fdd640320ea83635 +size 25614 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 032aab5d8496..0b817e09e767 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc9b57490ae9bc044bd53fdf3df06e792ac6cf44c0f30ad438b80f9c7da7519e -size 25660 +oid sha256:5d7773944b0677e858816831ff86503fc1cb09c9a10bbf7852dbc6bf30d039cf +size 25360 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index b1249df87991..45efb18d6141 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7a6dccd2281651ecbdcfbd765cf6856ce47ac737a0c036cd216c14b6e4551f50 -size 24637 +oid sha256:5eb68e3d1f8b2c25037ed1b7371da16ae3f82025740b120c60f0871d14aef11f +size 24491 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c43fbab94c75..c07cf2789c66 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e0b4cfc610cbfea6e17226cec46dbe73a72bf3cb739fbcb59cf4e4601cff2789 -size 21804 +oid sha256:6c9073324ba3a16bacbc8c77a948d5cc1e11aa637331abbdffca6220180a0ebe +size 21777 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 147fe0df92f1..fdf00183ad66 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e50b976571da5ce3f03259c42f73214d886aa46b9d938a282eb2ca35c4d065f7 -size 23067 +oid sha256:045ca58e2a8c8e3b7b132e5d879694456d151c654e9c4c6bf4319f1baced9380 +size 22902 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index ed5ea1345a92..d6f4dfb286be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50941d8346fca92ccecec5c8dc35079ee047bd4db708de9f369378b54351da1e -size 20037 +oid sha256:2c4404f8ffda17013db650999c258bca02838486cc96bd5ddbc570197dfa2da7 +size 19907 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1198ad941714..7295517c0096 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5410109964b690d12b711d0d2e7ebd552d62395e7b6192b11bee4bdcb23cd3af -size 23696 +oid sha256:1801af931c781fddfad826987700a20ad7eccb5e1a55f727a863093ee98c7bf6 +size 23554 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 059dd41596dd..cde2e4c15f7c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:777402e399e4df0d248855d8da540599d37b83fd810dae273094ddcc2baa0bc5 -size 20292 +oid sha256:e58d94abac323401077fe80f9a30da432f2ee8a49c83687853d1e46bd0e39903 +size 20170 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index bf87639d64aa..24a8eb06059b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9fa2fdf09e76f28a60ef710b1bde84b3d5fb977f7a7ab23fb895da01bfaabde1 -size 23321 +oid sha256:88bceba1f082277a704975f750f3711509a123ddf5fe49b366f6270c0b45c0a8 +size 23139 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index d154e9443254..169dd23bc07b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:589a80311ae6d5b48e63460260ef6675613f0fccb1b55cbce4582edc1068c79a -size 20435 +oid sha256:68413b06c28e83bbe5931f3ffcc9624790648b129444369415cf233ff4b159d4 +size 20318 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e9c8c8e89e51..d517c015a062 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2babfb40a1b462d0871c16b55b35fcc15c8e190ebabf98e29bcae47acaeddb67 -size 23796 +oid sha256:bb0809aef35f6598c464bc1d740fb6f1b282b809bdbbb55f853d542732b71f96 +size 23686 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0d73f40ccda4..f1d18e50e422 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb90a7fbacdd5eafe0eb6ceb8f2e4f9c4e07936de8434fda7273e781525ef21f -size 20882 +oid sha256:6f425ad50054969f0d91e8858ae34fa40a7544bb396e98dc079c8fb54a97d189 +size 20750 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2741186ce2a2..9133e6eb5e96 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:197ceb62ca0186901c9e1b3d3e0a78f181389daf68864c132c76748a7420c1d3 -size 26740 +oid sha256:c777d29e997434d6780b559d7c86ef3cebb8111b02be107d1e70495bb26a889a +size 26675 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1d07546ee5f7..26496d26b9ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c035ce04285e32eb31ee99d7310fa51f85545284725976639e83e42fe631cd7e -size 23701 +oid sha256:f74230a182d2b760cd150cf2bd85fd45605a4835e82504d6614fcb7861ac36bf +size 23720 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2cd118e7b53b..a9c0d39728ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:884dcf21b0a559c0221d78d361ed6fc120a863c1a94ed9bb7f4e4114b35ba452 -size 27345 +oid sha256:ecfe307447e6727a1aac6ff5d871f25a4284262e37f10ee3ea39e52f1dea8f1b +size 27216 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index fd929b4224f0..08c761de01e7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6d006e2c779a5c8237bdbf20367f8728ba5927ac3a9636ffab63b2da0f7d0765 -size 24297 +oid sha256:480111b215466cd93c451ea28131fb4b95837dc60a697bb4460b6bfbf0794701 +size 24201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5061711ad8c2..fa19379cb624 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d3d1be52f61f0608e7fa819628737df6b28d8313c67217af0e005f0b120a9b8 -size 27417 +oid sha256:6af5334350865bf675f590dd025eb636135435ae4bac8d104248549f7cce7d93 +size 27314 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index e9db5379a67d..e24dc8de50d0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f9097dec1d15b7ce951e85ef1366187cec8b2187b3e9f47e495da3e1200f8f5 -size 26961 +oid sha256:a92e55c29156ea53b304601b21d5c182d06417946295330f0f199d34e31f1de7 +size 26923 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 633e7560eaee..06a1680521dc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b75ee28a7199f7d81f813dea543d63d15b492b9bb3e9e335bf863855924d3035 -size 24286 +oid sha256:a7777f5a0b3fe53852d22f3fe60eb05426a03aba95407a4a9f279b315b824472 +size 24179 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 771f079cad6e..da5be556cb2b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cf9782b2fd6340195c03713577fb58bbd478ceec522ae35764343815326094ce -size 23827 +oid sha256:1a2d14238a26c085c86c805c2686ba38ceda49c4a211645f90b53e73fa7d7323 +size 23837 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1b73d31fb85b..98c5c3c6ef3c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8e51794a596188d959d3c6ee6b8c67bb671ca998c06e56f597364fc9a7f62659 -size 24315 +oid sha256:c5fb8b0a1b3f2c90849a15d2b326741129af54fc718ce9e2574b111f606e5b5c +size 24330 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index da3695107120..de5543a154ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9919520bb564290f532fe0ea11e54300f28196a3a4fb0305a9eae1146c281e7b -size 21203 +oid sha256:79d3c7524ef71ad229ffd55da434a83f6c0571e37d391c2fdf6ec288bdfe51e5 +size 21126 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4930b51661c5..ece826d907ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:799e18a70a3c2ca7733c0bd25334b72543ea57c0ca03472ab880074fcbee4a49 -size 28293 +oid sha256:161ae43cd407664f54a0f0624e864788e34d9ded4799a58ab3d3a5e06b083d4a +size 28219 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index e38654b88e4d..a3be134e1ba4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:247c548c103ae39427a162281fe6e1f7044b48cad8fd732af6a1aab66ee88421 -size 27645 +oid sha256:56228964ef2b8e414fce40069358b37a28f1244ec237835b5379f44f7a335eca +size 27444 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 03bced778ecb..50360c2e8a81 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6a85592a717899e18e503b7fc5d175c293b28bfe8e60290c286c4f1adcc9074 -size 24755 +oid sha256:3550af72d8e0bff8b9ce5b0588ed137615163461f70d760f3144c6be5640b78c +size 24550 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d58ebc403d9f..fa5f3f4c38bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae3a8af818aff1cd2cbc0e94a33f07962717e3362ee5f535890d68644646dce3 -size 24419 +oid sha256:3cd59daa69703b284e5dde356a4f7e8f93005b8894d29d2c859ab713953db3a1 +size 24260 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 34e4e3e6136a..47cdc9ed92e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5dc96dd741cd107fa59ce388f16bb1bb55295388c24ec6dbd32e6fb0d437105a -size 25197 +oid sha256:6f8d86e5abdbcf79b33854d2b91640842788d9950795ec6c60ca31c1595dfc94 +size 25230 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7c55411b9be7..089ce74bd4c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f3870e3b67ab4be581db0e3bd8caf57c80c41a9c54bc94aa0eab4ca8bab65133 -size 21966 +oid sha256:b49d2881ed913f615393916c46cd4214144c3538ad03fd03a9c56a2a88bade71 +size 21819 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGenRopeQuant.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGenRopeQuant.cubin.tar.zst deleted file mode 100644 index 7db06b6c0c71..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGenRopeQuant.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:cefa408f80869c8c267a04492d3a6ca48a142cb5daf4578d9fc0c652ea9d2e64 -size 38734 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGenRopeQuant.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGenRopeQuant.cubin.tar.zst deleted file mode 100644 index cd7de842a0b4..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGenRopeQuant.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:add4ac8e519d5ca02041caa00753481eb2f24bc236fc4eeb887a09ad28da97b1 -size 33814 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 73b1d5ea4903..f00d072f0000 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62d2aaf62d41ad1d057fc460bd4f92f260428eb4ce68ee5759d81435c7b0bf17 -size 27280 +oid sha256:21a62beff86487b163e9e4ec9460d319b983ac3813d84aae35c53cdbd43db6ba +size 27200 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d53ff47c03a0..adf3275b45db 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:761f683128ec07a5170179869b2e9f85b235c5a661d362f1ff58cf3e7ef12e06 -size 24536 +oid sha256:441cf98d892cf1dec7ab1769448ff10485c02a12fd0e16e13ab1dcc459dea6d8 +size 24413 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 649cb3f31168..c24aaf6746ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8795125fcff1d540c56872e6680917b74be9a9ee6c6b7079a6af2c0756d19057 -size 28164 +oid sha256:6fe78980936dc57b8f154ec738570fa2bfcc9602f967ffa5de6851b598270afe +size 28058 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a95863032ac8..86662cff0852 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f68ace9addc17ad55f6eebd780138f56b2eb3e63a379a706ba919d00e1b90830 -size 25060 +oid sha256:c04690e3006697b14546ccfed46452a52c4bbd6cf2c37fec5426beb54212f2f9 +size 24951 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 30e2573ad8b7..b4a3630db380 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c01ad0dfc48049303dc32665fe7c67d3d44c0c84f8990af322675417c96b3672 -size 29280 +oid sha256:d3965fb9e6a216eae5de8c2f96115fe236ec326f8c4c3be3db0a413405c3658a +size 29196 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e03936da925d..d10ed12369ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d9afcd0c18aa35a7033aafd886873f0dc92232d7d518916518b068f32dd9983 -size 25025 +oid sha256:44008487207247bd7d8d460bace05267d2afb1769a207b31142b933a2b056ac9 +size 25073 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 102b2557bf93..0d703dbffe95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d2da894dc419092a2eada520a6d1a4f885a54049cda5d8ce9d66736d75119df3 -size 29801 +oid sha256:b55f9de150ac5246c90d0603ff13e47684209f1c1979430702cc9f0897562a5b +size 29690 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 338d2066e0b0..eaeb0953f79c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:085a8cec7ac4d89523961538b0f992b338719bf1951ce79774b610b912bf06fe -size 25774 +oid sha256:8e7fed03e8172290d923d87771c9efd3f173e1ea72c007cb2dfa3c79c235b6c3 +size 25734 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 686704fd1879..e478fab41db1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28f778b8a372425ee6f1e2e26807118a2269798bbac0cfc08ce952d83bfb37ae -size 29216 +oid sha256:1d5ffcdcae364c02270cc5b6326c0969b0b1c72c93e4d7a5401cf5170c2d1b2c +size 29127 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c8ae4d894b35..0e94d3b56cbf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:debed2646ff07e7bac64e01bfe38e192b8f0db1f84b227cbc81553db9991e3b4 -size 26263 +oid sha256:e6ddb026a101d950b19b24816a288fa66b0a5601f716dece054f767754d217e8 +size 26082 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 616f77f3a54d..df0547634264 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85b6847fd27fc05f5b89bfb10e8a2d31680987781a1fb29145e2b1a40a3e12ed -size 25426 +oid sha256:2bff8ffa145c1b3320426e4af567119ce407ab6390c41a2f4eeda4fa3bdcd354 +size 25346 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4eee7260849c..8a5bae801256 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:31640e2a2119b2b219912817243fc5ab367bc5616537488762bd511a8acb2b32 -size 22362 +oid sha256:5e1327d67df90de3f0000ba5f916673fa512c814f6856f25df2d6e48779e854b +size 22352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 0d9eee8f3e5a..6023e82fb1d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:540e2f9fcad4cc7e49afbfb7a41a122fe598b7f1f5c0a9545450cc7a88424540 -size 29623 +oid sha256:d5dfd6eb2d745af9e2001af8412be7524d50d55ba464cfbbda676c971a46183a +size 29594 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index bd85090507a8..cda2df5e1f3a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:388ebe4071b754b84d0d418505307c77d7ddc36865c7fc239386e572db13f152 -size 26668 +oid sha256:0409facba7da60dbe5a53bfe8679645cbf1c62597077cd7277d758bafa31adda +size 26560 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9ab207ffb756..c745761b83b8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f3c2cbf69b69ea320d97e066179609672e75a06bab09c51e33847d432d69ccf -size 26022 +oid sha256:664b715dddc88c379b08a20bbf5d98eb257de2793ec317b921ea2bc68b9a894a +size 25949 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1c59d2d79e93..605b2e7f7ba2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9f9ff87e6fbacb6fa68ace9d51e8d32a9105535b710bb077236c8f387d6a41a -size 23031 +oid sha256:4c132514ef3fe783ed9de03e5d7be49c83c562dfce81b03dbe0393186ec4b803 +size 22999 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index db3c8deea170..0495350b04db 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b735cdbb561d957ef5423a1273cfbf5d08845202e80c9d1802a8c6019d28e333 -size 20947 +oid sha256:33721c781902375f859953350b29ce824ae4860d20984662812fc4ac097fbfa5 +size 20801 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c5782ec3df69..e0cf235be6a4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d2a5d36821e3643ec31eda3247aa57752741ae371ca587c47d1ac427d126ed9 -size 17906 +oid sha256:02a32d270764c89ddc7bfc70b4b086da68d30d5ad5c3cdab2f3850722120a476 +size 17807 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ddbf8ed534e9..897b993db942 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ea32840e069cdcf830fb99d40f7578af0871c4073826a47d19cd50e45b80032c -size 21677 +oid sha256:17939b9d5898703016f0772f2b8abb8b446965ec8567935268696043298b0995 +size 21675 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4783de2e0c23..92553a26c3fa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2905ba5af25ff5b712513364ce7cf871f42e905c0f5a00b4bceb101e712f78f9 -size 18909 +oid sha256:816d5a79a562bfffe9557730e5615714b48ff2e170a74fe33e6d3d6e4481d032 +size 18764 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 93ecca3ba06f..42c402ae017d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:792046bad31cad2d1dfe53dfeca2451f5b127f25ad663c8b2bd9e81550a9ae87 -size 20722 +oid sha256:769c70c941c6ca5085b82ca7a0beac7a52191be078d91deb10d8e4ab0a0f44d1 +size 20608 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index b7eb2764b4f3..d66ba3b2c648 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d9e57cf7e82027d08e0485d114476875288c5f1a3b4c73b80765fce123b3dc94 -size 27509 +oid sha256:86a30f56aa1d9fe9b188e717ff80a3d1f0b03ff282751f007562f58de48753c4 +size 27494 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 7fc5b439c1ec..446c21bb2a0e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b63b1a08da5046013185e6186551b2716e27c85658b55b2ae29a7b42b0c978dd -size 20614 +oid sha256:b860bc7b431cbd12530deab127dd7bd28f33e55123267b75eeff01966427cc7b +size 20507 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 3c38331088a4..2157ac135d54 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d623333b8a3273fa7d9b42160aa35c8127f9040fd3574c264ea79bb482d55dab -size 36581 +oid sha256:4c7b8c62f6da785ff709705c161cc5539bd140375e37edbc60b46a39849af531 +size 29762 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 9e57f3b8e72e..4e327f7387f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d245eca6e2e3fc930304b1226720e7cb01752942863f6151cf71a8b1fa8e738 -size 28242 +oid sha256:83747840493ffb8ff7c8702772c783afd40e0e3718263b92e58016eef162d541 +size 21946 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index e67fb8d46f28..311e060ecea8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aeeb787fe547941ecf992e0a940b309b908d173ab02fc82b8b1dde72d80709d5 -size 28949 +oid sha256:3ca48c75e7f927de7dc83cb58f5e37e780d5a203e00ccbf65e29008e11925200 +size 28856 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 7431e5789c6b..f175adfda3f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e98707254117357691553eb601a0289deacc87652ff2c6bad76ba23abf6c81eb -size 21562 +oid sha256:e43fe7f5afdf7be8381577f2b1eb8d23d0d8ee8e20ac3549afd086516173dd9d +size 21465 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 54138f447419..983600c8cbb2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59e9288cdbc29814a314c139eac864bd68bd849dea4c24ac3e5764e3dbace408 -size 17867 +oid sha256:f2aa7c5c7ab07e4e73bbf1de78c43a44ca37b278bf5ce965cd480ff4fdd76a12 +size 17734 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d7257cf73af1..b42e6c43e0f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ccae57621c46f5d2938316b6c51dfa89020b3b399941a8df32cb28cdaf7feea -size 21722 +oid sha256:e14d1d3afc9c3e0520089e42c0c31f465cf47cf356d3c67249a49813709cdd7b +size 21616 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 761d83e249fc..91af598128ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aef9b01061e5dd9338e6050ef770e6294b77ff06e142736d2c00d5029462390f -size 28212 +oid sha256:703146d410d03b0656af2c546ac71e98687e10052546e02b173274dfcad2a7c7 +size 28134 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index c6679a6b4edc..31b8d0c3c531 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ad551a555f6cb3114136f0b75b879a7b251892f611d886da85d27f2b5fcfe5b9 -size 21830 +oid sha256:7409cf56d3868ff7bfc5cc00fffc550e2370c9bac5fcad476aed837fc63bfaf2 +size 21729 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index d258ab475b28..234ad7dbd06a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b498b00de57bb440dc8209ae1aeae8d3e8249d921183a7c218d4bf5e4c25798a -size 36880 +oid sha256:582c85f9c1a59c8465e294e37cea3dded0365c67848f19195af17b86f39a8176 +size 29807 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 595e7f731f7e..544e1f4cbca5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b79ef92053d1cf32119b9886a3387302ea89f51200d43727b7f5151abc90c87a -size 29895 +oid sha256:7b32d9b175e8900934eb5c2128c9d52b75b41d9497fc25d53657243c76b28b01 +size 22935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 15e634d900d5..2ab2bff79843 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e98bd3b04aea8c2939ef9fd7a6d595617a40082343d16459fd7c8c7e3a1717a -size 29247 +oid sha256:711ceed8cdf2aae8df3c8af2f3e873e52afc4cb8c4aabb286ffa045ae5d5ac97 +size 29222 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index f01ef6eb8cb2..0dc8304aef2c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9703b5876ea00beb1d642b416261a4a2f56c0de79076169261aacf13b4635a14 -size 22732 +oid sha256:7a688581141afa342b4ba17f65651fe0c72a7fa21c5e94294b596873804a694b +size 22609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ca237cf71b59..c3cf2552631f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9fe24c43f4823f0186fc403ec8a2d7cda22976624bbd6519990f54a0ac6b629 -size 18783 +oid sha256:1962df7d43c42e53c41e9a4cae293d723ed480c8ba6165273205f6e4c917ed85 +size 18664 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 351db8a97eb3..9d9504f5a1c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8c0c60231c18a08f7da11fd843a2667f5d84b7ad9bc19e2b0da4a934665676f1 -size 22993 +oid sha256:3767eadeb725318e6d65d89f13efb78d3a6fbf66fed0380abfb2abf6a83a253c +size 22813 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 59aabd3dae60..a5e0ce597a43 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c34f5e937afbd95aa24c091a2dad81288144513540439f312feaa0703cbe2894 -size 19961 +oid sha256:332ff295a34660182d29aaf75b2436989a6fa971b1e38a50ba59a8ae9f41f53b +size 19777 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ca04524a174e..b4b81d94c6aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f84b3f26f34b673cf2ac20c97bde088204c6f0a9e9e73b207d313cd27fcacdbc -size 23924 +oid sha256:9d71812f1f78247c76c8b3b39b27a4d664f33dcae9384c1b94e79d51af5ac9bc +size 23709 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 3cf7e50e2741..1a86cff6c1cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:60d7c4cbdd7eb0229003be687a5d292ca40880acdfc7fc800af861672f3dd297 -size 20947 +oid sha256:bcb5ad321204f521d6f7a5cc9cd03a85d4c54e52a330d7e87be32e624263e321 +size 20727 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e2ed8d53fe04..08699c5837ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aae28183a3bb556d1e2236d4f5690d75951c92b4a163bfc4d71af835f572e315 -size 27140 +oid sha256:a4a8667726166cf7d8b1bd707b406b9beb4475c41342499a4c03d1803ace7bba +size 27066 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 8d11bf740534..75390b503dcf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db624487a7c165d7144fe107e3ae2adab8e48239f458f8b4b070966b82b1bad2 -size 26079 +oid sha256:b1ad79ba74399c6c5fc4a24094e34ab71637356a7c3fef31813aaae2cf3f1e22 +size 26032 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 84197f02517c..b92567f34dc6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10b9cd9fedeb8f5968070a4cea48eb34d086266feb3e9a44ad2079b99d38560f -size 28315 +oid sha256:58c5c58de660183e002854e1f4ce2a9563c5d949928826af13f9f19cddfbbeb7 +size 28303 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 91cd4b4095b3..1967161088df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53f6339396f701bd624837994772a9a9e04d611face947c196bc0614aac315a6 -size 26993 +oid sha256:725bdebe5a4e91d2497b29c9f8030400f8380ed009b581a0eb163595553fd1e0 +size 26925 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ba9f66961d8a..2f07124c71af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e474f4be23f7eb47f45cce1c04ad77aa8689c9abb56f9b1f2ab1878b0ebe9683 -size 26002 +oid sha256:eb152e99e8c9161154eb1f437ed603628cc2365fe7e3e0cb9055cc8c4a69a526 +size 25931 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ea82765b38f2..9c8df91a6a07 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:adb12e3e1895d432efa881b41a14554ce6f36b9befd8655c409ddd41862f377b -size 24764 +oid sha256:3275e2c7cc0286008a25dd9d3a3086523d784a162914712cfc607f1ddc9ed154 +size 24673 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d5df258749a1..b763d2526e35 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e133f1fc575e1bd3b7d7447f1d646eec2deace87aa3cba99d56fa433094522e -size 27311 +oid sha256:08792ab25cdce793f68d7041b02e4692a7fe7880ef58b2831d54065caea8b5f9 +size 27293 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3deb2f0ce3ec..a2fbe617687e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:23518a0860a74122d16c612efcc97ef0ed37e87360fa8b6600b5b364ad67e79f -size 25867 +oid sha256:4aeb9a7e21b0156544a7e3fffb988ba301a85ebe2ba4354f3ed5d65d77b2633e +size 25660 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 67b9d1f94202..f9b726c8c9f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4195b3ebfdfc45b17c043536517af703170dd066916fb72c38aa9cc307c32ccb -size 23818 +oid sha256:8805e02e4a25414d854abddc341acea00cb4abc4df691daac541b999109189b1 +size 23641 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9869e9d10af9..13a424213281 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4157095eb1f68e640c94f964031f63afc772a9453fc84c5807ef33830285adb -size 24187 +oid sha256:731eeab7baf57c80e9ca58a70a2954444460c94baf57af1caf66c1330fbdb19f +size 24181 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index b3b33cb0e739..9d0215426f63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7c06e9a538408d89010429303e539bb10a177a16b63c3a2f8169b050ba8f489a -size 21412 +oid sha256:29a2aea691d2ba840bc253e57577dbc1d88887625d5f30538708ed6c970efe79 +size 21290 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 919e29a854a8..cf5cf15146aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:82346e91e0200218d84d3a6f8d2d320de0bcc9a4cab0e18f37326b12f17fc55f -size 21404 +oid sha256:58e9090c28733b2096e1ed098488ec90fa8669deb31513fd403c7fe7e61cf300 +size 21390 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 765780c85819..79fe12254bfa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f44c771ec88ae9df799d5c161d0821cc8471d413b116f0c4aac9582afd0039bf -size 21851 +oid sha256:a1d8e36976ceb60ca8180cd4cbe40f81ccd17cc20eba566828cb6fa913598f42 +size 21792 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 608b7a708843..9cc2cfda3e29 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3091e50fc16c38a63c099596f56ea98f22deeeff9578099b35a2b8fcd3747c3e -size 19136 +oid sha256:e9fa125b9cbe9f996e79b832faff2e1cbb112dfa296666b7c149a6b089a047c2 +size 19111 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 5d751d04eb37..2b4bf8e92c94 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e42bd7d2651f807c150bfff4915c78b14be7d8c6c3c7e0107c3d7a9b19ba6801 -size 24851 +oid sha256:e9acc9bae948760cbc672d56fa67b43cbb1b79dbb9537de5fc11002ceb2fc572 +size 24752 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4594ae765b44..f953c8bb73b8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c45c73e2b3b5f29df1b4095c249e4351e7532858cde09a8e40d96767b71558e -size 25452 +oid sha256:982e581c6805a120469e9dd7f43cad2a29c19fe67d1913b3a5b713dd12a22f60 +size 25394 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0e9f9fd15895..dd93d361c266 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:851a89806867ce2d81302c4195d62975fe3d0a42bb546d870e835c39471f4622 -size 22376 +oid sha256:d1c1beb727bd85ae69e5d3ba96792d0a89924277d23326e40eabb30cb80aae99 +size 22251 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2c5d9f932676..e0748e0fa11e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9ad94dfde7da6d168b841223f004c4aed93c636f85d0046731ba4bcb296d2cf -size 22387 +oid sha256:a411eb78ac93162c283fd277f38bf8b749de70f53e206309cc02332947b369a9 +size 22397 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 71ec1998f809..c1d37e03af21 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:778a37945e6819be08b1840f366bf0e1af90bb10c748822da2c7200c2482389f -size 22884 +oid sha256:2c824c9f088c759ea96eeb2462da20f5595cb9063c5fe3ede083771d40653890 +size 22757 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ee053a366aeb..a86756f899d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca6de14851cad484eb7ca08c135257d58173fd7388d0350e24d86b2920cd35d0 -size 20123 +oid sha256:85a3169a9263c5fe69459001d76ddc7f9ef0ec4dc9908504507372551be11123 +size 20178 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1ba3d434c6bf..c737145e45b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2bdd092413af0d2a8e646b28e5499d0539cf0e6e2401d29f3d6489c64fae7ad0 -size 26707 +oid sha256:4b8d0225b26e2402c8dcaf6a23f0c04330799c10287ce6194566bfde508b842d +size 26651 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1c17623295af..281b19cb2572 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d47451796f1c1eac06eb16a80877733b267399db85ebc9a706afe740c57b3cb8 -size 28039 +oid sha256:66bbe886c404200ca56a7d24462030a0033d495f0b8d9343cf02f2a4c1ef4ebd +size 27891 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c160e92be0d3..954706e05ef1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a768ece8b80fd2b515c646d70cb7841d52b5832e45d91ad84e1cbdf4473fe1fc -size 25749 +oid sha256:d751861f40159b9660e7fff47a60ea0f0adfe638360d17cb3014728bf7e2a805 +size 25564 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2ce0dceb11b4..c32981d5b9d0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e881acf0759e3aa4abe29d092641de51a0a8ea6edb8f44e2585f81bce939d69 -size 27097 +oid sha256:8528e26028c2468c70aa74c6acaaefd5b14c84efeef68907c1d147bba4fc60f7 +size 26916 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 71b3a5bfd727..6d6bf3235f87 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ccfe1fb68388c5ed365892a8f630e7669a7a9b3cc2c768bde6678cce547e5105 -size 24051 +oid sha256:25b72819bf382c1b5bccf93d68295e42f4f38fb9f97e65b748ac1696be20eb00 +size 24012 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f97f095709b6..4e12a03a5ed3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d370b18aeba3706bd523eeed9b504a7bf56ae827d4b052b3811097cea57a411d -size 21038 +oid sha256:8acc1b95981349aeb7883ba35e4dc34b3a44e6747a8a83c694bac079661b1a2d +size 20996 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2389a0400bf8..8589057548c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5e12853ae45e1fe54f3c228cb9951f62c307ae3f1145817a4ede29a6e3c23cda -size 25319 +oid sha256:6b6bd4c1eed6037ba29003a503bfaafebd6af57d7395ac020f4f8878f2ce2a10 +size 25247 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4d492c69874b..8f6cd0a71cd2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d51acfdb898df706388adbf93610f1899618721285eaa2436efc2ba99b8a3039 -size 22275 +oid sha256:a198cd656996b80c6b1103033b8e6efa3bad53c7d1fb088bf7bed084c4cd83a9 +size 22226 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 538c72f627bf..ab1802c9ef30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce83cea9680794ec4d9418f8cf089a29bbe52552774e2ad239c5c34f2ca984e7 -size 23783 +oid sha256:e3b6ceed98d4b550ee4574dc11c0951e5356de65ac0adec67f302625da2e4aa6 +size 23650 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d7f2318ff2a1..65b6c8397e41 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b31493c92c7830397d0d9aefc5d04791d295bcc118ef9e5703f6e4a4f3359e91 -size 21272 +oid sha256:7230923648a884d813130f086b9934d0d07828c93eebb4cab9eea53064d87d0a +size 21131 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 230b24df4991..42686b9ead74 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3499334870946b4c56d9e8b781eef7f145cad6032af408ac5d8ec5e78e239aef -size 24832 +oid sha256:3f2cc4e57e335d46f85b1572e9578a52c1558838a562912dcc578d561342bc18 +size 24687 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 454c471adc0d..7bda4e2071cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0c78473d37aa8176d970b084ba3372fc2712a5b7c0d8cdd37e0922e7c2cc0c6a -size 22344 +oid sha256:946f382bfd88755814f4364f3919fe92e363403538c2faee8e80f496ec1a49f0 +size 22245 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6d217d0f1827..71f79db19b4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c4873a07530a1791a60b1ba69e661acdbb6b445e791add1354353efce55c4089 -size 29701 +oid sha256:1e68d8527c99dd784e6a156c7e5b7d45344acf47c21849acb6e53664c61033f8 +size 29466 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 64902dd38902..12d9ebdc2a73 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a29a052c7e2b8bc8523ae527718bf1684f28b64a208dbf4301b0350f89bcfe43 -size 26937 +oid sha256:334d98d40887e1b1752d4394382b23c9cb0b18a8bde4cb01ee0a6c05f8153558 +size 26828 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1c15e1d98eb8..df074ae50f26 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:64df0e3a0c7273745d4f34736ed433ba4999b567c035c24c1f8fee3fe26f6227 -size 30896 +oid sha256:5f6a04acc96af029da116c66bf0f2f07c1915e45fdf0977d15a765848eb3de60 +size 30840 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7d0b9102b635..fef504fc0bc8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ad886cf0a35cd30932ac4c5438b04a0a8a60578c7d09856e2f0ba79d91289a84 -size 27838 +oid sha256:01644ae2a7cda27c81e65cee71ff875c1e93df5a5a1a9169778953b8b0ed4380 +size 27788 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a9db169a9d82..a66361418157 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9bfdc09da4998550a93eaaef994cb56e19f051c301d94b815f507911fa824ba0 -size 28688 +oid sha256:75e7c40c9df56f37060569bc578d1f07e2675d49349b6e4c6e6ab452c927e924 +size 28530 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9784411d693e..47d1438eb397 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3b0d12a6876b438a384cf5276aae21c4abec89a8b5a54de71b7a5bb2a27786e7 -size 25766 +oid sha256:2c26ac02af30a3c35091ec34ab381bc81fa66b1c6f8a61a823d818b90b5c0c79 +size 25600 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0691209a7d31..93d959b9cfdf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1262a27ca147b67101a05f1978b3b2b7d888a3ade9b19a0d5312b1c923127856 -size 29887 +oid sha256:49aa848541e30906e754c7cf4c809391dddce5c6b23f01b90c13a398433be62f +size 29772 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ab6640400a0e..971d036648f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1d3fdc2f8f40a4cd3a1b5470c72566d18c4bcf53388555316e74aaf933ff5f4b -size 26589 +oid sha256:cc6dbd6db17b4865c3c55d02064c32ef603746fa7b7a3af571ae695794a1d072 +size 26468 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4437034bee51..2c4628750883 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:43931e681def436c1373e6756f58bcfb642a88fb3790ad545a021761d82efa1c -size 26452 +oid sha256:58e7ccf0075292d723f9967979deaba131fe7a9e3cd4fa90f5e4811b68d4d346 +size 26243 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index ac36080ab95c..749567b835e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7f967baf64ca5c130a7ada3b8d59f79cd6bcfc0eb2f8f7acfbe552a57d853020 -size 25571 +oid sha256:36ead2844ad754df34f9362d933f0b732ca082cf3d612d2da9f5c26798aec705 +size 25507 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index c9b8e66d9134..587ec3f7958c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1554164d815c2a7f59ec61f4cd93e8e061e5c9451da59abb6d578f902fbbe994 -size 24042 +oid sha256:1fbdc3af0990859c85697c082e6b97e2f46bd0d36d1b85741eb17b3101a50931 +size 23838 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 51f25fa5eaa9..72a150f92688 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85882c9420654fee5ecfadaa20ac6825819d1dc931eae9a6c08fd942f5ecfb8c -size 23707 +oid sha256:bebdbff2f5a1d98676a4b2e9daebd3477ec8a9bd1a98d1a11dae06047cc32185 +size 23615 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 7857c0161f74..e80c0a941cd2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2b5254ecc00fa93666a5f181908ff4b93a47b6155d01b97a0a4ec4b3ca897a3b -size 23152 +oid sha256:2d219d588f37909897f515f0a83ff41d9df57048d1f802c4563fbc93ab16d62a +size 23125 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 367195294642..173fb6953494 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13b28e178f85c8965aa7708824adbb67505254eaf4febde09112765de7e55df9 -size 20294 +oid sha256:c3ab4c670766f02fdc07dbeb64c939df34ad62f3b2adf6a1678102679b01d673 +size 20352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d2fadd3edc9b..76ca34d7f02b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db876317d7527fb23862e3aa1fe7116177b4235726b30a7668865f250559e8d4 -size 27146 +oid sha256:bc9587b9425667daed02642778a49da3ac7572e98e0dd4caa1c04c1622698929 +size 26955 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 0182d8d74e79..b7c046964c8b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:297590556aed9d118bb0a5856d0340da641514b54b29bcd37dfd1317e817c7e6 -size 26543 +oid sha256:0115a1febd65ce65c35c36913f9c43a8f47beb5b54040c593532c31c6db16f4a +size 26533 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2ec44152e44f..654db312c934 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec36ec60e428d6cbe94f91929c959273e3f85735a350b2dd687fad106074ff55 -size 25045 +oid sha256:e4afde4ce949ba8c68da9f3a74e2d4083b01b8c75f5e65f5ea945b43ddc92ffe +size 24815 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1aa9de25ccc1..acc1ae3e0c65 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94ab19b93496d34edd918d7744863710ddd474fc39d8608959fa6d33adec2df0 -size 24981 +oid sha256:0dd57eaa93e0a4fc3ee032c186e9aacd76c24a99f0b423dd1f319eba3731aec4 +size 24889 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index ffe5f5ba8994..b627474803a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a2a1860dba9c2704716f62bab0dbce2d8a529ee44a4f542be8c0b05a264c4e8 -size 24102 +oid sha256:77cf80f1504b5398d3d69fe6182e2f70e913c21b016f9c18350f734f0eb0909b +size 23994 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 96e00173311a..42136229f9e6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6aca42a4ca178c9254d82c14029c754fbb8340919e72f8ac7e98f3358de43783 -size 21607 +oid sha256:fcc9ed34c94498dae41a77d06eaa85a1654df547a4be982d1f8b2473839af6f2 +size 21690 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index ffc8e39bcd03..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:97b3ead7aa95d965ffb8b4c65c248d3cca1beb771d4c98b39b480241b3160b6e -size 33531 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index f691847c756a..c72ccf34164b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a753a9167e9ef375e1a3ef8204231f74c43ebc4c264fe14535da4beb09b3f73f -size 27535 +oid sha256:43bfc4dfae89f52c1c012ec64716812855b50734c6615ce45312300c6a19e901 +size 27525 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index f5c4a7f78daa..bfceb8923b3b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:549a4b5159a1c3288544640e78b14c64a6ab525c334d4909294eda894441391d -size 20652 +oid sha256:12d3c5d35d725705675a216ef364baee20a3067bfe8695cd73791f61f576d5a9 +size 20548 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 66969ead4d05..bff52480dd18 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:580f528cbcbcd7acdde22675c958336fa2d77df70db5fa431e0f38854d42172c -size 36790 +oid sha256:28711569843678a77015924899ae6199fdcc6080a6b589c983bec3455133ada6 +size 29828 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index fa01b2ed3aa1..fe0b3d025474 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:277d6e50a5e366c2867701e5d9b295b3b0157fe028ea4821fa7af4225daec87a -size 28234 +oid sha256:a50e03ab65007b496ff72964ad8f0cb9f9b51df3ed661cf4c8078230926695cf +size 21953 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index a4423bc100e7..ded213276893 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:221d91ffb86ec74fe6b6fdc147547ece59b5ce7ec3d117f7ee584987c4ab7620 -size 28970 +oid sha256:ae36bfe692123c06bcd7238caf41dbdfee983d16d9bf8fa7f6fde9c3431194ca +size 28876 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 3f9ffe6bbec1..4de70139286b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1d7d868107ae03e4b008d529c8d5bc3e3b4f7d19be12b15e803486e0d22a4673 -size 21570 +oid sha256:a7ce286ec89cbe1fe423a38c6a467be86aa04a28f021517358bd1527d7e78a4a +size 21464 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 47074a345b04..4c3bf15625f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:17bfb3549546c3b101259bd09a873781c6d70b53df0464ca9198889d577f9272 -size 28222 +oid sha256:e8b47e3cbaa4574fc1e224b65ab4a07d1ea8b7bde132523ad2f2ada42ee3f995 +size 28139 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index ef710d70744e..edba92597267 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b56d79f01e845e463ea95ac4af2b41d2d30af5804b1fe9011d2d9fe050ee2db8 -size 21822 +oid sha256:0ab161870046152828a4f3cd7060cdf10d4a3125a24c4acdd044da2baaf194b2 +size 21717 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 399b5ba89b7d..d714155685f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:620a037884a9903ade1d0312d5b461461fa5bfaa1749366efe329209defb86f9 -size 36896 +oid sha256:8e4548ef735391a7c31b3d04ff09b63060fde6cc86c98626fa172edd899028cf +size 29810 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index a52180241461..452e08ff94d7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb4ece7b829a3b11f8afa135b92ea7c6961d4928100054f94c41c5250abe73e4 -size 29886 +oid sha256:5234dea9a7ceb8e14e829eb01de873cafb08ee617e23aceddf6074f1f877f9c0 +size 22931 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 2413a1732b48..1cefdd2644f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5e6ecfc4487a380be6d59b82fef6751d33a11cc6bebdcf0f400afba8b2e0ca6b -size 29269 +oid sha256:e5cf3d65ab2b45b65b5d254c8c6d6a2432785a33b114ed488932e431c7139606 +size 29233 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 7eab23fd9ce0..8f46b4679427 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ab3b3bc4d20dea973061cd4cb54120977866132f265da139669c98e4b5f62b7d -size 22729 +oid sha256:8d85ddd2ecdab50e04dac2aec035cb1d06a172c7e0f2431ed2f88abdbe981f42 +size 22606 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 355cf7df4bc0..9cde7d4b66fe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:04f66c1adc866d82ba4ebd7533cdd4609a8e1a8215cca65efd71c827f352d934 -size 21548 +oid sha256:8a7759088c120f17fe4c1bf94816268a1b6d242aad9a466a08616dab8192e9a8 +size 21444 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 2251124c0044..e2573b003d84 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb5f5b50cf2d6a1d2628bafa22462cb57873305b8a8ac8aa62c5efaea225070b -size 18841 +oid sha256:ac65f4db480e0e05c09923a00cf2a9afc523a4b246179151b9732f7b818e3f66 +size 18740 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d05529c13b21..e4a489376ac5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4261db39fdf86ae5cdab951e7946067f3315eb2667978eeb15a2536584fa1d9c -size 22402 +oid sha256:6ab8b4846def0312de32d24efaf3c9f905785c81701845f375c592281dafeab1 +size 22270 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0997fe4a0797..de8f5be88775 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b0639f50b368dd1aafda259ca2ece753c0520f377db6ab725b7aef1ae5c093b -size 19915 +oid sha256:61aaaf4eff3a9729c9afefe8cb36a7465f97ac827122a10058d7f5930b507c33 +size 19848 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index f805716fb57b..f8d7620bd8e0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7f36e354021ac0f3da6e6492f9e5c4d70ef3b6e2d3aa444f169793dce7fd74c1 -size 21461 +oid sha256:da57c1fdc241c9c97c010ae2e66c96d153697fa9329e776d833d062b54834b75 +size 21345 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index b3eb2863b487..037e304e9c63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cd15e0239b276972c35f41cfd5a5b454622e6f0858aa04e43b253d5cd8455a2e -size 28882 +oid sha256:3dfabcf55d940c409d5744cd698a21d28adb86566a0f228ee28c8b03f9858663 +size 28710 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 2efb68a3c041..6de06d721e52 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c7592811cfe74285f628c39eb4e085102bc61c814c218b39801eecbb61552e5a -size 21702 +oid sha256:eef38e6701a2290d74c6f73f839376ff4ee6e4de4f7c1114c74eec08109f5860 +size 21596 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 5f1c01a464b8..909497d2ff08 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db2443865191079e82636c0b54e6e83ac573061e9ae677e3a0aaf599ba757ef8 -size 38030 +oid sha256:e50da735064328132c31b719865ce9551def580041b09d2882203f2bc8ec3c88 +size 30866 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index eece998f0c27..e26a094f6d8a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a4b83b961fb6d957fd40ed68a785a51c60d74f5c0e635de9a1fc2de5cca43f9 -size 29440 +oid sha256:cb02c34b62f73a2d70ce4f03785d4f38ed8950fb014d2b06ae20316979ead707 +size 23226 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 7e63f38a6402..3f3d8c724175 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:76e65673ca371852e4eb93683c63ecad1c4c6329385ceff1a9f7e9366317f670 -size 29943 +oid sha256:621b1ea37ed724ca5d231d1a6177145a1eb3d4625fafcce6cbd5ea5da070f130 +size 29957 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index a19bad1e3aff..0b0da0655842 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:58e5041277aa29fe12a52e137770421cfe5496bf4286fd92038e850869f868bf -size 22592 +oid sha256:6ee74026f77bdd0caca09d1e9763d7b2ee1d607eb9b92b91e6d181d499e82b3b +size 22477 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1221d4f7fe12..97b71055f6da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d1037759d8c573079896739ccec5529031f7e0f0bc404d3f3cb7cf38f13b3ccb -size 18775 +oid sha256:d8a02994a90b3b829f9679e0ec9879dc602d8523615a42af76ae0e9133c778ab +size 18698 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index a82cb39ac175..c485f8c3d145 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3dfc2a2aff7bbc4e6782fe0654902f32b2856bad29e00912b562400756d68a32 -size 22456 +oid sha256:fe6a1674baa4f9b099d4c2889e4881a54feced7e0080f0991a3bae6c58266f22 +size 22329 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index b8e188d28a89..1fb984b2bf44 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d7dce8640b97c9edaf6c13f3dea5a37b7f86384a06a9e72d0149b44a888d1d4f -size 29595 +oid sha256:cd6def55c7d0a0d6ab2fadf5ce7fa70adc30e710967f66e5e7473443cdf0b6e5 +size 29535 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 1fce900c3a79..49d0b4716867 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fea211c2f5d2812c49e051e8466783bb5e3bcfe1461467569a06572c15d7fba0 -size 22894 +oid sha256:e61be8e1817a29b191faaf385034c76eef298d3e4b4e480e51f8059cfe54be13 +size 22774 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 339a21acd010..e1917050c194 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57eeed185354728b5931f09a84f082dcdacc7f209887aa8b9e494ef0ae3d4f5f -size 38385 +oid sha256:ca5be5b5ff84309be922add280d15aea4c0ac6b46d454ed70e6ad18baf6f1022 +size 30927 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 3b7697244b91..4794d172d081 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0a1e1b723023a3a93aadf2f59eb8de754b590a2a7a544ff7ddd543584b63890e -size 30641 +oid sha256:3eca42e9d8710b009a39cd803673bbce72fd9fe04f770f74d4508dfe963ac3e5 +size 24047 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index f63238480534..88da929c61f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d47d1df944b4caca9c78da8e2acd709e081ea1547d11c8992b445f22ac946d1a -size 30464 +oid sha256:1f12933612150286de7aaa471988875f21d7701a3d7b441a39f4522882841edc +size 30441 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index f77032f4bc94..5ea97c73e8a9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:95594a01ada034bf05f3b7eb20d97809ec9c30c8ff9cf756ad8fa235e626ff33 -size 23489 +oid sha256:e24b551f00cc28a3f9efb79809ea09da9a71b5926a3a1b501e230d71178804d7 +size 23323 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0af73019f937..578878ad211b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f0245e3857d09deb6a52d9f7bf250abcf3e16080899358b05977d3e910ab2816 -size 19644 +oid sha256:f9628577a01615e90f5bf928b61e7121ab72e65456c1a7305bb833dba73ce7d8 +size 19568 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 27f479504b18..a5e654bedea6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ec3b9a1b655f0b15487d25e53a2eac375a1fc45910fc30c2f9863c8c814b1e1 -size 23678 +oid sha256:7b922f5bad36ae5eba00739011f04ba4a8664456f5f5b8586e2a2a663af49c2d +size 23477 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 9713b11ef366..c964fc3168fb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b2b1e3fc220ae1264358dafa0370d5c4f1a2fff1cdaf80dc3296ef10a0fddbc0 -size 20906 +oid sha256:bcdcfd2d6deaa78f0f9ab9e6cdbcc695d8edd9aef62b75c17a80e08dae9d2252 +size 20711 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 63910f52c34f..a32d520f9c41 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e938b8f684a1aa2984b76ff4f1004c65817899a84518bcfe22dfb7a34d342547 -size 24606 +oid sha256:062fe4db848c753dfc4411bb1eb8a90eaf548dce322551c1f156c68cee999197 +size 24342 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c33def7b3bd6..f0e4d5ea428e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94c20235d4a97900028b4f16fc97db36a5a765fa2064259407bf72ca653e2a2f -size 22038 +oid sha256:a2b579b50984a80facedb88a133d28fac6d0e91fc73d87d63cda5a39dc3277ce +size 21891 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 18415ccb0332..90707e10d405 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:86263dff00035a7d14762344123c56e5c2635a46ac2e767005ea8e40d8b902d3 -size 28183 +oid sha256:b8a142da2e15ab30273c907c66fd6e3e04b398d404d8e9066806f9063b08cea9 +size 28049 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ec17011013b5..e19754f59d32 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b5849be40b791a269472cab22998c304cdc95ab29d2e88e793050349c1d971c4 -size 27019 +oid sha256:b241d6f6c60941bf1ee33ba8f935309fae31f9e98cb595d02c586ed0110ff684 +size 26890 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4b6278c94f54..81fa7a2a14c4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:777b4de95f73e2a8592700dd4f7ac25d36589ac56f287c0cf467bfcd5dee3f59 -size 29117 +oid sha256:4abb8215477716ea97685ad50314010d90e3895781561a9287afa46dec94edae +size 29095 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e0ae110ea674..39498839fdc3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91ae8a2a68d581f059eb2c530b13505d04d4a3b812bf22e787d80e07c01d9a49 -size 27929 +oid sha256:54921d3ae0ffc836589fa5ac80045770625e7e264acf5166371afa79b9307091 +size 27798 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1c1f513797c0..69523b210692 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee2b31cc2c485d1a415cb697d04f9160464f79ca239c6994290e6b9053e969d7 -size 28923 +oid sha256:24f7283526c37281e76cdbd5a1f3f1000673e9d8b7b33b74c281dd5ef5e7af6e +size 28732 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 685de9a66fef..25da766fe3fd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b0b3fb15b587c991cc887b2ddf117bfd85e63ded5e3f00001fce5f787273c29d -size 25857 +oid sha256:726b2cdb7a5860f4aeb1587c1b193f49e3c3cb4abb7d8c58a169624f23bcbaa8 +size 25751 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ac850945f173..fe991c99e139 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cf99cb7a7f147089c38add3903893835e90dbc1414c93ac34b06af81c5f0b489 -size 29824 +oid sha256:11f55542051c58d069053151d05567f0b7a558a174da77917b9bcd2aa7847fb3 +size 29804 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 532754e5c345..c9cfd1d558bf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:97731a15a53d4a783e0795cfe2012a5880cfcc73848d3150fb9caad6f9f1305c -size 26786 +oid sha256:37887657f9be7184c8fd54e0340f92733ca8d7c6830d71cc13a5b18c9108894c +size 26652 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 350dfc515558..db50e6983244 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d86cc192ce48d63ee1b5490ca93e10ea06ca0d3de84785388a008e3a66a72ac9 -size 24985 +oid sha256:5a81bd6aa5b77dee4d6a1ad3c822aa1d8fe011ad378c79b4ab0b5438e522c7be +size 24825 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index cc674f55be36..48eca9eb97c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:880fe6a689b8a42695027a8fb6a6f821e5f48b5949eb9595064a91823190b6f4 -size 26413 +oid sha256:4c893393e2afed3914f9b8128d695d95b71482f630702373d575fb8921c920ec +size 26433 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 3a909b355b94..4c5e634d9523 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e5e6602ade3308ef755d3f5dd02d62e7ac5c5de07d35d7710d397f2be058a1e1 -size 22555 +oid sha256:6002ec6800abdaf67fc7ced381be3f6f4a37cdb703898b00fb2f4f811009605c +size 22432 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7e52b08f123f..c35ed978c021 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4654c9d73fea96638771e41808719ebb7b5c71d6cb78a12bba75e981ef318d5a -size 23696 +oid sha256:c58ec63f395ec819361a9a1969abb9a298d430a945966c145adfa4713c9bcd53 +size 23694 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 6236ec0717ea..c41fa166002d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3a01c7459e90ceedb0d8f4ae83edc997fda7841e4c6cd4710811ac1d177c0667 -size 22739 +oid sha256:60acc783fff7881e07ef28910c55e70a8635be029c1eeda2ed4ece0a3349cf96 +size 22658 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b50e18c2a7d0..ab85022fb8f9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84bb3319ae72d03556e78d7a36ae8e2a2bd35a530e13a77b8f60e6f2db982b94 -size 20375 +oid sha256:5f531b3a5eb617c3ce950b90c7013960cd7524e7417916d8a85bc45a29876b2b +size 20323 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index fa6162d289ae..6fb4cbc2e500 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cffc274f0811ca7b795433945937f881371e1f9206d684b0f33aac77d5e747d3 -size 26003 +oid sha256:6ed0a942a79d553d68d452f868126d437e1844c9e7501d82248dc270d7814baf +size 25860 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 5c0f0dd6357c..8a9a76fe4d3d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:69f3ceebaebab51d49b60534eed922a11a9172706fcc7ea2c83bc56940273d08 -size 27685 +oid sha256:2926fca271eb190b2c0ec8e6c853fe07cf9b82979d123490bdff173d2acd3426 +size 27593 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 81e67ce3a537..e96d52de3739 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4e949cfc11b950df425f56dd030806dbb9adbd49b07bac9a6d06c19033cd3d6 -size 23588 +oid sha256:7cc445d2a1ac1f0bf62aca7dca63ee4ad9f3fb19df8d98e00050ec883139e291 +size 23456 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1c2eefea1cc9..f27c2142834a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37003205e460026e60e3875515018df89838760bbdf4309c356c7b263c38017c -size 24799 +oid sha256:5b0be1ee0861e39d50791ffd30e602d90e0115c821273a300e7fd74cacf02fa3 +size 24737 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 81b7b6ebbda9..3ba614adcf99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1c7152202ef167bdbbeea376c7fe0201f6d2928c722e54408bcdf550a2930d9f -size 23683 +oid sha256:ecaf9f4a5a02f27854fc3718abdc369947711c671898b4d61e0d689cd3d66d47 +size 23585 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2a7eb720f136..96b0e4617fed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57e9125342fd562b0e5b65473804b7b1b198645f12c7bbbb1a079886933a24e7 -size 21393 +oid sha256:ea171f442ee6cff0991ece1c11e88768d050773f66dbf0e0b304b0c86be9d11c +size 21454 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b2ce6e54f458..fbdceff0ac7c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d17854234c2fcc7bb4f5dbdbe7cd809cdca24414c8865f2ca5399f8f0625309d -size 28035 +oid sha256:15a3975dccc19f97769c841bff4c9dedd666204b4f914382f6c4a126dc37941d +size 28050 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index afe7f18f0429..0f8cc95a3744 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:34b176eae05d3db8fe282f3bb7ec033a4b4dd60020321ed7badb4c020971809a -size 29024 +oid sha256:715655b41a042663c2f6d9046141ba02b45ef064f31ef26c502526c073efa980 +size 28928 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 679bbc99e7c6..fd21a453b204 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c85b9823134821d5e1bbe1db1eeca968b9f9fc78716b8119e5e9b9ee8325c07d -size 28969 +oid sha256:0bd798a8ae64b9d795f4893040a1485b7fb5c075d1bd8057d59d0f597abbd162 +size 28966 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 13b5d65320f8..ba843da161c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:da23771a4184990830172704fb6179fb3f2f64cb5163489125447ed43b4098bd -size 30241 +oid sha256:25f356015fc661b2ed8a2fde6e2742817179b9b42f6f442e9e2e58bbf4a8c989 +size 30075 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index f7585b65d185..0268f898c21d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e185c13cbe47e2467adf5f5d68e81c683326c4d3ca4c0bcceb55dccb6379b3d6 -size 26395 +oid sha256:caaa38b3b30713331060ef58814fa2e9facd7826b94f4583f3e4b2fe5e41a659 +size 26332 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 078e442a093a..bdafad411166 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9be9fd80f342f7e69fc7907417e4e8791fd6b7af6cf7cd3fdbd14771caa7847 -size 23491 +oid sha256:4bf9463d2836773d8cf0b20e5ffc5e316e62e3f47d5afdd246e1555806f6e8b1 +size 23476 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1d843b1fd7d3..7c1d4bd43194 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cb021f2f44661815d83f30bcd9d12125a892c86aadec79765e4b69532720e566 -size 27651 +oid sha256:d40afe5598799af82596e35b2d782a2d9ee51bc1c87fa8960d249a8d6ec7f37a +size 27609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7bf88083d034..050e29195e3d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2405ada079d0da001f95226cb451a61ab903e78ca16a305f328cda51bc2a5143 -size 24717 +oid sha256:16e2275fd06cab7a7602a909540a23f3c18428cb7cd8f2e93ef58a259e111a56 +size 24670 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b297fdc72cb0..e1e1910be884 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36cbaa0a0a87db7ab411d60366d1b97ab841123bd8a72a6d3a352cb283d7daed -size 24655 +oid sha256:f468686d1c12b44914b5c627e3baaac61e5180dbb8b319e00bb014289fb50e9f +size 24648 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d640de95a77d..9466946d7b30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b3ce6477f26cecaa864d92186083ce237b708ac08f737c0357ba44bf4b28b94b -size 22522 +oid sha256:df28d80c0c223ea527660e836b274dd2ef0cb906489f58ac233bfcd7c91402a7 +size 22436 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1dbfc403d431..2726e0aeae11 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b251edaa8eec81f9ab3a357b995d94c07be49634ffea75176d29cd86b73b1418 -size 25754 +oid sha256:15fe17411d76d856839170336ea6ec70d373b5117ab3cd3d0b0e469a0dbdc499 +size 25749 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 564711b6bdfa..17bd484e7822 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:431454fde59a295930de17734776291a551361c2c1fbe7d2d1165396dad5fe5c -size 23465 +oid sha256:a283b73d7a494649ff7b09a05221b2d17c6663a739b2239fab98593d883de606 +size 23355 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 5b767ef844c4..d54ec8f61c6b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1908f6c9d660c9a5c07753e0aea0ccf45ebc3cf5f0382315787cc88058ab30e -size 30957 +oid sha256:f37fa9c8d959d3b2ec25114bdd288a7a05bcbe9848e0ea94b41d97afebfc4d66 +size 30956 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1724c832e3c7..a8786238dc45 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e7db4ea11dc36270c9e2ad9e851dda8f322aa1fd79eb64a77be82e058eaeb293 -size 27814 +oid sha256:2d48c663e2bdec4f8c9516629e8800950173c383dfab7d8ecd39a09dea4e95ce +size 27627 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b6716241af46..a644178eda4c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ff7ab6745a1eee949e265ce2a4c4c604073ddf646ce168f2932f96b6d5e4fbf -size 31866 +oid sha256:3aacadee42191a5094a7a8516eebf9cac2550e52d8d0dd43d28593ced28bd05f +size 31727 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ea0a1f6d0bce..565c9f0b7aff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36f7361a1911796bd66338b043cad7fbca7cdf470e35f5bf4c5c6030d81768ad -size 28660 +oid sha256:d7e910cf329cc173d101584e056d00cd5fd588327dfbe92043bff2346d790dd0 +size 28614 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a1ba812df6ea..807a40ce0bce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7549d0dd6094e757b86405254199b0702ff688c467ffb47c46aebddee46ad014 -size 31879 +oid sha256:58c02a7a98a43e43aba6a9280e0d256699e945c254bbfb4a6b337eb4f0015b8e +size 31883 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index eb78a5ebd4d0..0be0f1079af7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee6fdfab33d6e61865312ee12befd0064c62764cf532441aa73fd451fee99526 -size 26720 +oid sha256:3a477e1759d79d8255916e2fc2d9d6bd9954d4567a3d654618b3805f74dca09c +size 26583 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f8c521bea637..053989d101c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7773f9b7c5b7fd5e1cc5aaf0037e57befb820d9328c7b87a94313d8083856842 -size 32690 +oid sha256:261a82881060f1d04d9a09cce0e6157c97a8a402d4461ae39cc229feba507aab +size 32687 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 59c7e19ce3d6..0db5dd5f3ce5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:089d1134e15334a7de407218903a5760fb96540a8c942704896e8da1385e78f9 -size 27538 +oid sha256:063aba89839cd2222adff2619b376113be713a7fb224682cb18668a61f2e2e4a +size 27351 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7641a4ba40be..7bfe1d64bd04 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a7018d89ccbf1fa38d32115358cfdef5e9a58e28f8bbb82b0da5bf91e95103d2 -size 27023 +oid sha256:9c8a12644742934c0c7d8122ffbdf25f536983e2b0519395f8b8bd80277cdbdd +size 26815 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 496644e315d6..1a5ede01ba31 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79ba91e41a1d21362d8bb2aa15255679aabc98a837b13cc8499f3e2a793ced7a -size 28986 +oid sha256:887fee1a536699ba2e341119da757e0f3782d31bb9760808787fa9eca8d073ab +size 28930 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 4caeb59f0b1f..b74f16484334 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21fab58dbc2151495ea65bb252f81911bca82fd1ef898242378372129c5296b6 -size 24860 +oid sha256:4cd014b3abe779bba6d7812c840e5f41aa2f56c69e22a515b8f3da3bee474837 +size 24688 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e987954bc7bd..ca429ca4af99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:29e529c6e53c4ec6e8b630e6ec9f18071578abcc0224bc668443147bf8f9e3b4 -size 26301 +oid sha256:f6f96212226f43c6aef4700f6da97557151ae8b3c361a62b08c9ff959f702915 +size 26315 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1a0f7787fca0..a316137d596b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e1dc98496ff8cb75f3c78aa8541237aa10df19ae852a56bd3953858b7ccaa23 -size 24064 +oid sha256:f24c6f99288b985e8b607fa709ef118ea3dd92bf1c806e4dc8355838a3a694e9 +size 24000 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3c5fd669c84e..70b9f2c6c079 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:46d8209efa7f5761e17b4aa10509d050c96280a633a8c5d7df8b68e63ec0b8c2 -size 21940 +oid sha256:053baabde66c3bb9cfbba29bbb46928735eb19585ecbd15b573dce29854bb1fe +size 21918 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 74391476ab1d..7a17cfb7b0e3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c95a1d0a6c8e8b9b267550d0b1e816f709eaa9cb52bc7152e6b2116bf17d9e2 -size 27959 +oid sha256:e7eeaa0bcf60a9f9c332ba8a0592a4cb6a8f45afea57a22c6d3a0b883498d5d8 +size 27741 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 22b9f63ae226..347215d0f1a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e348734cdf23fd9b68538d73b3b067d4b7efe93aa636987bca50c9f2958d87b4 -size 29840 +oid sha256:13892c82726bdf9c996a6a6a3ba23d17b7a5ff7b31ec96853c7d6c7c76e79d3a +size 29888 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index fa8aa62711d8..82e8dde8d965 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2dd92e1e9cecdda119247e083d10310ddf88cc06933d51ac75772aa72fbf1daf -size 25856 +oid sha256:69ec7c290ce338f5efd75e07f2b7cddf3855bf79af3702504b074ddbd359be87 +size 25655 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 04fd8fda22ba..87f43e0a30a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a5009404bc96e844c5046d75a564c40bc9f7000dc9422ddf16006f8f62f96ab -size 27095 +oid sha256:766a46649d02926ca8983446b031898ead8522b970d4d6afc6c14a1102d7cf0b +size 26995 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2c0b068bf5a8..d1a4d670d533 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:614818c81e49fffa0f0d016d964f4a933ec31013b4e32da2a2f3010d93532444 -size 25048 +oid sha256:443cb450f5fc3f4f750d4c780ea637a02d91b54473a54cb30183974871a39a64 +size 24943 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index bf3fcbdb6b1b..7c91ce15616c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1c750b39ae0040a9907225c6447524bbe723a69ee975a08ca59be14bb34f634 -size 22886 +oid sha256:2559a6c2b2d674088cb48ef8d910b68643eefac20a1587ed2bfb3c1d5b4235c4 +size 22981 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index e4a9c6279d66..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9524e45eb9533eb9033aaa377a5c0bbf785fddd35dc3fc863485585e13fcc22d -size 36726 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0b9abbaab1c4..02daf16e5ea4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e4d80b020295afbc2745256bf3f69150a0ef77a9696ae9f0c4d0d1b070af294 -size 21473 +oid sha256:69fb1d72552380eed6c530381d009d4ef6ef8bb28ba9188e39f6511dca1abbcc +size 21354 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 0995d1d59dde..611c7430e244 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e92d5d809c7f2b4369258abff4052b538197fd983e1b3c7467a2ea667e7bc316 -size 28849 +oid sha256:f8418c94c3b0a4be99153f056f08688cf9eb8b1ee73f19ac8a5c216eeda79259 +size 28695 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 6e616b7298ca..2ee79029de7b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:094ff76793e0d5ccc32d0840450de1879db3b0c4f2222671bfaf275c5ecf4357 -size 21749 +oid sha256:14e8c550f1f0f2e4af6d673236e7c0583a0b8a7d977f13bee1861cc66f936caa +size 21646 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 9eaa352e9eb2..121eaf46503b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:03700dfe3e95f25652b80285b4bf43dffdf0ae07d6eb8e5204806ba3c8accd86 -size 38054 +oid sha256:d4bf1b172766f7ca24f2fe5d24782050196c5eac2b86024da06207cb5d6fe34d +size 30846 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8adbb0653c79..7186c381780d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3488039a6f7acd5c9e7dffa9ed0e5f141f7f9bc25514ef0118dfbe68f331ed69 -size 29514 +oid sha256:b9efefd9c02fec0e8a250f395000c837cef7972926867317d63a6bb1d5b7b71b +size 23266 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index ca405df96052..825bfcacd7cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3f56651f30758d22473b26b6ff7f48ea7bbf32d6581c5653ed952d3f8af2031c -size 30045 +oid sha256:f9484b1cf5227468b63d228b9ebc6fdc9de901939163026bf8926343ae1f4664 +size 29996 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index a484aa8dea1d..9ff36ec4144b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4107f0aec5588cbee37d05529bf33488b9d19dde195738c9099e7843a3056fc7 -size 22629 +oid sha256:0aa1ff8767c7d135e9599a28d7ccd7011bfb2b49bb6f142262bb86d9fb3e0402 +size 22510 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 02dc8a9a1146..9a2bcd0702b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f70fd607fe8240d0aad1f2b00a6833a3626730020663abe0b97e10afee42bea5 -size 18784 +oid sha256:7eb029ae6a4e5279203779d8d83e807ccd5f2ead02dd8ab63be830a9b9ddb1d6 +size 18712 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 073fcb25f0eb..070c221e7a26 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5e2f57c9416b2088c681e2ac9ec6dc405b4c2a279bcb3d450fc9cf48215c1b12 -size 22451 +oid sha256:07b9ba82fa5bf21e2815bc9d2a0e7dae5020847f9395311d67ff1e79166f3baf +size 22315 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index c1fc25636704..4d68560827e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:483041e90e8d31300b461fc3e9e7ff7e0daea49d51c51af3fe3dc49a7517fd8b -size 29573 +oid sha256:60fb2785e0776bec06f3727abf8da02f82817c025164eaba537e259ac304af5e +size 29514 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index d17e2b2430bb..4c2ab80d05fb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9afb980733b829c518ef887539d19c23ba416db882e2a4ad102379b2eb0e00c -size 22897 +oid sha256:8d812fad109d86598ba3b368f8b5238bb68ff15d6e4d00cba897c3aa6d5cfc60 +size 22776 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 07697559cb28..dbee4817334b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b5fa4c75a5562fc1c9f38b26781bd24235b8bae4647ce2734016bb2b310028d9 -size 38384 +oid sha256:2b6583a58707cbab9a491ef54134083ccc581dd73e656d355d88cb2e704cebac +size 30914 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 71b9c00e81e8..f7926884d50f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d9f9a22224a24046da11585d94712e47437997433999f651972d946a78f0d02 -size 30642 +oid sha256:c6e9eba01ce59b466e34a3933209f43f3575a9ccf4a505c9d1365759bc842ad6 +size 24051 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 774bb9c95d58..d528aa94590e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7c17898008df055a471167035e4b4040ad4a8ff3bfaf102a8befff1760f0858c -size 30488 +oid sha256:e56cfa30ed77c861d333ac4e21851904dd837400cb95dc3efa0f5fa0af445163 +size 30447 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 42ec6ef55173..14adf90d9cb3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec97b30abbecd08a5840a19e7fc542cf894d75cfed117c772ac0554e8aa78687 -size 23489 +oid sha256:7226f24827cd3aeef2c9c47482f174ea9f3dc9b0d9297bade730faddf7aac928 +size 23312 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index bf9774e36c90..b6c2df683a66 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c92336d9015ac7d21849894063a9146a3b6df0d50edf16b4804ef74e83b06cd1 -size 19648 +oid sha256:c2a56814f4baf71f9a9e4dfe8c571ea512b242713fe1c0e53e961e7ce78fe383 +size 19561 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3bbbefc61505..d4a5bd52de60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ae8406000749f2d61f1d80e45c27c7f5f396a947b47ce786b332efe43c39312 -size 21031 +oid sha256:ab2d8a880ce11b5ae4ebb7e668846240cf92dffe0edf3f5250f02bc00360c0b0 +size 20897 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 3fcf0a700ea5..c3ef16517c83 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:19ddbea0553b3ef812ea8b7d98fe7232745ef5c472b90b0c57483a3ebdcb079d -size 18388 +oid sha256:bbe301c930bc7f1382bcbb11f0aa7788707eb56b06b230d82d9dc1657ee8ca10 +size 18254 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7c601963e168..bdbaca169897 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce086a833046ce5ffd16bb71ba8436dea812f72dbab5c873bf418faebfad8aa2 -size 21530 +oid sha256:1d4fa63a8e36a31d65b2f31c1244ee8fea5ad8defe04301fc40d48272033d12d +size 21488 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b1bd9c76b281..07289b787121 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:96e31f73c6345ffec03a98e6d26adb1730790e55f6fee1ecd00351f2632d3a10 -size 18861 +oid sha256:2738ee3ba893a21fb3630a16dc4c0db895760eb7b59e6df54d19f255da70f8bd +size 18728 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 741a53d634f2..b35a5e484fec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d50daf50671c5e9c15754706b9241a2cf0bb5ef4ac2dc43de6f0a32b3786f753 -size 21344 +oid sha256:cb509b8fa5b597e13669fa9152ad44835c2f40b11b2a3cb03c4364dd623a7b47 +size 21225 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1387a00420bb..ccf37b2c3688 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f4d60ad4b966e8a27008c6746d19d23a6e033f97d8c6cbc80da552d12cf799d6 -size 18300 +oid sha256:6fbeddc618d5143f624c5d70225aa168cab84ac76edcb243d8dae4e401dc9f5a +size 18173 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 925ef4998880..3fef657f02f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48a68fede4706fffdf7d40813090a99afcff287196f8e7899a9411592f6fad29 -size 21721 +oid sha256:bdbc19e5dd62d4acddfbc032f50f7ef5351f7a75ffaf7746f2e67a0007cda1b1 +size 21698 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0ea06b86ced0..62161f9e6f10 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d52524d1df981789cb8bc120f318d4f6d59dc1601969b170b0dd429332f2a2ab -size 18617 +oid sha256:7d781b419e7d9d56cb8b581a557a0cc8e046211f1002bd0ab3fac31c699b849a +size 18482 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index fa0c06452c0b..711c3000b0f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0d02ff87fcc55b64a742e71035fd00f9a2dfa62d9664d48593c20e64a9e502a -size 23204 +oid sha256:4b3fbcf4f15dc973d3b373cdc6aa137524fa3253d075d5eb42b425dd2b0eae93 +size 23006 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 167ba02c26ad..a68f41baf9c3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:11907996b9aa9a4af6041d5a68d9910421cc3674dd161f7a97b6f2db6a6b648a -size 20589 +oid sha256:1caaec99c63352228fac19ce371e18971c22e03e4b365df786a48e9a3cf6ba26 +size 20366 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d47c6ea1ac3e..60dda4ed7670 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:24b9f2e6dc5a6cee969fbbbd03cb350790ca6cd87897ce0ca913b505b48a239c -size 23687 +oid sha256:ceac1c435db9e640d5fc83233f34d55db88f72bda1f284eada7287cbd7135744 +size 23609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index afcc7a6bc8c3..3a8b7e7b9359 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:098e371a521d51a8233d1eb20ac90188588e06eae064681708fd0f45f2c4ef12 -size 21089 +oid sha256:710b5bbbb76f1dc88b08020bf30c40fa4eb687609c3bfd1bcb4d8175c9cdd03c +size 20881 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2e7ffed36aa7..0e17721fc772 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:418e28774b4ddd927681145648b94c18fcd7209ca8c79ebb740741f654a3c19a -size 28375 +oid sha256:9751ef690e34057a65f6394f0a9971227a09e770e4af3644559ef512ce560f12 +size 28235 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b00a5945c852..34da3b039e17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4ecda3b10c73382b970a8ab9f8aa942bb1a68f80e9184e1990e8f088edf6616 -size 25271 +oid sha256:537c99d09370c9d57d6c694305d3ea89318c88c19dba1cfb65bd7a2e2fa7f53d +size 25214 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7af3db5b6af5..c3df50ab500b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:653316cad70a5104280a365e166407df175e47b6c05f4a72292fe333a97ecd2c -size 28952 +oid sha256:a5fc70f8a4350d257e0a3faafb8b31a1d898aefe486ddc3e552264652f270262 +size 28836 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 989be7fddb4d..ae11e4abcdca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:548621c2816df39716cf63339ac63711f07a79db865fda6565148d162a046f9a -size 26065 +oid sha256:2b9b4c684dae04e0ea7dbce78f06189bf6547a73bdaec8b98110833e55fce057 +size 26034 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8b8f1ccd1474..bce6aa84270b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e2e1eb7dca4f5feea44888a0ac16ba2551dce516b7f713d082236b293d09f07 -size 26200 +oid sha256:5f014c02c1aee1b3113f4dd39b284ea8714d8137e32ac3b91068a8edaf28bb57 +size 26084 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 518985bb3c3b..9c98d712174f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e4534505a9c929ad22ae3876af8340df870a2d317598fcddc2c5ee076afb59d -size 24695 +oid sha256:0ceb4c5893a3b63b14039e0b6d31a84ed193529dd93e8b4773ead4d1dcde5bad +size 24570 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 187f35398989..e59cdc69ff01 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b4de63b1c5df0ac197c2cdb49c59e8379d00db3b449a00ad5f63934069c68f2 -size 27410 +oid sha256:1926eff9ae56a72d96a87f423a5e12bcb9c0aabd171733b6d213c94bebfb0561 +size 27288 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 398958a7d8b1..a95aa4139ed4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:488415d59ff0ee0541bd1738e139da0dfd93f3b4d3911a608100371c2068aa2a -size 25680 +oid sha256:004eb26e536e29b5c73282012ca12a6dd11b771124be49ff3c2e1401d1fc4948 +size 25525 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1749b59c653d..ff3c3fa05b7c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a072e79966508a0bcb89ceae5f1270fc4e2e7dd7b92cf386434a795f888acd3f -size 24812 +oid sha256:920885f427f3a3c69a0eb111cb97d2bc5c3092092f6565473a705c634acbde44 +size 24623 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4263bbce359a..c1898e52f923 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cca118668df0780f6b63db45e1784625e245e97605c7f445f3a66fab9c64ea8f -size 24606 +oid sha256:b080e652f218e7233f6a6a7b0ff5af798d385dc46ba02ab256f7a2230260f132 +size 24460 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 0d11499da23f..e5db786401ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:747bc01cdb922a491f4adb6e6609a60a6d49e6e8bfc7d6a340976111ca46ccf0 -size 22278 +oid sha256:e28f4b7bed172d5824af0b4f6efd3461285d2b329f0330bbefa62255d80f59c6 +size 22130 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b39f6dcab43e..df90a73d39a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f08f4ff951f96eca2f990d0a6d323267f2c1f52138c54000a161eb9bfbe636e9 -size 21698 +oid sha256:d615a572268bc0e5b9f699298de08732465f8e313928c4e31c8f2b35442b0c5e +size 21720 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 142a7b7a2cb3..e91a3ffee5c0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b59708b9d63e5d16a016d16598e63bc5eaa4753e525a8ef93784ed089ca08a22 -size 23218 +oid sha256:a5cc55028273a5615ebc30fe0348ecaab391eb8d299bedf634f23a9216286379 +size 23234 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index aaaf95e21366..64c2621de630 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4dc492ec36d79f1068b3f1c7c08dc704a4b22c3a7fafb1770b61116b2ee6999 -size 20124 +oid sha256:9c36525d0b298584cf7a5558b79d21eacfee69e749c36cce098d323b865d3d8f +size 20070 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 6646fb076b9c..3c28a073c85b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d37c4ff243e3fbf9f3e64ba7309b474e3e064e004a6274559e5284b295ea7bfb -size 25516 +oid sha256:1fefe8342977d6188b1ab86b28c1ce5ec9575be0e54879abafeba3e4a85a14ac +size 25369 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index d95818517041..5eccfa2a2331 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0f04def1ac319d47cefa716817cc3f060aadc152135585148171d6fcd22098b6 -size 25156 +oid sha256:9b3da735385cbba3251e2554b138466858eb38ece6be940d61f3e46f28894470 +size 24967 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 837754f04e22..9353b72eb3ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:247adb4d8b05b841005948d8cce05593e9b26be40662ff9a7d9c6096b1c13bc9 -size 22598 +oid sha256:0d434ce9d1a07f44278916a7b1e1f72c8c63c324c0111ab723aecb787c696668 +size 22414 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6465f8e0ab6c..b8bb4f1e9b3e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1e16337ec84e2ae218198afb9b6689920c5e607390eadafb589e353afe68134f -size 22428 +oid sha256:c8857a93aebcd5108056f7d19c7319e15b499a71bca60fb3ac7b45c22849093b +size 22295 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2f2db796b0fe..d92be3d9b5a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:066f64f7db116558c6f44f735becc67cc0b00e5ece04f84d886c38d269ba1891 -size 24266 +oid sha256:1013dc9e3aee451b1b2df58f65764086e2bf9d3bf97804d91615aafabb8e6696 +size 24205 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f057a89534f4..2a3bfee39577 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f40a22ede8c9f41a418bc8c47c86413bd1224999ca84355b19efb99b8a838cc6 -size 21096 +oid sha256:1ac43a07d25def324acbf1111c5b4140b86d416650a6196bb5704217e3ff56fc +size 20986 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 770a25788b1e..9f0e38f806cf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3af5bf68953a9fbaf80c096294edfcaadbcce4b8934bd9519d2f4b2302185c56 -size 24804 +oid sha256:ec5ecfc3fe336070b9bda004466e3d48bdeeeed92776767644b6bda7ccfb7f3c +size 24717 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index a8854fc54a20..b0cdbda9125f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c924c29dce7fa082a3ef6d15ffcc70ddda54c2e338065d12572f0956c5e3e745 -size 22059 +oid sha256:3150fb6331e1c6ca8a30e26f15caae6c306a4ac7ef2943aa2309fa31fd795bab +size 21905 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index aaf53e74477a..a9d10652b34e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:66346f0538c9b22ae96b223668fe826d95d97047ac99bb85b98433150af8f2ed -size 25430 +oid sha256:1d364ed631e55811147a132107aa7960b8780319b5960eccb1713acff9a0ebbe +size 25335 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5aecd719040d..c9bacff89eb0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1e146a0adc457470df0912b3442ee21958386584d6d98707640bed694b407d33 -size 22330 +oid sha256:bf73b4ce506dfa7720182e3cbc203eb5fff526317b9abbefcc4a941e9d05c466 +size 22207 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 5ee4241e408a..97883005bf43 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2157cb9f93b47c25bc2e7849405c39a9ddd414be38e9fc14ac006148bd13d55d -size 30913 +oid sha256:a59463f853b378c661ae46db156db62214c3b157a5532bafedd17285f2878547 +size 30767 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 31b9461c7423..acf65b3ce1ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9420f945b1e56963ca80a3e77a437ba6c318c9855fdf56f561226e91e9f4bfba -size 26787 +oid sha256:30332919a0b98188c5b0e3ee6b55045a101358801ebe3cffb491fbe25660a034 +size 26732 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1498af411fb0..b7284a8035ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d02f7614237ccd1b695a61495309775e4acede27a0bb81a131fe5388c5806d2d -size 31568 +oid sha256:b781eb19bb81d95ee4489eef092fc997c18338d200436e51cab0c9217b149c0c +size 31388 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 02a3eb020c5f..98894172f717 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d47ffa046e4b62263a2e2c60584ac865e336f4896e338ee9a3520021cd7ab98a -size 27531 +oid sha256:0b1ea7e05451864f00380a58add9501c485217d73a2b3796f6061d8f4ee7b5b4 +size 27509 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f7d43da0da5d..32c17e09a15e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c0d8962f81ef4189657f24a99131a116fefaf3bbc752c529c9b8d653dbbe711d -size 28937 +oid sha256:d4e6520635c8227f089306fbce998c4077a628823c15e295fc3323f871e70a7e +size 28801 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 060396510f88..274ebb745f58 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:227d576aa8d9b3a63a9b5424aaabb571abb455022502ce021614d30c3c1b42a5 -size 26342 +oid sha256:109aaa9be1f84ce64d7fb583941100233aa3f7c9f1d98da1e0f3df2ee9c106c9 +size 26207 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index aaec2656a8e3..60d648d6679c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:09aa68a4ec827c7ce28a676fc4238b58001430d6877fe7a0031f7245c8e1dc73 -size 29887 +oid sha256:be6918325d7f6c95932c6a47655ae72ca5180f8d062cd0d60f8bad8a8e539691 +size 29714 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e2446a39317d..306d5d34a9b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb6f37088796f2440babb327933a6e25fca0cfe6a995088d59c160e7da2e843b -size 27155 +oid sha256:c0f6d077bef4f66c9574e639fb7ee95d122bc0967d2955a72e79ff07c8e4af5d +size 27047 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index acba7b7d59d2..3bb63c70760e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f934664ed92383f8713a2ca8a8b575881a1de3511c7717a74ad30e7c3ceb33a8 -size 26931 +oid sha256:d0c641ff9b0663ae45faa4bf01fb31f352b835950ce291fcbfb503cf97c9b90d +size 26720 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 3a8b7059e11a..aba33a1d80c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:63e064f84cf15e089c96b7a9281affe06b3a779f7c5c56865aba69815c0f5929 -size 27066 +oid sha256:7650f8099cd54046b3ca30beb3afd73db9f610384c39be79658592d6d06d8f11 +size 26926 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index ce194ae1656c..6b8684a72128 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d5752c3253032e808e18364ea98232f60c3bce8c7dde3fb45457b3ec5e2ccd7 -size 24674 +oid sha256:69540f00c98c6bf3026d345aca192837de8d92ce51a9d54534178223c23daab8 +size 24455 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f89df45d0bbc..9c3507cd39f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9416991fa89afa53d5448844e4cdb2228f22505276d1723196b94b26cef8db5e -size 24157 +oid sha256:0f91997b09fb132e837d1ca362b818a416b3eaeecf9353a9d0d23358e0091f99 +size 23981 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1210f32c8aff..a3f1a99ebaad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:836b9b37bce53fb9ab40201fa6ccd43b6f873541f036337f1020e672c2706948 -size 24544 +oid sha256:04f7a9089faef14dcb73d81aa53a614d808e15e5908d9a043f8edb744f23ea5a +size 24464 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5029fbe58d62..bfcf92a70446 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3179b06d55efbd7c5c7ca91977217b8ddd5490fca67c3b4d5f6cd449e88ce37a -size 21527 +oid sha256:67020e761d1d15366894ececacc053c1fa146286198b8eb9e104fed0a7c79089 +size 21538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 92b85bfc88a3..2e456905ad26 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87ec99744023e6e6db0b94535995bfa727f50334f11105f75235fdc173f42b7a -size 27576 +oid sha256:e4167b105d5e4b9493525d2c37bff047426f0cc858123ee4257cbfe4f729f2e5 +size 27499 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index bd3533de1507..01dd43de8d99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:30a7880cfe54e89868c80576808f3a72596d2f01f3da89d2b88d74147cc04197 -size 27396 +oid sha256:707f210773e9134d719c005c69ab7c73cfcc165ed00cb1c66a01c7d15084e7f5 +size 27431 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 91680f3eb693..6634677800a4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:385f7db4ec0349ad16071b2be760a7392b39e3a10a5ab033e19fc72c0241c43b -size 25095 +oid sha256:255c4ccbdd268d1e7761198667e7371f8425aaa8106842645e1765c0377a5374 +size 24912 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index dad2f6b5d324..8e7da29f6237 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:51be676a062040c186cbdf2d39040fe6264664758dfca0f4e5298803a92ea06e -size 24744 +oid sha256:99c4c93778ef0978ab17c79aaa0a1619633d9a2c3fc6837740ec914cdf47d265 +size 24657 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index a4f7f086bfaf..1e95788f7818 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ef91b433a921212c96052361a2e05d56eeaabc2212cecdaa11bd3f91be3d631 -size 25125 +oid sha256:85be715d911b51a17377f67374a8dc1ead1de57cc91f3c768b02b31088fb5ca3 +size 25097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a5a62845ae7b..f1cd028444bd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:283403aa484b11834284247c0b99c893c4dd513116ad7306a0c4f236a5bdcaf4 -size 22354 +oid sha256:8d99ab27d41e327f280e629280ef3a0ce44954028488d0e551b9c0c6c0236522 +size 22295 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index aa827cbef575..49644d3d3ea5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:efbd8534c84208f8d8dbb25f0bb62cff9d3d2a6670bc72da3729d2ed1c5bcccc -size 22042 +oid sha256:ef6510558404693e903c182037f13eaed34916a60a7e19604aaa674a30dd50ba +size 21904 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 6942afbe5bae..bab228c78e5c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:809ed9feba7fd0bc6f1593a9633299b0e6effb6aba43cbabf8b122d7abfb71a9 -size 19249 +oid sha256:0c949ccbf0ac425701c17416f0667e6b5864ead38f10001dbe012564e3f69732 +size 19140 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 68e9d7df160e..dec650655948 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:370931a618e17729b97c5cd7f60f9382b44329022fa194e55eec66f2fbe2b444 -size 22662 +oid sha256:592e1a6c7c3b8d3cc461f0e3d218a22364032c359817c592df4169334f6adbcf +size 22556 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f53c54183bf5..8f37da1ec270 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84ac41f56e81976a679774ba0018ef0f3c73ab7c4d6a116e34120d25889a0167 -size 19534 +oid sha256:6521ee4b3242007b4fdc098bc5aef8b1d33b9372839ca6805f11169ea66b22f3 +size 19401 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7d28a7020f1a..1bd78fa3bef8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85d25993de1227c8531ffd41659e650c29949b3a51128377f6c28ad5053f5d69 -size 22295 +oid sha256:0cf8cdfd5d66ef5ee38aa77c2da0747d855f8e522077906a52897219a01e1d22 +size 22104 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1b72df19eb6b..adf201c729cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8d5ea160e3421042cf4cc5c6b2d2657a167eec12090fdc225848fd28e677baaf -size 19668 +oid sha256:b1749c4305a048c16200d5e22e7c03f490f2d02f5a3bc0a1f0147e47a8c5afec +size 19520 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ab957507c1f6..c0fc5570c4e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9bcdfc15b393910ef8857ce1b6e29570d3ad33e431e40df12107505cdc0bd87b -size 22752 +oid sha256:8fbab7566e9d01e93682ac3400556d95ad5eb44b8ade2123015d5a70c3884499 +size 22648 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7e4d90ecf94d..3b699d364884 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f33322ebe8107f11fa88d5d02dc7059d4ee565820f20e061dbca833b4359996a -size 20083 +oid sha256:423add9a0884cc73330db841ed14187ef6ee939edcc926dd2d0365fa6b03da3a +size 19946 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1747923d6520..dd1fa0538671 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8d832430f038ceb3bda540d4d2c60f1c4189e8ba2fa36f9e657f021d4e878823 -size 25222 +oid sha256:0d92fa9e88107b6c3e1e22dae9b4b673e1bb1e3bfc202ae8daf70754c5431e0f +size 25175 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2070ed4cd524..04155061cabb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc6a4385ca8d3a5919b8cf77c889fa0a91604fd781fe13d0fce534a154b8b8ad -size 23700 +oid sha256:8599c008a2751743af09f44b2622987fca233acef324fb3a3e36b8bcb89db093 +size 23719 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index de1082977a16..db7c0dc8d2d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33776de90b0532266c449db229546cefe87abff8287aebdfe75d8e8105c1b284 -size 25629 +oid sha256:0a68a15a4908c39d6294725d8d5530bb6df6ddb6831a39f517f9831f680d42c4 +size 25499 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 6a38dad5b349..6f40b851528a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fbf5b4f9271c8ec9b5596e7cb8892bce2f3811434b6c6335007cfb15029bc00f -size 24298 +oid sha256:cd9635cd3c39e620a55795735679be6bd20a3e3f036027dcd7d759fb6ef8ae5e +size 24201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index af60e5bf4048..fcfda4fca96c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa83a26fe46b1899e16dd18d6784adf76603f15e3a426f7848262c18e0282809 -size 26148 +oid sha256:e868d7139d8586fe3ca29031655acefa1a9fea7b3a7ca2f7df1549456b0308e3 +size 25984 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4d7ecb884b87..345b7e23e6ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be641250b21c18c93a8dfbb87b8b33bbbf34ceec29f290e6cdcf66ad2399bceb -size 26016 +oid sha256:126d97020c408656c223e22273206ed3b6a096bbd9e39c0ae24f175cbb11bacb +size 25906 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 22e835d6ef66..ca1a1cbe7dfd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ad1752b457e57eab0e72fe95e69a8738ce1bd27bf007122a138e3f55a8e0698 -size 23657 +oid sha256:8b44c1c6852c8beab29974d6872404b0260eca8050f7e539d49c0d9b3ec58c0c +size 23548 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 584835c03268..d7be0c8371e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aaca3b525f707beaaffd586940e770194149f89f66a4a0cf9949052466583848 -size 23139 +oid sha256:d2e4d86b23d3dc7b567ef852935d07f8cb5a4589c712a9e5c7582a88a4360777 +size 23135 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 88241649226d..a02e84adff1f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5afb8be2cd13ff041a5470e987d4194d40c6f1078a8581086cb9b585e2a3e7fa -size 24753 +oid sha256:d8f493ba2c230b0e74fb3e438133fb2cfb11ecee70ed01c15ece2265e521849d +size 24779 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index fbeefd1cd4ad..50b59f36f20c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8de2e1d606a4931985b78d07506fd2e6ff298a409fb07421385d631e0ab1c698 -size 21694 +oid sha256:d0e92ee904244a1100349e3d57fb8faab59f2ec757d279516e6535532dab6bab +size 21669 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 824432e5a9c0..6bb11bebd9ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d4a2a6780f054cf37bd4b4055916ab48a19cbdb5a2c90f2cf1e9e4902f55ab2b -size 26911 +oid sha256:32928f317fbe82bb3765207dcfe7916137001113a2fb2609740ecb3b902061bd +size 26871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index e961eac8f384..cdd187760f44 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7fea5a82a149a90baf64ef5e075d0e8ae035f39ce0adf85c64d94f4f069f70de -size 26598 +oid sha256:5aad70a3141a2d5be7a7786e47142be8df2a12e307fdbc9650d449d21c3b5f89 +size 26350 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c35bf06be220..6f1897a71551 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:95a4878b1c5135e9cc04240230d72a78103bdb9d759a69981f2153b3a1f2581a -size 24111 +oid sha256:f635a452e5ba54c0ada1ab74db73e8da4aa221d5968d14b2635f716dff35a807 +size 23917 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e8ba9a35f319..ee18e66be0e4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0cf6ccf36d2cadebb1ad52e72e1250df5f949d57e036ac8f890644b4230f028f -size 23765 +oid sha256:d1ff7e167aebcff890bd00ad7d307749ddfd5dc6545a59cbbf9e7a52b3157def +size 23601 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index c4ca18b504f1..b4cec3189410 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cd666f33e5c04883aeb4e10c9e546121c5c79c8638bdd5e3ae45eda380e0d882 -size 25687 +oid sha256:627b85999b84ff74ce6d1b869dcab535c0fc3c57c04ca4866b1007380acb0c23 +size 25639 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 379ffa8fb6a4..fc0a2f6dee7e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:470b13a4d6bedac78b7c2b0908b33c2c043c1c96c2f40587a3113bb5d6891087 -size 22465 +oid sha256:76f42475522126a88901a33e706b67bb45e7d50f9fdb9509dab24fe4ca4bc113 +size 22327 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 16bbc41ac618..1017692ed5af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8a96a9bd944f9af6d6123614a1be480fb3abcc49074c1dd1ad63f3608e5b715 -size 26349 +oid sha256:c49907eb1b303b700484c6559a2f0e41c6ff689462fe0019a00f60fdc1e1d24b +size 26136 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 11b44bd44b47..f5dd55c9bb22 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0335af3c63fa4c8c008836b3554b5b89e8222e64b6eb8644aef04f31677f4ccd -size 23932 +oid sha256:66456604369c420ea674a829e25f008cbd13204d9e43b6d1764c0a6b931dfe45 +size 23805 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 81a957695d1c..9d01c4d41b9f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53a1f804906aba43aaa73308f6b75712654ba9c420ecd07b001783d5a3e7b35f -size 26959 +oid sha256:aed034fd1abc572f2f3b3abd17c8a544d582f8f41a6f16db8e7881a049907cac +size 26905 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b141ad8c5203..bf4ebf2b3c49 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:abfaf5254b45a0e8beb84821471f064c5650f7b7a825f55d8423e75c55d3ec3a -size 24394 +oid sha256:d7c92fea21c8b5e055f71b2ea49b63a2ca1ed173b89447e63d92ed017c0fc489 +size 24267 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 34c10c3a07d6..038e5649e614 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:67109e25e8ec9356d1f4036df75160b1529830d0e1374b8c8f3bed29922483b8 -size 27882 +oid sha256:1db51e86da821001a5416ac63d5b064a6babe950a673168a22c0dc094449e282 +size 27731 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 9b78575c4a20..0debf66a0956 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6161775b9d5389316074dd33ee50cc8c0d7c288edf3182dfe0a84e44627199a0 -size 25025 +oid sha256:84bd1458e364ef0a761b156a17de5d9e86a364bcdf69970dd3600257ce1c4749 +size 25073 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ad4825aa21bd..2a74c2b4fa60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:74b00fbd8e94aab02e79c8c7cddee3f49446f5196770cd9e93efd14fbaa2a0f6 -size 28230 +oid sha256:18135f0a2d38ef0d341dd65ea0213c024432d287b44a1f7cf14c7b8a09549e2f +size 28055 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c7a3b7b2d0c1..28b29221c62c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36cbb55000b9900d775416d1c8e2498c1fd3980645693e07bb6724985d9530d0 -size 25774 +oid sha256:47096ca123dc5afb5225987458336d66b6a1e99b9f76c4c14250957129ccbff6 +size 25734 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index af7bdb1cc54a..85482c8e0b93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:266cbe74a1d3f92af3fb4e0c97c38166ee781a4c94fe890107db304e389abfa9 -size 28283 +oid sha256:311cc85ad33e515b13a93459daf5167a6014bbf90efe0f9ba849b5ea631300ca +size 28191 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7d1d1e6f92af..ab8416a1e01c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ba4c039f90bc150c2b21e68a678cd784f4b006b11f607aec1d9d598f05fcb506 -size 25588 +oid sha256:051f74f598bd8f700e011f63e2ee3ed54d60f9a5a5ec2aea0cf2976452d170d6 +size 25399 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 00dc082820be..00eb227abe4d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9dfb66344ba43182b6022a7f8c78fb942cbccf08e74114da033e9de1cdce37ad -size 25920 +oid sha256:a5632fce21f7897a832781c6bc1849de5967039776e2731422470bb696ad5e66 +size 25808 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ad010604db24..62e169ed87f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:254ec8a929c2c27f0cf525b09c180c6140eb06fd6369994bb823ae3d33497cc2 -size 22902 +oid sha256:995dfda71699ec0394ec59623ef24136a9fcc8ac95ad692f026489a6cf2e474b +size 22871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index e92953603598..d2acdfd89843 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50bc74bb04d9ad74ec102cf9111af8345435ca6b55dc52bbee0d2ae3808276f3 -size 28681 +oid sha256:de9c4b8d57292d76cd719f552b6da2c5367f8c543261e65b5146d73b6b92e27c +size 28642 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index dbe6e4d4e498..723bb9b75d3d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa1deb43fef934e565671f92041c935e3c593456e205630f5e981eb09d6ea4af -size 25969 +oid sha256:fcc6041a92ac183fd982f20a6db83d0792227d41ca842decd5fe4ea2b3f1e9e9 +size 25848 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 5cfd6116f0a6..6c4bc4553ef0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef90ed7d3a90b293cc38fadc02597880f161716d843acd0d9bd2af540979f1ef -size 26551 +oid sha256:6cab5d790a56bf00b988881b1c8f532a0909d874115d7aa8a16779a2fe4315a2 +size 26538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e15e66689d1c..4198261489cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d23b22c9c8fa4f19bdc7bd3fb75d47ce69bcf0be03d6a957715f1653f01eebb -size 23531 +oid sha256:f832c4a6e11a9595f53c5c339fe8f01c67495f132ab4d5691a808578ce409a62 +size 23488 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 678c5942b311..0a6ab2d91436 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c636c9a9d46cefb3feb7ebb136bde18e2858c4f25b72044f52618c94f57fe65d -size 21387 +oid sha256:20a2fc74339cd2093e6ad010abb08bf479d2f33fbf792e46d677a33e1288a861 +size 21283 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c69ce9d0b5cd..a3c53ce4956b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a7197e4a76ba2716a00b762b6e37259ce07b2076bd138f31eea1ff436dbced66 -size 18371 +oid sha256:8e8adf5e34b6169ae8053bf1c52aca78b0dc22d339df191872677284d8e1f3e2 +size 18262 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index a681fc88a4c2..93acc721a596 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:75acbe8b49a6c0fcf88df5d2792f993d179dd1f8ca064461f6efc67ed20a90b1 -size 22147 +oid sha256:28e1a763912119b47b9fccd5c110e1b365758336108492e25fac2c1d81254d3e +size 22129 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index db53cee74d97..db7907ab785e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e91f4b0ccf1628d62c323ef727d7e7dbef5b3773a076d549ed73006a35e75056 -size 19392 +oid sha256:113715730662fc013d04334ff8d9da3504a5e29f95a62bb2eb8c4ac283ad2cfc +size 19236 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8f0c1fab69ac..c09938ba601f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:138e4112ed14688a2cbf3017a32ba4016470fa0bef2f8b74753e0b06d18005dc -size 21149 +oid sha256:0e015eab607442ef5010904cb8e6856448c0866e80e5fd5dc67dca111aa1e2a5 +size 21051 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 114b30d6d460..810a06cbdf12 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff933f77576f6da62d70513babda91a2145ad279781c170092a4c86a9e5fa9c8 -size 28429 +oid sha256:2e3d796f0729c3ad73575fe4f4fad5ccb72546ac29c616f3b763e9ea8e5f8dad +size 28428 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 322157b21edb..6807b76f20be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef0c07d3afba169c541b9e806c63ca194d156f5a87c6bd46477ec5ff97255c62 -size 20754 +oid sha256:0a753138618cee09677365b9af44a3256389594995aa879dc04241b11d9fc8cd +size 20612 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index e1b2447be5a3..4b2b43055348 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3cfd945db7d05783cd302a2dc5e6814eae3475f07b1f8dbbaacc1223ae068e93 -size 36682 +oid sha256:60f259c5e8aaa6c6f9711846ad53907358a89b91e8c159d5d4456b5020721da4 +size 31397 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 0396dc473ecf..6cd0d38eaa93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a8f884121105ac61830609ac82b78062e7127ee66985b9376436a56846702b4 -size 28528 +oid sha256:685e9b02b0e7461a658335fa0c9e962d43fa3addc9fadcb739d25bc42c15cc02 +size 22120 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 334e801385a3..594447e709b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9c0ddc49a4318febacbcf8094306db236c6af849b53595b2364e868bbc62b70 -size 28811 +oid sha256:33c39337d3585c308e48e1cb00df27749f72ccf596ec5b4636ff164ed4098427 +size 28793 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 54f5f3a30d28..297bccbf2b79 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c2e4379de4c9f9029042243465f6c6b971a09796781438b6a130f562f1685404 -size 21598 +oid sha256:4938e8923c9be3e903078b00034df54ac694accc0248c157d2119f5929be435d +size 21505 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 9a719f04f536..37b2deccf40f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4792b5a2a56d136ba62ee694b08c71fa9f5df17acee3329e104361c8bb1ba084 -size 18261 +oid sha256:84ea6721103e32cb11604c98fbd634e69cf45af98718617a13475ff50707c92d +size 18123 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 82626f54713f..730a5e82627c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55e33f5973cdbd79de8ec66b3abbf1652ce22b651e8de26d460df28017350e2a -size 22195 +oid sha256:15313ef820220cef1c705b8a411eb0219c3bc3939a70cac9012f8f7f4798b8bc +size 22077 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 4bb40f9126f0..c8a66b426f70 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa0b7604a26ee4be8a6eba7c15411104ccec640f0cdf4d5205fad7f55cf960b0 -size 28104 +oid sha256:476194179d0ed57b24657da8b816f77587a1f455baaeba6a8bbf7e0744f0ce60 +size 28132 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 6e34478e1cf5..e146a76b79b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0cce474d83ab7089acf6c26496e489d3387cac147cca6e9e99ff471e69d9ad92 -size 21900 +oid sha256:ff4f00ac7f00958229d2bc5a45240751f3a3991a88b14ba918d7fd93a45e0fd0 +size 21811 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 2d2e8719c0c6..4f373c848025 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0df3990cac35faf56878ef5aa15b9380c8245c0f41d3983a03e76616d4d57701 -size 36965 +oid sha256:02412051b411fd09218ac063350bcfbda37f37ff8f053aa7282249bae6d50269 +size 33828 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8ac8a04b1a05..feb6143dcd7b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:903a2b20e87bb45d126da482469320ac5466c709e0be0d23acab3a89eada511d -size 29822 +oid sha256:edaf23c52c610a0b9f7942c5ced16292b1698b121e42b6ebe10f28341aff1532 +size 22968 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 9bea5bdafe92..da2be4d84f79 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2432e87b1ff8e8ecac0416adf2bf0f690795435bf35968d4a9ef276d3c639ded -size 29745 +oid sha256:dd1ebac7e4c31cb587e628a62cfdf9f537ad158e8f2821e5115ded32ccb60c9b +size 29669 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index b7c18cd564e0..5b79cdec8326 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00940570df87c6ebf23e8e5974af75cae5b3726868a1648a68b76a199f95e007 -size 22604 +oid sha256:e22b8697f9adf4ccc43f4686d3fc296c4ab0af32188f501a8a81697b3a3b01ee +size 22489 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4eee788c731f..7d61b5c212c9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fe7dff754de8333a8458eb8cda59083825e439202dc385ef36541dc327e133fb -size 19156 +oid sha256:3209ecd4d113fc81122dcd760102bf6d1c16ffeea4ac7360dfe1cd908e6de5af +size 19027 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index df2263f61cf0..ce078472f963 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b80e3eb26924dd30bc247619385ea6a2b74f3d71209806aa16785460c585a0b1 -size 23451 +oid sha256:b2982750c6ea15e95cf6b39ef9987380596d55bd252b4622502338c7d6957290 +size 23268 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index d66d1f41678e..f430d6c164c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a21e759458b9f90be7c0310b55b9d90d43a5f0d28d6da04a00a2d8821c5713ea -size 20417 +oid sha256:7b0fc466a22db34711575c236a6d78b40b95f4d8f2998944ce1e014a56bb1627 +size 20222 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7f9469839f96..c9cf9369ada3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:937bb2e7dbc996912d09eb0dd391d4f0eeb00e78e0220fe1c58500aefc223181 -size 24339 +oid sha256:45f0e8a7c3dcb4d34e079ff1652154a45582dd7675c3b06a17fcf19206debb0b +size 24124 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4f96365aed68..f6239713fee5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8bada5e91d03018e301fa657454fbb26ddb7796c85216e6f0b00a5883c1da63f -size 21382 +oid sha256:bb183a988804010589bd311d51bac428934cd0355f1e3e999486da1cc6b14df7 +size 21155 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e82a97e90939..07d1a439dade 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2740db4975c9bf465cb9f0eaea3c5eebc5a051e53452b41b46dd5f073464cb4c -size 27150 +oid sha256:d546890bb798550abfb5ef6ed01cdbb0ac700e8a1f25f7a9bfdbc257e0743c1a +size 27054 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index c6fe7cd77d34..390a70f8a992 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f6bebff98602336fd162ff5a585183316c0e090bf79d97d8d4b874ab15f33cc -size 26068 +oid sha256:3af6ab4dd07849f920c0897b68892ecae819cd0cdf49a028f63b57ac3e9578d7 +size 26153 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4ca95d0fc439..ad97c7acfbbc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4199545f19f951db1cab5e7d044c0ac08aab5dc967a2988ca226e4ceceef035 -size 28338 +oid sha256:34be75e884f0587d111296ac58ad70fc47aa5afa8eda4c0eb624b8e377b305da +size 28329 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b23b106b8ea5..422a74eb20a3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ae835cedd885f4db301360e3a634150a283da7af91d087e9d0627d1dff004b7 -size 27090 +oid sha256:9c9b34b4f9209b95a2ef690b5d1dadfa58fbcf879846b90e44d4657d1e499644 +size 27048 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 64f55a941a08..2235a452b3ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd24e75fa65f8bbd76fa3edb5dd268048ffbb57876b86050cbbefa47ac051013 -size 26043 +oid sha256:324095f76e3223b23fa10d3df7d3583bdd937f4186e61e8a4335aa720948ddcd +size 25943 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 440640f93840..92a47ab1d0b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f75e997bb2d17180bdd10b4f2ac2c5f26ba3e2f9f079b244f6bfa6c84b889e61 -size 24763 +oid sha256:f91e7deb294ccad969b40d1474e892480c1a6d59c833593c5b15ad2e0ae2f6ec +size 24599 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d0123115a813..427b49ebd3ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e30f0bd9cb14b37bd52430c87f473461bc071e8064b54be337aa8d9937a7e7df -size 27360 +oid sha256:60d856effa4684ba08f91d41369b76b046edb89a3fe67d7f09d4b0adfa8ede3f +size 27283 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4df8b8a6a805..9078fcd184df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0eb7cd629016ebc5056acc5ec5161995a61854ab77002c45f51e3c5aefde5b4a -size 25857 +oid sha256:342b6a5cb7c22acfc98dc6eb267cc1f345920cad9c4b207dcb0e82c4fa8ec11c +size 25663 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b121bc3a2509..0624c48d2ddf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a690eeb2e4614c11977b0e848ec051c7a62269b35205ee6101c90367e1094250 -size 24178 +oid sha256:a58323f35e05ef9aebabd64160a48f45cd0a0249405d4151b0361e05927b0a5b +size 23989 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 77e829733b72..ae3a69a656cf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aec0e975547d8e374f708167d553171fd8055b3665a5a4b5118111eeefa1fb7b -size 24435 +oid sha256:dfd9799b68d769e29096c8f760cb6a325535841dd641b50ccafbf844b2e275e2 +size 24386 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 3adff8cc5194..e13ab76b53de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9cfc91866fecd5147f04dd102131d3ed462165945e1ce53a75fcace399807167 -size 21946 +oid sha256:7ea749ffd20d735db10076f28fab3458a75e60784f33b2c413c64b50c42c78bf +size 21821 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index dc4c1b1664cc..ac33b15664b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88555675dc7604bf78403eafccd58dd37911e4660391cd59becdc3c405e410e8 -size 21699 +oid sha256:80365ca2d383f8315ba257cfd394eb6e4193890d92110ffd6b3fd661fbba2729 +size 21667 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8d8159d05926..ba935433cc87 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef59eebf98cf862e5fc999254cc75225f42c2c98feaa01a2044640d21800a8fd -size 21946 +oid sha256:34b86701a96f16a07601ba9a274dcbe7eb4cc9327446effa7a6bcf6608ba5b46 +size 21933 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 772a58e513a5..37467ac3edfd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6444905ab71a641b09a77dee9ab4be642445f01dce08638395fed4192181e400 -size 19312 +oid sha256:0944071d58816abfc9f2793dfb225e9765dadb9a3feecb01eef7dc3392bc20ce +size 19232 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 32d20d5d07e6..f8a2e78f49dc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:482185a5ae8eedd6a54f21b55e2bbdafa73fb929ac3b66abefe914199c2b058b -size 25314 +oid sha256:31a77cd0b2453059f8046ae4dead001d862ea0310fc535bb110db0598a776ddc +size 25180 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 58f5e57831a2..aa20a81858d8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:579a59dbe34197905ad2aab181d83c1ce1be4b2937e9e517b877b7595b2f299e -size 25470 +oid sha256:286eacd4f2c69757c114b67cf1ab8d351c5954d8b2dd4b7e02df2776d46d29d2 +size 25466 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2aaaf9ab463d..7234059704d5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a550427c2ab4fc361762c93b0ced9f09b4838b460d3159a88cbb74f71e82421 -size 22845 +oid sha256:ebdbf173fec57a80618b7c9687e28231ac8041418af946f06d1a6ac958bf01bc +size 22701 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f4ad09d29828..b8b8a7477dc5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ebc9147e7b8e2a5fa0a78d5403d538bfae6c66f4e78e3a05b780dc129568b54 -size 22888 +oid sha256:e7793de1657dd02f4cd6c8f1e3c787e7d8074c5c4c9b05e74f880c71882ec4e9 +size 22837 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 4b03099aee8a..a99c2f6e3136 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7c2616de11ca121b7abd275c3833c31af2895a3213378067d5e7e3fc83042d32 -size 22977 +oid sha256:970ebe529ab1983e022dce66cfa4526e748771f4d3c43feb0d0b12981ad08bed +size 22933 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ceff9d2fa3c0..e33f39df2500 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:313fe337ca624d5d7d9d011f95699e1b87a244b99883de3d0bd665882be52368 -size 20314 +oid sha256:6a5fb2700ebd2c54234126502c7cb06c7e1add12f7edc4419c0ae245eec30ac3 +size 20284 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 562f06185019..06e85ca8f4c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8d2bd1017b68b6636034e47e0d2dcae1a04989d2a344038a061afe78e0d0e3d -size 26742 +oid sha256:46960c8d092f258fb4b84b90128e677623894e827905e8365d7b959fcf03a095 +size 26650 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 46c7ddb9b32f..56fb86be1f84 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b1ee4c3968c6d4edb6b2f009bc1fbcfe2b7143c72271afd988e06cdfa7f08491 -size 28045 +oid sha256:574c38b150ebda97fb20d02fce0528d434d60616422d1d87aa36073b82049651 +size 27914 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0d6b28c7ef2b..0b4f231272ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9956b523df1bbdff88d6efbc2236a41842ecb9814e801b6f2b37dde99be703f1 -size 25741 +oid sha256:43bb566146c365f6604795d83d5289df327fe932d5d39e57e2b1bf32fd803249 +size 25632 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index dceed870d1ce..2bda28548b09 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be3e8c58311e3f99fdf1d1f42b5ee846bcee0f42e211aef70a3455a053664f21 -size 27109 +oid sha256:aaded1e598f3310392313833f920dd3ed3ab12ced6884f26e53b23e0d19fe6cd +size 26949 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 66027377932f..10cd4a400ac2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9f648e964562e185508fce93b73564dff81281db6e51ac80507489d1a4e823c -size 24373 +oid sha256:1f1a40c2f10cc4dac6e06d2bdd0cf35c71da8c836c426c2ba3b398ab78c00b72 +size 24357 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index db7deb57ca0f..45b15183f4de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:396094845e2eb466fce6073c26a6a1df6ab854d5bcc594158081d49fbfebdfd8 -size 21179 +oid sha256:bba75fc44edf306931aa6f893da922aee0a1895fd0c88f1b0fc664528769bab3 +size 21106 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index aba38987704d..730eb66669df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:69f00e09c4b1de2c66828081123ca03d6f949f662c413b1264dd853dfc66da0e -size 25577 +oid sha256:fe3e5932ba391a00b490f5ff0518f264d668ffb9effda455fbd02182591280df +size 25572 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 269bcf067c69..d33d54cdc42f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91715e91dec03ecf0a71f154d7b2b5ea43ca05ff40ba87bd7875ed6ec834ea00 -size 22479 +oid sha256:47d750df09e6ed477f5d3207da839e7d13c4da136c52336df381274eb58bdae4 +size 22411 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index bcc404c640b1..b6ca82d7f895 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:795c2a0dc85472b5897f98e10dca62520b8aa4f3d2676ccd5285345b84edacd9 -size 24177 +oid sha256:afeb1a54058f8606594b86cf534435502d74174515893fa8b18086a9079f860a +size 24021 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 41e4314d83d9..a720b146c697 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4fea14a3f85ae433bd4335475a84f3b263556cd6f02aceb29f3a9de0d9dd3955 -size 21682 +oid sha256:e72c3bfa74b19611e6d1744be9eb3475e8cf9718ae0dac02afff230a26a15046 +size 21549 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d91a57ca608e..7cddd8999774 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d20ca5c581f453bd56fc0b639153ace3210040a2bbdc72c233004d2d38c22f2b -size 25338 +oid sha256:7338f606f302471d4103b4b23742e1645862b7068e2394333b5c6d2747cc14aa +size 25166 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2f91da56cf95..d0d03eac15ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88a56e38c901db8636e963fff557682133b426437f87bc0c99fef326105a10f6 -size 22841 +oid sha256:1f79178b2be2f20052e737712cb597ba718c6f850520eef220971ad2705dba77 +size 22753 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index bbd4c9fd412c..bafd0693287f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:afdbac2e6919fea94d129e75bee307f5f4d86ce8dc95a07f8fd8c92060d5e1d2 -size 29720 +oid sha256:70046d73d006f64f62befbe076e4ecb413c1099376c1ac481be86f5b178b7290 +size 29504 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 01ab53bf64c0..b5f040fe2cd9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84afc82c420c5a3e960c7d4f1a2efd85ab328e6092c33315f9eb0fab6b6903a8 -size 26903 +oid sha256:b26847f5ff68d61b4791cb6126317d69ea6fd8b46bb0e58c47c85580be54bb56 +size 26835 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 84551ffa96eb..8d2993a32d59 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e0c7e774b2e574fbdc5fac2869fd2639876506a44d11bea11ced776c07cc010f -size 30898 +oid sha256:081bc2482258ce314b27f3d737ee13bc19c26109dd9a3c6dc5c53db0f4b728fd +size 30868 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index f6bd7b7481b8..13e3c2e41221 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6c12bdc7f3d970584272dab93a9d6920fe763ef010dd031c6620881570b73894 -size 27863 +oid sha256:5ad6b6627b985b3b98f534215fe485b1afedfa4504ad1acf045d3329d9ae1173 +size 27834 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index af2df76d3348..48f4e9d46dbe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b1063989516f9311b55feed9c46c9f961f00d6d46da16144f387e3bc3ff0e2a -size 28761 +oid sha256:1053a226bb5a853a757d61d108b2179c1958ff5c5501a96323308532c02743a8 +size 28552 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index ec2882d4fa45..ac262070c3ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7d330f2f567561240d8b42b33bd4343a0f16d2201f04d7738fdd8c587bd6a90a -size 25816 +oid sha256:490fdbfdf7e862c43d2cea53b66e7de7d094fb6d3c675827b321f4b57556ad9e +size 25612 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0f978f2ff1a0..6e8f4d4f9bc6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a9b3be2f7a9280a52500cdff65073b797bf5acb2ddc9262cfb733afaeb31cfd4 -size 29893 +oid sha256:d93b8966167a4178ee3bd217e8b0d24760d72a4b2c7172908a6b5ebb41af7d22 +size 29807 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7aa4950f6d59..4d56604b5f35 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aad35c00fa62505683ef01d32c1c44b176602bb708ae09378f4dd20afa289a0d -size 26596 +oid sha256:e95d096e20c0eabe1ea2f1c72d6b150a5966a9b8e79eca28f49df600c0c1a20a +size 26454 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 437e405e5c18..9f58e58b9d3b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ee5ac3a32fbf56848716ce5e11d5dfda11e0e13de98ba3729d2019eb382b853 -size 26953 +oid sha256:9e045e81ea37b3e18d93fecb50074c69ec30cb5c75584ed280e8c6bf0fb717d5 +size 26749 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 01032c64b2ac..2d71b793527c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7c26f79bc923b523dd679a71892bb24b5081c1b42e3622bf6e259783fdd4fe82 -size 25871 +oid sha256:fe034197581f0a2b881676d89392760ada152f737a8003588f093d16360b7318 +size 25832 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 3a51f748b471..5939e2a14324 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5723966efdf369454218c73ad3a8ce27d0b38923fe283eabac48620fd1dc7633 -size 24425 +oid sha256:de638e4a5f0942b3f810f90f0918b99cf73eb92f8af8570093c1cc2569129bc8 +size 24220 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 5655bda001dc..1e9130c8f0b0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d3f2025c4634a9a80746c4bf70db22461d4a118269ff717bc6e9fe9ce70a74dd -size 23921 +oid sha256:0c8bdab16b8125eeb02ff9f7a04aea30031a57cc4eb4073192e972a94f4c979f +size 23901 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index e63534a54cee..86f468431e6c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db31d8178d3ba29fb98aa649466d4e7f3b3e6e8ff619cb1ab6a467e44d9c7b9a -size 23288 +oid sha256:a9e9c1fbd29fd605e42408e31836264cf5d2a26438a763063f70cd2708925ba0 +size 23276 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 85b180d722e6..298ccb84cd05 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38da3b7698a623578ecda0396088b3fd08bc3b0fea1facf3883282eb36858800 -size 20439 +oid sha256:afb79272500d3e562a4ccf0687e7ce923aa485b2405251b6a3173310fc32668e +size 20447 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index be3e2e42fd61..11a5a2adba4e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b35f68bb152227cf926464c71625035fadc54c5ea1a57b04edb29264e8ac47e -size 27565 +oid sha256:1cd3458b889364c4bdf455437d8e8f503159bc41b6afc7ad363a1b3626430cd4 +size 27348 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 273f619c9b41..e1e984c7a3b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5e09b1e60447ac2d753dea54c51c849bdd8f807085ea98b8d39fe28ebc832539 -size 27062 +oid sha256:78d1fb76d21a55de83c16d12ecd7373098321f047204a7a7b3317b2dd4ba8acf +size 27032 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 18eb12a5f61c..93fd5fde5ed0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:39df16f48eeaa2e0630966581c13211f5fd1de194012debb5f5b490725e38385 -size 25352 +oid sha256:98d80794425f85913d569d289c1cda64746fd0d3f73a3e054b36c51fde09156c +size 25107 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2fbe4a572fe6..ee6930972015 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9517a6c9a676afaa09ca65749e098ddb6ee55c538b8ab5b0e032f6c0b52f4262 -size 25180 +oid sha256:c32f4b224d12a2fc6307f2754643ed8654b0776f6debd2264ea81f1b14354402 +size 25163 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 7227644f35b3..44b71e5e3004 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:186fe15fa6562b8e2090611488bb28fe1239f1d443ad4eefbe5fd1abf9f65844 -size 24215 +oid sha256:65dc7cc34c6379722ecc21dc2e77e8a480f8ec38a6b432d45b72b6162fddbbca +size 24089 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3d6a47e1a13d..6bebd53d758d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:832b38eba22f4681d13379ef838562691a0a3132800a3c4c9dff02dc466b2e07 -size 21751 +oid sha256:73495771f692b30abfc4489e8c26c113cb1217389659518b70297b52c80ca470 +size 21754 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index d8eae9ba0a33..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5d4e8d3e2ad38f20fbe186019d168beaed09a9c661cf8ec25afe31cc9244b45f -size 33523 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index d334b9f22795..0a739753d9eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3925127143b785325a51b8ff48e2ffa0b262c298bd06705bc2e009832bd05431 -size 28423 +oid sha256:a704a0936e5d6f0d237c75bd0721c4cf7df98dfd317e37ae5b72991d95f38f20 +size 28443 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 92b09694aa92..3fd03abb9d84 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d02669be9eb607614b3d436120e9fc9afee72cf323665e4ff828b85e708597aa -size 20782 +oid sha256:293e1a4f0f1c2ff2f33acf78d7151ce1ae30eb859d5b83245579b07ed920c6b1 +size 20654 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 720b258b6be6..621f6dc03cb8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:312cc48a4dd559bd93b52bde12acc3031aabdff23e643bad7a45c2b37a2367e8 -size 36870 +oid sha256:84d3fcc08d8fb89c45375427c811a683893e7e5a70e07c61ab397c13babf55e9 +size 31495 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 673771f4adfa..3b014705189a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b2377e81b2ea6e280b86973edbafb03458a22ea81f0d636034ceb99ca678930 -size 28534 +oid sha256:cacc5cee4b2610d90d7c1f8b4f394387d2b493aca93d14f2045bccd64b8aaeb6 +size 22123 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 5eaf33998ade..b546e9f5cb85 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13829e3c496f1d2b416a102b598b76595e20355f4bea9f05cd9dc5c6edbd96f1 -size 28811 +oid sha256:ad75602b21ddca557e0171cc8f687895b38a24bcb395f7c5f8c3744e2e562389 +size 28791 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 101fedc88b77..f734fcd9e9b8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:77956d2ac7db6df106beafdd0b75315ea1dcef116c7a9d37a4d999c4b19deaff -size 21598 +oid sha256:fababe8f5777db5fdc44444b5b7ad0bc14eea809fe09f481f9a1526b1190f51e +size 21503 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 7b28871b6839..74f9e0fb4d59 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8b90ab0240302695063c5d88cbe6922b6e4a474108bf4f57cb5c854a5e383c8 -size 28117 +oid sha256:c881a1ac4220dcda6c446e95a837e9d565d647ebd404729c2c51bb7398059d98 +size 28142 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 5077cacb7bac..6923e54dec86 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ccda5410195b01c072cf7aed862502e6b4510e18eee909f6d2d3f3648071a46c -size 21890 +oid sha256:cad06cf3ff5cb586cfebb5fdfe96a4538e8ca5c8f01baebc6a17c3eb58f7c7f0 +size 21792 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index e6eb73ae45c4..e707f19e8fa1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6c75a126feb0b3baeac5928f898092a81d59d2fb42eae5ac2ec16830f83e0c15 -size 36978 +oid sha256:518817834f5f0c72fc4dfaaf579c84125b45b3681fc5e4523e262bd302f5eb62 +size 33833 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index c2cee4fddea7..7d333f6b8bd1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bea70c983a780aca324074a23892de2ec5305776f79d6e13e3ce62506ecd775d -size 29812 +oid sha256:9d1d16cdb8f8bf5bdae5b318b270bec2ac1541d831afa015ebc0d38d77e3560c +size 22965 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 1b10ea1fb5d5..f5bf66eb85c3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bee85429bb6ccfe3d34ec464b05e7b31c1f134b0da910111f0786c07bbe284af -size 29744 +oid sha256:c80d5f6c52ba0248b67e2236d6af16fc69ae6752271b3fed760da1c3e8718473 +size 29673 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 4c5a47051cad..deeea7a31ab7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:943774ea1415834814d2f2eb7a1e31e007a253f7df0a3f79dba423aa39cb9dc5 -size 22614 +oid sha256:424861a19e14b7e5a3bfd7702d97bb814b955c879910b6c6d153d88c1a8d2d7f +size 22508 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 83743901a45c..30a5a39d01a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:627b96acef027d60a539a9ba0b5f3a68f4cdcaa54dceb8996564f4e7ba7a86ca -size 24532 +oid sha256:dc5906d04f94fe270ccf0297ed3c03efd13682da6339d6ac772cb7ae785a3203 +size 24460 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index dd5befee839f..68bc31c0ea82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cd9078e9747789c91ac4dfbdd88468b9aa339e3ede3e8194bc0f7fcddc45f542 -size 21232 +oid sha256:94310b1db24803b5b98cf752ad40e376bec9fce93ba905a295039f9736bd13d2 +size 21102 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b5c0372b542c..71a5b342f9e5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:514f3a725454aece3aae4c04afebdb8fdb0ea73c8454405ee5815c4f1bb8769b -size 24562 +oid sha256:9c8897a628682d19aadc97444c7ac991226a3d6fbc7d83aa8d8e5e7acc4c0193 +size 24523 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f3c7ca7e8a79..b04c3eaad81d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:29e435337f24315660c6e5b86fc0bdffbecdaa7215078e33f6f3010a7f741b78 -size 21621 +oid sha256:0e41a7d884e643cd4832c9c8403d790c7f9b85326ca12563dcfdc7237d3ed256 +size 21494 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index b1a9a644c8bc..f976f896a11a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88ea3d489b972c89cd7836afa08fad73c7f8bdfd4b36237aefe4a8fe65f8b783 -size 23882 +oid sha256:67cc4a0078b049cedfba15c5cc18cf25713f8395f84b03abb2006be031ef0acc +size 23815 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 17137d14b058..27da62533e14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cca4432507f02d524c9afaaf97f094c21b4970324f727fc0cf8e2ee722107c64 -size 21080 +oid sha256:131f9c259cdb40ededb993e81dcfd6beb3189609dd6d2a0ac595436e1273cfb6 +size 20951 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ee354f4907fd..fb20efc333e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9003c5a6613331dd4ab1d68b3a0ee1fba72e6ca01418a5c020b118e7df5e38c -size 24016 +oid sha256:59ea95176974ec2bc9dcdf37f3a6e657f3661ff87d4f58406564f8a7da829100 +size 23841 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 3b5ef2903cdf..88fba9e84817 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:15ee16a2f5a09ceb9cf5570d7fd7b128336bb034a70839c12a53ff95373f4ba7 -size 21434 +oid sha256:da1f2e0de52d94c66a1b8dabc603a7eb2b6d7cdb9fc503fe9c79fa750fb33ee5 +size 21301 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6bf9151706ae..54cd06b1d92f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83966d37c3b95a575d0198731ab0a2396b5bcfd12c16cd792d2839cea1da3245 -size 26009 +oid sha256:aab957ece41a1c858225d51e02b8e7904aeae83ef15edfb121d2a9bf76c1b32e +size 25787 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 88d08e8ebd71..606205daaaf3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d2310386db26b8480ec6092152e91fe7879defa6322bec80a77043cca07948b -size 23270 +oid sha256:acc98799ef1f0d81077913bc49959e356acb13f0df8bf1a69adc0200738f9e03 +size 23035 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 198ee9161983..24f0b6521739 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:964fa0a5711632cfeedddb37263dca883aaa4c8d5f4262cdd3b62c7d4db78666 -size 26273 +oid sha256:de4772cd2435fed49967b5a5376e01a67d6f4a81753feff16813d711c542b71d +size 26028 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d3402f6b4d26..56dc201cb199 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b639829620c32a6582e5e176d95d745828f2eebf794288f03ac8113bdb2d7c7 -size 23988 +oid sha256:9581ce412c53246fc648e7d242640c16e2448d8297e56b7ac896a725041748fb +size 23794 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 325ba5087713..c999b07a645e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35898387349ea00684bc0e52679a8db57404b63358cbabf07e159727fe3a0704 -size 29974 +oid sha256:75b853f6e3b6aef151194b076f235a9bf2a203d30908f5751bce5c3c5e34c4c5 +size 29922 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 44df6d950bab..4512315c51bd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4fb1eca2732264c3d356316cad127738c16a5458949c773ebd3d6f250a07d3fb -size 28225 +oid sha256:7a22cff0adce44d3ca7fc221988cd1738d88955a24c51c79cf8716311427860e +size 28194 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2c73679c4b2b..b6bf67f90459 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c3ccc8b9701212d5d729456b4e6ce4bf0a2dde2c87fb662ea383cd519fd7ed1a -size 31928 +oid sha256:0cc9304842b3fd51ab1b949adefaa3bbe93d9633326222002701433581dc53f8 +size 31713 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b478aa9e1a06..94d300fc5ee4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d011f8583af503a60f0656fee5f437f99a11295a153ec3e9c26e2c57d8a359f0 -size 29138 +oid sha256:6c9ce75da3e47961577cc70d0a8c93f3f312bbbbfb95dbe0572232b0e0010afe +size 29045 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a9b7a8f581b8..c766b93b822b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3497d3c53c2f99a1265365774f1601dfd56e74c908092ba67943647cd6414986 -size 30750 +oid sha256:b2dd59e4fd377f77ff8eff39eb63da1441ac88220d7521e633c9471a10e24fcb +size 30642 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index df0a0ec0835b..6b8a6014a5de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1dd4353960330b7b346eb42b110c33551616a5e6d2945358bad2c6b99249624 -size 26987 +oid sha256:b82d429908a7a2b9bd5f784ea41b08c7546b01c7d8bba8c049da0a06c1adfa52 +size 26949 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b08ec3fca383..8f72690af781 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26204620ca8f82ec1f4747dc281ae8f1673e506fdccc669cae7b6038d0ac8e9b -size 32751 +oid sha256:f9e893bd813d2d60896663707e83207981265f98a5b4d64f1fd1b09db8afe6ff +size 32564 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 77f0c184d6b1..c69a52ce083a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87640838f14e57cb2a65ee1cc22b6db884f49505672d62db95a8b4e1266643cd +oid sha256:7f35dcfec4b400f511b4860527d127de2172e0e1ae274108f7c38a9171bf446f size 27902 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5bf918716d5a..a169e40d2573 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f31181ec63c45781456f8b7db7101b0bcb26ab90f834ef693ec897b7e3899516 -size 27684 +oid sha256:60ad119be0114b022490bd6e3496b028e21af180d5c03157cc6e772595f44e5f +size 27615 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 524befc59229..e12d9cf39683 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fcc585dad9414f2a2267d784606ac731cf3776e101d7b7a29dc442d0830a3860 -size 28511 +oid sha256:c1b327e298ca976c9a8a2b7f0a696f398311977957f69e69e08096689073ea7c +size 28388 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 46be04d5cc61..ebc2b8f90838 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9964f6990ed1e308d0d7306fe415c5f10f1f14089b66498f3389d572fce5d90c -size 25278 +oid sha256:ad672f6a9b2c21a567cfc9f98b822972feac040ee9329344413b67bc389882c3 +size 25111 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 91e57baf68ab..ed7e66e38729 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:061e5281d9305e9ef42a32256e2ddcab13e0c65a661e9b3063cc7dc42467e6a0 -size 25592 +oid sha256:9a278fab65454ffc2c07f72d414044d15e05e732c8f828695fa5472d9919fc2d +size 25519 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index d3e1952b7093..936a66b0d6f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f2cbe3c5a0e8068015073395629cef523905798f802c0d9ec3a598a6635e562 -size 24589 +oid sha256:a0f26c88cd1550dc9df869adf53ee4f25b79f222e9c2124246ec5f4b721873c1 +size 24551 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 19dfbcd42574..9b89362a35e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:05377d9587c41c28e2d7a733d337a84fa635a50e660e05110339202043132654 -size 21465 +oid sha256:6a40f454c0d26b5cbd702f0b16aa7b959e665e2af5348297c27ce2931e4a83a4 +size 21433 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 34b485cf2d39..7ada16d4d3b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b9a0675e5961b207017f970d05c53582241d88612faffbd5540725d77c5427c -size 27744 +oid sha256:1ad15269c28125dbf76f48c13333f46fa08c46636320ee0bfcbdc7650b3d6f11 +size 27664 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2d4ec14df29c..1ccb0d73cbf2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:627e01131df0fc10de359dc84ec478da3da4a79dbf99a9b5e57bb138a80d3c49 -size 30265 +oid sha256:e594d82948521e41d3cda284c99fddb38e5183a14d7c9738411ce1ecdaf2ef76 +size 30214 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5ce2223534e9..257589744969 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:18a2658adecff150b3f3a98972bb4ce7f742bedcc92654b6c0ebe851edd4b3f2 -size 25605 +oid sha256:c083907dbab07c0203d4553bc683976c14a6ac3387e84e455f64b61714a01a2d +size 25433 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b2d944dd00c0..f252a2c2b9e9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1b95f34f59de0456f3742f5eda27582b43971f31e7a7f3458ba14ded553636cb -size 27427 +oid sha256:da4f1cb95f74e0b623c0ee733cf5ef6470ef50ca519f14c49183e4b041fe27ee +size 27317 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2decedc5539c..8e41d31807ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3b6b105301f9c0379d3052129f624fd68f4069a2d5c1994bec75f60fa5b0f421 -size 25402 +oid sha256:305d86297e2eb20e703684d03fbdc3d9edd7e1fb247848b41e5ff01b4951c555 +size 25252 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index cfd32444ad05..3d96960739dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee64e62ff5af79e95ef73278d419ee13ebb51799428d6a76fd134f494d5c442a -size 22460 +oid sha256:0ef1bdb351d507e85d90b0d4754ddba9af29356849c3e272425f89c096c881ab +size 22504 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3a84728e9b95..3c8dfcb6d63e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:437f7ca0b717fa379fbe11f6463fc49f4b5b2005644ff8e90a8d5dd98b61385f -size 29686 +oid sha256:5b10b23b017878fe5aa893dba0eb4f3b4f567cde0e3f50f83e2abb63466147f8 +size 29719 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f810692f3b38..8b0a7c73ee6c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c705c3c13c21c944d19c1add0b012e9d5511a7eced1554d28eeb48c4ee421e42 -size 31752 +oid sha256:960631069cb63eb6a1fa8c0d553ceb1cda48d3812140b73cc28fdfa8f8e5e520 +size 31686 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index dee3d5b4aa81..8c70e91c1188 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a3dc462c2a765ccf7db8737e4485572e93318e02092d8c359f27b1f1fa7938a -size 30823 +oid sha256:afe11780db154ab79cf02f1c62e049dcd67c9c6bf8273113d46563553355da87 +size 30759 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d09cde2ef105..d3f745b59d0a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:875b639eb87cb45614b0d68f1ed87db3cb83fce906e052b5c57ad454df91e7a0 -size 32764 +oid sha256:a2e3260a56b445e09b5f41b9d61988641daad0f85e8e332c10445a68c07a0fa0 +size 32598 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 27defe557c4c..843a9bf30d2e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:45679a5bb9d4be5dfb7ec06bfb1d7c34b913a3b43c50ecf852c7e07b9ed01355 -size 28524 +oid sha256:59dc2ba32b7b5f341786ed94bc430d7e5ef5e127a3034659adf51e0da8f1bcb0 +size 28480 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2398b2c1ecce..f4a5f909be06 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:da99973976d82f69fe8eab18daf62a981a79d1c6d0ade71749d7a7d31bca0171 -size 25549 +oid sha256:74228f69787d10411291bae1f6bde6316098f7cb20c64dbbcb60af51af93af99 +size 25536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c48461176923..82a0dc505a4c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:60556c42ca29fc87c47618f2fccdf96c39932bf8d98a61764255e6f4ca4aeab0 -size 30233 +oid sha256:96454ec1de42f050bbc5f9dee1e14288f5782794d214bb8c3b527f53c4fe27f7 +size 30185 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 7c98fd5637ce..a6c852dc4034 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9e63f6242f94f7f9c84c3fda86909bbbdf0686ad5ecc6bfe9066b175d476ac09 -size 27532 +oid sha256:2b04f414949369a7382a108fff8ae8a5cfcee94b48e353b4e2b44546d3ad9520 +size 27437 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6c826dd7cbd1..38329fd2053c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54edb27d639739324d206aa83e9cf96299b066e1f929cfa3e9628286797f8bb0 -size 27607 +oid sha256:6d5235cd4697bce0dff26743562ca20b429f90d85b604d3b0ff9d0bef2b07de2 +size 27443 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 7213159b8de5..ee6162d89878 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a3f2c58334b0286990eb0e04aa2841ec19da599d338528c455af7492b4900ef3 -size 24993 +oid sha256:4652a70a083c3cd37311f686e75e686a24628f6b2459b3427109d7bc60a3cdd0 +size 24864 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 35a3415f6954..aebd34a68227 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3a9b732588b0ab9fd94788e42890d091085ba9400e01865b2216590ca29af02d -size 27569 +oid sha256:818319be23b306e8c17c577020ea2495863aedba5e55cc701e5c0753b2a13d00 +size 27521 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5d5715b30256..409dabde15a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d8db6edb11c18c9a1ebd2228cff68236a8c9e6424c20abeadfbd3756b5f52ada -size 25363 +oid sha256:0417692bf2c025229dd22c6bc72dcd37576c2464c2abe180c30dbb5e2b092f19 +size 25258 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f5a38a582c44..2d34af4f8fd5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79bfd8fcdd65a5ba9e00284125ca194d6f6d94c9f7399f0582fa37bde9d3b741 -size 32699 +oid sha256:59c55b4e3e15808b5f85144c0bceaf55774c86692cc27722903aacffadbc166a +size 32541 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 11734163c8cb..6ce8a5b9ad76 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0263fd03f652306db88a3e5ecccd10361dcf7edb0033c8aadabd6dec3772bb2c -size 29171 +oid sha256:2c7036e035f838780c032cc12d3262125bbeba66b8fa3a99f78dc472f281ec0f +size 29147 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4fbc0d014e65..1396df925397 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ae4eea520a6cdcf4da745fc4dfe423f5981388fcf31f2b81c65184e840c7aca -size 34701 +oid sha256:bf12cdb450eca0fbc4d19954980fbf374d5807cc63cfeae066566679484cda79 +size 34536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index d52b257ce391..8fc02bec388a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:03b0f10f8ffaf238449220037f45a10d909c5095e225919ae167b8dd97d8f31f -size 30028 +oid sha256:73857ecc4d73741de8c5cf6f09934e7165f206fc214198e772ae789b8a9a6a3a +size 29984 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e74d9f6abfb4..8ca5652a888d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c432dd00211c3d4ae5d7322fbba2fc47139d2b7f66856f57071986f10bff329 -size 33530 +oid sha256:c5f329e84b44094555fc97d72bc3f44fe66fb0677c740104fcda883bbd6a2575 +size 33380 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 7181be07882e..1d82d515c478 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:874c3d6111f0d4a61b2cb1ee4bdb37321a16e07db68570d6ec50dcedff6925d2 -size 28089 +oid sha256:96c0176bf80704b87139f345815d4c1a28170a1d996ee770a957e366b284166f +size 28022 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 31df5ebe98ea..140016de8400 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:257b09b44fe3668ead265f30a7ce620f64a90431a6c6ad58325a37320682f467 -size 35509 +oid sha256:240a6efff3fb95bf47b3df36e0b7e003b54dc765e8d5cc1711e71c3a9edea4b9 +size 35261 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index abca69c6d624..5a9f85dcf68d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b1cb6580104da196c87e7a2eaa26f17443292da506c542ac6a102942f0afa696 -size 28915 +oid sha256:8a18e5a9c3c6d2750658a535db7edb18229b62471a0c9fd853f876c449573bc9 +size 28840 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index fc0c30dc8bd0..ff75d9f4ae24 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d144083de12fcf55607ce85a6f06f740629a8fca97ee9d60cff628d1a6274da1 -size 29916 +oid sha256:698e0395e712a2401667677453b85bdfaf6f15caa017dc340361fa777ff2a407 +size 29707 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8408eb0f8237..846290ab300f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0cd07c821c34fa6cc3cbefc5e98f84a996aa11dfe9e9e425f2d94e8a8d7a77ef -size 30819 +oid sha256:c1c53d25d77e27aa3837d15f837e70be3284fb5d6ac77664e77786a099288bd1 +size 30678 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index dc7c02fe8dbc..ca7addd12a13 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:457ba655b98a6cd04b1993eb1d636879466279609968e71984d6130affcfd347 -size 27103 +oid sha256:c962d41ea370fa4365fc35fb573f2abda95b4dc7bad3e815bcee832e139e994a +size 26899 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index af656689b564..6910d4c05ef6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80c6182cdbbf52dcee189f13d8280ae6fd76224e4f769bb7e7430b4d889f9044 -size 27797 +oid sha256:5ce3869bafd2729d9a2d425aae7532ff2b841cdd8240c9aa67aa6de5b8c50380 +size 27745 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 3fd5e564897a..4cba8a8a18f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ad47602cfe2d332aeedf5856d5a63342d373185d68f96a388df21712c117333 -size 25846 +oid sha256:0c4589e88ab77d45df222bb19f023278e3860b6227d5869eabfdd7733fc1ad28 +size 25663 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 84df89b9fb3c..c470eec87503 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d1b38719d74ffa0bed1ffb6faeab2de981e572c8f436be89e544adbf0c9cdf00 -size 22890 +oid sha256:ef702c341c1fd2aea33c3399b80ad9257c7617e5aa9e116e1a67bb4f31acbb9c +size 22963 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e5abd871dbfb..7c87aa0c4c4d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca5675e4d98335d7750bbeaffa60093dcbf6c6d17fc7d7898ab63a9959c40170 -size 29477 +oid sha256:69038f129801783896fa49465699e212300fc55e3944e84debcd0c6843251c9d +size 29253 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 85e81fa5841f..614991fb1be2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d5857356fcfbde5d51f27c8b6bdf591b6e92f0f95fa74758d120bd4caa9ecc12 -size 32515 +oid sha256:446e568f089202b408380975c2527fb3b6c6205a9adb656356c9b2e54d0c4ffe +size 32416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c7c1db261c25..d942584802bd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5a075d82db7022a4c3c46eb1e25429dd311b08825db7285ece45b128f042129c -size 27621 +oid sha256:2e26c1961be7ab26000fd02b013f9df9763319f976b24f2da49a80628821160e +size 27401 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 994ca3cb51a5..4be2e577a28e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1294b1d0afbb43a4a6c9ee073dfcaf20c699f39fee08f77dcb26f7eac8b93883 -size 29504 +oid sha256:fc8c98de46cc9e71fd872a710409a5e02ed59b0e235357e7f3bd2fc18f12188f +size 29282 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index ffd37bf3d629..c4b0e732c593 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2c1f4da8d883418d73bcda035905e5f5cbe3cc88fa63381264328bdf3ea167d7 -size 26616 +oid sha256:7920123c92ae74bb8c95b759758ea36c8b01dee0d599a55a2fb897a45b7f7893 +size 26512 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 29fe0b7e7ced..41baa8635acf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b05ef9eb27548cc5504ad3269e36ca1b4f989cae1ecee9598a81404f548f06a8 -size 23901 +oid sha256:22b9859c3eda86611ac70d2cbb74deef3a4d3f5732a913521e201b227fa8f6d5 +size 23952 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 706608255592..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:fa9990f5f6120eb979d90959b261673e192d9b78f78d0f6a08db790abba19412 -size 38801 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1b3f7b6d6526..2c3487b940b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f1a48ce3d2564cac40868947cc50eea7adc3cddd92e3311c5cc9af022c57189 -size 23896 +oid sha256:b2506aa58893d5a0de0eadfe777410d0cf6c3ff25b7409e696849212c671bfd9 +size 23824 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 5596298de652..006f95fa371c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc1489baa3bac897d6cbdce84a82d8e7f0f220c91cb283bc7e7fcddb4d1475b1 -size 21106 +oid sha256:8467f3e1135b98f7c63e27599f52d6099a13c605f27374dc8816b34eb41186b4 +size 20951 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 94ab44b47be7..9c091ba5a635 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce96dd77734beb04efb1930cd5abcee06e81ef73063d0853f699887bcc110a54 -size 24042 +oid sha256:39b094ebd384ddd1e4a66dcc4e66731367440bfea057081cd6df9e90180e96ac +size 23878 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ad5bebcf0e29..ea14d7b494b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6c02b80244b2042656a673819d6885223f027587f59df728eb05db11485112de -size 21423 +oid sha256:48f6e4d49912da49fa4f0e644f107bb900a3630c3d27234ca53c6f292faaa23e +size 21290 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index f520a5b0b5bf..fa2eabef0fd6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c58e5605f2dea9eeb7e7a7ed2f267dc213b286226ba833257b3e65aed50616e9 -size 21719 +oid sha256:cfd3fcf76544581e18da62ea7325f2a9eed724decd7654650f2ea52bce2f11b5 +size 21603 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index d58da627a0a6..9c5981543ce6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4361d77070f40c97c6881cb30e7182fb38586e0dc8e1a669cb74bae63af15644 -size 18861 +oid sha256:e2878699e5b36416653dd6416144ff3be6eb3d4121e29a42b7fa6703046a5b35 +size 18718 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e1294b5044d2..e248717e4415 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:636df0ab335d0885b874915f1f59427d6b7eec3ad1bcc4f352ef13111a0b0e64 -size 22501 +oid sha256:165af6737338275ccfc6d7fcbe3acaadb69f8d19e16d98a834466c7b33fd679a +size 22446 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5a83c7916bd8..6b37a4696d26 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80cf4b33aab05aa420f5fccf334a8de06e7716785cb347670efb046643ffdb90 -size 19513 +oid sha256:8d4f5ccc14f5c7b4c4cd12c927ca6c8d403e17939c3bc89f65cf2dac08195212 +size 19473 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0f5db55b995b..d8958fe18ff5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6cde20fc7801ef7c64cbfd5889b6f726ff3ef52a29bf3d9dc4efca038e26aa78 -size 22016 +oid sha256:85fec5e02d3dcd56e3d6821bda2a1065c244ce32a89af195aed07e1c48205393 +size 21865 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 12fed65e1ab9..ca71d0d0bea8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:295099aa4685ba69ae3978ee178378554f1caaddf683aebd85b3696c90f770af -size 19389 +oid sha256:31813c16876d883dae0398f2de15785695d7ccaaa7d76890bbdc694c4ed22077 +size 19277 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c7aa4bec0e15..35ccbea144d0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f767afc5682c24b9633492943ee3d20051f2a749a990d78fed15e10650df422a -size 22466 +oid sha256:ab29a7c06a0c5c541ffa33bbc3ff18dad3cf258e28dca7751dc7077d7d9896a4 +size 22303 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1217f78127fb..1a040b0451a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:687d808ccb14fda7b0ee0cc0b4b69cdd6cd143d293dc660112110eeb8b9140e0 -size 19953 +oid sha256:910ae284cbeb8e4648012590d20312d3e2a9d8a3ea5d12be1858a866b6fec792 +size 19837 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 433ac794eb66..d0766c5abb07 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aeeff1a1af7d8fda80ea5cf12e24519890c7f55520f90cf25fca1ef2ee4763c5 -size 23960 +oid sha256:13e061b1167a6d2321a6082df25dbc03be839e4265a60c88bf0d7a488e67ce79 +size 23749 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 7c78f30411f1..f6f2046843d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f590521799d6fa73f3460953dce8a985f383ca15c174b89446dafa5c49d78f16 -size 21087 +oid sha256:abc16c5a381661b265e1da3848730604aebccc8a748dd0b88862ce65f71d69a0 +size 20933 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4095bf1a882b..6e2c13f9d8e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1bbbf7b1bbedf1c0f7948a6220f2dec1879ccb07d933e40e2957cf46404e9cab -size 24459 +oid sha256:b0e5bd557879fbc478912ecd3c2a5036ab505092f80e76b6f9a0bc5f27c9b534 +size 24421 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1105232d7096..5f90c5b11131 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be781e8ba5443fbe4677311707f4bad3797c2c8a2871acaba1f3db9f2a35ec48 -size 21572 +oid sha256:218cd0c9faf0a8d8f8e2b71e948cc6272d0ce9d2cc1ae81380007cae0bed5188 +size 21555 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 14cee31ed18a..e895c3aae46c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb494348f43057737dc03692384d06adf963ec05bf64f682d799dbe58e8f710e -size 29668 +oid sha256:5eef2b4c30337ab43581d445f84fafee2f26ee8fee7149d2c98b3f27e3c6d978 +size 29544 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 2c77669b2098..52bf674c372a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6202b9a5494d1d7932eff7d3753e4b128be63c39c853e1f9cae8bc532b278fed -size 26781 +oid sha256:7bc200e55e51a7dee02c875af3e24021d70f00b694a88243c0eb7cf06cbd7b1c +size 26740 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 93f454199d9c..f7a093eb896a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c93896e7bca8b53a17f29bb51e0ea8d28827aa29ab2a7ee2445cef6ea07d62a9 -size 31326 +oid sha256:306769daecc30346e41ce7b630c2bb9fabbd7b87ee38266e469f6bbd155bf2f4 +size 31253 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 45646c12c281..508923efaa08 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7de77611607b14430812e7385b8daa4c8d026cbc8d134a6cd039322a0d1730db -size 27667 +oid sha256:8ca481003a25d0a20b24728b5052aca7e48cfffe779dc98c9deb2cc9e1a6fc49 +size 27573 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6d061f34dcb3..77b7520eeb2f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec7420292644414b842a8a18b7f7f15eeaeb6ebde79502081652dcceaca82365 -size 27553 +oid sha256:982b0b56e20a14b7acb1f1c9267d81d9a563e2ba1fc454684e443c09f16e7415 +size 27486 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index dc422bc2cdb3..3f537257694d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:11328147e10791df72a7af670a7391fea6247cc8c7b3bfc9e8b61bccb6c3dd5f -size 26064 +oid sha256:1f0398507567e798b47cb48c453236d9424e9a9c7fe8a9c0607fcbf2a263469b +size 26013 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 4032ce17a892..40fd5bbf109c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b840b686c5b46554384a030a4718bc21bb8a26ffed15aec274dceb08c7562756 -size 29486 +oid sha256:3c0332e73beee51a91172e61f566d0e1a3206a42ff6225708d480c43eaef0da9 +size 29416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e579be9aea36..2b57f65469ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0906f4d7e87c7909ee9b6bb18e32d1ae9b9b7e1f95d51f7af845dfc3dbda4981 -size 27077 +oid sha256:00eeb1016e94647c3a9c52d54abdc48b12ea226de514e717aaae1b7c4935fcce +size 27058 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 059d40576969..293826eb6537 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f82cbed3458127047c57bfba098a721ef2fdbc1b6f3d1d990385e263acce049 -size 25761 +oid sha256:adb4ea245a1c2e7f5f3dc70667afdc465ec827874bbb37204a38b8aa0f5ca39b +size 25656 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index d7ac4856b8c6..35179efc05c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3543b6a4ab637910b7f7a50ab02e157d00c9d0833ed3e88511f5e0657e968a4b -size 25600 +oid sha256:5286222f8af52b7d9c3f2b1cf9de1de04e8a8759a5fae529f9989f32fe9b13c5 +size 25495 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 4d718f740f8f..303aa0738f1d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca4736d2c7ab9ae0c48a899675007746bf5adb8d35e241bad114dd7c8ecc12ab -size 22919 +oid sha256:2039189fb1641353b24cc852089af746e40d7568fe7dc569c0ecbba4af3f634f +size 22805 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 91bece52d5ff..15738e482f03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6eeeda49a5d8a451552d6436a58f037ba4ad60552b41ed62b8c8e9d4e07c8107 -size 22411 +oid sha256:ee5c2b4e33e3d007c86dbdae14d1e527f57c256203be51996c8a7cec746485d5 +size 22326 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8a72f31dec2e..37dcb35f7c75 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d6a208d5895866ea918cde096b51843343a010879375ebc85b7343e5e19c701f -size 24223 +oid sha256:752676797d8631922d73c3d1bbb5c12e832f626610ca5f35e7d33f5afff986e6 +size 24247 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 94de22ffb163..040fac9fdfee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:11388c08091e66804902c2840752494c059c751095d7f15a20f38cc641df5dfa -size 20938 +oid sha256:614e2bb450441ba92b316f478ba7500b015d3a6fffc5ee4f561778393cc9a438 +size 20936 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b22b5b18f8c5..e6a0316bbcbf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5833f7509535d7f809c2afb9a931a23728ba43e2d1cc807868795271bcd544a0 -size 26465 +oid sha256:b4fb010f0bc9d1e3e36daec78a3d4a5d0c9e377030de558103bdabfb39985d8b +size 26322 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 1efb610c1576..f1d39c6d1c96 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48fc042b7fa5157a12c8ea8651d7eab3406474ee2f652658cbe2f027e845d9a6 -size 27246 +oid sha256:186a73be869a8297189b4496b5e87fb47015f68f9e46aa2cf5d17b13978f32df +size 27112 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c4b4b5b3d57d..d8f5fc58bb4e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ab89d7d908c1bf4da130cd01ea27c4b8fc5e69c7f7020dc7b18245fd86833772 -size 23658 +oid sha256:10e2063dc9ef41740e58caa0960620a3b08174d60e153ac8be97eade6ba84a94 +size 23656 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6ab20789bd46..db3f03308832 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10fff5706489561cd52c62e3a4cc9cbcdc7e3c524b8ac1b82e3949ccbefa997f -size 24703 +oid sha256:9a2376b729479c299a6decebdcf6758ffa2ea021d4b05059c65d4c85a4e98b2b +size 24590 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 802066fe7bca..f3458b3e444b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b5c1f01452f39b7d15f11a4d98b30ab22c2866ef1263b63a72c4d187ad4f90a -size 25230 +oid sha256:d12561f26db0606548a450f25048f255288a55c9d28db3bff9236663c2a7c78f +size 25146 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 28d394f48330..594b9ed92511 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d19638b4cfd7a51a34a09ccc80c822cb4ad988883d6cb639601a50f5b9051a0c -size 22433 +oid sha256:3cfac0bd6a1f39f10c60c3f796d1017827301552a11f5a8e31c5fc347aa68d22 +size 22410 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index d88d3cd44a52..8def0d7b64cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c318c65f192396c6f503f2e04ae52039739aa3f34d88903e50e1a93cd92b1e2c -size 25609 +oid sha256:ae7dbc3fc1da432a3d2c9084d6b1e09c58a6e67b821176c4214c3910a2b009c0 +size 25535 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 33f202ff383d..bed669c894f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:242b4f1ce46fc1c7395db3a5cb7d05424d3a2c9030bda6247371d0003959e6f5 -size 23273 +oid sha256:0c97ed462911ac3e0e554ba63183b2278fdfe94b6292027217b05a0577006de3 +size 23176 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 298b5415b37c..b13aa96c66e3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0a6ccd374a1a6c6912b1433b17a6f7edfbca85e76edcf495bbf7eb18511b288d -size 26280 +oid sha256:b8ee780eb099833ec4837ba3787ff793f86a8ddb43139bf874f6cef5352b47af +size 26152 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 3867c3d188cb..149e0a455b90 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3f5350ffb4aaf42af11b4b838723bcc25ead66c644b87aaeef3708d1c6067a4b -size 23896 +oid sha256:336cc462c5760da6a9b1eb9781bddd764c9ac00fc6f049265f730099301b193f +size 23753 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e893ea7429ed..76a22afe7458 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b837d3c2b29c42abc018b459bb3a8a38b6d56c7e5cc6979f4d7e86dc8c6231ce -size 32203 +oid sha256:c742919b0eccd49d6af4d8b76df75e191bbaf1155554599aefdc3947606d5679 +size 32006 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 1a9cb927cc0f..8e0beb02c44a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:66fba5fc6ff3f53ed4cfa65d4b3e547afac9cdde636c465cb579fe49f0862d82 -size 28338 +oid sha256:168bd43f4c0ed6596012ab4c362768ace7b65b1d89d10cb652573dcd259eb337 +size 28252 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 2b883ea8da88..d8a49166b6b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6d2a4caac4d346ffd079cc5c410747271c09732c8c06ed60f5212fb1d4f3079b -size 33162 +oid sha256:4536c1f3c6fdfa139f38e210291989ae75faa0cacee72fbb67375339c52bc93a +size 33102 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 43c9161c1944..8587589add4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d9d2d0f1c24b4709f83b6f93a4e8e343771bc695ad7449ec1156d8720e2bf90 -size 29190 +oid sha256:9d41c63856e664cac55c14198f11360f7fea64613db2b32f45929c010e06729c +size 29134 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 1d480a0d1de7..d023dc741716 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6581a70492894e53729f15ce61a57d23581cdc1397ea80b62d8b4e544ba52f72 -size 30226 +oid sha256:1a6972c945df9eb5fc24b205c3760458314e1ec6c6255c2585b53cc6ce1ff949 +size 30059 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 6b62f2430f72..a259d9bbdc5f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:251783869da7cc0100b513e953a3efeb52354b688062264148d151d9ea877736 -size 27690 +oid sha256:3404a87f3c733ee2f606a87c8f55a1ac9236f6df612dfeeead7d5e0984b811c2 +size 27664 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e895804baa81..96b1beb60017 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53df2235d462c667515ea3808bb19048f65c9d6cade1e77e636a01de553e9bfd -size 31316 +oid sha256:441cce1d694fad642fdf545e361e14cdd3077a03b18debd0298c4a4464146bbe +size 31208 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index af8a6299a592..2fde9876d399 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:536a9c624fda17eaa5402b4c101ad7f7cff9aa8a72a6895773f7c96853229f20 -size 28593 +oid sha256:5955e2a1499fb80434e4354385e6eb31e93c42e879e505c6ab59074b5a235c68 +size 28650 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index a3d8ef9f6744..791164c69649 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:60f80d1b3c3aa339e4de0e2157a5686c52d278b8726c69a459a35261ff94900d -size 28252 +oid sha256:acd31ae21b435e120a56494f814ff91ce6cd71db9eb7fca45b89ec2236968559 +size 28241 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2bb45f357ca4..a45de512e84e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:010d155f03db65bd6b9820b08d31a90b0851ad1069ae317bcc83b613f93ea7e1 -size 28035 +oid sha256:5e61f396d07c858f1a2cacd19c3071e429a3fdddc20a22154c37a4aeb7a602a3 +size 28000 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 5784bef5f9fe..9ed2f1da2c68 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87183c00394dc3b9817a5be22bfa9a212f5144f69b683cf423e0e7abcd4051e0 -size 25422 +oid sha256:1020abf10844c6a74ed1938d7291b577b78db5070f3d13a953c9e9c9d1ca0976 +size 25207 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b196fea45401..27733f112b60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ba4dfac3317bb3535e1bd77906d38cc671ac5335eed62a9f88ed9a67dfe3c55 -size 24579 +oid sha256:ed4b69c2e6e6c8d957f723d074312baf686c80a8f4535495853f553b570a8447 +size 24548 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index b77833631e11..416adbc40618 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ddde50c6134c8fbb5c183fa8594af4fc392cc5a28538549cc8047df7516b1c1b -size 25532 +oid sha256:98ca1f4396efab5a21e0ff68a70895dde98335e13d438a4b1c951ce9699316f3 +size 25499 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 3d7a01df7e5d..88498d4d447a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37691254e50bf9ebe0aadcf9cf4ccf6029fbc5b286b7a4bb363559fcbb318253 -size 22327 +oid sha256:b0bdbaf52e10490655e895fe62a51615763eadef1cc2943e9f14e1b959fcad26 +size 22349 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 18654684dd01..da040b4e072c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fe7d20e7c24e2c5f3c362e1c1390a9563df9bd20cb122b58a80c21a138b4d6c6 -size 28952 +oid sha256:04d42f43ed11c819161d1183f21f0577527ca86577df5434e1d5a7b56ee10647 +size 28776 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index b1fb3ddd1054..c719182166d0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:388b291da62f43a626b66a4358f7274c9a9bd46d1e28a3160beedffce50ecfed -size 28779 +oid sha256:5cd680468ff3533d5d307ba062e5fd6cecce711f4d3d28f085a70a7a4742b636 +size 28617 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2a51bac5394f..3eeda022ab01 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:01144ebac82ddf46694e8ba6b7b55dc0e44a231ed91d9e840f9c7241e5f9bc86 -size 26018 +oid sha256:35f7d6d2ba17858d2422cb55c6b744460a36a1566f885dde2bff2f3c38d5a18b +size 26014 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e2d1785021ba..189bc6366e6d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9e506450f879831d44ffa75a0bc8a3dfb7e3ae960d8b0f972a2aa45491ed0d87 -size 26227 +oid sha256:861c9875dcd99aecfd17ec6558f197849fa1f9be287c6e4d146a6c282b6c84d3 +size 26046 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2b14ecc81b9a..a133ae5cdf86 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a069ad1ba406330fb19316b7f4631d3ba4fb02a78df0702cbf26820d02f1ed49 -size 26232 +oid sha256:0234c2cb8935446e799a39805522abe375796896dfdbb6a3356df329fc1dbc54 +size 26115 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index b4f45084a448..fcfbed26a661 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ffc816df3d4aa2cb6fa571cd5d8f70634bee60fba9356e99e33b37f35e6b4041 -size 23620 +oid sha256:f1d6a0f317da94ef2c9f4b0a29fcc85250fbaf45863520d6746698b917aab305 +size 23626 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 19f1d007222d..8ec333790bdc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e05dec666ccb2e9c4a31c0430ed70d5a3afd7dc17a559c001bfd192f4cc517eb -size 22869 +oid sha256:4368d37dbed4fe29d44010532954a209048e282682ed201bcf6439202eb77ab0 +size 22751 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 0fdf1ff8ee59..22373e85b808 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb1b64c7ce7fbe75061f90401770f2dcccfa2608ae1f2f0faa12e26f740405da -size 20353 +oid sha256:ed67ada448732af156098f17319fdb6436a7c8ac3fc4323de9a8ae4a646965cd +size 20239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 59e50ea7497d..d0cf5394c95b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:624ca5001824cc4191b051a854aa973ace4f622b0a4cbfc730ad8779e6586eff -size 23540 +oid sha256:6befffed0a33eedc362e838ee86200ba4f572b6a3caebe6b26598b0ea092d5eb +size 23537 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c49decf7497a..f422d7663091 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:583ef3eb70adcc542950152cb6ca7ed0b12b576bcdad9152cc2f5d73b9130b30 -size 21075 +oid sha256:6b9356bd4bab76d2af6290f211d68e9d0cd1b403957d7eef2c907f55dca85e41 +size 21086 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 968185e47a89..3ff315e51e11 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3597317ee902bd9b381c61a1ff9c59706c1909671bde3988bedac5474f52c841 -size 22821 +oid sha256:db686e082a65b3cbc12210ff831b23fbe7985563d2a02870f636fd8a1460dd3a +size 22681 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 2ceda4eee6a2..93a69ea2cdc9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af1c3bde1dc5ba9e537a9e98a08c638cb4bc2803a26e2398e71fb503236a31b9 -size 20721 +oid sha256:b0e146d5318b946c659c8f2d639ee0ac52b4323f2ca77e2d8174daf6e99b84bf +size 20593 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9688a5abbb0c..8f5ed9fe6eea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9fb49307ca9ed9bfd9e58e52a62d72ede4ca46b396d5e4b13f022fecf6609da3 -size 23659 +oid sha256:e2f47e0487cf5d031fd442f90a51e98e252e763389513b1730979d63ff41af1c +size 23480 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d3c7b5772d4b..befa17485f4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d222ca20b8eb0c7671b0bc862c099b12068762e04aba6245041478bb9d6e4226 -size 21235 +oid sha256:5bdc3e2dab059ba0c44abe129e25ee587910a4ebbf8c900be465a54e94231f0a +size 21140 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6e46e29f1f0b..dfe3127889a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e4507368004888aab97591cde8983a0a70f0d5da68c104943ec79018f6a309da -size 26403 +oid sha256:cc89114a3a8039a3189ffb5d30164974fd8747b024006a9131e7dced28536aa8 +size 26399 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 51bde2d9932f..69f65f3e2326 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:708d11a105c0d8fd91effdd029282a2f016e64b3910c45c84034ebfbeb0e92be -size 28501 +oid sha256:60308e17ea2243efd6a474527f76f1aa40fc58b6123670458a425498af39d0ac +size 28445 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index da8c2215fc6f..e517607d9bae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bbb3dd889bf73523967ab4f952698319595be8f81873d03f0960ab57d0247ed2 -size 28454 +oid sha256:0bd13b35fd6494625f7f016aa58d28286a75c19e961803545fa9537b3654726b +size 28465 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 172971799859..1b2f65732d51 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee89d2dea903d8082260728179b5e74958f9a472c9683882dd4093446a208b18 -size 29018 +oid sha256:ed1a9f2d1543e2c3ccd78f922c047bf2d5a05c6effa3d4e09f197a66d81270db +size 28875 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0563082a0447..b47e06bfbdb4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb95be58ae84da4e839fe34a3356ab83f7fd1ec4dcf48bd87beb7db177e5ea49 -size 27655 +oid sha256:63e5961eaa1035978dbc8671983a9a9799d095a8f44bbe4056a557adf8676614 +size 27568 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c0a6415f00cf..78c5de5adb99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1122f18699e004eec36ad13f3bc245fa2492a68c51e66d43fcae49c5385f4a6a -size 27718 +oid sha256:613906dc5c25187278126be2245e5183d4e80fc990781268abf8985cab85425b +size 27613 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 2b2018800fe9..6bdee640ddad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:674dbbe0f88b6d5ded86a0ee855fcb7587a209c15400938d5dfbdc739a613f74 -size 24942 +oid sha256:f3ec3afedd712c0312e581b9ff768370cf8cc3263e350a19b260f7626a4c6533 +size 24754 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index e7bf5415ab74..f0295da08c08 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a464149d26b86ac3e0c20226e6a76f9c71cbb370d0198594428ad0c2208c07e3 -size 24361 +oid sha256:ea384436192254845700cf000b839d483318c962c1c18ccf66cf6747f1d5e04d +size 24303 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index abc54ea84d91..460a4ee78525 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee1ee334f07cf20f679e71773b4bb9ebbfc1e4d9a43471894806018bb8c7cdc5 -size 27270 +oid sha256:edb4e69be8b8b363a6f7d4c2f90701fcd1036d9ecd3998b01234660e9075a803 +size 27280 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 43f88352f7c7..8fe9b144dced 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9cf0bf9fc4394ee7842ad39912cb7af25769033c4665d930b399068f2dd92a6f -size 24893 +oid sha256:d48ac16f925f154504d84dc2f2497f2df655d8912340293120127d78c92bfeeb +size 24824 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ff6409d87c4c..270ee2293f35 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8dcd13639f4e380af28dd7922640851823c6b12a60c167150c981d6ec24458b9 -size 28476 +oid sha256:e1ce255221ea20ff12cc04d7be97ce881ceb71115ba580b21a84ae731b96c7a9 +size 28368 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index fb16774f4d7a..5b9446aa66d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bf308a8665c44b09a4b7409f97f5f07691e94d958fb68d074a5addf80da9ef80 -size 29209 +oid sha256:0c881534665d1868f90fe9b76eaa862d1f6b981e3bea764762185284d4a40868 +size 29053 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 8d8746e94817..994ef8441e30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:66a7f77cfcc3acdff224db97ec5ba2ffc1a5663b647ba5544f0c4ed2f3e4d377 -size 25867 +oid sha256:c3f62583d2c1071df25665b8b50157d63752f5b33873a254055885d9d5278d21 +size 25800 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f530d38cc4b6..19e1cc481bf1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f37c5c3a10f2044d409fff98217373f2ed078887f99f05f26ab2a47953afe22e -size 26749 +oid sha256:3006b8d04bca5c97b673755cc31278f1f6acf6f26a42dbda8cd1ed83994afc8b +size 26653 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 02a41587bd10..e78b04ba29c3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:70c2f412eb36caf6b19ce0fad226cc46b4f4108a765e0ac58da061aef5cc4e51 -size 28352 +oid sha256:84561bbb16f579089cf10fbe5eccc809329cd7b0ca5acf766b79ab53260d3f2e +size 28281 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index e613519a6eda..f3257e17abc9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ec5815a4834382748ec96abd56ca0f3f50d3f6e55f33b59782035f5768290a7 -size 25049 +oid sha256:c44616e1cc609aa8dd389d6207e704d3635002e235b919cb40001ec8799f1957 +size 24970 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 92d1216bdabf..58c535f338f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc04f43ec57e3f9cc9f7cdec2d454954183e9f33dc673304a033a97ec48198fc -size 27578 +oid sha256:50cfdb063d7d67fe3f71b2750b22ffca373e248ac6f021f9cf0bdcdd7586f325 +size 27432 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index a95fa40bc738..4b37b7d6dd00 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8fba8b9572487c1cc5012e12d21747de7bad49dcfbd22310923697668b329316 -size 25128 +oid sha256:22092cfcf0e9ce7aac303bce141d3ad45a6140ae2a54815c93dcc6a242a58e59 +size 24922 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9f49aef3f9cf..77bac69b0ac8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:acdab2c79b4f9a7a1df1e1d8b4c39ba33c313c2addca96b8d49ffdfd2dd5e2b6 -size 28310 +oid sha256:e26bc3a71a22a60a2d3f5c1dd4ee88c0e4a5abb0bf0113061a45e49f6c7cfeee +size 28211 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ccc69d484021..3364ad926014 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ba211fff8be06ea4b5278c48d62ae8f5e6a223aaca92a63d8b371ae9afd3ba0 -size 26106 +oid sha256:72bfcc9f8772ed71a7707b9ab96819f07fabf2be52dc572abf07a19ba39a29af +size 25942 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 6e0e8d74c537..edbb09235704 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:067b48c68481bc6a011ef21fa5c37bd651a28dbb1ec52a135583f0c607a0f309 -size 28944 +oid sha256:07f42a97301d04541fc9bf128b7dee137f7afaa39ac1b8307912cfb458805190 +size 28798 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 558d279d7ae3..44675ebc5a66 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6a9a32e1f994394da30f06626433c9c88974c9d61b2424a1c3b75791f9d6c4eb -size 29807 +oid sha256:4e7401ea6d05f1c84e3b2f116b05218e0fce3e01ae32006550bbf4b51f816cb1 +size 29796 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index eb6512d943d3..eebc009a84be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b436a7c512215816169f4e04ff9ca3e0b61f66729745116e6afa232a98cd637 -size 30105 +oid sha256:9abdecadda22300819e70217e4d059751c9969883f0c1e84632bb9abca8f2bac +size 30084 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 288c68d7f2bf..8d51e6e85fa4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:352d699e8585eca9e3c089b8559e890f46979be04399179464bbc24b6c8f6a7b -size 30307 +oid sha256:55ec15fb6636bc342d20381012806f27227b67c9c49515f248e0a2b9fc3308ed +size 30142 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 49528a1642f7..bbd8d3a04699 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:64fddde64e8f64cb5f47207eb30c64ad91c2f22297246abb659a0694adce50ad -size 29857 +oid sha256:121336fe3c74a3715f061bcc2c2a016f2ad4b02ce4bca1f7a35a6ea1921d197a +size 29747 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c31351ace7e3..04c9d95c2969 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d33ee2e9e2d3ccdc9d2f981db3d0163de10df6f5e59e1773196c6448d542c5b3 -size 26646 +oid sha256:549c9fe9e2b5bc57260b216c15355646bb1cc99378ab5bf7305d41276392e3a0 +size 26538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 2e614826ced5..3777624373d0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d7c3e10d23ed67f3787b7b4e74a3e91d91239c7f16195c9fed690547bef24fc3 -size 28371 +oid sha256:401e26a4214170e33e39f1c0c9dd48757edbdfc2da4807131038ea83475c6408 +size 28337 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 852edf34b0da..f73def55da4c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd27b5db60bac179cc081f25bffefa05b8fccdaf9babbfa4e274d6362b3e73e8 -size 26224 +oid sha256:448528f0745a35099942183034eda075c271a48de8b1466b43f499af93e55997 +size 26189 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index f7dbe8dfdf6a..a1ff50df5ecd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b792ed81049d1fc1bea5e9030fe44ea94191a9db283fe7dc8cf080f417a18071 -size 30614 +oid sha256:da49d1a099fd2f22a81bfa33fccb516345170a2f60556978a2b1b02ab1b96961 +size 30486 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d105b26e6f97..ca7c1f8ea72f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1fcf072557701cfae1b0a6cd6537903701500dad223311ca5d7967914f7ab9f7 -size 28061 +oid sha256:efe248e7e183995e97fa36009276a5ca07a4c1b7a520df601c07681a8d6d0299 +size 27977 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 065b11c76a7a..b4a878f53e6b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6c59231422d532fafbbcb0e16a473f64bb033cd12b01ce92cb4098bb30e6a9d7 -size 28973 +oid sha256:a97a30e73469e3871c83e33fa273fc55d80359a44886df0d273829b86c78f4ce +size 28888 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index eacd6ce6693f..1290087955df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:acfd5784f8f572f6dc966cd4eee2d55b43a38ebaccbbf9658ce61c16ac1d38ae -size 26353 +oid sha256:c2307da8ef487e1967040ec8d69246ab31bd0d056fbabe96250ed20feaf3a639 +size 26321 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 75612a399a3a..de09076fba96 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d9fe6bb771c01a2ba92ecbe2129e9a569e446af6dca308535f449deba4dc1b9a -size 22864 +oid sha256:ae7cbb44b4ae4bdd4e5bf21cd48be400c231d5468ee9b8bad618add7fc111813 +size 22825 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 4aad61b3908f..307da0ab16b5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9c06c16255479974e88b51a1c60fe243c0a34afc5c66af9494c363a4bfdfabc -size 20032 +oid sha256:040ea59d4d38d7982c6c0ab86443ef85c9678a265c0fc49592e0f69039b8f3cc +size 20004 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7b882829d936..0b361c876f82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9f48b0ece16309d2ed3b4bc224272695f1c01d6c4624a19615e48ff399f45556 -size 24129 +oid sha256:9c6b06a80eee5eba089beff069e90b74d4168f592f0647cb05fe82bea4e2a7dc +size 24088 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0d372a00d405..de2ca2789d7f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fcf97eac14a956ae88dffdbd001bfc3fd20965a08d827d9b8f3228a92616e293 -size 21399 +oid sha256:eaf70c968bc703d1c728ac2d6d5f53cccd9ad4d583d42b23d237b7a3fc0a7364 +size 21327 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1c3ca246ccc9..9b07549fcdd7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dfe14d34f05fab9136684d73f9bd07a303b6494fb12a6755c52d207501eec695 -size 22784 +oid sha256:9fc7500871ff3fb0baa79e8eff36dd86ac5d13025d05e7a8dc43db0060ce2f05 +size 22767 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 5e35c9e7df22..b17fe08f5108 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bdf91a0e7fbda36eecbac7ea5b8df56b9abf424a942e5377dc3dfbbc0f6afd0f -size 19752 +oid sha256:532c24c39391341becb459b1969494b60f9606f9c75c495e1e130bf34ee7a6b9 +size 19707 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 99d4d9d1be48..4fbd9c607db4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1e78034ad3515ffc30c92a8cc9fd75d09533a39683b6d8994773432a962329ad -size 24166 +oid sha256:389feac2834abf4b85db41bfe14f07d07f24735a6b90d94b5d776351a526be13 +size 24125 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d50b9c73c8c7..32c387b2ca98 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28e018dc2e841511d06c6ca3d03cfa82b1f07bf1f53addfec4ceba194c950392 -size 21047 +oid sha256:39b626be3f5920e978c63a8727f065b591edb69ae0fa532814cfd8da8fc6877f +size 20996 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3b3306aace63..efd202fde5a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:edc5faa35886b8e3b76b5711ff29c6b210327dd5dc086b407b6921989994e62f -size 24961 +oid sha256:8d9474fe0e4e55d83761888ae64725f7551a45380294fbcb69b4f877bd3a072e +size 24947 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 366c65f71124..433a2e02a5f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:25061f86e19d3d44896128dcf7ff0d2654658416972708f2fca39460d2185194 -size 21964 +oid sha256:d360b76fdbe6c5f662844a745f99ea3d5b0d40c0e1b85b004c75e01590b8d43a +size 21935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0a94fe35b0e6..a6e9cc7c0040 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88c6dff3fc76ec7485e1a33349f95fd381672e25f17b4d78403cbb99879a8685 -size 26197 +oid sha256:5e9aecc9fc3b34266eb95f772969a1e66a52ea22e0a43233a0e8b70e53ce41db +size 26143 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b2b3de65d961..bec488c1329a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9e5b8119db996d63e8e3afa96d0fcd238ce5c29902f071d09273fd01a63fb37 -size 23636 +oid sha256:1afdee5f498340ab9c8ee70f2459e4401243542b445cf28973d0dbd5949e1aa7 +size 23575 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d53b450f87f2..e249f5136536 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:23260b8d4c383e91d784742af59d5cbea613fa337b705953f733c16299fe91cf -size 28623 +oid sha256:21f6f2a88c715b0cc8229974d2590ba0d6b8ce5de8297afb2990b1d1a42a40f4 +size 28516 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5d7ee45bf633..830bec930288 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53c818496dbe86a41448d50bf5251789514b7aef775b28ce4e7979961747ac33 -size 26986 +oid sha256:e94ad911e027cb1064a035d3eba1ae54deb24886f06c7c96927d2963422a3cb3 +size 26911 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a65920fc7e8c..ed1b78bacafc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f6cd068f0b56da3d837fdf2b48c044227cc18f7ef7ebe024b15e26366a1719b -size 30348 +oid sha256:f807f70b029708015169b9c062c8e32430cde118bce8ee871b307e6f5b46f9f2 +size 30202 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 4b1cdffbe3e6..c4f9ff5a2b25 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa356387c7529d854baeb09411d14258c8798457d997586ffd331a137dc27de9 -size 27812 +oid sha256:0711ac678acbeb0161c6ed85f2674279684e9d3eb2aa937ed64c1d38316d7c98 +size 27700 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0d3c510693c1..2b93a39b7788 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9beaea29ca8ec48ef84f2d85daa3cb9ca1aa1e5e9264ea900e471e6e6b8d380 -size 27492 +oid sha256:ad7247c55e8250af8b40b545ae8b8d0beb09d2de35d5e09a0b3043c03ba3be56 +size 27411 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 404422f62906..46970a6d2e21 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f2da8ca21e654d50fd480ce1cb7e5e9f027a9c3cdf63347a763bf97e64df7e1 -size 25582 +oid sha256:6385b339efef325e26974f11116cb7e77f980b1f0eda5e90d23f01b4905716e8 +size 25439 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c750225f5eab..746e5803a596 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ed38cdab5bb8a08326167dd39206a42f47f26c6d1610cbd47169d490bd9507a -size 29334 +oid sha256:bee500e5d1b50f095980d02833875ac828fb5e9a28ceefb5a14a1050ce91e50c +size 29101 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 704699c32177..3203aaba0596 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f99a6d295048aa40e3410f475da7c114a5fad93c7c2eac1c0fb23d80ad94b5d -size 26332 +oid sha256:bbea6e465ac2e175ce43e746e4bb56a215eea06b7ee7604346be5c909baec4c4 +size 26253 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3000c53d519b..a9236b78215c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:780fb8cd2133ec11e664c143e5b35956d34c5657cdb5ab5b87f7ffee93cd7a4b -size 26252 +oid sha256:ee289ed6e37dd64463b371efe127c24b6f8066fe69030eb935d3de1c2b9c0d7d +size 26216 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c637385937fd..0151f11774d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55318c1ab85c67892ff4f3be7e1c1d4ed80ad4a06fcd13bb0e529591d175b74f -size 26092 +oid sha256:cbc41ad42b9d83aecfcbb8b2b357a8a87683482c886fe79b1c050ab9f20b67ee +size 26037 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 5e42574be673..cc5e2f0734b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88136d97afba138123b906de7b43e746f64e927d9bf76a48e7fb9df01e1d9c96 -size 23704 +oid sha256:e27586f883f1140aa22e57d15709ce3df8f6c611ac74a6ade7b8a3cc77fea7b6 +size 23686 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 475988e87ef7..33a20741b57d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55e43dc2115a13309d89d4c06e9806122e0226e7262a92b105299cf170daa0d5 -size 23212 +oid sha256:6d34781ccd4c93ea67ed429b690ad49d6563cf4a8eee9f2c90fc1754705bc4f8 +size 23187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 56d37233221b..90dc4952cc73 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:000bc304dc7d2b8bd51594bf981792283dd2e7a8dabeb89dd692fd1f8d576716 -size 22703 +oid sha256:d1f6e1952a6819e8df3f584847f35044bccaf947087a3d4d940cdce5bc0c12ff +size 22604 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 643a4e852e03..d5e431b13bdf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:52ffc371b5286c95324e8aecea0d0b7e0d35ff43c9e79b1e76543d92d01a25d7 -size 20158 +oid sha256:bda8cc4d17e1971b411beaa9e3ce7122e2a7690b32e70473076009bd1b22033a +size 20187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 899fb3f0f23a..587b7bf485ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50b6deabe8db6781276f94594677f3cd6c1ebb81ae925e944e6fc20d9a58650e -size 27726 +oid sha256:11a665d7925173d6954de41480b5b76324ff1c4dbf6d3b051ded3504fa83d0a4 +size 27663 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 9c4773fdf70c..e19c8f097184 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f12da090053270093ae48c5cd32f2229c6467a73cd9f90c1f27a7ad76d7cdc04 -size 27645 +oid sha256:5661528fdd9ba0f382731411cb06d648d84cad40bea3217de3c17be5a5801bbb +size 27509 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 96a48edb6bfb..1f4e7a1a65c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8d77790465cffd1884cf39680b58e7fc00a9e3a0f1fcddac01efb1f9e0c3d752 -size 25035 +oid sha256:35b08c791bc6ac830087626cae2d78ed45370ffe93efe878a1ba08e91396bab4 +size 24941 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 99ae40139da4..4da8a8a34120 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ac87b365e4199c74232844ea58525522ebd8caa30a1a43a46149a8fc8eeeeff -size 24876 +oid sha256:94f16370380657db8c6aa9d92aac880c68e9040587fbc40d2e355e3bd04b3a09 +size 24860 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 0336d4a06782..37670bc6fba9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f44baf166f2cf60fca36a04d984e38aa5ce280ee9420b7fb5c8f67fa8a9bd22f -size 23625 +oid sha256:80dab8af79da4eb02dced28e1d0357f75bfb4257b80a2d189ac5f68a9b7d9218 +size 23502 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 04a993bb1ff8..eda262370521 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a8698f391447f15a87deac3866fcd9e78def603002a1185d5135e5de705dfb9 -size 20968 +oid sha256:810eb3e5db51b8c9c0cd1759e81298cdc0b8212c279ade090a4e000d1c75824b +size 21004 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3b51738ea3d4..18ad26867701 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c69a7f30be724dbfd0e4f23dffcbe6adf2e1861216e8bedf1cbcf37e1a0b67f8 -size 28122 +oid sha256:e97899e9dba36bd36591a2ab50eb6a4941b912ce49e272fd157de510d54bca76 +size 28054 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 8d59e6936b47..048da480f940 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e24bef8e108ce601acb4f39280c7257213bfbb3649799cd2eb210bc745c3227 -size 30195 +oid sha256:a3158659a51a946e40ce9575469e799183b99edb3937bd0651bec040065040b8 +size 30065 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0025270ac3c7..bdb0c46d4499 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ba11c70579c3bb33e6bc1666103582ef1faedbec5cd6747af8518fe07737f0e -size 27278 +oid sha256:b4c40903fa634d084951b645556b09912b7a243d99fdc480b5c6687bc7274f78 +size 27119 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 214c436c37a0..1d4417276c95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:20ec5d0bb856fb29f23f709505601b8b2fc14e451d071d0cf021decc0e0c44b4 -size 29370 +oid sha256:34bd1a4fdcd0c2fafc86b2f982b5c5a34333eb1bb6dd0b39ee2fb151cdc912db +size 29231 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 8cbe2dafeb6f..f6577213e401 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d58ba089d681a12d0d306c4e7103871162d9bc5fb0bfcc09d0fe31c27f48b2ef -size 25956 +oid sha256:c922ea3fa0ec5ae0a6787ee4d66e09fcbca2b7a70014d6d871bbcdf5fe8e22a0 +size 25900 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index c269e9647258..14625599bd48 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f769d4fc89fe01b94d559f2515c6af4a4924e61929a387d0e12ebbe3d140b029 -size 22687 +oid sha256:beb1dac1d7720ff68f233ac9f0286b6b11b2ea6f64dcedeae5d7554070cceea8 +size 22650 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index fd95f1580b2b..81f5b1aebfe9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3c612ce550b247ad697ee68c9d2d2ef14a6398f6ce27fb0c0fb19f56f7c90af1 -size 27582 +oid sha256:e1ab619955443123dc7b532de11690fb38bc5dba011f00fb3261563bacc704a1 +size 27507 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index d84abedcd439..a05bd3893b03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bbf53ab913da451e816b8fbb63f24ae72b39406611ec5390d99cebe852b9818e -size 24483 +oid sha256:9c5f9cbb3276f681b3a0cc5c55d12849c2c2b73317a7d2061dc59f13fab9f45c +size 24443 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index d297070e9e60..b0aa93f11f53 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a585cd189efe92e35249d4fc1d5eef8f89ca1112d3fef7a60e86903c977d75b4 -size 26118 +oid sha256:c0842c884bfedca46be121e6bf5c26e73074447cc801745d0d55c9ac7e45072e +size 26079 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 9050f9cb5f50..a98430cd7548 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85c97b52b84e5e12755c631e81eac21f8a64a66260c957dd11053f120c93e632 -size 23425 +oid sha256:33fe75be2cb32c1ce508ae027d03af404ed5267991ec4db7ffe26989ad8fa282 +size 23456 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 939586133b0b..fb05d3de8ece 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b5deae2b5928bb24fb8283622dccc8220e5c31c96a493593567a5326141d36c0 -size 27589 +oid sha256:335bb6888929e5b374f19579fa03e99d612f3d7e3ade39b1072c7fdc98719962 +size 27539 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b0b9c5260c37..e14928ba395f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a78d9b8b8c45dfcbacca5a53fce871f2096d4c8eb171fe29c135e74a90592b31 -size 24840 +oid sha256:1b15829991b76162bd4bbeb62eff3633942f3cfc2c68bc13d8f9da552f76d8ef +size 24760 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f72fccbac4dd..934adf5dc90d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:de7543ea35592d4c8896b6ff9bfa709ae3034a14cf6eaa1bf992f5fb4c06f0c1 -size 31295 +oid sha256:5c4fe923d23ebcd03342d825fdc2a661ae1ca570d86d0222d2fc5e22c4a86d9e +size 31137 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index afaf613b5a28..cdaf0873d0ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc97d732795e58c1db809d3605541e92fbe5f30e06f8ae42237ce2d59016e497 -size 27796 +oid sha256:fb88978b01dc80d5906f5c1dc27950fa0c1f6042e76bee4bcd916337c49b6a33 +size 27721 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 852141df4509..23babccf8c9a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b5050e49f1698d42c8ddfa637cfb60b08c0731c1061f1ddb4f9981cf5bef9291 -size 32940 +oid sha256:f9d3b0e2a2cd0613bd3b02eb7731f1b04f1bdd0ea42fb6f221fa0753ae7d57c8 +size 32807 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 16fd0f813b98..833c03721aff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2cdacf5e9da3b3829e3b795a8269469ee90c49ee5dbbecb8dfd7239f8e13b23d -size 28511 +oid sha256:8c1eebe53447ad2ebf533ffed58ea0a48acc02a57137def85fca448a02b86fb3 +size 28465 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index f5a0bf9e747d..f0d618cfbb37 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:71921c9a0b4468b13ef0ec3b2c9e31c29d8c10ec300fb1cde8f528d5fa9013dc -size 30221 +oid sha256:4581deb0ef05152f751d019ec965a99746bebc4eef2d6fd4e0fb54da44ecef11 +size 30158 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index a900f7114b68..bd61a2292c69 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:acb3aa14da643c560d95f62f6e14c5190ef84ca1e67f314e8e9f121892bd4997 -size 26675 +oid sha256:8136ec39f10efa7730729a5586ed3bb95f8fb77e9c53d800a64a0b5ae074a177 +size 26614 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 3af5a903665c..0fbc1b8195b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ded1c8cbe88dbe785f313bc2d15599648452f5998e0120d58b77459655c1f7e -size 31989 +oid sha256:e96e76baddc74b33bd9c91d3441560b7b2911ed2beb6a41d16cffa8ad8984d6c +size 31757 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 5c288f7b35a0..83e7818057be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9dc54208f10960920f568878eb59f1f22f2a30c42362d0bccf9d8a10d5729c0b -size 27333 +oid sha256:3e5a40422613bf663ad487abb02dc5cf05d882e20456990e17aa831581459101 +size 27242 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index a3245be1c321..2a103bd4c0c0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:122f7352ca88efd1c175fdb3d6f9dea590ddbf87f93688fcec1bca836cc6c0e6 -size 28354 +oid sha256:a35bb5ba7c319a959a7f93433977b45f5d8c8a0d0566b608e143557b1bb91c43 +size 28332 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 5eadaaba119a..127538652c52 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3a19d171e309cc8836219b27cc1521448b8f25927f6f2929dd21984da812443e -size 27260 +oid sha256:78a28f6649408a1bd9b8848508613c2152d55968b3996a5bf3afd689bfab591d +size 27290 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 75dfd2725785..15dfc353e35f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9d1a3b92a60e329f8cdf89b697f4420a9c2dae4512ab8b37993f99e9870af72 -size 25549 +oid sha256:d88a6df54bab709f6d9403585d4d5916075dfa60d92cf2b22b363811bc7a69ed +size 25650 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index ced66188a7f9..e3f5aa869686 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e6fdf115d4e5ffb315ef7f4756fda2082e9b88092fdb9f01c658616dd07dbb69 -size 25625 +oid sha256:b389256b42e631ca0cb37e365cd1909d9ea0862118dd189ba59029c9e39fb95b +size 25492 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index 5bbee6e758da..d45fe861c587 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0081a17573a2d36e1f0413c31cfc03b05b8e3f3f1777702c3f3c549228614270 -size 24124 +oid sha256:f35553547a9bd9b3d3b4b53555e7acd7125948d2499152fbf839bc840ec1e0a7 +size 24035 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index 272a2421bbda..f1e39567be98 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:397a145c2fa8a74013476a1104673b2cf6e6a9bb1f55d9e2d8f080f4d3c4673b -size 21254 +oid sha256:a12abd25db195d5a4f38d00885848e3ee6f04b2f305d668b7963290ee3c42f72 +size 21341 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 2d2bd7df6408..f85a68b74700 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:02769da1c8c3dee7163b6a1326f97b76d5df088aee88819d00e35bd94abdd414 -size 29611 +oid sha256:3e126a4320778c2342478408ebaae3eb0aeb8c15f163f563487f9f1198ecb8cc +size 29571 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index de125645f4a9..5038a90aea57 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2869b1f0fe8c8e2531f4eb3e9e0b3a49de644917ef4db00e744f7150152b22bd -size 28906 +oid sha256:ea765441b9ed2a60f4811206dac1f4ce01f4282e7a1d63d86ba40c7d6bbd16e1 +size 28909 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index cb28d1011deb..ad5ef2db5183 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dbb767c78b5bac4af5a6b1da9b9b8719dfd6c66f6d7d1b724f5c5eca74b3d8df -size 27324 +oid sha256:389705547680887212e9606bde399521833c27188ac47dfe7b69b7ad3f75b934 +size 27245 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a884c57fafcd..92b92ddf95dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1e9ef85c8e9f6d05d9604df32c53aad65719db9f13e2f2c814617c76100e2f69 -size 27081 +oid sha256:df3e4f6c7e9c0df09342cd09e5e3191bc0f7b195837a2de668ee7fae0cc9a078 +size 27113 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst index e6a6f3b2ee0f..1afe89d9ed14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:41503f6dbf83cc3ceb3bf4d739140456235cffd805738dc90ed40ff37ad79b51 -size 25017 +oid sha256:212d521f9ac53210f6ad61cb68113d01f9689f0e521964e3abccdb9800c66f8d +size 24856 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst index d6eeed5011e8..c780e1991154 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87617d4e03c3776e5541a512802965a0a8a364d02a6703d6d460b3647e5061df -size 22619 +oid sha256:db4da86e206503ac9807a69d9f1e7bf73be30f10de933c6b32436bbf9b780def +size 22605 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 12028df65e89..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d476f5b26aca72a2d14bba499310c07852c4a5f653a3f670bf9b7e1ef853cb1a -size 35326 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 39cb5a21bc55..98fc85909125 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1dace0054ce1a13c8187ecaff31d21cfa8615a205151077afedeaab11604ba6f -size 23570 +oid sha256:bd127790e1d081197c1f40b1ee06e7360fb083bcef782a9e8c3870d028e73802 +size 23482 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index b7e0b8f660ba..2f37bf70d6aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e1032681c755fa0b66c2a3dfc6c2857c29c0a6bbe3ecde4ddf96a98bd3c845fc -size 20596 +oid sha256:6d5a3b97271ae3520176d955c70d2a292caec24851a3392b5844460eee35081c +size 20589 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 322a705b93a2..8cb00c0afc41 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ebb3a8d20b32cf087e7c7d4f185dadaea382e45fb8e2e331737912d81f48c54e -size 24742 +oid sha256:e20d2febe61e21aec6deb5b834faa9cfbf15b951243883a9e829da8c0b80aa8c +size 24695 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1630e04335b8..d8fe0b4e9acc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce8164e043c5e6fa6ecae2364251b60e7fa38eef1f72233f6420cd8edc2ba45a -size 21987 +oid sha256:e377659024b2252ec69ffbbcfd552a26abbe95d1d76d062fc88dfcb76e0e20d3 +size 21938 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index c7d8937ef215..8e93ec739732 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:89c48bc0f899c073258e47d3c23a7dde2f76f136cdbcf0bc7a9d9b09ccfbdcae -size 23513 +oid sha256:9eebfadfbb49ba66a1f0df2d6009df528025145a1fd315fa8e4d1b5b7a14e57e +size 23449 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 95b2af52576d..03d230930b92 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:40b8c6e4746f1ed8ce179bd3050a50d3a16025d7c56e75f6a528a8ab4762d4c8 -size 20419 +oid sha256:c9745fd18461bc504db8dd1523b3937296472c9fcb2244fd195b5643ec6080db +size 20421 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 02da7ff3e814..2ebbde20c09d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:665ffc29554f324c3335c52fa623e68eabb7eb61f72b71d39f9d37c7e9563291 -size 24722 +oid sha256:f9528229f293f592da0ba37e2c909496c9aa3cfc9aa540a1790b78ab6ce9c2ff +size 24657 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 8f6d3231f36b..8988ea54b88d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:15e43e30ca079ba81f3913cedeb12e004a11658c17a1735cf96f0fc0cb3cacc0 -size 21797 +oid sha256:8578e9ebc39974aab09db3ed804f4de19677d6b86e7cadf5b8ff9a7391b7f892 +size 21769 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5d3bf0fae14d..4bd7e0ae7631 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db435c72dee50559ed03011227b76f6a82e2d29990f27f4d54f9b2316ada852e -size 25786 +oid sha256:58272ec7e45fdbc40b093d7d23d2e172f20715dc689b8fc17c039d3ee68bff51 +size 25654 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index b6861a6d6ec4..e54d324bae95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f289626cf7ece5e4a695d062e21bc5ee4d9df0c92d5caebfdb2a15839320fd6c -size 22566 +oid sha256:3eb3eb1776800176b57a8494c205fa0b4b6b49e5614d5d8fea7f5adc057cd479 +size 22549 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index baa7044ff306..0b0c03532e9d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c1c6b47289caf49b61270fede8114c61dcf26383874665f9e1f3413d63b87c0 -size 27020 +oid sha256:a12dfd6667463bb19661b512eab572f1343ba9f61f249e63280c057b34262ada +size 26974 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 6a446878781b..da7a4b8c44a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b8d8201897fe976b85da452bf05c1e36412c20081672c3b45308eeb86bc5502 -size 24106 +oid sha256:3de2083166931efa3b9d2c6bcac0109e597ab9269b284da34061bb515575b381 +size 24044 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 35bc30dad017..9197a05d9235 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c7ddb57d44f399464d1185854b978f0352d5800b439994d0bc5942967c2098b -size 27145 +oid sha256:e17745bde30a37ad62873a8158b6b6256049eaa1eb7a3d162c07aee6c3786d2e +size 27102 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 1e2b6340f606..6e4600049c1c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57b2ea0139fe57c51822109b44c35ed437ea39d607659a53eb15285c7a64f911 -size 24649 +oid sha256:0048d191994fd5a6ec2df3128d35af16e652ff53e057feea06f526ee4f8c396a +size 24630 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c0d701b0e7df..808b69e5ce8f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:75b8e2acf39797b432c5afe99bbf1fcccab40b865276090633a6dcc9ffeb2f38 -size 28362 +oid sha256:84d031c26e16f09034742820a41f1cd20dead013b7567bd1dac3686aef6a86e5 +size 28329 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 46f6fefd25c2..2e5a5e09b6a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49f1cf43ac3b11d186096290eb51dbcdf78e732e6ecc2a127e94ae5452cf24d4 -size 25969 +oid sha256:c38b8de4bdfd285bcae0ec60032734fc0c381325982d75f5e8af48f1a6f8f9f7 +size 25885 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 08c9efa96723..8b56ed7e3fbd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a641fe9f3bc6455ab0c6d9f58ba82bf8c61f2501835c1fbed279df7db734682f -size 26904 +oid sha256:563c26bafe8d52ec673cb522885007eb381a7836be889f17b2ed58d67bc914b4 +size 26891 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 53156ececb7a..a61cc345489c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:75351350a6b13a30f09c6b15da9976dc7c3f737339cdc51065c11d6497a92f91 -size 24387 +oid sha256:651cb936f7b875d08734d977ab9cae8abfdc783abe44a67e71c93299c1057bbc +size 24381 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3a705c1ba32e..b20854ad05b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:12b035f2cbcaa1d3a82e3d12c29a3e4f3c345830f5db5488fae82841cf3b9abf -size 28295 +oid sha256:32ae9c6e90f78ed71910cd0e8a4d26f5f5e5dff4eab1c0d452205b5a1354ad55 +size 28201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e7cb1481751b..20151c5c88cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f00f77e6713fa27461a90467594aab1d0e808f5213d19d0f13796d0246c65130 -size 25779 +oid sha256:bbf02cc962a0eda8b2f5bd44182446df27f54d82b548c29d91d82d274a2ebdcf +size 25713 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 213605c3ae34..11dd83cf590c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ddf4299bca57eb933c26f5072414654975f602e415a83695cce118f4b6b07523 -size 29078 +oid sha256:f77121b8becf66be773fa7e6e04f2734cfecbb9aef31721c0c817f0af77c644f +size 29084 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 75ddf096d6b6..4b08476a821b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7115f0eeaf0ea847ed5e21e01d6fc32e1e389e6c6e78a2dd6c8f457a3668c79a -size 26413 +oid sha256:ebae275233e784f05b5696e0a82ee2b6df58c8c134d5f88417960b981c9037e3 +size 26366 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4189538e11c3..83a13492dbc3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57259f8ba05cc83a60016b49b944de5d9b7a8afa17c2ffde7cdd1d5bba4a5203 -size 30443 +oid sha256:fc7a7e7bd723163065569b8b64dcdadfafd7c34dfaf4e2db9a8f2f381b4af169 +size 30338 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 8bec9daa85f9..3e0b825b634c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e98d5c70863a4515ba6b9e3c39941d35d904cb91904fb9799eeccb5260b855c4 -size 28272 +oid sha256:e1c01784f8dd9cbf1e9a7f8d8090f198da8639143baee51e63ea80d93844485f +size 28231 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index b004ec649ad0..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f4b2382202a4d769ab6d19d0e8951dc774b616674f8458351fde4c2d79754475 -size 25413 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 5fdc2864bba4..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:87aa07c27e4e6659dedeab44042aa9a7f9fcb218de29a988d705b02b77da2dc5 -size 22841 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 34819ed3d313..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3f488928f6e2cd9f99d3a5ff2b6d29d51a368fe44afde20ecb4e0539ea8f94df -size 23937 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 687696a3aa21..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4bc84f61409e2fb925af497cf69ecf7555652328f292925d7f47e302e4279329 -size 26081 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 8b53f9b9b3ab..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9e71dee1e1e1ad374228f92e770fe4f31d6d9fcbc98e6a03057b775cbd6976e4 -size 22128 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 7fba17a46e51..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e9aa7e499bc3553b69e40db9032554419f892dc7f96c35d2a042c59e98437ddd -size 23542 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c6c095fec329..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f75a37f0a0268843089529bd5ba39cc9fb741242f4567097d347ec7e19faaf99 -size 25512 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 4e8391821055..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d61caa13abbe5491373c6e1b620598078a19c087a68a8064bbb80b750bd99c30 -size 22019 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index de7302516ae4..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:fddc9e0ef041987dc5a0f9c22173f8fc7f01ce310306fa4a26f5a04692ee0028 -size 21167 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 4af453f70af2..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b828df556b3b6a056bdd2eb6f932e1b61a95f0616b2a887cb5642e4c41404bfb -size 18556 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 9b36f03cfeac..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:1f5ff9ab4f667dc8612ee26a5150108815b93dd39b6969dd513d87db0252bb65 -size 23408 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index cf66fe8227c7..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e5877b74daf79e50d155fb94b7ad2def1ee51812e6ec1dce8f57c692b33f9739 -size 20716 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c8360588b30d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d1eac9a62fe8e35638d21419aecf0cdad95e20a7be4a80d690f5c9a9ab1cf8c2 -size 19923 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 80b493cf08f0..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:19a4cc54c0814a559ad5c5ad399f42c2b135741abd6712e51a59fb1b4d813b1d -size 17474 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index ff70315ea2ba..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:7996c79fc625c5ba3d75b3bbf177178017e97f28ce6cc35f8496e387f96fcf67 -size 23590 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 58350c5637e3..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:bf7dcb9c2c414b9dc846d4d8f2fd68b2c58e810ad32c781e6d5259d9988f164e -size 21699 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 1c0cad945253..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a5dc2453ac43330c8e02e2ac904ddaa8c822d4baf20cebd9f1b89f866ec7509e -size 23102 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 4a6e95710097..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6af621960e6b7725df5fae2a5fedeffbc14bcb22ef6f5e81e51d9a318052d3b3 -size 21726 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 512370da8d7e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d7b76b5fd9d4fedea0beccd2eae516438c121ea4fd6369519826ca9d23112156 -size 25421 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 5a3a54f542f0..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:68016c10ea796160144d8878481a906b8f5749d45db002fc9eab948979dc1e79 -size 23090 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index a0c4a1fe4f70..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:83d1b902245c777f62d33122624ee9c92cc72cb39dfcfd143cb9f4c030c658fe -size 20944 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 6bcffe3741e1..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9f16bc5cf7bc93adc93c576978c7f35826db6d2f04659c6e4e2ec517b53516c9 -size 18558 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 27daeb7a6be6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a51d3a2df48e6a891115387e6ccc886ff4fff8fe2ae8f0e5883b3c6aadc70b3f -size 19802 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index dd4d149ae8e4..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:29e6a718477a5fdfdf96d5be10ffc66053de278fac63189c974d9d1996b6aa0a -size 17422 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 7386f782956a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:66d845f094af2601929f1ac87d9e226e459ff8fc9517574cb0617bb228ef1654 -size 24622 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index be94879e2dbe..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:83e61de64ae870473d9e6262e2bd3cb8084fe6a65d97b2736df0eb81478c0e07 -size 22370 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f7dfd3f68e8f..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:534023c20359ad61d665ccd8e2f468796b731014954a1f871c0965c84c70ac3a -size 24151 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 656439268069..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c63edfab00fa47700f3f19962189d89eeb6a8958634bc64f8a31081aac07fe3c -size 22827 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 838ffde6226f..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4dbad7dfc8ea7b95797252cb79c3dba49b6ceec62734ea7093f3a530a5a13335 -size 27100 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 2c166800763c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:48b03e15101bc982eef3f377f4fe8911ecb5bf129d4390c8fad525fb6946f272 -size 24360 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 70e1dbc9bed8..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:0663cc0ccf8a5f64b05d15682febdd4f5a8d3ae13f1055d0dc238c6197f35cae -size 22022 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f954e4fb4f7d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:47d2e973ede60fd1d8e16d5c58ef3a8f5f6f2f4a42a390e01fedae9cf7c9aa73 -size 19071 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index e4e2e22ef00b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:842541711307a9a57b5c97e74e9d51bc5d40fae4075efb9dfc579a13aff0dc6c -size 20747 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index ac5ee30c6336..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e519eb4d49bc2be981075cd8f9959b398a06782b1973289afa8ef552b0039a5f -size 17917 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 51ba957abf33..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:1f1b2e1100d93355c6913e07f0603f0caa41117a58043b7dd307ea6ffd8b5e64 -size 29577 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f0d46c2f1da2..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8032e501f82eb2bcbd5af4be3b957dc3937a61aab2b865c235ec8df4f2e038b5 -size 24697 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 5c7cba4d1bfd..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e1322c93ed52d1b0c30d5080d4a73c817518872af0b31b4e6d398c2621f632b0 -size 26776 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 1b89e13b316c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d55222d637e61e7821ab83ef1274a2edb1255e4580b274ac8cbe3c5eb08e4e03 -size 23153 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 3a9327173e0b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:75f43aaadd334b98cfb9a4bdf25815296ae82e92d1474ced73d6cafdea2f3dd0 -size 26388 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 00cc58cf2660..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:afaea930e3e195dcf61a51b0958a645891a5a313accb5dcafd2a99c0822c7ebc -size 23832 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 2e4673f347d8..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:0e3afb80aca2a80937d03d4db26a964fe5cd1112bfd8a51f43308f5f38d2f1cb -size 28498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 0bbe96acac0e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:19920f489bdd66ef4438148efc79a3ed498bd3252da1582ae65701ffcfb56318 -size 27212 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index a98444b989b5..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9b08598bf7c1510ea5c63203244ced9760204fa654808251854e760d81cb968c -size 27785 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index dfd6f44c1b2e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:096ecf04cafdc19e6434eb13b12d0c73656652227ef5759006b616e39de2befe -size 26909 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index a558b305b27d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:2de0524c7065713481cbdf11abf5563c8a388b2475c6314c49f1e8d859a4b862 -size 26819 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 9eb9b05c1b31..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:991428e8bc93f4e76a8a948863732f538128b62ee46855ff619ef180ad36796f -size 26260 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 0259e056325c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:26b4efe28c807ef2555646df391bc9cf21c26a1c70d6b0762746bc043a1ab323 -size 25986 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f0889b399e8b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e5849ffbeaba963ba36504812bc2b7ca249789dfb66e4c5492d24eed49ccb26a -size 23208 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index fb6baea09b98..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ad53ced7a5e2340a6c5ff9f50dac65b56a031640a2330e963b9eba2051c6ced4 -size 24716 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c9d72b086b9c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:070381579a82bbf88509845485167a8204acedc876df99e966a095f5ae591087 -size 21993 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 0b79d33d2fbb..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:790afa3c309856d6e8696ec32ab6851ee8354106e6de5628cd45c773685fbb3e -size 29649 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c6c01fce92d5..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:28f963db878cf562d3cc42256682da2e43ea9cc6ffcc3beb2a92cf7eb9b97bee -size 28294 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 8cd35f5cf6a7..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3dcebe6ca9b12f773087da02676a63889c8b99ade1e1df634e71c2fe1aa46507 -size 28953 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index bd4e048c6ba3..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:398499fb791fba17a66f9a3ea4a0d06bff7e3079767a01a91ea3da245587737a -size 27878 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index b74efe2c6c66..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9c00c1500c8f8f343722b78b81a42aab51d507af45a8e78422d2d54298d20824 -size 27848 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 79dcd79a9bef..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:86236da529989110210601c08dddbf981a5243ec12cf043865a6576a5e764c37 -size 25336 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 6cefeed24d29..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9d2a7b48c43eef29bb1e86d299d769665c58fec46beac1369c8efd096817b5be -size 26937 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 04bc77693d04..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9d7a4e922430a2967c1fad687424785283bf7c3195326b20d65a0901562e3ba5 -size 23669 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 8ebd1046685b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3c1e2c08fa3f9b80131a7c55360ca6d9e09bafafb38a6dc6a7b16cb38f55b3a8 -size 25774 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 6ca0e573aec5..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a3380d1e93e916dc4c256e0e1e862c1b818b5a380441b4765b071657f0358159 -size 22380 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index c0a9b798b93b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6dbe779ae1bda36bf503edca7ea27eea4a5c12d64e379d0e3432913624b25e7a -size 23249 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 36060af88d82..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6c673d190ce61f1fce1776ee04e83610b7e068e26798fce36d9978a7bc6a9fe2 -size 20997 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 54990bf01672..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:72c2a899632177d6895dc109127349c2ca022ad8478ba5d35fcc12de55d691aa -size 21158 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 543dc7766d17..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:0115d90543aef530578d301535812a2ece4f29891b2e9693c817bc323e723a30 -size 24257 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 755fcb5d8463..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ff3e64ad7f6841d50349bc0afb51d52f632941b8fda3f65068ca4b47124b312e -size 20415 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 336ff9582950..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9d7eb553be556a7e982c967b80bfeeac9d1c08dc8c901d3472e2bee8d6721ddf -size 21490 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 9dcb66db11ab..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8df7fd34c476205bc2e44bcef5d7d79b55ce76725e66a0e3e63988f618273343 -size 23774 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 42523b90d17e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:510e20ced0c20c58c4e31690458233e0ef00639dca0831223439c66cd6426c1b -size 20692 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index ba0243a1cd6a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b7f2fcae267dd5b3536807d698a5768db45b48f86cf1a0504eedf454396fd73a -size 19386 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 6d5aee3e947f..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e1413d0a48d90720f325ba4fa9c9029ab82c8e5eb00c0597cd4d755f6baa2398 -size 17251 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 079db768620c..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a63d23dcb7519f906622d62498fb32aed35b2ed204738525e5f1e00c18dc0a20 -size 21342 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 237d5aba9902..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:146f4f2a9d914f41e1a516ef287cb48201adbad88134c951778d52444c94f34c -size 19207 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c8cf894d3ea6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ee9fea34fe670473d9b3aa575733790410e83a1ce109a11e36c13f0551f9512c -size 18641 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c08295e64f4f..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:7178c109b5db58165892d80d49ae2f9969f5ac9f1ca71b0d60704b0abe9fe05c -size 16400 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 5f58d6f5a61e..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e71021a9e034b781ed537922657ed1173f49e1186e42d3012f84e64971fb1bdf -size 20661 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index eb992ac7e2b2..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:adab9a320ddf1c2aa28c4ec590342918e3b4fa9ae74e1543cd923c9692a9652a -size 19947 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 9fa364143638..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5e3e202c316eb67f35e7d15be53479998c0c0f2e1f59d15ed710322b53c6fc6a -size 21093 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 96821cd496ee..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:de81a54d2ddaffb26bcf2fde2d379ebc45905ae306272f09b90d180fae1120b6 -size 20361 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 9d61d06347cb..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4fd1af8979a4a61e4f89bb8524d0f6f300542c651f75c63c9252eefe49407a56 -size 23184 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 71e76e424907..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c821494308022da38b0be208773f9e1249a292dbf7018f28d8554a350d530ab2 -size 21187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index d83ee4b5f515..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e96269d43b22e8115bb807b188391b1f657d575b68343078037feea19d0ba19b -size 19348 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 8ae5f6eb5cb8..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:64acb5fa2c51b36e5ba60bd4e270e1cc3cf8dcd1dbc1b8c51fe68fa7a16c41b1 -size 17144 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index bb8c6348b965..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:38958960b7fefeff7c8eb70ff4c7c8b31167239c7c27c58cc4d1b728fe401116 -size 18572 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index ed31700896ac..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4d04ee0189616a26f130dd822e2b7192867ffee22c968697a1496208db8b4922 -size 16261 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 0a33b1d3c3a6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e1ba3c5c9f7689f1cbf202013538454a208e2f579d1e056e8a7fcfa27091cfa2 -size 21838 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 830ebf8fd531..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:1238a8beaf60ba0f753d79a2fb31cca7f5497138289a47f32be668f29732ed51 -size 21107 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 861832fd0606..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:26c4ea55a17490d7f6f39b984c324cc99535225355aedb5d4bb9905fd7d302aa -size 22249 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index f6fdf46fefa0..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:9fbd3de980362bae34df15e88d77f4c7416aee22054c1f4975e24b7e0db0856d -size 21497 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index f97d5edc2fd7..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:62f0e99abf61105b23ecb07d3c52e308b0595131af0b4389272ac0a2f89b699a -size 25067 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 77035456530b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:3873b21972f2d301b72aff5ab1c94e6aafec9abd29d8b8a41f7aab2d229976db -size 22427 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index dcc16c74bcc6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:c0fc58be2ff6c6f46e86b35f3a224a5a45413b920ea464d471b34dd8f4893320 -size 20229 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index bca0153ef18d..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ff7f1886169343f024279b965d6a386bfb6d3e3a2ed5dab8e9f0b8d4d2d41969 -size 17710 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 3641ce2e326a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e3b9fd896132d85df83573b7abe6edddf3d40912d3e33ebbfbce6591c9c4cb47 -size 19181 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 626ec4f4ecdf..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:271967734b2fcbd3863f06adda9d8c0b65b1baa74c8c6a6f2a8eaa00c5f89977 -size 16935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 85fb111bd0ad..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:cf1da98d1f0737144197920b57afb9856fecc5bdf28e00dba07e2ac0bb7abf7f -size 28429 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index aac2c6a99753..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4afd1b0660818e329b4ea7283117d5101610e8c3c68795d5e0ad4bc69e685d18 -size 22641 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 558395527697..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:1f39405285e289b8a6447ef783487b10e187052abae576795abe47bd27031f71 -size 25197 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 2394bd35344b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:42962ae280995b1b28c896dc6cf099a1522456af28922e90a94d357861cfc750 -size 21842 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 35e41aa51f8e..b5ba727f6d6e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:02c75d7b32dc35eda66ae83d7cb016b397cf67c7ec4fff17a9a2a1efb962568c -size 26387 +oid sha256:98ded521405afe01b99ddc35bbb54cb0e0c2a0ca13d246bf8c293d1f5866be94 +size 26379 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 293890c386fd..549b61de0169 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8db057cf2101df8a9d8d49461a67c496e0c2f072ec75193808891de540a56fe0 -size 23692 +oid sha256:3bb6857c7f89a665ce4ac50f0e2ba2c41391a5910f776b314fed8a2acf579ce2 +size 23677 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0486ecb5bfd8..6cfce0582e45 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4f47ae42d046d4d5a5d23c8ffa0582507191d01a1507fd1c41d443270423d36 -size 25460 +oid sha256:9f2924e759475351ec1c5227c2e3bb58b724977fe53a2fd2aac704e83bf7e039 +size 25441 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 0df9a54bec84..1b3600c32191 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0a2df729906bc2284d7441d9a6d25113f56e260266c432b9c073c8454bd0c83d -size 23963 +oid sha256:b2850ea26c16199f07ef4cbdaf37959b4f3a9b3075ef369d510104b5643b60d9 +size 23773 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 417c5023a8c5..d12a736c3d4c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e95c516fda5550ac2a807bbb793ab8cf9b82023250d30170703e9fd40f6abbf8 -size 25957 +oid sha256:b62c3d40f544d2a99460aa0abee76d8657164992cd378a1cc4ee63fe676b2403 +size 25865 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 17375818d3f7..e570457b0002 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f9972c5d58723eb37ad18f38186ec52176b7a192ff5205fe82a512fb64fbbde -size 22125 +oid sha256:c86cfe1655b74a11bf699c47f6a871e2ef332e16bccaba3b57b3090a66d4166c +size 22081 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 273b8e52a669..f9c561bb9e3f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ddf99810df4f218a4073e364f80fa92a26ac847a0f69b20cdbe08e8333996c16 -size 26107 +oid sha256:f9d56800a33ca7573f20d9e0fe735ab28b22054f39d4acbf3102c8abbbed8044 +size 26133 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a021ce91a2ce..0ce518836f19 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00ede9a834e48272ab1f0d7bd6547874aa15c3d02865e57c62cc516c7aed4bad -size 25039 +oid sha256:aa69c084b3a7b1e20ef26f8ccd994721777fec6d7b9f9a0faa8a67c0b7ec1fd8 +size 25049 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 67c8b443c2d3..e51a746200c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9bef66313218910b0f35120b43c7d55d2eeeba5503fa23b17d37509cd9eab0ac -size 23345 +oid sha256:8cb583e95bf3a4be46262b2f9540fc8a0c92d4ec33d70555aabe229240ff6c7e +size 23283 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 884d5766423a..51a3d208895b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:286f49a87c5fe4669dc631180fc657a8650206392a79e81c711b06e71c05606c -size 25378 +oid sha256:87100c7ff5fefad426230c480b4d30382ca501adec138632c4178b0cc64e8306 +size 25330 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 293a4a0bb8b7..39e89edc86ca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b7050b30b618aaa56b2b547c11347cd25afa20137ac8499f37be9a01cfee5460 -size 22002 +oid sha256:6daad164c4e7386b13ab3569416173d075f4ef31d794b8ade52aa88c3d970794 +size 21877 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index a6d4e4474509..b094afad9987 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb194dbaf49e321043c6810f92d9eef385a10a37f51abc1ddfeac5ff1eadd91e -size 26005 +oid sha256:b6500015df916f08e992e9678dff2d641de7773fe1fad141c20c85d098788e0e +size 25986 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index 17c009da9388..cda2cb61ce38 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b0726dcecc1e1207113c429839ce23daedd201668f9f66755162cf7d6d438779 -size 22413 +oid sha256:19bd9860bc519a2b7f578b95dc6b74b30103daef91cbf09afe2c98d7b3ad550d +size 22415 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 63b89033ea27..380deab8690b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a913313bbc83560e0c3aac4fbc5284e0f9f87fd8ea07db83cad22d9ce010e1d -size 20331 +oid sha256:113f42b6557f952e3d3e5f1b65e11586febd94c6deb4aa2bcfad14f85ca8d322 +size 20313 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 822827e2ff7c..57a31996bc3b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dcf15836cde3ff85306c74872d7fa8686e8bd89e00f2923513603c2ceb1853b6 -size 20744 +oid sha256:13333822761f1c9fb6fe6cfbaf59822be5d8f19c3ae301d8522144325d4694f3 +size 20747 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 692ceda96d21..37949035de63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3547be34d4b39f26383cbeb650b03f991e3a00ee7af113deda67e2ab9e888cfa -size 18336 +oid sha256:3de3b1ac1eb456a43d5150d0aeb2c81a1f61c91902a2f34ac86b5818fe5ff783 +size 18337 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index ec3b97ae737b..5d92d66c545d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8b5405a6adabf44d76bf46d2263d5f8d15836e0b9da0b19e06ec7712ef71530 -size 22776 +oid sha256:5d66b4c9e60207f5c1e1a4961cd166293122faa3a173da66f17ec4b586903e2b +size 22355 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 9ed9d0e0cbd7..9db11ca32945 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cee0dac4fe97c49f089f6386fe377ac583edd2776b37e5d947043e95a68a06b8 -size 20289 +oid sha256:58a91d241d37f142c070fc0db904b6082bbec42e054334a753090b1c65f8ef22 +size 20302 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index e1d68880943f..d431194c3c40 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:05e7950a03c0152563a4bffde7f699238fd06a19fe51e0ba4fb7fe3cf7ee7945 -size 19792 +oid sha256:2a8c26f4b597dc13a69f564943fb45ad7c33457bfd811ec5a9c06d878c893605 +size 19791 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index e5a938e3bafe..397ed4ea7ff0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a96f1ba6fe39b14c6609074dc215ae89694164400da76dd84a8f1d83c632ca5 -size 17405 +oid sha256:b06b48f77ff180cb51e8b35a1c5b11ac2e73833baa3319228d4efe1a931bb8de +size 17496 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c157ca1cd336..e947bb6d543b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f02e56d8aa6d1af6fb155cb4cad8ddb25a14feba2fde8347482580e09ae51476 -size 23534 +oid sha256:1f1d242553bd109c55c05944dfc94be6b4411aa2bb31ef5702530a5cb49b1b11 +size 23463 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 0e4e673dacf0..1e6d552140d7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93b66a85ba569af9fe1cf8cb4eb56aa10be3fec0fb3ceeb5737c3250b3103431 -size 21138 +oid sha256:efc9333131fc23356e87efd362834b53a0b02a09443300add018b4a7bbf56144 +size 21174 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index a8b802764f16..87a45237115c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:24d8b5bd814cb061a4c8e17c85258f8fdcf7bfb0b6b3a2ef73017fe95c0820c0 -size 23496 +oid sha256:d3055a6dd21a4bedcb84719081a9f1c96361347f6de7c3f810ed3095e952c076 +size 23429 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 142dcd2409cc..4f82be4b6703 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:526b56160dca46181248f9515db2d23e2236374acf6859015d76e569ea3dd7aa -size 21668 +oid sha256:a36b695b26301254d4e5ea034e7f4653745e2501dd51911dc7ae00e120d15c79 +size 21744 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index a8337ca816d1..957c84cfbaa2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1263d4e06e8ee2e0a8f610e0791740dcbafd239ed8cbc13b3fdb2790b74a2334 -size 23155 +oid sha256:bf4a217fc091d61a03b72f96c1a0ce588bc047a31af353a78aa49ad94e71dfa7 +size 23058 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index e87247c0fc30..096048b67d03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:27c573471fd3b0eed90cb52cfbe836c581f542b8611ff84961d0650957806679 -size 21685 +oid sha256:6e19a2db0b4d733d403047708be639035233ba2308efcbaa8b50315a257f41f1 +size 21637 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e6780e0075df..60e07bd5a71a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc4e965e9198c9987b85d421619566f69f27f9e028157151aeabb34e315a8bdc -size 26031 +oid sha256:4f855f3484bfa872f26f2b8a12a1f38e799a5b3ebfe94e103d3858462c35595c +size 25974 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 63975c5ab1dc..39a9b1ab884d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d82a518656cfc03373db8b333d862e9ecababacd531c8d66741055c00b7e3782 -size 23910 +oid sha256:0c884227c3a87e0f4a979d2737c168b3b21aaf1afd2e2c08364b00e2c8de3279 +size 23891 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index f2dd8ad9215b..a8d46eede06e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1806dad9498732b9d5ed453cee09b7dc8b9765388b18c63a5b1dc09561d0185 -size 20578 +oid sha256:d9d0acd3d53715f2324b097d506d8fa6a20cdedfa0935c957dd4b41ee9e7cd1d +size 20538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 5f6fea5731cf..cc1bda95ecea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:737f367912018bf6f158895d06a82e986a01077ed3e9e1fc23f3980f6e063998 -size 18147 +oid sha256:13a709a8a611fe9a309558c4c6aa4115cc9e885e4c91b04631de40230c806b76 +size 18150 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 0c9bcfcab357..3149e428dec2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:490939a60d92b4ed1ff419317f06fe21973f96d6b8a30354533cec0c8ddf21e5 -size 19699 +oid sha256:090062fc9e907d08549811a76e255c1f267284fe00e4ca0f22590ca38f9d995f +size 19685 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index fd15e5dd3199..ed75614cfffe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9524d4af13374566444c8e92111453d4ddc70d3f7a0745d0ff63df6c9a2e17c1 -size 17403 +oid sha256:5e26d244c84423fe1841d8962eaa1c26cb2c85620598f1fdf151cd2ed8fa7838 +size 17391 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 78add708ea6e..657abe851748 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:759b97d50a9c2a39b37a79b0432fa612a86544e49b9dcf34fcc2305400a8481c -size 24680 +oid sha256:ca77c1f7b76e66b499ae094372d91cb47f4947dbb53b8123c2ba6bfa03248f18 +size 24641 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index a8feb1492d25..c2882b2e684f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e859c3d85ecb1b7bf3708459bb270e1c430ed02feb63ea78f4f7f8150cdc45d -size 22266 +oid sha256:fffcdd36151608c6f9dd4fcec325ac65a3ea5c33d6bfdda8ade771815584973e +size 22309 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 001a4263479d..a0adf39fa86c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:24283d51eb0f924bcf1057518a0bba7f08b4527378d846a751dc843a98cf1cd0 -size 24153 +oid sha256:4c56efcb40d1ce2ca7aec1a1a147b5b99ff1e85a89e671912e7ff732ae0c0031 +size 24052 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index e36b439f3c70..2c59b571f0d3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57b2375866fa5396df61636fd1768471f9b009914d14f3f4d5c036bbbcf3f645 -size 22774 +oid sha256:8bd0b0ac2e33a93ab6f4c0545ca25cdb164a055bdb21fc631f3decb8701ecae8 +size 22721 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5841173b858c..7dea76b50317 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c41223464e29a9ca6474cb418d78e9851f4a1f9970add06cb6e791062d7a46c -size 27981 +oid sha256:8d16c67574da792a30db988e83f792f1dc34f73b0d99ec11e1535b214fe53f9e +size 27922 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 36c890489822..0271f3599ad8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57bcf2f922c94721ffde3518e74c8e4f0bab342c1941f27ef5eb7735931a1e56 -size 25057 +oid sha256:13d219b457b2dfeaf2228727955748231225b48a7f05f45e6e103f2d737c8b7d +size 25049 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 6c1f0a10edc1..182d5e3ba187 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4cdc3c29c290ddefb76fc77822ee3471f9dc76f44d901e5009460f941bafc4f6 -size 21560 +oid sha256:ad310436a0ad6426cb14eb8e6a466fed0f6af0ace6ed5574abd601a942ce3895 +size 21534 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 6101682d7b14..43b53f764e0a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:df47a3a352ded819e1e8406ee1da7850ca71165f979088efaf4e3b23fbb4b41b -size 18748 +oid sha256:8785815b5243a201b321c18abfd5837732dca7a075a3b0f3e854b71173c275d8 +size 18733 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 4cd3d62136cd..bca819235fec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8088fb3be40be3166fee6f479995643d79558d6277f1cf532cd11c63e11e955a -size 20598 +oid sha256:ef08bf585d7f5953a1ebbccdc81067d954e8976e3f1de11147b4f80b71c3e423 +size 20604 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index f6aada20d4ec..111a4f113008 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a6fb7d24bb32b0b072594d5982c0ec5bcf12ad976626654f34eae8094794a77 -size 17843 +oid sha256:8e50142efb6a85ed8593d8883bfe91960f10b8d1c13b62c8815eba808902f704 +size 17763 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 0f3df5e492a2..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:024b87c1384473432f3fcbf828d48f60567882b7353bc3fa314a750d7719fead -size 30098 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index e5fc791b3186..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:34ea1bcfe4173ae852abde6f52d0b8df840070d941b408cea4c5f372fff482e4 -size 24626 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index e6b353e14369..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5107be6fefabee0fe5fd9207a90f3a00b5fc066e0975c8bd35aa6accac2d7d94 -size 26855 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index a712b98bb0c7..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6a633df8df4b8da8071755d97cf2e4bb6fde2380b43758647a85ee993c81e9fc -size 23081 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 07998369558a..4874bace3e18 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:18a0daefec3d76b9c8722a3d1b01e0295ae9940780f03aa33ec46f83a3019de5 -size 26962 +oid sha256:27f9acee30851d7e8367b9aa59341dbe5594701c7cbe44a641e954b921f5d463 +size 26935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 85fa25e35eda..ff80ccbc99eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8e2275405888e63516e4561ec3a827c5b596862a66b30fcec63f7d986a72ce2 -size 24230 +oid sha256:7e9102a87c2fa359fca1147afee80c164a020edc03cf32044af46636567ae61a +size 24216 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 4164817fd4a9..48a0a04d0c93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:959624ee5d74b121dd430cff57c86bbaab77eee6c5760f490541b0691c88c477 -size 28548 +oid sha256:d56e5cbb380758636b9e080a563a6dae48bf2fe842822a0a977ac4ee8b76d741 +size 28453 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 7d6cd2355457..e17db754667b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:51f10eae103afa5db1a4d717210834f06f2582365fd3cf23e4199f6c6d0fc27f -size 27075 +oid sha256:75f20fa6bd8802a63ed3712d5c33d7c738dcf22911afa04a755ceba9e65c7022 +size 27039 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 5ac5d744520f..5acf69f82e99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ca3ecc8a5de461df5cb57031b4624a9d28e959b291a5507482d41e913b88ccc -size 27730 +oid sha256:3508c7fbf48c8115d416159c36c6a5ec2ca0b627857e05be79c9f4d7461908c9 +size 27621 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 3f2556b9a429..894969524f05 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9206adfaeed34da0113c8ea9e76a512c07b76ba3135086058a5b4b633dd7a2c5 -size 26839 +oid sha256:2a29a021659467e5d8f384552f77002b7edbb9a5b186bd39563624ee784c531e +size 26773 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index c54bd5a8f173..b7271ef4e900 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8407f8f1cc722f344f8fc124332029d9d49a3cb64a7ea93b9bfcc2fed87b54ea -size 27653 +oid sha256:a10f5ab2a23b61938d8aeb4206e6d3fb67c53e7fa17b2e7d002bd6cfad638450 +size 27643 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 0c85f9b25e8c..3f316c756628 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a63a5dd1e5bfa2c65d901c75227ed20f9b86e13724720d22370994086aa210a4 +oid sha256:f552252ca7401e6284f64c023552ab4f573a02eef54d9030fa444a739f3e1b97 size 26542 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 5f709bf012fc..f8595d7faaa2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1a0f5c4ed98121f39421e4ecc41f7dc3e181d1af9ffc1003a96cd18e30411fc -size 25510 +oid sha256:d2265cd05b36e651c94e3d0a2689b2118234d69c6e089ec21aeebb2d57ac3324 +size 25493 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index d2a134614dc1..3d7f53aea85a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3bef2228e10af482accd46b54dd185a6aac2e725f9860f45bb4e2080911c21fc -size 22773 +oid sha256:91cf2a03af3c6f82cd9bdebafe74dd3eb392b43f2318f85eb2324e7d66e363f1 +size 22718 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 28b1c529934c..ec224a1dd537 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9dc93e13e01e9ee4e67af54b10d14b80ef1a998ca551f02dbed773382f1741c -size 24467 +oid sha256:45ec157c737882441b5dd2b893ce89da72a29c9a95c0c13f90c21bc88578b9c7 +size 24501 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 1e9402ecd2a2..bf96a136093f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21a35c1b57bae20e92c097fb1efac30bd58e4e19a5b314dcfcabf7096c007fe9 -size 21832 +oid sha256:b1f053c08885295f6c4d1be49a30cf2ef744c6bc6a3a1954ccce660ed2c78e5b +size 21776 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 0d7c34fea59e..c7bc088dbc71 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4dbf1f0aa824df45c7f4c0c7e2ecac65cb818ae1b842e55b8c99ff2ff8abe551 -size 29578 +oid sha256:5554b2dc1ee4e23e5bac46dca47f115373ffff7501aa7d73ab3db884390fdddb +size 29409 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index d253b60ab7b3..6f64eb69a20d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00bd40ccf3bb24870c4efb69ad7122a9307f3af1ed34cfd8590758bc010f9eaa -size 28211 +oid sha256:d84e652c1e3de9938bcd2f6987760394e5ad4b714e03506a9f039eb2c8b28e93 +size 28184 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 2b444a45a075..33255ae7b9fb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:874a42a6dcf721349b75f3b4f7da856a51d2b37e4952e7c15779aa262377ff36 -size 28869 +oid sha256:ff20c9e416901f58644726e1d6ef2e6a6f6a32ee18140597519a5eea0f9c5285 +size 28749 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index e11818b98c28..e0e9364fff33 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8bc37cc380efd457675b48e011efca250e73178c1e1f5d4a0e3c68c8b698d91 -size 27826 +oid sha256:656407f45a7b76538523ffa8981b573afa1fb60cfc8e0aaeace1674993d60697 +size 27707 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index db770ca68477..011b2b701a6c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ee52b714c9e9b1f487ccf1571d84592d5a5338a7513c06fd2f8255b614809b1 -size 28710 +oid sha256:0790cf4e52ea3a7f771029d31a7adeaff573eff1869a2a37e605e5b2b1696edb +size 28742 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index f19d45933886..db4c36d110f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7938399020d36dbc9f66e64ec0c545750e317d4c3ee9554ee5a9dbb1a8785bf1 -size 25842 +oid sha256:e189107b812d0a7b131f52dfe493d7265ca4a6f9be8beb68ccbb14525d9705c1 +size 25830 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 3dbb3a0186a2..33557d374d38 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d3b49c49a79325d75a155faa43d78db42a6a280f2b20294690d320f5a8838562 -size 26596 +oid sha256:b8d1c7d92e3dfa330250d9fa8521ad353a27dada86bb69d5eaf28edea10410c2 +size 26636 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index b3ad582eda26..c45492dd23a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e21f5219b425d45e45177e10e54b2613560bbe1251c657906afd11687e908a9 -size 23249 +oid sha256:a5348e783c177219a68eeb978685c4b12f7b42d80911db0b423485ab4beffe3f +size 23195 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 703e565758ae..fad69bbff3a3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fd178693c76324ee3e40a7d934b78eaefaa48699aa90d4f80091dce4cf59652e -size 25476 +oid sha256:1f3b86ccd7ca7bd8b0285749db69506223c0e41e74efde922ddf7eae3a37dd7e +size 25487 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 783a340da951..59148d5013a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:19a52ea04d06adba81f202455a35cbe057cd909ebcb4e599c718b4fd9d05c887 -size 22248 +oid sha256:24473052cf9bba0b89adc0aec99e65af7db5247158176f9ce86f956d2f1dbedf +size 22250 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4123bda3b5b2..237005cdd3a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a6ba0b74bf3023f643ca787b96e30180333e0ad6f54f0c123b9266a49431d44 -size 23060 +oid sha256:71afbb83284b2afd301c54a808077ff49b86d4eace540f7c7942e8500d678185 +size 23047 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 17f3150e4a63..3300fc6b7f4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5629b0073f1dac38c6e056358bbfdaac8b8b8490b79bd12778f6721d7bafc391 -size 20900 +oid sha256:bcbe7137bdfb34f3281d2210df0d174e4f7797c30261035b6fda6b439a795b01 +size 20850 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 122a0ffbf902..91e3e07b2480 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d26ec617d30aae6cda9620aec30c79d2673cde32bcdfc79bb9db7730416169d0 -size 24213 +oid sha256:4293849933a9fbcbf2892256713c332e3b1df296bf2216167f49c2c1d6b5e2dc +size 24238 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index d45c45602e48..05ad60613f03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:894eef2990808b3c0dd448783a52478660ed1d3aa78e1ecf74aa058465cde090 -size 21740 +oid sha256:fcfd341e8eef915d9202df69e94d2a758bae4c1356aec6d52dc5a7d4b1f6c382 +size 21645 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index 8df3c38e2cd4..03660b0badef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b271286a6f2ea0b33b5a5004ceae5baabe39e738589c9daac0c8c7d579e3da08 -size 24218 +oid sha256:73201015473716ce578dc7c262171a488e5e94388bccd1ca20739b72f1d73d44 +size 24357 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index fa29249433f3..121aae1878c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2124c873fb1d3a07e21369d0b653a0c125fd9833c3acc5cc3ea12792eb80635c -size 20855 +oid sha256:17441a4a3252f613a6378470fd7ee3fdf426a6deb4922ae6ea15d1c80f308a44 +size 20834 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 650e037bd146..c0e5fbe35f06 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f663fc24a8b567a16f6edd7f8e878378d1f2fe9fb1c2dcf02657d093fbafc9b6 -size 24892 +oid sha256:04c82dc3036aeca8db721ac2b98b0d9dfe538a03a1bd88e8b1850a97f5fdffb1 +size 24991 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index fe6eec957428..c8250a55fe59 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb34bbcf37640c45bff1cbd6df594f0e0bb7d28a5b9c9531185b0b8037b2c341 -size 23424 +oid sha256:d64b7f65da64d1bd443d2013e477fcc5f8c1c3b057508bec6278787df046d91d +size 23422 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 779f559f1655..0b5b9a10218e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a875390a26e3cd4f3d999aef9c5c44c2c16e5c8a309eeaa348f84ce8f19e4bb6 -size 21404 +oid sha256:5ae4706928356486e898072a673e7dc8444dcd7140057d0cd6062cf701fc055f +size 21296 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index b4c327b45981..9b600626cc32 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:da5b5bbd48c11a29da15a4a8f23c77d40035cd4242bf5b4f3e3491b47505da7f -size 23577 +oid sha256:70d38b648a30f593196e77b40924f21a91f4bfb14e63c6afdd8522bec54dfb6c +size 23490 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 76b6b92c7531..3259b86d14f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bfd21f221f71208260f973a7de704ed9af92630136f7c3569d0620f8f5b90dcd -size 20620 +oid sha256:235872a5f8789c7820d8ccac03c0d03aed75e2a0197053ff8127ceaa6a6507a0 +size 20475 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 5c43ef97aa4f..62d1a805fb39 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8cdf68110286c65abb8b9923109ffa72009c77b502217c5b70649700c1f52673 -size 24404 +oid sha256:b74fc450190891130cf4eda90c81b31f1d2ad38a98a675f8da1f251ddb224713 +size 24326 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index d68d5f767841..714f378befb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c83dd4a4ec4a225a6e998c34ba87359e7f532f132f8f06bc61b576d063373148 -size 20812 +oid sha256:6f83c4d982ffc9fb8b8f0edda4aa5d57c14940fc4567839351b44480d8d52b4a +size 20799 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index 94a21c8153f4..f63960f60bc8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f0ede5cbe5cd63e8f5ee4ae4269b878cc237f7a3c3a4cb6dc57136f620850539 -size 18782 +oid sha256:6cad4fb6a10700486cd4b170d2b8e4f2b91d99957cc8105d2b72b0e9f6aa1eab +size 18749 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index 1bfd74b1d69a..94589b4917d6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8ee5827bec3f5ef014b20643b9cd22c0582335d841460a57b6b57562d5196bc -size 19348 +oid sha256:6f5f0cf0f0cdf35cf0822038074775aa07d2c930e88fbccf50e0a7c3ff228bfd +size 19433 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index fc3e752c665e..e260748b3b9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94ff70431f24816fe51561791e4fba2e77bfcb12f5a659d4b671395875b06437 -size 17166 +oid sha256:8dac19aec78de946de66e8daa367e0369fc777ae84fc0094164b988d8921e59a +size 17255 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst index 7b8e9d354119..46f08d8f8791 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4fb90c6ff00c7a9bd074786b3182667cc64d3821ebbba6069aed787766af11c8 -size 21248 +oid sha256:3f3687c070a30080a819ba9a312a6624f784243cad7e5d9fd9a6fc1c2c05b5ed +size 21304 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst index f14de5cd4423..1a5c0ae5db34 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:745c8910dd12d304acfffdeb55fd859905acdaa9e060f98d2a54e02cb344178f -size 18958 +oid sha256:782fa2afe4754db8c017490b49c45fcc33bb67e1f5d5e2ce9d12014498aa6f72 +size 19093 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 4bb4fc7b9009..c0a257240cf3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91a497e2821de6e5f7ee38133be04b2f4a01f6ba1c4561a859c63430db521eba +oid sha256:b1c9d906a8786fae715932bf30d62bd90417856e2eb389dd42951bfec5a581ae size 18637 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 6d5c3e9da8e3..f9f51a91636b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59945e4cb0f896db625648d239689e0caa1e5c2e9bd2229d24ca390f7d2ede95 -size 16453 +oid sha256:0bc843204df47b57f89067c8405f30c1e84747d093cba2d615333ff725563875 +size 16533 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst index c23b7c23f016..273363381023 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00ff1c0d47e28a02e6384f626e3d2a263d9ff1ae6c92ad54e3195622cfe0c8b6 -size 21908 +oid sha256:59447bfbe8e3114147aa355c7203a28e2e63aa67bb07177d35a95327e347e8c8 +size 21878 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst index b208ae909a2e..9cf3a7e7f73c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c816be07c76eb9410e6191220b03b4f6681a4924f725322f78f0819c6ce53a1 -size 19800 +oid sha256:adbeb23169fd8dc1addc96cd12a0eb896e086dbf24dc4821cd3663ddfd86282e +size 19778 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 8594421442e6..6f96191a3f88 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9e0d88505469a2516c0c23651de6cdb359c72e4a217f6a91e5ad66440495a060 -size 21355 +oid sha256:b850ccad91fcddc4c4ba5739a81b87bfb00c0b559ac355e213b9c6aedf89efef +size 21322 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 8eeeb35f7f1f..9ca3889bc614 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1acb53ee40ea0ce76e63a61076852d48340cb220790f7af22b6f6e1eb78ef16c -size 20503 +oid sha256:6aab34193b83981528274a8866b4cdd0dfb8f26471631ff66e26f2f3bae9c0e5 +size 20518 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 4bc5223b1912..ebf8c48abf2c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:44e7e25c170e16682257d59ad4ddcf99aab3b02174cf58771105067cbd516105 -size 21082 +oid sha256:a8ad51f3232fdbbe6d55e42b67365ef811ddca22a9aa392d0d81e37f753f40f0 +size 21058 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index d7f0a5973142..f9881f5911c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:31b7189e4140ecf37396319f65318c1b8e57b80b0ab2565b2936234e156e72ce -size 20241 +oid sha256:c858468d0f6db8ce024d74e6f3aa2036749480c66ee877f8a88e73ea91b1e56c +size 20207 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 84e7065fb580..fd5cfb5d3bc0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c104dcfce2c5481613d5ff6fa3923e978efcd2416c369cee6d0a5be96fb0619f -size 23120 +oid sha256:19ba2faee9f85c2fe3de5ede5d85189bd0d09d056b705a7410323f627d5cffb4 +size 23148 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index fc2673067031..a8eeca4a3ef8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3312b41f1712da43c4b6e1f66a2609528121f977ef3ac25c4ceafa19c0e31c80 -size 21213 +oid sha256:3428f8c57933022ee6c741b04e4420df8270bd27334bd13b0f76fd85f2b51801 +size 21140 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index e31dd10371d0..98bef0513589 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2322f3e44a20be5b0aa2793cdc5b4e341d468e52d9c1b8a6bebfe641a8d8d3a5 -size 19339 +oid sha256:3b9fc905f9332d773d582d8450409160a3e18752af1d1bfb2dfb4b26fcc7dd9c +size 19305 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 7ef38012cad4..a79bb672a998 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:299c8aa7112222d6ce82da33c499371504970cfd6c3b1e1d7d9c15409778c9aa -size 17120 +oid sha256:9aecabf4979ac44c54c64344af42fda83cc193e2740d366465c2db652e965f1d +size 17098 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index cfbdd9bab231..82637eda2536 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa9d996723fba1960f487fe80203172eb44e80a533205293cdb9d5b236ec520d -size 18525 +oid sha256:acb65898d655e412cf9526e516ce6b241112704b174c05aa9fd4148f356c5ee2 +size 18518 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 47367fa5818c..786fa88e6d67 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:389d61b82508a6d7f70c7afbd82f540316f2a31e7202cb4ccd9b87935c8179a1 -size 16285 +oid sha256:6c7f84587072acded609571fa8a8a21c16ab1222efb3ebfec4e924b5ed913134 +size 16272 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 74fb7139e943..97dcbf6839c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8bff03b888cc9d12462f63340ae217f9fa077e086fa8963c5f36a7b0061b241b -size 21866 +oid sha256:26e218acfa5a92272a6740762f3de60902bdc155b25ddde7ce9f83defda7a5de +size 21816 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index c5f8a76de3af..2e2dfa4905f7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ebb135fc4b39df195178f08339a037782e875108efc99bee7e399db0e9d8310 -size 20994 +oid sha256:ca9ac485f2059f8160736ecf6e014299cca2d2ffa437e5ba778e825f59554513 +size 20948 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 04ffcb74f488..82b82a0d3f73 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be60b687b2c4d672e7f2022249f72d500646fb9c4bf158d829a63f64858f41f1 -size 22173 +oid sha256:d0c3df151cb4ea840606632ecc03ff50e55b5610e8671ac90eb64fafb0821a4e +size 22074 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 186c04d0da47..e0e202ec30f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa8950d5d4cecc368b3c06510760d2cce3465df397bb60d9c27448448684b502 -size 21405 +oid sha256:9b3cd3d3ce6c79bfd1418ef1b6215a31ea1859a4a5a780f5e36bcd608c2c71a8 +size 21341 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e2870e99785e..929112df4ea3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f0dc7de81472905ed199c0a788080ebf271c36ee71dfc730558130cd08204522 -size 24914 +oid sha256:44031699fbe81833f101558c1bea464673f5bd2cfa0d27c556c235957fa9c00b +size 24904 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index ec9c0d206e6b..c51debcbfcd1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1744f66104c1e49aae830b6d46844775b28b6106284ac79bdeca832a7856ccb -size 22332 +oid sha256:469a60a4bae7485cd10758640575ffcfe09888a6ac64e7340a4cdc920af45a17 +size 22297 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst index ccb77fa58b1e..4d813e014bd3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:688aa370f94f72b253ac94655e1da2ef94066a48dbd70863bfc63dc2a6f0650c -size 20235 +oid sha256:91fabd0cf902aba153416264bf65d3398ed32de96a81f5b8968c185fd1c6cef7 +size 20220 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst index 7f6c3121e4c3..653bfe04c78a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b7b9a2e8cada04d3f0a785ddffd1406c231ed17f8ede50b0b2e6c61399d5eb70 -size 17642 +oid sha256:c83a55d934eb28e59decd0791cd41a6e7412f133c1282fecf598ec11aa963022 +size 17611 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst index 47553d712068..2564e6b10efb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f8dd75b997e59a696d8e110417138f8c41935eab60adb3e28532c75679635ec -size 19085 +oid sha256:395fa1427d1e14fd74e0b32042761825754ec88f91fdeff2179d069fa6b28aaa +size 19054 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst index 61602ac7e787..c9545499ba21 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8e0ea6e6806aca323541e10a31c2960b0bb929ef7f4b263c32478f383d57ba38 -size 16879 +oid sha256:c925be2cf3d6b1604bf58f6f454fcfcdbb0e19de54849c2e45b7f43b01ede822 +size 16855 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst deleted file mode 100644 index 8be30efd405a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:101b14fdf55162a0e161648fc28f83834cc5de8c0469c0ad48e86de8c10366c6 -size 28458 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 30679a378ca1..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f242f0fc3d93a29a09ac49c257a3a33e348969d64581ba1133b3d045ef0a5ed7 -size 22545 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index 0ec443cd4ab6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d0859aad6dda646ad42399e4c47aabb9e586b7323cbd649dfe3e5f88acd53ed3 -size 25115 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst deleted file mode 100644 index c364e0464225..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f2f53d645becbc9b59bb0d9ea30f939cb1aad72c72d33e9a6c31f5538657a14e -size 21792 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6ff8db91ac58..b4f7579c0ebc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4fd8c09f4a7602695329555921d5887b173aba696f8670dd91dfa223b6ef8fa1 -size 22419 +oid sha256:3ca34e3b7f772eb65667a04106c133f2287e23a95091c9e9da2bf5f9f4e076b6 +size 22290 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 53c309919177..e33d8a97fd5f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0c3793de0f049b8cc3d85b1fe085f681336dbe1a893af9cdd8b9703c8357d3fa -size 19228 +oid sha256:3dd37caed0f7fd6a9e0dc6403f424baebe8ed3ceb459fc27ee288e00dace75e3 +size 19125 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index de68396b06b7..2c87fde95642 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1f9167d2b5f2fcb79a524ed4c47f0833b576d621ef49f11ff31240c3e00208bd -size 23150 +oid sha256:1ebf924173b03b6d0722bc168ed53ac25d1a5b5edae3f279edb8f49cc2ee333c +size 22991 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e1b7dd94ca00..06cd89062b23 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca269c43d69d438d9d89c08771995b8591e478f493ddf322218463924280ecc3 -size 20160 +oid sha256:125532ac187df2f74be74d4312bfe1fbb57b87107a2c7d91db5c6d82521e71c7 +size 20063 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 503a0814dec7..6ca0cbc2218d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8c8b66b0c1eb1dfb30b2d2c6cff36a75c144c7759746f63ff88690b401132838 -size 23597 +oid sha256:a85790c9b72f3555389d4c5dff5794bb660dd6db266e53e7f7cc6cb69e5f2799 +size 23539 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index d78c7a2d4a96..cd60f51ae571 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1f22028e1bb8486b7a5485f433fff499363edc23e33000ec1b1a3714b23dc86 -size 20290 +oid sha256:d0a205486191e4ba351aacd5eb3188a58162bd6b5e926fed511b3caad5bee706 +size 20295 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 6d6eac8e570c..6832cec5f5cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee6dc42da04342d150363c010479ee716fa70df3b574b192c862993158e0a40d -size 24340 +oid sha256:dcbb1c231b4fea3ca5b7294a70da3bafe1558ed50209088a85b20d7535f3c2cb +size 24285 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7a61affc4167..36c5c5576cec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:05abd3eef059a9c7cc205feb49859706eb6da7f21be34c140a2c64bbb0d29168 -size 21449 +oid sha256:4dff7ab004b6cd66c869978a7e3ab3ed1348dd398a86d2c805ac4349292f25b6 +size 21373 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index abada7623e47..fedea4439e59 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:559af55d716891ec4a37a1ff3e70dbeed621e600ae91867aa4f86067455e7c52 -size 23673 +oid sha256:c47de7151fd778413d753c8130589f4234e250469cc076945099769ccaf12dea +size 23539 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 7c4665f95073..b903f303916d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b1644c271239b0472bdae59337ee593cf2b84feaa64e363462a37fc02766cea -size 20487 +oid sha256:71e3aeeea89cadbd10eaee1d11e475863850b7333a1fa0a2a6d4e246fe42edfe +size 20415 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0864f518763f..8aea67bdf465 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e0e990ad45d6133824f1ed150e7d6b7a5bd0e7c4492fdbca801b5ca8e2080f0e -size 24475 +oid sha256:b86cbe047693b0704ee76e481587ecaf3fd9b3c389223942e203ce44f1eba010 +size 24344 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index df0b3b663097..ebdb2342931a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9eab1a5c39c13f50078b030be58aaeff649cd046cbc6d7cd73656681281f242 -size 21655 +oid sha256:ed0d967767a87645e5302bbe8208414ccd78a3516f335fb845cb9c269cabbe4d +size 21611 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 609cbd05966f..3c4b8b24adc8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ae22a876dfb095784a596ecf8b36fa9e7e07bcd8e943b818d3c017e7c0657b7 -size 25460 +oid sha256:beb4a260436e469ee44d21a737e8b03d12dfaf41e6377167fdc39c9c39a23031 +size 25234 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 396702af4a33..a5494008847b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:452c13d384e079ac27ded491afb7a4a5466b7950182fa73f201cce41ba4d7d1a -size 22634 +oid sha256:f624c5786bc733f1297cc9958d7d21d97f9c11fac0e7b07fbd85cc8db857a704 +size 22579 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 825b8ae6cdaa..c35334af256e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0cc561e0079ba19601594de2bfa00f3b654c3b1ca9c633ecc9b79099fa81e698 -size 26339 +oid sha256:6ee4a25a066d8fedb11517d2330106b58715db2eb78a6c276dbdae092beb5f11 +size 26172 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 40561861cb78..56cc7fdf28af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1351cd5258aee9d204245b413dbc4140f9b572df0bba9cc36344e23c89df07ec -size 23933 +oid sha256:17fe83f1d06daf2672193254b41e0ab1b43256d3c016a915e10ee828768aff36 +size 23957 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index c01f4b73b1a0..4b8c541d73c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f5f1a4e3d7ccc75d11b4642d545586fca0e81bc2e5d504bb25408fb4d2b599c4 -size 26685 +oid sha256:9d596a22515b57b65ff8773743c63604a64c06bbdbb1702cb5234ff4e229721a +size 26515 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 973912eb6328..fc6de6bcdead 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:69f25b83e14203b2e58b16e807cae580877b15f207b329236c9fa5e1afdc3efc -size 24068 +oid sha256:369fc0c78a2b544b162aef557c2da54d328995a31089b57e540c65dad36c0bad +size 23962 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 572e4094dfd5..5dde3e694a67 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e45539c533a75df585dd406f2395b5cc041fa4b20cd27463c8e95024bb82091 -size 27617 +oid sha256:95b1942ce305a609f34ef03f4e8138311efecbdc17a4a9387ec30fe8da6c20a1 +size 27458 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ec3d06cb8b73..1e673e622a88 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e55a98035076be4e7517dd25ab03c4f042eaae16e9501d266bb5de5e92383f28 -size 25306 +oid sha256:a6c339d6545d0c7324c297c3e324ad58b937cea86177a916efff1e4ccb82eda9 +size 25206 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index bb0fdce81649..acb42bfee581 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5e85fb5109133d92b3be963f1dbc2d4877b4c1bfbd23bc912d91124ab42a059f -size 27045 +oid sha256:3a94ce05e09d5093f50061cc4663769aeab697c078bacd2d3f11049a03ac12f6 +size 26983 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index ab8e0a6b797f..d8a9d1dc4e03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3c6734b8ba8d73960fd4c7239ed14371b21143cf74b1604372542875385b54b6 -size 24269 +oid sha256:20548596825612fd6f58429ec966acb4d52d6cd5a0ddd32210f47c8c248e80e3 +size 24098 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index bf0f690429da..6c8db91484f0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac544251d022ee2beff5f2fee581a3754497309f1fa1e75b0dc8df8aa91c9bd4 -size 28045 +oid sha256:99d967ae79b6932fa50ed4c3fd594189b984ac0bf25c2d91434be2eaa1f0fb49 +size 27939 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index adf192aa22de..c1a9436228ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:20429f1f12d138445d3d934db9e418a7dbd00c2cd4b94701dd2a90767d728f4c -size 25474 +oid sha256:d4b52d5c29e13d7e44c0bcd10ca765afcc14695f1ceddea9ea093ceff99048db +size 25326 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 06b3eddebe35..c51d9ecf5543 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a46c0f3846b3b4b526f732c7ef02ecaff237dfac9ea62f7ef924d48dd5ff6ea3 -size 28894 +oid sha256:fa87f7588b0f0673741f249545f56f7a5d24ce9e521ba6dd1d80c0c90cc289ee +size 28777 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index caf05a4f50c5..6be407a6b593 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:347eb54d93c2accc63a73e274c7ad9bd72e9cbc617d85bf3357d5531b7bf8f23 -size 26260 +oid sha256:4699628a411a0c7c53c9536c8546e9abc8c0a0ba7d102e0f1f8fd6667c759941 +size 26241 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 531d7c52ec5f..03c524f433d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6fc287beca16b7d04b742a977a92c4dae7de5f69952026a444af7725e37e9eae -size 29951 +oid sha256:4a3f54f3d76270cf81f7816d232d2cbe4f69da727f1173a8f513de7843b35cde +size 29806 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 03333efa564b..917e36626046 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7d5cbb4a28fc58e503e84d6d4a1b54d7100930577b1e1b6f223cde28a1fc902e -size 27568 +oid sha256:1b60c89bd8486c4d1188e07c86976f33c417121ad252546f680fe690de8f3376 +size 27533 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5e87afd9997c..5fd976da072d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d3556128cf4a6e48bef5c5293ac377bfcd092d601e34677c4dd5a26e44581e3 -size 23451 +oid sha256:0d304b95bf41bf254a406c17140a4342498af81534500f7a37ed5df3bb5fe9c0 +size 23304 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 185387d7e587..531c517099cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2966cf1a8abc47eb6a0ba416d5252b8d4de9a0887fef78e308195256c7455a15 -size 20334 +oid sha256:c66e3d88213b1c7170626b53292741d0665fc3f77ca5f211f573539480526c52 +size 20307 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 8140ec718e73..28bbea0c9852 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1d3e7e5e37a08cf6974e36c875d33454c38d17d11c0eab84eab7b67c84b8026d -size 24478 +oid sha256:8ee499470bc8976d2c77f8f8f93d35cacb1291f2e9003be7a6ec6853e2847f95 +size 24308 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 6a1c8a4d1987..b516e765751b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7dbea23fbcfa61bc07969250a0faa66916f40cba7c0e41fe584b87da95e8ae60 -size 21456 +oid sha256:b1e34cec8d15a404fad068b56e4a493633f2425a3d01c047935340e65a644297 +size 21416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1bfcd82e977d..8455b174a69c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5de3c6d2e820dcdb976b7fd7d63bbef77db271272b5481a394daca8ad912ab72 -size 23652 +oid sha256:274bc5ba897cd9c39160e7af90b849998f2dd647ed4dabcdc5349c41dea8105d +size 23530 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 570324ccd268..64f94e1a8db2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4fcd9dbfb5f28d92207866a88ddfce87f4ccba50d5fcfe26dc20a61a17d1ead -size 20491 +oid sha256:96a59c8fbe34b2ca3df7d5eaf82b18439081005c628f77ea49690c2f99b256c8 +size 20421 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f83296e3f895..b8f4feb353b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f40b7341c670d5b8d597af74a690ba973c5402be84938b13235d396088c3f527 -size 24483 +oid sha256:b4e232d53aae7d80e10050bd10e483b82c131b9c5105325c9b013e0c0ade114d +size 24356 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c5684d580b6c..2a92ba1e1c81 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8917c849f9f46ddc1ac173ffbd03645d90686597931ce8cd976ced12f1f01970 -size 21667 +oid sha256:27532d8e1e2e7cb07b74263a14b46d402c6c61428ae1d8b4ebc34460338fd2bc +size 21618 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0bfb0bd233eb..20ad532becaf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e6eaea509ecf6af94e6ea8b791c150e32607eae36a5825d3a4a8f3231245df34 -size 21615 +oid sha256:c10e9941f39afd900c63fea07dfe7b36c1d1712818834d17641672bff125b1ff +size 21501 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 766d2c270baf..e666c650ee1e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88e71a0ed00c772ad092b9b9ac7f553b5036dcd18c9d221c835120f478743059 -size 18856 +oid sha256:04b1e0075dd0c2e0137cbd31cb242ac9d4f1550d619f1e143a89b2f07cd567cf +size 18726 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c9ef0e163b62..e59aae0d4456 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f065567abc9760ffb0f9c96b8b6cc93917174b0b627fa812fae413de70982f8 -size 22447 +oid sha256:016c136660c67af7e5fc22f66d10c1de77c7d2f6b4dd095e5f4f3ce76a676b9a +size 22389 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index dfdac2b691db..50f695734ecd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c7da15f851f8385350cf9e497248e74b4ad0b18acf5519a6e44817769b122ee -size 19450 +oid sha256:2d3f55afee66fd12de2efd1ecca64b6ec556e621639c480a8566403f3f8cbc55 +size 19380 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5b8e0cdf09e3..b351d2786b7c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d159d4107bb94d3944bc6a778ca86b99a844a1aa7fee68776e22da0e7bef303 -size 21931 +oid sha256:5bd776c55c8b3693c50ecf3b58c96a46e0c327d115e876301c7c8ca41785dd22 +size 21799 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 251a167bc168..002418e64500 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a35b2492c339942e3785620ebe1101e1a6035e61ef8a439d67b5b15eddab882 -size 19430 +oid sha256:5232331499679042e67d585d54c72dcb8c27d1dfe3a383db488af860fd791c15 +size 19303 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 63c2e991086c..683954cb7a81 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2029e43359c5c6f3b74c3530dfb410b9fa072568a506f9e7da7360e0a7611e71 -size 22433 +oid sha256:5ee393196a14400f65daa815195835d3cd8db841f91b9ebd8af2c4fe6d52997a +size 22296 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5e46bc8dd1af..ff4fd962b64f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f99508841e29761ae8af975f7f08ac6310328af4d7fd8100928ec47faf5929b9 -size 19965 +oid sha256:e70830cd23cecbca7e4c2f7ef845f76f7e17647261d88296fcb327bf070a4d67 +size 19809 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 2ead8a236dda..46b790273d0e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d797747abb50d811f5e15dd5372d4caee995d9cc910116ffa5829bd3b6e9dc27 -size 23856 +oid sha256:7470b1ab46f54dc08a9b730acc07747845b01614e86dff50ed800af40171d808 +size 23591 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1e4fe7a7490a..e63f53ff2fff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e6c97984c0c17e12368dbb81d0d9fdbe52c431c5c57c39bfc917dfd279c036e -size 20999 +oid sha256:9ada0931a1d7697738059ba7e9fcef6a7249ccaa1fdee32cffbda14a1e0d3414 +size 20801 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 80024d56dc1a..6a7db5ee64b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d3f64d9e5ffa1de1bd57609ff3afee6b51d6dd9175cbe52238c83856b02133e9 -size 24459 +oid sha256:4e0c9eb9f62f9f1b305be7c81c1ebb81adb69360c0dd3c12e1a7f88d06f29151 +size 24401 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index fc5a8b7f2cdb..e84081cd4b03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a91bc9a76eeb3f6e9d9a01ec3f9d62ccb01f87c09efd1401bdccd174b12038f6 -size 21413 +oid sha256:6f58f6e9de3558bd02e88e79c46e71b1478b25f3e090305a519038b98bcb7c1e +size 21363 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index f218c7a1d2b5..552543e697af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1e2db575ca928e1d280dd48b1509a4da32779983a6c8f970d19b11f6e158ea3 -size 25703 +oid sha256:78ebebec117ea52bcec5d72de896d62da816d3749332d6ff53e5d55ea056894c +size 25570 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 198c388c44c9..fb0dd293ca03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1b6bedf4dc091a9123b71c4fdaaba5bc427dd6c2ea73109b392fa4248ca17d21 -size 22871 +oid sha256:ff2b3d16a2a545c9a047c4d1f0be9de88f81227534d3f40f1c5e05046ce057d5 +size 22755 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b64398722157..4af4bf88a9e6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c18c4f2b85ce0ec224920398385ab8db010c7ce9d6c3ccdff4f0bd0001e692b5 -size 26418 +oid sha256:2e996b507bb036d7cfdefa027f59fe5c7443c830dd8579b7ec3438025380e59c +size 26404 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4d5b130f0209..44857132e18b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1fd446c2258c2496b124c0cc9696c353cb2df3ec042b674d1a2fbedc82f49070 +oid sha256:1dd388d3d3cb70720d3b3371931d449344749c58c4f3602e4e6d430ceb9077da size 23589 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ab16a81d5daa..e887fb96c31f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2db47c3d5b141054c849757c1de8843aaa499d7a217be1c7fe9f7b988db3e3c7 -size 25650 +oid sha256:9cb1c33d2b738066e3bbf659e8d3404b398c40f1fb80f2a9bb227f04b576a441 +size 25584 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 0bedb49e09b2..d70c0b28e6e5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f0e2462b2c4f795af334584b6c6bc60890da946153fc6180f30a1598f336619f -size 23231 +oid sha256:f8decd5bcb8cb33d3636b9cbe0371da04bcadf5c6c2f814cfd051d401abbbd70 +size 23118 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4a35911c15fa..9d65e19fe090 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2da9cfc7fffd080a4e149fe8c19fe2d63d550261937322e45eb60420c79be8b4 -size 26362 +oid sha256:16e47286e26d679ac622f312acaea22087f5addc95a884634d071bffb2a9bfcd +size 26210 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 92b72859c7c1..414876f5e87c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8cad9ce7438ccf165a50a2f3abeac1b391f8c6909482f3ac0f0184662d912c78 -size 23920 +oid sha256:6319853294255a5940b9d722c86b81e6c422d98aae605d7d74757e70d08eebc6 +size 23791 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6b556636ee90..3c15608f9c63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:483c9b00425616ab3398332149b0f9a14debe5e16be592513023ad03bd0a3eae -size 28334 +oid sha256:49ef742d16467e5c693532a027c3fcf582e278341226c8e716c592ace68dde36 +size 28138 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d753f250994f..6df258add161 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:241d19f882d24a0d0bb810d859fd60ffbdfb2d1fc52eb2af22cae5aab8c014c9 -size 25395 +oid sha256:91b9f0a7effa302dcb21e3513b8fc80d1b77f7a47515cb9c7ac718185730ae23 +size 25222 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7340d1ea9ba9..df3390f4813e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:19d1dbc5462a74498ba99e9025847ebe7c58288e1771dc573900c187210c1a0b -size 28990 +oid sha256:e81442ef012dce73df603523d9b2c9702101bdc41e74a21fc4c504a1fa046118 +size 29004 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 38a6a7090ac7..a94c9d5663e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ffb863b27c51a96c269276a3379c03e67deb14a9b1f39af42e7de6471de67d0 -size 25813 +oid sha256:020e3d10e69ccdf34fb60452de987777af449b45588923b92c9f2cb7d7e1dbe1 +size 25779 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index da68f3c14530..6849379267b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b3695602544ec464d87a77f663835e496e78b17bcf1fea75c9d5b796460787f -size 21549 +oid sha256:440f3013bebb1df95df75f1163d8d257d375c4e1ef13d8ebb4f2fe7c345d3399 +size 21426 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index caea1d249c0b..69c1fde54220 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08d4915268eb444209577c66f8c059a4ca0e6517b53ec04e94a6a6b4f13d1ec2 -size 18818 +oid sha256:78c8ff21224cc833fdd4de7209217f1a2c41bbb4b2974c6f171c5e6cfad19712 +size 18670 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 24cadc3aa20a..4c0e29b3fc53 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:723526fa3ce0fc81742c11c95440befb3aff572101e0b587d7916c3d297c75ce -size 22514 +oid sha256:e925aa6e820c320116aac697a6d2a2cf5df828bac766a35ae2e952972cbf94a9 +size 22394 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index fa12c7ba4f69..6b059e3c2d08 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c6f1a9dc55be99f90b20ea3c881bf76f911cb22654c14b3b105883fc0aee4bed -size 19393 +oid sha256:877f0def2f05679394cc2a1150d766aa069c9694b81d2dcc29671cc5fdb2c730 +size 19336 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 14f1f0d2bff9..eb06946a0225 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0c6364e66820d8edb7dafc0f7c4f3c554c8194c3ca920630df9907a6a62eb339 -size 21905 +oid sha256:a8471de0a41e1be25c52ed6b67bbe12a7ed26950b4134a7e7e6a4972457981df +size 21778 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 12c49689b826..e6806b13ae62 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0636235b78bb154525ac58447f32936ead2dfdd0693795f85f900a089a63bffe -size 19413 +oid sha256:f23e5065bb63723f81a395a5979a73107eb9d537b222c4249562864fac73b30d +size 19297 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 51469d44399e..871648561074 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1baaa25a9dc6e3cb479039c50153b67827a0799a7b63cc4d40766100b08a0e37 -size 22444 +oid sha256:78b35c651b67a00eeda850b218e4769454ffc870e7951bf92ffd4187de4d117a +size 22304 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2d62c695b7bb..413d9ebff53f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ef2161941fd8c4f9560d0ff137ccf3810b098618f19dc1fce03bdd84b68ddbb -size 19965 +oid sha256:c078c0be0ec8685f478b756233c59344e57176987f0d4981a3aa25dd7cc14c94 +size 19812 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8ee7cb06b628..d3e3a3b7490f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:473f57b4fbf9e30dc35c83f09c40a6f12ea6275ef45e2002f54f4ff00e3cd8c4 -size 22944 +oid sha256:48b12cba05bfdfaea8f780cce451182aef4f0d74f76d3e680a6d348a0c7c53c7 +size 22809 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index bc7116ae39a4..f636de02116e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb79d88a394ba4b7acef7f414db4b89c629cd059ac6a7d1fd92c76e59ae2e1fc -size 20464 +oid sha256:e370255cb7d05b1007953505dc48df1cc095f057ecda7a59bf8917ee5c946c27 +size 20310 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 94d7df7f54ec..91b6870302ac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8556a28d9335ce06370e167247111c78b4415a57e01a6398cd04f3f78d3d5e3a -size 23496 +oid sha256:e91eb538af60ce6d717a737e61e13812845df0a6638fe5c63cca14ec02b10d30 +size 23445 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f4717942e57d..550303bf9a6f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:457dfa0c83df3ceb59d749c56526f4c7a642fceda9c093bba956a2eb8bd9d5bb -size 20947 +oid sha256:79beaf4e37c27f1b5932de8fe09cd920473a3e2d5394ef023627a70d02d32143 +size 20910 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 63ceb3069248..1d0cbfd57a3f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4bd0b09c7c8719f4994402e89904732761529edafd91788f4a620ca4877c463 -size 22844 +oid sha256:79bfbd5b89350234b555aba65f65bf85b46e6ec6cdd1761c8a804be9630ceb2f +size 22693 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 5e6bfd5034f2..725ac7c72189 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f64b3f772417dda5a5d9e582de553d952ded077dcb3836a04e8af47b272b5d71 -size 20883 +oid sha256:f0a39b69f07da05703c3a238f205fe43966928c0eb2eb76ddf578e4de55c747d +size 20762 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 06ecfb60725f..bc633d6ed98a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0624574a21104616b4fe4ec52f0b9c902a78af37ef5b0be1bec69199106d8e29 -size 23505 +oid sha256:ae680c6f05db5f98a8a71f14e55aedcf5f24719aa3fa138f17bc281bbf69f221 +size 23353 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 232ec51b1fc6..09b1aede57cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10945298a4a62a0cd382f5a84e3f02fb593e268cc3c17155b0cefb22265955e0 -size 21151 +oid sha256:cd372b00b1398dd79f79cda3f00ce00f06936c81aae422bbe86bbd94cf2d5358 +size 21071 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index c33b745907d2..d24d6343542f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1858ed5b5caf25dfde18ea9fb023d81b342815ce6089035ce142708854972e0e -size 27755 +oid sha256:c8669fc5edb4424de8b1a136095c7eb0336b7a091a498a9cecbcb301919b25c6 +size 27591 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 98bc0ee6eb17..d3ebe2ec39ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d5fdf5804ec96ad5589818b84b5b9bf7a3cfb92ebc99303b738392d65e01d40a -size 25122 +oid sha256:9c045c660e780049ffb1f3346727a0f703df3dbccf7993de5a528ef3f708d330 +size 24924 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index bd6df60fc258..f5d4ffe87d71 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c2c463d1d5c60adccd58ed98004549a3373f2f261edc75dea75961aefb1ff597 -size 28444 +oid sha256:462357e4fca3fad8ddad1ed14c529276b1eb71f8172d3cd18090a000625ad1b8 +size 28375 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ec8adf3f9686..635af039c423 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a6562c2b02437a4f465678a069dc345bb9151871dd36932db82511022c272df4 -size 25841 +oid sha256:8497be290b0fa048f81105345f10bb7cee66f2b04df6ac05586233c561a16d15 +size 25824 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index d2d9968803c3..f4cf6e05ce65 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87abcc43e8fcadc2106ec4f9a28b2eca205ca9e8819dcfb4a0fe92d2a83b9cf4 -size 27728 +oid sha256:53e301c3ae1cbc5ca8188d76e73bc7fbf4a49b09dec7071f59e9ed7d0e19cc46 +size 27568 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 0f4249b68e65..d4cd10f40b69 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8861413e6063a78539bea1263ce9dbce47527fb071cf4306123f03f047caa613 -size 25226 +oid sha256:152b6248b39306ca5a768576424ef36277f3ea92838f22a0072117149d300db6 +size 25034 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 213c3ddfa2f5..b7ccf28fe157 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e1db55fad7775ff31e01cfaa284bf5cd243bdde056f5d95743db6e5295f4e61 -size 28417 +oid sha256:ce54dfaf0891c9699ac8b3d6db4064820e11653199986b955c6bc17c30d2f08d +size 28269 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index bb3b9e8604a2..fb963484404a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:23393a173af76c1e31ef35f05d88a09629b34aef97822e5e8772ae82228f8580 -size 26117 +oid sha256:3d44c7606d9d9b955a6b0d6b92ef833a79537ad7bddebb29316604dd17714142 +size 25969 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ba368370571e..135e9f827890 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:22f47585ccedf8943aee2035f98a65e574ba4fd496a785507e659671759a9366 -size 22568 +oid sha256:dd5fe086d4cc9aab9b43e2706cb2ed62b317eb75c0220dcd115df54c82e089bf +size 22504 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c3e7f9a1a411..87f96d6d4082 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ac1600646dab784e02806cde472b0b5abbdd0c6a3727e9dfe204ede9453acfb -size 19518 +oid sha256:0b2937c73f1c25a97951e9f5d74472f1e2525692fb568a11c3f8a5dcad3cf6c1 +size 19494 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3090f7fb9700..aebe49fa8a0b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e4db3b879a0be61e0fa23ac26fe4a37f604f60e7534483f36330b1f3d4f7cdc -size 23460 +oid sha256:8aa09b0e42c037253d716632aac37eae0667a899aa8a364e071a5de2be21bf17 +size 23450 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1da9abfb0094..0abeb3f0582f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:276b61064e20eceeafdc56339756aa3423da400c87a72bb2ea927dab38a49804 -size 20556 +oid sha256:8714aeabf346405e0080eab5a9826881587d6cdc8f3e9ac88815325f079815e4 +size 20539 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 72032c628fb0..464e578887c6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bdf463ed29771f34231e22ae260f2102b888d2dc08b3e66f548cd344df612da3 -size 22342 +oid sha256:3e776e19cac66f69b258db1b31f2673c6b95a189e56a8ea276c8d64eac162a8e +size 22317 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 8843e8e040b4..8b7922623304 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:39c9c1399ece53d016f158dfa3f6c032d77d03610b0f261005ca01a9734d3c77 -size 19431 +oid sha256:ea428bb88bf2824225a141e2779d69b8b5ae534b54de6deae6b00808e259afb8 +size 19418 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index be74ad5e284f..dd8eb1989d91 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a5ac1083f3fa3984f778efc3188376f4b81c85194b0679a64c13c5164a82ebbd -size 23633 +oid sha256:e6cca1d8a05899c9c7c15ee501f1701cb3c1d410e8175596511fd581fe926003 +size 23582 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 557a1a5858fd..bef48d9f910d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:06a582f9c1a08d51dca85c1989c0db411164077fde5bd0e5117b5ddeb3544d28 -size 20401 +oid sha256:10d082d2cefcfa3a1acd4cc4582b9359cd6671461b6d5e0dbb39669c316e097c +size 20410 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0be79edbd8cc..74e120a7e45b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e0d73c0cdf0105f8f924a531a2542a4a28f32a43de377f732624478f95f930a -size 24600 +oid sha256:432735adcf73bc3fe7d12c272bd4c07a7f3ce6b13ba78f13888a8b4b14c26087 +size 24563 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 17d7bf2ac493..ab127076039b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7139d075ce833006516e8484d1662b6ed2976cac0def73a74feddbaafad4f8ed -size 21745 +oid sha256:455ac60b7d09b1d67ec68551f201eec7e73e3b9ae7321ef0752e97e00f25755a +size 21748 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index a3acf2a4c64e..4a60d42567ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6c38d8001c77a41f038bc3ba358cb1b610f8cb723c9fc58864f82ddf9da013d8 -size 25674 +oid sha256:d33a916f800575b8b59bbe8eb65d638771359c3c74a694d249f8401476aa4125 +size 25625 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 792daa60e4e8..7db0f8a49ba8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f09ae4774cfc2a232439ff72876c58807dc61f98adab11651fa12a3b5dd572b2 -size 22734 +oid sha256:63f33920495473b8d5be4693d3b9060ba7aa93a61c3a5ecb1cad4c11e76664e1 +size 22728 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index dc56487989b7..511f4492cc93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1f2313005f0c79902d5b459b58f5f7f9086ad640e68e685b9f4bbd63f6411c52 -size 25623 +oid sha256:88e67e979305a1f6678f7651a913d26dd0156574fb5ef808e697521bf35c991e +size 25579 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index e82a6240f6c2..a30262233154 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50a9e02c0a658f5c441a7a2a469c230f2fc469f61db169d8f92e1710ce47c298 -size 23331 +oid sha256:8b0ee4ff263499c59fd185a0492cebc1ac5f5fb365a81f697b368e9bc80abb59 +size 23353 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 15e90b043a62..0bb91ab81ae8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cdeb38e8a9ffe46bf3e4dd43fe73a96152665c49e9d289ec50ba0f8433848875 -size 26683 +oid sha256:e941598c3c423447aa513408b58de0cbd06432c6ee45562faaddc38dce039ed0 +size 26665 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c8ace32eb3a6..2700948f6547 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4540326fc2117d6aa5e8550a2b911a4c2c3ad8a866b3cf45ff002653a15dec43 -size 24375 +oid sha256:39951cd084a73be4baa02cb369979cf75cbbd9d1c864a74c213383a74a96877c +size 24333 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 225b85e62356..3a991c4233dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2503352b93ba2afa0c78230fc90c43297e5e327262eca1af9dd4afaef5852750 -size 25561 +oid sha256:49acbe470915adaedec3b7a05d2be4597bb272393bdeab5ddc8907d366a7c053 +size 25523 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 15b8e2365d25..8bdf4a919684 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d9e50167227b642ca31e01c384371a61cbfafda98226149feaece1ad07b09350 -size 23250 +oid sha256:61545d416c222f421ed1f1474c904c5a81ca808e2e53919a3e102204f2b32125 +size 23203 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 287d63f06503..c4c611447da2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4a6d9e027c3bb6ebbaba105c02354ce139e20d66a105c185223e9195cfddf593 -size 26878 +oid sha256:0422fc9f4f23ed0b46aeb0bdc5ff8014defefe3ff181a6f069475ef81ddd6927 +size 26814 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 67a0760b50ae..b52fcab531b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6c27473ee56663df670ca004a0577634c0e13e0516ae94e05b1e8ea3d4e3726 -size 24233 +oid sha256:e87a96c59cc66faf07775aa2df777d33bf6aeec230f6777f7a14b7be6e51be5a +size 24239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3d4b3fdbbd37..5f98d6481ef8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:163d1e34b28b1c6ef7474c8ba995bee534393533f93bb243eb4fed0cbd3d3a53 -size 27905 +oid sha256:2ac2ecb4d1b4c67b0887ac4f4ca4c0300cab2b6c6ce993f82640221987db12ed +size 27838 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 26db206b190f..eee2eca82b74 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79bfa773e5cf5e63f6522d78c05267d9dbe7f7e80150f0aa793d609c1bad62f1 -size 25523 +oid sha256:956d0e40c96c98daa20fc6e23365b86afccd659b014c38ef334d2e7d64012eb7 +size 25506 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c474066871fc..a47aa9707bb8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:322fc62a0693dba6c7cfae76528d6e7a6336465cd9925df853c95f245abe0edf -size 28707 +oid sha256:5eb61e70ab8e1e359f5c8728c9d2f0fe7dde821acc073a1f00f3303710c54b4d +size 28572 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 8fa1af4e8183..1569cf096e12 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b06721c5944034c0a2583a1854378762aea2be90174337cef2bad8e899f2eedb -size 26658 +oid sha256:9cc32410ebaf52649368a35256ba14b232c05714321a903f95463aa71e3c528b +size 26643 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index e086a161c92d..22ef8aef624e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a19e36dc775a693aca2d6d052591f0ab1cfee2f28eba4bde0e9e04071bbff19 -size 23253 +oid sha256:956b1c802259f63d7480558007c42eff65ebdbd6ee1656725bcd217dd36e3d58 +size 23127 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 0a8b2c99fa53..5c1c14698893 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc014c29f11821da4830b356929938689fc63a6e7cbd940d38c2bb079a9ad14d -size 20128 +oid sha256:5063178eb89f38fd068f7cd673a35c3df16c6ccd21cc13fab31b0cd55bbbcf60 +size 20097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 07d95a81b746..3baa295b0841 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e4ee34690843f17c878db83b11c865644b925aecb66849173995abf6ffb8bb05 -size 24190 +oid sha256:7b13831ee84512804eb5eb4c4f9c04c78858ba36a23e5ae1315a9a930cf84a03 +size 24184 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4e460b19763b..093ceb74cc5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42986e5c3018e63652c04ceadf960b1d502e43587d92f1181197bfef89bc47e1 -size 21228 +oid sha256:35078705b94614d3cbb1a3b323764b3de28e6d0172fb4bba72a5a460be615062 +size 21135 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1a2f054c4c50..e12ca674d059 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7cf3ab9c21e449a7b04203b0d591dd1bb03c6ce22dc43a208f703fe1590a4c53 -size 23074 +oid sha256:dc20bfe5857d3094a9e7aab6ac79ccc4d1dfb2c617a206c63d111e89ce49a479 +size 22995 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 55b178667cca..4a97c783e0f4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c95e2a2a924151aa597a9a30fd0e01a1854906381789e3afec361a73d5e7c7a5 -size 20071 +oid sha256:8fedb1b4b6b4b13d154a457d94ac9de10bba30e5e64a735ae1d2c60d17675297 +size 20098 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index fc04e7cffb7a..a0147404fdb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0117ad4e19d4823042b5bbce3334ba206e20c51640d9a4a146f00b5b45792a34 -size 24196 +oid sha256:dd75324e909ddd9213c1554d95e5f805ac6c4ea3246667a82b14da0c8fe6b528 +size 24180 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b0f2ac1a96a1..57daff38af36 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79ae0c93b466839972f132ed3fb8b7beabdaca3afce49f55b5dbc4a67ae4ebaf -size 20993 +oid sha256:8fa75fea180b59a0ae0878762db722eea169f2de36662d8d77870274448eada5 +size 21027 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index c65687a8195f..f2cd2930b0e6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e56c4d98642ed622ac7c6a303e7db3a3cb281b22f24b28d69acd759f68e0ff66 -size 25317 +oid sha256:9c939fad1f6754a5ed04a4bd0436d2a122452dd53964af852691dbaf6175b765 +size 25141 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 3bb78c57160c..40c609d26c82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28de834d1bc87ecc46930966f56f4e51c3637dc6de72a3e561b23ec46264073c -size 22248 +oid sha256:f7e980551b836fc5ff6b3bccf76b041347c470b299f3577f41f4ac3e24730d1e +size 22260 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1cdbaffe2149..3eb0e3862150 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec4637eba779710d96b8b26d4acb345275948272d01e2f79d4303bbc3c64f68b -size 26231 +oid sha256:db9e556c4c79e695ce50824424d615f1c7e6c8c1925fc3f3e98105f7360dc9e1 +size 26247 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index dd00de1999e7..eb71e0b3aaf7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ffd14d29a2f9cc5bdb9a9c61427b7cf3b44794434b049ff37c76f17f1b907b0 -size 23508 +oid sha256:2a006647915d6bcf7c8ac3f0a862d712249652dc0a3b0fc1ed72ad0da024d89f +size 23515 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3f8f656f1f6e..0e96193ddf90 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:665a1633d0ac77d9147eafca801c032136d2b22ee8964805beff479a3c35194e -size 26460 +oid sha256:a5d210bb9ee942323783fee2aacc14a21497d993410d96657e11a964f71346dc +size 26413 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 8906c58536e4..f032b7d1099f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:78bcac7a5e996525a26e660b0801189a8a17bddeb621b73f829728c33e3784c0 -size 24206 +oid sha256:cef4248c6971f0532a851db5ef6ff1dce01fb2aa06f0c2f0d54fc235cae5895d +size 24207 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c110a6ce0984..f311e03e43c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb047a19555d0884057bb421b62785326f76e77a275a50688f49459adc520870 -size 27505 +oid sha256:b1eed1b6838b1301e2a85dd81e4d4619318378acefa49e09d089e74905198d34 +size 27513 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ef45504ed008..f1f2cff9a298 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:edcb3399430e2aba658abc7389c9d29b70604946955cbaef147b9668fd0a7ba8 -size 25344 +oid sha256:19c84ef1dd8c221eb8fd88e346ed83c1385a87da83cfce843bd0c81fc66683c2 +size 25288 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index c9954a0f89e4..493983eb9e5b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ef3e732f7d26a3e8673cf2b1231d58be1faca17705f52565439b2a17a31eacf -size 26635 +oid sha256:dce6d030d30811467cd82b33d26cd7fbab165553a2b0583f897f87c447e17c7e +size 26599 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d19a5ddbd156..8fa2253c0503 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87f8b62fbc1a9a67fc05e3209cf5d1198ba035b54c070d4157534244b016ef39 -size 24088 +oid sha256:593fc831d65e607142e2ca74d0660ac8982b9f3810b11ce99062f924ec7f216d +size 24079 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 56ab7ee6a32a..ceb500c29834 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83a10bd1099b804cbfd4f9e6eee6b6f36ec8b8b5606f1e19e2168084b31bad98 -size 27599 +oid sha256:ded7ed17a681ff607182650374f150db47167da7bb28290be530f67af625a5e9 +size 27552 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 124b6910c31d..6dd3b02de62c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0c6710c0e760073f06dccedce87c15af945b807fd5a0c9ac52cb4b00469a205e -size 25151 +oid sha256:994eb764fd329da859139156be3df32866ce728a49fff4b13eddcafd739d2ed4 +size 25140 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index c14ca1b5c3f1..88573a9e0b95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4522e180bca5081050f1d06a9e38b512ae32d763fc99713adc4097ff2fb8fe95 -size 28711 +oid sha256:ff20ef92f76df8fcf097945379a1636f12b6ce72f2850a6e7b840c48a7ab7239 +size 28640 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index e1fb4405a5ec..f5c0a76f5dc6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cb855d4d90aacecd2fe0c454359493e646d46c942fcf1b9a2e530e39cc6c4dba -size 26359 +oid sha256:a5940fde1a6249fd6060fa22b5911a29fc593cc2dd70e0f48c0f364e1b621df2 +size 26332 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 51345ceeb305..9ed7acc233ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:887142cdb820ba9fb0776eee2e206925a1bbcea7c012f1ce84d25623015daf21 -size 29459 +oid sha256:8db7180927ee7fb7cd6775ea88ec99e9af7f17c68198a2a8739aa20e74af1d82 +size 29416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 503edbb5fcb1..12bab5292cb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e5ea361e2eff60e0c722a4bb13c7fbcf8500128c7519dfe151690645dd6360b5 -size 27521 +oid sha256:9082937b7ba27cb200fb41247e526ba1515facad4da839b84bc591f85acee228 +size 27531 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 9f4b86c4f331..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ef9f4cf8c3805955612a2b5735ca75869475ce286e519de0652a73796cd56429 -size 24148 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index d13c50a1dacf..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:d5b14288168fee312e598a867dc96079d4c6f17b085af75e8419ce40ac042a48 -size 21022 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 716b34e07bb0..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:91da6fd6c6263ad1c4291ff212bf6c0e24adc96f572264199589432d9474b93b -size 24775 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 65cfd107de89..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6f00b359b4e12ab2355faf61350104dbd0eabf789a7b8473d6526d959db01b4c -size 22230 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 7065ff3f1500..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:416f3e5c23011eeb7369e3300b73712afd731f137e5466d74f2f2d915de9eca9 -size 24187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 0fceaf2eee1b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ca813905b87e9c12735138d2fb99775f3161a265d381a346acfa70e34f05b3d2 -size 20830 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 3e2c1b88768a..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:11ba91d95d815ad5c80df1d1dd1e15dc3755f1b44610e8feda9cd1a11dd0204a -size 25190 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 255a4b215b72..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:24f886713a3a81f4ee37899c2127a77aac0b2aea188b2c4ae0f1acb0567688b9 -size 22168 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 9e012db2d35c..adae7cddcd63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f513a93435891d551f2d92a02af23e62beab0e29784b5ab8c97ebea802cb9f1 -size 25141 +oid sha256:8b3b225ed9d9b9a701c85b6dbe74deb8510e660096f1bafe893262237e40869a +size 25017 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 3c8d21939e6f..0b98b5cbe916 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b3ba26695c1aacd6f71cbdf53d7598cdcbe914c859be64a2d90ec5d85c6c54d -size 21931 +oid sha256:55fd29b8ae1df32932a9c4761f282875e56082977e8e017149ea742965b5504a +size 21815 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 14964d824854..0cc724a57126 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c8ad52731541f73e8098702329e2da6f019481f766786c1acc4fc214c90c4cae -size 26049 +oid sha256:a109f510fcf429056c9aedd269ba0e61c9b9cf51b3632626ee4147f1818999cf +size 25861 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4996ff3a8825..20dd0c25db2f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb87e8220f97acc8f599fecb15bc36443a38f3a681b39d5a163a25dff26a3a2d -size 22695 +oid sha256:6749e5d2327e4d1710b4b0c0f398f52fefef544f53bc6622d646009b83bae4fc +size 22707 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3c6f544950b9..82af327d7722 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3cdff0212f5f6f28e7481365dfcd4e060e3f348d75dc103be865c7a7104d7cdd -size 25332 +oid sha256:e6d26982f17eb8db8d61d8ddb9f69b622024e93a635379464f6a0cc7f9f7f144 +size 25186 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index f41ccdeb4aa8..a5a978eec05b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5a9d3c9d2b21558078cca6362c7a53400804c4e59a834466eb0d36f5418fc5f9 -size 22409 +oid sha256:fc77359f12cace93b99ea1ad88fb9341f2683b773adf3332eb21a69b4330adaa +size 22272 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 6d6ef2968f1a..b614eec2c381 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:064aab262b985df1d3ab1a1c90ee498eb5bf75b2bff5a409d6abcc1124a5fd9f -size 26210 +oid sha256:1be3f98803387e4b48060f1c49d7ead38afa0a05bfcab325cef2985128ed3729 +size 26060 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 722e864f3b9f..8e2430d7a4a8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9fbb4e9d36a7bd5df0c90c6c16a0b9d6c9d64f1a4756dcc71b81fe6b9253eda -size 23487 +oid sha256:3ec2a691e58b115efe52b0dca1055024fda191bf056985273666b675fd5f4a64 +size 23455 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index b483fb16b350..4c8807212903 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b2d6a9c2f1e2bf44ff79cb7510ddf0d5b825a78bf7e2ef64efbc71fe39880def -size 21610 +oid sha256:d83b4c58ae0d2afc29b2d51e07f562e76235e1b8c0e63f41eaf9eabae0c6212e +size 21437 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 0930fed3ccc5..6d32091bcf6c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54fc3177edbc1140545c85ed92ea08bde885731c752c879f6ff8241aa6817422 -size 19297 +oid sha256:7a932e03021faf24775c30a229acb941ee35670dbeb71c3d088485a529c90c7a +size 19234 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 57aef9c41a1c..d6f0592fc7be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6c93055a8236579eabf76feefa2762a1f85264e4eeca089540dd1ce6c479345 -size 22826 +oid sha256:1f6921063b310da9baaf927c1456660e941f22fe368a13c68c71218f0704c1b7 +size 22792 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index d84ddef213f0..26d7020cf947 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e673bd41e756d01168d0a9aa247083d6fb20974bc693ed171c22535b75ab447a -size 20201 +oid sha256:d2b78119a0f52bf96619950c95b488d903f3673a1abe018a026ffacc56549eb4 +size 20184 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 152b1b20b64a..a9218c0cb18a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:251d75f34ef4598aca7227ce16dda4607709cc78a5625a5dffbac527e1cead5a -size 21825 +oid sha256:7997e95fa52cf78e3bce6b8c7bc92e55f21d18b6e81ad7644ac6ba5a5bcdb375 +size 21682 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index b39494a2e53e..dafab500722e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:52cd8b818ef8ffbebd08f603689d45e9fdb9b04df3f8e04123129d18b0685c50 -size 30034 +oid sha256:8518103e99fd03be291830447898c36bd90d9a63059ee858c84438c433a5f9cb +size 29974 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index c3bc72052753..6b66615dc759 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:34c0be37f116160b345064492c0edadc98e86a851790ddbe9957f7cb05d928a9 -size 22966 +oid sha256:a82d1b068cae4e71bccfc9862cd73180e790c9f06111e7fd4fec687fa4be435d +size 22942 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index b5afe55a3f8e..179ae53e7e28 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f21ee45218504074443fdb51f6299f282d871f4eb53153e99dc5a7206383f578 -size 39122 +oid sha256:a85e41e64a98c93bd5791689c4a6e7abe9203649e284e2e4a8b28a42e3a999f9 +size 31059 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 6fd733067f5a..d0678b91cdcd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a5ff1c8b177813c84cbaa109ebd4d24baf02f87169f772ec7f1181224668e999 -size 30858 +oid sha256:8841011023d1b16c50118f52583e1da643bbfafca879c27b9cf58ad2d65db1f1 +size 24201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 29f9b98ec79c..2b3036b3cdbc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:60cc38a98edd1ea83dd7d47174e36aaa2949df19f25b42f2f6ef9b147e93b130 -size 30871 +oid sha256:18c42941ed81a19f412f3bc7941e559351f69472679c730be917af21cc63fa05 +size 30851 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index e99b9cb3503a..7581eca7219a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c6dcebf7ee9b72a7058fee5a080cce4dcb170696298f4dbad850ff2f17532057 -size 23627 +oid sha256:d5206fe4b50922f5516626eb51e235788e156b5633fc8cef8f7b79b693cd904f +size 23506 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index f5fae9744d48..c8111a671245 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:45b2a40aa609ba9dd860c3edd60ec5992915592df5e2abfc5e50f61452694641 -size 19211 +oid sha256:6cdae0cbd449fc3820b08a60b13fd93ced15d68cd3a5523da4faac7469aeb15c +size 19156 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 8d4dbc9a30dd..31561b765c23 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:401a20cfee2d59d1457c71e1014edfc80296b7c3c50634d94cea4eafecc9e003 -size 23157 +oid sha256:b737faf6107d2691dbe4f91663a3789b9b6e2825a035cd2b85efcfe7cc89ebcb +size 23030 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 4f2281b623a9..2b9c7653d6a4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:107d341d6eaa487aae9e56c7a170f6d1da16fbed6b2d83f6a776f3a59ed74fb1 -size 30629 +oid sha256:75b2b7c21aaa43e44c4baef98332a50fe60bc79a176db376c8f068dba3cefba7 +size 30542 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 6f0fb778e7b7..4b6242a7b996 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:95c388bd72d4303c8b05da299f77fc139ccc422b58b8a48c88f67703c50d1bf7 -size 24028 +oid sha256:6642e894630634abb351a744038752f4b225dae2b8e8cfb2f7ebeac4f9ed393b +size 23880 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index db4d6957c839..1fa4ff252ef8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8613e1f0956a7ec0554876fdab23f65864ed780c309785c7faf9e07ef8baaa85 -size 40027 +oid sha256:7021194dda3ff3f58eec9893a7a37067dd3cedc505bd34c436eaca9f726ec196 +size 32307 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index bc1c112bb0ad..3266be255f54 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a9dde8866c9289f309e3197c5bc44395fc6919f93e9d49633e4f237dd7a5ce1a -size 31933 +oid sha256:515aa8470b7774f61dcd48fd91852848f8bd58ab9aa697a14011c3b80c0d83a5 +size 25028 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 28d5cdeb457b..f5a9e2827b08 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2efd1cd805655537473e711dab62667a3cba742733d431056c92409ccae065da -size 31897 +oid sha256:414e3ae8a9d9dd55f56e6625d93e0a5236d8679eb5fa4541e6e60d73367eb7fe +size 31947 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 89b423e69e48..dfdf01ca987e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28576724d85a86aef6eb6a1c2b2773df96b7634a204ef12f4a51a65fb4c038fd -size 24479 +oid sha256:d9ac5ce8187bfabfcf8984acfbdf2846dc763cc350d6d5cb677f5666dd8f6f35 +size 24391 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 80ace958344f..cd6ac37610d0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:119e416018f1fac5999063d8fea64794aa37dd0d09671ef27bbfdab8f6c8917e -size 20321 +oid sha256:3cf5b55e0d6910e5ae7cae439efa8743e5c349b16200fc49d6f4abfa67d21674 +size 20291 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5f39a80006a7..aab456b4f17c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d521dea94539cf4ed81c8ec680ba39d5251ae7813d15398509993c8c28eefd9e -size 23616 +oid sha256:529146e18dbe7c821f6b7a0f0611d544180b9e73cb90c4d491904907a17ff82b +size 23406 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 586be5245d3c..9cb5775003d3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f4fd775d3743774c872addd86ce6689c992ef185c432d8003bf98898b87fd72 -size 21318 +oid sha256:3c1b4d4b4a5cf6e6906506127c3aa5d6d962293be1a73c380c4fb9d24406b074 +size 21306 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 533aea4fe8cf..055fceb95149 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59d89f356ab6c458f213133fd36041591520bb63a028b437c04aac6a4e0fe88f -size 24717 +oid sha256:60600ae31b1bc6c4cafa66e0ecc0ee740ac66b3731e2f955c403f7fc3cae6d34 +size 24554 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 01bea6b8f518..80a1afe82e1c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:db1dd298eb77bb12d98151916140a994783f44a6ec4a25cc4e92ba8463914f48 -size 22276 +oid sha256:ca475075c642b4d067616cb5b4ce735547a49968141d307c71aa3306a476eef3 +size 22278 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 120502b00d5f..773ec97e9be7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:164ef97379d60d4fc79f39c8615848a79c6d08c598a5fe1e8dd7bdb194886ccc -size 24778 +oid sha256:06804c0b1a3f9e0d265daecf58b695ddc3dffe730dacb469f0c232311e400cdf +size 24647 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 78ea34f565f2..3731225b4dba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8c568afc543f070f40558b39a1a2eed94a94c2eb419ce4858c7e680af533715 -size 22788 +oid sha256:278ad73b0000256898590ff9db1271e9e5463f3a708b6d6c831c8364019fa969 +size 22645 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index dc83974880d1..0f5d3e28e1b5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0bf02cfc14d994a941f5bd312556efef6a65cfc91890dcd3517604f68476f71e -size 25854 +oid sha256:7c406eb15aeb71ee6c030a42bf79d2cc392209edfa36058181f0d93abd22c6d2 +size 25813 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1069a402c7ad..a3c451845903 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:15942c07a7dd115210734bbebe1dc35fab05a038e348b698c6ca061996f85fb9 -size 23548 +oid sha256:1da25f3a2d24047cb8f11c7fb4886342e0c714d5f51caf32ec303964f8bbc212 +size 23445 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 77449405671e..eb3732b27ec2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:76840cad58bcf918a1028869d2616ba306a03c7d5f7ad613545adfada0070e6e -size 24831 +oid sha256:2c37a317d3472778e76f71d8fe5d9e3c920ce2b84f93d69e4df47155d3cb31e6 +size 24710 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 06dbb36e2d46..4e29ef2b46b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a06f0a655b7d734fdc374cb499e7899ff3a5eae1b9376adbe523c63568fc2340 -size 22861 +oid sha256:143af325c49505d894ca60a2f06525b40f6ab141cf7611752abf98c1dd02a32f +size 22790 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ad65c372e10e..49a8606626c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f249271d91bfc4f6ca4f04d2fc999aa3251f5e026aa2176e6fbc905fa1aaf19e -size 26548 +oid sha256:61375e9b8388f503d8d4ab50f3ef9391eac0ea708f94eaf687de7dd1911d12c4 +size 26423 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7414b14471b5..9c315bd7d5ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:186863f67b0ee84e02684b581f98dbe2c9e69ade131160ee67caa4a40f281e1f -size 23962 +oid sha256:0cd52665b75bf9a6218a6bdd608e8541f3112dfcbd0cb53aed2cdf5796b90223 +size 23874 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 35aed7171e1f..642f7a79a979 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7154202f5ba07ac84ea814982ec1dd7618e702159ee72b48622e56c440379e6d -size 27031 +oid sha256:ae5a36c78678d09c550e9a4ce1fc9ab768895cce918da79a27a4dbc19fb12d7e +size 26839 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 9241a20ed2a9..cd3f89148c17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa032d832fa3bf0c75bdf700037752aa48c8811b08607644e7293e0a65864876 -size 24817 +oid sha256:0a90992cf5df51b6aad1ac5a101cd83ca918325ac97449d127d1b7ce602ce733 +size 24806 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 32a0094404d4..6cb582892039 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:14fb773c215833596431620a7601eb37d5fadf8199c5ff76b7dd73ab8ccdef45 -size 27895 +oid sha256:08a9459eefef03ebe0c30b6ff5c9f182da7adfa6eaa4ca1f5cc4e6c88571dd2c +size 27879 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c04bf2b1f833..c0a98ccf03f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e5694047271ed0a8693df4542837ceeefbf4c0d0c413367af0b54502d25ee0c9 -size 25844 +oid sha256:1b756ac9d3ed127a0efc9b34d156a432e3040f647213a33a7db84f225470b23a +size 25820 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3aee7eb84f1a..32aa241576e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:73b70b9730d646754ea94c56d20fa3909bc549ac2f17d84b552a3235f93b5a52 -size 21647 +oid sha256:e61baa793f72527cec7ab5400beac6cec47e14008a32a4e3c375865add711e08 +size 21489 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index f107330d8286..67d64b5f4806 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37143ab7246653326919467206fb10e49ef2ac807051ce30d3969781a67d2ed9 -size 19265 +oid sha256:ee581ee6eee30df9ec691ca82182e7672b2cf44b0f05af1381fbd9ae14d4bf8c +size 19236 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d2348a5c1b55..83d1e19c7508 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79205e2dcc64c39a4e8f2b739a774fdca3f2210f8bdcd15771123624105c27f1 -size 22855 +oid sha256:78cdfdbf1e8be4062965270af93ee928f6ec1381e4055b7ccfe667b660996e65 +size 22720 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 84c3a5d275b1..22bc91fbef19 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:30f4673d25220452898915a401f2dba2e51680974ad0baaabdf858689dbe2d8c -size 20201 +oid sha256:4b5e8d0497cb2b587c68d9687cb18f39d0a7b946beaf396179d9ce01975497e1 +size 20187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 55b21005b801..9d9040288f96 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:df2893a75079d90ed470c116d0834c75d9d99e17926f9fc0473f3f53c921505a -size 21822 +oid sha256:f91134038395fec192cb9f8e7be22bce41cd76c6389e82283686418f1b95fee9 +size 21678 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index c2ff81f2946c..adfea200ddff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f4926895305683ec6249fafd822c1a2a3511a2aff99e52a9a9ef3588585ee54b -size 30036 +oid sha256:20d4fe2ac6e043dac9a791bbb5e70c2b0521e1cbb475a342e586ae0a64766ddb +size 29978 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 2f16335b3bea..d3077ae5cfc8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c2507f9998833bef3263ee32e02bb12c97a3c59d25278edd9978891a828191e -size 22981 +oid sha256:d3b18b1b772143f7a7201182168dc942c5d705307ce7467d6534e2b3addd6a0b +size 22954 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index a067df360632..ba8414b7dec2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1b28b6f70e8365f32108a5152319d39bac53d77201139e45a59e9744c5e2a3ad -size 39121 +oid sha256:e3247c49ae98ac0acb36bc533a13922e6eceba7d93ac42c1eb36a36b9b89cbf0 +size 31053 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 9b2775be5c6d..bc360a0d74cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:897cfeb5192fa4443becb00b10a955c89e952c44dfa9042d319c155c45467dd4 -size 30860 +oid sha256:8c6d029d9c035e50e897108e8f6a44f5f2fa8fde5ac663e2e60881884d9127ce +size 24217 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index ca371cdeccca..14ef2c2776ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54e9382e65b47cab8e144b694811b50e021f85f2d2c46dc7b34514dc1e1f883c -size 30853 +oid sha256:5243f6b0f1622721857eaba1b5a99f1322dcf8286ce76dddf59076e4cbe2672b +size 30835 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index a93704172753..007c7c46e5fe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ce20849a03eb2d3f764c2583b2ed96933b53ae4cd5604a50f4735df31afe7f9 -size 23637 +oid sha256:5f6a37eeffbfdf42c4493752e1850f4f58838901c6e5c4163d6f95eff44ef4b8 +size 23519 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 5eb6a4219410..c585adca1472 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:596e656c9e610ade1b2a90ef8d649421b4d576ad904997fb6c48b1bad51871ad -size 19216 +oid sha256:60e8729bb43db7c95905a8255f45ba415fcf9deb30c17f4563c648592ccca55d +size 19175 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 94553decf3c3..7cd64c99e251 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a45f2fe730e558ed1ab4e2ac3d211d9d713bc66eb525f99d35a496eaadb58e25 -size 23165 +oid sha256:4b979d0e6930436b5b0055b78443edbebf24634c34998581b3a5d9d94239e1d5 +size 23053 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 45f4389b6dd0..e1bab117878f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a72c4559dbdcc37b004d6d977e0d2df4042aa16cf8c4ea1c1d40f30611927d2e -size 30622 +oid sha256:b33b91f1cee264961403427dc1434e15745028aec1ab770647c56bc16eb0df40 +size 30551 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index fc05a60f2e9b..f7dae9c3bbaf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7a2c7ba810c5a17495c8fe727a7461d77950971df1f34dc23821655a20fe9992 -size 24025 +oid sha256:e4a9362bbf70c150ca7539e015b39182562b5d7a8575fe03400038e6b6f340c9 +size 23863 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index b50de3f7d7d2..b611c825b6be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7d314c8c9e6bf87d4c1705f971c69768bfd0e2d4a1359b802576eb1b3a4647e9 -size 40046 +oid sha256:55f130913d0bd0a38cb965bad471427da2adcb0197b204d1a9fecca702f2ae46 +size 32326 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 44fb232dced8..1fa7889c2762 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79fe94f66230fce7d59edbd872882beb5baf74ef4f38840a7cb707ac846c906e -size 31933 +oid sha256:1ce577940574375d1e0c4b449bb71d37d5e4b1795f18b205b730b017bcc14ac6 +size 25016 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index d2de752a5a1d..4c7b0c4e03bf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ed1ee88c761cb2a2a8e087a70f8902113c94bc5acc0281be3dcd52449276dc05 -size 31906 +oid sha256:7854568e5439326774bf03f0159631ccb829cf73695ea20881e36407b31b2bca +size 31955 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 1d33073baea1..aec5bc12d4e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dda192fb54979c58192c0d0813eb772f29056806d5d8728f7e20c866346063c4 -size 24486 +oid sha256:6e2479c6f614c55c3f4ab4d3f21ad698590760dfb19721afd2a19464af527562 +size 24393 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 016395719109..bf2c3d4746ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5db01826cc1f21a161eb8c95dde7b28d3f0c5da03766b877a258c118876f5175 -size 20331 +oid sha256:b7681b660f05bc382efd3dc4527414af6d0f25ec0f8ddf1eb614ee238f0b599f +size 20306 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 877fc124ec55..21997ea39e48 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6209c9287106d3a9ecf4675b0b65adcb8237edaf1490f3f0a97168f1dae6156a -size 21246 +oid sha256:7d05242577993f2e14a77a6b5a8a64071718269d8c75037951f39338209bcbcb +size 21111 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index f52d51ff94fe..39014cc3c474 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d21af67e5d048242e5ed1459ad87e42e9231db0af5138cf90d70cfed910af5d7 -size 18584 +oid sha256:c006dcc007eb0c92fdd3cf4102ceee8c95fa74aa8e7e68bf6a98892766d0ec26 +size 18462 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4c39f03199a1..f2d942a4acb8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36455d53a0c9427d92dcce9a45c5a01e77a4c21b89a935e9dcf7f9a70cc506e5 -size 21800 +oid sha256:5e5b905c92d1b965ed4ac8cbe13f00bf8526aa0746514f9972996b3b2d1a6a02 +size 21785 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 8966ff472195..82d0d32dbf89 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e808af6803a5c57cf71cc4b3e2308c524f806793cf02b0f431681e696054f680 -size 19075 +oid sha256:4579b2ac9447b1370f8781ca3e30df2504216f31976a0962cf5cb7661a704e7c +size 18960 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6a1b998711f1..3fa3dda1c798 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d0202f994d69ba68bb4fc8b4da7d2091f0037763cae98e4b73817c2de846e877 -size 21466 +oid sha256:d8c0b749fe5acfcf632756cdf53841e0c5540dcee8a585965ec8c1643f08a856 +size 21319 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index de39f64c6096..aad7c44f455b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:24cbdc92f2c71e65382d921c32f591d8dc7387e4046425334f2e51e287255081 -size 18331 +oid sha256:fa1116707bee160939024a5f29d6df7336e919fe12350b27edbf2054f9a7eee9 +size 18200 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e95d4c17acae..cf802d3e9ff5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a52dc90dc0ad6daee37faa0212bbf05abad42eb15c9930087f748e2ef7fb599c -size 22030 +oid sha256:82fabbfc164b2d8d37b5705326cbbbc09cb6d67f0f1d2443a4b49eb050a9c500 +size 22028 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 52ebc55b4a04..7e8b68d4df38 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8360e2efb70faccee310be0a03a8081fdc2098ce711725f6131ec3b36e42d76d -size 18984 +oid sha256:cb80c6811899d5fba3000fc5d32c1422c54ce071fcbf0ba9c9233fcbc5a25dd2 +size 18876 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0f1f3acb9d9a..236bb97e680b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37c42153af66346da4d077103058459777a9ec9baa37e40bbdba61c55ea4a3e2 -size 23325 +oid sha256:805da4cf6ecf829147bb8dd53820d55993bb636c60a4973e66bdbe548259f5bb +size 23095 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index bc7771b6f170..03fae9bed378 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d2e438fa260b326a62cc4b2e3f9265f9d9ecc004d3044aa549949d39cb7f329f -size 20678 +oid sha256:212a922f3ff81ed876c4fee4076ec330d8610ec3512f563553fa5aa9127359ed +size 20448 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 04ecdb5626d9..bfa96e344b06 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dddcfb2316eed7242aa684da9d4505ccc6c6722ecc60d51f880f597bb8268497 -size 23853 +oid sha256:99c1cfe79dadfd34cc4494e34e03f782fa2b63a7edbef956bed85babdce73144 +size 23833 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 52e833a55e90..a7665d20ad6f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5e0e4121fe401c0b026b23c92c23542b7e9bb7a2de7b2dfd1ace85bf42a3f50d -size 21202 +oid sha256:74ccfc8b6e9476b027597c23f63cbb5d01a590a490199ac16ac3d6beeb519b6c +size 20978 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5d0d7c30f5d9..4792ec06efbc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:648f8a50d849ce0f3e75c8166cfc327454b89b787be95fa12b350ad2bb1c15f6 -size 25015 +oid sha256:2fe9fa6605cdb9bda6d6e67f7b66ce00e8fbab0645b7d621ce89fddcddb950dd +size 24832 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index da0a47a830fe..c3e7cadda4b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d89137d792681351870f8b2f044be426955b62071378a4d4f0c2131c4a00fc39 -size 22326 +oid sha256:f782daf31dd5d81259b7f90092f6143c2db571905d2f731c444749d5c77155c1 +size 22175 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f74da71037a4..1b2baa9110a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4619273908081087c30f46f8e8afb92e7f2ca8829806df2a1952c9a66823ccba -size 25931 +oid sha256:1225c6d238e7d52014c09a8d08a01011a617d3d542f03affbb1d7750702f50ac +size 25806 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 16691465fbde..78a0c9206156 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e5636e4784e314293b3877259092b1b20d7d4eae66047ded681bee461617bd63 -size 22889 +oid sha256:4db18ad912f3a8f71ca0a01a4b2e726651ff1287187c902b13040ca7f55e2081 +size 22727 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b87ea761dc60..f94aca1cc603 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:63743d1413498aeb5a4472d8d791db29b6f849b3b96f3c75fac5088dda10f544 -size 24927 +oid sha256:012420597f462ed3c68924f9ff9fa8988ed35a7180ddae1cfcd157c204cdf64c +size 24808 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 5ace9dda3b6d..2553cc617b1f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa34513767abb93bcdd88aa0262829ffca71747f5ae6d941b73c2c2c738e8ee4 -size 22217 +oid sha256:ef02b0fe2192cda77fc8dead08a47d0412bf147abb6f90e10d8fa72ef47b3e23 +size 22053 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index cfac8dfa13e3..ffff9b147c1f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8141715e93b4c508aa90094e6f4ea4c25057d7b3ac2b358e0c75cde4f541c7e -size 25838 +oid sha256:55b7265855c148e94889701b8b2dc7123c9021add29597401d41740052a967ce +size 25757 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2c83e3a6ae52..09009ee3fe15 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:222e09e10c8da72c3de076fe617344e66692d729c76ac0a32c50a7816d52d5ab -size 22703 +oid sha256:37a3cd6c71457412d2a595f160b35ab64c3137300abb1659be64e4fa4279cdba +size 22615 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 09ac61e0dcf9..3920992f5160 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7957e950b81f5bdcfe62ca1ff54e1339d42fcbf79950acffac4c8f8c604d4220 -size 27352 +oid sha256:06daf56023bb93e35cd53a8d47fbcc2fa207543ff024b1909f7009a2283528af +size 27140 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index ce74767a53f2..7ad91f52684a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:445bad6e40d358faebfe96b544c44223f4ddd148f0aeb4d92ad66ec3053c1f72 -size 24914 +oid sha256:ae63fca4863ca17b75524003744aadc335b1bd6a6c889cb806f400cee261046a +size 24647 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index d73e20e284b2..d1e7e1a27702 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0017dd3bb56a706c38b7a01dbd9407dc259697e590a567dc0b6449857ed5687 -size 28251 +oid sha256:59b3a470bce1be3e602fbaa40400ae5212eb308fea36b17dc368804de9091179 +size 28055 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index fb4118f487eb..f1786b0d6ddb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e55228d706e4d85298a17fa745e4f8d6707b434798fe464803e9d14c2808bc10 -size 25349 +oid sha256:1f64c7e321fb05652169e4148bbc529be180c7d4bba7da1468170a9ef9e94c87 +size 25111 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4ade36b2f961..551eb2c65f90 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2c20e06e87e46e6112c1594f6f33218f1a50d360c5e8dbe9e06c20c2cec4aa46 -size 21177 +oid sha256:229291511e4f06b30a1a30a9bee1d8435f3f142cb209ccc080ff52ba3cd32e4f +size 21053 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 709dcb1419fd..54c5faf6d416 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b57755ef6b74581a12c4658a358edce719301e7a3e36864e312212e5478b5d2 -size 18509 +oid sha256:7b42676b05d79ee14f41092c4ad6422ae3a341cbc78643ae911c3e64d7461836 +size 18408 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 58970c07cf00..7a348fce08af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b58942109ad460bacf870d05d555622f95d01c250160bad749daa555a54157a -size 21849 +oid sha256:ed495cde0d71054e1e8589e29c61fd49d54b871abdbd3b1cf6396075652e873f +size 21801 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 431fe2ce779e..fca32bb7722d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aad49c6aefb0a194921c4584e6aa3c7146651d6c8af0db11eb80eeb0f76bca1f -size 19130 +oid sha256:94e134bee2053c595e7b5507616e0cdc08054aeba970c338806da8fced2a0b7d +size 19082 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6466ba496d78..5a4d34ea4a64 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb25a522ad478aeae2b8635f5a81d71749a63df0c3463df76aaa2434b9189bce -size 21459 +oid sha256:7ec823f3e9dbc620ebba63afbace5bffd09108faf819f0a2fa6280f46bf60501 +size 21319 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 8ab76919c82c..7284aea35fc7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5cdcef0139286c21a8e00f6b8faff7f2e8b1ca04219bc4320f7bbc43e23b84d3 -size 18318 +oid sha256:4c17c02bacfc51ba687e9ecb0f3cc735d0c20132d4ad380c388fca65a8e628c1 +size 18201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index cd2235197720..30fb347d6374 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ad75a4ffc89c39fe400146abdd49fed279c8b4527350962201e0335d3e55f975 -size 22033 +oid sha256:c5df5e4417199adf0f2cf89a09d082c2fecbe524eb517517d3742735106eee8e +size 22037 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 561067414227..157d508105ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:030c21eac556e9298b74c8ef4ee015f589ba1653053ae3687a9c9800c4f41f83 -size 19005 +oid sha256:86da0ff78ec3935cbc21a445cb18ccf531df38d8ad7f9f3bf67bf340a60c949d +size 18888 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 2e9abf58a1b3..bc5b4ef8fbf6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d615ee7f1d9c7a705c890ae84a97653f6880f79a551a3c3a288a1c10a5f8dbdb -size 22273 +oid sha256:7eb1b046a5a7b674ecc785542d5af46c97277822647611d501a35ab569ccac3d +size 22158 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index f86d7e5aa2e8..7dbde089d64c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:476cda3286222843dac8e0e88949c8c58868d821e0b6bca3c570b5cbfa71cf40 -size 19388 +oid sha256:d7b8929ff559c873ee31eefb8c4f35a770c70352c918aac1307402809b529fa5 +size 19261 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0077b8449d38..793db567b721 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fe56dfdd0e17995c4f92d99ac6723e9e344450b83ef2e1d8d0a24d3a75219c0c -size 22947 +oid sha256:cf32098a5534a69ae8890f2a507bbe30dfb75094f595db5db1fbed0de7cf0939 +size 22831 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5862f75123e2..397cddb83f4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38da81d51f6e30e82a9bc90ed8dd13e77e9bdd25347fa34bc90ae7bd5733b565 -size 19903 +oid sha256:bf7a198d45ec8613a720eb7fb6461847e5daa8bfda1de23b45ce9533c8f55ac4 +size 19757 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index d99f8e8004e9..62428a1e33d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cead1fd80a4362e7f6be320b7f5ca0d699ea3a574f8e8b899689e2accecb71e4 -size 22546 +oid sha256:e2428f9218a077ad002e65153bf3578755291c5e9f0f76a9a35a9a674caaa0d3 +size 22366 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index ddcb8e84859f..79c24fe40a51 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4faf85d7eb1fb3ab39604ad671158bdb9ff0f2069347647c26b466189ea12ecb -size 19937 +oid sha256:872a5466af814d5bd48c885199ec6dd0211f3c6fa546c12539302d7ecd06b73d +size 19834 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 68d9b2db37ba..ec6edc10ba16 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59483186ab00be653b019574689522a9a8b5d7d9af52c46117a4d88e7d256f2d -size 23044 +oid sha256:7931c9a729ef392825afd18353af3fdf4973f76479206c0c2fb41b910673b799 +size 22957 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index df527154cec8..d597fd87930f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10aca4cae70f424204d26365d7d213d6f56739690b92838200aa39f7db842f2f -size 20519 +oid sha256:763f2e88dda0896fb909ece60cb87ccf1d98191058129f050e9f0c796aa2381f +size 20388 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5279273a3be0..95ac700a9bef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e3196e37e313d60dc5e0db9cab666b1198cc570e590e5457d385a8ba7ed6cde0 -size 26567 +oid sha256:9d98e34e8f8d766f1ecc3fa5af644d24f942742cb164278d87b55861a2670c7f +size 26388 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d07385ae7b8b..d434150f3f02 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:04f23d5dcc28abb1459392949f69826048bff00370dffbcd282df2ced9199069 -size 23806 +oid sha256:59883ebf1dfacb74a7e121ff067eb1ac877d27a2c8cfb2f917003869ec358255 +size 23714 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 430682b6e17c..5ec50667cc52 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:092ac7f19c97280e3377073f2af219d527da367f9ee08fa900b6f346f7e5ee52 -size 27445 +oid sha256:7bb5b8897b8cdc186369aeeb94bd2be3a6761827da5b51d74805c649531cdf33 +size 27462 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 02d5915e5323..cd58234de220 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e227e02cd4ed3b1a1f84f87db33ab200097ef13dc5d7d9a9a1e9b8ec103c5ae -size 24358 +oid sha256:2a9b93670662b34f2e07168d3a7e218b1cfe4a14c881c2c818422254e8daabce +size 24235 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e112bc32d689..725736758a15 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc864d6a0a3de2a225ebdfe35e58800a07624fed2a8f7f651da2a8d9584eaaab -size 26651 +oid sha256:7ec7df300b412e6bc6bf38a38b6415577502218a960ca58418cfe8428882c931 +size 26500 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 696aeba2768c..e8ef544d5e06 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:733f5aef04921bf336006c77b6fb011d0232f7786802f2324931db2586b0ca0f -size 24181 +oid sha256:1e1643e865315b3ebaafe73f7b2d3528018cbb2e4495fa97e7104b0e2e462971 +size 24054 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 2df513544527..29e186aec354 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:903b4e2d7f25c6d970a23e2bc2608143352a9069b700e95077a155b998696c37 -size 27309 +oid sha256:b21d6b39ca89b5bb9870e869d036e6d97add39c32d3be5b4fd27823d37bdfb10 +size 27246 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index cff71f5a9eda..abc33526cdb9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:534e3e19dbe181e65f537da1b941197782763792fe23f14399ce888a2176ef79 -size 24945 +oid sha256:ac71cc2ddd49ff0ab589a6b6769aac7789a6f1c7beb68688f2062d5c15786948 +size 24847 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index c8e470674f86..073c82cc9723 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1fdecbb52931b3af51c6bac19f3897c8548b992132d47d0553e4a7927e95b36f -size 20849 +oid sha256:f94435ab96c972cfa9bcadd5fdf86d450023dc2f7a18aa3bced885c02fd6c3dc +size 20721 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index a1d310a27283..44189a275e57 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6eab5addaf6e990ca483cef3476995a44059af5806ac2829e1b50e50a4c4c4e2 -size 18099 +oid sha256:902c8b3e63c25c41ce3cbf91d18d872bbf0a8e7c0a1fc4cfb01f82cdaef432b1 +size 17994 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index afe0d6a7b90a..bbef2f2419c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6c036e2da2833f0cfa298674ef30a52538bf708180043e7e7cf69d586c4905a5 -size 21851 +oid sha256:2e8363cdfcff921451beeff0b770554f713f36c5760dd3a4de733c31ed0c3eea +size 21715 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a2c1af0895a6..b5938168c092 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26fb905fee66a3384ecf1e9305d3048eb8770a8d9e432b07cfc7b46eb2fedf27 -size 19029 +oid sha256:c00a855318d57b900c0e12bc84ef715a55e6e9cec1e66bb5fca8649f2cf3e12e +size 18910 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5cfa33a7235b..ac90e0da8fc6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8d0c9636aafa7a2d50e4b2c8cfcb2f1cb5a3075def98bafffcb95544bb3e0de7 -size 20980 +oid sha256:533b313e23ebe52eb5f407a91c833ed2960ffb58c769e89d4ba86d748cb83821 +size 20820 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index c494de5ce818..56c551db6b17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7271a29431d953046120b32beb570f59cae863e8cb5863c94be8937c55ae17c2 -size 28510 +oid sha256:aac8bdaf095dfa7c5eebe7e997f827e86dabdc16d8b86eff8db35aca09930860 +size 28542 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 7e0f29cdc10e..8bb129af2ea4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:06211260d8373a182f753eab765b431e71da580a44631fc08b6bf1e09870c14c -size 20836 +oid sha256:bd898e76972897923fea1fc3bc115091f8be2cde7d27486fe9c6f0d64dbf1687 +size 20738 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 59cca5653ecd..6aa743b6b50c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:47c9ca36ce5092f693dbae23c3a1488e127e6561d6c2cd38ce45540295f5f345 -size 37051 +oid sha256:d8855d50932f6c5343e55a1d4620c5ece5fdb179b30b724a50d7d862d923c422 +size 31389 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 0c8bb3107beb..4d88f3d63d22 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec1e8e70bd20a12cdac00e6a1d3e09e3df5b310c02bb77627c06892761f937cf -size 28326 +oid sha256:733dce7ae815c615f9b98dd8ba6810145b1b203159b919ea0f186e98ec7261ad +size 22232 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 82aba1f6986f..fdd90b679a4d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3547574f27da20725228469039b792b0cd11152c7f295557632bd87841b1d1ed -size 28956 +oid sha256:d3cd2261b8ca271ea45333e124582fc659a9cbb8f157ed7a9b911512271e9b32 +size 29003 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 762ecedecce6..957269af0b22 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f5bfe8315327192d1bcd3e8160583d0d59e4b53c2d4fde65da4b647635552476 -size 21735 +oid sha256:d13837cf0afc4a62a490f779c03377a8109f78536eaaeaea4777786feb626bc6 +size 21661 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index f31615169664..526c5ad0d01a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f24c4125c29a39d7f1efe9d538872cd6bc3fcfbe8e1615189f76f12fd91dedb1 -size 18052 +oid sha256:c4dc6586037cfc0133fe16e2f799f35f3a8187710ba6d110e4c0928d6d003d1f +size 17938 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1a0976d16c26..0daaf8c90c95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59b78383fa82004dfaa48046d53e6df00e3d79cd9cde419c631a68e07944e785 -size 21879 +oid sha256:80de222fe28bdb9a5744f1290996e24995fd6e9451f851770e76ace1c72aa17e +size 21738 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index e0c8872629de..adf4f4523fa1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00a69fa613b468d5f434c0f67a8bcc9be6f12acccf142b315d2da7c7e2af0ad1 -size 28264 +oid sha256:76b98acd3d821838fa6c8d60f4a8bdb68b65538d58f6818e37144c3db3f7bc57 +size 28185 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 791560ae4fd6..0050d7a875da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:61cf78636b99f13da2584b258d94bc23db2d8fab89dfd3b838bde85c37201ed0 -size 21923 +oid sha256:01a292c40a45bd1d07d7d27650024232c99ac590c9dc64a4ff588b752550646c +size 21850 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 256c48c1842e..905788d45676 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef9e338845333b698f3792c4b0ec9843432478ec6eff97cb8a66122193d4b117 -size 37206 +oid sha256:3daa74a7ca116919ff3141821b974fc3116aa1620b3bfb14467596f8f7751f8d +size 34168 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index ee14bb1dbeb6..b7cb1f16f26e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:04fc68ecf56ab93a857abd6204b01f752032808922cfde9e239d3b8d6c7d2499 -size 30172 +oid sha256:8d8b4f05d1b790fdd6bee6dfa100b865207adb4e5fec1e55cdd149c7bff20db2 +size 23158 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 239d375d44dd..78aa9beebcbe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:625a43b492a6a8afbb04717e2abcef19b253ec0885e319cee40a68fa854c67d6 -size 29659 +oid sha256:3711d1828eb9e960d99d19a30e273f32cc39c355e4ef2e6bc47c5cccb4f633f1 +size 29593 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 9c4464a9c797..eae8799cdb06 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:20c3d55a5cefe0210f404ffad8c16af988a8df7303196dcae5ad08502061deda -size 22803 +oid sha256:7e23ee8beee98388edf79aa4f85a4bbc6e98b4ed769a7d99865fc5c5c1a89f65 +size 22692 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 6c5a12628735..189e63def7f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a897f4675c3b84fba369db20e6357eb32229f78e24129c7c992aab8305b369bd -size 18918 +oid sha256:5394d326dc3224d6f0c8f9a044b42cd2d56d40932ca90aa274f7fc4a7d09d23e +size 18875 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7962105b21ff..453de9410cd1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cee53faaae90a92d1351dc9282864eded11bd8620f52d4a00b2958d30c568275 -size 23071 +oid sha256:31f8dfb39142c9710056778a1b42ccb908e6b08003dcccd47c57546175db99c5 +size 22844 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index a5f02c34d7d8..9273cc378efb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e19794afd10b63881f3795013ce59ab3b22721dd0ec2a69d704aef2500577d5 -size 20215 +oid sha256:1f5e01770ec03f80e27cee630f832aa2e5846b584f68448689895799aec5b110 +size 20049 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 17afeda7c417..c1c5ed3f913d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:929ff6ccd413b0884e64bc368159d71bded77692ec43b9837d89d9e957606e89 -size 24053 +oid sha256:c57dbe53a0121dbab2d9d6f9930369c590980ab92c96515c2cb0127910ab41c6 +size 23854 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ab29d8dcd5ea..7c4811f377a0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a90d191f608335f04a2da702258e5935b04226fe51871cfcfab393652d72ea47 -size 21206 +oid sha256:67557e83c989b13bea89c4029c8bee325e86c81a38bb5a1e6f5a471a2cac0c2e +size 21061 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 9714e574d34d..7f624030ba30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:63c0dc480443ed9d9a18a433f34d1a0f056ad66eb92205ead4edf43b465cef19 -size 23837 +oid sha256:72ea1d6458a04f9bbd5efe1c9d68828949847e72af71fbf5d79dc6c2132454e3 +size 23676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index bec44ec6d7dc..3aa11692e12c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e63865cede62c1d6719aabbf8c19cef8bcf1e60790ebb801320e709241a99a2 -size 21684 +oid sha256:261768d55a9fb4f3cc7511e398b4035644fc8c28b6ff69fd6ce73978d04caa0f +size 21604 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7fa08a5c8b2d..af89ec042ab4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:223087b3effbbea89fa2539e980db966a65f0a12e033e6a4e241fadff3b5e1a4 -size 25045 +oid sha256:2ba31752aee7e221d9124de018ec68fe4fd5ac2c289affbddc40f24cf26d4f47 +size 24871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a49f0356b1a6..77df2ebafe24 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e50a150b08a7edd2b7f4bb45f08edfcf7d326f76c016b50db63de9b38993f6e5 -size 22653 +oid sha256:12f34ac21d272644102ef0da641836a1c094830b6fcf34daf1ba7a9dd7d66dd8 +size 22584 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8ac2d0bd3b62..fee284f7ec4f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:808b5865b27791cb0894631a33734028b26d1e6c95c8ca76314c332911908450 -size 24217 +oid sha256:437ee1124fc6ccf62eb4a92d288ab71d94d6a336fcbf5f49de1448d51c49bb81 +size 24080 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 7ce060de6451..fcbbe6275037 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:576a655b4cf47bc08248442b5b734bd17159df56b4931fcd06e6272306f3a10d -size 21839 +oid sha256:5c82745c8df375bc455d761c4377967a0f9ec799f47e36f0f5a5e8d022bbbe5f +size 21728 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 145b6923a0d3..a345801c899a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ec75f7cfff0664e1974a1931be6f997729b7e769d1df4e55acc82236b32588d -size 25193 +oid sha256:29470ae25d10e6b9197e3b88b30c6e4917f34720eacf10a64b3fc21233c6c1f3 +size 25027 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c253b90b213b..4714c4792af3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f1fb49f873ceb26a0bdccd1f42ac25bd5275a20747db7584e8ab4af79917aef -size 22846 +oid sha256:57cbd2cbaed53a2eb064dc9a2b8ea227471f411507a06d5f328c74f50f35d654 +size 22747 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b05e12e41141..e3031956b641 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81fdc21ea31003166afed437619e73adfaaa2016e2b2ebebfddc626a7662152c -size 26419 +oid sha256:cba5c47a0c1ea5e74c23437bd5430c61f13208899b9a544979d9cd72cd5007f2 +size 26282 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d5644026baf0..267d3d284fab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c18eb510aa9b21d107ff95a5b5f02b1e71419538b552e7d4f9daabf94c6983b8 -size 24392 +oid sha256:e26f77d10ebfa3ab2d897aa0eacf8957f6202f072b93bb11cfcfed343f7d5991 +size 24215 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 8eeebcd82304..3ea9db019609 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e6656e389d173fb4bd2b7008128d75aa902554a2d66583ec8cc313797a89c91f -size 27368 +oid sha256:d1a35b2a003e57bacc7c4f2a30d7989b386f6375d1c5808fd8eaf158c248a94a +size 27151 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f00f7fdabe52..9b57d2568af9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e67e4f761ae4c294d7c92be00388db87817138c7f021a97f2c563ef1f842ea5 -size 25277 +oid sha256:b1bd502c0bd7e38cdb861c0801148ae8a2ea8b333fda4adf70738560857729ca +size 25095 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 96dbc599b2ab..7b0db2dcb75e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fdcb903f33af10d74b7e11010067050718f0240126da1e24d131e914d13d3f07 -size 28501 +oid sha256:8353b96d4928e8e4e2b8742ab0470a9358031533ad8b3f42cf0a170b231cf3a0 +size 28539 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index b3afe72204c9..69d60aff00eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a6381eebbd5837ff28985f90b08003a67d00c74232733e93e2113c7342e8e499 -size 20843 +oid sha256:93d9223b21b08e91d0cc046fd9d8ef37951733c86d343ba61c87a35cd9c57753 +size 20753 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 3d3338e30160..0b60325ce43a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9fafd392d4dd4ac90ca54825f7fcc99c64c71c9dc904a5bf68949e124a754453 -size 37074 +oid sha256:f70295e112f9a442001629e95256358d862141917f961cdfd112ca554c2773f7 +size 31377 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index b0a2bee560b2..499d87765152 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7eb34a49543b084dda0edcd8c7b47f4815c142363255c34ad9cba80766044678 -size 28344 +oid sha256:4db7b8720b492f8280571739c55eecffa5bc6b3894fa34771bce4da26e009a72 +size 22252 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index d545a7857d28..172799ffe4f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2fa5b2ca29b55146d714ecaebadd2491a8a1369441860f5c7b7e51307c011c5f -size 28950 +oid sha256:55c8bbffc2032d5e4a8c5c775ce29cd71b6b201bee8f6c217612901dbee64834 +size 28990 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index ef85d83658e4..debf2aa918a3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ed108b9877070712b6cebcfd7f3cf7ba7183c0efc332aec57c48fc0df4f8f3c -size 21744 +oid sha256:bf8fa317e28304fbf3ec83daed4741a23efe25641f98d455c1ff6ef3c1e0821d +size 21684 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 86ee76cf50be..5264a7810f3a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c779dce5a8e48c62b44a53ae8a11e67bb52a5825c41df1b4ff5713027914f5b -size 28271 +oid sha256:91a1fb8dfbeaee504b7015e83c19f86d2d551afb84f1d80c784c7ae5169d8406 +size 28186 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index d35cf4792b35..f97eaaa33ec2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:69cc8bf0afa614ac415d8c03d9d9420a216b85c5e2b566efbe81507b4dd19602 -size 21927 +oid sha256:0a3581e55e877324f77bf9888a78aae3ff4356166d0d9817b703b690318f6fa6 +size 21844 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 88a1fe4381c4..905a238e92da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:51e26796dfc5d066a0b587ffc18282b40eee60ea0a507af0c823bb43af09ed0b -size 37216 +oid sha256:46b5c60bd5d88795642dfdde4ef67ab5f8524c6b7870ce40eb504929701ffae8 +size 34158 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 342c2723cdeb..285c6d55b768 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fea37e08d4ed3f1c8c9bbfca48f0ced088bff38696cad33c1a4d17a488963079 -size 30168 +oid sha256:7bfcfa1a4dbdfb6dff94ce1143a2259f33cea26ab5dddfe17aab88b0bb74e850 +size 23175 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 18f15e8448ec..7a1493891440 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:610bfdbb1c3c7dce531df661f83ca945d072a579e6d48f78c9acfeab813e3036 -size 29668 +oid sha256:3bbb57eb8b37603b7ef3a3f9cb5b4a6755e2087cc3511920125fde8235cdc2c2 +size 29590 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index e66ab77a222c..ae46063e9fe9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e7e0e3c36a4992823855ea9aa1d875fd85a599a59d89b72c0a97f70dbce0431 -size 22797 +oid sha256:62236237afa66604bb575c24d737408b8315678cf866d63132d92389e9065ab0 +size 22694 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index e90f3f481bf1..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:114b67457e12ec0d360d7aca6f3b371513e6041a4c403ab2ceb4b57bb784f917 -size 21251 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index 504c778892a5..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ea88e99a5b8a63a6075acf165b8ba44d04e55761960f508571ceb4ce063593af -size 18616 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 7ca65baa5756..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6401d895372b3c1c9cb9892ee386a91c84236fc851a4c74ffd9845d58d019052 -size 22337 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index d5a2eb83c80b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:232bd125a6aaf71deed8cbd590868debc4bc5eb61f8344b24a11f0d3e64ab4d6 -size 19512 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 189806f053fe..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4f505601eb2e3a95b231bf77343f9a8f8962cbf595bbd421571051058974f0b8 -size 21643 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index e62c421f71b5..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8d45b760fe561bf933e3141faa0927f18cca16425c2bebc216709b84bb5d24f8 -size 18643 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst deleted file mode 100644 index 51d06b521dec..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:bd3e59fd9e07cd405d9430b3daaf2fa198b9d2fa7b9eeac7135d75b704d34e34 -size 22615 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst deleted file mode 100644 index dcb06e844ac6..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:79538b408e2285ff03b3e2b48847eb9201363f0dce553d99247ed61baf7df970 -size 19587 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 325f17914bce..fad5b5d04d3a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2c314a71e7776e347559232c7c68d6a5ea1102c6dcb78310ad24e929a7fab543 -size 22693 +oid sha256:bff4c0eb67996287a093e63e4b7f5d3b6b9529e7426bf792cbbc47246f3606d5 +size 22553 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 68cddb1ea871..83cc2a325570 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:574a37d4ff69f96a9c8f9fcf7f5d4af378a76d10d67da777c0bb79462f96943e -size 19950 +oid sha256:a540210ed0121cf338f4528058fb543c1451e31f82b8b1691a0586561302c372 +size 19865 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 90dbaeedfedb..caabdd2d7072 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:525f77cf4f1c7eb81205774bd838522749b1e07abf3ba6abb70e5b91f23e3aa3 -size 23903 +oid sha256:cfd7cfa67b96f01f79a172e51c15995df562ee3e85414e30bd9a9b400a32bd9e +size 23767 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 67c2b13c4b7f..32700e142d7f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:866147f5e879794db86e976a627967475800cd6252cc3408265b30d1952c6c32 -size 20766 +oid sha256:6ec56c05c71d99e04f7081ce90115dc202a6d52178a0dfe1decdd5964e14d163 +size 20705 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1146c8efb813..e9a990caf555 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d1680f79639970fd14abf9dded326dbeb682123541bd47c92627d6311d87ce0 -size 22677 +oid sha256:d1b2f2636123e7393c29984c16211da48275f82ea49030baa9ff9253c28c42c7 +size 22544 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 54bae1bd31f9..7dae874a94d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4a28d111af3e3e9eb7019046af61a8e91c4bf91fb655b426132c15e650e79e8a -size 20565 +oid sha256:c7e374eaeb836afb9dd2cd220a3f0e72fd0a9754c9b44a30e10837311ccc095c +size 20445 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0ec7e495c044..b1bc5a6bb9c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62ebc83305e45f61b8bc919dbeada39ca1ae18613ba994c8e8684bd88cae85b1 -size 24102 +oid sha256:317dd82ddd4efb2e2cbd1b71ce079a9f7f9565cd393b93fd1cdc66c1d366cfb9 +size 24011 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5b92c1addb21..b224b0861f8b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:356f19f24207cf5445e5915e962b24b512fcd917b9765c5691dc6d93698bd9eb -size 21412 +oid sha256:a28ff82572ca68ea07edee7bb690db2060544a11a4c2b80abe78811e67f53626 +size 21352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6415428cc61f..3568826a0dcb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca3bc63dacf60615209d32a67a420e6770b44490c760f7e2833af685e29e36cb -size 22898 +oid sha256:81b338c27fc0daf37fef514edd5aac96fb1dc6f494fb46a01a2f6782251e14c4 +size 22779 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index a65967db2258..098dfb9cb9fa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d05467ec8b855c66855e5039adfdf892790186a50fc139c3aa030d0724c1a61e -size 20411 +oid sha256:aeea9e24bd319cbc01bc2d54f91ff383fd65a42c054b0464d94937c85d56a69a +size 20383 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 5032dc441097..918925b77d14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2150b95b839ded8bd71de1a8f3b3df26e093eb0d9833e7ae3ce5fdfb7df70964 -size 24000 +oid sha256:eb95e35ae3b403915e37bbe31fed54b8d741e3c648d53163fb318b72d47ecea1 +size 23942 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c86454e7c94a..2ed761829329 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1341331d57bda42fbf87588e87523ab11123f851d386dfa165f4c21f81007f64 -size 21322 +oid sha256:0d71a09c538847197a6e0a26b1c321916f566109305c94b4a0a1cf578109e751 +size 21245 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index b6b7ba4a7600..15ce2450bbe4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c665875479195f33e1981d5c70574eccc2e74a1befae4be01e4cfdacb6546adb -size 28338 +oid sha256:7e9f3797c9b9e73a7e35fc113e2442577ebb263756324790bc148e0f103e5b43 +size 28291 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 977977fc13ef..a3b0be41a1cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ed341ecf72f0abc8e0365958bc80a699daa7bf76c08cec0ee02a0969137120b3 -size 31320 +oid sha256:2de5c37dc45dbc48aba10a6f6b5c9905ff4660b9de2c3fd6c8a0bcf8442613e4 +size 31184 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index ae68fb3f52fa..0f0761b8e11d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:355091201f54a3a40bd6bb6c86287cfae15973e6cd8ac3d88bff5f6396a45776 -size 24357 +oid sha256:86d7c526a6b16038ce97627b31849adc0f7ee13b7723c91e180e5d1a0f7b3231 +size 24331 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index daa53fa2e980..8fae4b7fcff0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d5a24a88c3571d6f59f06c844bd042e4bbcbe87ccf6443d063a4baef22181f36 -size 40461 +oid sha256:616dbbb4646670efa0c2f3a6cde63102db028a3b17f5c3ae86f29d126a088553 +size 32308 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index d481da271a60..fb162f128609 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:16a5036520da7746812aec4d5134cb76d0686b73f248005864aff9c3aa90fa8e -size 32207 +oid sha256:5be8748677f3489ecea876eba100531796cf9115470ba14b91e619f1a5906a6e +size 25585 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 2e6fc0fc7cda..558ca9e5b57a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:92de453a3be410e55d4aadf4a3f804e1ba63e6a7e04a129cd24e573f002ee33c -size 32098 +oid sha256:9c7de4d5a05a21d5c029683a226393c93493cfed0e9ca25882eaefd1859055be +size 32047 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index e1b843623862..e4d768149896 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5be3272a6e9d7b9591c78b94b46506d28885ca224896d69e8954008131e5039b -size 25057 +oid sha256:9b90b4dcd08c8238db6e2997a97d9d52884492af38d9f2ec72934d8b151ef0a3 +size 25027 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 223c6b90f701..cb037706c23f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae9e235137a49c75071fb9694b9c6fd4bf3939146531e432d84735d5ac6fb15b -size 20287 +oid sha256:fd22f7fb8c7704d565adcac6fdf8642af8e174ab7b85bf9c1122ccf2584733e1 +size 20185 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b8666823b2e7..73258f5ad4fb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:256844168e5304ee3ba77850dd4fb70c62057fd30a1f295fadf2a65cc3f82a1d -size 28873 +oid sha256:1a7e7787d99064d912ae857530f2d451f0182f23bccb073bdf1986f663dbdfb6 +size 28862 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index ddca62b8cb9e..9d8a51c409c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:75a4ad6a3a7d7a9714200f34929dc261f2c0aa8956e95ca1efa6e4d57c3a8119 -size 32007 +oid sha256:f617b86ed21d4e90599f5f1d6e74d999eb06431ecec8b4f6d2610dd1bf55bfa2 +size 31981 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 26ea0413c669..c13a5d27821a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e33a3a5f944e09f38e68a15d5e09ea304fab440b459b2adec38799f0b8b695d -size 25516 +oid sha256:66a9983dd46423d3ba9930270498692618dcb1403306a34cbddfbc0f4f440ab9 +size 25424 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 08249f05187d..e59322f23210 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:34410330e0531b5b44ce4a87e425254f862d6e686dbe618dcaf61392a1fa6cb1 -size 41367 +oid sha256:e947cf0969003916c6952de1ff6eb3cffd58445a3a99cdf79850e453c232e26e +size 33609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 37951a3a7f1d..e4dcd1738780 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b01d069e3084cc0e754599bbdd61d364da9304a402b0e31ca6d0e37bd0f91dd -size 33439 +oid sha256:78f953547e71b4e64e636f235859190d7397b0d9ab8892357168d1afc9e510a6 +size 26348 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index d65c5d4debe3..241fd04a8134 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f4fcd217ce101c63f671e53cf151e2be133d72bbd760e7e22a9ade969a30c665 -size 33217 +oid sha256:ad95958f7274d2e16518d3572dd84cabd4e24ff69d1f3bb8be31d0dd5a367c5a +size 33233 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 81940a040088..ce94fceb26c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00da81368859635cc2b28020ecf43166c49d36c65691ccb1b2eee0f266538f49 -size 25924 +oid sha256:ee004a3985e27b7e85dd8524fcec88b80eac388e344c6cfd5ea1263a5944d931 +size 25876 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4af7647fa1c0..be77f43a7980 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2c2132303cb4ac08172d0f9d4ff99e71ac7c6fe38ff82dbc2daab6fc79528ffd -size 21439 +oid sha256:24bc2181845085fb34eec75825455d256cb0a849ce25f62ea77cfc435c6d5ab4 +size 21340 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index be4358c13192..bb393f126c12 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c236001e5e67d41bbea931e8626b1a33f2054ac785856e45bd60e63b5fc8d0be -size 24857 +oid sha256:70445e394e5472fc3475047b3335f486773c96bce4586442efea921f77142f7c +size 24708 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 051e818eb0cd..4d94d2eaebe9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aacc5de3d5634548ab64c2a7539b323cf88cde0eec32cf2cf999874442fe679e -size 22563 +oid sha256:b6e2a7a65acb9cf40b3b972d624ec107c7eb0610a4cafe4105ae11ac5775fa82 +size 22468 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 8e92c2c72f48..476ed44ff3e0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dcd74499978b28c545ba6b3b309fc9a4a9db55f41b03db1595849d264137eb5f -size 25998 +oid sha256:0079da141e024209cecb951f0198f0b9756303dbaea774cf3c6d6f10229b4ea2 +size 25848 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 457c3e13bafc..31411503deb6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c58c07f1d4f576aa4317c4df2c1516907935739c29aa936a858de4cd03122d5d -size 23496 +oid sha256:f084050daf5b85db2681b9ebe72264513ef065266e95a613c6988b58859db925 +size 23501 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 329dd44ca484..2018d042a298 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ed17e84af47dafecfc6e5eb689d26f04e9bfc0e77af75f5bf2206b90e50707ac -size 26043 +oid sha256:5973249e52ecb562e085ae9314e64f64f4c6369dea6f6f19de324858563acf2d +size 25932 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index b07828d582d0..22fcf3c70a50 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6725cb7d1faadf98e8edb007cbd7da24cd20d9077a9ee5918c2032fe07e65a03 -size 24041 +oid sha256:571a6e38e29f801465e7c590ac59d1d905260b59eb008f6331a0f9886b9e6c4f +size 23939 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9ca078ce1c07..50e3dd537e62 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f7a7c61fcc0900a64a6499ccbdba20e19782df1201062765a673f8f37a109c80 -size 27056 +oid sha256:c633d6c13997a07adc724a464bb10fd54b235a5bec737838be0edad28cb03c68 +size 27012 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f95fa6b127b2..2b591ac9d25c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:547d690f559c7afc41ddea32c4cea52f27774234bc0ff233561058aff3620fd1 -size 24811 +oid sha256:a056e84878f35013ce9c0ace3e6634c65f39e180885837b9c4cf3b74ae68e5d6 +size 24818 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 85d1a9de558a..8a92cc2d8728 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a24dc0f1431c4afc6e82014be06e1449c53a455204a57071fc104c3d2b9ec86 -size 32127 +oid sha256:5bcd8d0b78401e9b611c224960ae8bed7646a9cf4845c160b58c206ea13dd9d2 +size 31987 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index e91f22646023..93df710c3128 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f3fa5b564136384e3cda541c0d6019cec7f456b84ea7fd8d4ac12d2971da5b0 -size 24006 +oid sha256:bde4723255dc64546c77fc61050fe2cd96c53750347e77f1ee770bfe3573fea8 +size 23888 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c1078fd6ee93..941d6e47d740 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ed29ca56be7def1d9a7f0ddc5e00b3eb1ec25d0c415a536f1a59347b2be63b2 -size 32532 +oid sha256:99d6753fe518b2c5230e95d72e0502eac15d6d6552251b67acd106d25c396b5c +size 32473 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5a1d4fd2e509..d5d1b6198ceb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:31612813c648023b2dadcbbdc3545508b656afffdbeb0dd7cf704d54f53edf86 -size 25062 +oid sha256:243b2140447fdbf98fb93d85c212c5f17340b8c43be99d82f616875f204aa351 +size 25003 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index d869c770c16a..c63d9af74d13 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ba224f84e1e908ff260d213c6161e91daf4f9a7ad3a8a6b7aeec90df53300b2b -size 28103 +oid sha256:3fba16dffbf1708f46c9c3ce42786f0cac2f591d48934918964d7c01c0af92ea +size 28083 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6df8c30bf253..03a9647dea01 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fe67ea85bec5937f91dc8e18d82fe926ba8c053958f94f3c75b2ba282e861326 -size 25993 +oid sha256:ed9dbe685a88e9272adbdac926f18e5a68dad5482fbaa01902acb6c9a89c2fbb +size 25944 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 88ced6b5f72c..7070ef079f93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49319e2d33b4f5b0ebd5cdc13266e52c398e3e6200f0333725466fe0d0a2704a -size 29129 +oid sha256:204f68141f7bfac9e3d6dc2e77d2fd8a73868c79bb1c56a2ed5d55b0a34d011e +size 29121 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index dba0af23d1e7..fb2ce3c7c8f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae059cd3f07870e715229aaa19c474ab6356baf67288aea71555ff0710328f21 -size 27203 +oid sha256:485c3ed3ec7a80c091f7442f845336416055d408c5f70dacdc3f081ba1ae7d15 +size 27196 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4d67e7109c49..96398a980c2e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:091492ad02dee7267b4041692251b68db78ddb1c47702050e5b8fc9730de07d9 -size 28345 +oid sha256:13b7583fecd9eb3e4487163f65b9f653e04a1e591db15edb43882afb359a6a19 +size 28305 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index aa10ee26877a..fa067a840d5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7a839cbdc3739e317714ac98d728d2a28637056b4f3cc85f5e84e353fad33a77 -size 31263 +oid sha256:839fcb7d54cd864b205d2cd8d4d3ab5f5574ebac6004de3afcfd2eee209aaaff +size 31142 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 0bec6d33ddea..6b2ad6f1b8d5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:370211de66f8b59e437807e2954835638370569fe4079065e23741aad4b358df -size 24419 +oid sha256:bd68f39fd348f8aa7cbefe1d0f9ca5475613ddc6e11ff779784eb9e32ab936ae +size 24382 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index dd124f6b0c18..a3c582898b8a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35b7bc37b019d583e4947a7e34acc9e884842e744aebc7ddb8fec2e6894e7321 -size 40457 +oid sha256:394c99780b9f3be6cd87bf3a5e273646ecba24a755b996f6820db4e114826af4 +size 32317 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index ff0c25a4c6c7..feea2a85a58e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c53593bf53d292209385872532ecfbbb7f51b2e612984b01fe075e6318f045aa -size 32198 +oid sha256:7053a5d41fc0a21894cefc3966120423d956da655aadae7010adc8c0b6c782da +size 25627 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 27d0ffc336d0..09f8030ca47b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ef4ec45b38f502ad07cbce980c05c50d0b66df2a1df0465cfe9c2c1e6518b7c -size 32124 +oid sha256:5db09729fccc1ad568d8f1eefd658a053aa15acfd19b7176b2f086b49024ebb5 +size 32077 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 8b6b64ff9796..3991660e6c21 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f3fb07c4f3a533c753ba9d36c438ce82c7bf47f31bb1a7961c5ec8f27ce00d5a -size 25101 +oid sha256:8b53d5ef0f850917fecb6bbb55f85caa8b6f812ae301c19776139d78b1db490e +size 25068 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 8e8c6c209e11..12e4098a192c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81e95f5ba2757bdb49e61a988a04f73cc60c61e86e49e743378b38ca11a7456a -size 20302 +oid sha256:2ca813bfb48ee77f1a23881c0856608a3071a7e3b185c483152b901635cfa268 +size 20200 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b9f25e53743e..c30322822330 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dce416092b13a7e769075908da6599d57f42df6cad556b5c13171caac3814469 -size 28898 +oid sha256:c93bf718e36a2e6d4b46820ecce90768e3be3b7ab79634a6d5b9b6d38381e82a +size 28901 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 040d126cb0b7..99fcf62f56dc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d8c957ec55aa7c78671a0153932ec4abe3dea219b55add7ee7280b188678fd70 -size 31983 +oid sha256:397c8a4863820de5a345eba38f2b3a559c2daa5e8c05b9510b7ea90ef9695277 +size 31951 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 23c27c275731..2cc0b70c7b59 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:adf36bd59b5b3f9ad9fcace2583d0f2595e69ec659840bda56bd201641a9fad0 -size 25516 +oid sha256:343a28973339cf8b04405c4bdcfdb25d72cdda5686b785b894a41bc3f3ef4758 +size 25410 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 77e898a5db2c..8333ee591588 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93afbbb4d727d2824841331553776cfbe5e964b0d5428ec953f7b7a6d96d58c7 -size 41390 +oid sha256:5e6d47bfaae49b76928a8b6110cffd1fe5b5af65b48a8033e3afb841477049c8 +size 33625 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index ec80642fbccd..c3d6d5d9f290 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33e5cc505f39a348fc7f5a06f0dae10a760ab502ae8812345f7ed8d4175486f0 -size 33437 +oid sha256:557f2bf1734a1173738d16cc4aaf65ca9ed25b9922ae05638a4527e47c1fe22b +size 26331 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 7b778e19e96f..722db29cb223 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:53b9864c7e983343771b8b98b72ce0f302c3e7de5d7f1e348f627e1013f9199d -size 33218 +oid sha256:6a6bf47adff7047d70068a717e332e821fdc17407a9382e7daa037b79e815640 +size 33222 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index dd64bc0f61a1..8c08e4511631 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d756e9a99c67dcb9d0563f30102a1d0745a5416e6c67c25073a9ee3990a4cae -size 25920 +oid sha256:15e2d292b59cbfeef3d65be025d98fd182bf3e58e21fcc798aa221a219b6b4d3 +size 25866 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0090a7339a47..03f6d242f230 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c493b077b34784de301c27ded23ae7389b5b0e951e193d380bdda22237f4feea -size 21418 +oid sha256:3af25f820bfb092dce25a30fea97147f11d84aca8b35c246d96894ba88a21096 +size 21320 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 30b8ba99e97e..bd227bf2766a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cf205f1b097bd4009d7f0727d2f2f9df33be7f47963714568237e755f54a33f6 -size 21924 +oid sha256:3dfa6429b8ef808b1a01a8d4ac88c70ae91ddaad7532b1d990afb5e9247ab65a +size 21756 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 4e1f5975a7b1..6b848a4e3fe6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:329b60ffa81ad682578a6be4ba837c16c783cd4839203a53210a5a6cb4d0c9e6 -size 19369 +oid sha256:67a893922fb2bbb2c0b47f06e17c8b7415c4dba972737403030afa171dfbddbe +size 19325 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9f9fd4edd8fa..0f6b20017271 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:932050e044e06a3b7afd4832aeec171115315a325b547d7d78e86a94682dc185 -size 22505 +oid sha256:4c11cc424677f943e3109dbd21e015ce6966d248fa16949174f12eac008c72ed +size 22471 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e03ddc6bde17..b3015739cb06 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f72ef8c00cb65493b04ebc7b6b34b8411f52f71f5c05797bf0ebfc7cc5115f7 -size 19880 +oid sha256:3e25b89310f544184f56c02801f858acbb8d0139f369da0cdf66d91f98ffcfb8 +size 19756 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index f1506a488e33..b842e72f0ed9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b198f2c3044638b1a476789f37e32b539bf814d2ab6e123f1e5a919043fdaf5 -size 22137 +oid sha256:a5a7cb2a3b423bcee6c97f22cd89c8826e5103350fa114b1926fffc687281f7b +size 22029 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index feb910a7405b..57911939c916 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ff19baa214864eed397a1de67f48e0cdc84e498f106f7ef4f67398715701bd7 -size 19112 +oid sha256:bd61faa6e18214d5c42aaa5a028a034b8f9b961881d10643f16d51eea02a7113 +size 19006 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c5ea585760cc..d6e5b2c19bac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f7ee088647746628ef71c170ba6b46be8fe0b9f82d725fa23f67e1a5870c6aac -size 22696 +oid sha256:49b996a8a8499001bfad79aa91f98c0e5319b7d6dfc2ad00b26a96ddc9a7e9c9 +size 22662 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b82616f0c5af..66e694e655b0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc0c4e439aee49974b407c7c787c9b9dcdabbaa1f13476bacc2df2c85d4b3c6e -size 19987 +oid sha256:0314b1979eddb77b0c1e30483ac04728879012ac669db6683ac44405a66bf851 +size 19855 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 55b62c7ff6fc..90c864f9089b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0cd772ecde3ba15981581c3e90a6dedcc63b6a935bc756915787a4e882981305 -size 23974 +oid sha256:b32a6a92829ddf6f3fcee9ec4e6d0bd20bca8fad323768a37a9ed0b73e4ee5d5 +size 23760 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 475f619033bd..4e8f1ada7b13 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:080ed1579a4153f233e50f77431997dacc7e0266873286350235373dbf076f3d -size 21497 +oid sha256:da7eac5e1db5b8e688450a15ed42390c0a5af8d9cf415963d8f6049ac83a5e3a +size 21279 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9815c2c368e4..a7c47f451411 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:71266eae235cbb5260f731cd4432308d36b09c328a607ae0ec165951d257c6a9 -size 24546 +oid sha256:1c9cf67616158cab50092f1fe8b644c0b520fbf60b74de9094724f151d0296d7 +size 24522 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e12d0b4093ba..09d392dfe53d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6bfdfd62b00c6a0679bc8a0bfacd5b65439715d14ea460e73ed5593dab17903f -size 22185 +oid sha256:e0aefe4b985bac3b64a845d7ff96e1f66eb9ccce1c1a2b2ae65beb816de84eac +size 21974 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 866f48c10e3e..961908e271ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:394c1726748f4ee62a40731dec3a6e16ca9b4b6d43e3b44f8cae5c09ebc3bd39 -size 25677 +oid sha256:a49826e7c251e5c8cd80f5eed5afc7aaab5d873be6a35d771b1222a90d259ca8 +size 25548 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 1c19e6855fca..0ec7ef6f772d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd84955c6d3e130d3ec1223d32e801e195d66cae5f7848dbd705352e95c8ce7d -size 22971 +oid sha256:e78e328a0b27cf357f43c49f9f81eb4d4d6fe3aa3287c32ede7f5953d652abd2 +size 22817 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index db50af5bf8a2..d3654766c3e4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d43d83f9f1ca687296b647d3447154256402ebdee187af214380a041a4250f0b -size 26530 +oid sha256:c0e1134b3bd0ecbd38feed094d64f584e934fc9339674bbdfde6508d5474d738 +size 26377 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c7e63739b066..bc530b48ecd7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c4f4e4994b03ce96b6426f2aced51ec790ae1f7935cee16a71bc0b9a9d97cb81 -size 23418 +oid sha256:25af84004a69c80ed7d69e3753917f5b10f487e6924b983cf1e054a671a1064a +size 23293 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index c19d8f00adb8..ad8e15f097dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8e81bc3f979b2e01e625ef8ffc73bda29f676768ee6ea2dcc50c6d6d7b6c168 -size 25557 +oid sha256:ca06474150484ca58aa8c7bfe013f149a935fb366d595726eb3be3ba3363c4f5 +size 25555 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index bf578af610d0..a48639a1233d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dcdae04e7b58febe7c7801fb30854ca64d0b83457a0e9898afea4397a0577600 -size 22782 +oid sha256:67a395280613cbd3799e0701f0b06d0eedf044e033f21baf90c9c81751859597 +size 22632 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 86d459937a6c..e58aa230c36a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f4a1161f272d6ef0b209422532fb02e807d0507ca1f8412d0604ccd46c30ac97 -size 26372 +oid sha256:7cf5fc3a20d3dc923762ccc872949f42d8d202f45716c1829583ff78a28cbb01 +size 26207 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index fa49c391e418..dd8fbfbd13fb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:043a75aad57508f6425a228b5a7a04a1a0c5f623157e7dcde4e60ec0a93b3c53 -size 23317 +oid sha256:d80cb7c4d4f4b16a52d6afea55a47f633226a0f02fb14b803beaad57003032b4 +size 23213 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index fa661093eafd..5e97de76faac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd8d04f56475e445599cabce99b5d2315cc2f10cea45fd2dfbf6a177b5e40ca3 -size 27892 +oid sha256:77ad7c63153aab06fccc1d79f61f901f9df625b300d2d69b030c6a4270cfa1dd +size 27705 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6b78bd0b0455..0bb5af409454 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:17a002eb731e7a45aad6dcf389d4bc32541be8d7870adcad089a757a7bf00136 -size 25391 +oid sha256:48c29e48d9bb409435f82783d08a1932cefa364a9e529e5cd76def32735fea4d +size 25142 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 6148696b8593..36593109bdf1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3769ce7a152956c1bdc4b2370d1bbbdddc7718a7da0e276466ea0b180641e77b -size 28806 +oid sha256:6d0b7d41f65dfa32310098147b1248adbc229745cd52cd94419ea7e5600fc12b +size 28579 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 07683a763ec5..0c26cb240707 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af68d330a74bae38cb725deb3579cbd7fd1a16a0695694d5048c925b21ee1617 -size 25885 +oid sha256:f1cfa542ae189d7905f33ee80d2d623b2c597271c41a70abd778d961aff2f8b8 +size 25670 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 44c506931336..cedfc1e4708f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ecdea71385ba8d51ceb5b0ccdd6c2775b5a6d1a9fb509208236981c1252de1e -size 22854 +oid sha256:7ad9286d79d8f339a6bf1f0c3e714e98550ce52d0e9e42b99bfa45698d12d3af +size 22749 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 6e0ca08289e7..5e4cdfd6999d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:60286caae2407da76dd61f79f6d7fd9aced66dc9666a68d3a026a740b1ca6cea -size 20242 +oid sha256:2d674849b491eeb3e60b44773645104d7788231df9b41a88c9d958cf9bb58f78 +size 20128 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 5fb2533b0e94..df9d97c36555 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af74e5e22678f67072668d5b24087e25c672e156c8ffcc1edba405270e661271 -size 23667 +oid sha256:670640d1603f5bec4bde357755795886e0b0f59a5366f84dacb37ec988bcd3c7 +size 23542 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 9cc61eafe724..02a5e1f2b07b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f96c752b939485e912c29cd1237b2d364cbeba21d412fa8ade29aa8b06a3939 -size 20700 +oid sha256:a74e9286cd12e999ecf69a3de570e0fd27a2bd66ae50624fae61ca8aef99c02c +size 20559 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 083ac8cf792c..a279bdc53f12 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c7fff4e5b8cc9082420a32956a739849405d0d7e8f717c59ba20b3a25f20852 -size 23212 +oid sha256:fe86a5f755a0a8d65fa2935f4e66813298691b70b627d0bb55202c7038b1269c +size 23061 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 661aa85a9339..2a63f04a8739 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ac4cb177283bd24118ffcafcbfe0e7a1cd2d5c30e38584bdd2c9080b535292f -size 20690 +oid sha256:2ef0514b701191dfb8f2d1207f7b2caa7b625fa49e72e72385e0e3333c2c9dd0 +size 20580 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 105dc02fdc76..42aaef29af06 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c3a5dc7ecd2061bf83ad429fd17754e8de44f0a46829e80f794f9b2492a6097f -size 23765 +oid sha256:469f8c7c9829213b97cb4db626706c8f44007726811ec34252b768f30dcf1402 +size 23676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5454d5eb92bb..9f4ec9c08dfb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a31bd249d1c8022b80f82f95a0f80a964541f953aa3c4b5b5a86ffc7745a6dc6 -size 21527 +oid sha256:4d060e93833aa97c127a3f8a2cfa8e27f0e717b0143d34cdb6d07465d63a8797 +size 21383 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6fea5d721fc6..a624c5670530 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:920dae119d078f90fbe93707ed1b6d1c27d383e274e48baa06d35ff100b067bd -size 27093 +oid sha256:c3e857292d2ee53715d9e8be5505c1bf0b372cd512e2098bd8e3c32674c8aea4 +size 27117 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index f7906852f2d9..8a59d0779947 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f0b51edd7193f4154ffc05fede5d7d9cb7f5528211e97cc516130f9131fd55de -size 24457 +oid sha256:d25f186eb523c0c3f9d6bf45250d339f6ea4fceb4c557dca0020f01f76a71795 +size 24327 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 880d03ce4205..ff989bcbae29 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8412f1ba3153a61a10d52da6ec8d840687d9431ddd4d4cdcc7bacced411e51b2 -size 28098 +oid sha256:5cd7cc9205eed9ca2c0957c7efc0ec0443c5de4aee6af514f2652217c144a3f2 +size 28135 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index cd8a4d915b40..033a302b3dcd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:da274cf7e304951622def3123ef59bd6966585a7e6e8eeeee1277c48ea618493 -size 24931 +oid sha256:afb2d6bcd54d5732d3344f9b070cfc7759c228e3bd8f98c2595d1b25a10609ed +size 24791 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b8fdb1ae0415..f42cb445862d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc104a57d76533bce482de62a6411aec1bc7cd4181cbb51b8274d9201c014d27 -size 27250 +oid sha256:7db68963bec9b1694e18a03c927336b8fad8d50485415203bec55e0ede99a3e5 +size 27114 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index c4f5b6e04c65..06151402e09a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b8655b678484cdc3a25dd48be85587b469febd297f283d8360765fa69fdf0e6 -size 24826 +oid sha256:77c80f75828ca1f2c67f1f357e319983f3f2d40c454335632a70aaa979fa3ad8 +size 24676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4f5b985a70dd..97857247cb75 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fd86c2cc0275a20509105b7f31ca9c548832eed4d4db7bdb543932c74e78dacc -size 27941 +oid sha256:cf34588f63e1ffe4d0c4d331b838a2675c431eb112847609fc36613c8febeabe +size 27778 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 227f7d1b2883..827e1df7274d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8402c8fd6671f0880c234aebeb87939d8237c5c4ffba1243989cd74978575402 -size 25439 +oid sha256:76878737eb4594a46685270d6a70deb03858e15e78cbb62cbafdc9ef9aa7bd0c +size 25368 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 41f532a7f566..6f52d110862d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:990eb27d2c746815a48590c3cce2056b1bf5cd2800401fc2faf88efa78b4aaee -size 23261 +oid sha256:fab1c407fc5ad882c351702ff2c75f0a81107d773a5377aa63e4c49646e978fd +size 23218 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 9b821e61fa20..1ad142eda362 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6555d088a32e17df62ba428bf108e17e8f6f39bd9e3d40f180b85d8b3975fbdb -size 18890 +oid sha256:d27592cea2c7bcdd1249a6ca61bd443ec82faa4e9bde8ea92162a5103106b829 +size 18785 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b90ad30a6fef..510fadc6593b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d94c999f2c3ca542dbd0678d7c3eccbc24483c0cbebe478eb51cac6eb62277a7 -size 23749 +oid sha256:86d070bfa299df3a6af6a140d23d15714acc7d4e5185cce23b6b0ecc1d047a84 +size 23609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 74c509bd50ca..e8f6cd009d50 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eddbc4cda6a072b055493d58a6be41080db06b1c9bb850d8e09bdc96e2e55995 -size 19778 +oid sha256:730a803fca33322c905a19b9254d0a99d19ae47552fc3dc0e5ca420eeafd36ec +size 19676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index a5a8e34aef27..c89f6256f0ff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b0efeb95667d26662606a6bd6a75d7697004bee6f5d5712e8010ccf37ddc51f -size 23248 +oid sha256:0670b2ec6c714e781a71d7831f3b8d7d740772885be8695afd3a19f43c061a70 +size 23121 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 299063b1ea01..c23c7233f2de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b3c758c285575ed21ab4e4d0915c3afff5f48961ef1a09975852c48b29895cfa -size 28996 +oid sha256:5bf4f9647d6bbabf6fb455c70e4452d93ac3a1ce4a27c40f0b9da5517ef0ece0 +size 28971 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 685b6c3de280..c0f99b0d884f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6134041d9f21332b2aefe8c2ea4adbe10aa056b1d6443ccaf17f44ba06390656 -size 21949 +oid sha256:83d5c448d68550b256397fc26cb0aef9f6808dd6e32c49b95de5698dc61b9c4f +size 21870 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 5732c94171c0..3d45ce2d279b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:68b48258783082c2312390cfcd685c4aa1504df9a400624216e71b17cdc62b98 -size 37914 +oid sha256:c9dde5dbc4cd4e59efb28f478122accaece0b31ff45683f8342ea7ea439cef37 +size 30827 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index f3b84fcbf496..c5553ff5e6d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f942929a26883b6769d8dc82ea48fa5dc367dd6c97c9e01e7e8395bb8492a09 -size 29571 +oid sha256:82274ba263104ae4ac8ebf36687dbf26543e7777a480dfb19a952f918d532804 +size 23541 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 7edafe32d1cb..d90474445ddf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c70e9afc9157bce192bded9df3c36b88a15316416cbc9f25ce830a820c1be87e -size 30176 +oid sha256:4addc8b9c1fe2032c2d07de60f270454a21297126d324a668355981c86d37391 +size 30104 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 64680c0a5c10..39b1a896f876 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ab465e48e04ee142717cfa3b238d1b6969de19ea9ee5d08ee070fe6938efb08 -size 22899 +oid sha256:8c6a44fd01b1822044ccce190fd80b3aeed2e9a159849dd4df489da4072dc37f +size 22840 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index c88a1e75d8f5..f6a5b04b2efa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d163220660715bb77875c4066424cb2f2187ee0f34f0353ab257740a82eb76c0 -size 18757 +oid sha256:fdc3333a7fad984820fe6d964929a8b4b97f0dc360ccbc809ae9eaded3796842 +size 18665 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 35d203375340..7098dd35c6d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3927189121bee32130118bd3840f9c4929af919ba312e8c6ec118d097d870153 -size 23719 +oid sha256:0ffdb6c6e839a8212e33982b64a13a670aed84bc3102c9fb1e38679b65ffc32f +size 23634 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index e24332e0a093..5b8a12e5fab0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84d704c2327b7221667d7752a9c8d1502cf4ea676d89ef9c7bb8a8224f0000d6 -size 29628 +oid sha256:3070253cb28ce2da29e3adc8d390f78c770990c700e6c0840a3cd16bf6325c17 +size 29582 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 8927ca2c9337..5c5fa4730af4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4fe58944d6274c2ca8aabdd58f78b25b9d8bb4670df86c4e8379e6fa5131e21 -size 23124 +oid sha256:b2828755b2af23a7b170ba23d543678cf91052b7e32ccb53fa2b364daf573bd1 +size 23041 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 6d94af31e7d5..4cedc7e3d5c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2b03ac8efe05073db043e45470b0d98bb334ebc584eb881c28127753f2f0fb69 -size 38390 +oid sha256:13b57990711bedc3d48d3dca9f23ccefd545a2c83a123e219d45b4ff1220b2a6 +size 31070 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 0903bd97411d..7268a982466f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2fbf581649c4c5bf9b41582fba9446618bc15e1a2b4e737012be0e53ddc31f4d -size 31191 +oid sha256:888e762bf88a42ecf62b33cb29401823ad498e0207f2a7b9ea57a67a738ca521 +size 24478 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 375eb0b80ce2..0e836db1dbc0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae20513b11f595aaca6712d9b75a8b64c084563fe326bd16cf149cd677648346 -size 30351 +oid sha256:126475d0fae30dd1c256c92de3ef3dfdb2af7c7dc3223765cc2a10dc30c30cee +size 30311 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index bcc21f24df88..9744efb4f151 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cce323585be8602f2d52487b1a7030fd878544fa55080607b652840eb603550c -size 23969 +oid sha256:87d9aae06e60ec586839d5df21da10fd0d7e3188fcdaeec6c8d84e8483452731 +size 23871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 73f9026598bf..9bbf8520bb91 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:63881d3f579d0020b5aa9400607ffb0e50cbf9a34e4545b2d975645fa209b245 -size 19536 +oid sha256:6a5c9d45c0397def55c41af498085bdbad2e248a2e63c09861bd65c7908a72d8 +size 19528 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index e64d4dd81e16..3d6656a6be60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:728a070c02d1b4bd3eb309d5961353907f5688078ed81371750815b4183785e5 -size 25909 +oid sha256:0fa77c824cc28a3a47d7027729952af4c951f8653e8117d17bd7f4980440a2de +size 25708 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 3073ac63c9fc..0e371cff10bf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc03cb43767090b436dd64240e6a3eac236f39287449a0eda7c1fd8e3a245abe -size 20942 +oid sha256:7558e410e50a50270612d8868ccaca47b11b2c9a470c0df8169952cf7dbaa3cf +size 20923 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b488ecfe3c09..392473d84f9f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f822a1cb79d21a985caa318ba1f0e6cdbd964b0c2c6fc39af39e25de21d9fe75 -size 26104 +oid sha256:a379633c55a6e408d7efbf3f9f76d38a7be5792357c06f673663e5ae763b7779 +size 25906 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 339ce7adad3e..341ec5b14bca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0edd5bc3d619682b97b89a8311af35ae35f9da537fc48ff3a2b176a11bd70ca -size 21925 +oid sha256:3177f1a73b7a15558be1c95b1119d7859c3fdca8b08d83752a572f3ab9388661 +size 21784 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e6bd24c0b18f..6c7a60230972 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:335ea1c0854515bbed6ab56e3673598f20a28d8700d957c35e8feb38522bb0dd -size 26290 +oid sha256:f88c6ae2c985ce2f8caa32812c8fd2c32c3197d4efe7df5ea4ce2e57eb8493ea +size 26139 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index b56d5439ed82..2137d204f714 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2436214c8df6097b5d57df4b5ca11724c57548f2f8924e8a8073288a6c43c8d7 -size 22370 +oid sha256:24ff90d1cbe8d3c9483cc50f94c8121fd217e62167f7b6c2edcf6670f87cac42 +size 22291 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9c6d46fa572c..955a9c5db083 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d2ccd5a444b73aed448693050d22f1bdf460576d46a9c49e7e1664dae7d20300 -size 27052 +oid sha256:6ea74889df72979e57371e0e4be1011b7c5bcbeccd6e719535d1548a4f352e14 +size 26892 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ad20eaf56bb6..61fcc33ce888 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48e76375257ed43aac264452f7b999b3743b60e75a3a223585f15ac7b8cb5330 -size 23262 +oid sha256:28af4f8e810414197e121c9904ad36166a179ca5cf0115a85814c5a72751184f +size 23174 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 0c0731726bae..cf7743f8c182 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee62654350260c67acdfc0f7c330947338aaad56bdbae9ed2c869e8ad67cbed8 -size 26598 +oid sha256:532a2e27f73aed93ca98f42353c54d82743416537558d1ce716d8ef5517efd58 +size 26499 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 7071c54c9b93..f27cf10142ff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d6fad51b65f7e1fbaa9bcdb99bac375866dc88a3ac3b4dfac984aa2bf5495c51 -size 22501 +oid sha256:7d53394b0bb70cb9e7ce9fb717a2598f79e50469f57d6e525f8d6ea4892a9abe +size 22369 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 65c2ce31ac13..d0a8ec89dad3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f73b57c776851b41298d9e460b657090467d14e3a722746e05ef96da5e704337 -size 27188 +oid sha256:592ceca9e846b0e8b562246087ca87feae368a3affdff889960b371fe142714a +size 27092 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e348b79503b1..de415d36a07c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:712f131495bae4a2c395155397df071b4d1fb00985de515499ac5b8f260e0707 -size 23456 +oid sha256:2b2d3eab0526ffc734aabc0df806809a01a4aad2af90e8876e590808f06db0c8 +size 23361 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index c00c81f19d55..9f31ae855187 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e60228433433adde01b4764098db5417f866b07a3c201d652f5daa16dc73b95 -size 29218 +oid sha256:e1b5480a51c9515375636f4147e96a652aa91578ac2ed59baaad10f25a6524c4 +size 29007 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index bac2896e6e94..bf752f6ff0d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4590cd97572f970065cdd7404013b2b6dd720d0857f48c3be6c9ad9121b77465 -size 24955 +oid sha256:322f96528d8481e15dba376f0389056e2ade949bd78f85f0e321022a28345309 +size 24812 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0fe0ee425ec3..2efcc71bf0af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e54684c4e5db698c581ec43cec49d8e166d31d276c137a8040adc027bb1e9d0 -size 29652 +oid sha256:8c3e7036ba4b2846c6fff5b7d611128320ec6e49a765ea7f68ef36bcc3b630bc +size 29430 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b5229deb0aba..947d860f7fcd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d505150db9f8ad67db93e3b28ba076460f9ffdee1058fd52a60f2fb3c26065e1 -size 25860 +oid sha256:b2908df1e278160c960ea00da42e26f21280bae1be8808bd080baaec83672ebe +size 25691 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 68856deb16b4..194c89051d08 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:234881ed45eb7e010a2951ced0bf1963b91f319966e00dd1491ad09b92a225a4 -size 28997 +oid sha256:f4118ab6e59d4c5e45f85a93a08e8169878f5c374a0e9ea3d1cccc6cfaf44928 +size 28982 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index e4d1e9fa0f0e..772a985be4fe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a237bfe8127438c63bdfe87f29cde1ed9151f1761aeda440dd04bf4f419efbb1 -size 21993 +oid sha256:d18c004e131c0de2fad3234d166c9d9868d5bb52948b646a8905b4e241f0d9c2 +size 21909 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index b4ac0886757a..a7e40678e8ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d8df715f7dc88e143d4d9a5a93d3cae78d288669322e1e22e4de32b9729df64b -size 38067 +oid sha256:57143b2f5bb9e742a570a66eff02aa1a6c7dc9abfd64bc55ed6009fc28e80d43 +size 30893 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 9a3e01cd83f3..f2b08b92ae42 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ae7afc8b6fc56304ad1e926857f6258f25d3098cfc5bfbe27bca12d9871a0f9 -size 29591 +oid sha256:6a71faa101d3ae57d2dd3802244ea4687d3841c24d0b35f8c0520bc8d491274d +size 23541 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 1f8222a03f91..2bd4fc4817a5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ae5bb6bf3eb9bff8b9dc01335cd4aa15c36f3bae27c42c7e0a8b85cf0bab282 -size 30220 +oid sha256:6e1d63a4b81db64c766fb206fae2749d71f5f10ccfa61e141737722924359885 +size 30149 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 03c43faea36f..b193ce0ee3ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c05c20aa61d4bfcfca37515c8662db26ea0975743ea4b4544d2a1ef468cecfa -size 22901 +oid sha256:157616b8314453fdd4297eb18168ab373b3bb5e12be43af843ce592bf9766b96 +size 22840 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 4b12f2d68386..0fa9dcf1601e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b5ab90c64947240fa717c29cfacb3202f3fd21de084d717f501950c387f82029 -size 29637 +oid sha256:683d5f051a2fbf34991b88970080cdbd5adb3f9dcc3c25bd4680a9341aa14478 +size 29590 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 9058e33a287b..cbd10493e72a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9871154ab4eee6e1d8e32738b67c1cdb236c4535328ef0c8ec4f4b2ac6eb1d41 -size 23125 +oid sha256:ca8a95fb6d4340fb457cb90d878d846ae0e23996fbcd8a6dabd4bd065e997426 +size 23041 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 7c3a105bbd6f..dbad0074ed8a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:759e6bbfeee9f8afba3407d71532881af15174cf49c8f6e16ed729ec461aa0e6 -size 38385 +oid sha256:058df8a0a55ee8bec9631bec3c08d479957a6207d3487672f8452b1d22aa4d86 +size 31085 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 4212136217ff..23919453f051 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7a2888ea692ae949cb2c28cb9e0e03a47497d70c54ede0bbcace7d10d89b0855 -size 31186 +oid sha256:3c65158632649c9fd61709a6124edf89005936bacd0bf6417dc40c7a11c46bd1 +size 24474 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 248138061313..7067a95c239b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3677a3a8ee081d6c3685cd7642828a0b8c8539d4233faf329f53fc89a368c181 -size 30371 +oid sha256:047108ac3bd57f40ca7bf0871efb4475478a56b067573f495732c529bea73925 +size 30333 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 584c05b621b3..ad6eeef18c9c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb2cc1b76ac4b9fc17c88ae7c8a68958f69a133f14eef5f9af1b967cc4068118 -size 23988 +oid sha256:7bbbfa3d58a22b26db5fc455667353b4600265491d0617ba24655505a4027aa0 +size 23891 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 564fb10f3221..03df3e58bf72 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b71684bd29e275e6190fcae73de1eaa0bebd35bbd93de7028f92e19beb5a1fed -size 22507 +oid sha256:1b40e17d8ba08f29750d80cb10e0a364b8d7f99f5ea043b9fb661dcfdcf3388b +size 22395 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 7da5b8dc3583..a93d3c398e13 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:07ea75e08c1e16462e779f571e53e29c08633014b0bd2c692a4580fa7ed73e24 -size 19932 +oid sha256:b40151b20837c8def77caa4e3788171bd9cb38e08e6fb1032cb6e41081bbf504 +size 19868 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ceae99d88e1e..a61c1bc7dbde 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33aa279fb246a3132abd1574956ed0c2c44a73b4f1125538a746cbd220cfc72c -size 23665 +oid sha256:fd22a1fb351bfde4c8cee93347226bfe7d0341f0d9fd75dde255b70f22d8eb08 +size 23581 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0dedbed90d33..e2bc26db748a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ab7b8ecc5cff7d0f0362caa1119788a5bd3a323db897b780b43881a0121c0e0 -size 20829 +oid sha256:dba057457b4d7147b4850849c816f4a83fe4c74df26a3dacd3a543216fee97c0 +size 20788 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index d7969a8e11b0..625eab3432d0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:411becaf63952cb5c636b583bd43787532dc0953a3488250847a5ea526114714 -size 22589 +oid sha256:5a68d0d5928fa7394f9431c01c7ce9a60b18df74907d52ad31671256232dd467 +size 22450 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 3bbf9e01e0fc..e81aa7768a92 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:114bc40a596e327e993ccea97d118d2d12ba3719931c68ebf8de6f5cf47701a4 -size 30586 +oid sha256:cb279eb3880b6becdac80707b18e27290e0dd7557cc3907dad3c8242a5254e28 +size 30379 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 8e2f284edac3..87959d77834d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a0743dc78f5732f3fcd0828a46273c6580f460172ddbb9eb86a8b1a39adddfa -size 23493 +oid sha256:eaf2b7b84844662eab6ffb9d8343649ea64a8ea6313a1584b60992005141c345 +size 23406 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 48827f383c57..0260ada65c57 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cdd3425fef5eb08684c33b544696e6e05f1eb9b1ba4a177be97a45d50f257dd0 -size 40019 +oid sha256:63072f381183cad974211743b08e3d84d305bab46cdb378aa302935b66d8759e +size 31461 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 5c0c00d2709f..73437ba9c3bf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:234ed965bcf885de54ab5c3f4419fef2c6e523a461faddbbd379d5de1ba92232 -size 30972 +oid sha256:389e32b3cb369e66957c1184700ed3b7f8971269c70fdcf4579f6de2a6263635 +size 24275 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 5ebec1e4389d..df138e4e12de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:98fc5e9f504e0f27724180c4daee16616486d6b0d43fa8c37a5e9e3d583f2a62 -size 31178 +oid sha256:28ad510f65393eb17490dba4fafd680fc56e1d41e7e57da1017eb6e51a84d634 +size 31154 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 6ca87f0d8ca9..69b00c9df016 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cec0eb2e00424b774a58f235751e4154bf1598a5d1af845489f78a8142f2f95c -size 24173 +oid sha256:549b2c5bac82eda75b5209e0d2563badf07d9465153e40f1731e033e25d54c3f +size 24074 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index c4ab8150a193..571c45af89de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54f4ab165f9151a9b7535bfbf0c6f115bf248a785cba1622cd50ff1f1378820a -size 19915 +oid sha256:69d1654448c7923802eadb81e36ea74a4deed48d8c13830f9839a53b9aa0b300 +size 19880 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4ead2cf96ada..04f1cd600e67 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cc3989e9f75cff72099620a7be4fb7f18fdf2bef8187cf177d2c2c6262385214 -size 24103 +oid sha256:25548d9d4cd62879ee2efc73546b1fba2d12ad96106370fdc0c2ee8e11f873e5 +size 23969 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 88eb75469005..c3397dfc7296 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1e5a366e9a307d56f9447ea170b45c57156b66a7b8de4356e0d6d94a78801e54 -size 31064 +oid sha256:d3b32fef1f50f54018517ee110bca3c9ba8112f5004453f97af95e944e0caf8f +size 31033 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index bd35d65f96ea..27b175cdaabb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0a718a018fc743e96df8c7220061aa191363cbdcfe7d2ac93df09f86ce9fd46d -size 24598 +oid sha256:8a1ed08258c24bcc1012bcef8262407fe6ec70aed389d3d46d25c3c558028feb +size 24439 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 51c97d8b0c31..e05a83cb6a0a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d2d8921cd4b11c0098b7aca34130621c79fc89ffa449328d3197539e78e63903 -size 40856 +oid sha256:a13fd7e556a6e14c8a4c8f4673c511b74011345381a8745024734d94e4d48bc3 +size 32791 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 4f36159d3e5c..a26cd449754c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ddbe223eef001bf7acc72abdccf2bcc9ce9e8b8bdefb27cfb69ab23526418f6 -size 31941 +oid sha256:f64cae1ec4a057b4423480d039b3c266f8782f2098b3dfd8b9cc9a853096442d +size 25134 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index fc13612020a9..c4e18b3c7b7e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1da6787663f8a546ca2f76aefe95ee2b255f43b9fa1000eb6a3e590043c118ce -size 32263 +oid sha256:4ba4a3adc3d45c1e6eef933bc34bd63cfd450b3b4f4b1f881f0e9540e2c0e9c6 +size 32295 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 70187f5f2f41..66491f31f240 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6603aff2b5945f797eb2505296ad30971d68d121a2a2085bc861b700d4a256c8 -size 25082 +oid sha256:1b18c671db95c15308964dec7b57403983fa9867faa9b1e9ff199076a4a2abde +size 25000 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0b33ecf51142..32f5350c1d2a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f55acafa750cb742758c7136a87b117163f5ee8afb44a4d902ab8b0a18d8e143 -size 21015 +oid sha256:b2bb93d3ed3e779d966e59bd92b28917554fc10bbeef0f6e6fbe93910662fff6 +size 21017 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4b0f111ff440..62ffa8da893a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb9acd20fa63dc2ddc1d556a4863b5387ccb6061c99d23818813c9231847f5e3 -size 24622 +oid sha256:78db7967ff7cf03957a2ac33bf9d7b5bd65b9ebce3a8173f479aea2ee17c181d +size 24396 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1885b704383f..71cc77be292a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d378dac50e5ef9bd5ee39b232b3163065c17e0ccc991cb06465cae5c3b919ce -size 21894 +oid sha256:2971fc4479672a940cf43542d96596ecb7ee7db2097a52289fb97088ac112c09 +size 21827 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c859a9ecc49c..0c02b0d382b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d0082081ad3c23ef4c7417041bf3f34dbf8a640b61b9ca8b4424594764c0ed2 -size 25655 +oid sha256:bbb7595e584d46beb09ae0f27e88f81b19c4c8deb8210b32d0360ea734b84347 +size 25467 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index efd14a09897b..df9cddb26578 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce9890ed7bd60588b8402abea6428f84ad8da4a0b4b1d43dd3b2385f720d90eb -size 23091 +oid sha256:01c83e4985fb7f267b915837f640eb3eb34428f2158daf70bba989f52aa5a459 +size 23110 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index d8c0371a2a0b..eeaba0815cf5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c949bc529dc151b01e199cbcd2503fdda28c6b55f91ed22564acd2197dcc2c2f -size 25699 +oid sha256:9934988a64a8c7c93256ebfe6a731db0add3deac286e25f5833db8d5749acd49 +size 25622 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 8231997362f8..489782c8d820 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc5b72959c1b30594d44abb8e8c2b6e4c115e4a95badbdd982518c05ba5550ba -size 23607 +oid sha256:5ab7ad1bef88dfbf31221748b1264d3f8c91ba955d36ba3c0c44761e6d88504e +size 23502 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e0a290523c7f..6f53ffd9e021 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca240294e4a154a9a072770af5952d63e82a61ff1a270bb3244ad4debc1115e0 -size 26776 +oid sha256:915923f7235cbe8b241fb76aa0ccfe8e2273f17a00a491e1db8b9c93cd0e415b +size 26679 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e8fb8d17d4c7..303ed8ef7624 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cdc2999b1f5042b1b81c35cafb64f5bfc2107b2bd607daa7f84278032a911b18 -size 24539 +oid sha256:dc0f8fecfc0f2e201551f6ce33a1181d369af72049bbe1c3c3be957c8fb4a153 +size 24422 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 814e6dfc3852..39d5a0c7d23d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a5ed46e276da7a40ecae7523fd9d69df4d3b8b3175aead1de30f06dcad33d63a -size 25830 +oid sha256:e2f626c7a997c52676164406d5e55433009666839af5d7e4b51b170e85a49c82 +size 25712 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index efd4872b7d51..5d4d6ca8bc22 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:639327e95a7bdfce75650fb8ce0d9a74ae8bb82c167dc4395f5e3d4f400ac57a -size 23742 +oid sha256:6bfb70f96645bb2e646cdec0b382dc79d16cd37d5b502ea269235f97d26db171 +size 23634 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 042e3c2bd189..d40d0c19f44d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:64a01fe6cd13fd56a7c125e44e970f12c52d3d0d18e40759f43fbda755229294 -size 27360 +oid sha256:8e866385ff4fa958ffe7501f23a20a461f94a93cd2ace93587157f47c65c487b +size 27225 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5c07b5c355f8..fa78671ab75b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ddbe9081d8a3cf0a54ad0513a43640de5088e3087a2af8f46db62e425ee8ad1 -size 24868 +oid sha256:dee704c699df10e5da94b82614ee94485cd34d8030b1c4274b7e34c44b2e03ff +size 24784 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4800add7fd7f..a9b8bbd5843d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f154c5a6c90723e6b79cd4a4a50c6ab3867cecd655cdca6f94a1591dd9d77f4f -size 27916 +oid sha256:cff1efc79816f1f49a5a5fb1ce400cbfddd96fbb561863a61f1b1f5715e0b333 +size 27703 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index e77866954c11..5cd0b16b310d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a929575296f9c48b20ed4811d6e505a11fcfe8818b734aec63dc51719c6c11f3 -size 25640 +oid sha256:c4a0c73e9bbc5066db2107b23b8491a6cbd7e21a2c77971f005aabac79eab626 +size 25606 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3b62cc119f8e..99304108cae0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d1b5151c9005afd76bc51f03213eba7f55828d0fd2199772b50ece75d80b5c19 -size 28810 +oid sha256:4955e49c733707e8ee540eca96739d6653b9d01e78f9b60d0e61f35d386ea992 +size 28761 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a6103521814d..d2d101750b64 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:34a4922653d45584f448c860d88aa5a7f050db2c148848b04b49f928895a42d6 -size 26706 +oid sha256:26bcc1e73a8fbae437a59f820edb73a10627689992ac5b99c6a8623850d7f729 +size 26664 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index a96b1ade1fbc..a41b54bbf977 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e31bd52cf7fb0baf9b534fa66637de03d34c035729cf951477b1c0a33bca513f -size 22587 +oid sha256:aadc180035fc30ebd1295b695d9f25ce461fe3d060fb48bbbb2c6faed3865e18 +size 22447 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index a6ffc41329bc..127a8098a0ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6ba75b485067c40d602ba8a04c785d31457c4687113a0f3fe91b2b13886a6d19 -size 30557 +oid sha256:85a52f9e6d2df0a91daaa3f40201e3e6b2e2a7cd34192966aebd1ccacfc313be +size 30352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 980a315bdf88..77de869c76fa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f792627a4405fcd5ce85c87d8a30161624567fc591c742223c059f15ed9ed746 -size 23551 +oid sha256:8486f8ef632ed5bff1134e5a973682a26dc9144de7c9b6d40f30b32668428967 +size 23469 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 9de05f0dd36f..22f0f828da33 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6dd5193642a8b2fe94409d68c901dc48c4a1eaa051c53419271c6f02393a57fc -size 40016 +oid sha256:a6f5b0fdbad182baf34bc552346154e786493b8514d64104f264547fe74bb3c5 +size 31495 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index b214fbea367c..c369a2efbd6b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7a0fe39293c4ff1be82353321c92a791aee460c207478a9e718006c520d004e0 -size 30951 +oid sha256:c5364d650ccd20fe3b3272f1ad838721ef7511f1f966397a9a2cc76827e441b2 +size 24304 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 020ba5d0a3d7..738c6223ed8b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:782595b1fae6a6c914b5d28e280f1aff8cfb90c6d77b0bfe34f048aeb7e5561d -size 31218 +oid sha256:2ba6b1020feea7819e240219e42157a545bde0e401256e1bb1bdf2fbab2a64e7 +size 31193 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 8c7e5a522a58..8baba94459ff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c48cce35358943d8f21198d30b385199b43198e3b72ed728929e4295c85b4053 -size 24219 +oid sha256:22e8ec5a7a987e252b932b09d05d32a83c6fca001e438a38478128f2556dc357 +size 24117 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index d1971b85b975..d378a2c848b0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8e9aff3009ce0c2a1cce07502edf4ed47d75e415bc4d5227d26c1b05126962d1 -size 19946 +oid sha256:6a51412623e064e675e68dd517e9789bb4bf5311515e72e01c0a466fb44b1c6a +size 19915 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9a033336e564..147fad8f505d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:56885a4fc09f5e0153fb1a8bc318e5e9a3db6008f5547527bf968943bb856862 -size 24111 +oid sha256:fe23f83c7120c517f9f6dc7a34dd69a98b1dbf0419fe3265521d6fe05f11548e +size 23979 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 7145d0d2f568..fb314a773b0e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc0ff4d93f1ada667243e2e740e6724656a07878406866e10c734521e6c52ee5 -size 31064 +oid sha256:ecde77ba89cb6c41147c514563affcd5900123e604caa992d2d1324af1e4d8d8 +size 30992 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 3c34ebd45999..9798655facdb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13bc721ce6f2f468483481a872dcc905340db5efc626c2d46526b60d8110d3fb -size 24603 +oid sha256:306747349783a78751d2fecff365615e9c2c8234f7a6789a28f11a1d34086f3a +size 24423 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 622007655761..ffda4022089a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ca1e59ec7f3e6b5104a000947a864861cf5f0cde2fc4b672acaa01ed4923909 -size 40881 +oid sha256:c89ac725bc56d6ccca30c98c98cc51e5efd08f359e73f0c6cab2a1818a8cc2a5 +size 32784 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index d7e45fbe592c..59424c8aa0a8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:553f701ad28befb82b74f3cc2fa03c59b6e3ab510dd04651077bf242b1c15e02 -size 31929 +oid sha256:c68b61ee38d4b5e2743f195696f9c8848bfed54c2070bec8e91f8571281c8420 +size 25123 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index e3876b2050db..c39fb84dc752 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:315aef8e48f100d3239f3225efba3c7411ebea985c4ec1e97521a1c8941fc11d -size 32265 +oid sha256:2c618771822755609e06bdf8d5101c9f4a1ef177c38623200a1eefb8f1a7a65a +size 32318 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index e5db0b87fb06..ad0615b69818 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:16646c0dffdcbb2f4b2389ad978dd7a2bc8f86515925658cb408497d44637e45 -size 25077 +oid sha256:d5262d10ed8abf5b5ddf3c4ee69440195ee5f3c667830b1822206b3656837fa7 +size 24987 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 24cfd6b0f0ef..b827dbcfc2e6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa808e7e543521af62f1d1cb3f03df1cbd1fecb5eb83eb39d4765f345dc0c2f9 -size 21007 +oid sha256:eaffffbfc3f62ef2c3cdb0ecba067ef206dbe8596d174814973b8fee558ee8f4 +size 20998 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7148cab94a26..c6ccd0a37506 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:86612b5f533c5f1fbe0d90f9e3cad331f3e6fa1779aa07da727fb3ea2b69cc9a -size 22444 +oid sha256:4d656f983c58f6eecfc7d2e6755a9c8a1392c4f2f0dfbbcd2df558f8393b6e29 +size 22293 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index ea64198d9f4b..18e0e496a050 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2c6b5ef5c42435e1f35d7604af052980db74bebefa22b10c4dc1fbfe174561a7 -size 19690 +oid sha256:295a53c903621dd1d39a1d88b4aecbd8035972bab65aa62cbe542e93c770cfdf +size 19529 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index c25a1a307293..7c56a45835db 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:02b5a5f7aab947454d6e2afe65e33da56d415fa34c62951f8bdba21b5b017804 -size 23066 +oid sha256:0970c50619a8019bda592301ee5b1673d700b03c64c0d7614d435930b30c3023 +size 23008 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 844885f4be1e..8bf6b9891ad8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:14f252b6489a04d8e8439d09e36dec6ce31d8be440f8ebc43ddbbc322b3a7e93 -size 20016 +oid sha256:be9644ef7c434acb404b6a3d7238683320662602e92d94965e5fdf91f910f6d0 +size 19898 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 688524a41cd3..dc91b1c47072 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e33a955e1e3e707ef5b09027051afb77198332d4dca3fabdf95780fd23fa7916 -size 22653 +oid sha256:b04c26b802665177357c1349b4c9e3d7092cb2fcaf12071c6dc3afd80fb3891d +size 22519 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index d2be51661068..bc6c1eeb50fd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99433c9b2fe79c8a791fb02ae9e454157b77d6c6210014dc402a8ee543c14e6b -size 19382 +oid sha256:42de2f2eb9af8e6eb9342fbb2e5c6e64f59038b01978aafac2bbb736b222e84f +size 19259 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index eb1bee5b039c..5d22a707dc93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c83b496d3ab9fc0008d43d23a9468d25bbc1b1df154016ba4be623693ce65d7c -size 23254 +oid sha256:3068a8183d948d822007ca32f2be02479e6db70af79aba36df0e26325204c19a +size 23224 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1e0358e0c329..faf6aeff5345 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d16b0c9c82465c4ed00df9a18ae163b7ffeb3439f47abe41f717b620fccfd5c3 -size 19928 +oid sha256:4e9c79f33fa5617ecc6001465415836bc6a8286d7420a162885adb3835998f5d +size 19816 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index f87e6bf27e04..ea1784707d63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:712e616c945eae955f4e33be0bfcf91e0ebcb3c0e2f88516b3c54bf459e71202 -size 24559 +oid sha256:a79b8b85f8f3551c6e7b39048097126ffa921ed2329253d60cec283e256072f4 +size 24299 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 6386f9a3a091..6c987a986c2e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f96660d99a18ad7fed7913aa81b5f23b26d00516b3ad221a8fd7631151dddd93 -size 21734 +oid sha256:01dd420f1b86a3b55b16f4db613627fda46d0872b33727c8639c0f0605b0c8b2 +size 21502 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b140b349c6f4..30dfadb5c0a5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:833529377294f3255dee3055c3511f4b64f7c668186a7635c2dfbe1143113ffb -size 25107 +oid sha256:8e08344f3aaababc9e6b0215abbd817a1c99fcbc551334bc9fabc73b11549649 +size 25065 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index fdf150afb699..fcfcb5fc8bd0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2fc67a72868e2ee89546017733dda6187edc20f2877f3cdca0f1d60f866fec88 -size 22207 +oid sha256:702202ce5031ebe00b7a9ad134123f49ea70b747f7227888211cde6d079b0d72 +size 21966 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ae53e0fc6737..83a48a249d99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3875ecb12f8522c0238f33653400f1b8edfd1ab6be84b72a846af11e1d68ffc7 -size 26284 +oid sha256:1893bda111839ea6e92f907a4bab35f6ba6bd3206a6ea00c2740fcc1e80a3558 +size 26121 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 84e48ee2d369..f6a12cae9f35 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:489a74f6877f67b5b9b6f0057d97b8f2deb83457a957a971ca250a3cae3b7f84 -size 23304 +oid sha256:532afff778d1ec1ae4bcd31a59c88b8d38a3911343906121684215a3c1143418 +size 23116 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index feeb3267213f..d1ddbbdb6709 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4fee81577021d0cc697bdd17a402dc9632ec69691797357dc27f2085e5f24ad0 -size 27089 +oid sha256:3e782e14e388d946f1a91387d130d50615ce36708198a5a5c0d98bfd572b1ce6 +size 26980 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index df8746d55995..a92629640a6e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:820a069e7f8fce4ac8ae37d43571f9195aac9d5b22fcb9a11e73ad10cb5c9f1f -size 23716 +oid sha256:374af7eace25e6da4c9d10cf669bec8cad34cfcf7258a40e490925684b991983 +size 23590 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index ffb1a93061bb..83f3773d621a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8c90b831de46a4da55ee6f06248d7bc863013e26bdc1326f2c17dbed43a1a668 -size 26112 +oid sha256:e7b9c8cc662ae1193e00c8123eea26a14509882c799c5927f33ee267fb175629 +size 26030 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index ef581037bf69..89350075696e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d9617b6e11d9d62eae9d8a9e9f1e632a1dc9dbd297c0b945cef7934357f3a821 -size 23201 +oid sha256:884a2c5ad1ea2c4e46b2b5899967d684efd5c83d7ba29d94667fecce782e2e19 +size 23048 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 08c2fa020407..835ecef8cbf2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9bdb6b2b374380910103eb9f54328cde00088c022e76de062be1257d003adaf8 -size 27024 +oid sha256:714e57cd3eb9437d2cf6ab7760a1adca4142cd74dc77d9e113ac12dc795f625e +size 26950 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1d77dad6cac1..a0fa1d45a2ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57c66f13d7969f3873cc5058d374859ce7d3813e006d0878843af76c5f587383 -size 23632 +oid sha256:082e0c27cdea58c498b05c368ae1f36855f9e7078c2fd81a0e8deb5619517b71 +size 23506 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 2772b53f994f..7a5bb711a297 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa3675fe885f5f55f2e24bb73a817636a8802a1e76c9a22fdf005e780ddf990a -size 28573 +oid sha256:90a62b1afa365840764aba190de34fe83bbf3a32e0d0d8b78768b7207c21eb02 +size 28396 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index de04f56d3df7..d8635e9bc5a4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:927ff4d6af40c374f8b4f4e2d972b753affa5b3c34049f29c3186cc80282861f -size 25637 +oid sha256:a7786bc5da657946070c44f682ede51be7598478973a2fddb69c3a7abf0c761d +size 25399 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 97174cb3602e..6ce03e048ab7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:204bb9cd34b9f5a1bac45893e5db53becaaca05c71251fc675a95d1e2301c9b8 -size 29434 +oid sha256:5cb43c307e726500cd955a81e96a202581b097cf8c374e3d7c7b34bc6f4d4d5a +size 29302 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 281b25a2700d..9dfb490fdf13 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ac93cf591e11cae77a742245d1ebd4f4007d8ff9797debf6ecf120c5d5fa7c2 -size 26211 +oid sha256:4c62df5b1b534983072348badc6eb13bfdfa1e510315dd3230ffa5147410dc2c +size 25989 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1007612e6390..9fb094d3e41d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b208ef1b8d39c109fcbaf090ccaa15f166d6948e51d2b3649385f0a8f2a89d13 -size 23224 +oid sha256:bc14488063871813ca1d910c852ada1be0fb523a2204993a1b406b49f9ac42ab +size 23085 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index d4912f4289c6..eb2108d93e1d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cc7d5b34456e1cac898b01bda6a2d9e191f32924e81a203d70f7354ed7e7ab01 -size 20262 +oid sha256:835c2ead2e2cf6d09aba3f2f0c9e46cebc30e494221bda45d15e10dab71030af +size 20109 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index f5746952cc8c..cb8ff6a881e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bae051f4c9387f80f9549a7655559ab49a9c5d64ff1b49d47f370fa1c9621900 -size 24076 +oid sha256:ffab9086579d55c80f0189150cc5e3a12246b3eb518e220b7cf5b7e77cfb580d +size 23921 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 3e34398ae39d..973a5906db86 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10c29fea3e7db99c0ec8974f704fd00f3143cf9b49884079ba13468fadca63ff -size 20708 +oid sha256:38a4d6e6d828eeed5e1eca0da1b85220bc13de70304d98eafd4ad80908a500ac +size 20587 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 3c0bdf4b35d9..850833f7b948 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e90517e2cdb14b5a0bd573fdf12ca78a2413b1ff6baf6c64e3bc2d5c9445d8d -size 23586 +oid sha256:1262d71b25f8340a2d1cf560a7157aa8b4999edbc56f00d62ff52f9871f5dff6 +size 23439 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 12fc7398ca57..2578ca3de74d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7dc5c90304d5d34669dd200d8d81b0adde5e6fa7be6a8eb5d010983da647e6c4 -size 20738 +oid sha256:57c7672e7679c0b5e9a3c023fa60ad5c6485aae3dd229246987eda7956c900c8 +size 20626 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index e37869325c17..966274a528a0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d9fa42ac4a176abf9fbaa751b7920078b561552507b5d8d7a54970c56a610ae -size 24151 +oid sha256:4352676a93b1245892f2407b2845be3cafd4db51a6068959e28ea737a992c018 +size 24067 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 08cd73ee02a4..758de13e742e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:44f57be7e5052eca36507372c41516bf8d41e7d475cf9766e5d407e26046b1c1 -size 21384 +oid sha256:a222bff1d9d44570e54c35fe06b7d74dff30ba39b1d4ddb29ce792f1ee603e65 +size 21258 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index e331b4ec0c41..6af3ac17d64f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0666aff2f1ab0cbb33ad1dcb491586e59ed6367fccab5f6fad6b61ba9c83225c -size 27445 +oid sha256:0e8bb5d1168c27f9ba0c973cf5479c0903fe46ec11897a32c78ae615983fcee6 +size 27423 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6587fe60dda0..07fa733d620a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5723c6ad4e3657c068c6f9bbc7ee36883733be7b02e62919c1c3df93d8dc6d9e -size 24502 +oid sha256:bddb3d029c135eb87c1333b9b0578bbdcd6d658a369749892fe5b0bdc8d1d6a5 +size 24401 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7fb0da20251c..6d911a636469 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80b2e6c98d838fd9c45929c94f1735f5d32c8d9220cc9852abd8814318d70932 -size 28755 +oid sha256:4ab0551e0848c2eaf75dd2f62be5c3ca721497eb2ae9a4da5f10e4dbcd1d1af7 +size 28805 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b9a195f1c6e4..f41b692ad456 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26d101949492d541d2cdf3ff8aae45f2785bb1b283e1bb247b8651190173077c -size 25112 +oid sha256:28c9b5595496627012e775abebc55d8184bf12751aaf5ed153a08aa7bb8990f2 +size 24960 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index eeb06f8613be..b17828345bcd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:318a8be178cc5309b5d89d67b9db3fd3edca3617425c87a0b55f6757dc8cb92a -size 27569 +oid sha256:8045ebe9205320b4dc2281c3f20ca0b0ef8d8fcee842fb3b3d14b949d67f9da8 +size 27541 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d36991a5b409..6a89d77b688a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dffbfb5c7dbdb97bbf1331f64ec4f10ca09e1cc8c157af6a16e7633f1a69a348 -size 24869 +oid sha256:65dcd8a3e8d3c68e5397c839400ecb77399b4fcf63240fc6be2191986c58a547 +size 24769 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b95c090c400b..5837009acad8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e55057ca7dbd2a9468a4007b4f387d68ad7c9c643b447ccd79ae01a42d10939b -size 28595 +oid sha256:ac68ab0a735562f19b12db186f09925c5d7e9141b7efb230510fdd1f5480f697 +size 28510 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ae1f4149ecba..bcfdd90e857c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e1aa73d98b5a2a9285ff77e00f4716111dffe5c0b0121ab880cd11024677384 -size 25672 +oid sha256:6894cb5a5b6787800291de051e9d93cdb4aa43030db6952936b4c6c3706b3679 +size 25538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1e0a7735d920..f54f667d9652 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fe6a9275fb5af253ebdcd9f8146fc8bf7a5af5196de71f11ca433f5f4f7ed744 -size 20656 +oid sha256:c39128fb8b56226137f75f1e22b5c2e6adb225d01804fd48c720882752f70d5a +size 20514 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 994d4d3b34f9..a61b770b2fb2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0c4b4fee0aca754388794ee235ec1e7bde044a5c545dc5a0020aaca125a045d2 -size 17967 +oid sha256:1bdbbcd2284a30046cf639e225e56a8ec20ae4afc64afed5e1ef6b963d47bed6 +size 17885 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 2ca68611964c..0cdcacdeff85 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00198977559d984e5af41c2780aecf92e2db690df5e6c2b438a6f15658e82145 -size 21602 +oid sha256:659af9cd558363a60f5a51751590d045aeea316d1f52bdeaef2f675271865465 +size 21458 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1c5c71596a2a..9b33790cf364 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b32cfef4431d9cfa105d86ecf6533e5a9011a1cc16ee1c52ff3fefa20f11c197 -size 18879 +oid sha256:4589e669a0e7021d032b7860aa02718611c4288fa3d9f7fedba0262da7dc9b85 +size 18765 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 62f71b456fee..c6407e948292 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d5482630e6f94b4cfa3997ef6ae7a5ee4fb299f243b88eb3ae2e0be4579c13c5 -size 20835 +oid sha256:7d21e7a0ad360b68e3aa9501418b76614f59e142b2b0582d4dfcc4c97d308668 +size 20642 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 18d01cb193fd..cc309fb22981 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:efda402ee5b4ccfcf057c0b83b4a4d38399441bb68c8510fefb4ba42126f3afa -size 27928 +oid sha256:ed1c5de16c037dc048a47cdedeb7ecd3dd98899352a2b9f831b7d1b27b80fdeb +size 27891 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 3827e6ddd614..6d916bf09c2c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54ab375a547e6125446a334c1f17432d9d2bb6f308f1026462054cea749acd27 -size 20750 +oid sha256:d0e6d373d17ef05dfab3d14fba492e8baabbbac92597a594e7b92d61950a991b +size 20656 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index bf35b472514a..c02b49219719 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bebf96bcad56833e7241de95f72becb7762e40d3a6c4e8fdb4d9696c5b7fdff9 -size 36901 +oid sha256:eaf949f4efabb1f152a31403b4dcf5dbe55c5e16abedef7bbf2b9be4f4856ac5 +size 29716 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index cb05ea57a2b2..145d9a9e49e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1dd48024b40d2af387b9fb89b2891a03c98d59c17a3a3e13e919a70b46973e0b -size 28388 +oid sha256:6da81da21467fb119d82fbdd28607cc5079ab4c320fd73ea3db56405380f6546 +size 22107 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 908fe018f906..a48d9d5f7d79 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:481d9feb7b0fd84daa1d7df3fd796974ea2622b9bd32c222d24fd54ea989e726 -size 29044 +oid sha256:66578839b9d0f06426f58b5416b8c21184bab3b86555a5213c7cfd615c23a4f3 +size 29031 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 2b42bac52092..66e2ff179e08 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae643a632a76de64d98679f268ad345e9f97da01e2468122cdfcbe08158d42d7 -size 21637 +oid sha256:68eabbc410f719ac060d15bae20722b60bd7fed1aa9433603bb5384eb8a034ac +size 21537 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index e1452853df80..2cf4e82d0749 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bac98faf4dec345c428469db7e83e97c7ae56c6cd4203c57d6a854bee830ec4a -size 17905 +oid sha256:40d2f723075afc0a13d459f1adb08f5d6d510222a945dc6fd29c49fe8a82b7ef +size 17791 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 99305444c6bb..ed22d681ac38 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b53bebddde8dc329c6c95be861fedacb6c6afa86030b80b14f4e2336b2c72957 -size 21687 +oid sha256:35bd31fec69bf53824ee2eb87b0640001735a15e282c4ef093a213689bf582b6 +size 21543 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 562753a3f2ad..156865023141 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b0566695f7b5158593408924bf565789c0b80d93675e6d6daa77ac884cff3535 -size 28469 +oid sha256:06a19a2d7551ccab5c600f8f259cf3cd641aed7b6cffe904d19f60d40dcbe7e2 +size 28431 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 10f35c5f3aab..4b20d15fc45b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21174983270aafb183094a312985b99f46b2517bcc80d0fafc7b676170724882 -size 21921 +oid sha256:ddfb641ba7024ce2db0dd39b69fa7a29e1794193e7fdd825a2314adbefee90ff +size 21866 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 6e8c52d84032..6bef6cf440b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d574e48020ce00cd0dc5a27bad43fccb50d73d26fdba6ab7e9e0a0dc6af976d -size 37334 +oid sha256:eb745bfa127bd13d432bf0b75fce1ccfb83a12353b6396ea6f788a7e57a72586 +size 30006 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index ecffc346f8be..0e7d01638415 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f97c1ba08d1fbd9657943590d4d8b3598226aabcdecb537f8f1a55019d2c0c5 -size 30064 +oid sha256:ddd6a3576c4c285488b717d81cca9cfc2d8974111fd189153117051d9666b712 +size 22965 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index d2fa4554c348..057c7014d766 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e22a6e13dc3dc7a5132ff9e451f9ceb210f8e31f2a139842c95f0d86a3a124c1 -size 29333 +oid sha256:b7e5e5173731ecdb395fc9ef1ab94eb7efe858edc13b1eccfbdd834fc63294e4 +size 29223 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 150d8044c096..caa3c4eed7ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8717d7ef06ab3cc4d8cad5c0459191c17aace1906ffcc886a2a8f4dd03635994 -size 22858 +oid sha256:a9bbc26f6d725ca7943b5d910dd85925ee255354359b9134ba3dd2a6dc0f3dc5 +size 22760 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 874ebed71fdc..a795bf0745cb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:03a9d92c684a50b46cd3a67f11b9593dcb7caecf65815e353bc4101d4b5a50d0 -size 18778 +oid sha256:d8e752bc0f6bdb38d791552ce7c0354a1b4c5d1df3dc93d521175b19eb5cf59a +size 18763 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 131a2f180754..9bfc74bd35f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e14ca135161b2a6b5c4f26dde1b90193a49e836640c79f698a7497868b5db851 -size 22908 +oid sha256:56eeecdf99bb62141247b36c6d9447da53b2e1e16dd088ed2e9246aced7b7ca1 +size 22652 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c1477ffd17d2..1e533ded1a00 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4b3e5fb025abf46d83ddd7a7f419946cc660e4e09b1d7a65d4053d43c9c34d93 -size 20046 +oid sha256:ac45b542ed557cebc5bd2565851787cbe4409bb90cdf628aed2566a31d28ad8a +size 19896 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4a00741db2fc..9266c4cee795 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:127db0a49f74d583100d64ae14ab523927191ff3769dbdf34dcb311e2271ae32 -size 23878 +oid sha256:0ee7196d621bebcb9c55fdd107889913c7a3e71c3c33fedbabbb5d013d2a9524 +size 23669 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index eaa2c55c3d69..0538852cb5d6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:070e0b9b5230b39e48176d68004fd23ceea9d99636da94fa36fe141aca2d0d7a -size 21090 +oid sha256:ffe35c39af70d747316881dad3de70bc493d668b9272d1c9bf99d3fdc29bbf74 +size 20935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1a65dd567a05..021f184e19ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:34289ff91cdc43690e39d958d44ce27e6a8a188bf5041dc66c34bd59b9fa6311 -size 23658 +oid sha256:52cab408dfb8d43154dd591ff11d0ad27a05cc008a2b8b6d2860cb0e846f2825 +size 23494 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index aef048b3ab61..110ae627635d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fad76ba22890a3bbf235ee0c082dabff0ab6aa2238e75fd3c20a7fa66d4384a0 -size 21559 +oid sha256:e0724bab2921d1bd6d245f2236b6216104a4548e7a83695a000ab2e762003984 +size 21483 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 34113a3ff114..2de12962e56a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:16d84dc43b4d9484bd75271b9eebb78a50aa2d438d818043577fb5026f7b49cd -size 24737 +oid sha256:5219595dc37ecc6da23deefb10e8178a1eec0688759b00baa762ccfa2cff9c22 +size 24574 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 51705dc43f32..0431c58968ca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50db1c0974b655f2e2cdc6b40214c5b263ac5f2790d3f7da12518f13a6863b87 -size 22532 +oid sha256:2e7a90e12dc19b48783e45bd3ab856a8490d4b299dc130e669967c040e6beb65 +size 22455 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7f2ac4d02be3..74fd18acbfbe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99df323cd0b6bb62a42615da86e9b97191c67a5f54b6b24270cd2894823e6f49 -size 23983 +oid sha256:c4f0f7fc9bd9407ce7aefde81fe08096dd83cef40dc20972288c6d63cba07ba7 +size 23839 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index dc1d8be5de32..094bcf5fced8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b77a51eb198cab50ebe15d45efee32e9f9596f11335a2751e40b77c004b6ba64 -size 21700 +oid sha256:cad11c7078f820b9c72e160a17b2526ecf0ce187de9a722bdb8bbe620268693a +size 21613 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 269142d9a5ea..3c14f0b8cb9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2af1866025c25989d4363307aefaad328f87a705f7cf19655fe5a3bd4d70ed39 -size 24963 +oid sha256:bf97d15997f04119d27aef1ba678b621aa166a9e335495e725a5fd1e4f087f63 +size 24787 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b25bc0105fa4..24cea2734723 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a28362f663bbe9f2eed3698f6cd2e84dd6df0f66d0a96e2b10dbb4d35f4c2a5f -size 22687 +oid sha256:fc0e53113ffbdf2bcd2ef2a241fe02b0a0ce85ec7ad61fbc049bc3795c7c99b8 +size 22603 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 364c869cb92e..98a07ceb7765 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6d1fb9f8bb0311f59ca09627b577fd10191814d2caa21334e28c6724075ad353 -size 26191 +oid sha256:1aa01bf4e7879c3d0ae91c2a3d591355039100e34e7b2e571e520e3bfe72b54b +size 25993 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 67a64bda334e..f22ea5ded412 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1dfc85c66530a261ab4d8766a92a32eec8502f2b507ac6226ff4e63bc093f8ae -size 24144 +oid sha256:c1fdf5d642f307a1ec87212784d2d5ccbcf18c88441469a78f5196c98224d436 +size 23965 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0df7908bd575..f65ffc434ca2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:373ce7e86a4affefa73fb261aae544bbbff0f9815fd8645aa7b29679bd8a242a -size 27187 +oid sha256:a0ae408115130e3b82676a2a7145de255e9853cd0da02820051db354ff6c5b07 +size 26969 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 310050b26ccd..4c10a61fd132 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b3546fb9e7b86385f5e16827d8f6738c5a64536fd935f433fa860f9153beb84 -size 25106 +oid sha256:8e1bc11a44d259c683a0e724c67260cb32ded37b9e13e50236a2d22022aa04c4 +size 24906 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 2a4ae3dcaca4..9aed066df2c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cc7bed1c4b8c150a06d36002b62d36aed7455ce5b959c363128786b84d6b17b6 -size 27945 +oid sha256:1610e8df4b747a8042b94c6d1d3e174f0c80bdddd9f353b257e83998f69e9f63 +size 27901 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 832b131c2872..a64e9cb321cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0f7b4a18f4e5980f7c16151f313f5938ddb9695141ed9389662f83d37235a832 -size 20783 +oid sha256:472f5ffeddac911bebaa3bc4186ea32d66610b9564d62c79015487e86c565e8d +size 20687 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index ac8c4fd428e4..241eb9b2be7e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:950294d0e9b5819cec4c3ae324d5dc130df1fc64b38843aa4090bb11474ba983 -size 37030 +oid sha256:0e2f2718a126215eefd72e061c5c63273322db79c9fe5e5d5f5296dfabe23971 +size 29771 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 05b7f73f09db..9d12d27d11e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ebc46119d7646555bf8d5a8451123d536d7553d52b35da032bab2bfedfc1d59 -size 28400 +oid sha256:6b261ce2c6101b3aaf97936f0055124c8dc8b1f0288625a3b631e2ea7d91a932 +size 22109 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 00f4232efdb6..170083d22605 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d2d72c2af630500d7945cb43feff2b322c6c064f60df3e22f719a244ee564a2 -size 29096 +oid sha256:5bdfdc4f8317564e404bc0bc4f65831979043375af53b1312bf91924e0fefa4e +size 29069 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index eabbfa5504f6..4798c240ed03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62cb32c14b8749a80cc4cf847383aa207066219bfe2fe98c2d4477481e35c94b -size 21644 +oid sha256:4a92f158912956c08ce020eef833c55124ec41643d9830a6cd5c75787a30b625 +size 21545 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 40c846ff20b6..a488411a2d95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc0fbbf57fda445090e4580c73d24126ed7db984293e271b377f8073159691e1 -size 28482 +oid sha256:ef3a0ac17e4bc7d03887bcad10146ff3e7098333c6206a29f7cfb5e37e25bf7a +size 28436 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 0455f42a9cf7..212c0b9cdf0e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:925438cc3d145e02e9af70db517d455a42297e57bda69e30bd656a8a3afdef17 -size 21917 +oid sha256:5cd824d7b96a97e24650bc0739c451c2eceb7847bb624e96d532618649ca5685 +size 21838 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 26537ad268d0..83252984017d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cf31eb47011bf2f953c6c7a578e29d8f648b30faa2c72da87b6feed6d54e631a -size 37335 +oid sha256:d7f6caa42cf902f9919e6c896b9aee38992846fa578f7dbdd9b1e29ab2617584 +size 30010 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index bacb2eafc862..718bf9d6ce64 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a78937619fa9e5aa8b69658be4c0b78c4fde1705f2302e50c1a6674f436d3a3c -size 30066 +oid sha256:7dd19700badd27d4b1a8a5dee3c2a96d7830798b39438956dd516dd907469060 +size 22982 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index c6bd7150209a..a5bb42a6a307 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:177a0bd8b6efb92ec09f847765d3442f5ce67d5332974026891b58f7ea2f5448 -size 29326 +oid sha256:67d1491cfd3f136d6e766767428b234c2fd6e90ede7abd3b1436f30f43be4f31 +size 29228 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 884148e789d3..37f005c9cb1e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4a419a4582abc9bd065d52e19b7e0e835a61faf5342d191935bfb7c6f1c4c811 -size 22871 +oid sha256:ca4c2ade44cfb4a24d9069a45267b0dc5e2b0fa025df248f5a217940c4b97382 +size 22772 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index e974962a8d0b..cc5074b70973 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c64d2941d06be4b686e05049e162212bcfdb649463b66da8d903410cadce2272 -size 21804 +oid sha256:ab784ae824457e5d06875e1753f8a0e4840a6ff6de0fef8974ca5c2d8ed4cffb +size 21632 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1700d8d03df3..55c7dc74fbe7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cfb6d609862242dbfbdff934bfccb8fea06c424241e91b83745810760ec1a406 -size 19311 +oid sha256:6b0636f8d1a4f1323a2d412fa3ec6ecc796a04804fe813500e5638eb39367bf6 +size 19196 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 5eb44b615cde..4b4d85e33d0c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3bb502ac3e539b9b0d1295ae6c251bcbb817b377185e0f6a4c1fc5daf4866395 -size 22924 +oid sha256:9aa5588304ab3267264a5b8adce595c8f66810902bb0d1d8f2dfd286ea51e09b +size 22877 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 792b24b894c9..61a636774fb5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d578c903bba8cd5b76ecfc617a8943a9a7e0fc8a8552de92c7fa1662b71af5a6 -size 20316 +oid sha256:1889471b4889ab2c7e996be3c3324ae0409bd83de717fd00994c1e91764dc5ef +size 20287 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 32ca386dc280..feccc0c4b904 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c16189d53435bc3b2e5ec059cf6ffd598bdb1207af7a8d2eee033f3ec0fe4d12 -size 21952 +oid sha256:00ea9515fb152195b68a7294de794b0d2895d751ffc488b87df7e22b60354e8b +size 21829 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 823f3afeb74c..9d96c25e2043 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2bd71e40a84162c177a936cb5cd3594b1f116051bf36fd90f9884d980b8927f8 -size 30183 +oid sha256:f8655c09b0e268989323ce4917c3e57143414f5ede9687050c72975b25160aa2 +size 30118 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index f8d31162c5d7..9476f33ae266 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08dc5959400b89d6e21a8b18cc18ce08cbaef39d6db710b1e3c2a4a699be5dc9 -size 23025 +oid sha256:e30bbbe105e6602550f1e91178ee407c8cd549a05f9278126df772ffca9e4ec0 +size 22954 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index cca2d9d887a7..32bc132dbe37 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:04cefe084b7a8e4b9ac3e6d12a894be2cd16581b616c8ad6dbb2b5fe5eaad5e9 -size 39181 +oid sha256:e769df83c283b22fafa810b7ab741bccab7e33d9d6ec16f93bef9ce5078a8b09 +size 31144 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 8d15bc9c59c1..8f621fbd076f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b6f903d2cefe5d748e176a56aafce50851063e39e83709d3de8f13cac46dba1 -size 30798 +oid sha256:b6c72717dd2a7613f542053ce3374a685ea1a87ab4cd011a53d154c2539806a9 +size 24239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 36504ce0a463..e50c6eaaf405 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1eb3fbb54383bc6fec414940f447e6fcaa56460ef1f7a4b5cba83fe38361a761 -size 30971 +oid sha256:70e2cc50ab8ca10cf26bf6aa1b36b1a05d07e6c67b01a2d815c3ce9e8606acc2 +size 30929 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index bc039f539432..4c9508a7fa83 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d0199ecaa1dce9f1285d0da8a27bc280bee39766a1fda9e9d7688c8add0b8e84 -size 23723 +oid sha256:7ec7ea19a5d0e34b849ba645e92d5e84d5f481ab33e17f9206535fc9f2df270e +size 23611 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 0f1ccac64640..d8518916ee8a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc9ea7f11450c960a2d753e81801b281570389e991c415ae338fb855746412a4 -size 19206 +oid sha256:9455f4819670791dd89b89d1126d64ea33a65183107a16a6500c0923eac7137b +size 19095 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 7340691f5ed0..7ba1793b4b56 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b6a500cdd898caa322d471cddab209571d6cea7c47d43f795a021f9a4f6e785 -size 23183 +oid sha256:e938dd988f6cd5a8f826b1921693d1b11fcfb16c993418444e8eeb96beb6267d +size 23059 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index ba1ecb030b76..6736fe3b9bf0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af93cfe7b52e4110c9eaae2bed9bdf0f71dd27752e4febbe1dcd183bb4d86eee -size 30755 +oid sha256:34452ce38809f2a4ed36672dd3a80842a22551add3d7e9ab715bc47ed020884c +size 30661 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 5f903f6a2674..faa2043a5076 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:614a71352949980cc05e9990eb67bfc712a1e9f6f41f5ad54143c99283972cf7 -size 24147 +oid sha256:5d55268d6ae896f2eca32214817d7b1d1514d556efe8ccf60a515718fe35371e +size 24025 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 22278068c24e..3c62fa862cbb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:947a38f8b36f0412055ef196142ae6ec759a18598067ea8d7585c723813931e9 -size 40080 +oid sha256:043af56d83a972956d116732ee443f16004e9f35e78bc0152aa638639d938a92 +size 32414 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index dfac20d2d018..5c5382399ada 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ddfefdea845d423edd22f984a652e7def7054e105dca3a92a15265309918477b -size 31807 +oid sha256:4ab0dfa48b93b6fe03a21d8cb6e836f01f69a16489906ac9fa73ab61b487ab47 +size 25117 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 12dc14e1d6e6..cfeae1e06eb4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49aa369ccbf2df291d566d3f2dbe80430009be77de51351e508bcb69ac1327fe -size 32078 +oid sha256:4b9cfe42c73f1775e61a9fb6cf1c551fe0220789a4db96f4558ae956e7ed092d +size 32043 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 7a55ecc2062d..ab3f8755af0f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:059fd35e57690405c72cc25990f6d032045c0ce8344f49ea0ec229b5ff301624 -size 24599 +oid sha256:479076e6eb12e05bb4d14eea16514348e679f0e86397033e3f4fddc61e0c55b5 +size 24531 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2337e95edce7..8d923c959fbc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b92c8f81c14930b1515900ef024715a86aac8a42dc84082cc77120226fa1c2b7 -size 20345 +oid sha256:7175d4e7dad718d1a843618e40e6923ff02dc96ed29a10cffd4204679a00b680 +size 20349 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8441f2e9de2a..99a074ec7ba6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b26391c5e10349b38ded8ed9bb58d41e37ce762052dab1ed698e388fc35f22cc -size 23821 +oid sha256:cf6ee6cee5df851a3f5f49894276aead3aa80ebd57e451f557d6e768822ff054 +size 23613 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 3af5521a5e7e..084f98464142 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:688e2cf25b7fa34aa58779cb1abc24141370582469c51267558318d883490f53 -size 21376 +oid sha256:500f3302da9007e0f700e00daa4072d195de75b9cba497b1db8f1d6460b677bb +size 21314 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1075629e1f94..0f38a4f944d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d21a7f72ffb2307616955200ad9a08239c704eb7e7ee29c60e596a135524744d -size 24954 +oid sha256:0ec4fd3b91250ed9ba09de6d3abf6f40012f8f6c81e0a4235a1f466d3f7affc7 +size 24765 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0b57c5b03300..d145d3b8e1b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7eb4bdcf085d475eba6fe59803e765c14fe17363a27df7af23f52933bae4ae71 -size 22377 +oid sha256:b6bcca710b4c98dd35fe7eeac620b1bfd78aaf3eeeb4114372dc8d2dc82902a5 +size 22393 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 4ccd0d629e9a..b2a1d9aea7e0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:083f94b2b55ba74e8517a1009fa9fbb3cbac900b6f4c5c002afccd3311155adf -size 24954 +oid sha256:dc96a880a6f3277aad4bc6cf98879c5a2cbcd71c3569135bef9db9431e4d3401 +size 24815 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index a16ac0517760..67cdbc4160cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2807f6397678cffdfb17c1069cc223535f4f5fe783e6d3c4d1a76e790baaf412 -size 22978 +oid sha256:c9d382a7bcca0100d9402313a195c578b2d03993f64942e11d5b2aee26d71706 +size 22840 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 2aba536b22e4..09e20c56d80f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ecea0e5d63183568bb3e504aec3b755377d06e029e3d6fbbb598cc764493325 -size 25921 +oid sha256:c363bb9fd50a9f8c8d693466d47ec190b2cab6ceecec3f4d67d4e8744c53d3a2 +size 25900 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a29f1901afdb..03cd13406ea1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:015d82aafc183aa5c07b06dbe2675a0b79d317c719fe3272f3bf3041875ababf -size 23759 +oid sha256:d54a7a27aebf52729a6c26bbad64d25e4031f5d39dfdea6f68ca6f1d2e0ee578 +size 23666 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 2be6b9d7cadb..fe3675f0eb4c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38c328cb2ddf0b4358092a01b1f3db2007d23c29b752fe9b0c41666ccdadc746 -size 24966 +oid sha256:e7261bfd2d613a9dede4178c061498be2f53d820075b12af55dcb921df20a8c8 +size 24833 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 08af70b1663c..31002a49325e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca6a0adebf23d11edab080c54241f61d078bfa3005643372dd33a93d5988ce41 -size 23003 +oid sha256:df4463122bbba7df8012070be28e5ec464f0ea6444fe10e5c313534ce1555470 +size 22890 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3e5b3c17217e..c015d6adacf6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0c9adb76150eaa8379ab5df546c1d86b182537f2b09eb756531b3f9e9c35e0a1 -size 26597 +oid sha256:a1138ff08da713938a217155138fa68bd61f1f14c8b3476c8da864d23b42ad82 +size 26476 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1cce81345bac..ad4c86800a44 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:172d76309b8db1d59f90ed0097c85e2f2c6ab311a1eb20f92a8c774f0efd1a18 -size 24089 +oid sha256:e320c4bcde3d1a4d63cc6d1fa4d15d4e4a7f6dba46f77c8ee80697a151127586 +size 24008 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index f52872824447..72bb1d29c4b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b482a584ca8cf43a15e90c6c03864c5e4ca9f1e6c1d22dc10db06e19d1a8c57 -size 27175 +oid sha256:94b81fa9b01b24b07733fd15a0bed85dba1ff7831a587a46ce7217ff5de175fa +size 26979 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index c89b15fa12b0..dfc671618804 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d56560c8489dbc438538baff73612cf48caff51ff9ae31fc9cdbd9806d9013e -size 25033 +oid sha256:7d060cc93ff463303650eb37854955ca909c564b8ea93d237f068aaa2cf7f09e +size 25015 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 840ef61b3fbb..1ac55cd83cff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7008eb0c1fc0d61ef690237dca869ddec3883ba9fad3488e48adc35e48238af4 -size 28092 +oid sha256:0294809dc3e4df1d11f04ca7f23edf0b3b753a4327d7d5902dbe60bf27cd416b +size 28072 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 56dc327b546d..ff86fbb7318f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b59c3a193148d9e72548cadaa0d6b92699f3a781e57cd37771241133fe76ec8b -size 26123 +oid sha256:e3b8ab1611113fc937a24f78f7f1b7b1eb27fedf35c4343c4481068c66d010ad +size 26091 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 199bd2b989c6..db508b12e95c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f79b3d0a5a5bfe83e849a0f6194a6d47934bf73117ad9b570ba09c5e220e2be -size 21941 +oid sha256:f35122b4bd2bac5a0057a85e4b39bbf492b705c8e526c52e6fa46b16b2d38a65 +size 21812 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index fff22fe8038a..8dce6e96857b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:482fefc010f2328f11d97790a79769cd1cb59ad34190aabf71e69b8b96046d86 -size 30137 +oid sha256:62661878be15479ef7cb53eee7c88775bbb4b00fb44848412a7427857dc22ef9 +size 30082 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 249ec9d9b269..a56f04266260 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5dcb925ac793a74b6a938a4d1684151a7c6f6409b6927fccd54bd148e8be567c -size 23078 +oid sha256:fbc55207f1f3fbffae83cb14886ac719e5355d81e2519263df090a893461dc4e +size 23010 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 95da0c5924c4..938ee318febd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:819096825c95cf10733989a0bb85148060d68f409cdf7a9f70c2edd751525179 -size 39213 +oid sha256:921d3d0a5dc6b97f0640ea124d1a5714aaaf3d281e77552548d4db09479d5fdf +size 31157 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 090e6631f841..7a2a378d0fbe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a6ddf66d04b74146d824a5f17332eaf363c40d575fbabafe0335be8bc934a413 -size 30866 +oid sha256:196cffceb3a55030b0a713aec7ff7a8f789a22e7a421824c7bd069cf14c68795 +size 24283 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 20f1ee238427..89ffd22d45ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6aba7493abbed8dab29bbf9e977822d3fac74ba6a7ae5055600b5d07485bc564 -size 30992 +oid sha256:1ee470df8723fba9345971ca14f29218d2ba8763d011002dd7b43759d50d1aa4 +size 30947 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 3a10f4149966..e52881747c61 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:940bb727ea31eb91f818886dca9b460957a86f611f03b138fe39eccd2ca934da -size 23767 +oid sha256:54c4ee7d5bf6ddce6a1a32e62c21158ce86308ebca4f3d6814b8464444ab6594 +size 23656 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index f77d8e23e1ea..7fc965b3ff6c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85fbcf39e5f8f5c273f158ff8d7d86116514804d4ec0f96776345dc700e85811 -size 19223 +oid sha256:60de9c7ef20b99d3c3ec2feca37b6ba8da20bfeae4e6c138c6c1a4b84d38efec +size 19108 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ded2d62b5e14..41597bbbcc90 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cbbcf4227fef34480aabea0e27fd56708a8fcc2d34e2d5c8cf43789e4db0215f -size 23196 +oid sha256:f5979335490e0332854a9669b30e07077881006f5a065d7e3ff7f1bfbca13e81 +size 23074 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 52c381bd3716..c8177850c651 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0230ef79c7c5c21233aab6685e09b65c8fc32f25090e5b00c278806946bc641 -size 30721 +oid sha256:9dac5f14114a33479a6edb7c24252d9d16b0125470855040bf70ed7a27983ad5 +size 30636 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 8b0b3cd912c8..e9dc2d13d07b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d962f32eb81c0e15594234ab28e17cf5ff5b470e4fc07872d4d245b6f3e6f29a -size 24149 +oid sha256:ba8aac383188f28a5b87a9cb04954c6ce3b173d5b9a50fbc78b4bececeed7c52 +size 24013 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 9cb1a2d99d84..b58e8a305bc8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eefc848e21296bc9513d4cc1b4c03142f77e8f923623c6a7bdbab139853ba180 -size 40086 +oid sha256:44714b6ef9109784bbbdb1f0028fa9faefd169512e8be7bb07aa93188d39c133 +size 32418 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 317fbb32ce52..f3a0c7c0d0cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54aadb4e56b5c9f4004c57cfbf7975ad713d3ac69a9eb39fee60d97382eccda5 -size 31779 +oid sha256:ff1147a7a2123b6e0bdcc72c82474635b5135b9129a0eab2553aab53f6eec5a2 +size 25103 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 3c05eed36bf3..4897a5b731fb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef50173fd20de95c54f4d386013ad22abdca6de6ae6780cb051fb056562690a3 -size 32102 +oid sha256:bb5efd190b5fd6806a320a5e37a052f9145d41b5b522516c069fe4fe5fe6819a +size 32043 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index f09169c8751e..d4ef50f53316 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e5f5e533f16f27fd47ee5f4d7b82557b45990e5ceb705f0a202202ad29f3069 -size 24591 +oid sha256:7844afa9bf5ac100bc3a4d579c808958479d19edf2b0a9ce633b592cb177416f +size 24519 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 19c7ca550a6f..f328de0292af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8080a781ef047fc2b1c99b53ba9ec9207e8d404b71ca270ba3ab189f4bc45185 -size 20340 +oid sha256:0fde377d761bbbfbc5262d32020092789c91e67197690c75a871c3dbfdb4fdf9 +size 20339 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index a8dc3dca2c95..b849f6d88012 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48157c9046231057611d2bdd8ccb020ac1f25fdddc7c9d02c2da2d124667b073 -size 21288 +oid sha256:57fc0ab156f942cd208d72f702edd0202d35902141e23907cd24e9ab7516ec56 +size 21142 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index cef06d49cfbf..b78e16a8623b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0e91232f8a0a40a0e01fe4098a96703384f76b633e88537fc78e0520f2f0099f -size 18656 +oid sha256:07c583e2a963af5c733dd0b912bd9127a6da5e2809972aa3a6c26fc857504652 +size 18548 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 224ce8a9a585..a179c6b4f6f9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b48933a8f67960460577288bf06a67077d810d7a195514c94ce3b399c057dfe0 -size 21804 +oid sha256:3087aee444b9de5cc48c6e42c6f28dae6a38862d2bfe4b6c89a62ae56e742f8e +size 21795 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 14ede0b4e63f..c2a8b6b7a659 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:179256d497d2db8b92f635ca66d9fc5a625bdf7f290dc4efd737f0f9c87fed42 -size 19133 +oid sha256:8d5cfb038ce78d194cc28dcbbc0bc3b6bab79224f737189bd6bd695146533e76 +size 19014 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index fd862b944ac9..86b832fa598b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:205bbe18a278721e114505d294df5bde3fa6a731ac25cb14b6c9566e555c906d -size 21512 +oid sha256:75ad4d60e302ab695a8501fe5b6c16556943563f7fe68714fefd6da2432fcd32 +size 21360 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 470d1ef882c0..52b234ed031a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2255b5c708a82255e84e9640792db14a1d257709a981fb02d892b242a43e2203 -size 18388 +oid sha256:b73f33e0900dd251c249f0636fbf0b089c3d097c2b652a0a26e1f35c10586bf7 +size 18269 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 48d0564eb01e..1adc2d659c93 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:51373153ee939ab0f5ba8b2bb209c0233dac4e1394ff95757a20ca1087d3a32f -size 22027 +oid sha256:157a77d35200f4f22fb462fd8fff929f7ca93430ae56584f6c538bb82db0892f +size 22013 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 69859d335fee..7d4a61359dfb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f95663ccc683f06eee937aabb367ff79f08097a4553accdfaf857ce2062cfd88 -size 19048 +oid sha256:a4df42b8385e5011254580591e443694931bd246636ab6bbbdec0ff0bb734df5 +size 18921 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index c5319b17ec28..38143c7793dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b152c601af56ba1ab4bb73ada501d0ad9b11879f7b0bf1f93422e90f59476496 -size 23329 +oid sha256:f341b0f9a19013ccab1e9cc7b12d49b8b0cb80d85f054e758021b079fd079399 +size 23099 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 87855345fecc..5bd7c5101093 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e364a16d9835c2be5d2d5178a325be668199acdfc9cb11da2d71721738a4cff1 -size 20726 +oid sha256:5260cf8d769d32c00e9ce6b6bdb19ffc6f0ca38661e247e7e75771ab488af3c4 +size 20498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b626a405dc6f..8da8f470d3b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:75e682e3b3b20ae3c51d50196b49f7c4ed32921751c50ead6048391c02781f9f -size 23914 +oid sha256:121201277753748e5cdd15e7d393d78017b32fe0df096602e4e5d3b6319ac26b +size 23908 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 141e2ded1d59..642e22a5ff2d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:699d62d7b1d98c044eaf6311e6487626c3aadddd298749dd4be733759575734f -size 21268 +oid sha256:241f578419c1ded5b272c2020ddd5332ea03c9194e9b4c451227118fc35dcf42 +size 21032 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 98a821ae41cf..1b8816c742bd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3232bdd80220b86c5553327b9195c7063d0432ccafd2b45f70bef479b0e3f8ac -size 25056 +oid sha256:b22ea9d09d8c384cbe74f817bec837948e272f2bbb4a98a30a18c87e1221ea05 +size 24881 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index d111a8cb32d3..7aa4f00fc30a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c93d453d6de6886e0cb099cccf87b5694163b2d7e100a414836daf4ec953762 -size 22427 +oid sha256:ca4f86fdc0f60c4116ee95da407ba23001a35a12159c529a9cdb73a9156fb6f6 +size 22275 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index eb40c87dde0e..bae5ed64ec69 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b0f295ab10227ae62013d413ba5cf8b541a9df920d36aef2c4c6bb33e0eecf2b -size 25956 +oid sha256:45aa14beba28713fecf793d1cf25577e57d8aa67262bdb92e31b02ac5584649f +size 25839 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ee713832c21b..eb4ea45bb03b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:88eed088851e8a34459fdc095762c3e56cb3017e63144cfdbf18734e770293a3 -size 22959 +oid sha256:184d3973a70a833d0f7195981aafc800deb062fb5475bf153f02df757278d420 +size 22803 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b8da9b25ca64..f3c69aec5e1d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec2274c3d29e81f9b1c699f654172a37be679247ff20366ad85700fe709b4b8a -size 24982 +oid sha256:d5641e136f487066284ffe017f576c84785115fb65643fd0a17fd358a254447c +size 24870 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index e7e315c05b8e..bc9b821537a9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aee8ce69da17cf0e2e03efead2f8b485caf17cad69f03493d63bb4c5ff2b7f29 -size 22285 +oid sha256:b194b0b126784df06e1000532e50f369c7e9bf6a20ce999de2edb66f40dc77f3 +size 22126 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 6df4f7d792d3..3efe5dd24db4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:00de3fb7099fa2be2d63bb2842ed0ec1678d6a00f1f19395018a6ea5fc01c1f6 -size 25830 +oid sha256:d5abd9d9069c2b4e20b69086b7f68d066879071af1fbb4e016cfb76c68f47274 +size 25754 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c25217b6ef07..2aefc681f93e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:77d5e3a3346a7c55449c02ae2aa8f7c1a752126696a9953dadc1c2feac710dc2 -size 22845 +oid sha256:f6b7b733aca5de68efc601ee88a8a36ce1b8b175d50e462353998d8ef544d828 +size 22727 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 234e46f26c6c..a2343358e060 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42e2088ca4e3ce4c4a527eb1eb39d8ab54a497cbe615c65f46956c90fd980441 -size 27430 +oid sha256:1402bdd2208cf044daca3c3266268003732dae387ed0146ba7830d111f0bf650 +size 27217 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 5274a1bf032d..4b8a39a81d86 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9999ba58e88c5a0a1f7a1216c9983796c58c46802419253adb2509511af2d8cc -size 24930 +oid sha256:ac707a7c3d5c8ed40e157522d2c21588ace709d6cbfbf495b188abe4309145e4 +size 24681 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3d2a1b0b6b24..a9dfea127956 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1a237549e2115570e07a109033cf976167937be203ad2b2a0afac92b407df79 -size 28321 +oid sha256:eb10dce1675907719a2966a43f94b1ac3832ee28b5db5d585ccc5a30e62c9f7e +size 28118 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c7a51f887289..002e53be9e32 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1262c1058e8c794665aa3391c363d75e05a2c437f5879693b66e3a5db004869 -size 25463 +oid sha256:fae1b06ed876cc71ec273645b4f1f233a2bbcbbdcc5a9168aa229f97db716326 +size 25228 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 435e39429f08..c11aeeffbbd8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83a4583ce7e34f054c1e4c001588ccecf4bf20f6d5a75f5e60a149301aff1652 -size 22255 +oid sha256:187217ea0fc7e9a677b8ee1a1bd205e967f68ac3020ea5021ca235994a003fea +size 22136 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index d10bd637a951..732f00b6114e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f4cfd7677449b31ed4fd8aad519bc15d3bea580eb8bdd3fd20dcba89687bc2a4 -size 19498 +oid sha256:80c1666b71a0510887f3342b811b6b5933093a3cfec3e51e6597fde99d1ef717 +size 19364 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 11772f1647c4..d824735df2e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26046cb3fced4afbdafebc28bdf139c3a51263aee634312022cd538e891a72b8 -size 23002 +oid sha256:32b010a89298c25fa49ec24552ddc8ffda1490e70aae9eb01466a2623e8d561d +size 22880 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a2067ea1d17b..c0eb8a579f95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:40ad31a2096c66a2deecc5ab3187ae24d33247122727a8d62002c55b167caf43 -size 19965 +oid sha256:d23e401d239aaab7a28059d658d4c267b091ed09541ec3ac27ea7916ef0b47d5 +size 19812 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 85f43327ebe9..03a3d2fbaaf5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ab8406841c5384349adfbf685ff0513691262417a4ebfea1202ed1082c447148 -size 22586 +oid sha256:8e433db8438bf871670cfe913dcfc28b36e536f959814ea455a354952fce9665 +size 22446 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index a96695725d83..3f809dbf120d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3437d1cb457ea3fa476a9ca4e4a1621f53b373797d2a45444cbf2c7194634683 -size 19997 +oid sha256:1632b5d335c9b9f320311646d7718f38c14e67a035466ec1a13b52aee44cf21a +size 19890 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 0de27d0ba369..22e7becc7eed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b2fd4c9ee436681272d0c0aaf760014104f823e51d38bb41c6ca23cb49f9b056 -size 23056 +oid sha256:58ecf8a7416430f003009db7919c5ba26d3edcd1fdfd1eef571d41de46f7fa4f +size 22975 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index deab4baae42f..5dba02319222 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33a93268c454a022eec6dcd3dedc17bd7d8c299eb6044e134363bedece228481 -size 20603 +oid sha256:e55729111bc2724b368b5e6cd2e1b5c634b4aa7bea0fdb2d7efda4439435e4f2 +size 20456 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5dee7e95e9bc..6ebd96e8153e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e57f73421546966ed63ab1f6a17bc416fe29171d65927cedd4f0d3a854bd9644 -size 26597 +oid sha256:0adfdf1fa4cbee58943dec99ef1922e78cbc30497d6fcdf311900491894c3eab +size 26427 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 40dc44535da1..4a2137a28620 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9429d5c911749ed4c6c04d0bb450866e29fe365e2513c19c5db80a94b42b66b8 -size 23894 +oid sha256:3bc3c0ae44a222071ee7fb5dd6ae4d7d55ca38c7aa28550d70be690802aa7ae9 +size 23782 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 21cecff3dee7..7f6387e8e5d6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:45f7d2f3a94490b1ebec62bedc732d1a06641010e686305b7cbde14ef505a2cc -size 27517 +oid sha256:7e12b2e0e08665086b9a265690b74294457d4b2dce4fabdff874f0896dc994f3 +size 27533 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 4eaa42073465..7db0d9390521 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f7954469578936f7c1e5c3255954f074b738038f8b7c54b2cd1f7945b7d3e2ea -size 24458 +oid sha256:5ebd9689a52377427e1da18df20924ad0c8b61bd8c19c8711abd4b6c5dc240a0 +size 24312 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6afc74fe34b7..045d0bed8f4a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b3bfe2bfdd2b3075246a8e36898336e617c92045cab68af1a439a477468031f3 -size 26696 +oid sha256:8fbe98d151eaafbbbdd94263c800ec724180e342e6701b47c0881ab0321c6141 +size 26538 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 0fe00b4627f0..767e0ded9928 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e057a6d3642fa1e35914806aaa66097ccd6da9e946723df3a80bc5e94725dc7e -size 24287 +oid sha256:2de329588faecb0366ce85ac86d9ca484ae809a78685d43d05ecb5f7a5da2020 +size 24162 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3ee1aadac63c..138278d9ec2a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0db4ed81164e6548f50649418569cd376ddb4ba28f0a23a168f58eabbf83823e -size 27371 +oid sha256:02c30f2d644acbe266f4e516acfb87a831b857fcac1c566845283a16abb10e6d +size 27315 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 5436a6c51990..5b45bdd972b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c36985f534f70e22fd1825651a14c9a6bb069100c61d83f54497af59ed424458 -size 25029 +oid sha256:fca7534a1fc83ef0853d935a9e718ffbd66addf6887deaf79c6d39baaab9b832 +size 24919 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 09b0e36cbe9e..28b54ef826aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93520b90aa55c2c745ef2fb20ac7d2588e5520d067713b16faa9cb20f4c61a45 -size 21066 +oid sha256:f647acfd963b4ed0725bab2ae199d78b824c71ff28e27ce297ee179d2bd8ae1d +size 20947 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index e6047ef24ec0..0966a9141c4c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec41b6452eb61783afb9bbb1a02977dd5c351cf204ce7546e9c7248c0189c6c9 -size 18367 +oid sha256:57cafd69e674ce14ce5adfa2cdee54c43dfbc5e18ac0a03ac4d2345cceec0fe6 +size 18281 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 9d1f7c226a3f..c844e2f19d24 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f47f796180979122ed34966abf3b7c8230228c50cf78ad3358bdae5061e93e10 -size 22007 +oid sha256:2e0c2637f51907c8eec569444a10b2b585e3f01b95580a3424df84b29af9e8d8 +size 21877 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 446ed73f096c..679c2a4cda80 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ae6733112d543d2071d7304a3e624ad8dd9e2e65dcd366aec4f30309ee9dd18 -size 19286 +oid sha256:0c7846f45284a15e52889d2321038dc1077985bb22a03bf857330e365fc5725c +size 19152 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 9c8437207351..bb8488941283 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f114229ba13596bd60ca03fd42ed54b0a277ab7191a11bb05f05a1810deac38e -size 21172 +oid sha256:56c0ac410decad0aef0c98e24b433938ad8588068ab79f89e72f04b78085ab40 +size 21012 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 16fcae3e157d..4882af08ccbe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4206c8ff8cbb6810be234786e140a2930ea23b2e42f23f4dee5adbefffc20cb -size 28619 +oid sha256:e70415aabb8c8b3f91a2cac63a6b848243d035398f8f4df01b466ab3685d96b9 +size 28663 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 2ee1e1c862e9..c8f30e67e168 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b2045fce17c6a3cf47a64425fae75741307f2a4dcf99a38fb592595308843f75 -size 20845 +oid sha256:ae6d2dc998209315645986aaff2e3a20d33ca5cf8ec1ec38e983a30d28704203 +size 20762 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 8c7c702e6309..427a9cc74636 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91bfb352dcf6f408fffebc3f2811169acb3c359b4973ab3d44a43e7a1b10d0b7 -size 37106 +oid sha256:4189c1e92999ab3ccf706ca9a36a50157672729f6201605a80e8dec05d2300ba +size 31516 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 87bcd8c7157f..2c3a38d04126 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0398af42fc1e1b2fc2e82fc35ece9d504d6bd259b953366446b64a6c49529366 -size 28552 +oid sha256:2d23d08a4048b8e3cd96544ea82449c0a6be1e525ab0363e6dd2ce0978d29468 +size 22287 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index c828f7bc9b49..55c7d475fc63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c7c1052ab47b7cecb9a2f98a8d149d294e315157bc246b3118b3a9684befdf78 -size 29033 +oid sha256:bfc2ce5d7e2ae7cc707f925d074e51248aef8d46a3cc5fb741b1c751cc387174 +size 29075 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 63f3eb659559..aeb596d92b2f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:315eba1bbc193d555d85259a66fc548d59d47796a5e2c4ab4cc13573555ab120 -size 21694 +oid sha256:74d095cddb9cd83913a9e5f03c81f0fda718945903c6f2ceef93cdf138a5d483 +size 21624 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 449a7f27aaf1..4e9bec3279f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8fc14d7f8db314c6ffa7ad0bef8c03565078f9ecde1051a510c4598692ab2c1 -size 18226 +oid sha256:75ad39a37eae285a89bdbd1c8c471480045df3437c9a697aa875bed0d1da6ec6 +size 18123 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 77e2c5fa8a54..bc121799ef71 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f66789f991ddb2617854056219dd6329f78312eb1744ee63485b48234375ca31 -size 22105 +oid sha256:155497471285003bae0c1d4fae425301abbbba8275b7d708dea371460e99ec4a +size 21963 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 49c9577e847e..bee12df043df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d18a17b67856dcf052533efeb7ebd2487543461e1ee7c098460fd69639550c6 -size 28403 +oid sha256:cfdc0e57bfaaec53aa81db93eef7b3a2a9d856df57acd29a466fdc50a09afa3c +size 28333 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 90ac25ba4b91..b4412964c8da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1b186e43374c152a0a7557e23c93536c2dd74d68e6cd89908dd6bc5a08e4cc8a -size 21996 +oid sha256:7f148ee357676dc5e6bc6ff9e754dad4f4fe9ced64cff6427242b943412f1ffe +size 21904 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 76823b76661f..ee5b237e2799 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb56d296053f66edd10827de704a9c65da45c60dcd0fcceed700c43a94d502d8 -size 37362 +oid sha256:7f1be33cfd2aed77663260ccc434975b97c12cce149ea202e0d644f08b601790 +size 34147 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 1e4714cf459a..d10c31b14de3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08644187a08da3a4159310fc1a989bd45c3975ebeca30e6c1ca63180981bb01b -size 30249 +oid sha256:7372bca8f15b8154117ef46fe8f8c8eaf3f5f250155873f88a0f826deda3c4b5 +size 23230 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index f5a8a2e8feea..045569b1f2d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:913866c219feb922d3e5bc4bfb9b2c5644f89a90956d3aa42be968e732a58e1a -size 29844 +oid sha256:96b575d2cb812c039bdd78d18f6d1214a3fee2a0a834112692f918047a728259 +size 29734 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 11d4aee4e5ae..70941365b6f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:734310b51fb4383f7c73d530c621060be5140277a5be9a46dc60c759568a40ff -size 22870 +oid sha256:af2ac9dd1559e1d5512e1beb9ef21722c3fe4b4b60761ba99f8a0546ebbbe64b +size 22765 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0c7d3bd99544..595eaf86d0e6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ed0e9186c27019e87b5fcfa39b1feaac1e16ca95f49186a558ab5ebe71f023b1 -size 19185 +oid sha256:4172386cd97774936fdf77ccfbc5d291747c1f79a474e6699da19654ce5003c5 +size 19143 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8e387f0b50d2..dc53765e6cd1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9922112142c61c2c5faa98ee8a8084b2031ed041257142f7c4166aaff7fa3174 -size 23308 +oid sha256:ca19c833083d754c271a3ba7567ed99e7ab0db3ca439906a0bf49b1d9d06e207 +size 23079 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index c1d7e3e2b69d..6e59f435f680 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f6d0cafec753493cfb312808d791d186eb494a1c4344887018b57ef07cdbc07 -size 20391 +oid sha256:fe757a2cbd44f57d86229bec9225f382003ffa94e97afc4ecc6a30cb436ac4b2 +size 20239 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 814df860dfe1..3128febd7449 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef2041f0dede8e7758108ee218f374987444484354a86c611190fc9d6254e6cd -size 24222 +oid sha256:61e03c62239470b81f02c21396fc2da7c017990c503c8e77a532ec773312feb2 +size 24037 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 91e6b3d32327..d0e1004d0e5f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:32bd4b43a442f07a8871ef1e9e4ea4c9544223215588c57e9e5633a23d4d9a89 -size 21519 +oid sha256:1240befeab388d3c7ef9e42f7859b227f826a2841e889d08c6e64af1bfafc6e4 +size 21368 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 79029885015a..d0b393c66438 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d0f8b261f0fdce35c01c71fefb18228529fb5029432ff837332cf14dc4eaac0a -size 24066 +oid sha256:d1726e629f07a73c64730f0b6c078a22f745dcfe711ef72c11b143902af87ce3 +size 23887 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6fe953c9eaca..1fad34ccd27d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6a71d222c8c5c3afe9a9464ebf9a78f2754c89e25d5640af74f4fecb2870de87 -size 21913 +oid sha256:952340ec6c53769ee740d3d2b675607c01c80fc1745e1612635fed8c4315eccd +size 21842 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b7e622d382a7..d6789c42cee9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8159670eadc1fa6f4143b9eb5e8a2ce734da416ee7d1ebddff11e201563dcc0 -size 25132 +oid sha256:4b97f21f1adcd4d3d1f89766538fae48aa61b57815d96580d64c5fc76163a4bb +size 24961 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index f37c3193317f..7bf5fb375ebd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d151ac63512126ad0dfe9c24b9f75b7c037d65ebcf0207e3377c1daa6d14e5f -size 22900 +oid sha256:7d49231ba88ef14f37384fdfb86b84d5dbbdfe5c3b5ce3da1fdd512c23b7c1c4 +size 22818 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 1fd7b054307b..0186013c4232 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99b5c905107908320c729bd98176feaff7d372c95cc0733c99e8dc12db1ce859 -size 24410 +oid sha256:ee1657b83b7f0d6b9fae967d611246bc0e02240a1d79ff05fe43535ef0417c12 +size 24280 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 4ccecc6c8785..bc547c7aab46 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42562b1f195ea180d7ff74298ea20c6877bac96b4d4a851ced4b396ffec74643 -size 22093 +oid sha256:e8e4eb3dd0d15514058794800f6ea097ce2ce71674de45f2e5e9e5e40a027e31 +size 21972 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 2f59690e35df..96965117f41f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0366e2c8148ad124b5bd39e65d959381edc47d91bfed1800195b8e7c61964995 -size 25371 +oid sha256:07a13297a977a276548d519b75d5aadc7c7590a41afda0d577b7838d7ef6d535 +size 25208 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c572e849431a..435934fff8b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c7fc3a71a7c71c4bbcbfadf176e66ebe9f1a64a4297cc9764a95651c2bf63eac -size 23075 +oid sha256:8d99fcc7c61538b663dfdbe5cb3096b423e23aa618108fffd42a3e846df065e9 +size 22959 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 5c006ff3002d..7dec9e3aa24a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ea0fb862b942e7ecb3a97d684185ea298739cf1ca0733e0b90b07e7b9bb03f4 -size 26691 +oid sha256:9af16acbed8e6c77a757f43b9f9ac4b1e65cefeefe740303d7e772492e2d789b +size 26515 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6f44071f79a2..4cbb9f0ec55f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:881601cb399339d12399a5d93c9cd302dd3b73e1d58c5b2b92ee8d0c7ccd3728 -size 24587 +oid sha256:3eb005d8215a5887280b57961384867ca70cdca899713df98b5b38546ceb16d8 +size 24403 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index a154f7ce62ee..7d67b961c43d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21142e74a35d1c191304aa0089b44c0d5892fbd8e5584d139f246351cdb4bd4b -size 27596 +oid sha256:96c9dccaebfe878e15aa40bbab9d74ee1029e6fc3d455e2018efa9742a9b11d6 +size 27390 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 82c0122e2b68..f7b25623d703 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e0a43ffe553643dd840a06690ac1ad4189f89a4a7350bb614ecbd3898c7104fc -size 25494 +oid sha256:598e57d7bd2a9a72206af4c0c28cc874398234b12c0e6a4acecd2ac974269af7 +size 25300 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index 07aebde85c85..5410fe680b44 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e86b4e5c62af7d2f534f69c1c8f03f7792e36b7d3fd6db354de1bb4f05c8b038 -size 28638 +oid sha256:44fb1b82d822004539f3435d23f967cc52309bac20246e31f153dc6ff6146e04 +size 28676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 5cabe2ad997d..a502fe61e70b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4737cff655686055ae58b47424b90ec02ec79c24ad3f8183c1999a35acce335d -size 20885 +oid sha256:4be70e07c1910e2446693505d090c0645fb3a25df9e3cfd16199348fb6968fa4 +size 20805 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 9bb09d885006..a9412c1d43b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:74c888a1bdeecff1a09f5d123d3212ba5973cfe471cdf54c16c95731bacd7cd0 -size 37238 +oid sha256:d1f742d755001d6ef1cb3becc9cc8e515a5932f4fa24bc569684c0d3356fcbe6 +size 31589 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index cd8c6599d93e..8e3e88d8ef45 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4443d881904b07892463d105d04ead68dfb652bdecff489b5a2f7af65a91795f -size 28576 +oid sha256:fca913b7262ac96f83955ef7266cc5c9b6e28cd12bb0a0bd62c6979c5daba72f +size 22294 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index f2add8476106..d36b44c50f3c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3983064caece590f43773a3e3d1390ab0451e989a89f95cdf3ed13953456dd8e -size 29065 +oid sha256:66d5074268a2e661c7d5a428188db49dd123c1f18da3478471301877556f4125 +size 29102 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 1a5addf2c33b..8a5a328fb134 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99b3173be362051395264b0bb74ecea6cc73df5e99acb345aa61539ca8fc9c04 -size 21698 +oid sha256:01850207328c633175fe399bfb24f8a444a713addac4d75b825e431d60f8282c +size 21620 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst index abf580f1deef..4bfa206de762 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d7ac52cc4c989d499c4cc4dcc01b54f202c7d14dd5f1f0a534b0c861e4797091 -size 28421 +oid sha256:633831a9d04c0f9e31df11187020ad72828d14c3910e308825015d1f08f73ee6 +size 28334 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst index 71f846f811de..e0784a3570ac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec90423a03a1c9a01426608a6bddf0262c6d41563087fd3aa80f2012d69a9656 -size 21994 +oid sha256:55c32faac4afe5d54db48868e54723caebe0907a8020cddbe9a0bd4fd92dd719 +size 21912 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst index 5396790485cc..d692f361b6d8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f53fb72561589d46abcf23a1df57e007235ee6a780a648f8cb29c3aee3ee198 -size 37376 +oid sha256:649319e7fed2ef64149695738138ae49ab9bca710261c8cacb2b412d096b34f6 +size 34150 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst index 02aad978739c..b20649b690ef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:29cc91768251cba9c16ee59becc0b57e50e17b2da42ced7e3959abe1a9e0844e -size 30252 +oid sha256:133ed9acb7467c2ca70725e23f8aafaa26bed5f57d5f448e5de3d68693a6e0bb +size 23231 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst index 565ffb7f86b1..4a2f09e0c4dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cb74a476da6858cb55fd5599692c9fe8a422625bb3c8c4d1a0685d10d8691198 -size 29823 +oid sha256:d9b8db370d0eb2f67475dc0c4adb6431f8f60862e324e7ed2ece268072268de5 +size 29718 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst index 42d4268f6581..8d78197191f7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd06ee83c7b97fd0788b99823595d0142e8e3bcb79ce229145a23478b45b12de -size 22871 +oid sha256:2f0fffd8a6c1909e5efdcb12b75b5f259d6f099efc94e25f967ce85b50122b32 +size 22771 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index db390fc5b717..1c40d5790585 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c06b560f6f7bb6860ab30ff43aa027314e83fb47c4bc92568e7d59350faa7a35 -size 23919 +oid sha256:387e4f1ad5af2d2220ec8547fe5654fea3e33852e7e23ce8bd99556fdf51c040 +size 23829 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 7b077b6bc81b..a14d1afc46a0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0985b685bc3e28c956bb6bbb25a0e783cfc1cce963ca315ead79af5a2745aab -size 20522 +oid sha256:80a8f3c2c2d248a888bec39f2341ea10b262d6c78d772b5ddb7ab04f348a900f +size 20536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 469f8fe824cd..495d1b0711b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:95d570afc8b51bd9871126173840841690617650f6b9d2cc6fc5ff2e0bbc00e1 -size 24550 +oid sha256:2dd9e4da23e496755ec4576990a8cdcb3344635e7bbf7fd7052a13625dff68ed +size 24469 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1baa2ee9a333..62c236c35415 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:828147a1b14f4483c5532a9d7ccbff11d7ab524613651b3420a95c9b8850c2c1 -size 21701 +oid sha256:e680e714939f9300efdd6fdab5e3e06cbb6a7a72857e02068c2337e28b621d76 +size 21690 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index dcc394c7be59..394412b67c2c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:05c0bf4d072ca6a9ff6282b8cb3eb2d32532df40ee85498603ecd34fd0b83c00 -size 23940 +oid sha256:481390ebb03aa2b1b43499be31653f0c4efb48bfe1d8612b2d55ac940864a59c +size 23870 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 67f10fff1f2d..75368b217189 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b0f2c474502f074bdb070b135bde8861a494d1636fad6a7f873f2601667dd139 -size 20716 +oid sha256:6973926de45ae3759c1565598faf136be957b9a845470aa516a6df748c0dfd83 +size 20657 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4a647f4665ec..b67b09f7b042 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a85da17d4e587fc23121698bfe109817b64edcb58e5fe27d658d78345e8e027 -size 24714 +oid sha256:af60393ce48161f39f051199043003ba62b095984bb65c289b2025f569262520 +size 24581 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 460b6fc98054..83332c4f8420 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:09d6920d8abf27c739e2192c9d68c2c68878192401ee030c3895cbd7e9d80d21 -size 21896 +oid sha256:f3399d8faa7f6fedcdeef1b75e3a2cb19b8a5c4a26aca215d260d963c42e2b14 +size 21805 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 62e0fac6678f..40de28254bad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93764c76a864efd5081c02d24e654fabdd79bd454d09b3382f01c5fe532bca21 -size 25810 +oid sha256:4e17586fcae68fe561dacc135d7be8ac9de09b19b4a0488b4e72495441dd8d69 +size 25572 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 613710718a3e..f17365ff6c55 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9bb5245fc131de26752362fcc663b2d641392f29776c6b9cfbeb8f4707964fa7 -size 22755 +oid sha256:f956cc908c4f6605b51a3247004dffc6c1468e04d4370b5dfcab168888898b28 +size 22695 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 2375d59d6694..9345c5ee6515 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b6e546ba75593fee6dbb3e9c6ad04897119afa09e5ef5c407429763e8760535 -size 26672 +oid sha256:005fc7f9f3cd4c97855c8cc1181fd457cd7a59150eba8e145162917bc5495711 +size 26489 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 443441541b20..9139b9b35508 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3306ef08dc83f0e311d411622dbc6b56a398c83775227fd64d2867f937f84068 -size 24235 +oid sha256:376ab42818ec6c3cccd381e11c2875104241820f4b0b35c0aaf884825fadcc40 +size 24250 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 87a776109920..39ef3fea3ca5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b03fab127ceda5e0ab823047526973591b6e36d09c4abe2354f9ea2855df97a9 -size 27197 +oid sha256:f33eff475e7dbf2b11eb73457d2c6c23fed5ecf3a24caeeaead0f1276664ad17 +size 26991 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 42de69f832f2..6095909a9a56 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:922e32dc79b7010b8f30845a68eb981f96dafe791121f7a2d2420ab73ab40fb1 -size 24515 +oid sha256:0f4dcd2a224f8419b138633963046ef3f526f54c5e05254a21060c848ea11aef +size 24414 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 1b2735c594b9..833adfa15d23 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:557c001fadf3ee5916a46f9d70238ea41560036a82fbe53102e9cacb0024c086 -size 27914 +oid sha256:673583c1e960228c85822aaf7a30bd022574144dccee605219a31335fe91ca8e +size 27737 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 301f46ffeb5b..88da45897f8d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5522810b52450037783d43c0ce765d8102240e6d2b62f058994a1e9222144913 -size 25727 +oid sha256:9c8eec93134506a1bd5e572ac22ee83dbcb63a74b7ddd3c04827e0d2a7f2bf5c +size 25604 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b3ec57b11403..09a82ce0a230 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc0c450bb60cf6e25f757f1b66f2195536809816a5897189c72f86f4f0490fdc -size 27459 +oid sha256:905b0a07c978cddace09e4480a0a641f2b0a16b561196cbcd0b5543a4bfce803 +size 27377 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 50a80022f624..e473d7a33987 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55bdba6f036722fa5a12ad7a5c6b1a810f11c4232dc842da4a77960d1b203b15 -size 24652 +oid sha256:7618d885699cd93f6d5b69b1653d7165bd43b3ab0efc250714d9d4e470f39c46 +size 24476 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 3e91d50d23d1..ed0d0db22b30 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5fac7fe8afc6139346957fc04ed67a8221269b224b439de9492a57e60b4d3525 -size 28289 +oid sha256:06c362acc5122221e2ca538d74790efb1691d43ab284b63a5fb92a8cfc4aabb2 +size 28105 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 0d42f0007243..9eb04091167e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4383f830b26172643886c933dbd24b5372d1a2b963a98ef0c08e0802d2cd57ff -size 25936 +oid sha256:19c4de99f8f9960f15e73e2dedd37f419141633df8881c68a272b59a539b1237 +size 25782 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 22dcbf62b779..a38610bde30f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:27770e356a2cb38676d4b1eb77ceb047dca7d688559282416c22113576f1eae3 -size 29286 +oid sha256:518b7caafa4826ee2d82d1c409e54459951220cbc440f9b36818c09ec8123928 +size 29112 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 6b27b8048fb3..2373ac43eb65 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:477f8cf72d003b29bbc372aa30563f2be289797f41f68758b291c3984786523a -size 26669 +oid sha256:580aa4fbce1074b0b40cd557062911b5ef463826551fe011e9181a95329c3fc8 +size 26666 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index da28d582c71a..c82ca544c3b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca83d89f922adf999a3b1f8eb4d2f63997275d2e54e6e585ce95ff69e9f1523d -size 30131 +oid sha256:045e65310146bb5190b821466c1f5b7ba9702888fa3989b5fad23f22454396c2 +size 29948 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index a0d3ec8bb1b6..e19f6f467c79 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99d831e1941c1fa296410b3998ca9a4998006ffb30f3cc11cecbba659bde6985 -size 28065 +oid sha256:9555657f92e3b51fd15de65e0c3bc898cf075abe5ad669e67d1bbb37c18d63e5 +size 28011 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 8bf7e7c9f7c8..afdfd4842a7a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:503d1fe287c01e267566942a90915d678349d2894ed060de498a1bd9e88d5045 -size 23958 +oid sha256:e5ce4ff27539e233de06682df78f9a7f2d03dd3ecf8caef3698bd656a91cb0b9 +size 23886 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 591fd2884065..051220a16db1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fd8c9772b2a7614061e9ebf3d65ed8e7dca2840db0b6664c3604e49042ac9552 -size 20740 +oid sha256:502818894d081ce766d8275d415ec687e8b7a2c791d9a1639363d40ce08df818 +size 20692 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 571ee9bbcc4d..2429c5801f85 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d446575fb4ce342298b07f032193d8c1f80322f08cc4c9f9509ef4903f8c1b07 -size 24739 +oid sha256:3c3a981f6ccdd1be4d4a0cac062211b215fd388775ecf64944e19430cbbdb9c9 +size 24607 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 13ef225c010b..f8924bb5832c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5905eb17713e9308fdc64f0d7e5d0b3be3e2885bcb4562461da594b097d37ed6 -size 21890 +oid sha256:a6531b2eb0ab37905978601f495874962962005d89183d836493e5c6e1a28d2c +size 21804 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 760b44d40947..36a75118edae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ae19d18c878b9da0a39a89c4466307f5a9e4d32dd3faa0a79276d14e0f17bf9 -size 21941 +oid sha256:c37601ba0a1d290f5058a10075144f4be4aa2920f0c4e61efc261c0c41c333fa +size 21802 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index b3eb4edda1bd..b7898fcc8961 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9393727fcc2588bb64b474c447cfc2f840452401487a6d0b27b1161eec01a550 -size 19154 +oid sha256:c3448f4ffb9c4eacd9390b098815113517f513ba579f140425ddc8dbe4f1a7a0 +size 19031 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index bbe468a96001..99bb0008e193 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:99151ddd39eaff4a60b8ff8fc0bfde19f22d69a733b6fac1baab5d4eae82ddda -size 22745 +oid sha256:0531990a211cbae1e20cd02cd5722fce29bb3e5664d91b3c3ad05945489f94a0 +size 22689 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2e6fd572d4a1..bb37264140af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb013c2983f59c5a5d4f3c84e93d4183da4dacd993cae7d59552090d561ab8dc -size 19764 +oid sha256:324b83e5b37699d4c3d5aae6aa679b5ab81d0f3f25ceb48303068a32298edcc8 +size 19695 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 29059d108d41..3ae9d9beaa8b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:445fcf9105aee5ec824db03eaaf6800885bdf11331553f8f16e8bb84ea3b0aa5 -size 22335 +oid sha256:c2462e164da0ce60bfb7faba4df9eb88b612264a86bdef9ebffdc1d23abd1e80 +size 22202 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 782afcbac1e3..8035bd1834cf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c045efab68d59f507a065e27e112cd54f209e10f5194b6505a67474803ba2163 -size 19728 +oid sha256:79c855157de825cdbf596501f63b954c790de9a8d52de45389d3c38af1d7b2e9 +size 19629 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 756d5ecf0328..4e3cab991c3a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be45486566b11afcbdb230882dc639f71b29043872ce0d94647dcb8f639093db -size 22729 +oid sha256:56d0e4a2f2813ffd4586d53e3122c6533ba4d09bc6f708d3d48c0687ef165d78 +size 22585 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 9ba17e9339b8..80a2b492e59b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ae76c8b6674d434860c2e2dcbffff54df8d4964d659dc8055dc95d2509f826a -size 20265 +oid sha256:ecc95d505dd83bab6f720dddc357edeb13ad9a73c54bda197de3edf2480b6194 +size 20119 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index ea4a2818d625..2c823b0fbb99 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a6caa6b93c2e9eb42186488683b51a4b379b966a0740cb36d8c552d724f6b3a9 -size 24079 +oid sha256:b1a02d44ab3f2ae52b38edb75ddfb161e2c83050a2e2cd86d59c223968c0c762 +size 23846 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 5e4dfec891e3..25712ebf913a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9fe30d60e6af9c884e154403c5988b0c78d7b4b4817d21b978c28a0cab66bf19 -size 21312 +oid sha256:159d4e8f2c613bfaa2b955bc9a614fac05142b264c1d0fcb6d409bed1e795bde +size 21094 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 22bc3de65506..6d017b837041 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1da0212081a03e120a9e67162e902cb3b5888df0639eb08aa100fbf3e359591f -size 24815 +oid sha256:57e3e5fd209dce3685c676b6d8a45901fa81409c1c23319bd0f98f0475433de8 +size 24753 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 21f3eca76076..3348a4794985 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b92e04d809a89741824119941b0f0b9b4f9d345a018fecd4e937081c4a607765 -size 21805 +oid sha256:ff90c33e9aa0487a19f0207e615cd54c3e268f9074ea83ad766e098cbfbed684 +size 21754 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 7b8fa98bebd1..ec03c00deb4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e92406425512504d75167e75b3e40e92b64de42bc19ad62f1b309170461463c0 -size 26045 +oid sha256:aeac5d92c7e75a31df9fb0ee2987cb321ab89a1438e2a282571779872aeb3b5e +size 25900 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 0cd2dd2b142d..56508b3c18c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7c7822bd7f2ddf75a3400082c10d5a207c7a0cdad1a8e72f2d257a0aad396fd9 -size 23179 +oid sha256:56ba2ad414123ffecbd49cbe5a22f1650b3c7d6af9ffa5f4840a47ebe4c0c17d +size 23041 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 03ecfc1adbe1..24437de51437 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b1e990c079f234e3314381d514c19a7c3ffe9beb84bedec0273cfca7518902d0 -size 26701 +oid sha256:66c6f37e8a7304c322e44726745018591e7034cda487aab2817e34ae757a2e48 +size 26661 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1f2534d7aefa..e0469bf17fd6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:704738ab90b2e3f378e58e76b697be16402ae7a4bbdb0e3e5404c3d9cd93bd25 -size 23890 +oid sha256:152ec50a32839e5a7a2d544498b970f6aaa048d4d110b0731411d01508466468 +size 23871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 368236f77c7d..9519b655c20a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:536c3e589270f136bd451945f6b75a42a52d70523d7f45e7bb8488ac3b7ebd28 -size 25973 +oid sha256:6aa523f7e03880dd48c631ac29ebf775be5caebaa713cfd75ab9f6978d3354d6 +size 25863 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index fb04f0d7443f..6a9c024cb3e5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0856a09c89ed3b2cd132da34bbed2ceca9ddde4b1c7f8f5550d5076fc2726d8b -size 23657 +oid sha256:7a4471f80335e976479a2b13936072a3803ebce9d47c48673eaf36d75e96cd32 +size 23545 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index b5474725d3f0..c85b1da04757 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:21ec1396caab14b2b607b392382094e98166dda0b735f6faeaf91a92c0c1f26e -size 26652 +oid sha256:f7dd7ae64cb669976c2d50db15b0c0925f92ca9f2b690546f28fb906849a7a53 +size 26530 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 56af01957da4..0e124f515ec9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:05986c37e4a205be64695427e520f481f95c9aaa18fa9ceac182f2d1262a28b2 -size 24240 +oid sha256:3debd60e60600857f95b2d23311965c9921907e8e64f20f0aae217b8e7181b4f +size 24111 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index c82f7f22eabe..45208d61f229 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b17b52b18f2b424f88bad1d03403d661726871f1c5621b2bafb27749bb60a368 -size 28569 +oid sha256:e7f9046b732af9a331bd2a70604d7b87a8436d603b067a8c6040d852ad33e57f +size 28446 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index a3e358e469e5..9dfe528e4ac1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3f7b5678b4d4b0d0fdaf2a262098a4703c72a366f70b82058b50417abc206e8c -size 25670 +oid sha256:45191d91d160b007443c278b72e186ddd44a80f32f45dbfb1bafc88ebdf5d0cb +size 25484 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 6b24654444be..cda69bd68a03 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:73fa995e86756cb20e7e4b4fd01be43acfc85a70363cab9d69923308e62c10aa -size 29278 +oid sha256:e3f107dd258c4d65e3d2a78d15152fe455fdb33c21116220f03e8d0397421260 +size 29266 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 6ab16330af80..d309aadc09de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d93974d80a4eabf871b71158b6f21f1561889296b130bb5dffdc6975ec988845 -size 26281 +oid sha256:dc8f3a828859dee0aba009bae7510d94945a2400ee5f51cb027d1421424633ae +size 26234 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 05f0331bac0e..180f16aeff89 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b11741c6d77ac14ea0a0ac1b5dbf270d87e0458df3827422208c935f73ff420 -size 23221 +oid sha256:a7272af10e1205a4f30e063efe8920a25a0cde610ab0289e116f630aabe8eac0 +size 23060 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 8150c080fd6d..592242a95fbc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:084596b2e7f504c44a0845ed93743863c474352cf0bfd81c2b16ea4689fad7f0 -size 20724 +oid sha256:857db65023afa4829b7ea9c2aa58acb4b4cb7f326129caaaaab9b2f477a68830 +size 20559 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 64a31ca64f70..2e554b258921 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c299f22a268e6119f58a231cd3f75767c6c8269253676b0f8b6997865d39d2f8 -size 23823 +oid sha256:2d599c6a8d786e265b28e1f10e60c69d7da94e5105f30e602d7a7e2839149d4c +size 23777 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index df6917ab4c40..4ea9402f1749 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4cccde167e0043b5c672c737b55a013c00c70fe1ad79184370806924eb810f94 -size 21302 +oid sha256:764b832290f9012e1d06efa2e2e496f03f39680520da2723293da6b2f0c58ce5 +size 21277 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index e55dea844f6c..e9062b292c50 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc0f19882624ebe4b17c52c20f625f66092aca1957e67bcc12264100064babc8 -size 23185 +oid sha256:87ed99f49f4915b553c62514c44c2cf53b4ead0717d2cdd8ce767b28b235c3f3 +size 23042 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index ed71c1552e1e..32f8c5af81f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:503577ba6ff923bd2f8fe5ac6eb87f4579c2b54bf4e43cd172261233f80e1d50 -size 21254 +oid sha256:f80585ec673d2cb15fb0e77c00d53e311dd5b738a79588471c3bfb4e814d5c1d +size 21131 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 53d1d8e62b04..10c90c5bdfa6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:910f6d4ffa9e1e180bd2b436dc1a8febbaaed9c73aff5fac81d8be45cd2c902b -size 23826 +oid sha256:13e956f4bf5a5fe9a07cbb0092fa2f70d517693d0ad7614fbebf85c9210a8195 +size 23678 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 2728cb0eea38..0bbcfc0e47a8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:40ff36df85564fb88974b0d5de948bb450b5ffe64f52082803b6e25c2f76c240 -size 21532 +oid sha256:fa57f683c5f52be0c4eaa6b4b8865e3bd77ac10d93af4c995ce191176554a3e7 +size 21429 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 286a595c77ab..7e0d0eaa7cac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0365f7e143872cc48c847303675f24d07ed0e72fa29ceaae3e9be6ea6ceb889f -size 28032 +oid sha256:2edd18a40727d97d9cd90ebbf6fcdfbfe107094ccc80be7a3483cc7a7757d0fd +size 27850 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index cb3812d8e05b..1ae1b11f98da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8eaf4903f8b3b94a926601b17eeb440350e9c4800757ab4ba88b95179c50f15 -size 25382 +oid sha256:b8f3a30c0da04646692a2beef117bee0b70f67658aeed5d3e28099278f940d9b +size 25182 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 89ad75abdbe5..333424683f28 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:247f6d4e3f6d3139f412b6a7c0bafea8a4487ea5ff33964b2f32100bdf02eb38 -size 28813 +oid sha256:286f4f9aead65ec6e62921dcdeb0a226997f1f162d3117fda2df450cb5b204aa +size 28747 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 3bcb879ea6fa..4ae12cad1a8f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50581af25b30260c69ab6a56795b61e42d1b31c309c4bcb4d7634b1ea5042e01 -size 26211 +oid sha256:0839fa590da8d2951b21acddad94afea2c6a9563c5cdfa3d348b8e0d2915000a +size 26188 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 82c8daf6b353..1f2fccd818d3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cf787ac3f13bc008e9aa822413dc1be2bd6fc46464e514bef8e131a7a7cdabb2 -size 28006 +oid sha256:addb79dd425c8a557cba9a1b9f3aa5ad299206bf64e4ee67c25d5e7b7dd3d726 +size 27869 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 479241254577..ec0f2dedcc95 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b2b89b83e7de54bcccdc16b01526fff2b51be15cb4843928a4652ba9cd36222 -size 25582 +oid sha256:359bc7c7371e6319c8c8bc0aeb8d8298328da9585b446373180e33ecb56d05b9 +size 25390 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 5bdc7c2f950f..b44c17bcdb29 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:866703d94c05dcf03948484ecc8a7f68b236a1d25998dc03d208c774caf6df6a -size 28796 +oid sha256:a7ef24fbad5b10613116ce678fc406acb631484d818a24ffcc0e2b2e271c7af7 +size 28657 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index c9599434a594..fc5c8e6ca0c3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a31f81ea8f6689c341695a7303af2e320a017d0c000e4bde3de55c5cab014938 -size 26514 +oid sha256:e61660035cccc140d5ea2e6b52a0088abfba7347f9698f22b73040f040947b4b +size 26470 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 6cea98b3fdef..40b3a322b547 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:01b365ce1bf95b23b490a24526643655ae46708ab8895ac81565483bb7615f28 -size 22893 +oid sha256:9f341d1319474c06f9aa25c6ca40df2af63816e375f733e92f8d987b9744cd34 +size 22840 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 5327ac6773c0..d8a85f767ad7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b5e5a6f0ecb1109df1f305e846fbc805eb5126b16382d5b04de8aa1bbd064070 -size 19952 +oid sha256:92b3ee0f4c1a53bfb7fe4235f9249ee3dbf707800484c9815bbcf8e1409eccd4 +size 19922 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index dc134a7f403f..479b1c27c3af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e696f16463a6bb7671bb1e618b32f3131f97c73bc1c58091dd1a5e10350089e -size 24306 +oid sha256:8af8ab335a606566be989d1a8d4602c88b722bace5cbf03e994496b3a6fe2a34 +size 24247 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index fd0809a3290e..e7f11fb0a8b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:083b36f310a02edc7ceb3c4a99268fc2946df1f0744972377fedfc18b370ebb0 -size 21519 +oid sha256:a9672d847d518f7920573dbe8b7c8267bbd692fb40befd12e662255f27ec04ae +size 21452 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 81e25dbceda0..bbcc1fb4c68b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1746a700faf0ea6bf8253a4cef7a472bde600320f47d60a18a61df9bb1424b1 -size 22784 +oid sha256:b48c674541879a37885b34ab58e291f98cdf1f555deaaf9152fc949984bad6e9 +size 22722 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index d24e38843bc2..8e042c8c83dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cbde06ecb94178f2c60bc7895ab8bc3f7bb6fa25c862497fd67ade0ceb2cba1c -size 19854 +oid sha256:4a809ad8c7fe9eb798b5092aa340a90e77171999cf58942941359ea957bb2b9c +size 19855 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 97820af98848..f6d1a3482b77 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26834a2f516a4df0b52b0f22dc0282cc51cd0727fa9e778c0dcf9573b19fbe5b -size 24497 +oid sha256:4abad6c1df77a340a25d296a8de3ec49e1e2839320f8c7b10ad251518b6fcc7a +size 24402 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 13b6f62b1026..46db232bef0a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd2f25ba5effbf75a179b01b787185ea8ec205bc1df4dbad9838af729cb57d2e -size 21423 +oid sha256:2799a5a9d5f414e3efa6660b8575186e68595cc2da91ff8d6a250b7780d41766 +size 21417 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index af9509c35e7f..bb899ee8903e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e6534e604a86eafd6ed2e6f63883bc3c40830fd04641e73d6cba8bf1d08c979 -size 24943 +oid sha256:3fcf082185de9833639f1eb3f8259bf88e84a7b01aab61855e462ba27255f726 +size 24904 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index dbf6f6265bd3..a498e75746dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:22e868beb9c5fde9acd28803a03c59c1819c6384e68014498ea336d3dbec9e44 -size 22122 +oid sha256:5795d66050277bb3b72008380b2d044f14d03066adb84104825caa335e5be0df +size 22108 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 109837abe8a5..70ea87f1b8bf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4013f1b694c9b278731a3d607a417ce31e1ca50018d3503ba1af0f3bea7d9454 -size 26590 +oid sha256:64651d43bd23d29ba22853fdf5d07b7f7bc43909caea1cee559092b3e13dffca +size 26532 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index b50db1d4cba3..123118c35d47 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8aaddc86c0dbd6a900e2770d99ae350f378620f309334436d4d892736a08ac87 -size 23751 +oid sha256:afd24718eb201d67dbdf2aa00d050196d0417373dcb8c60fdddb7edf77773e75 +size 23729 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index badbb69502b8..3662bbfa8675 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e522b26b0db4f8297d4316e697ebdc6cd95e1bfe2fff429e9c5760df44bb3dd5 -size 26027 +oid sha256:7b23af05aeb106c62560c67d3e4b96a042eb35420e0cbc8f4c4614de0c21e33a +size 25988 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 45da9254e6a2..42915d8ecdd7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:32efbd8d50f1679d4bada5c54af3e345a053b949db4e657b7105bbdec7d8d866 -size 23745 +oid sha256:24bffce6fd0d50b35ab8dd3d2466db7a1bf04f524b8484beb1486be3b871e7ce +size 23744 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index dcdc314bc32c..3163ff2835c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cba25f6ee4879b6004e41d3a4117d7bde677230c81a98a46af84280e01a47c6b -size 27578 +oid sha256:417862cda75e723b79069a84ef28113615f2496e5f5aa8e9a96f7dd246cfe9ba +size 27509 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index e0e686363692..067da085987d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7794df0fc9c452b4f94b0b16bed29d2e423852f31ff690642a29675c3d7cc272 -size 25307 +oid sha256:5552bcc1a20d6185d77464d6295e12695de9ea1cdb315ff4ec8671770bd12d24 +size 25261 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index b830ecfc281d..b7873dbf3a2f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:065a9206b5117aee291d354900d8d278d6fcb797c099c2f199651e5752e27692 -size 25938 +oid sha256:9b322e054775ae42a8370014fd94f2ffdca3a590895b0ba4409780f73ee3cd05 +size 25891 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 1ad2891a7c92..3120d4762289 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f38d1cbd23526e1cee68822ea556f8ed46eb58195facfe281a8638b7f5b55888 -size 23651 +oid sha256:be733df34567d868846f79d776e600c35187c11ede9c36bc99a85e2489de3c13 +size 23623 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 632a4f9cb97e..8518c1a10a60 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ddc30a7ecf1d941f36e9ca444bb54b6203ea13e9f3e5ea59349d24ddb5030f0 -size 27834 +oid sha256:85a13c7e3ad3e5533562902a2a3ddabdea135ddcbb4718a3919f70fe1806c34e +size 27701 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 7521ac2bcd95..be580f9ebb7f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ad983b6ef52ada7ecc402c9a0e18ae1e25659b7ad9d8556c7f2a1f2105d3598 -size 25176 +oid sha256:8e71a4db88799e83547533fd819ceab7f634494e821c4aeac4611d9a4914e573 +size 25147 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index f6cf9b9a00d4..9670c87d263b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28dbfb14d1cbffe408dcee701850decdb18789d83a4d1bd8fbe0cc7e1f84984e -size 28295 +oid sha256:adb967c34f1a467c6fc04b2b292e6290c9d0cab2a6f4a1472b79bb3ac531cf08 +size 28255 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index b1f43b52d00a..01b28fd56f7e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ec2be008525922e74f58a84100a6744dda0bec328702dfb621320195d41ded7d -size 25835 +oid sha256:cfc1deabfc24376c1683b342aaf198c6cfdac53d2e101e2898fa9aec5cb55ed8 +size 25818 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 499a493b1c70..6c01d8cc4215 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f7708b437c5629d1d755f266133f03c37a98c8e05308be07731ea6fceb98fd8b -size 29595 +oid sha256:c102c7c00f69e8595d40ef3b379f074c79a1acd15cdd3fb2ccd23aa22e97b0aa +size 29454 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 127d4bac949d..db87c61acd11 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:19eaa48c274d2301cbb883f339ece634f9f4f5bb25e9e51b8ccce149320ea27b -size 27669 +oid sha256:b1dba93539f8e6e353d18e8ed5910fa7c48a44b32857d19ac8145474caa1e522 +size 27646 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index a525d18406d6..c7947a420f65 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f10cc5a6049d2f037b08fbdad3518f129a5d631d74ef5dcacd865b5d89588d00 -size 23691 +oid sha256:b2e55e82d32089065a4da68c54d6797da964a4b32d0545065cb204dbb71a1172 +size 23587 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 34bfc7fdd669..969c3e9dfad0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81c8862bc01f676015e6bb9fa415b5a4320846eedff6c022c0416b641c182004 -size 20560 +oid sha256:5dd8cc534c2196c55b7e159aa3a445c55fa829dd93d8d9fedfabec9db3056af1 +size 20510 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 61e110bfa7ef..b9fe55364257 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be0076c9af8c13e3e7d447853004c9f58f0f92540695b57a4e706ff340f18470 -size 25121 +oid sha256:d2fcd07084801431346913a9149fd55dba059d224a9332f39b7ae754c1ec10c7 +size 25088 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 199b86866a4d..b06fc7450ed9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9797a226f29b42a862ea34dfb9dea54445d0163012fe5ade6299567e098146b -size 22239 +oid sha256:ef677da24d919f08720b70d3e3f72d2f1e25e603394ef20fd1bdadc1b7328572 +size 22187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 2419788c6dd6..bcff72c7722a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:05a3528ba72ee4de9936c22a5f5a51974c21e98b00a5d12b2afdd995dcb12fda -size 23657 +oid sha256:7346eafb0d24271879f1584af741bdb1ae326d6325a6c53fa33c2f8888c70bba +size 23609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst index 1c83e9a8e200..dcbb2bad639d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e6108464cddb199810784490600d8a45767bb10bfdfb578a8f254fc111aa7f49 -size 20580 +oid sha256:db4a31ef4ea9a6d309b1eddebe17afd62736fc0fe61bb7ac810a474376bd266e +size 20609 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 283d558a6758..15b36e5cd733 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:994e03ffa132effe99967ea4c67d22844bf3162424efef3da3f40cb761ca2855 -size 25162 +oid sha256:7e19b72888dc64206321abc4eef51ac8872b6a5844710b68591a84491f13ec62 +size 25097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index ad4ed4ef1ce5..c8b4b536745f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1cdf0d590d8c83ec8fb617e9afe3bc05a4dbba77cb648861d5821808a87f2c86 -size 22045 +oid sha256:1539b40e1821184cc24f702a07d4fa06ca3c09415946e3fc3e03ff9d2fc208e1 +size 22024 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst index 985ae82d5211..a8a6bf02857b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0c19f2080bcc0a305f544e0771f5e45232ceecaccbadcad543ce7b1d9ca9944f -size 25738 +oid sha256:6b2d7ce2d1f2469f32907d31d3e1b603fa52922033bb69998bd70563b539a66c +size 25554 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst index 0a6ff42d8aac..efbc01ee65b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9280477efe89019c3b6e6fccf0b2d3e96729f97081956ffb90583325ad296aef -size 22646 +oid sha256:6e1099bca59689d5a1ee22a70faea4b2e081695656ddcef3cdf56cb9c1766c60 +size 22634 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index ced9afef1733..5461d9bba71f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6192e9d82a671760d08ffaddc9659a637306988a6f9e268c76fc86632ae2a015 -size 27179 +oid sha256:5dce65453c850b217975bf9fd5f7bd16711919e321763f43683b3f728b962951 +size 27152 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index abd4cba2cfba..beb953c63dc0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d173a936cf8633d6103ef1dcbe1c099b3782e5cde428d34b7283a48ebacfdf76 -size 24609 +oid sha256:81ead28f98fb942ce081cc7c32a44a18a54cca5d90c646f1efd7885ab0a3ca0e +size 24556 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 17c5df17163e..b0172159dad3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:87d1582767f14e7a9ec98490b223a40fe332232f59384be533ffb6c18893ce6e -size 26937 +oid sha256:d28fc9f836686fc5b65e60643f9694f22780eccc4c53e4f42b8a71b73ffdff1f +size 26866 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index 18f9525c3f8d..b14c42dee1af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ad8aeb6d568867ecf9c73ca6c678bf24652ea1d422466bd391c0829bb8f06306 -size 24706 +oid sha256:1bae4ca05616c971262dec732e89be98a5a34bd64e21e33acbf46f342975b96e +size 24687 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 4a2c7eb2340f..b0a7c12ce2d7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:27f3583dc59f09fc71effe4b787388264a5e39a609a2c29eed7ecc0d988eb38b -size 28473 +oid sha256:6f53eae273e0dd929762a284d9e2aa3071098d534ce0734f09f2bc3b29f89442 +size 28414 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 20ad8060f84d..5459274288f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:756bd31992c836b6d1de22cd841b465e156fa357a184d74d672ea25a443102f4 -size 26293 +oid sha256:c34500634c4eb64a3ff27402feac1964f0f8960b61b980c44ca0184f60ad22b8 +size 26224 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index fe2e5e2e57f6..35cd14281b8f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5c74ab313d79c8b0bfa203ec609adc992c32d183ec01a1b46ad06752f20cc21d -size 27082 +oid sha256:dceb2fcae9dcb1dafbd9dc535d5b9d7214eae830e5cf0ec8878d89fc477a20fc +size 27053 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index cb11dd894c09..a9d9c91ea1f7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8f3ce952da35e0a49ee866555a0d6c579e29e41ec778e35a2fe776768bc4799 -size 24644 +oid sha256:b1c6fd734b2837153c6743276de90eca3bd517c00c5e88178d3f2265232465f9 +size 24618 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 91587ffed499..6d16b9040c6f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:47fbbe855aa655442533ab62ce0747393bf2a5e5c9c65e7bfe3235849d27c3b1 -size 28630 +oid sha256:f74a521dddca0b3a223618441585b27fbd5da2b1688881bd6ea23025345eee01 +size 28530 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 1a7f8b484bca..f33495647182 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9fd9878c4ad2fed3d2333a233f24a50d1eb287b9118cc25c2069117c68a5c44d -size 26172 +oid sha256:adcd100ff10872badaa3de22a12c20c95f1742115e25952b02e05ed3e40e6214 +size 26142 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst index 96f0322f79c1..184f530fb3ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e052fc6fb1ed5d0f8e4c6286876f3fdc5ace7d51bdc879111a24e49feec68fee -size 29152 +oid sha256:ff7e6cf7ce9c58c7b40fe89a20831a7f9c58a6dc85a4132dee700bbeff907c4e +size 29075 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst index fe20bd005390..7b8b056330e6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b7d78ea5167d52d790a21db5f927ee5fa8ffe06ed772b60959c3e820f5fa8bb -size 26741 +oid sha256:118074dc52fbb322102871d1ec457e7f9f2e5eec937409abc103436e1901868a +size 26732 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst index 29614750d433..1b55a0b446a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:553ea32abc98e4bf8bdbfe449d543f9446a754654843acb17e9bc724df28cb1f -size 30340 +oid sha256:774617f118b698d09f6b10096a07ada3ba193726435cd92680d9e1caae92ffa6 +size 30357 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst index 6e91f2729e3a..fb34421f192b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e36c157cbc75cc86cebbcba34f8128ce18220c9ca286c3b6e5647d4753637fb1 -size 28546 +oid sha256:fcecfdf045efd73fc6a458b8eb64e7e93c5569e9e02e912b9f8dead1f0323a16 +size 28518 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/kernelMetaInfo.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/kernelMetaInfo.h index dc4634e6c6f4..4978b63f11c4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/kernelMetaInfo.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/cubin/kernelMetaInfo.h @@ -24,106 +24,8 @@ namespace kernels { // clang-format off -#define TLLM_GEN_VERSION "2d2032f9-dirty" +#define TLLM_GEN_VERSION "d46391c6" #ifndef EXCLUDE_SM_100 -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; @@ -166,10 +68,6 @@ extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOr extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; @@ -234,10 +132,6 @@ extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrC extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin[]; extern unsigned char const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin[]; @@ -286,104 +180,6 @@ extern unsigned char const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDe extern unsigned char const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin[]; extern unsigned char const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin[]; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; @@ -426,10 +222,6 @@ extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrC extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; @@ -494,10 +286,6 @@ extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrCh extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len; extern unsigned int const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len; @@ -549,104 +337,6 @@ extern unsigned int const FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDen #endif // EXCLUDE_SM_100 #ifndef EXCLUDE_SM_103 -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; @@ -689,10 +379,6 @@ extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOr extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; @@ -757,10 +443,6 @@ extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrC extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin[]; -extern unsigned char const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin[]; extern unsigned char const FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -893,14 +575,6 @@ extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidin extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -1061,14 +735,6 @@ extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseV extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -1613,104 +1279,6 @@ extern unsigned char const FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunk extern unsigned char const FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; @@ -1753,10 +1321,6 @@ extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrC extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; @@ -1821,10 +1385,6 @@ extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrCh extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len; -extern unsigned int const FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len; extern unsigned int const FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -1957,14 +1517,6 @@ extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSliding extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -2125,14 +1677,6 @@ extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVa extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -2751,7 +2295,6 @@ extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidi extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -2975,7 +2518,6 @@ extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidin extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -3000,14 +2542,6 @@ extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidin extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -3160,7 +2694,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOr extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -3408,7 +2941,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrC extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin[]; @@ -3421,14 +2953,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseV extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -3581,7 +3105,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChun extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin[]; @@ -3770,7 +3293,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunk extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin[]; @@ -3859,7 +3381,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChun extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin[]; @@ -3956,8 +3477,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCa extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGenRopeQuant_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGenRopeQuant_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -4050,7 +3569,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunk extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin[]; @@ -4139,7 +3657,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChun extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin[]; @@ -4328,7 +3845,6 @@ extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunk extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin[]; @@ -4405,7 +3921,6 @@ extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChun extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -4570,7 +4085,6 @@ extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunk extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin[]; -extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin[]; extern unsigned char const FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin[]; @@ -4663,7 +4177,6 @@ extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidin extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -4887,7 +4400,6 @@ extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSliding extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -4912,14 +4424,6 @@ extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSliding extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -5072,7 +4576,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrC extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -5320,7 +4823,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrCh extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len; @@ -5333,14 +4835,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVa extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -5493,7 +4987,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunk extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len; @@ -5682,7 +5175,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunke extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len; @@ -5771,7 +5263,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunk extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len; @@ -5868,8 +5359,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCau extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGenRopeQuant_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGenRopeQuant_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -5962,7 +5451,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunke extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len; @@ -6051,7 +5539,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunk extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len; @@ -6240,7 +5727,6 @@ extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunke extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len; @@ -6317,7 +5803,6 @@ extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunk extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -6482,7 +5967,6 @@ extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunke extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len; -extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len; extern unsigned int const FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len; @@ -6549,3256 +6033,2995 @@ struct TllmGenFmhaKernelMetaInfo bool mSkipsSoftmaxWhenPossible; bool mReserved1; bool mReserved2; - bool mEnablesBf16QFp8KvKOnlyTransform; - bool mSeparateTransformedKv; - bool mFusesDsv4InvRopeFp8Quant; const char* sha256; }; static const TllmGenFmhaKernelMetaInfo sTllmGenFmhaKernelMetaInfos[] = { #ifndef EXCLUDE_SM_100 -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7ac4a5d0ad1e2ea76e17e5472fa03cfdfe2e48bd60e2613e8fbea78ad6cf1470"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "96809c71bd2400fa84dd5f67b2448a3a332a673417e2eac2aaedb1fae32f44fa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6c38736967a7d028fce178ea7ade4bc38b848284ea7945b6195753e4ac0a8c13"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 209944, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a7846f6df6cba2435510f8d5e6b4e2a3dac8799fc35b92feb94074147cef9ce6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c5147543d1bd42cb7f0e6e1799a8f0b639685e64a83197ca05cd17b45fc4d9ce"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d2e9869d32b03de93f2501842d54676e2f145d293666b946d08d5e486065da56"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "535e2c2effef99d9ca9f2325c92efb26f27c170f74a539347bbd92ec9a2bf546"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8dda63ede5dfce839077c5fa1f3e9412f5da095d838ae4a35feadbf9832a5d52"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 171392, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ef63e6d63a723a588b371b0c7836664a6003350b2324d07f3b8c4e7f58ac9816"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1a00c7d0ff9a308ee4042b3c0387dfb56ab838ee13fe213fda03f3a35dbb627d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 184448, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f938ec7315972aecd531d2c5393d671a1ff870605ab94d70382cdfbe78f33f7f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d511e461484fd44251f4ce9701c42b94b43e2404bcb7ea0f77446d62ccad541c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 164864, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "eafe5d29f63771abe46a17dae9b7a2b97ebaf9e4a0f510299441cd1be30c2ee3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ad1b533d053000185e6a9761b86b0468a4edc21313fd045f8b8319890bcefe93"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2ec7678203a18717c48eb877af8d058ce4fb673bbaf427ef155d4cf02763b185"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d4365db77778a4ce73dc5d91b2ab438b878c5c18bc47700b8559b83be18e7624"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "17c6f3c7f539cdb74ac02adcbe99df9e5e5c266eb2d915d141482cd7da12e372"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b4a0784a4ed8de799e94afa366439f5de7281a4ebb2855fad84d740fe13e337b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f87d8f469f588fe083fd8b4e4bf61f08dbf572f0d328ddef23ddf65c8b2dc563"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2a67ec482bd7845ce4e567b6aa7800342bb6f0470d821fbd3dc23ae2ad10c3ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 171392, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c9726760fbf18f958f9b986ed4c84b2b0f82e04548b9338f4a07ae229db1c576"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3ef3eeb11d0570d4b207dafd6b5a8158bbed5716e7acbb6d7a6c6d94a0790e62"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 164864, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "28bd0fba818edc58aefaee052284b70981803e599bb203d61c5729259d85b776"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2d18f9d7ac3aad2a2c3ea0505650ffd7268382277148e2031ef99746af8b463b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "deac596ba0db430fcda4ce036a0cb78d8e6cc84a4835c7f67865371077024c4f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e31ddd5c9aefb89282346e4497eba0d763447aeb0752f84fffe242610c5b46b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "af1222a6d45c86c0d485b13cb2a0f6a75e29332b69bc85720e55a66cb5d16dde"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f7b2be81d736ac697d961d70816964eccd4844e27d7205844bb8277654e19a11"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "34447293c906192007e0a832e0779dae4df42028cdd61f7ceeb63bab475087e2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b2288f32aff1338d6b8259b145a3a2ee2c75439844e86e6c25e2bbfce91fb0a0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 171392, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "4ceadfb2e1f6d02980456f8ffee428564d3f6d1ceb7597307d9d5b607027d7e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8949d8423544be3f8af7219620d760f4a0ea16f143789bb74ea6edf82784f7d7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 164864, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fb48e9bb7f1d0c1eabafb5c82f67f10551a97c00a9ab0a6974488c77af35e24a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2bef3ec5ae7e8e957919ad144a5c7ba21cdcbd07bd285be735e5e5ed8bdc5de3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "cbbc8e962d4299e162800665cd21bf4e8fe317576020c8b18c645d8fe0bbcf14"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "30cecd4e54ede5d751d016a1c9656d6f373425deda689e05d710648a1f9b063b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d48c1206576b42028a706ea7d390dbc549e0af5662d4d8c21bcd1ba17fece110"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e75f71b3d010338b969ce293735cfbdf7af96a625e5fbead0af2ce1430780b28"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5217ba1476428625352721c599941c4b4cbf237cde26adfe2e5724b910f16504"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0df7ba7a6ec7dc7877ee3bce3e635d20a16a58d440c838d3de1dd6a3163d39c6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bb74b3c3c9f5f0bb343ade783b493fd0cce1359ace2925092e48cdc08846110a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "93f54ff44fb8512f2a1db79b0cb5dadd46d7088988fee5387d1935e1a44d0f14"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "14ac6fafb621d6c4be9a1ff153b48c51622b3cf25a7dbbac9c040048a9f68b2b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9d9d47d3415f4e27af3e120d294223b7b07b6a106c6a07fb0ae301828a50dbd8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dc2a12734bc9a959bab44c3ea2f8f3fb9c1a56828470751847f0bdb9281c693d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b74ec4102a62182b10bdda2aed5245655c8ab89966aa03b5bc56a8343a1c03ae"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 184704, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c37a4e81164b71e71cffb171169ba75861a5a274d3c5463feed00bdf4ec24791"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "da27c3676ac70e1a2ca98c663b3816fc7659035b1d7d4c8ad219447c3ef21a17"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 176128, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ea04dc318dbc6dc0f1fbc478f91ecf353ab278dc61e69d5a2815491ca4d6d43f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e3a229bfe7529ec628fcb7d929aeed6cad0914bedc332818f12269d861c65911"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d2a9a19a663e565829b759b78fdc2e864a4621d9f751e538e546f35d963cf64c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bfcaa804ec01491f1c539cb880a3c4048b0ce3aeab69fa37aacf939a404ccf7d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "47ea881f25ced33168431cdd159898f9a873c501ac00c6fe74d13db131a16f73"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8473a4547ad9d2ffab98ca22d99f99246fb8d3fcc61546badd0b2b17c6c3862e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "69dd63f00f00ceb0c73cec95f46267f4cc21e773f558248b8ec3b3ca25595231"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ddddceb525022d9b558d2088ee49aa5a670e7b31709096e76c9729de8c07fe4f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 184704, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "4be90dfab4682f688897da1e8596bdc9d7cd7b40a425cb260fe43525a4e78edb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "4f7522e7f67c3b166080425b1a71ca80acfbf63d71daed1a22b3f1a2750bda79"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 176128, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7387d80f376edf2b117aae6161646c7fd9dae82c656a9ace9aa1846ed6208d23"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2fa831424fd0a2fe466fc56c8f6f1b2b3c486654412a158fd1bac4dca3b59517"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e54d2ae2eb8f51a4d57c6456124be455f0ba3ba0b52224a5882cc2b6440d2693"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d0c4d88ecb2daf9922150646732658ec9d8d9a48007c22e82bb3a2ed28036841"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "40070f71d787dac3036356362d1fba7d651f422ed1622462c64514e1390e9a46"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 165912, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9c77745a022c3b148abb8416699f342836a62e15f5c0b7736547b01227313995"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fc86dc0b8e40a44a54104975315b3ad5aa1e03b2143ba8bdcad184be652e82aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6d930395e172950980ce59b3257932ccbf250437c4fcf9f8b1b373fbe8493c99"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "553cbae1e3479d7568cf075af95c81bb2be645e32b660f7b2337c760763c68c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "4d7b66e7ccb7d4bca164330919c6b3199b4ae5b4671080a13fbf0ffa5e198cba"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 126336, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3ad0a567fe846ed5118c0371005a563da4a00db5ae8fd0e5a1ccc2b6bd30c987"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "afa2391fc236e83ffac3d2e59e673705b47bbe1a9836e7f0826dbe8e37e5470b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 135296, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "46e70a1e7ad183189e141d4b570f9a44b60148d71589c0ffbee9d354af83ee08"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "940e1870cf45ad1264716e9a8db4e0ca1bd4c8289fbe94c0539b45d72f1b7a23"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121856, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b92af7aaad1a25f0c7683b37b92c17b803b877097282ee5ea79456320183cc0b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "45961f5f22372682ded8723709116e3889829d88c5edd4d8e908f3f6d047cefa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9e01faed9f19c7ccd29f0596cc63700a22eb6df7e58d8b93a03b31433482227c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fcbb66f54d53fb39057fb60667dc01e15e55780e7bb8314742f91c4016d615c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "90a868948117dc61da90656c504e9a4d219b2fcd109534360b167f03296f7574"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a1334bab69648b8afc70974592d838cbe356500707a013bcec62cee7c40f6c5c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "27021cd308d0c863be03bdec889ae42081b350a4e760ad4b38eee49aaa7cefe1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ca163732f12c37c0a0dc2dd2abedd49d0c13e4bd1b79be5214688f30c0bac035"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 126336, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d74353393d0a37f8200404f639744edf3b6675747fadd2dbdce5d529fe6d95c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e020d0fc97f7bb9e2f24d0556deaa9e8de12e39773041a7fe189d4b46fee75e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121856, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8e42433cdef36d5618edf1082f0537f06e362bd76d199e3126b23bb631bbc9de"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6f85c0504baf42d614354a93af03fb932326f28e7ec4b7fec6a1dad161a96add"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "4ab84f7bb9ebc16e2fb8c21895022d1b84fddae5201855b32369d20a62d451bc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "721d44a6fca2b043871b94a7f23ae3b36e50ef371526e47489fdc4e3976a18c5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d668df4caab6aecd19d7c5905bb7fc18ade2314daa2783b115a039ad92ec3607"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "389315be7546c091c4a061bc90e595bd03da2cd53bb88583a2c0651d8c5f845c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f59e8527415f5f0815ce4e8d72f60f7758d2d960054ab0529e6fca204c84efd7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c8a146001682ec730aa46cd18b67ddb211b43ed4314174442bc1ba3e0b9e7df7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 126336, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b971f49c0f2dc8b3f740803d2a78f540275f697d7799b68c54445b0749d42197"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ec1fb7d0425c540d521880ea8727e3c70dc204df541ebd038a5368405f5feec4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121856, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "14c16599f00b887f2afcfb2a52addd82fb95df43da09cffa6864a040c5d196e7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b5ba7b03ce120739c5edc8e7264504294f478db665f31932e2380d7750b21ef7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7c6d805a4cb61c1c3a4695cd2876d975fd44e9febf7c5fe3847409e45473181f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "aa34fe597c8b1ae3162decb0094d9ab783c822ce020de1858a32f2200323090c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "22cdb5b4b273473571e2ee756870c007dee8925bef3a6deda5e3848cea40d9a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c71d7daf9793cbb817a97f7a2d0289398f3614719f2e5fcfffdfef943cf534a3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a3379fa2946e45f15a70ec2b1fea33e15e89aa2ace1c089fb26e1c8f97d33567"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "847faf9a1d5bd2aea86746810ef500ef8312dbe7328da4da3b9724d104208a70"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 182576, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0e9cc2a5542fc608b7232c63b04b48acacbbbfd67528b40b8fd4967a0660fbd9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "cab0eaa58241355ff0a78c44a31b37d9a25b1088ff3eeea192e211c1486db088"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 209944, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c5acbe452a4aafbc10c3e8565e94c9af360f8d94087a3edea6bbccdf1c389612"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c2b01751744d6524aa3701c00cef6687ef70259868ddca9e9d5f1e1d48d997e6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182592, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6cca553f2c9166dce8e8bcd67eb7b532bffcfb81880e88bd6329ae3a39cef877"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0b9c86bb3e73ae58a6f2868e284fed4fb15faf1526d0bf8c8dc28b43cfed416e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "daafe86a7152bd6fe61f4d9a55346ba7e3ca48c38561a0631e344860ca0c9b8b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6dda127b19cdcc265c2c06c7444d2897f831fdb05423846853282b52ff167cee"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bf11b0c6c61d51596ee46c7f429272dae7fc75b931c9247fd476bb6ca2cc274d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182576, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "eb22df80eb2c78fb29f6b0a657b1db15350cf9b084dd188f94689c9cc98c897b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 182656, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c1ed08b890fb01ebd5ce0512d9e1a9b22704fa99041987504d70b59812112586"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 182544, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "bdc2a3c27c4fc9ecc25986e781b8165fe1397cd0949ae3fbce82c5187be3c909"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d555ec87c4a64edd503112d549a60508dad558db9c7d33fc79765a4906276d1d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "360506a084e8f2240e0b7042da30db383a512d9adc2a53efe4a991274071798e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 180352, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d05a01647647be71854fcbf9c71eb10013eb9e85fc0ee6c84bb72b7f7a44cf8b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ef3d44646e380c24230539b4de35ab332b082e5b0143fca52284b08f7cbdb7ed"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c37edbecb8a9bc097e0cadf178787b4e54ac8487baaee204e906bf02df1f5b8b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "57aba5baecac0bb5bb66d2b5f0d98f7f8acab5d7e44d549df5238d74889c4429"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 182672, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "662cac92f577df03fef8f98398762c86d3561eea4137b50637c709449c848f94"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1bd617119ab23eff727a6f73045a0581cc3a2e5fff4643a384bfd745aaf20153"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "38b04240ad63fcd0cc2323a18cdacdd27bdf706a85ec9b72bd48374f6da31a61"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bfe5eea07d9a6905a5f24987391b43ad5d433bc97c56a6d2127acae67a67d72b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f703eaf8766daa2c11cfc1476cdcd9572f006dff1d37c4bb5804f5b47cc644c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "44e68d7ed3b49b6d116ba8bc09ac707050051a1b123c48f625b3bf555e22443f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "07c87336c3a8ea50cba5f266bbc0fff4eba20a8c645292dfa8bcc7997ba9ba7f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8db527d8c7abb1fc94126cea697b912a0d15849425ab467f2b5003d1d5d9f83c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "031dfcb4241fbaca8821a10b6fc6f5c98c483ece077c050191f942e705c9e715"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e91a3d8b04421ce7a2231f42751125d14f538f718fd893bf31d18a1bce84d4cc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "618606e7599bf77732c27cdd42afa63265daa769c717af964655ec8dd7f1b1e4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "08b6a4317224904f60e9d380524257366b48ca3236751a7d873a727e38c7f365"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bfcc5db308d912dd1ece42c3f89a9914c8c0a036ac40095531204c411fddf939"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6a011edf48aded667d01b3dd36fcb149868bd1c7707af82a6e4de65703fff744"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "51e2c5e78fe9ccc70adc7195611263b50b10736bb28b075923924fe7d8c99ea7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "4e32e28787c9678a252aeed12cfbbf94b4937357500b8c5e4ac4d62e74c9f4a5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a3123d4f38b9400d5cfbe0fddb463bfb9d941425e6d5e4a05832e059728e24aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3d594f2e0635c221f4db44afa39ecc30e4bd01a77f2a1bb3718c7adafaa36679"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a8f2b53b062f8381a1aa280739744673bd49135f7f081fb23c6e69c2dae96c5e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2c3aa03e61563cbade1c7bf9534c33bcbc9b6896021db7b6936c66c87bff65e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7b33f2349f029d592143e7860a42ecd1c317a997353e8d4583459c278de47cf7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "08cb2d00ec041f45246ec4c0dedfc5a479405b0b70a87f05c5c3cfa5ea7802a8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "63a1edf158d44b437894da98e382e8e16d33ee0dcdfc43a5867b8a5a1d692321"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "225e5bc8aa8cfdd074a6271949e40da4f9112addd47488708d7f5ec6b5104a67"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "82704921aa29fc2bd0cc7eb62e0f54dbcded7f7accfada9272885ba49d56904e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "319b526bbed869f05b6c17a8e376e06e1da09905ef81f77bdf2e5d54cc043f32"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6694b0fc1a4894a26bf455051dc415a3f40dc4a0b86a8762545e048766d414f1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9c72f50e72b10bd27df9eff33cbca933d6ae562a49f2333201303e4a8b75e261"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b8ab1f2c1673ef4be86cda9f3ece0f7286e236ffd114660b6ebbf83bfec43e0e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "da5fe2e3d80d4e161d1c27562af74a31ad8ae5bbd8d3f63d019b20f49cfdf0de"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9318934945037e2f2776f914c03f153cc7fb261b3a4189f2292cc8ba4cc6a3ca"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f930b9283f411f5771625ddd1a0d4c56bcd961cc5ee89a6368893a10db1d2a8a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3029a200cfb141c18234b8c84606f1ac99230b3531e236479a3c67b70459fd37"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ec2b497220b771879e0b6c689c075bb05aca51dc33b9679ddc2ce4ddfcfab49d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 182656, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "07aebaf5fd7375430ed4ecad6c8eb5fe2e92bd1ff875e5bb3c07f410eac6d4cc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c7428e341d41c2e8ecd81ae0205fba0dbe869b5bb4705d869ddda8a8a9fbd3f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 175104, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "41f88328db9f04602105cf73f50b638987fd6935994a5792c0bd9e33723bab96"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "df4dc51ec1d55b821e943e662fef1d8458ff29148c36dca20ee088a10c6448dc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d81211f98e88d695f92edb3ba694edabf2f40c3334cfc6987f974bcc429a4b54"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7e0687f3635fa51c3e8616d8941241b30282916df9aa01c29c62a427051dbbb2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "670c638c11f71d2e754e81f4f5564c09ad3c58c433dedae9bf96e373415b6425"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "002bf0751a716b01c83c350732e89e208f5e71d19257feba2e2e726f75210c8f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "853be711a90be4b133f526721cc2e94a3e4a1adc282f06f760f1822fb3b5f567"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ed63ac202edce8312b1894b84d92b5208c33ae0de68ccca3266cedc15000820e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 182656, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "360fd5df6222c87e57bd9e74f91827c8922bcc05be3098ff22df3c8cd0be4c1b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6e26020b8a85739da12d22d7d871d4c3f1b6e00dd4f2110c0eeb61abdd7d9692"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 175104, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "eb4127e32de7db99052ea8d9520d64f6c4646de6fa9a6f0da4d28871f4c04e9c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e0f348de8058cff3a4106462ea5cb4bd99b34c297aceb29b24e77a7c17afc7c3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b8f0d74629baf35199500cee629659fd6784356a140cae3d8460cee49293238e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4b72e22c459f1da588411afcb163fdaf0ed0edf93a822e6dcccd1d9e8f1179ec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 133680, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5f4a46808bc25952b4d3f1e0f48fa5908361392140a1788403e19db667cda2f1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5fc75980a18b4d377e28bb26b15c99ce0d5c4143ef49c48fead95adcc97b8c5c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 165912, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d3ee775fcdac0f57c178dec16d4ce332afe4a9489b3c7889b28561eb388f6fd7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "72c7400f49827d4253c57430dcbdf196094b5630b65608ed4e06dcbe974c0b2f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133696, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "35b20cb5ca97a9a8f6e7904ed0010e2605fd10b93f6871a6c48878ad0736bd8c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5055b9b874333e40ae0b0d3aeb7a4ca938e9d87f34bf87ede3342d041e50a1cd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3ded1e4ad3125dfbc712b8b1f004e07c583f509f56141d520c9a705a1465074b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "dbcdcf025f4d96ac235ac19eccc714381261cc6baa9b8928bd49c5fcc65e18d7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2969f2e6b5dd1afff1cd5840f46f105cb3105c35198a9920b6510ac642b74b9b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133680, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "67b95700f5e989ea0f147befc2ba542be57a6c00a66bb917a35269bbd01b5d4e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 133760, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9a578256bb540933f75259c4edc9ea449fe0cc4a388b97582bbd7b54beda8959"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 133648, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7c0f931f5fd7bcdfbafbd5252ed04ab258c3dae5c541beb1e105cee4626aea80"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "80241ddf5f079cd0a26c32556e39a61527454d33b2f2b5548f2ebb53b453fc91"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "0206db9f57da06079138bb4067f87b4fa024a39303cc6a2aed30bf04e7cea105"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 133248, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "0033cb0c37223d8fec5a1a1ac9e42466a7e103446a25d7f5aecd2c70265e98e6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "68d371c60e57f6a8957c7047ea911c7e0efca3482411f0299bdbacbfabd2711b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fea1673e9ba8ae271facaddfb58279482c9750d638af9dca4e93c29196268d8a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "617d56f7e7afed36f0220ec440edb2b58585c4cf63481a5c928fae03a85bb5c9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 133776, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c6714f9e0bd169d462be5257865a89f5b4578e4b8051a5ffa320c6a7fe549d6a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "101dbc80e50949742bc3a88e1cc54688bb123e2918021ac24181ebbb2ae4eda5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e93cc8ed41bf65c487dc6fdf788435b17ef2835d12ff86c3d751b33ca4fe99c1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e0ccaab5a1437441aeed0648554c99b028cea7c084c86ec0f2cdaf0a98671197"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f690ec5214f80e836ba4a03fdcb29d8ae580dc37fae6a2b4d22e163253404449"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "99431b8a0764c70b189a715c3c68d95b75d525ac8fa25380bef64830825aeea3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fb8c1c8e4095daad14ca5ae01b348d39f191f8ac79d24d04701d76cb172bee5d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4c6ce0e395be6147163a2eea59768c8f2b5138a0b8a20f175b135c70d302288d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "31787516e4d29752ce47ceefa5ba62964b1de8b40d41f4b09f6c84420021eed7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "76e887a10df1d6192c7cef0a8f5ead970f400ac41720d9e67892d3b2d5176fda"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "85696b25a53020e2f416af08302be1a7e3292070e824afa9ee6a7823f59a93ba"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f985829d38d36940a4c4936483dca9a774ae84712e1a5692e04adbe59b566c4f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "425a9362557de8f60f26ba30a79d1cd867d2ec3b6292379968a7cdab424e8c62"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3b0f373d7d8e9797c36164dad247ffe369b2e7fcd1e08fe060ca21601c8c65e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f059083862e2552d999ef0ea15549b1acfc8d7f822eed34e320faafd741fa7fb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "94a6590178dd2950afa92172d936e9b2c58af5766f67249c6dd0ad3d73f311e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3834baa01de7f39180b5128396ac152909120860c3dbafa96fa959676543bac5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6a673dedc233154096380245166740143f971281a04b0b905e10b2220a6a6623"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ed5c5cffc3ceb942e864a6e505e67907d9bc6828d2547632f3f8f4d8b8e592a4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fee03ed1d9e5be4b609e56040e732b9e0540a3b8cf35bf4fd1fbd71a95c777ff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2273d19e5e9206653ac3798e6cf606f039bb51ca18fb8b87c50afc4540bcf423"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "867519dd838ad73d1dddc8e6a6a97ec9b022ca547b1dc0551e73e6fa0b3efbd5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "41f3b348e406df3268b5bec57854b444f6fe51a58508c14e23819528201d2972"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f49e27e936cc6a5075df94eed6a8bf332096848a6d91ba61bb7562b03a92d541"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d697d187215a2cc00a71b95d55c87968c5d0002e83e349a8f17f8ab7e6238268"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9877b6adf1d9b62820c56d2fe092d434b10c7e7207c4214c3b0a07197e34f3fd"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "7977f73f57d96247283d893aa5c6dbbf279712427f05b83cce8f1189b1ffcb74"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "6b7a144bd6006684dcfde1e033d219c9ffd5a40d1607de337c000eb2334e4236"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "85f762e1ad443efa2b11dca6e4084adbf2fee9ff07f3e8321c63ec022d59a725"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f0b1b86444340a51ffd2c90d49a15075d1dfe971775acb90fa8227a125c784bb"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "18d10eac3aa67c0e3460724407d1a4a5708dcbdeb8566fa6307b27374128c7bd"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1d1e9b393028778ebcc93a6fcbf9886b791d8fb745a49607605d7a0a04432088"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "4abcc3d34ff14a53ec6e40c78fd8a56d5d0f709b0c17fa76a39f008c41ee3a94"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f627fed3ffcf9a0650080fc519b3a8ac14f00f730fc6e072509fab70ae2455b0"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "b592a2f6efde8541c3b65c2249ad37990e5d1046af5fdc834c54a9b6b1768200"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "18ae09330e4aa219a6195261a510cc4b1a84108b7905aa05f46a605034713c67"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3d8311f3c02b76ff0ad316e1c7e7f87be252ff41b6056a6b16fb2ce09c22638b"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3c8c0e8ada3d3ff4ffac8d8e56cfe5f79c92bec374568f5a1273290c0e2db49c"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "98ffb9ab301db8988586682930aa78bdd13ccbe02b9d81f84b403af5007e9698"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "3c33caa41b808afe20186f4fcecf1d20cf70ec4509460408f7f212b96a198acd"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "fc3dc5489e002152bbdc94b4f50600d7c5b365d4dffc5ceb424425923a2acaba"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5823b3f11b295447e142a11bd94f349312971d83daa5a903ba56938b86ab9a97"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "d6194bf907f777d6b2f34fab67cd6b2c76764e5c93bf88ec28faafa75c75c928"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f536065699ffcb04888048106f95683c6b5653721376a16b86b6ac891d0324ce"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f4935133b784086b27929f2360c5fe518c49e8ee898e29470feb00690899c3e0"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d023d04a4e038902f33fd3f8c30a8b33252f766fe3944ba969a5eeec1482e06e"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "b17bc2c360f1b6fe04077d015191bb8445019c55c673c112874aaba838518843"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "05ea0af6b52449cb71c6d0a4386592fe1c6c519add8a7a449087eb60cc8b2687"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "aa6891245f4f150b6765b40bb72d27d9949708b9307e20bffb87e1ab392910f8"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "1aa8bde9e72e35e95d1c3edc2a76caa339795cb24f872858b185d92555aa2b2b"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 67904, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1cfe06ae854eb4fd6c2a0e482f8645db8b36a7704e039fe59da62d631b1f6e8c"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 67792, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "c279bb25483537ae7dc0caa497854c55c32f49f2baf83cfa1e296b3319b1050f"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 70784, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8a20f8468b28a14d8baf59c0f4eeed95a9936460104647cb052006d49f1ab534"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 70672, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "0e1d53be99e2429d5a32aad15c1e599d0ae4a6a1e874cdad2c0d0df768908c91"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 68480, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1017e781a777ba05bc186e58fa7edd0f284b5d10b93e29aae02a8efc423671ad"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 68368, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "318d1bc36920b88fcad74c9fc064dd573586b6ea2953b201b3a897bc3879d2ae"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 67920, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3fdadc140f93297ef5b96f70781df17841996b181dda0a2788689f75a5f628bd"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 67808, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "02ed3a7c15bd86554988d5dfe5a75b77d89efb88a3c2ad9c6d64a92245b54e00"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 70800, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "4535a1540b7e910493f3a4816a122a2da9a2d567c03acfa561fe7d43d06389f3"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 70688, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "00c26c6626ec52bdcfe2bea926d71b4b33a6c26006600d82c26a5d4254ee057b"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 68496, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "61c4253fa90bcd3cbd62a262cf49f4a8598931e98667899911f9e682160cd9f3"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 68384, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "ab0feea2012f9aafea99de9796934ac5bbbaa3be59a4eff1495869dcea37672d"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 67904, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "0eb5fb6ea0d83526881172a5a62fb597ed4e3b2056a880883f37a51e001e398d"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 67792, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "54582e33c32ec8a1f186acccc2f8946cf1876ae3f973ec88ab305784e6641ebe"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 70784, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "c0e44cb47cc70edd7fefe296dc161931c7f73e73aaa4e1629d3fb1e12322dada"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 70672, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "12aa866aa48ffadb5c1c1302faca3a18391b45bcc15ade76eb424172e2150a72"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 68480, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5eec34e420de3bc8399938fd967168f398eeb499ef34803fee9cf5d19dfcba1c"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 68368, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "ca5c97c33bb2df873a251efde9c149aab0d98e0fc493418c88a42fb56a85ebdf"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 67920, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "cd0c7d616d6a4f86b0f08c09129ab9abd7803cb672684aa7256bed28b48b6353"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 67808, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "7e9480c553e022f94a2013cf6f61a29123a8e0d3c05569aaa8a8e30d7e98b9d3"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 70800, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c1ad2167bb0b8c6f54ef1b2ef62d59d35f1e308cfc6bc200832d708f7556c5c0"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 70688, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "11ea74249f1a3cf4f178c4aa7cfa7e3f54332a5ff70c725d4d30dcbb85a1fa8d"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 68496, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c0108d48675baaa6337631affe2d06345af8bd8d37f929fc673179dafc44698a"}, -{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 68384, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "bb08a06fa001fabcbd57cdb4c199f1e2a89481fdb7a796a7f0efaf215a7d7957"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "921d2048fa3b0ee370cab501e4a532af67fbd4922c02b7aaa60e8305179d0940"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a31ad14ab217e9e1e0162663739950d3f46d772fa85e4dc1eb259c383ebb18d5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 182576, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1e9646a118be245f006e8ead43aed10df100cf0c24a8f7480a5c634ef2a77ca0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "11f3db234ad8180745ad06068b3de05bc615c5a9645ac609d82f74d4cd224ffc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 209944, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "9f7f4de768be5e592ab66fba5c3360b81c5b6735ac6f07cdc8eaa520b8911306"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b819fcb8297a22db4fe075d7b7d4a9cfb81b806f93973641eeec6f81fa045dfb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182592, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e4a911d22b6c18fd6b7c233c1673368691f999d5125aa2210b1f657c05c01b7e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c9e84206e13dce849aecac099c32fa39e864eed5cd1f70dd33935521348fd6e9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "48beb4a8636226013fc88908444de6035cec57043d2cd767151fbfcaa12fa08f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "f419be276533fb03851a98a846259fd984ab52fecdde2687d46e2252ca26c2a6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "cb0e84ea49f0dff34571212ad73c0835f46b298f0ad5a1c3ced9771c5eee32a2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182576, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d0e62809653e9cf1253adfe461156f48d2fb2ff662038aa6efe37a10e319f449"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 182656, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "78eba7d545277d61bd8ed5552095e319dd0c27f662f243b9b9d68eddbdd173a1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 182544, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3a82ae4fbf25ff9b82c3bce00b36df51e397cbbf192bc7eca4d4b25284ae7766"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "4d851417c0c0aae8a589c8685aa84cef769700c5e9ba44200a614efac8e9958c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "96b7837d89c94e384c89f5682231c5081cb7607066c8e2e1160128e33c98cb59"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 180352, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "15f6ba738f4640b0b311bc2a5e07c9ad3ca6344fff85d3aa69ba40cf7f7f1bda"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b738f6c0dbae0eae06c182e9c5175801c979dc353990f8fa994d1e57926ff232"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "50182df5728468ddfef91eeef0c4f565a03930a985e0cf73f67d2f21329fdd4e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "de30fcee5a815aff3e819089a9cf639642fbc5c044c5375831a0300a84c1fcc4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 182672, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "27f82e865034fc361adb729eb56083577f86a29bd22d4fcefd13b92ae3ab5141"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c7bf229be2162271d6ebb587b8f94feb14c5a8f3a753a3ad29db31ff5a23d86b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "4fd7f283db79ee0783a34c1e60a2a423a7efc6b6b9014e7093b0680dae1a8b32"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "c7a28342737b6b84a7556a0505861e1e8e35d435c614119ddf4cd04c09d11f72"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "8f23f086ef43032872825d4f011cde21b06df28aca147789bed0fa9007bf8639"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "bb78f20507853bb905d1f6d0ba3dcc4e18790df9daae7b4bebec190016824abf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "98f6e558483cbfbc6565127451e9ff92247daa70ea630e95ca97b92cd545ab34"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f549f5ff3da5693f2218464a6d90298988902736b2775b6603ac24a96d7cf686"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "878698ab61cc63431013a27494ac5560ddf6fbed7c85fff3a24201af55dbbe73"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "cd54944e93dabc9e9604dd1cd375933c5012f3b2580726b4e5d891109e2d62f6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "c541a13e0f9fcfe0c6485b25bd042282e7af005f586e0a3b2ef6d78ca3d44702"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "e028dab91632c2723912b5d5d66543f0a2c5405c9674eda3f957357a1299febe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "8b531beb6bbff4a4e7959fc6c82ebf0782ef7bd0c9974f417330cc49a1cd922c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "baf01232db16808b80248544156ad7f93c4b0b60339be0a0ce20b53761dcb88e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "6a9ea3cca41cd1f9b3f8df7ad261f4140757d62b57fed5af9c33b1a4a499ad7a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "939159793d03e359af530b6efd57f953523d55ee12b5e7957de4d5ec26fb8eee"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7d5aa3b7c6417a7cf056267cb0b84464365f26f614fa44ab4cd8df67bd69c7c2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1a489cda5f2fcd7e53eb754288ed43fde9c6b177d3601265e07b3ba0b8f91152"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "3d6d26b436f3f6d5ccb75e5d1adfb5f736c012dc66b186cbeabd17d86f98a75d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "7cba82894f1651ae64d2826144428a790be46b408dc0b665436482a6b6d3695b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "e209156e3d77e9c4a8391aa253185cfca505edc72f84de742c1e3e6d233dd763"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b8d38b93e14a166617cf0f8a7bf625de96f609bceb259449d82c21ca4a523c2d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b46648199157a59ef8d50188e3f5158c8af0dec83dea38dcba2780cbfa386bfc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "598bb4ef0bfe5ce61e67ba14ac98f7e67b2d2318733a4898be026d5b4b75a9dd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "120a4b5f4e8bda1bd1b1ad7af9fcb22e3dd13c556833471e69ec5a767bd25a30"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "a020f92f3fe7eaa16faba4a3c4824eea6de27d489169297604b31c62ecba0d0d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "c7aac2785d02f4c6a507e7c15cf0daba84557ffaf7f3fc6ace61a012a9c0cd37"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b31b3b55c79090292527e00ed267db98e6ea237ec1a8f1d9d4460828c9e23a2c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "34dbfbd281d4a525f7fca77cab46d6c23d78bd2b70978409640911b380579e93"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9c2b823475239b517d164988a236bd25a1f32bd8fa8ae0cc1a70080febe87fcc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 182656, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "641bad35ce0f00250da97ae5cc65a25cffa4390f37a5007e42d302681a287d3b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "e679ef4e7c788541f051f2732be744ff5a15df22aca69ff5ef0d678d9cd352eb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 175104, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "1bb37fe5835e089bd6263a6d22b25d95ac1c27daca30c0eae7a514a0ba8f8478"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "714f51e34fd237e5bdb8c3dd9d67c8630fd69b4525fb52b961cc14beba6ad953"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "89c432f793ba66b42275fa517dcb0bb2f3ad877c6332d14d6aaadb7a52fa6791"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "1f1ce6a7620bccf7cbd73784e176cf613a3d626ee87feef4645ecc671de35528"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "99def450558afd50af0e8b761fcf6e09b84a38fc7bc5fc388b6cc8f61e3e95da"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "af2f99fababd564b27d85bd5d567dede376abd2ed3471974db447502ae254e92"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "da96ecbe42e2a73034de45cdd4596b8076401dd4abd2275b48ad2c4b717bb5a1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c2c06633d3b48a9ec56f324ec33ad8b73d4ea63a3efa9b03a4d887626ee16519"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 182656, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "1c46e272e04551f0a098dc2b2478cc89463af63d76f71e9a517d8e4cbe366565"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "0c165ba2ad3db2cf5fc146a3f78ddeb77c66a8b22070ce08f5a87af45f8ac031"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 175104, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "549421995bd23ec5021880b63c1ebd2c14cf4aa580326912879672b37ed443b1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "955be6ce9ad617ec6ad56370e5aec7e50d3d8b7562b971a0f7b59866e1110062"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "86a64e6117de092a351bd3bf1eeb69d828e15a724370c5715fe2c7d27e44b362"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bccc6d43ad917ffff34170502ba1f44b5e59897703c181856d4bcf0a1e93cb38"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 133680, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9a916967bee4ee2f01d31e8daadaf7ca3dc8009f84a01c135426d68978b9f5e6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "a48398b4e7ef0dd45a05fc60c8e3a814c1a59bde2d5cfe9cabd61671588527dd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 165912, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "3c9f36d3f20406b055459f062d41bfb742d080d9f46fe0c83b9ff681378b470f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "7cb09b4431033e7313f7869aeeb39717607e0ce1b679f90fd4fffdf11978038c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133696, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1a308880128d6364c51f80be23f08c366f7324da83df1ef01e5be818cef0a7dd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "49f7e577051a034022bd40f91b6c7e99deb35c8e52b9de5bd24f34626214f348"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "1f48d34675e7fee4245dd98810069192288d7e67d58a382516322febc6fe3e23"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "6f9fbb6a7851edb0d1b4735e370da28ccfc014440e3d33870b68c6252fe898ca"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "eefafc2a99db68ea6a7d2c1e234ae2355eb1281469c19fc6d0a51bb9a4d72fb7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133680, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "2e9e5fb8da11d48a0ae24608a7fd01b89a86b6363467dd82fe379d43d4acdfc0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 133760, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "79c69078b55198ba6543d21981c90c741097e109dc3f7b2f1d96f2b16025b9ae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 133648, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "df5620e50f2f8fe025643648eaa66f0ce047e1cc581fbdaae072e1f1247502ad"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "008c100b8ecfabb1565d415895a4cacd61c46d7ec47e4348a0d6c6f1c11f3c35"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "87a9554faeee5d2b40558b1d847890aecde430c049e7dd500f6fbb9b24ea1797"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 133248, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "d0330275dc135d85fdca36d5896bafb270b3773acfea0950d66e5ec54bd417fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "1e3d6af627d1518f5a814661eec695bc1af4e0088653d237d76ab3ca7ae26929"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b91d8ca4c15781ff5499a6d9b67593bca679c712ed1de0859335cc1264484a5a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "6d76d84eb2ea76799632b23b2f615623a5b01ed8cac65528852ad89cc93e4365"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 133776, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4253cd65b20e20ef55d72ba2af7613f37893d17c8ba6e49709d506e17bafe6d2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "fe795d415955cb3c4cdfb58c753fa912f1554a75beaf9198a3926b2ab4d6eecc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "cfebdc160c9983049a49e6664ac2b8c4661d76d67ae5854ba077cfe6e59332f2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "1ff1751e2b75deb7e036e89bed5163bd5ef31159b8a6e2cc2aa9b376902ced4f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "d8ed055107b2248ff342f8844c0d6dad8586f3a54d27128ab748272feb24cb7b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "2ad1f662411800d0b32adc5b01f0a52190e71df13149a2291ea0cf7018ccd063"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0dcbf8e040865850ec11276879b463260843ea10263bcedfdf69bab38f946df6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0d5b0bfa8ee33a684de0c9f782e40ffde119d0efea0f4f25f3919f302f398644"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "47cb8d63b71a269a9cbdd87aff550be859fd1ee08d17a0e11798e3e99aed36f8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "4bfd4132a73ac8e4cec8b7f0f4908cae5a86a64f0c7c1bfb1d698bf7292a8d58"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ba71c22dcf72a9994e6bd09dd2b693a6099396c10f5fc01283e9960fe66f8344"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "cfc4f709644f344b1f71f0687a13005a1bbb40d6e5898387af53cfac730d15f0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "6cdebdcf6a6b2a8aeb9f6ec9a61274c4a650ec7f3d7bd3a4cad1bf478e4a5944"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "28a9df09b3a649143e6848096a4b8518fbe002ea779cc5ed681ee209194678b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "7d1c654ed085eafa0f9d24387d048dc70da1bc54618cd6768bc617ff5105bcaa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "412fd74aa1199cf8deb9f75cf01abaee8c9274f5270110daa9abe8b56bd53169"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "90d6b5b3b05800dbca392e7ec1320c97fd8aca51377002bf2161466494403c74"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "18b5ab482daa20860d6fee911ac0ab34a9da978d22004f74629d0ac43754a27d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "a528e4650b72fd81520be7408736c4d8e917d80138b3f59c57c5995aeb68bbc1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b59f1feba929c5179df20cbeb445b486435f69438577513d33bfc66f0d2fa2ab"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "089a66f03817987ca0f5e24060b0915ae2654623bce0be5826cf25ae290e517b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "8b460f20e8e7cc4e7ce4760fd17649cd65a1aa6b5f3fc51a555ffb48c1fb78d0"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "013533609842002cd538186c4057909de4ef497331f862338b2316336e8fc84e"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "96105382f7a47b27cd9ebdc2e7f05d16435994e18b28e9c025a1cec345046216"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "1c5cd8dec9beb3297bc49a55ef31e2b6ef9b7a824fac29621839107eaa2a10be"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "80346fa6f0eccdfcd29acbdbc054706c0eb3accf7d39bf23c0ca516bc15bba57"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "da37cde8ff3c18d8757aac71b93d70801f301186e0f05018d425474314ace240"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "88c9123959b9893ed026abf7d099a93e479579c5c3b142419dc46855a994cbe3"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "974f8bcfc2b2c0e038b6fcf912386f1db0e50bbea8392feec1a2beae6b44b025"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "cd64b8fd2b850ccdee7f270e540d40adc0ab2a308d079e4edec0d2dfd6a26a69"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "628e6b9d90d3f66c138d811f981893f1c1a3ca0d228d79059a8c046217fe494c"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "1d98e4b77f0250e167c683502e4b667d87117a9a847916ec0baa6e0e1266c7b9"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "da9fca0184013d278b9f1d50434e1513fe0d7a4e8887fc66ee7bc9ceb9838570"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "d2a1cdd004436631fde68b031d8b70816f51951ac35343c81cf1a46cb9c13598"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "7e3d2217a782b1af07cdc2fd302baa805ff68af44d44101fbdeea9e753e6d16f"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "ea7aa818ee06cdd66c8f5423192f93f3e28d044a3257ca64d6919ca16fe3dcb1"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "fd8023d53aa223c3b97d5c2a92e70f0ddfcdb547d4c8ae5029de9be9f67c180b"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "dd221a53fc49394a1865121ad86f3f4c64d0fa078eca6b6d6b2f13ba94ef9408"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "136ac8ef92d09825e16aa332994acb9e5b0b09dbe1f082da8ae8acefd3dcb352"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "e299ed4e2577ee327e9dc1ac44507ac4a487425038b47b27bcff28cd2d29850a"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "4edb6475b04de7cb7bda78e3393f1f8dc3d6e9a496701a99874a6d5ecb55de34"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "1edc1dcfc69a90c36627e78552431593af0d1d842d5e1be1d494767623f7e82f"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "699a83d3945cd9d838f1d1e411db0c68082893c3381e67ff6b8963c9a2a70a83"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "fe2b7f11556b32023ff4222c59fbfd8d72d9de088e0312d1747e08b36f65dde7"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "4442575dd510b1783421fcf3cdf161c5a87b76c2dc3e7dcafc7efc13704944b2"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "680257252dbba05206e3e2c5ff34cf9e75958a366cd9c661c437770cd93926b0"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 67904, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "575efe421073b4c551948048d2cd3e9de6d49da5474d59cdb9c337c7e955cb64"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 67792, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "de40b52c9e2e8a8ba0cc2784e11091efdc43f9c36c3bf05846ca8cad552ed448"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 70784, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "fc8b906d515a4281e3888bc497d4e1262ebddda3df2045ec3908d77e64fdf059"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 70672, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "2a1dd1ab00c66dd26932879f9a605bc236c92914b0cc33372fc11ed6da74565d"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 68480, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "5e0226ad16924a3926202fdc1d4872bd7bb579ca4798d282d8ef4e9ee1da0531"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 68368, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "8ab3c52bc6d0482e5750253412c5a22cfbb34a3ae3a803f5ca19a4332590301b"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 67920, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "03bf21d46d7a48353d2b28502dbf4b982e70b95dcf61561c20436f3d59862781"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 67808, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "2ec1aeb2d385ee34f5f6dde2a6336161aaf040a8db4cb670248508aa70532a21"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 70800, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "177f2f31fb43ce2f1e734aa618eddb772cf19f93433a8718d4c53ca93e16dcda"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 70688, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "94a621da3a51594e4e253cf824565a97955b1664e1be2e7e11ad5a842de5f194"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 68496, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "2af17eba16151157be691966d62ead5ffe8705fb9a61463f2c96b63db4d725de"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 68384, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "a5b066b28ad003064e6e97cc056cfeec382588607b409104e3131ddc86b158e4"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 67904, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "5fc8009a9479bf1e216c762850a026ea1a753b987279ab94b400f4cfbf200767"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 67792, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "7b2088820a25e013af9a60dedfe6f4ff9262ed06d91cfce0ec79bd9678c49990"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 70784, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "a34022cad26047c559e1f41f6b8e621b0793b29f155fe94428e6ecab59661d1a"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 70672, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "3a28cd8498a3c4fe99affbf0c92d62f009582d580f17e2d83278915870eb4427"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 68480, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "c65574684940c4a34966ada6e5cf95dfb66c7455b1b88d069137184d2104d7d5"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 68368, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "7d379bfd12091963504e6db77852e971920af5d092be60d936e25745a7e50acd"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 67920, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "6f4bfa6d9eeb92775a24e832d291ca6a12527eb3a7b49a8341147259250d493b"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 67808, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "0946719978c8deaab3390bb91ba308b68a921dea27c1f33e2e95d8c07ca38e41"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 70800, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "7e6cfb09cb7ffad022591d06aa87d37aa236ed74c1bc621ba367b2fdbbb23994"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 70688, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "5bae8fe8536258c4186b28238978a1b996d51ad50b31da20b68d3a337bcf7c66"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 68496, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "1f98b350c130fb51a4225c0a40285186078b17ae51a0a6106a5d04e5305d3922"}, +{ DATA_TYPE_INT8, DATA_TYPE_UNKNOWN, DATA_TYPE_INT8, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100aKernel_QkInt8VE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 68384, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "f7318537a20a695121cdf613165ffa5d3dfcb689fb74d2a9c3f32a80de21dfe6"}, #endif // EXCLUDE_SM_100 #ifndef EXCLUDE_SM_103 -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c787b655e151e866e2f9d47837ccc44154cef6140123d5795f103685ac4ae2c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c09a727d49839d9b54bfdf8398c43dde7856890e88eaf27a0f3be3db09386382"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3e8f67b70ca80aecc7b18dcca9052f08f8221e46215d77e6759baf5701c79aa9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 209944, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1844a1b6b41aed840fb4c18afaa68d55e76409c3a82edc75927ef234dea41efe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f43dfcec24160828fc10db0e872c8ce0e6c02f6505a0ab7e6bde8dd048ca9e8a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c0777c1569e79d9beda71687b4a80083f839831a115fef7bb5281467c2f7b550"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a4b515483cd00e8f682349c3d9032955104ea1e88322581455aed880a9704ebc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bf7837d94a30066ecb6dc5d07366d5f8fcea736add1907bd0580698c553d649b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 171392, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "33f927252aba8743819a4bb5ca79ae6b541c402ab31039a7777a162454e771cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3eb0d90628eca6bb236084be51fbb086c892a323238bbdb232f422368f3137aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 184448, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "67e369c601c28b6f120aea62b06a3e88693c3eb197d58e974bf0d9c1606c0811"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7789811acd1b4626b4f089a876d6e5cd55dd667cdd3f07c525d4c498a522624f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 164864, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "279f261192e42097c5bb9009d7ccd6b759c1928d592d16be7c918ec50aa109c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a4216cf3d64ade99382941de1015e302af7cb29bde5c80660ecc9e5f96d7742c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7aa99617df4c8433d093d412e004663093d4a442e143daeed1cf3add05e23d18"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7743e481628600a60a2468c189368af26ff23cd8acef6ebb8493718bcfb2481e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "288f418dfb3da582948dd61b7f4bd72dcaed8872ae77b279d15835226a263460"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e32e087f37a316290d3f7b2cd9b8a31b56e8c095679fa4895a475ae1356637ca"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0b6762a6b74a046caec733effea9d1a2922426e95735eec52b80989c5c04ec54"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "15b256c8f4b8836fc8e5ebea0ebf624ddf912cfe93140b21fb19c8e423f44b2d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 171392, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "136a15cc127c197534382697d6351b882676b10b13ebba94f7e4eea615c7702a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "0893835bd962d2c5b3c3f4c106ef5ad21862fbed954cb1466ba4b6eb8171194f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 164864, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bc5b4951b09c045121e65860190083caaf825d43a36d9a259a63f54b3ba96694"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "273be0d4f950e3547d905310ab417b50174450d6a4ee9f1aaaec075f5d8bbf67"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "72ab17e2c09016a2d24321c17e4e9210b2a0aa9ddc4119c1201681d87b4d37ec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d2ce32e572e22e2b2bc0b9151183a30f1c286618b3c03c2e67dc950a3ab5f61a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5409b44b37824f525d66bd4928a3967e6cd38e5a64967cfe8d535e9b10fa33dd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3cbdc7031b2f04bf7f139bcdc2a056de129afb72f989a987e7ed18e19630d1e9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "851553b1da2ef94991e6fda1fb4e615f3eed7ae5183ca556feaf8e9eeddaa06a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "96304cb330dc696e5b06b418aa50e503612ad37f72b72c65b81a7c820add3b61"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 171392, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3e62839c19481ebef77c1d04d7b0df3432fc22c90b3c7d1158478f2b6e34eac6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "aaffa3be9e90cd1fb62c79cb68add487739b3e6ec0dc2461cc7cc054b5c7a027"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 164864, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b446869cf56275ff8788302e7c959cd9734dfc29615c9dc2cece084375cfdd3b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bbfb5944ac7d657e87809f049f9edff93e1eaacc82b90c92513c8225f815787f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2a059ca26dec126f97a876c59a9925ddc0f1798d311d3f708e35fbc167dcb8be"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "67b7baedcf65636f5dbe8177de7681016bb7250e70f833b66f7fa31a5ecb32ee"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7798a24100a07673e6f2dd79a8d5162a7511ee0bcf97f09bdf4eb1a3a53a91fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "24fd28da88b5bb2867e63b9a27464e5764e0e54a6f6f0087269999a0ec7d561b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5f1003f0af643d8106b7dfbd3e6773eac995bc26b68d3f014db0e55c7794ee06"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "542361edc12d9123806dee9e0ac3f100f36ea27a7c39418f18642c9d6b5bca28"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6b76b63c9eea0506f7a427bdd27eda48c85203b37dcc97a2a4be04e68f1645ec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3368a657c72c7f1e8edc565b65d5e12892f2f1981e3842d1756d5d0f71e80720"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "783f7b3e0509e47238f5c2fd98f0c4d7ae588153609353188d087ecf33eba48a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "4854e4303312783c593f9f50435a4d74aaab4a5a8858aaab69f2e4bd2ea5790d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "75d128faa99773749c12728896495b0c9457d623079b62eca24fb0315755a1bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "902b0e97f18162dc0cf04c01d73f41e3c8df19dc34384061d9dd8553c7423791"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 184704, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fdf92ad916258bb4f7c2c1d779c6b01777ea6eb33a1b2782d7f8f211c2cb9471"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5f2671e4f8956cad218d17dc02c47560588d256f6cd0a64d990a966cfcba7cb2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 176128, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "55f191e0bd240be1c22258fdd57b4a4d9b2bd72e385eb3bba38c5ac5ccc0abbd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "552f4733b10e92a0afec71b980a779f5cdf101ad5698bf342c4dded47b06f731"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "12d2671820fabb44fd683fb47e0a0ef7735fb826b57c1de39f82e96da912824b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "365ca4e357a8f8c37af05226f11aa653ff1d124f5dfa13247dda0865a2013b61"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9a9bbca9e870a76c6fada9e39b808b76421928ebe0061773c835729d968510c9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9edb6e77514301526ed014b9d28a307c7a6d79d4e22f772a182d2bc5955e8d92"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ab457086780572cf47f39e314bf8babfca45f1edb392b7c800f7a9e5a93c50f1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9d5ffad49e747eede0e99a9e6f781f3803e11f291dae734f7274d8cfcf6e3ad9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 184704, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1780c53fe16b4d7d21feefbe048e720834dcb12ba1bef629745185ef531c5973"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "172292b013505afe2e92cc8b93617293fe648d3a915e61a74a4b7470bae48cdf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 176128, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "535830072365ff0d3c1b7a745181000c0dbeb329940f5c0d4939a65bc9ef3b93"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a47ba099db7a799816a9ec8d11ac7120c96cae8257b4b696eafa410feabdbc93"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a5046aec9c811bd8ccb448b7d342222ae816d0873650f550f12fe49a7ec2e542"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "cc930712a7897086128e0f08353d23c6386715c7bd4633406383e78f10bc8378"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6a14a9ffe1ad654e0ba1c2cf37e9de9e9461d523680c0c6c48666c0ab29ac8ad"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 165912, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "586ac550130d2b15e7e5cad293bc79e295423d306ffebeef5553065d5f2250c6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d92df2056231b653c5f8ff9db5404cbdde1aa58d78caef3070d6a607d7869420"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "32810b0c631febcb2d0aeeaab0be5335ec438d827c05eb073f0b506af602b951"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7533f4eb89aceb6978966c4f6ac17d56f8bb8b2d0d8dff64578b91cd33c114f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8a8d375b6919ad3110757baab3196030875c78ef3a862fb62817fa8eaa08b156"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 126336, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "dac1f842df08e204a9e23b25d2318aea33170abd8565c9d4dcdaa7da0cd7cce6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2a13c8271742778f8a04602bb812855b984b6854c4b90a29fd0d895ca02303da"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 135296, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "87e5e5653160b55f49875817bc3ab9dfc36c25ff03223a582ad9f6b01cbb57dd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "442f2de51d32c4edafed3a84a41c829e78fcde438c532294c74772ac6f4533b7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121856, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "77e43b61011a0f0473f4fbd46ac0d2a2b1da7274089fe4ce523b884d77d5ab3a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6369ebcf5c3b9297ce37a832e2fc7065092804d460ecb62193d299931a9ddf11"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2923a0f139d7a2a75cc6d50b7da0d3170d9f238560b7627c8d2e3f21fbaa8d1c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "197a9597fe2cff8497f12f4adf6907679376280fbee897c0b4c4c63989afc69d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8106a44be6a9dd12226c64a9f1ac6e5caac9afc27794ad774ce649122fbc51af"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3e7eca050407b0a1d33c1051363a92f5f22e09224f19857b5429ef38c6ca1183"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "25e373999904f0ccf030b6e51daeee4c3068d10301d82b4a3ca95d3d6ca03881"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6400c927a5ec686aa04af673871ab24acbfcf838bc72fc116e9b6e16a7b068b4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 126336, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "dcc09da0600f86366a5f4200d18884fef6eeb0a719c64b37ff8e4169f5094eb6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3da310fbae9bcc8ecdeb4bc9370de73409eadb254ff8cecf75f8bca56d017f57"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121856, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fb41c9b35482758984ebb55fc4eef8c1bb1c699dd1a1d61c22a896cc8889b1ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bf9932eec4cc0ecd49d66c8073bf6c4f7e0ae6d90d437d9906e88e55e01962b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "14aeb2aad1ea82410f52f424950d91db6baf173353d86d366906a94c5d2d5fe9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6b69da7067750bdba107de342ee1cc6097913e4fd2c919b69fcaefd250f9e9bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ab154a725c2411322827a23a14e09174aea9b42eca7e02cddd77f4f12cd37175"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fff410ccc8a7ad937a9c30705a562a55a53571f4684baf32ffd273dbd4748695"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a506927ecde1ae734d741317ea937be640da3781f83547bc8822906289a68f56"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4df124703a50c80dd4aa37e0676f6b2abe25cb8e10a065fecae0ef0aeeec647a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 126336, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "927311b6eec141ae258573a07f6e5ce79ce5b2812cc267b030ffb1eed159e805"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8979eb4ed2635299da756895807380f8ac255e1f1f74868b49f6d8737f97b6ff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121856, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "99325df4ef72a4733c1081c7ded85747501e3ee920053c0a74e3c5685569d3f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7cddac8e1cf825e830d5b30915f6c98f45eeb301c3b50872446fbd82eb3c1e73"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1c735ec37141d086655e2b061a9577cf50ec89e561c114da22c7a99c52d3a6ad"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ecfc910d3bb2adb8e526061021cbb657c48ca7cbf7df1de4fe7b143617b896b5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "81ae14a8607ef174c1de0487bc4af70ea90af5c014ad68e6e18c594ab81a8c77"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_BF16, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "48374e662d3e7e1e6d1f27920b426704d48176f89189f67ff1b8c71c19d82439"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "95ec79eba407076d22b25101dd5e6566f4cc21e15679371be435014c68597a40"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7114aae45cd80954c2e5a4d118ab99d1d2f1859ce85fb3542dd2f2721c38e508"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 182576, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "67021b10a4b46aaa05811c0c6ff8ef6fddc1745cf79d3ebd6575076fd4e54d99"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ffc8ccc0c10e29b2839773be3ab912f68753637526fd36e0b29489935d3cbd93"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 209944, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "aa48ff63bb38deb93afaf012c92ed0698dd5e7e21fbca3af39007432f3b12cd2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b10bb9c5090f51f0e333efe53dc305c29816a65882416804ede65c98a0b082b4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182592, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "015e6211cfed0997051ea5a76b4c44001dbce3a1d53da24a6619b995a4e751f1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4a8d7f67eb4ab61c162b9b864e701ca4f14d7c5c245bc78e3d20f83e3a50a2b6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "34313a5dc3e6cb07b5264f6e121562eafa34f1e9138f9e62ca0931a0c5dae4de"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a51d249638944d46e88f51a68571e7046a1cd9114bd0b2e09857102af3806486"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "03afd3bc3793ff2638c47a34008c6a4b37d8d5ace68a26a0255fe1f62bca8441"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182576, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e8d66d43523533284782bb0e8a41122fe60d67f72a0eeae78ca3fb67ce50e0e9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 182656, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9c048832f64c5bf40fe5f39c1675a761d67d9040093dca6f19956ab0e9eb12e9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 182544, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5d93ec6b645b48f46cfc68ef61360e030c3b8d239b3cc2082c5e4b6f84cfbfed"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d07d7a70be7bc17bdef1eb3b78896a3043449f03c822adfdc1aef39518ddcae6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "82b988505da6c18462883c67668911bc7e067b11d4f2d1b4cba4002fdd60be6f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 180352, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1c444e08ec921e05aac17c58ddd0660f932b96e84d5e83774a79240eb46e2add"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e016ec8216151d5e1af1c0145794d9bd199c7fc3d54d2cb0881e243ec6f32448"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "de283f25d343b03ae4f8db747230db3b1f2521ee7c087c0564797d0d9b1df444"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3715cac62121e2ecd02be5a0b93f56cf4cae97961821b7780b320b6d80ecc706"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 182672, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b048d1a06448dab6614b9425bb8b68eb268e7bb49b45ca6b7f25ce1ec03e506e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c11dcc1ed78323ba08d4c688cc36956a3be914f3a72f004c8aa642ff00814acb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7d51a99fa49d1cde54de2b86effb59a6bb8f31823b2a7f91dc95e13bff96cbf9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "270cd36060920d875f2a15e718b44ee8b24d91bd6d176ee4630e95f8a6cd4937"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6ebd11637a53062705cb574580795f0b0c3d5a118ea747bd401f74d69f8fd68d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "cce2c7a2c05fa7babc5d649c8ff36af8a73c613cb3709707cbfc8074e19c3107"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "efff59a540d561d7c103c43d76c3e64afeb7d36f3e99b4aa38e18ef8a6758953"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5ef035ab36a43419086c7e354f5f8bf7ccf878c8d9dbd02ebcc56779defe9111"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ccfc09ad84125aaf03806342e4a1f50d83d1a5e83e288dbad397fdcfba38bedf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5be96167b6950f7d31358be0fe78fe4a24edd521066227ff560b8b550d30e37d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1d3ba4affddcf9a28df0de928f5856d956bd96377d05d4e9b5f100a7911363b8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "48acccd6b622ca48cc77f8db5066ffc4792cef7dc1c0d6268a78560da295023b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3a11b3657eee5701dabba4858a96a62a00ad1ca09fa3e8334a8b6e9763b5ef92"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bbc7aae182455cc8334da6deae8c10ccd333d4c179b342f823e1a1a82fa5234e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "df95e31e6d879547e970cbc7817a86927daf420c54b70777c5bb4ca440afb067"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1b69156dad49b7ad2d52cbdd78de2fe5a85d978d09404c7b9bfdbcb3e6e83945"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "81e68700f29a459c2122a391b7b2f9b91904b5760c5b488f0416a1a432edd4b2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2d9abefb66de87f0b1c3a09d0688fcc6c820891c5dfe8e5d3aaab02c7285c5ff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "90c044a2bc77cb49600cd6c15793fd2d5dd3800f8eea91bfc60a0088af02414d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c40feffa433bdd3c516747d2e44402ceeb73640b693839e8f12b317f5141a7eb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ce4d915ed4fbfff05e6a098bcfb0ef3962ebe2c4e78d4e3aa8618f92ee4d469f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3ff60bfb13ffd65aa6b89586f3167aabec34e812089283871cdb4034faa6d6dc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4e0517c3876c45e20f47939ab659b0df53b19cc0084fb4b7568655da37bb3bfa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "14aa6fea42660807aa48bb5e080ab7fd34b232f675deb0d64b2a610e14ac4a12"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "dc7d2c79cbfe44259366e37b7b1e8fc81ae29107328dd5886679ae68debcb7d4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "63a677905ffe5d928d02d881c7c0a8224e9e17dd81d33b7c81aca968c65f3a62"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2035e1bda7cde77434fd06146567aa58a4a18299176cc759f4c5411c67273c71"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "66f8ee9450ad61b02e04fc3371dfeb8760e492fbdab92d9cac31a401c18385af"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "0a6404eac5cd3a65450916eddcc25db5bd4d7e0ded147b404d0d7cb32220331e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5c8d7e11a821e3868f6cd4ddc476c68015d9e9faf62c31d43ffe5454d9a3dd40"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "318dd319803f7d8338114b5deac970d5c2ffd6ea81701673f4ea2a1a292ac6c1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a13e6824328a29f224196235809f5d5d6a8016b1aa37765af510d87be1226abd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "403e01b61072ec006b256a6ab956b34ffd2512fb54bbdc4847508f60ab87e6a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5c2fe70f848b4cd03f8dab9aff4b6e4e8c9996c3e111654f27f65b84c2eea02c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 182656, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e752b6b7ee066e2e0d2a31da82904947f4583c2fdebc2cf99bdae597ec64973d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a14d3be5cae55ec649791537d2a3ae0d7eab8668ef9f4aeab5b6680191d65df0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 175104, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fe0cd4749eea1b397823e80b8713fecb8d1cc51c9a71e0391fa8cb2bafb5abda"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fab6f1b7578aeec12e7a68a42378abdbce2358679985ad06c5f18a3a715e4761"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7199ae058df2503244e8961fdfe055528deb427f632b717b952a21c242df6b11"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7bf708bbea491e35b9f03a45908fcae463c044d900f2969e00f5994019856128"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "67a53ce895f289448848f4628ed9a03121ab4fbdfabba9b987e6e5a629808946"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "23727b60e2f2f1f1085bb2c051b8ea183793c0d97793493564d03a9f9e9f28c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "771d7078c83035c2f91d43df48417ec4a9eb68d39a7d9373c6dca2ec2549490d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b4ed410582ef6f05145d2c91c3613dcb48005f2335106c4b02da02022f67b2d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 182656, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "28dfb8b6192042dda68746b95c9a5e8d1c66acfd4316cfa5cbe585c84d67e6d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "01a87ce990d07e8f7ce4755c7c200805c33aeef345b8cce900d81745152e2760"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 175104, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "33716696e112ae943bb0f3a19af72851ff84a8622f5455df297d472ffe171f24"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1b1ef8ac0bb39e1b455dcc96efaf921f0fdeecef49ad5767e1177b79005cc32d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "99e7ef9fb1a6216ddacf8d0dace8cc8baaa53bde18b149efe604c4a227c188e9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "95bda692c4af03205c7e0122e27a903de556aab84938fb9b70d43561b4a54669"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 133680, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c9184516aa34ee6fa1d8335970d5860c282d8fbb9383b5d3d0733e520004a0cd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "973293cb3aae9b29727056f57687113b22dec39f7dd3cb580d738139f3dcc09d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 165912, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "084bedf54897a30415c04a242731930ed398d602a4ca35bf3afdaef49ce48586"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b7db56dd7fa108760de3a807d1e2e985964901815ed351fd24633f3fbcda3582"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133696, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "09f4d894f6cbee8f4b57dadd1ac15649fb492f908a2df3a978adf50040351d16"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ddc573467ded5a3bc3d2f17df3cde12f8a3a1d9d1412b6f5ea0a1e4799d0dac6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3cd0508cdb95b1409a60ae92f18be690d193718fe81959fb0d91de255a97c90b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f9578db4084c73eee250dce8496639f212ed3c8bf1a1059e1ea20a580c640d98"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "61941247439a926fbe1bd28dc8a1806d9fd2e58083b376af8b03acbbbfcd7870"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133680, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1bdd81077f8e4f791e9b56d93b93293b78933a87a19b7c15c757804cb1dfc2a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 133760, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "bb72c1913aca8a7f195bbb4270f35bb313d18060e8dbf5ee76905c6f17ece4e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 133648, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5661725ffdf6c3b9150f3e99ea83c0787ee7c36641642381d19477252772341e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "15a76a8684f9e7b40eb1dfe63675f0fc68d95b9a7eaebdcbb267c5a58cd8cc99"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "80fbce8c2b4986ea7b0c4b873e658b59b3617cdb76570bfd7fd36ddce077a979"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 133248, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "904641e087d9cf70ce50c19916de0b51d7dbc11f35879f6dbc719b52c5a04edb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "033403e6566babcb5c3510e8ef6458852dfac3d272c939ccdbf16c1e535bff33"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8b9088f933ee9fd390fd867631fd8510c2a9a421a4e06dd2da1e838784f82248"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ddfb2afce6870ca32fbe2c992ed5018e117b1fcfd4926ebe4f464503275bf616"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 133776, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ab794d9c0ca3ab16a0423a8b22b0633a89bc2525d4750cb768dc408c06cf66c4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5f5ab7c2cacdf8851ebcfdf99c9cad4e2eb9cd26f93f0555dab73ad98a3cddfd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1d5cb62baa2c08e72f83f99217e1b8146ca4d9f62c1e19692c8c0585be96551e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5a493a7c922e5cbb0fa689c443f99497d34befbca856fdc291ed2e2592579045"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bfdb66d4fa139dc8b0753e18140940eca3e8f9237aa6653370640d4259a36819"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a1645a42b4e1d1196ca1be00e3e1d13a711f5ef63d188ce857886ea3ca0e6c93"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "56eb2a562ed018ac9eefb6b376ff343702e5c7623e954a4d3686f4a4306ac58a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3dd51d8ae51dba40eca32796f5fe2c0c49d9365caca5614c1259c9e38159f947"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b6e83195dc17d1295c42e36f899dbc9f3316e7d412e8507db8248a7f04a1b8cd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a177f3aa08cfd0b618f6c7f17de773de3ff449f6cb2708c17b339862e1fcc4b5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "cfdbd0b8aab5c3a655f936330a1a4690ee4981d0f7dc3261ccc1082f1396883a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3f08a8e03a016e651f60f314862a45b3bddab783e31160bef742dc9272bc6dc6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7e9aec8ce96f0933f7301fc197669ef5b4000036cf1f5a31bc5f1525adacdb67"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "87803db65c15a0f36ac2eccc243d553788cfed73226a91cbee717542cba41534"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "6254496d06e5d9de110766ec4585ce6c4592262e2f7fab4944ced39ad5f9951e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "fd41d9f5ac8387d64c97ec89dbd95fa7c9556d4a01dd52d3fd76edea8ae82163"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0a71e093ac43394b6327afe0c4937a7b1be376fc4aff5a36519a5d46d1b10328"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "72b0504ab5a5a700052d6b3ff7331b074347c02c924a5481e8e4111b6a380bdf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "728d6d5efd28fcbabbe9a652550db13dee82ff110009f36819c776d886c913ad"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "bbd66dbd0532ecb79fd41c3711b4de868cd81892be3b5f5b0bb6951c252d78a8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7805c42c88a453949e4a0e5c948a275d436c534695196de3d9d45c5a78a8957e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2e1bea7ce0fe3aa9e2f8adb4d5925995dd046cda391a8b6f7e649e80cfe7e7a5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "028b0f0da8fd80e227fd2b2ecb7645c784b9a24b892153c8babfb8dd4e392241"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ffd984ee358e1975c93600bc1b5ba8e1e1cdc9ffffca8fff96a36c47278d2914"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "d5fe7c6c498ae9dbe7f771fde6327bfe0b29e8740e0d9786c0a1003eeef299d8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 4, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "c3dea376ac28ebc2496ac484e923d2127061b7c8484052c6a87e9c93817f17d6"}, -{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 115328, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3ed9352087b96e3e7a924b8a9e2a3ed974f50d2c66e030a25a2e0e5ed33d0258"}, -{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 114960, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "465c04b644f373ec144f147e66a21f09ede4db9eda06138a0f769a06f445ca40"}, -{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 115344, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "eca3c4662af4ecf1206beb0383e652dcc2cb1a05f2268739994d175e628fedc3"}, -{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 114976, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "520ff51e07120a21fa5d71e429ef2baca2ab05994f3281dd77a758af38ba9838"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ead3fe96d253caacab5b084e0872ecff25da97c6dc648722ebea5ce3cb953332"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bfd23fa88fdea66f282ae10de87c30ec1d1a927c48392c3a84c5654e9eccf635"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a5cd776b92be6c8b47e9d1b1ba7fbc61bf260944102ee96fcf7d6ba535282267"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b6e6dd2e50c2ca6bc864994c3b72d7317acb9fdf4936b00c86ca4e6ee67ae02d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6f43c13068b668b927f406d874be17e25221bf091c6c7ae3d425daf2233ae806"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8c5f696e7ce94be6dfced955ce0c59bff0aebfa68ebd5674cdd49475dadfaa26"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f0efe0575509492e221dfb88fb912436092154510d170b6bb1f394678d57093b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "440b8d9b5225bb5ed1c2abd44feb24f647a52d366fc3e37e59c06154bec3f107"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "450d11ae8cc8cf29cefe2f4abda1517e8d77c79fc1a5492c6ae0b0b5da326efd"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b128c15e0f4b44d12bb58a6ffe985ee948f508f6ef71959441adb4b9cb251ac5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b15fc4c885f53e59a22d00bb13a0080aa299796cbf031d3967ec6fe341cc826a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1002c180a27d83f7b9e2c4b171a6529f0a2e50903a29267937dc61a26e21216a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b2acb7b30f7e90838159f5ea0abed9c4031f04405052c4e2f40caa31d2d1ceab"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0f32b0bb24d8cc99d93fbde08754dbbdf98fad134a2fcb02070193e84ea98c5c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bb3739fb892bd8b92c0ca57c94903275e0d67dcd144cad7887240bf8c0435e1d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0b3ffb73b396324cfc0d62cf82b6cd39af7df5e80a54cae77d36a3f56a260f18"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "339b67fdccca1a8fbe3e7e1d8b4ed4b1b63fd4951b04d805656ed3932189e417"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "aee7a91802b254c95e60cb2608700209ab79d3c0f079cb12bdbdff8b18e4b41d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3353c12c5eaf6e379ce7eed3043637b9cfbd43fc2517aa536cdef0a6b6afac6f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "16c90be07071247bda8dad783204b26cf2a10368a3e2e5291267f650b0d1a03b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "07913912af66dadae40cf5d25edd7587d59492d94204ff89ae27590100dae691"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8837ab4631d164b23fc6fa85f82be747ca12b276f65dfdece4fcb20f2605929a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "48a5575ecdf238763242ea2dabadba7508bd6db909a2296ac9c6c9c68795646e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "50546230d2d153e7507c04c1d022baf538ee53848f02c6f3c3f391fea8b8a9c0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c0650b4175275050cbb387ea19e50ac0c35f27811b1cd13817e5348ac21381ca"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 166160, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "96ba8c835d66ccb64962648edafd6fd04df23b5ae111d3309e11560985b2b748"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c06979c96e51f9ca9e478abf3f1aaa1f433681a87c43e3f451c51ea1de341fa4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8603cbf0ca7854cd44a9b57f4cb7ab3f63d2686704fd709da2cfb0d3b11fb0a8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 166528, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b1366b86ec155e9ecf1c857470c2bea3cc8f53ba33c0962d0231d8c7170da279"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 166160, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f13b478369585bf735868ac5c6bc9dcbb7cf1565dfcd17963a56d7d0659e1120"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "49ba1b885f81f1f88aba51b4bb5b8d8819693368ba3c3570ca7f09c1be38ce6c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c51379022d04672011794a164c73a360efdd1dd5f59e36efb31597eeb2a1305b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bf49d623df3ff39eb30d410b02cd51db021131de09c9a232baaf89f4e70c4252"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d083b88ae21cfbddf36d28b3bdde81fd0940ce7542697dbae60b36ff4362554f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6af598f7ea2a135a3624f2798ce0a1be22caf1cd88e175a1ab3038bfc8c02f5d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5bc5d2ba46c398e0649d7495f74e1c73a76e41e3d41aae2742929ac8ae51a932"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "30a8072c22c4aba9a19c9c0e16bdb381b460541a6ce2352ab14af90269749fb1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e6768d9b0f50696060140e51eb96206c90acdc5ba9c1d90af9130da79705f4c3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2ceff83e35c7818c3f109e43851cfbf26a9ba76e60a07be1b3217620183e9eb8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a43db92d48f01989c0f389eb42c39c6e16c4dd53491e6a112889d6e8013041d0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f25616875c6bf869d6f7076c3ab87a3d87ce9510a41f2588edf6630459928d68"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5184e9475d576c7bfedbfcdd00d175e8fd6c3b5c35705b9b412a1a4ba2e97443"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "489dbd660fd8af65a73a65984725d176c2fee9862be492f313ca1f0a25820199"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1660ff75c0eb75403508ccded437103509f8c24cb617f05ca3eb692b2143f4c4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "97e2cf43ed5957ed1f1a6efea7021ea1ab9875174bc98bd3771c01fc3c2bfddf"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6d3175f6340f6b43a3d31c8d4c4fc747c5c6722b594567a54ed552d97f7790db"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "18485eebb06844a2219fdb0cb08551b720885960082e06f788c0e39273ac4183"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7e22b6dfb46008ec773050c8c6997def147e1c78e888bd52a9bb64526e459e20"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0d09cb95e9f84a2520143921ebc47cb1166978d48b92a82e4dfb434abc8526f5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e66697322b5498908506eae25fc4935845942e4d9c7482d0482e9270884edf68"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "27d579723582ba4899d93a813dce2085d48a93ad1017d6bf246a76ecc1202a36"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "edcd1781b7c2a69ca6b599001c7a9dc70fd2b5f8dab7d19caf6009ab4a533892"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "86270c679c5829bcd92af02fd29510591486d185b62586eed2688d97f08195bf"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c6870ed053584dbe474a68febc2e1c100dc9b93b4e8b0bea9602e8b786fe40b9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "84f5b2e7be48a6d3550882d74b561980a7c690a89c21e04d603a5e2375db9eb2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "37ee13dbd2ee00b85f7574ba6901f0f36a8f42dc32a09d943778bbe1ec4f8090"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "593177799c470205ef40f0528a8c1dd6467845561bbd64bb522e5eca2b7c0281"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b551db71cbce04f9b141fd06376598216de48d7c20c93858b3f5765a3cbd71d8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "74c76d2c11a24a3a341c70018aecee7d0028028516279446a48ca5bdbddacf40"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9175970e04ec8e2fe7d3036b07f3584ae67e846066039661c98546f02b9f7481"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "deebccd04d087e436102a1e273c06a5271d0cd3a8ac5c9dd6d02697e6d96cf92"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "379696776301dc7b33a1dc98953eff8026158f080679a0f7b5866035a9036a0d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b1a9e9306551ccd337544d29f04e19c23d0c0b5c305385bd506770a85e13ff10"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2492018f7862c0aaeb90a458f67479a316f13d77a39d6d382cc663fc015f8e3f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b72f236d00f2caad767ebf77ec21753383e1d8057575326e99d562fd08f3a940"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "17d83cf069eb4cf4a6576b0b0c474cba9421333bd8ea5f9ef6967361205253bd"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1c837f674c93c72fcc98051d730c3844340260b686e2f40c6f9a443716ec29e0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fd367a5eac98c4a0a02d91263cb895cfdcb3f8e74c5593ea55433310db388761"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "50a6656b0501459f1d2f95cba0538199426fdf49a0de328a5d5376e29911612e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "371d915e11f086d4c8c60982c4d7071f9ada29eb91ef9c69d9f5b011d9463e20"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "db25c61abae3985c255a502fd33359e5274a712aaa5637609875cf9dd0999337"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cf2b5efaab78dfe3b4017f165da4c4f6cc4ba274c6e83d56948700e3b8707315"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f9f54f8d3afc49f3b235e11f84113dba586b776b518992a66e8e56b206c8ed82"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "024cd5cc0fd55f1b1ea45954466f15f66c52e154603dfe633634d4b9cfc6d3f1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f2e377cd381d3c54ce8386683ab207abc391f3acb14832b91f3dab5e82fbf2a0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "11ab3d66d0611feff25a8000441842d16c033ad9a67f384e5264221d429d3239"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "41f22d96c3c492e63ede0c7dc942f525c8916d8411887084d9f9e74f752be8ce"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6502c30fbba05ed7210598db9ac098113202b617060ce56f089951af893f4f94"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5e190188a1fcc279d6d82846142d9acbb86b7d529a3351ab2713c6384701b45f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9a15e89330cb5b54417022bd0cd76d982dbda3297196227b421af5b060bdac18"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c941e99ef805fb840522c9b5fd539c547ce1a19892ef4e5dcda518c595052b72"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "37814fbdc1aa1009e71544f30ae93fa229929ef02bcc1db1747c3921ec4f94f4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "276bd737176fa149a8ddba10c3d3ecbe7e451bf6c14b85c4cd6820ea56058443"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c4ad2287d63db30acf28b9f72ce5678662e9e07080c0cbfb5cacba8e72cf6b8b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "018e9a3dbca7d48f0bf99043ea76853643a27a903f6d9ed81fd61b8b7dce6504"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a4aff62406778c0ae43bf75c9934c89c0b93fcb499b6a4b7d0945d290d021017"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "23301ab62ff85b417dfe1d046cbac427180953777b3eee5bb9a949f6278e073d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e06d3adf53257ec7a3db8503a7d529383d91c00051d38b626784b42dc1e0b0ad"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "700f093dca766a011cb2040933f8629ca53fa7f5e2ebb121dd4bae0ab8a1f36f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d831ccd26b30e3c49c5cf6ea3f6488f84ddb0f174ab1fc8ed1d360b3ada0969f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "85566cb1714f56b0d1b2c4b44a4532981dcefa737e67116fc0a2e9e4e95a756e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2b0869aea9b7db21e8b4b039eec6b927dff7871bdcff5ee56e258b6bc193f963"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f3cdf3b3ba0ca4db2c181e74bec31ce8cedcb37c91ed88d09da4138298799e10"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ab96441c18e2dec54041f001c1e7510d6109634835156ace3d193d8834110218"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d4e76d7c957829b441e60c3ca57eea0c2a3d3379e9cab1563b0b3f17b04a3d5c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8f538ebedcd77a6edd54f837d0743d8c0004b128147f4c27c20fb82ff2ec917d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ac635419d51c21cd841da031195469c4aae4d77356a8bf392734b7dc9e1553f5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "32de8778ff86512c58ec7ca178fef8f8d77e68c3edff80cb5ba2df2daa6d8f03"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "eb1530ef12c152e53a46362a907041f67dd702a1d48907e960e6e9b4d658e855"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "571b06f0c6489774a15158aac04c40967568ac4ab6939c2e87e89c42f2ee875f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "cbe2234f67036d06c38cdfac26d6959b5e57a3e2ed462c377d59a06b5801e27f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f87915e53813c2187bc99fc443622f23c2b8b5c50ec61485b37ca6e7ccba62be"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "aa55aa5db7e0888f17980bc0018d07a869224c603025ebbeaea1bcd130197311"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "323cea59b9831ed41e4574946f0b1ce356f33d92646ac137832a95de04a6a2a2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7c4e7e58efb0619c718cb2e6c86fb22890a2bfde151a0dc914dbc6f24ccf20e2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d79cb34227ad5fa0c58f7f6b1fdd344958d05f01a5bcaa71a1c0c575b868da1e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "03e666c49e1dea7f1024ecb4354793c3a58cac211fef85219f9f66357fbaf939"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3bab9ee5a26dc888c09fb258a23579be00f4887b422d96d3c09aedb903a4944d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f6a98667dce42603c5a4832ef4f950ba83786b2f3f296294230dd4804fd9057c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5467bfe4f764c1ca22617472a17e681832388c0693d933f47e050e36a1e59f8c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2d2445facc0ff92ebd934f8bcbdf9de50fce5a1fc6e5654d0aa5b7cb85778c65"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "87993412a362d37f5bc34758881ae7436df5821aba118c9e7028bbb8bb6dc21a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3cbf27cd53259fb44a2b18a2f8471c4587532296bf7f9cc15a69080cecddacb6"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8f3ce069c159ef834eb6e06a6e821863563460f21f4062a5828fbb8b52d7346f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "96069a0a0e7f04d66f28057a3c00adf059582ae43c2a2a6a2bb601ee1ccc364b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "77174d4863451b760f6a4d822c2612dc527a4489b3d16f63a0815e5f879a31a0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a0e17828f4e25659a70908168d1d38cadec43ad4dd0c2cb0e90cd01935dc48b0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "73b90b7ce2960c1fb65ce57257f1f81689f638b14cb3f8e037f97606c76a0b06"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8b577fb9356c7d4d937036d2340eb10598d258ef614c676cb909fab411f2142f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5b632ed196d977a0f01c58d15ad07a86cd2719cca1e5301c7709f39019d76a64"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2f9e45cab16b580b38233eb756aa2b9fa12c741782ef2f55a09ee592462bccbd"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "03f60e827010920abb74749f66c44be78cd75fa0bbab9d1d28f57b7c553513d8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e578bd3a34312a4e0c9332135351390bacd93536f3ea9e6ce44d37de13c0bd22"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4f6b134b6eef2d8f167833d3d8ad412ceb5e1fc7b61b6a1b71049e9f12844e6e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4b7087e3a29badd944736ea03f5d302a41a99f2b3ca9895475fe226af84436b7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "61d49917dc483cbea02a00a53a1b78dccb75f8c6b127b16cdca9df70789f0107"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2ab06920c9d9d5b18ae94abad88353a7e40de85861f5f94b4b1755d0213a1919"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4fee808f59ae9e619f232935f5b2c3ea4496460ae529215d56bd29e54b18ddde"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d71ecfa3c0571eb0ee5708fa095f71c06fad25509b0c6d14d0a367c3071b4def"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7ca29a7a87be0b7b53a3c54339c7895b04b0b4d5cdb8c3aa5fc055b6916c25f3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1c7e6e57200fcfccc0a5072bb399443d309d2da954ad1ed163c0b5fcdceb35e9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f1cf6983630c8c867c7d8ddc0323ef81d21b8380f9885f3e135217ed74b6a8e0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fb8fb055f41f33596a56f4b207dbd1bd4d62f948d3052068709f2a21117858c4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3749115feac24db5eeabab393fd3b6519467278699c4edc36f14344a9dcf8817"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d3c7f1a283a65b14eb084f7d272227bef0d31cca40967b63a9d22674d93cf8a5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9064c63dda4a60de9c2ca6e65dd8fe711a5275298ea84ad23ca5650b4b3edbfb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 199296, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "de0ecdca93ff2a50946a9ed68130fe8d1544b9cf7bc8b0f52ff0bed137325668"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 198928, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "420215e640fae6fcd0a3ffe8863e33bc675eb2b172b57981e6ffc1e1a41bda64"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 199312, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "78be7515a704b2b6debf27ff594fcc6324186fcbad900650c202fe0c9537feda"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 198944, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "139f5d18616df9afcd59467aa0535f7488f4dbcc159ed2c3e31ead5532498de7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 199296, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bd8e4224cb35c3d4760e3772c468e947f2cc59af3570ed8329379c6ef82694d0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 198928, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b331d581579c98b16571875354d3e1daa52eab13c690827d9ad5d21f802ae486"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 199312, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a414e980f0d4581aeedbc354f585cf774a38e971dd11c0ee007f4bf0932df64a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 198944, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4c7bfffbca6109de92421125f500282280bb23b23c3d1f96ccaa56004fb7908b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3f007239d11aed3324a0f386b27972bb43d808619c93de0f5b0bc840458352a9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "74c7166cb788c25a18c76397df7fe0c24e6ab174bc5bc87f0e8f2b2411aad638"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3baadbd73fe29e07bc2b6e1485964a8311d0629f0dc265b5cda977ee7bf4b039"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c05320921eccfed792d54d0f829bc7d8eab2f6b3d24fa47ae90171ae2744312e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "624629174181fc48e7d44581862352878b8cc6c612cf55daa2cbcd1eb2caf07b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "eee4956036ce32f30e5b18d621874ab91a81a8fe9fe430936a70f541b5949f98"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a55330ddf812490f76a7ebaf771b6a0b7831d7eefed9f2e2eca475e6802859bb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "609d15b4586f629e445bb098b78d1b58a8af16a3b5da2091f20297198ece9147"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "20bc9b2ef28c1645106aca6957da5e64764ed4bc70ec457a91723ae71edab659"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1f059578f3090e53ac76b0454b1363eedd2e2ac85799c54a46817319d62cbc79"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a66be79a3dea2715fbaac20a4884a28f106c8548d9fabfa6c80874e8a221c121"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0216fe2fc3171eb9e26e82736071ba9cb0c28f19c0c4539214f4b20936fe2f06"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e2de3b13158a9fa7b25175331256cce093dfde3c411fa7fd495977acf3141680"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3af99894b28a2a34835a2fbf306363b2374b803d191d9f86b0f68e8785be8453"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "47d29bef2b3be68df935a81286ea180b4857d2c98c9cebd4c70674db662ed8e8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2f6422f2ae0561592d52ce07681b1989b20942ef263550c26b08ac6bebd538bc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "98c823381abdee0cd55c257bb711699e117b21bae248184fc6e0ebe70f89d7d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "a896eb55a8ae52a05eb7018176d227d5b45644fe3bdc77a34f38e29375aa9f60"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f42a06ec88115dd1f88eb5d260a2b1a80afac40d4e2d2860408a044e6de22513"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f4a3f8877d1ec729c6600a676331623f7a24bf76f9a1400e3908004ce375e640"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "10e3247c45e73060844a4fe1dafd7a91298cd59f462b54f73631731d10788755"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "907b324ed75c0410f1105abba9605c49b6cb6fdfc5e3bfd7166989f06c35b679"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "17b1b170b0add6b305daa3f5bc22eb59cae46c3170f6bd06a3170ca5fd560d22"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f86eb9cf8933f084591920f20f177fe268a146e1ac0e6255afc17d7700514a7c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d42c34066be7c82447e167018f1d5acd94db37d37799c4cfadb0f8b0f7ee6098"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4a2e74b3a152ee2036220bff7b697def2bcd03a5d09f638cf50212f1b87263f9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "43effa4f43ca381f93d95d4d32462335ff261036f751cb2c4fe4e5c587a3dace"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "552664ad3caaaa9f2930ef71b55b9810d071dff92b4a031b13bb9ab7c2f7f25d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f913af34be279c4b99eb0adc6a26ceb4203251edb37987aeea771f363952d658"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "460d5bbec4c99c9f87abf62c98aa97099b4ad98b84ebaaa51e52cd52eebede13"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d1d3efb8b25e26cd19ac8c4995b780ddb608334a621b1f6067fae6bfa215543f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ce6590c755da330de917d7379e1f31d6c0cc5cd7e739db4af67c4848cca906a0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "68d845ec378f8a487ef5fe59dcfb626cbfcbcb5be6cd393b58df36eef6466b12"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "149d5bf78595cf0817d93d70a2f2dad380f87a40600f9b79ab7f3e509f8981bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cd517e639a90c01e66a157cb3f659dc227fab0c3aee7c09042d8e1caba18c196"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "00c38a3503c70cf9cc6be2ab7d494639cd393aca352bebaf82e6d8ece27d8ed8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dce261a3e5c3de28250b8d723a9c6fbd774410cb73390db07c0c36f6afcf07f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1fc021731c6c7a78d74753bdcee72965f43e957313d636ab88c96a371926d4f5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "be830f58c33a6d513b41249a776847d2489282d0cbd0f9dfbe9e8eab924eb44f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "095f2bce1475e1a84d2dc52e585b679e18ed20fa1cb4c5096a373d22e356090f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "06aebd74d0048ff61c2f1f7ca3e4204a30d3dc160c022c10eda9de0c9d489dc7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "246758914d0d2951a2845935d53eaf08e88f7196a6288a7d6c0b70f02e7bd1a0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8e516de392a1d3dc558b6e70b26f395acdf7a06b8713c52b6ebf5f5da671df0c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "2a81525f2f2f2da3107342bd3a0c3f48f954f241333504de69c902f57a0a1a2d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f38af08240314b9dcf6fcbcee0580fdbdaded8d0b8f1f484d1899eebe34f9846"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "4aebb88bc88f6d1e18476438b10cbd688b33b4dea70a4e6530accbb96aefd654"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "09ae1e1f9a08ce8427ebd658f821a57136d35890ed181346ed6b491badb156d6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "162a29bda748c7e91198576852c730ac388ef4e22991e531f7431030dd2d6ae3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5d90a83eb97fb77f376cecfae6dd7a3a8ca986506b5d47a882974dd2724197ce"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "271198e02ae33cd31356c4be70d33a251f50b1a53d4101801bca331e210dc6e1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2041340080d6527264136eb1028d1f88d713f3c8f58690db1f0cd39f35176bbe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "9d7e8124502aa8cf22e20561c5c8e738a2f37159a72b679003e25dffa85eae4b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d18908c7218e861a846ef883451c9d9c29346425b733e0db61ab929e7facfbd1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "61b995d81b826ac7be948d90099182f13b815348a57bd50487108ec9dbf7bfa6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "e7c98b3beb8a1b6efb82af3e1e68069a2494fd3a381cb99f68948750cffbc027"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f9482ef943e5e2baefec694bd7f25446ab142536524b73b2465768f173466c8c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "50fe5f19cc0016ef3fd4dd381b25687adfec11d577c175b4c94e100ec7acd5f6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0400d659d0b0d4cbb0c3dd16214d27f3d557c09dc9cc341dac465ac04547bf50"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "34924b12e091dbebe5a439482722ae7b677c3fc521e53c0fd3e4547bdf1ee354"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "08b42efca2744c842b7c530d89700edcb47268b087f9eff76638d09ac631e343"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "01f04bf4b6a1acb7c08ce4a0244d4036c5feeef362f30bebe409c81055e984f5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1bd19189326796cf954435f8b54572757919e11ebb8fefde52966e4d58652ba4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1f73077151af8f2df598a5587b5c418d87cb9321c1f2ca1141642732e0314898"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "78f6bb53f7c94688f8fca041c2a0f3934f440c4e27c17efc04bd9606ebe422b5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bfd164a585e1c523b92c2c0c5a0e37dd2bf58a252027ac7f8fa4ae7c0c9171e8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0a3b819c943e1a30e39e0c1630116a7f6d05eb63a5c751caed80eaacdf3a9db7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0b082636d5bba02f36bb9de9e81caa91bfbcb6d4a8bfb7bc9776c37d182b7b58"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bf97d285357dbed4c2c91cc6f777233e5b3629242e9e4e215f63cdc1fbfaaa8e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "790ed3a00018f93ea1b74017f4d19998efbd62b2a840646829157c2bb9ed0dc0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2e159a9b5358940da1a71efba0cacbf5778c13a1bf014d767682d05540696510"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8874809027d263622c9a73da53c639f4f12b86408ec71d51cfee0dbfdad35d57"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "00b8b7a5c0db11832939013bdbb090bf0db99ddbb9e8c7b897816998f6e8c587"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4a49b6e836a6b17fe1974b79005f7820336a6bcba278e4bdeac2e5c91b6d96d0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dc9f570ba5b9a7715ada2f6a7abe6642741802888720b7c7f71b4e67e256d077"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d1f3e1b58ca0501d75872ed81048c9508e363384deb46b634915941d4a24e73e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "309a02da5517c937205f8e5c7572b123e21cc1b8ba5b9ca03437446e275e7ecb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a389db688e2e476f801305aa2b0f2b28b9ed3860825c0690d2899fac84b118fa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "31de23b416d7429844b43a7b0aeb6218ea66993573eb6072e8e445ae64518952"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a9648a0140cb64a4c2208ac9937d9688e98e3634aef069aeeca259769bd033eb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9cd4c4937fafa0e0122de635c26be0e5ab320eac0580c7cc48484b5ed7c989e1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a7410f85363d43b7e84d53deaa744867ca92ab8ecaa2b09e39c52b9807ba200d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "91db0377ed52fae7c78ff445bee811c85532f91602782767af426e08c2b0010a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ed475099f51197971589331a144b1d5287f7f0b2779d140017b752bd3fc7bbab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "662ccfa5e022b5e5cda654ebad2c6a907af6072565b94e645a85e2c5712200b3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8ade8f466efc3b2fe7a2be245d064db0a0532471e10047b1f1b6b8e284f932ee"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f7c41bdea35ee2f05a2c92b1861a540659ea064b3d654ddeb23679f0932dfd8e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1ce49a13def72612916bdb105c0ecb2124b8efc722bc7da6f04b400daa190a8b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0f44f2e0d27ea568713f52951626837d5a6f961b05480cd778946f62e4d9aaf5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4ef8582731907bdd08fd566ea953dc71e8524cea82e4dad5a81eaf22f378f5b8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d325b8ef9af6a4d42dc5708be625464f7584602d69b285abf73e98fe220c4ae9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8b9e3b1dfd4b343ca91c8ecf23642c477585a8141c1bc9bfa2de0dd17ff8bbc2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "711cf3265d678f0885e4bae53ac72300bdbc113c7c38d35bb00d1c7465bac1ee"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9877f45f2be14f4eb6ba78955ffc5109f293a884d203e7ad56b3b0fba13ac4fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9862f0ab7a3c79dbdb6001eac4fc8d43bac3385267e5f22bfd81e64eb06e4600"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "56af2269c93f525b3dcac56a28297e66ecd9b3b695a4ddcc0c740f90c336da5b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "41f5a8cb4eb2a1915e36a4c00b1d6cccbb5c88b020a2cd8aed981b2a579d455a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "66936568aebd6b14146959f76ec7a325bf37e0570901473eefee13dd1fa6822a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "035ce11c9ba6fba4b9d75c6677468bd07247cba08c863a6b5e64f110de2677a9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b27b43892ed11a475796b856aa2336107944281b94db4c3ca7cc088d9a68355a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "869aa7e5eaaa53244c9fe620c85b38554de77e7336062f5561474309e541ea54"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "10f59b487ddb7b1d60fddc51bef5c138590478f03abcf7e13d26f64fb887cbd9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b0808e77fd1edb3cc4fd342631599a50c68696e26b2d20990446e2cdd5d8b021"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "655e7a160b3956a50f5756928aa5f68aab2e379ae28f935bb52433d2beb7ced4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d0142e9fcb4f92228f4fb6dcdc2fe3551e130068e8ce677a83761286370194b2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "56f227167c0e58fd3bdde651607752f951b1765b3249ba57ba7b47b749e895f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "48cc93f638d82b3c034d2ed634ca5870463b73309f94dc0de8c23b1417eb2cfc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "892376d8e08524cec69c9df359f9058b5e3b1f3ff537bf522612824d5a1f4465"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d3c9813c0f6e1cae1ffab362a99a09e0008341f77ad6f22f5e350869d708bcb7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f8c8b1097a5c1fc11377f3dbf8330996f25b1114c939e5e68f4bd92b45156760"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "75b11d38238e68fcc66ae0d2c0b1b17ecc3893f9e301b1bf14beaa14db2c73cd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "166a4c6e088b242961c6a37664d5bf7f33015ea7680a3f774ded1a4c30fb8fac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5c3b45c6aacb72284ad9c91997380a931628f3afd9bf7d7399fcee6504c4193a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "2364e74298a70cfe103b05e97057decaa84d4d53d63b3bd8105c6323f96e43b7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "4b601318afbea749b6fa3d975f028fa3570be5d730890367cf0ff8aa6536d579"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "62ce9c95766749f1b336d13c17e20b4ff0932c77f3acb90a896660f3e2be0881"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "89fb9991ee230ff02fbfa6a4b61f5d442025d6f9acf472bc28759c2d013c71d9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4771469baa69a9ab3e1dca40d42f7868b2f7cee8395ffc03c66c71da19c3f577"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "613d6f86c580b607ae0edc7b5a8c1981b2f577fb24fa9f579ae50d60ef20072c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "ed712265520f59a1aa6cf9d91fb63175e2b1bf102cab93b2a5ae5b04a4a5a86a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "e0920022c3423e92c3af8751e00d50fe3b79e95a787e441a86082be04ab80a94"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "e8c53938822c6268185e9ac588103d4aa57ee29e4e1a817691fed69ccac858bf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "b0204e70289a31859eaffe09986afd8eb08f0c8796e9df32e318a619154e2d29"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8a58f768f9b6a4636ed330c291b77a909cc32d2c7fcdc0172e95189bed3499f6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "62b5bd59067acb42958f5e26cfad234c5f31acbc1bdace822723096c4c8f15cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7897a64f5674ee02d9da1afe7238a7531ad326c7aeed950d4b0bb89efb4cad27"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "08334d3bf906073df67314340a47c79b266b93eae638051e27fd34fbcc4a9f01"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "54839ef0a4842dcf2dc4f7fe92318dee43c357e18dbc578084adc489a7ba4d14"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2bae3bec844d0db3fdc01f3341fbc1d1559aa4c61555253250c27bfde4164620"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bcae9d90b901ec1c4233736fd35e3b7bbb1abf39760af7523240498c882bcabc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e655a765bdf0dc2959a54a51b2f4df39555296ad960ea5420129a31313e33b5a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "83de56d93fbe636309b958df3957974cde3036dbba90493cc5b8357110d81c7c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "afe937a4acc45cce780a90b436a0064130b38a5b318707bfba61de1932eee88b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "51372c814cea5e990e18cc05c8f7a51c92addf5892d93e3d9abe76dbc2b21445"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "301bfd9895bbd33010a6ec4f9ead0b871ba3adb53cac9046952950c2b5f7d9ad"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fb0349975cc22aaccbb6f90000c5513759d31b7cd0cdad16b898fb1a9bc44c66"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b2d70d8984b5597d7e2debd69e6872fb6bdd6fc86628a603d6485677588cd5ec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "eeba470d7574fd5df4ed93c20fa39be38f8fb11a3fac34bfad2ea836a3313404"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "54b5d7c8f1c125706196e23143fb05e4e485b7df97f577daeec2e53ee476dec3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "963d3ed64454621ae9334694fe4e42779d922b9185fc047d8ab404af0ac26629"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "604f87b4e88f3670e9f254eae05335010298e695e460a38ba901859bdbccfa84"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8f0668fe709b08a2bfc6dab048be001aa96f159b197ad9e457ccdc110ea68e07"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "947616212175334ebe1d6a95ab3ca7bcc1ae857af5a78b9ded842b8d13841a7d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "bd28fc80d3f2b910baee4197185e714903127b0ec49f0fe86257f238a161e0ea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "749d25b1e4af6ee72e7ffdd8e07dfe7a486c26d2ad77c63fd91b46502cb4ba59"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "9171a348fa76922356966a628643a198491fa250de14817fc6d13ab964a4cbc7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "b26d9c6d4309c1bc7bd1e8f604a4c0bb14c898d4ca15b8da22478b0622ea0387"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c0dcf83f474e0fb9897e2859fde6f3dc443ab4329e78f4408b55928eb1023894"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "b0529b51e43fe1fa3e60c340b280cf357d80bd773ca4bfb77ff814c2b5a2f112"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "19812569b42fa3fa13722578343f277f455835987bc22114d1b74f29a50d6245"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "32f0f76edb2a9ae0a264bfb4dd4769a62d9ef36a497b433ab4c7b1e11918e75b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8dab1c3b916343a6a966fac6653260d24a93963cd3ab56f7b76ba6cb2b15ef80"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "27a69794a51a95cdee32ceecff2387649ffb3f5312fc55b7bfabaf7035310e6e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "802446e4a3641b76dc56b4d875f9901c78420f6a4e43c4bb7e917be64531d6d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3f24aa2698d373ac52622bf055c937e5298b88a39009f0078cbf17e19e79fe98"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1a739ee6d6c8c3fceebdf90aca47ad39f7526a783ac9d2735036975e103a9390"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "21834de4275dfcd2daba6296ccacba54652cd37e5fb36af87259be6c4575c1b2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0bb94eb70ec07cff81995ac3869167f12604cd374f821c1dfc4f50fa67aaa81d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8698c446457c3629acb48ade30e3a75faf1e02a92e4dad262f510b8c8a21d286"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c611679d5d30e4b698920fde9fb631e8521c53c8ff5031e7001b6a4f3ada8910"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5f78013788a59948f9338693572d0be67bf21e8e16b48935f4d66b49d4a6877d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 117376, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ad9152ab32a66114f3a58fc5faf29c1819205e4a581b52a28c86b63ed23c1c7f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 117008, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7dbde6857c4e01230307736594e066b5b8945442c42bfbfbb879d59b75f98764"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 117392, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8fcba1e95dceea6534c036f88d0bfceeef015374ff1aecc610aa480cd29ce926"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 117024, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "adc137af586d31b47de36445f0d41869fa8caf4af771fd6169a13302e1a307cf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 117376, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f4213b9815fb503f1c44c69f5ad5d6f28261ce712e21170b42d5dd455653c7b4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 117008, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c88676c8542b953471ddf0c1ba28ea60861e461402aef7a8bb2d096ae7706a55"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 117392, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5343b7caaaa443de69b7ffac16c50174c0c50f0f91c0efe28bb5f535dfb755ae"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 117024, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b16e302c660c21edc14203202189943050f8ed409eb24375e760420e9bf1e557"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5d856769ab4b06a00a1c22568a1c3043c8de924176f6724674decdc412966bf1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c5a87208852c2c76eeea33f9ac5c219747e1ef760f287736dbfe3ebed0509ebf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f2fa7d60d9926a773a54c549f279f74aa04b78802da987bbcb00656fd23278cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8558d7e7a347b86b7ffa90f75b136e34580b129b6935431ed35b6f3f23abf19c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "007fbc182367f287c6c39bf2463e17a17278c9e40a88faeac941ec817f448ef3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "3e82a750cf47f88d0f6c37ef2d60c47987edb968131c8390d2ba879a575de740"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "678c041f6f49ac2d2c7c15063d6e455968f28542d5944d5c38f7fb1e72c4f850"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "640f3725c5996c93ce5e451563b849dec4ed5f0542b30d6c32cbf25282012b6f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "7ebeae92e7b8edfb4cc103e69fa045c0677b4f712ac6a3622688854ee75437c3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "fdca9b491228361a24f3faf46150ee95d3c783df55287963f6b374abf22adcd6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a335b7d241ae1116ff899d32c13a4102886ee68e91cb3321a85930e78d0e1238"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "35fd646a5136266253bfc9bba3fb77290ad0b81bc0999bbbcd482087608a2b74"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f931188a321dbb1d989e7df8ce5e4d464659598178b393ce7f2bc486d258e446"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "a778edadf9d53856d04df7dec7b91cb073966d4c7077b31fc75135ed7ccb88f1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "1f484b43ac0166eef96ca2878e330f6277b6a367154d89938029526d2c2f8928"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "cb974d68a839b9ad18128b643bc211ea97a0c7e655bedb9d8f41e6f01ceaf855"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "759f352988baf684e62b327885d75b546bd20c4943a5623f7ddeed83322adc33"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "9db79d3534c98305af19b379442d1e63db58a8f4f37a0d9de4d35366edb2d582"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "36b22b2fa7e8a29de90bbb57ef7c1578d59d1650f4a9f1b8ec2a9e07839c7352"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "67465ff706d7086762ae75f221b09ecca5433c4e1de712e838bc75704bae9468"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "56a1ed3fdb0003ff6a3d27a2d81fbf05b5abe4129613933735f902af885c76fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "557a94e3ca84fdc49da900d37feddad2b8b8e3b7221ee46b1f3fd96de22b0520"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "66173bd87c4ef8056d0ff749b87f7cf851e1400bec8e4a50bc054400484ee951"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "415eb4e95b1f472f43100873a6e7bf7fd27debc86c4cc6c6c8656620f6be9ed7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a5a9bb304f0fe5bd80001f2302c2036233ddbc06a4449ef7a878446570671612"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fc597f6fbcf9f953c2dfbbea8fed423552e89cfa83f041fba5a3a1cd7385da44"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "193bcc30a173dfd97fd9d7a9f89a4ca5d9b073321a53814e1496a20da0e431a3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "632d81fba4c2cac3dbb7a9137e828c65da19f1fd7dd1d690ff1a344379396330"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "89313761b2f61bb72e776a696a3f5ebe612c8ac74c0130557ad395f1bc3bfd17"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "736e7b924a64b32c10c5837531d949771799d4dde0286d4a9c7d46c1237c58a3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4debc5cf26f238e156242029eb26ed29301b1806443de62565728e40c5122e66"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "12eabfea363eec79cc5d51f1925d7fd9f5dce8a89b08168312e50c8671e26103"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0aa9e88b3b001d9c7321cebf75fe9030a1b7ec84d275462faf0a3290b723b7d2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e79f90bffb5957efc7ae1131b29247897752f96701499971ed00c0542eb90e77"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9f11b294c068e26c1e71f70461dd7eb908ac64541b80e534014ddacde36d8730"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8c0a8120bccca93453511c5b93b32d3a3985a4a9ebd242e55bf092781a1cd61f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bc024ad37a4b504fda6d940d3a646217a7203e2f58a86b0a469b010b0343758b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "4270f82093de38a5b2f2b35571cfeb5bc4742af17e9a4f955191a914ea48846f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a3fd5349b69afc5ed646041e80efd4cb35bb8c424b89f28009a4ef6884d7bdff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1892d2333e87c6f7b6fe27d095d2163ed7c16bfbade6ff2adc21f5b6ad6d7906"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "6b3eae82a506e97f832b49426cc1feb7f68854fc8cf15084575ed671e437c429"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f82190f98858747c9d8fded8a69444995b115cf57ae0c8d35149afb2810bcdaf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "91ae11456b4c837f1f49e31630e7445881900a2fe0821fd8f475f4dec92e8e84"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8b08c49aa367aee54267c572a3572dfe3da93605e5d44e610599fe4416c46bdf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8af3d1e7b2007d0fa010d7d2f7c0a9099605c1f37c56dc962972c9828134632e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "222e78b18be6999da42a6442acf7d35c76d27bf462590a66e15e89f7736e8a5b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c33e1b004aa7884d139210c927e4e8ebb7017c12314f8cd5b4a1aecd27f2379d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "a0c8bffcd95602cbd6718357847e708742beb3023e36ec434428aadf354e9db0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c03cf6ff2026b75199510f8443535d58df69454d284a58c044e7c8db54c47319"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8e9c76a16674ffb0ac22365653ed9e3783ec609a70641f8c7a34e2d1a955430f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "329f063b532a20e64a337ffe592073dedf27042029410c1cf00cc1f5f720972d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dec3afd0a7b5f05413fbcfa5eaa278c8b858448bb715a1800ed9698e0f4eb7fb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2cd9e6280acfe2b3e0cd9a686dbabd726cd0d969c4f1e92c5cb48aa307a7a26a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9238da0fb6939354eb5e278ca3025693fafab31a2bf7a51ece014406dcc8dd7c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f1bbcafa60ec82848354cc1c5d27d96bcefab4acc399c3f8c1a510e9fbc36b58"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1b565bf022d2373dacc1dee893550aec6f621739909a2d54ceb0f3db21b327a2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ea9daaf29f022427672444b6a165445bcdb041c3a750d89b3e15e9ff30d9d18d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "799e864a1a91fe80c58c29be68f2955d34c27f862532671241ee49b62fcde084"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "44e92d63f04bf0463a0ac890816f5672e6504e5372723ef51050e33fd9a95dea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dafce4e10ede45c7329acc0afe43dc20f986cefdd965bee51bb6f8d8bfa12b4e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "94e56d032a91fada4c2f13e91c73e8a69b4cbb0759879c5791e0e7b8a2e3ce84"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5399c4ee998719ad033fbe96876b809b87822562612731dfa04a979aaf0b8881"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "aff4d30c39318f89aead238e66bee05763f0b6201582ab50be0a73987076a04a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9cf978ee90e37ba0c8012930196278f08cef0f5598aa9cb015d2298cdfe08dd5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "baa5f7743e1492178ea10ee761a6ebffe45116982654a6d7e56fc190d305376d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4ded4501d552ad17f58b53abd03731118a84a663f60ffcdda56de8ba70706686"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b3aa70b846b1743320ba0668efb614014e200d6fffdac9b4de7f9c7b69c22109"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9af7e6170df086cee1ed35c9eaaba546f40f51028ddee5edac6f9bdf3e72c02d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b599aca38c327f87eeb731774709381f2503a33ba06abd1f57444f833ac233a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "32eee20231fe8feab5c689b4d04a9440a0df33e013cc3c9ffc4065ba47d7b1ad"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "87bc972481b32122a943ad95ebf1e7020f16eae8f79ba2d80409b8f0d40df1f8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9f555e58ffc24f126eb48715504ec487c01f56f4c823c5b74608bdf7d6a914aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8284c60f6fd877b8dfaeb46f19db69463ba9cb78a1bded172c91c26497ecb472"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ec925980b56cffd8c2353a06e9291277d015f43f4ed9fc38f50d5a805cc9130f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b5cf3998f91970137ccd105a182003dda4f4ad041e91620def3c93c160b860d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a0466e0310b1fd322dc8272a1e37e0ddd01a287f4ed446ad67f45068f03b059c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "51cc76254214f0ddf7176292e5e4bb0c51fc728f1ae26814470dcad61a8512fa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b90360fa1bbfa09b005e4e75a41872390a31bbdd35bdd7492c12d320c78f76ce"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "027ec0d6fc669b78d1ae189936418bf317eac728652fa52ac12fe9a3964e6ace"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2b7a9c78dcd0a0c17cee828ec55e1017d09a4c7ca27b41c6a6acb43086440d2b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "765b5d53c05374a12e6690ffdf6d1f5cf3a26630bca3b91d07466419e6460151"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "43d2633feffbdc422d37b89b931bc3b5624e3ebf6f5a19367afa00a6b777031d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "db613b241f664de6ca04b012db34bde1b5918151ebcb1a06d6eb487332be256c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1793030d404798e182ab034213345e7e3965c6e035d81c4bcb216a5663a3c977"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6b2cb9f904f577fc6571516ea34dd60fc3015b6fe5430ffafef97762478831d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0093590fb3948135e47b902a1bcd3e1dedf204ecb45d4b7d4641053f3d3f9279"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "25fd4c13f327e5ba6c03971278064b7bd0a7ff33baec09c4d9342d373d963055"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1d35670cbeb7df179948734cf725e7b5140a7ff925a11525b4599070a5f31de4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fea24b2572b51687a743c0a8a03f9efbd8a5dc08bdef571e228ebde57c776142"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3ef1dc3ddae7b1ed18ce6a7b9cfe9db342b493859a607fc7c223edd5c97c0416"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "035b2ecbb11f46cdc43740e47aac761c9f7b25ab118371e35a8070370582571c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ca742b8e94ceaac2100f88cda1bc128d61b7d1ae0f95b1a258391f7e6e7e9281"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1cc1c5f22a939d21783ae19875546b17727633203275178e9d2820c2b000548b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "28995429d4796f00e5b6ee69dc87de5632389581bef2fe09c72c47430644e31e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "814966d490f5fb3d6e47346e396104b7e041fb2f0b7583cdd1ba74d8b5abc007"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0f456374d410a962a4f1e27039b6c26d9350849deb0e8c8b4d605f2b039ade9f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "19a27143ce71b9319a060040cd5d9580c1200e0bfed49a844ebf629f06b3df84"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "fc16872e5f11dc3d72f317725d3086c79b170c4266dfc383f452ef096a4e930f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "81edccecf86888a0b6b9a0274980b00a5b8c853719255fc46528bcf3ebc15aab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "219c0008280cdf462939d5d042922f8bd74677d8deda92bb3e4942fee4c589e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e545675d12cad64daba9b394312807c70096dd6f0022397e6f3f67f0e7f7e053"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "2da173c304b07976792ad2e4099ba07af1acc59b6cce2231f8f15280f3f655d3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f5c7fb3e8f707f0d5ad6ddf9dfede4585c7943a3a3ff90c0183baf4c006ac840"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c1a3ce3a6e37d97bc8146ab59a312ab1c1a47f8055f0654e562fe6174d283b40"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "67253bc95b45dd93a093282f8d9c6dae2a71e83c5dec89167d6db96bb543a9b7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "4c91b9572188610881a14c27194b5bb58ea422a9e5df1a9f6e0caa60009d21c0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "518d7cf505f2d5f74fdfab77c298f27a8daa85a286087d922da6a7d57ea0d91d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "dcc8bc7ad0dbcd01e4b00f72b0d7b3ea624fcc6e77718a3678ae967a60af63af"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f0b154bab6ac3341d3f0a8eea86f5d9ecb2f1e5ee61d07f1b99df1d5b9f75395"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8bfa52368464513a265a324ebc34d6473e1cdf367e4349cf91d20f0d278d5309"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d48e07045729b2cf0d820357e9b5584c1e5921a7b613e48f5b3f476244021522"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fa5f7edfcd6dba4a817021b6a69761d34558d5f5f361526b1bbb5e82a02aaa06"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f441d5f8e233b268fa56957326bd537dec22234e1dae382ffcbd4e35b1024ff6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "06a830043405f8f18e045f2416f14e393b973ce55b8543724d9c842662a8d70e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1fdbf19dd9dbdccaa0c0b0dcb82df0742a867cb57d51c0dd7f9cfa8d3b30ae4b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "477bfd2dc7408cef598ee7d224446086c5750e5cacab40836939f8b482ecd44e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9edbf97d37c5736df0eb74e3671b589478a2bdc0149adef3f5eb6fb03a7ef767"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "633ef4b1518096e7d5cb71ffe10a96abbdbda8793c63282691b3d4f2a8288ae0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b8155657bf6fff330c0104ea9d33705691b5b5d7b86494fac5cd6f0bf03b8b3e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "89abda3cdf19be1b05949c0679a9091ba4cbe0474aa08673a017f5e85935d92d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "77081839bb44f9f34c8a142f58b51cbc2e963d2d3a57d003e6d45597e5527c95"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "19c7022f20d3242cfc5dcc6d91b7c9d134d1eda2db7c23abaa21a265f389884f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f55957f877a41c6f8f6720cdccb3b597ef3b2642d25a2e5c6aae813ccefd450f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b2192c535b7ebdd885dfed2a943a9b8f0fe4ad432e169276eeaa75a55e835e30"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "591be7e8f2bdccd8c52eb2e7879e4004c01477dc2380d2f455009e330da7b350"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "738b387d253909f80918230020557379c3dd4edf044663130433c41c9e625790"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f190f0d2f8ed981998e39d0e1238335d4e1c471ec01872c34b466111c26db6b3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "90e43769fc26fcd42b33e8f122857b538568fad4c0af495e156f1035037b9ffa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "57030ca51cce592c39a1070455fca6aa579ec201dd6b83aa71ebb12f9ca9ae84"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a6d021254c24417322083c1c4847a281d20bf51e194280e3f47a85e7c3f7a73e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "95069291c6261029b3324791195e8e0fa88a1bea7858a7ff39edc3fa418230c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "61dd9a48c6607e809eb28622969b3b3a4b828af4c4bdff3dc6629a0a96051445"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f9d58b184e46d8a9129b308afa0c018c6f3231238504314ab48029ffb52d6621"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "b83786dbf9cf207285f32fca81d6b2c3aded6d840ea5188ea8f4787d07c7800a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3eec2f6faa8ce9583cc50051e97a2533d2f18bd0de52bff606b3956b5e892a95"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d1607addce99d2241a1345c53635fe26f0dd2d1e066e546be1d0bf971c74e019"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c78123bf2735ec34de2e01de2c5e2fc76f378a0a23c831717b6b016f66f79a68"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "6c5ef98e6039498173c38bd42753cd8a39e04c32bfe843384d6000c3a1292251"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "1d558e88052f710b0cff7ea2aba7d3881e0bb937ed1c9f80270d88c8c3a7d6fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "e7eaf7730869de21b2dc368b0022d01ad6b68c27784f33c78d62480421263c1e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "37f38e15594764ca6bf8dae9452a59f24b0b8185aa0166be50911dee3acb26a5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1604d0549e93cb45e4c6f9440e7a8c1e1800fccba544ba0d4ddff12301c3e88f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "29cbd2c9c96507cdf39c9db8680f71094d67675e391682d823cedce6c0aa7af3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5f906145f38dcd8fa7b9a625a58f74d4eb5d0a4362fab956ebdf1f104189cbfe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ba42ebee094219cc43eea6c5bb28aa6529312735d48e87a93c90c43c3942c6af"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e13e3b3bbb708f8426eb1cde42302ff54ee6556445ed1c9af0bad3b114743364"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "233cda76df9c84ad61369a0efab458d2ca5abe28438c146887517c88ac1e1fdd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "987e3c0b87f186c5326a28265884502553c9331ccce63078946cc32a96127f1b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "0b0caf52cf5437af446208ec6e0f5ff813289a04026cf5113fbf61deb17287a9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "629258ae06c7695effe0a14d3c9346a5fee7ac0cf85f01b04c18adba0da89e9f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e31e55ee4c1e6e78c79c0f6743b92a40886617aa7acfff44cb190e8f747b4510"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3d991da14c4af8fae5c6d2b9049d826ea4991bf167976ac98632c5a89cfca41a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "087cbf5c0d05a209d64954ba6e3de60c2a3a1067c441a46782f5c2e29a5f76ad"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "81a57b87810b735b0ee7fba877c6b186ba96a28faabb250d41b5ded9509be6e8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "9b5b4f00dcc49a6be038c986f9d301eee8a05a28a17a167277a48611a8813b2c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d60c497a26ca0e6d7fb7ea2435416477f99c19469c753729ce1f9e2789aeac97"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "5512175b9a704f6b720843d052cfc590ab43085f3ebbdd15a7d717f9e0b45c22"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "47efd100880bae7e478b0f85d3b1637b1fb6b0948182c254bb122ed7e504e4ec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "e1de4df2a9075d19af07a707dde647a9e1978927387822fcbeb0b43c24832e02"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ebf2b88ff57c7c10cd7e5d04c913157a873eea76529bc54bb484669019e78dbd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c4a4c65e1e79df7b0586f8d87a7ad7aa582ebd5712e813c7124177b30df07ced"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1fb00248a21f26838ff51e4c3b0f70f3fa5301afe222cdf92d35758b58d353db"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f4e5e185590df6dfbbbc57d1a346449a2724791af5845361a5dfb229ca127777"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0e14228e44312f46103576ca1c19932f29cbdba8955449f61e89653ec52675a0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6cb8ae71fc922f88777531cdb589989691f4370f5bf371da883ddab51b4a05c3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "494e4fc338e74f8af786643e0d51b27feb2c281c094519fb6e991e6ae8af2463"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "91b64d41a0e8636e3a7d383d41766f987598984f671ed6f9194feb3e0bfa8b2a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7cd5d30ba26702995eb794b1c1f5c09351fecf64791cc906ff1138b1b97c5ed5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e4d92d17492f8e2cbe43f085764beec39e42da0ea436aef4b3bcc583d9231884"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "52f7343933d3c9894993215bf947460d40193c370f9ac023550e48f3121c0880"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1b8b4e9d3a786eed1b3d736dfc7c5f581d5fa4a346a4f0efd0ecf14b85a9a018"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0d4c73b99c04e284589438f231af812a657e73613cafd009d1a8cfd319af40e7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0ba2e7f2654187700d0f93eb239d70cb03cd84415efe44d872463877cbde9cef"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8cde7907b5337de4e3fd566fa8c38649a82aa2153c396b23688233150be3a27a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dff51cae6792d761800f72e839ae9ca2669ac9cd74f885daed15f9159452e1a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "14f69e2ad0f26f96d45c89b146465d0c6aade1f9b60912c151214815c430dba1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9cffd87f1a064e0565ecce68c41f4cf76c5373e777e994e97364858891befa84"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "bd18e3898adc1ade11fca410c7ba18b9831dc16aba7367d2da21ab3d17da7957"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a038dbf2845bbe34e9d224af26941f3b6af5f54a420af8eca652e490b699bff2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "3fa0476530dbef745a033946ab063f513b14589af92151b12b1d5e5bd1d1f5d9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a07edaa36e4b2f96009810d90e3e520254f9e30f58f79017dec50f368bb1b2f7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "ec0d6e2285d9d0bbd03db37f49bd3d35298f27ae83db2ae62c4b99095d35ccc7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "c3bf8bb36a5edbf83bc1dc6e71bfe0a476adbb5c6a387024e81b364accb7317c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "223b927cd13780c1fee5c8c65998c59407bb1fb3d89a45332d2b31625f0a18f1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "24f9ea065b6d466ca20d9765e07abf8b0d51d5997a15c4dd413ec761124da613"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f3d0277fae7a74bd5a02c880a1a66196c4d02314c57a91fa293480ff0bbcafe8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "91cc2475e55e5588eb294a2d08c05b1757fe721a1ce47294d4202bd1a95367df"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "60e38f9bdafbe7332f232e4888103efbfa3f867919b40253a268d06c38f2eac1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "4045b9a7847856a833f94b05101b7d51f86b69f509e1afd7734d50ca03678928"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8d31b63caefc4d80d3882eab47c07fcd16a287eaf6829403b46b301863ffba4c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "b86ffc264fc89cdf3e5e497244f0fc9f43a2ee38aceab450b69d77e59fda3cf9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ecab6201d9f717ed0157a5cef661f27267d9459bd12a460bf48e3254ed102b2d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0c6206c81779e8b9f7874d8d2937b824e3254d56bcee101d214bdd25e98e34c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6a116c569745f73a5c2ae69df7539cb3b2f8e496387d3a814b6beb0a2d653763"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "19b2a3df1fc570e45be4f5a23a42900827f5131da433259f9d390c26f0ea29d0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dd2771c8d05822d296fb4a743fbc4b265c04f033a57eb7876a65afeee6656aa6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c4e3e4fc1d559184e1f63d81991270d921777fa27b4b486e84020d555e0310e0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bd71765c59804957eb05b40be29c0307118401f9c9317620d72ef6b916ef17a2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f6ca4d766e189b62390d0fa25730cc9b90bcbf10c7656c0b40b5029644f85855"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f6bac664108324cbc61bc8fad4f1861e5f0be9509d4f836b86f8d3a6265fbd2b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "13907b6471a767e80a869177be4d5b80a9484fccfe636b94d6c20d3be22114e2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b378344e4703810d8caa5ded48372e2877f7093b8a54eb24689a3eaacb43dbcd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "41a3afd923d20960d5a6ad72d73492ef53229b09df7de87f8835583d078f302d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f0d6f9a756daa68acbd069b65aad1a1f4d3072d5eec54813986a9268864842d4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1b9e7b5724e6b9fab2a711c394fe469366b4c11236c055a184c0343cb70e7eb1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c7ef6ec96527d618156233619235825428e6ad04290dc24071a01f9bf3a29167"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d95a837d2db9550db8fc41f2b4623fe97ff44ae805f81898f99b76b1550d6a35"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6a3d721ba3269a22e9e7de081ba8767ab4d2edc76de05c9156ed49c33692233f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bf33b5cebb4c9001296518dd7bee94bc94a1a9be979f39253ea22666db249c29"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4c3ec16b270ac4736db41c12c9bdaed4331316a52570c9b4c5e9d1c8f8a79422"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d4a670efa1afb29c385d7eb46cb659cc8257ba8ed6f9e955ba987a80f48da764"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1f75b74023d6f8477766f55487a7c929700c3f2707af1d56420b95e947fd6c83"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0f0dcfd9926c61e42b4f20edeaf2d4f4ffb78a43fb91c8045eb6ff4679880e10"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7cb68a2026d8a987112dc6b227308abe2a232389e7a164de83231ec59770153f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "442d63f0a117e19cc077e233f26750cd95000d6e2ed080794c8e671e2849a616"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6353d9a9360cbfb7271525288e46553a5c26787505d311dfe3d3b799a3c518a4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "11a1a5065deef2e8b0f16079c4c54ea1df73cff27c026a2225a5c1484ad4a493"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "38cf7c7d75a440e70b6d6c1adc17f0b6df31a70e71162b7d340d65b29db06989"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "066e7974e72ebd3fc5c2b86d1fd0b38a37a6c6080f2449dec72cf72a4e225b3f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "385099a96f348c91eb751b981bb937e0e6517f845e26a1188f3dddca7a86a745"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f0f4f49915c6e825e4528c026decc78aa1a0f6d650e5c4a4ff0c30137a14bb50"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "facf319a891c6c4c5b2145ce4a2d65fb4abfea7fc043817b20a15b5a9dd7e5b1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "934e2f8b40ad7c62500900028965af7da072ec90fd41c6c0a8caeb3eb88dd8c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "12137b15af4ea8866112d24295e6d697f6c4bea8af4147310e379f6dd032cf29"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7d33ee471b8edc9ca58d942616fcf5a2b8a3944d55cd554bcb3dd0a732a3c2bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2100718ffd92a96f1248f169ea29532af936b7cdf8c72f33557ba28c1acba942"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0893c225059e8bc537063079179204a778f6e6ec1f543c5abbf8af864f6382ce"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f933bc9d4f0c1b3de2fc232a69e71918e49271be092e12d2bee5745824c48625"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "041ad064cdb52046098d639971262d24337132b3d912e1e3b7632d78ed5cca10"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d49e3a74a4c8c97594f285d8dfb9f418e04bd47b7934d75f9c517e8a38f6133b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f0d18745b2156320f8ec7bfc24e07b6c3aae2bb3aca4e223aca823aa8ef928b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f1c5d1c41f8fcb2762b8cf9cb3e0a017ea4b10c9002f967d32c6167e32a0f7e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0a572fc228d723214eca7b0daa8f428ff37de88876fc797cb24626664549d70b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4362747998c8b83d29b75e4f19141483aa78750640906e11cd6f141ce935896c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "de030a2a412006e2e01c602fa039384ca64c15502db582d8ab13461fc2f4bbd1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c81f4e057c94880e9bdf065f2c0bce67df65eb9282945fc8bcaad42b4d988834"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d0f5b0808ff48431412b3c158349aa83e5565df616d9a8f68cc33aa36cb3e991"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f86e511c75e2344873daaef30b71de692fcaafadd72c3aa5baddfde6edee09d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1bcbfdbe480aecb45dd3a9f8e899f51ccadac346ff531775045a0c97d7305ff8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "209373efd0ecc9f6d083c3a932a108643a5f44e31895ff54a05d90e1e84714f8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1930ff48671af32c321318c4929f9caad1b1042b5c4d59646080a8bffab4d931"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e99ad5ae8c043f46153e06cc8a66385653eaf61711771589464fd98fb391cba8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "19ae7b31fa9e18c1308b269995c976fbbc809271bfee3ee1fece15ec97ac37cf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "80a7ac4f37211ab612d4e6e9eba42b5aca643322e481335ddd478f3c268f595f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6a9c9c0f52627fce2e910e2d0821fa4303c76e7992c27dbdedc7792c4938f76b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f39286a42afd9ba8808320225b87a8e0df40483e504a8ce56484e4df5d86be2a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "cb91656b8b08a3c73fc95eccb207767bb99181297df8f751b4167fbc9f4d54b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8221fc1b30e4dfc6d56601e16efeb8dd9e13b2886817f9d4519214d1c55d1762"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "5bf04a27fedef7f5b6efec872d85b59c85758efbffda965a32f957841db2455c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "ebdd1f55bebb2d43aff0db2bb9a5aee2fd6cccec8157198afb783e2a3b96b263"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "60562c84bcc2a2a65929bd5e7674babb081ddb9d9e685ef0069ce6120221b5a8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "28a23987a29ab436211e87c4c6203d224825ef23cb3513bdf1c5510334db74a0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dadef0bb59d5bb7c6f88fe03cae4f29ac5436098cc5f15576c1bf7d3050e8078"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a5e50bf1c0c8d1fccedb72ef1f8dccbcd0d6a0c6c62ebc321d487b6cccdd3fec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e22b12e5c2e0c7fd21d3165fda75c228372d81e1d1cd9819df2b65965dbcdbd0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "57c6477ad6e36da11c2ee728cd9cf0dae9bda940ab61fd90260819acb4b51d2c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6c8dd92f7bd8b5ebc6042de1aaeade0a30d3b9af0280626b09675d410a0781a4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "86066a93590f834e59e8717058f84d4040a0a645f0fd88d906afbf661b9b8095"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "118f0e055f22b294ddf670e8945c37e9902f5aadabbe74bb0f97970629d849bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b2bccd6309ff90e49a497bdb422d287ce77754554b51a0699ebeb8006522ae61"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1989dcceb503cf32b6a9ebc28a51f24a459f40fe2ea16cf49d498e6dc50a4389"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "34de51871d46e726b6aa5c9f7d7c5e0d0d7a9ebf693742c79ef8e6542e3f36e1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "27a324319a93f572548271ffa200a0fb38d8d8b872f28f4367c94fb25b4846c6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8b1809e69edc7c8ae32a3f883bbb4200dd063504a9655cdf9784ad66b34d23a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "000794d51077a7c40a12523036b48dca7370e574a14c62b41f955874c8be2efa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1f28276a85c9e9596da9a62c8129a2f029b5bf882fb320e575361237575ec268"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "50c123a188449c2efac3e19ad6254b4c1c122e34f09e1477975931fd194c158a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bdf9c33176fe0f66154fe9e45201664b5c950910c19449791fa7c63c637bc869"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "9831a0ac3047bf95cb4ee381e9b8d95a764dd40f307e284a3cb628f7253edc26"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "732ad2940b92779efcb8b999bf21d633a7748f4076e767def42829c54a056bd9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "6c34bb2a0cda78c813d88530607c2ff0b48c177a1f116deeeff1281004e1f3fb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1db54593839683869ed61c814b5f6500e6cdf22209de4609b190c6790de72ca5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "45d5ad967c6c4d73d2c448bf827ed270544cfeb06b0af030a0c792dc2d32741a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "9a4106d988b2d58672daf9f3c3bc5191841dc9c54f6dcf9403f642f33208f5c4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "abbb35d65edf4ee98bbbfb00343ea89d1c644778493e260dec4e3b4fe9c3b7ac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "add3368a66be202e1a99ec900a4b8fd957d4dc3eaca1013d520f0f6b8533a360"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "700cc46ea87b16e0b6a7f3d261ef406fd4ddb85ffd760590aeb2d600f063e46b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c97d6f09fc9dab2ddc2b28066138e589dbfb92f272eef58f06c2bdbc03b40dfe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "cd8a12f697628652fe6e24a63484346fb04aa910cb476b602e054a4452f2a870"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "5e9075dacc53a8a0f0ae9e2e39bd3abdaee926d480609de8d2714d810fbef11c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ead9b1fe3c6c3f707d3b1781e2dc3feda14392f9c34c2dc1aefa462fa18fc92b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "37dd3c8d17b9ecb4adc1e5d90b2e4713a3bbcf8a407f5b629c007a879ee9f45a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f531b014b497ebb91e7d373e8030ef599498ee2c688d972b6ad767b0b1523c58"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "41d57af0c66bb10a8290aecbbbda89883508683a1ba3a5048d9682d60fee3e2d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6952ce4e59d09ab31be62f1423109a65e0e870d071c7d116be62915dbf418e84"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5ed730b74483258901a0ad4cabf892961220a110598be2368f6d90d76cabc4ad"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "d5eec0014bf9c3dbfe3c45d0857aaa876f57128a99227567e86e0361b26c194a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5457a147e19f22b61142142b0cd3f19c7c9ff366a7aaaf02d50069fdb3faa93c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f7513fc4a9a0452f0c60bd2e2990f257c3bb8294dbe3e1cf4e229361ce792e9f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "d68099132d3994f04b98d9a66839c4956d90eb90ee26d0754e94ce691ccd8e28"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "2f3eb88a80d2e3515b748ba1d70a5c011e70909902b901527b676050f7ee9eb5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "93ca02a0564bb84644045089f9abf15ce01e2c8af8daf4abcd1ee39e25fc84c3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "121f982c0b62fe7af94eb7266fb403992fa33a35f54dba2faa4026a2adb327b5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3a0092e4be6835b5525f785220b5ff4f097c2353feb98c4ed37a2f0ba64a80c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "4991e017bf14709c3758769162648416f6fbf9bce4a2901cba4a8efaf73160a0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "ddc7b33d28d3a985263a883bf0bf54ba346902788039b544dd32a1703ddaf20d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "216ea426e16be7677c9b286ca4bf58bcc53d614fcb651791f29743e99fd18259"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d7f8b9ad178360e5e4c41f4007dc0474d5f6b9e3bedd085c0a1fe1e619da1f0b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "cf4a94c34e85f7d17873b3899f7068f5e435d1035f2ce30b98c40c01e650a27f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "07aa167b152ea72d1d549c57160321b2da3aeaed586f31d018242124dd834c25"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dd5a3cbd03fdd84597ee6346cf714a1adf51c6764fa29c3ad7dda3efabc3c029"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a6b04fb37c3ae703fb401057c67de4c160336d47a823c16463872e65010861ff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "01f3e88146db56949dfca5f8be80744a3e8d8a8c0ccb1e3a70b37587937f3f1a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "05a7baf85ef86d63c077e5fdef0f378079fc4a09f9f8810a420b0ca35cad6583"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3a298c75856e0a0e1219f932025ad072fa2c9a8c9390fa0779aa1692d6eef683"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e4201ea425c0cdefac6e999cda6e320fd7bdb88a9b3bf8c82c1662a4c3cef01c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fcbf84b0850a02fb715bc7fb584a3735e1fdb9a938d3c5ed687e58d72adc5bb6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "917b5b2817c421d133f366d4ea768d7a00836630affab446ee17bff130ccbecd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4a9a3e2eeecbce246d83ae00fbf476e4cac4b06d13a411ed972386c9d328d377"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8237abe6a319758b6c9afd64683f9d8fd6cc40d07004a0aa007394436f83e00c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "88a45e2bbf3e2488cd2462efc84147c81681f1770fe1bac74436d7a7cea1cab4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "28903ab536d234c751dadd3ecec6a032ae10a457445d2119f0cb3bd9793361e0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "81a89c5daefdeea6c8d70c984a6b201fe48457c6e1a8f724c778e597dbea0cef"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e550d9c9f9ab738f274fef7ffa6853074d1cb42352b6388dfc047a08946b7ab6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "21f3dafd66ca782e562fcac689af006e7d66a47c3594996a781a59bdd0a2323c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2fdc3f2f0b6a9f48538775e815775ccf0944a5bf8597aa7e9510ecc897367a00"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "78c578eb789bb787196d3c9f8269011caef47aee492dc735fc527e064d24d043"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5b4e1ee6d6ffee032613489ba28a9056596d33737d84a11cdcf7751c074dcccf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "3d190c572f451541283e3c8851bd6b7d4465856f7517382052fd9604ceb70dcb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "d04393739243ec19cdb3d4141c6cf5a99952485d9444223f4b12b32b920a3170"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "122fecbfc1a3653536cba7da90a0edbdd1b9b2c6b913961cd8125c6eab33d978"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "d33647bbf18027560ab23213113ac76042ca46178927608e6047b77a3a22c0f6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "4923b727518869153a3611af4c8527d7207554f073c74b80717c9ae30ad33a76"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "39f15732612bfcd1a380870ac6d0c7e7f39fe3075f9713e69aa6b1558b5aec6e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cdc1a6a10f4b92496069ebfab6c1b22dabd4eb1a04976a0120d491a16f510388"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "b33079c5486a1319f8b91bed88bb4042b73018565623d3325f8dfc19dc1d3c8f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "61a2a7e027e0bf31dd7811dec60883cb1bee14983c461018affd90f2777a1d56"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "22c6a7757833d9c5326c6d5ed828e99080700f1472efa431c34098fc07765d49"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "0845bc6c2bf93856d7a56a2b2f1448e7fcd7c2ba69a24d7ca688a8534f2e17fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "7dc00ad965c56b16ae8bf2680fe74d9eb8d143224bc4c3ad5d8aa267195dcdeb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c0693d6652c186a01c00b97c9fa511716879c702f16081338c1bbe94872e10fc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d6aecbd44d4e8d2191d4eb7b971345f35fca265667422fa35b76e3cf6122ae77"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5115be379dbd46b24683fa0e290e351ffe34bc8d6de79155f360ff80f6e7c799"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c5ff409cee8aec77afb2fc8a054c305e554e04a6221c18495a15cecd75e9e4b9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dd7622b877a56e6b1a78820bac363868eec509d2af193db09a3f2986f2c3b0a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b8bdb1342aea703eeb4022883aa98caaf5b44fa26d317007292dafd0d0ea1dfa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b5930978875c3e008644d066151965d690917b6a59ece0309a2e33a3631d3804"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bad1a43ff7f15aa45e4b97c7cc53ea3f7216901aeada02ecd1ac175e630f41a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "644c066030d208b43b05e2e79827c322a8b995aedaa340dfd57bb147b71f5995"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e3981816dc6ad4af5f9a2f6f3f0f8a5a9ebe9efed2d7c224e38c66da87e75d43"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ed6e243cb93c0cdb3cdb581854ad290171da1675e7da83a52afb1916959006a3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "764bff587674ec17ff8b5503f38e25db396cf18a07b6b030aa7d56311a50f3f3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "528caa28b1208e8591f10cdadcac02e65ab8178e1b63379197cb5c07a398ecd4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c9957717e809b8baa378c1420f33057b90bea4a22181c4918324f015125fa757"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "167249b3cf5204303f67b28b1811f529e409c0498bf96d1c9d1d3935de355e16"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "931b16f9d0f844dd8e397002afdfa2fb700ef6fc983177ffb889b5a967eb79f4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b56f50e271478757d9cea9a909b49e0007096fbe009825161a99988648283569"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "04500a490b86a34d41dc49e79b7b56ebfb38ab9dc44d6dcb9bfd25ac482a76d0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f8c52506bc1449a1b88cddacdb7ed1f822f806f1917b3be7a89eb654c3b1a0af"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "31816c24248e368954da8d0fce64df18f218f31f0c0c13b6ee3effd2ac820fd1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2b48f61903946be0954ea2df1f9945103bd13a15302309248b0e5657fecf4b56"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7ca0f525ee6387f22efc0a1a0382412c34563126a04d8da7a62f5e2b7bf3de93"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4d4338b36fed3c018c58dfe52be197436b1cbdd16ecf3dac83b0736509b1b003"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "12c587cfeeb984e181d99f09a6c1ebbad13a637bcadbfe38c71470889bb46a62"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f4e38a50e235de35c17b87641c4e3bdfc8ee7e8b5ae07305a0f8cbe611f41cc5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5721a0a97446c126244571617b32f74f74fb45acf7d0b018c455c1c5ebe2f8a3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9d33e491dff8d5d301c8402452fdd9b9ce7af3d878b88ba716abbfbbd01f292c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "22ccfcb430dae44d03539a5cd521c0aba1ca228cde32fb745d685dd27dd4caef"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "891c5541b3320fe7b6b7730958cde57c15329c18e15739b8165003466c1dc10d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c8cd48cc6a559172002822df7d48e260757f687011e923960b40215215f626ef"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6a2d13d9087eca7405eab9244552903bef9c80b752cab3511432929f72b75aae"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "81c8d7bfa2b7d9fbc29bfc7b6311bc14316c48b3e5d74fbdabd7ebf91cb5c487"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "65ab1f3400d2e297e722764cc2ede79ea7e36a1c91d3e85fff849a27b9bf80c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "53e5f06f37914b92af9dbff6423950490b88b6255a14fe71099099c028eb540a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e086127a917066f736ca32ebce8eabeabfed15892aec1e840e7a35ca5e077127"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "08344285154208817b6fa114ce72bb7d926e415726f7bf4c207b255c76b8c3cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ae97b4f83f8a35bff4f7afe68f9d2e35abfff1deb775d7f30967015df2304acd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5899aec5cda6943e7714a5d2e3efd09426d4aba24d17bd2b5c6f493596f0abdd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "25300a67117d66bc4a06c71a5be77f954281a630f55e150200bff4a1c1a14d38"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9efe61c42cf05220302b980a5c5c768170a1aee1a470c64eb60aa91a8abc3d34"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f8c592d623be79827cc2fa7479ed6f050af0448466347585b4e95e611156088b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "89a67cddb55bf9ccb3e2dfc31a957d286689700659a4016e21a984a799901f2b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "48603203b685375da03e11593a8c80ee84c341d787eec9ba3483f0e88a7519c3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "eeaa56b217b37cb954fd9c86d8c9532874a9c4728b14ca461f582f01fdf6d41f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5122700c1a5252e122cb848d0c59bae69ad03cf4bb8b64ded3bccb3861e2b42b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "db9e2490d4c8824cd3377a71a961a7e107cdb4e443264cf771fe118d56e4aa62"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e9fd66db758f6e841d2e5f5f5a4ad2805381a3a01d6d7c01bda45b056ca9d5e4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "ff4c3e0ce4dbaa7f9cb0f93dd83809d8a4e55359e80b38cf37d1e447c45082ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e859f9914f20db4e485d8b1dd330cf5f88813c7917d74237066cf722bead29aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "d2c05afc2eaf48e487ffc068089d8900994a5fc0d23fa1ddd489b92b89ce1bc4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "9616b2e0dca91d01c40d646940a65dde3de02f367e11ee555c86475eeaa59f52"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e710fcdaa1775a97492c9c40fcc49f2a11bf14f6c15c6de5ac0baf6301adccd4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5e5a03902770d37be612963e195715daa1495af84a091833910a79105bb6b381"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e14066bea4f2668afa038ece8251e0d42c9584fad64a34a47a1093fb10bb26bf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b5c4046b46c30a7433f54ec134868bb8dc1ce36a06d48e8642aaf9a7357a7777"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "1edbb5b3d0526615d06aa2583dcd214cb579c64a1740cefffdf623867c2830fe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "6d3705bfdaeac1259f193f1d7b050ee84861611e4d5acedf117d012b2289df58"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "0361592c04d7daf7844605a8fa15b8165c833b8eaf7b5ca7cc5f3f433c979fde"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8b3c722332f1554a69d1e28aac554bd34a605624a07d9a093b653ed397c93369"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d8afa0f0a1059562e552c1c8a42846e34dc1ee978c1a3e4ecaa884bd5ba84fbb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3deafda6f936c7577f6d60d44fc41aa8f511d9b6f3f9d9ea0c5f3088226d7f88"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "69bdbe7b4294ead052282cfb4dcada02fffd7cf277a59828228c18de9cd1cc2a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "289e012b6b47f49a6d7201d9a223e02ecafbe21dbfa8bf7e6ead19e1def548d6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dba136ce214ac939eb6f29480c1cc398eba1683e630a9501214e320b75049214"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ab80e85761ddac42791f6f734c0343065ebd23061aa6eaf35d4b95ed7763513f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "51adbbee4e13be5ed13e70ebcc07aa440b000cbe8ff50fd2ef33800310d342b4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "89674fba6252fd55eda847a0b2ae4c0691e3ba2ba4d8023ee8fa6b00dbe59f35"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0d618ec394721279b0a3f18cbb0a54fcf47d186e361093150c51185f35ddd461"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1afa0a51bd3aa4118ea381033cc1317295925ec3f550f67eea45d166c92ce935"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ceb71191fced5d28c857c1e2693715a06660fea489936de0a099810c8bbafaec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e8118a3adc59dffe0ce4e4843e8bf0499bdbc1a44218197a00cf73841a397a3b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1aecf5471ba27e6da38411368798b8ae6ab6c65320f8e307bd85327caf2f66c9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0579d9cffbc507701657838bfd4d011bd12ad817353f3fdf0fc260843844a954"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fe9e929ae68a63adc22e06c2e8f8863a89366a48c53dd2a6750f7f44558e83c3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "89efd58af2c51f7eefb6bc884283dfc590fc1d7192bb17fb4aff3848af77c462"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "769b0a889c78c58c5df3ecdb72c669bdb065bb2b28d80b46afb03d321890ff74"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b0e04ec99454f0ff4f22a62796d53e5273e3aa997963f0d8084a53f1d02608b6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cb111b6f0ffd2f56ea58559d33b160b77681b3976aa07de74eda450433dc6c55"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8d92d9157b3cb0e53be39ef28485548443dd56b17a32805cdc989ed4765f1b9e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8ac0083078e7a42c09a59ec4e00d5a64b7159fd2001307b97d6f2d26abbfdadb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "743d1cc5c4193a6321ceca9d4c68226947ff04bd4c350d49fd0f97d144708c43"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "06570355e8a19d8d2dddf59a21dc789577170bd2619470c59ac91c6130559a7c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "c62eb551f5907f87b470432242d9b7a67b0bed0b6579d07005ff69abecb0dd73"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "3741e358b2d8eba7ee3a6fd3906d5a915619792a9ccd2fcdf2ee22d54d5ac577"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "714b0ae34990fa625ba6763cd1c5b0b2b90f6e28cf53df251f4a85dc24a7526e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "fb52bdfd14973d71ca52078f688b06a4d217233f999c9696ea954449b1a95bc7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "63790adfa749665bdbc3ca20850ac7bccdbdf933a6588e76a0907cf20c3575d0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "accfcc31473861a1e34ebbd67a486d174eb061eb8004c3a96aea25bf4dc27bab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "ad9e8e0cf39b5acde618e750e97ed7c7d3b41dd91b38a613dc8a150bc382f672"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "ba663d5136eaba5aa34f944891ea3c53ab8486399b4f79b24260bde359a7ad42"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "04d6fe04efc2956c693ebf11690c2ed7fd800b07e18a428c5d3eccdd9da65314"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "90b563578258124dd0b46ccdc9608797fce41c912e0f13e074ea49085b2927df"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c7886207ca445ae3dd4c2f1a68ea8403efa68b10b6bcb76a4fd19ed75bc3d9d1"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "05e4129d1a89a00b6103f612f7592157477c511cb8671f8de3e4c729f139d7c2"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8c7357e79aed6b1ad22d67d4ca1fa5eeeffb152b1fed946c053d5265a4658253"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ef736f940f7383f36b3c153643dc27a1b754ff1cce34363869f7975d6acbbe33"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b9bb754887d5c1cd77d6b614b28dc5777497f7d471885fab191b454aa1f1afc9"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f480fd92f537e9f4e418cf0fc9617c314b6dcd68498b1605ead55e6b6072c8eb"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bb7274d87a6518022b14117d6f3b421b3a16efb190de8927465e0f1900dc960e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0ed8e7e2c0046d1c4cc8c943381224d6b06ee2ee8cf97c4abf347ee2405ef22f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8ab483c6307c7f5098026464a9e7e5382a9b8867a1b57489428e2d5d40be8fae"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f21288ac3e29ba8696817cd7dbafe6e39b61a92af7b94b48b5771d5112bdc2b6"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8b677c1b13d6c14f3a4c9d2b23b698be6628788c2efc369bace6ac7fdd4dcf8e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "20bb5c7ccf6e4073a7d80fe158d9e6e97a56da3af6c609e7e382871e5f21c73a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2908ad45211d094572be1cc071e841aee9659e54489e20c87001f852052badbc"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ab812618b4ead5fc50c3b702e21de38fdf5b88198eb431ff88915034385431a6"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e8e6d8af7f16be0449738dd8b6c2b02f0881e7c1b57f955ec64aecb1cd2546d8"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c6b9d5b291d34fbc67312e449638b0c37f07066527ecfe7eb4d5b30a1769cc38"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "45f2f32c462286db86db727c437f71164f3250e7624f95d895ed077009d46856"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fd035173550d08aac1a6609ba323d21879da851aa6d552609a5eec6453997168"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f9520d495847ffcb314649fbc7d6caedf4c5c8b72a70db4eee2524a20d23ccdb"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "648df75930f606b0baa64d4b163cfdfaa7a9e24cbb808e707621ecd7cd139548"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "023bf05d2f933504d5db1b457b2ada0f4c6a5de965576d35449f5cd16fa43113"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f134e74563945962c1a9f94128dfea8d16699d8d5363aa9521c67398dfd8788e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 166528, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "64523b44402a3fa8714ffccd8ef2be44daeb673fdcf5036e64cffa703c720167"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 166160, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c260c0153e31e157bfaeeb1fa45827163399471c73c563e6e782e306dd1d0766"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8306aef501bfb4ae4fa400f05eb42623ec9b6a77c894ec5b49c73a83e39852e0"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dd5dc8349eb9323997dbcd36295a2308a5f9610905342cd7ebb80c2771a97fed"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "14fc0be06788d080ab4b64e63f0e40d8054fe55f9d7c13ceaa4bca32abc934f4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "38149d67820e7d4d28c74f183f0bbf015a00716cac9b466264c5a4807311a43b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "344d89ff1c80802f8ed1a683d69dda66ba1886a534305907db720b5da76880f3"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "494f7ce3278a5cd3a0b7ba1781e4e298823e35d29ece6e3552f3229f374bce8f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "072f17262c8a5125515174eaa6d87b72941d1b4a220678672ac78e4ca6e975c7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b0131ff8ed0c1e564a68ee51a0528b4735512e9e94d6488f5a46c63dfd2632dc"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7e5f71cd2b1f4ea23e9e50f15c0e0f8ed3ffad24c05003579c060c3623ca60f9"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dd8e44069aaa1bad06f98c278e18d31025fbd43604d86f3c7a1111cef681a5a7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e56f524183f6b4736d0e626765cca9d5abb617307c95080661758ddd4e7ff879"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a5f547190fddc2d63b5a0ec697cc0dec8329f2526b056cb3b1fbed9f0951b1d4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2bff6098aa94bf1ba2962da23fb271e14d5c4e92022b535ae17f6899bef6a380"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "df500510f692a296780d6b29e55fb33be7a1917f760db85a40f219654ae0019a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "86fdc8a1c05fe17934d12cdf1cd73da37d0624ef80588c93718cbf03a9ff5c22"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6358a06baea72df0fc3526d5ee4ea2b31f2545f7ea11501a83bafc112b81c4b8"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "60e37a6ca9923b1b37d34f7057a8b33e91e0e4aeeeb04856f6a0eaf6d6542965"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f97099b9f57c3b01ba5985818be63f6ab5c8236e45549952365276f955a7b9a8"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "64bbc4a9e3852ae4999c13f3560b5c2243c66621169662eb6cac2a42208c9344"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c51794d1dad004da7cc305930dd88a4db1c2eb6837294a7233e05b33f35c42b3"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "003ae7555b962bb436b2a684be6bc8d07ef44d64507f7a47b230a492a2115a72"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cf7767926bd00a94ca5b67da0de1859a3f11e3c93b54309701af0c0b502cd9bc"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ce342c565fc3781158c84c5a52be5898049d1158fb1c9548fc452a651dc4569b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b0fbbcb3be62734d1028c18a33ac67e3585a027dce7448a080e05615c846bfa2"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3d64cf8ec0598d6988b8d16b1a05a29c7a10c2fba8e520576d0269fe96b73a76"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3b90b0cd8e3a3a113d22bad449d5251b2e30b132fc28b7e01d8a18f61bb0d8a1"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "147d2da345767e576d6a0f8cf5483e63bb7c37de8eed209dbeb3dee6e0e15115"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c24565239ca7f87735e3a20d8896f815c8c15a3a6e57df012297ebafd7fbeea0"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "57bb089027aa3bc2cf36d70a6cf32ad94359482f83f268ce18d9456e4aef6cfc"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "21811cc5ca33a6e658f15bad89db42c888dc48a010e919baf0eaae290a59ec9b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "477dd799791d11936ee1c0547078b1969f51b94e9e955471f38cdd6a293b778f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ca4403e5dcc62d0bd4df2494c95d4208bc35b4e2bf52f90b14eda2122d7a9f15"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "66763de620ca05272da3a935d4e16dbbcdff2b3f2a769e84655035fe00ca55d3"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2b9b8d2c40cc0c99c21975ff8bfe6ff663a74b44c63d008883cb95cbd57dfde4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "762b4b7703176590ca7d7b324fad1644f3c1f6d374f87195317abd647711f8a0"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a91c6e01ff1532715fa154a3e1abb39437d4ff53e38e5b51d51cc6fb9dc7c13b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c38ca278af157ef1265d43927ee4bbae7053417fac7f426967ffd9bef007838b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9920074dd64709ec9f066b44101e6104c5e144ad6cc054fc4ec37a46aa388604"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "db14c4d526cad84d824609c12250d988ef89185613024823af6904b6228390f0"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "36275883a43fa6c62902ff611a64d94c226e8dc4e2c9db18a48abbecef2e9163"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cbb00276042cd51df122ddfb3a47b65c81a61bec5085db6dcac91105827cd3af"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5252d3c190f1669836d5bd81a9e7f70b7859fb1ecb7f062cbfd768e6981aa1e1"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e4b13bc67f74ffbba7e8a061f1645526fe7ffafe01c5bb6476e0b972733221bf"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0d5d0b2c539b05f7a99b9bcf95c18770b475050c635538fdf48d0dc52654d980"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1352db6da75c916cbe05861ada85dec216b2f2851459af0493c4fb22f0162b96"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "32e7aaf924914195d35d24fa4e51710d2320a83cc5e31ba8dbd01314da8f0dfa"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "961cfbf28de01dff6ebb71006658ac5d18c8135b68eac9500411811ade238c4a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7f68f3c6df6944728382247a055cd7ec4e13322b4d9a064c7559ab9cef5361a9"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1e16de9e292fb814c78f5899bd6015615a7c401184fae71016571b413b33e527"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c50626346260486cbb35976cbc847c5b2e3f42bf2aa401feae7061bfa97e8d0f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2d426ce88b990ec6fc66b247bfc0691b75dcfacf2c78c76fad1dcaede5652417"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8f54a5a096150837539f9775ada961ac16896dc4687d196d5fae7313a5b52366"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "67a5f8b34154f6360c4ccac7b63b6a795f26e22391df921a99f3ff93736274ea"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2a9df1cf649a7ad62df12c4254a33106ab93fe7faf1fbddca49ae1a865b31b25"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "28d0604192f93ed0fdeae662416621a6941ac2de637a093a6a338cd3a59d7022"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "47ce8bbd87450e214919d4d80046db2bbe9dcd443143be5adb1362013ac4cce5"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8c9d7ec4eeafaf47c5fcb5b8f18f69b93e0d2a8ce56e5aaef373eaf4efbd8a27"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8028cf6a3b20f6d336470bf4179be20fb63b303d994e9795890a8c17dced9737"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f7b25e3d8904f754182fcf0ff4053b13f347bed60c004c0f0d32b007466c7b5a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4f8a754e3f08afa43901f69c41d850ac35cfc90262da66d1cee137c17e97e184"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "98c006a10c482b2bd2aad78716ad1180b5f27eaae6955a467676936c02ef7350"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0259e81796152cc7f6ef49b536b44ee3ba51e23e8840ff486dcb8d83100344a1"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0ab145292c087b208e3eace3908244be2b67a430ee9ddf6e03e7293c3dd81e9a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8164a4e6387c3917078710c554af86899284700c671b5189a44eb1ab103c758a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "95ff873676bc6d2eadf7c3ba5e9ae9e94491435f3add5a73fc3bf81ac87a3e0e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c641d9579e45bb6d4ce208daf1d8d6700cfabd3232b7f51b0c3ff44e3045b408"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2e02934d7fd006219293a0463883fa5f0b759243e0fac9497dd8b8089f6cd8df"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dfc51eaac361868ba94287044d11c3223931ce624acc0bc41123af4cab58fe4b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7688a2b50d626df85ee64aa64cac2eaae4dabbe2526f187a2899a702f43d6b11"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "391b3b0506034b0a22ae6d52badd81bf266049cae41439dc65ee89b1d0e1818d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f167bc08cb4143d0c31330e8a191316215062d9c3d16d19859199885cb6faa0f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "843f8fdf1178edca3db2e8fc90a47188f6748e612b0eb1268606e6c178093c19"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b4fbf63822a49f761c4bf118239eb25dc09081207b8ae0ef209147943b46859d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "eeb9343232a639f6fd059cfef45d032e0f4a7fbd256ad8c1dd345ee306b12055"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "96b7bb1adef793b8e8eb39d18b0f39484a3d2730a7cce04e15f0847cdb8d9721"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "488c01771fc93a80abd5f5e7255b53a508981e4d2e803555d494b5c08bc07acb"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "db4929e68e65d09f17ffc2d0ca9973c791dffad8957150a828d4364a56d54882"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6dff5e59bff3608c924fbcda85b68cd92d0565b604b292c278e37690203df747"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "512b80c8eadbc7d73471ef158aaf645d46b334bb73ffda4e453974418a534fd7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b57638ab886c20339dd34ba150a931d18b0414da5f488359ce7a7473ff7733f7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1670b2bafd79a8ce509b0d7d5df2f0aac80bef580e5224b6b7d8f63034e6c818"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "81fdad7e5726ad757f81c87462210f16dc615322b393352cdecac4907fcce004"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "97742331541d59401357482acba7fa9e1a18e3b8972f31edb6de895d80b69042"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8a9752993277c805382741637e0f26131ccf8faeadc5b75726cdd2d787ae910f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7ca74da98e7ec3d0d3b3c90b6ac12c5ccbafe77ac37ecf11f7f4a135206f69f7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "07f9a616ba4254f1232066d43363a3746bcdbe8f32ad7a8753850f029b197ce4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3fa59f32cfa61a4ef62982ef4247d8d51b6a67c5b68525fa4493ebcd5658aa42"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4ec40408bca50a3e3208d8edef21beaf73430d629c701a46f37c37ea72b54b4b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "864b4a839c3b96ea7cf871bcb59c1cd8121fed15940f591b69236f032b140e6e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ee8a959637768b1cce982bfdfe87b69ab2816a0ecde1363786dcd91916fb4789"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fda1d5d51e4ad25283f05eb5717d3dbfa4ac2a5273cadf4e3626a9fb8ab64dfe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bcdef6d29c3a66cd838dba2f8a5403ff91ea020e592a35f739b3b2222d9a906b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 182576, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a2e82d70a6595e8c22812f73f74197b6701b9616186735f25c283be8237da7b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "76a98150e63fbc0e3173a68e902c28a4978992a9a0ea1d3cacd9c9c080a5b9bd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 209944, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ae365c25d5bebec636049fdcc6966c3937dfe261246bb696c6f263843d7c4383"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "4263d7916fbc0b24742b8f807fea72ab9626a3eae5f97abcfcebeaed083e0e5e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182592, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "58ca7b385f8317ff2173f94084007c01d1e602fcfe25bea1f317dfc5f756badf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "63980b646e8122cd8265b3018c95d45f31665df36f87fdce62d1909703fafd44"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "76987b7e8d7a72709ce70c8e95c6c39749ef9351b56027d66dee0080f2b680cd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "700f8205c9bab89e59d59e7dee3c361918751f80fd6ba1411f1d982c40b44a9b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "6297a84beca54c61e79d1f5b983f443596e3df1bcb241a9bd7dc2fd3ce2b991b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182576, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "43e6599491166c6366e4aa3a1e5fd96e14afe0522a3ab1774935666e8346d8ec"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 182656, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "fdeb0ab1335ba60b19937daf23b30cd9b0a85cd9a4a227dbcc3d2d018d1b58f7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 182544, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "aefa0adf6b6db6ef0e5f2be4350254464afdcf0ce497a360b22d5f3b2c890d1e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "868496d726bafdbf6c7ebbd92c9cb3bbedee56659f5397a89bf22afe54eae2f1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "d09bdf76ef1c95e7aa3ec6159b5fb5d1103c8f1f81c5aa0f46da2f79ffeeb747"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 180352, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "3c9ecd2aa96b426a925ba7511eb2f1418fcbbe8d237f6510652d74e2aafcb581"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 176144, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "fac20a18144eefd2603fa1a1f458a23eb1f74ae1b70ee00b23fb0d8f00a4c5a4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "aa6d655b6375f8f082c6ce7229db8f80c6472f3cc09448d85566df530ebe1fe1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "fce00932c557449070f2c16e9e01595b858bb2a0498e681fa4732c9fcf8c1b39"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 182672, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c14a24807d3009303a78a1a4fd82cf4ddd5711268e6f2bd03f704ffde77c66cc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 182560, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9e183fd26b171c7ce58bf43d8682fbaf9427d4e1e623bb22c803cad5a336ff84"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "192e670af496488a1119b633a02cb3c16a195d7d689448a045d14e42258899c6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "8862815581ceed0569f3a899b9b9b3ed8c3ca20c85d34d1ee52761987bf86eca"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "35d5e243f8d4d27479d64eff23c9aa024748462a6ebedc7766be83732eb0aa88"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "d272d91abed07ddb808bf16e3c37660a2dc71aefe47879cc72293fcfd5038eec"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b1d0237c3adc3f221a307ebb3e1e717691265de5c83495510f301906b7411808"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1713d05f1d1aea0581cdfbb42c61e8aa3c18bfedf8949b58de251724498eb855"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "92bda9fa3c91ed8985800c547f31942e63dd9edf61db871fcf40176334edba70"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "5f9b356be3bcc54b46df10ae39b5d733255d794d479dcb61a1410963849470db"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "fb283c9b3686eaa56e4392f9351e46122da691ede901d221a00bd23e19ac4b06"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "f87cadabaa1c4bbf1d7cfe59ae59234e02f925db28a85b8126674b88265fd8fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 200984, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "f2c9033d3d361fd3828f791813a9d327f028df57abb4bb5e5a5bac767f51e606"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 196504, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "9f0f0dd58d21cb08cf815bffb2829a49b9ded036c41549f0397db91d32892352"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "978e77713c76914aedd54ec1636d6178c1d28266e2dee203fe30efdbce9ba301"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "201ad41834880d23027910dfb6f1c7f8dbcdf9fa51293ad94a15225b3dc1c26c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 127360, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5cb2f41eec57e89ca865255055c368c4c54f20c449f574de7120d8fe220e3aea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 127248, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1fb9ade6fe1eba9b86fad1e4c7ae17b87ff49dec9fa0088f7cc258ec56f9fe2c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 169344, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "22b357c7db02b3b01bc29d6564e35705521f3a3a5fbf402a86fd85d1b53d25c4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 167184, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "109857a2c18948d4569dde53de6cedff91ae0ec210f49734a8998c5474272aba"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 163840, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "facc920eef19be6cc5bbba92b02aba8ec9e30509dc86fd34f4caac196fb1e926"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 162704, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "dcfd1f5aae7db96b41dc972ad1a7409c6a5ccfd7539b4eb0a116ee6b3709cee2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b3ee4a67f05f6699733b7fcce3d4e846e69a32f6d5c3d9c44ec527e2d24a089e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fcc5dec4978e39892b72dcacfd937818a3fb34514879960178a764c5fa654e23"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "bc386f2dacc53e8503dd7d8a0b8570fbedc7531852207b99f95fd1b76932cb9d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "e7d5e6a8a74aa3030c50401144051826ecb18256a3994c644c98cdcd8e8bf7c1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "0873481bae09aef96ac5f7d40ee6c82d13e1244afc9ff0a590480fc5e1a50681"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ddd0baca28feee594ce7660338299ffbb4c9126c765b19b4cdb91efbce6d846c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8eb15ab31fa9ca4128dae23b82319b380a3391699ad62b555b73c8ca49f47708"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b26404adc633292cc9144d95ab463fea837890d067fb2e2bd8af8a5a8c8d902c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 182656, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "6d803b481eda515d8f0aa8b20e639793589b8559ae6e8ce7e11e3fa90da60919"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "7befafdae7559a1bf74bfdaa42fbafcb7a8a1965d5231d82af101c7b26fe7999"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 175104, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b749ba5648a21e77ef3c0d6c992fe6cd4912faa26514824b3ce410d9ded2ebbb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "caecb744c5cdf1f8bd2e72f0c1866596cc5e7a6f37f76ed6f726b8f1920af715"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 213784, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "65cea8edc5b9dccdeb96d008983ea39722f66281fc2092adb90702a7e6702a79"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 207256, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "7acbeaf7fc67317b107d61647d4d943116d16b3f1e91bb13efe499714917b950"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "10a880a61c4638a4dc9d3afbcd7c2186c0903c22e58e5d387b637ccb83a66e3d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "28c4c493ea0843df7517115f899aba45f1e2cdaa70a481a47dece0301bacb139"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 224640, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "83d58f7f298b381ebb7aa80ca24633325ff398fd409c9c0e6cf08967341d4c26"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 224528, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "94c214d5eb64c3cd001d324559bc19aebffba2893bb73a26488a2d6719e2f53e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 182656, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "15b1353f3fad182b75b588d672fc0c8ebaefca5410a5d1fc5e6c9e39ba3aa52d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 180496, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "5310a5d08a3b21dfe0414b1a818e747df05f4ff745e5dcae843f4b3dffa118e2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 175104, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b37f22b4920e47eb6aae3d4c780c7c122310e6132e5139a743493404ee9dd30f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 173968, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b8b2e2fd6f6f919f0fd550b6af8be17c4d9f0349edeab8edad446013dc50a3a6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "20c837ff0ef8c65af1016fcf6891145488735ef321836830f4d85ecce65b76a7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ec8370c1ce5bc58b47645cbb183fc96d0c80e7472c16336ebcc3d86dc09ba9a6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 133680, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a3dec200dd58b95cb54bd68d28b2c948e58e3ecf455eed61a8bfad809cf253e8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "a45e609e47f555cb8e9d87c706a389780a4e62ef7e77c9842d7716043469656a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 165912, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "94ed87e2ca23f51e91fc1c53a43521961dac7d950b0a94b622b2df021c7ccf19"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 3, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "db30b1be4cd75a6c4c4a932d4616eea0b4740d3840bf266d7220e6783aa03779"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133696, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a3e6565795f2e8e28531dedb289b3071bdbb47aac4eb3218f44092c788b5b3bb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4e0dfc644849fd03c6a497e2f5224088243d3c3b9de953d0e8e40d1bf6d71cff"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ae1a9e683407d265832eda2cd6a3b7cc989b771f4bc466755382376a5be1f62e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "d447a2f0014dc31dfd47c0e0d34756f618efdee30fc2233af591f96e6624d171"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "664032fd7760d72ba39613f98672ebe316ce17d7b02476d07fb04c7708d4b7d4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133680, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "63571b8ad8c6fc0da618dff03e3e0bf0eb7e238d38bb7936af6f9d3b627d9dea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 133760, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "38fa307f84193197d4b93ade819c823c16f255b2d8f3e79c615352f4355c341c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 133648, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "98c028da6779044fc7cb9d040ac0d3d84767a30247d869d46a83b7b3eca2a605"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "f5c31a3f78ef17ffd6fc4d3ac6bf12fa3c2d495993789f2d0937d060544d2d88"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "cd5157224f2d7a6273d3f35462bdefe1dfeb212d755f08d39e2ed2222180108d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128PersistentSwapsAbForGen", 133248, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "d83fade71ac30c797d434e5de65e75c6079483041ef03ce7020ca553690cfb92"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 32, 128, 32, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ32Kv128StaticSwapsAbForGen", 131088, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "140b21b4bdd3911ce60eb0927d9c9a29c4ee35ac04169e843d668982757b8880"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 3, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "aeb6a329f708047f8251708beb0f807d709b66765c9ce209bc01b77beeb5717b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 3, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "d57adadc7d98ab9f0ae18cc97f3b12d140b25dbe1f310315024d216e292df59d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 133776, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5b9ae4e0402743fbc16d55ea5b3d4f2d55490efb3a5e604020cfde740ff6ba3e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 128, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 133664, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "2dc0ed11d6621917a3f17a72a0bd4270f3c588eb5ef8d355b570c5c3457c58fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "285c5e680adaed9436855c4ef2fa40fff03b8ec2b085dccc48242c618ed5159f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 1, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "bdbe57beccbc359f84a4299e1bb768202ec5d3c933de8eeb2de472c722010001"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "989bab612a35d7b2070b991649f7ddac3750649a4ee9c15e3e1e95296284a6c8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "8cac6e277025d04a0cfb425fdaee12ca7f8caaa0f578c0251cff6bf1f8600264"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "19e80e979dab55555a1bacbf01761c7068ae09bbc38a23334b6f5ece2ac617e5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ec7ff599940b2f49a445c3ba047da3741b12222aefb51f48730efc74b1444494"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "676632cfccd56d4c0aa10b65ee37fa886932edf01713bc50b60b4cdd1ce98fc6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "79759bb636106130963e25b8b54d291f1d2f38cc5375f031392f8caa0cb6ba47"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 1, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "5038157c6e92f3e8b36894a73afd19dd96d21a0664481c9137e30e00ad8ac08b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvDenseP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 1, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "53be6ede19188bb673ff8f98bd5e60760a1939d22b404775f57985e0ec748578"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 159000, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "9b50eb39572a2623ab56f68da4481f07719a3adfd536cba005d20b7af626e2b8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 155544, 512, 2, 32, 2, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ec3df1791bbb0219002cae74664b0ec10a193c1b978ff9268fcc7117c4503078"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "3a7cea6d0aae2a1ea1476ceae568b873ecf704970cbb3e04089c51c4f5980ced"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "0a802ebfae30381661093426e5b495dc108a82f15cd676193b2343f998400b19"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 64896, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "62cdeabc0ba1b46806a01e5daddd5d4bc492cbf266248b2dc63b399f0731156f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 64784, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "aa61c3ebbe4e5948f28ea886c2214f87ba39bbd3d7adab1599c571a7e17b4a31"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128PersistentSwapsAbForGen", 125312, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "c415bfb560fc3653adc7bbc7a0853252ef4566743cbce75cfaa384b0f48178ae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 16, 128, 16, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ16Kv128StaticSwapsAbForGen", 124176, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ce82e0abe4397b3421273e3a36612d8b8bc29d38211bdcb98f8742088678d398"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128PersistentSwapsAbForGen", 121344, 512, 2, 32, 2, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "1dee07299d09b7c2322f306b938a7ab82ce29d130ba47b3eca896cfa64be0877"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E2M1, DATA_TYPE_E4M3, 8, 128, 8, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm103aKernel_QE4m3KvE2m1OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ8Kv128StaticSwapsAbForGen", 120720, 512, 2, 32, 2, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "08ce0081356eb3b077e4a5c6baf74d04eeabd05b38acff925c61d838b4fa09da"}, +{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 115328, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "da328a3b009cea8dfdc5f8aabdabfbba3fba74e413ff7b98b833e8a49d027b95"}, +{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 114960, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7d3a9fcdda05c5e9f82d58c23ce2a932b7ff5dbd33a471e91f53ed538d2e5860"}, +{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 115344, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8cb41402433d78a15e78acb298d6c02781e763c53e1388b72a4658e4b19af561"}, +{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 114976, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "826f83cf85cd4bcded111189d0a409b69fec04cbe4866c9c248dbdf6dc4dc51e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "367c7dac171f368108ed377ded21ca5dde0bc0240517b06be8d54a365decb334"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bb32abe43b41e9155ce91ed0a3e9fb772249ef49ba8245ec513f2c54bbedbc84"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d18a99fa54412955515694d82409b30ed867bd0ddad033402f449d1a60612f3f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d2283818b8f68bd06a894f0b4e6ff8ff6eea86ce516a65d7ef207a1d2824f8bf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "124953ec14991f4e29a197b6d6676a7556722922da35615798927660bff05a90"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8cc5168e225006b83fd3593710f15b429c3cef91bba20d072dba911ec8babb5d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0bc19a4c1d55bb49c4807e58dc606f6235a2539eabe584dba386f8918d022677"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "493a5744a55a5fcc5d8358a901b1bcc32ed32d3d2707d3fa7b9acf7ef60e6ff9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "41929e6cc749d1d90c202dc9e2eda0b08e476493b74960378fcb9a1d35a46fe8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "037404efe1a0a50180a982a51814addad1cd18b0cc8ca187e8d9557acb75918b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "83fada904a26f265a6d4753c392c7d155e36a5a0a0f8d91832b2b25880155fbc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "494638eef8e69f531eee811119a840e95a5c37ac0e1a3c3921e0d713204741ba"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d743c7e61b6a662433996c307f6573fb8040fc1df498212bb20afbe1eb44e94b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3fec1a51ff24c9cad1060a2bdb4fec5a2bf66a156c8c22ee3b079431356be24c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e4207515cb02fd4812072587294dc11823aa9074c8ba03aa434d5a5a2a51aed1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "43bee2fc3df3d892e5a13374ee83957c559d3a382b78c362daaeabea7bc4cc86"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e8c4619a4fcc2a8a2c1eb2adf966c523321d58249e3db283f8815474ab074313"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8d34bf2c00512a516fb7961150647d6b910a6e2caec04cb36b748e785b4c7a2b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e6287f32ec355ab9dded0c43af87b2d139971b00109260b161c594d451156c7d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6020643e31a02685fbf431c17e2a6a7ceba0d555ae510dbefa307e4c889015a8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "86fa656d689cb0e883b954f60524009d5e2aa537f2588beca774c8a2d1057587"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6645f91287ed35a145f82e955c09000dbae3075ca74d2b6bfa28d4464d30cf27"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d45952a1b8bbfc0538aa008a4da83ff8c1ed6971e89c2ba30f0d19a950438d15"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "eb288212a91df661331b715722db3e4da3bf046895c472cb5b4cf00a47753abf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c4feaa610b76169e770e29ce5c161b1aa7cf413c904e729b4d7a8bc292db09c1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 166160, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1783515cdd0af18761ecf91b2a7cf2a0760a1c412a2c0132ec2e3fecbaccb9c7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3c5e042d5b7e7c9c92eb9d4b61c6c970aa1c8416321b9fe3b412aa5f22aee32c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3c09e66ed18e195d4d58f3251e6dc9f6be9283d88fbf3fecbf82cfba28e2145b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 166528, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "160d78501b247ae9b0399b9d094857ba649ea0d7413857deed3f203bd9a7d456"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 166160, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "93aa4663fc8687840e0834d7de83aaea984c209ced2bdde1834106ff3bd0f61e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a6d8122d0cd13bea9cd0bd734d84ebd9dc66e634c5dedb536fe4a3b5ff057907"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "17c2616c7babbb913ef49d98dfa2de6824292f465fa14d826fc7ea2e291efa1b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "40fbd5c28766ff439834fb22e92dc8ab86d8032397087aa9396ce8096792908f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "639a40ed618a064f8626c2a24e37bafc00bbd637e58628dfcb623f3fe659cf79"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "db398c9d2dd1f8b038e176ba29517a96a479cb863e94ab61784638f563ca7826"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a4b1c3e6a6a76e34ada3d96d4263a92881804d8868cfd89396fbe315770a26d4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9651cb7370c3b27fc36f0e098323404c189a0000b8df3c9e551b2e0e7430424e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ecee2c76872240b8301b8b04c784c9bc418bb2f6b739c9de2779f18fa1d6c9ee"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f626ca0c2bb93a024182d8060ee351be7b9a824518356ae926d09e3b62af76b9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9ff4d983ae9a89acce5bc89daea4cef99f8974e4c3c4c85de72a96c79ad0cbad"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "13e0e0c95427801025ceddf2910840d6572bcb9ccd29feec439389e02ed1d7f3"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9ff45e2b335da0e1403f7ac0b96af1aa26a9cbaf840d63b0f59577c8c51614fc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0aacb3621f476f1883f44d858b7e5647b19acc69a6f6d42075e52927ee825f70"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a1161a5876a73265e54757dc6ae1298d9009616485c511d32b7e9c3b19a68fe2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "86372156d76ede827181aee32f487d89e3c95bf6b35508a15ae75f52512750bb"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "170822fdd745c426e46f2a63b7bf496b4ea4e2ce892a09accd0701996bcd1af2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7beb6e94590ed1e223dbb61de451994c38ff28dabb334ef7734cbacd9386a7c1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "397482ebddc01209922016118610739b9258d30d14ff26e84f147ecf040508cd"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6a32dd653e673772c97422c7a40b9dc2862b9e90fb85670b202e100a0490999f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "49714336cda9c77bcba1f1d946bfb43dd863f391662d5c6eba54a1d0adc13ddc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3328139b62a6bba59cbf1484cc7051cb0019fa5c2fe6e7df4c388bba62b99977"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "dc3d0e5e3af4a4fe65292c1c08c700331e7c06e63480e4fe75dca0b38e53b0a0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f6bc70f6700725d0d4c53386dddcdba25a525aacc066f7c5c007026dc3ce615b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4e505663628e3f532cca31e1d3a78ddd439fc151c0cb1d36de0e0add27bc5ada"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9411a1ee7197a76ae143e7bc56f4e6fc73a683d8fefdfc32f6ccd53466658ee5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "572ba97385d88a8c583993c82710281e9c95d1a6dddf00501b3c2d21fda5c83f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "42612049575f54abcd8f9fe822b4b4030213e211d4453791b817feb7f5cd25e6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "00a4c2177a90df91637b365a7b680054792fffb73a755be85eac0b01ab11061e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7ca65ed332f6fae769033f765027bba2c60658d0cba127ef4ede236fb81b577b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "16dd4bbc56640e64789350e4e4fe4da2ebf29ca221f4665ca74318f5ef1ddca4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bd69b85b387bf1a89c88bc0f056b7695f8ea0a534c5a9f83c55ac876c5c6a63a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4167a108669b3822409d8f7d0dde365ccb50a46481adcf8ed6be3c6553734b24"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f5520003f22c57768a33363f5b944f755d3bd209079621cc577cf9e0f1e1c518"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1ff08c35827dba6cfda4ff0a57993b662ddaba9d35d662f641fadabc6da8719a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "91e9f2b0acd0b91c3e97ba5182f6436e508dfed069342b16d458aaf261658f23"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "df6591922fb855f0943994af98999da8583d3beee7d6f5f8b37f1289fca3364d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9dbe906e60dc7964e3d61ccdd697cd7fc9d9f044c3207281b5f80fa09ac58ba5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "17e063e0174592dad41ee2b4b27b5a68844da26fa514420828f5cb459852860b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d7450123ad8b73ce27aa2596e35e5605e213edcd7bf31a80a984332f09b7c196"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b7cd57c3bdfa37ea00d9fff8be3b1bda16bbb587bb4334329e909e9ea4b8eb04"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a051d99c84563d0b97867922292dae3c6d98c4390e002f3ed1f813359352a77c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2f602e5b38aa8c1608b902cf507426cda7d12541a32c74aad436e9b1b30ea40a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a64a6e387a8eb20f5cc724060cb6ad458ac8c299236bdcb43d59aa29cb28878c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f0da86f36454aedfd4acf8b7c4b47c8e7cf24fcadb9b76fcba8ed4ded3232ebe"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6e2ee4d2755f2e34013adcd1981c32d19f2020d338cc7b49c4af251eb1175f00"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3a1352f0cee7ba4b4a4f7039979bb04f5b541d061ec5b3da68903da64013451b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "014b022c8d56ee6f5e55d46e94ccd684d8f1540c0da491f1c9164822b4771fd6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7c475421d38901b4e7a6920737f1373c4e9d33dda4c7bd4f18555f793d29df31"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7349b48e319aa002cc80550cc32f766d498d2fe2e282eeaae5e97a045877ce2e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4e4dfa9ea4035c1867df61e4056142f63c2cde70ed4a74502e7691a88a8fed40"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ad78bfe8e0346bf0adff904edd066333e5286a335eaa77bcbef211b1373d849b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "edf30e253e844cb583ede99fdce84c298caea0e6733a717974536b92a9c5a3bd"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4721140d96d6889c4186ad985c89b688d49e1e08e7f6fa9974ba46a9342e19b9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b76be9e47161177ef98e94b25ebc96717727efafe9168dc9302de9793f1f1477"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c88d7ab01e7836cf5c69e0527cc12c38d93cc39257edd1b425a1636fab64dc0b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "852119d80d4a3a1cecb826b53eac9c01a3866a1c47fe0aa0db161c841af9a494"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a2f05b1ef0f5f8c040fd3553d174ad31cd6feaa3c71b5cbd1380b9dee0d0cc96"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b9a2e764a4b0eebba2ee7d7746e67894fdd95e28f39894dbc158bb0484e2d433"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4af3e42a9cfb11eba39a15da5d261f7344675dc3d2ef2e1f682f2cf7706370e6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a143d879961c1931bf48fc2dce7609202efaaf642cfebff62328d873195f77dd"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8b883a6bf149a7e36c150c9a1c77ca9f4e46775cbaf7d8f23708ad7f99ad2f20"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "933116d1be894cbd45441bf690bfc855f86f45d35903e649403d2e2945deafd5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "eb70c6223274e6fd36185d4952125edb4a22c7349e19114567af3bd2ef6db137"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a44cf21a9021ae4f5544c244e8711e1eb352fc53c68bdb8905c39f61646b7dca"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f26400f701cc44db85888e1afc5785d907231639e91d2a57f2be6a13820663d4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "84ed43c4a3465eb0064ce8d7a6607d151c5a099a91ecf8207ef03fd80985197a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7a8611e4c9ffa9c8be4a4778cf4579a349c01df3e5d6d8665646c3cce6102e27"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "dfe5af6c75e57c36b0d9a55cd2f23868b66e454b50dd16eafccf4734ba9875a5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5317e7d4b424de7f596eaa5dc2fa94f1c136cd52397ce0dc786a7cdb7bb23df7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ea1034b094fb88069079dc497b0aa6c9e8d0b0a7c3d84ae04ca201fdf861448c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7ae5eaee3d3677583bd7032c8296d4e1becb287f6733becb99da8a25708e5917"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "788ae6ccc1d1c1dcb68c2d31ccc614ff1fb41625f6ff3abbe4119032389305ea"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "38842a603108a05c203b56dee19a3e4934978a70aeec39b878f298780884ab8d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "bd28f868d45d19586237d548136707ef28864b8ff86d69bbfc404c2405ed9962"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "94c4c80d0155abdf3b850db5638a18ccc346b522f6c42145a323dc08129c9cbe"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1fbce21dd058dfa603544f077c37730221b399c6d1e9665135f2769bd21d7962"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f366daf4082a10895b25f40c1a0118c90d871450b31ec68ceb99f1a67acef5f9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "157da9dab06c9c3c259d273287c983ea78380c368ff1113c72421d8bc6320776"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c6f19d732d1f7da14c5902228c5c392b99a7d2f4c9f2e6b98ee789c9b165fd88"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5bb44d0c344def0a3cdbf5a20175c4bcc658088079ab2b62f628fe09de3d0795"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "eba55188e1f6c0faed46545c919a762ae44542cad8e62b2bc30c78a27f48f939"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b0fbd69f439af55d67e4442f781fb3cc42f07db62be98269b5eedb85e5b1c850"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f33774caf8d6efd70ba0f66b6778ee82f64c9e08fcc640698b15c51991182369"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ef0df573a15af3b9847edd04206e79c4cbf2ea430ab9e530fe6184c87ead38ca"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ccf95715fdd20729d295f0a4cada4d90d137e9c43061e606ecb5939ce6eef4dc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c75cff35ddfae747faf094d693af3d7fda1907baf3c1cc0aeced9970b8a1e76e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0f8a8c161aab48bc24f836c373c2de3e9a8511afb2c2dd8a478a924f67dd6276"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e96d71af0117613b37a6fb4ec35b627aca623d114a16f66c3970e37e7bc03759"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c5ecea4a5a08aa91d0e55e8ccd42287388c7ef8983cbe46c265e0bbd98594d4c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e9822a2fd3c0e9c3b20d450a8803251f7cc6d896d99ae79da688d3b7d11491e6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e93a1b9cb8b6571e0573fb045af4f02d5fb41d284ec55c69f92f596a0e128f90"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1da03f1d8c35e24ce994e1adc82662641f6003c15b93d320e332b00d81a6f45f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c5778cc1a69e7cb5ba50d974934f97e888e9584a82be90ff853af204128084a5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c702c1d2c961c9454b8233dadcd230fa7498eaea43922e52e027b6e5565d4388"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "66a7e92b419cdedbf17c4d7f596e101773cfbda56102834c4d6a23616f02b160"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6dfd81137947667ea70d24d92c50378fcaf8433ed42f185a60e459b58c420dde"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "98822fa26f3ccfc43408dabf9115339a4f12175fa24aa79ec7b288d794735f58"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2f5d932aa91db58fd5032619ff993629a32d7de2bd848bdbeef86e8d97e6ccf4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 199296, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e38881ae00ddd0c3bd7330da4adcc8156206755cc9d1e53beb666fe6b1157388"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 198928, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "73c3440adf2475acd1d68cc1c90127da8f985364f2e0a8d98d245a346fdc5a70"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 199312, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "33b1c4ed021a9b60dfee7741527eff018fcfb4be7bfdc8868ed30b2f044e0042"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 198944, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c3e31925e1912ea47bb47c33396c8ea963a26e75a83e6880914fe3ccaafa8c75"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 199296, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8fd7a2f55c5d58a06f4344fe9846c45adbcc37f01172d2dd7b3e62aabde3803d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 198928, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d9b50418ced7a000d549d49e2c3f490cef57737f7af588662c9d10eebfaa9a29"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 199312, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "049d73476c433e7416396efbb02167ebe73f6434faab4e04ecd524403b2330ff"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 198944, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d3148206c85ba24dadad0467eafb00ccdeddc29e122c0d7a89ae56af5e685a8e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "771d4d679c720f7a371d9e5a32179cf122d3d6311132eb698a1b1b81c4037f85"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2bf97b22bb70e174fa3a199c565cdb4e3eb01dfd079cc70b8e550dabc7da1550"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "206213f0cb91d7137d3d6461aa61e9eb1e58290ad4c58ce493181b00f77a0c2d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "33d23b58c0252019c0034e085b063d31854804094915000f36903b3e7c741da4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ba1bae93264800cef7a74ee2b60ae1f169cee763817e967497c36ffd86119ad7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "3accb6e4b7e056fd83c81d089d539e01c900e339afa6d7f6bc53ae318417aaa3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "926b3d07575ea2faea8ff53edf49356ec362586fcb7246503b1d2e4df79ac813"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "aeb6b22fac73f23cc2669701adb3a985fb5d48f5f69f2f8f428e8d407a31ec29"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "1ef173e5c1bae27adedb0b6ddeed2b523c0f23be614e918c76bd058ce4ed608d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "922226f15365240f0063b0da16903a03855a55d1c993abf05ee8a3457f5bce20"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "6d1b842863cb66dbe143f6cf6f1af562b14f5e7f4149da22eb6527a41ffef19a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4558ab52a5781199e5849c6e625c9626601871cfcbfbea184b5cf0ede06da0c1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2d5c58a925687b06b59133efa930e80e78291ed29a1e390e909704ba032f310b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "2e4a8ea05238b154319025821a6b99f2886e66b5406d25c6675b2f807c1c8d42"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "e64da2ffe0afcf343827c401b38d72b170de76c07e54cbcf71489891ea705b7b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "514cc0d7cc665817b35e246adfa005029a7f9d6f8deb0f6238c92e6615f8171e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "60a64f1095b08a99d5f1f5c93ab182cf7d864a560a086c9c835048f6480f4b78"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "1de1c91e6f5429d05e52176dc41f3bec1cfbea7af47e862b48d24ceab454f3de"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "c11af1d5e44eb3ad6b2aa2c2ac759d7d475960c7dede787a272fafb3d1bb3820"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "86423782c27ca3feef0effbd17b5b39378f704e2bb1b6d1b56fa04c00abe079f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7cd8209221aedf3d810a0237e0379d21b9ac5cb5788863228937f920a4ff1ab7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2972324312ef5f02dbb360d2f4a39755cd68c2d94f06ed5ff11ab110e3606251"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f523a811331314ff1a36983f2d35e60ba48e1702cc4cc9a5d89992324e788ab2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b8395a1be37bea237471cb91cab12ef5232ac4c3df59b5e5b83459fb0d931c81"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2262415cb4440191590fe0f99536a4255c40142e8148280b7dbfc78a57fbd45a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3f8483823930fe259ea1222e1939d1cc412f53b831e29fa671f05c99041f996a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6d410e96f3023b6560191377c770d6258bba4dfdcf2906e818a2483d0c092c9f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c40a174ad8c5b384d09c0b8b0ac34cf4d4411a679457743a0b690e8634460c9c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b0ea3fbcdd543b69704ab488c02e105d4eabcf98d76d5dfcfc02616da2d1a382"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d957bc9fd3defdb6ff68f2e02b77edc6fee029d5a6e368a53e57cc9a79381b5c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "14085b9d984671242a6c9a1efd79f390a073f52a0f51243f807cda4ad619b8af"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "456b8a5f616a443425a3b68d38c3fbfe8d3a5bf89f9ddf0db2717f97d68bda7d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fae2a8d78f3fb539f133b2fdec41d92eee80217c0def9f60bed2d1a84527ba55"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "408fa419839db91af9afd9a3658369328593560ce6d0d39c20daf2098c79a534"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6a97a2d433019e68fb566a48d502ece6d1a5e7ac4d3bbe49884076f24ec1dc9b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9e80ad7992b027d3ec71f29e4ea2d44fdbb4f31ae91d81094595741f00479cd7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "08ca438a9398ada6e2ed444cf9fa32f9ec57ec71f091dc77e448e4eebe866e20"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9f852665c4b172eb6656389c8fe94fe2a1b1b8b89e605b39edc5a6cd765e11f4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "95bdfa4b0cb6f6c8fb1ee56ac5e765dc426fe60f9dd7669ff1598cfaa63b42ae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cd7fd65e67b8aa7c8dc07cbcd6fc9ed875df67c833a5ac9a1976a97f9785f21f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "796532d7d88e883c8f66521485c536c708a06cac88499e22442479f319331dc9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "f40975050af72b71b0bb636709847b2840335bae3b524c1fe77e204a49875139"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "21b66d13a907f885adc35f6fec0285568a8830844a238fdb81e3fb818b63bb8f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "611acb6163cef4c7d198dbb51e5d3795ebc0b84c5aa6f5469a2cc7a8cbec93e0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "1f66d040d7c79d96c070640a31451b2e5b5eb9c3986eba2e3e83017599e0a497"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "366ca4a2f7715e2d50ef6f6d3063c25dccb97b67217061776dd88b2777652333"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "60c3c655545bd5dab1ebc65713125358f9ad34f05b0567499103a8928bb088b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "21181cc4e2927b8254614771bd8e8344614fa3ccb593bb5318018acd429ac5fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c982dbb762c64e1c45fff5d0c918d6cdb971f963e9927d2f6d583b4469d64aeb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "93b662591067c207d345194a52e0e689ce639fe6d837e5141d16324223e916a3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "c33d29a9d9eea04cf4c30e1b3bffc16bbff4d3fd1d9519346f4142b9dcd54148"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "ed00436436ce8af5b884a76d412a0ae241d5f0665b46d4d0dfa63f0576e80bc9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "a2ffa7fd638f8f3a400cf5c55529e7f955ac5b4e530b97d609bc3d7865534b7d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "38a533a66de6b1e6ebaa6867d991f639eca2e88babdc0a9e716c81ef5c0dfe72"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "a52ffb53051200ead48855e94de8d259c1c1f28a6c574c6232dd90018f8b5460"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "826c975131911b9cfa4f49fd111498db50a10df70f98b11d69c74f4bb30d3cb6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2e167d759f52dbf53c4d0a0d6832078dd3720935324e28e46acd1c93dbea1688"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "271eb89fe452dbd959482f60eee1291a2a4490c22fa66a37429f83789691572b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b2a099aaf29056d9bbd511ec31f03324120cb9d81ab96a520052036e7a6c2a86"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d13f2fa19e0766b38c56b5240ae1c4753e69803082f2acba62489d96f4103ff3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1991adbba4218dbc02945983e103abf8b8c79b8c2ad393d8cb271207adf632f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f6f3b9b4b6f753e8fbfc22eeff92f2b5a3ee9aff565ae67748934b9ceeb693b3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1ad13d294ee73ab33bc3d06d076f591f394b23c11bcd3fe63dc1ca427dc6ad6d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "859424ddb565aa8ec41374f3bc4eeb79ca4a1abfea90c23e66e9408a9da2ed83"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "24f1fb813038d9ee65dd437b7a8456eb9aab2bd8e4f690bfb580beffd1536585"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "737943bf5b460d81491a2324014dcb433732dfadd985cc58ffc8cea61d2b3d77"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "50ee7cf7052da5de0e4e6cd2c4d706d3daae02e9087aba5bbcf973abc02d9c5a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "acb750b4538cac6893f6cf54d5303e4f24175c917a502dad1d73abc5a4df500d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fd468477235a6b02be086ebf83ee3defe9510ae665e12705de36006e27e20958"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bb2adf11df467ec5ab8dc301d904ffdcb9a5182614bc7458594cd08b46364b35"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d2b088b16c839c073e394742472f628cf1eb3b72c9618a8f7da3be75667ff30c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b035144df8cd1154d6cba3f2fc73dd4e5d751ddf49b4ea56d38fec7112bf0751"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "def6545f72d53fffa173e742c15df5610fd3f1c157d8d9cda3380feb56d92b46"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fc3cedc9a16f19565ea3c81e53e28b28e4536410f5bedfc009c7e77e8a595f79"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "77675c0ef9ce25e87942f99c24950acedf79e468c8ee8e300f026ce56828739e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b8e98a26140a046e7c54585740680d027f36487a88216dc687761a4d7cbd7f86"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "28a24ad8744e0f0e139a4f24abbeb46956f7b7664ad824ccfb76835074f9477e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "27f0f975e0adfaaf022359269dafcb95620a06684f7e8ede5c7c93a81997ac3c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "db01c37d8e68379a3cc457e6d3db23fd0c137d9443b8ffa911a3146e1629e1e0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "425e8e23e4ca07c0ea1cd457d12c25766db625d0daa38efab3635f1c8dd0c86d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7ff0a6ede506272916309cbad58908899d0d208d149ff44da675f97f8b01ddc9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e54ed42b931a6a521c3e649161461a657f4081b517bc9aab6c68f043eabbc0af"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "042f3e4f9f83d5919847f6910daf21f751324737d9d635f84c91431d802f0bbe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1bedf753a6fe11f87e160215e29e22b108cebe04288557e2eb89f6553467ad19"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c02fca600fd4334005a716bc10cad1606f74cc1b8b83c727224115d58e6d9f2d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fa390b8b7c55ee9507a999ff16eedcddf250b07b4b9e61d2f14af30f4b396770"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "047b2b68b0365f30784d0b6d32643ad339e462311ba365608fc93feac5e517ce"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9591967def99049bf8b48aac3aad6fd3924c228bdc64d5295ae70b8c46684506"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "57818bc298f3b46cc4d29d69b08fa3187b3251851ef18f1d5b6ebe9347da025b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "473fe365b1c6e7ecfa5ccd343490a8a576e8c39270fab4fee8e2f5645758fdeb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c2b6630457831df1bcf7b2459326651f54964469a3f6dbd76cf2ee0a429b17c3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3c5b3864307fe100c2a1cd3919217e0bfc2f67cc9c83fa65d45e2de4c7141b88"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d51295eff992e5b7311a8072bf0398d98662047d9fe2055f40c7b3dbafd69df3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bd2b43c4048f41bae2793a220e2d5b477086370dfa020e4cba5f99c3f78c87fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3feae41a881c1bdaf04375c8fd19c4106ba4181148971c410b93e75dae986019"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e12d1c007e88f33acc8b3a94970dbbd0b64edabb4b01cf0b11b7dd3176b28cbb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9db67d01cf500d4d581374c6275ab046fbf8b3c82c4cce002a0e11528f9c7203"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "dcbe8cbe39255aade8feaaf1b468618c9ca99b87d0a6265ec93047470538f750"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6e0c13cae9ff5cae2899e1a51886c048c8465750870d642e296e74253b670c22"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "44055e7eeca8b24aea61bf5886a49956d1583dec21f5ad97b69be32849ea511a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "09572d6ba5b6493873eb886c5311bd490078848c839b89d245a96cf11362bdce"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4bf30057887c0f425d1266aded51a26d3132df713cd32f2be20f595ae4f3842f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "df742fff80d81ff44b62edc7cf9c3a2b4a1d7512e535745daa9aa0df94622911"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0d270ea115e088e84f4df86a93f1692fc2869070ed623e286fb471e8dbc9dc27"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6f95b4cf4b4bf8ce42e2c34dde296edc087cff7328ab618f125b985723159a87"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6265ff9ca93ae2343c0c65bb601d3e49219285a8c7e1e705b484f0a73606ab13"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "053ca4c8d4f50bef56a5c1602162b23a5ae6ec31cda51a59980e383ec904d157"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "46f134d33337e413098360d4c2793b5203708025e9f44b0f34aea56ebe6bb44a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b92be448e7391a2d2285b092c1e53e48079302372f5288cf787b9adb89443f55"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "ec2b41aebfd7f555f4917f90c5566fc48f3370c0900371da4d11b892b27378e0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "569d501ceeb7565c5af686d334ddd63174cdbef34abeca6cc86d8b7e01405aaf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "f140ab59cac28584b93b6ff4bd0f3be0e7e1d6a7859a82920880dcadea3fd802"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "ad59eeeae33cf67719492aa3638b9057f6c6f451d82d7753f54dfab2a0c7aa03"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "3c2df8a8cda87a84abecb4fddc1c329d68d2a261c2aa1bffeaac2e4c06a587a7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "3644f1fab1b4e12cbfca7bba70bb965af6e6af4cd6fcddbaddd9499091c0b42a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1d7758857b949d81463a861f833831431dd3b06dcacd937f1176c2a3921c33df"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b05a718538fad79b0ed5574bd878d05f39e2f846e5f49646f083757fc32e4120"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "b30ac83f2b7aba8f024ae45fb3330434ba409337d90cdf70568b39d2f45a4e65"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "721baf0250507dacca38945e55c4d9011c0ee6b17c0521d36ad9f5635bc6f91b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "c6466351aa69984e8c7b1d6f227ab2a7f0160dc589c52d42f050cc485ac916df"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "bf74eefd88d4462207807646dbdb5f28df84342dfcaa47a31dbf10aeaf240ccc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "e8b8c8f48163ed47e5e1933fd972e82ff2b10edf6e4cf87c4c193287fcef8222"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "911ec8dd8f63af2cc45a44c902e7455873b711e93daf3088850558e28c984316"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7da66a382d997791c9cf3cad02d45b7a69c6d3a2c88c2b07a6ef4a5e0f92a8ae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7e9f8a8548b805ef4fa8f658a8ccb21ea040d0d028d64bbb9003d63c02f07f0d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4e877087b62ab462f17ba205277f9d93079f7429f40b357c45ffb59ae9074426"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ba14c46982884a4f8b164fd5b0e21fe654ba780eb53598fc85eeff29d1c0bd69"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fe116231360f0220986492187a7d3b670010025eb589079c310fdfa2d571ae9f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8e14dff8789d441af2a2523048169a64e12a0a287e46f640450baf86a4fe0081"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "68a0cec0a2acff4d49f6da3794d328a683bb1ae0698a1e14ad9c00bce71a4172"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2f58f75d273c5e879b1b10dd1579123cdd31af7400dcbba7de6832390841e436"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f941f4fb3268466e156be78aa9f52213da051fc949e0ce5927d6b4030154e8c4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2ee947e1f6b4b4812723497c482ebfa59d6f0ef6203653781e4bb5378a4d47eb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "82a654a02795b36bcbe633a3f31f40edb7435b462cc5521f89f93c3c836633dc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "484fcfd48bbb88a72c4f6612042e02cdc391805dfff0e18ae2e3c11ba24594ff"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ed89e0391582267dbd73186855d4faed5f7df0b53a54550d224235e506861a04"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "221311078c2c8e65b62b10112f833d6bf13d25de330a5d4dc45d31256ffa5c71"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0f5e7e679c471dfc9cee035c1f708b69956cb8b484bc439b13173701f9e65418"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5b0caa9b0d8326e39e3f9837640be031ee885b9682e62f2f063da1dc91def972"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "249e56c68f6c35f938c6a118ce3c08af72062930b1cef8a6b490433fcd37f01a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "1e5a0b1716d57fd0486cca4b809c051289fa46c6fdfd0d4b39734337d6de89d1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "b1c9e92379ddafc0e1a440f355bc090100ea2af2ddb1d7200940b3480da9380b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "37e808d32bc15764e13338e8fd9feb369e251a78104cdca38d30518c86448d00"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "06f0bbe020d9801a75d3f0b5bd3adea6322042518b9bc1ebb690b467f9570331"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "bdea21064f690f4ecc9f31806aab0b6e803abaa92881dc8b7d91c528a0857460"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "06296b7b969217b46566e4bd6327263b6dbc501e34bac55d812b1a32ce9cb56c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "d6de06f4850eb247aed3d33b1e085e684178f52aef0c41712ac23b83a88aaebd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "d23d6716608ba7eed74c313502329bd9b528f31b2c903c8ce1acfe60cd747169"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "3c7faf057665441923787d9edb255d8f9f20836fa70b123d6b461e93e721f654"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "8531762b373fb31f70333b47ef29d54490fd135574a028e07c6828c0e377aa3c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "d2469c3b437d76f0bb29e087d1fb3c329d41d2894c80d10e15d2721c3f63f268"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "9a286e3609a91c2673b9bbfc14773ea3c1892f7b343e0c8c238a5362b7d9c156"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 117376, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d116bb622dde14d3a52ae1865d41ff5613814baac55f74d39c116f203e430021"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 117008, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d4ce24340acb09460612a0e6a6d72eff7c2b8ab96c8113617f62a6feb553d844"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 117392, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2b55541c34b4795395066fbc4fbd86b2ec8998babd9c821e7295e148d41bdfba"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 117024, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0f40da6e64c116ba24b85b5b3c77979dc6d6105773fc6a8ea9ea6c74613978b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 117376, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "48c7290174af20335c39433fee4366b98fc995412ec0ac54837da86214fe641a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 117008, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fb4ab968b080a24d8c1d175d477156d9b0452eaf9217853ce2706d8b567c2169"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 117392, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6b5a373091818acbb896c3e8699fb9961bf69a5d2def9a33236ccc13b953b75c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 117024, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "02f3d0bbb1bfe6f837208fdc3980ae98327b81db95e980199739a84b7974789c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "39b42b27247fbe76d73d07d83c944a7b515ae015137114485252b91c220fa427"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "94e957dab083bd8c9d35ca7007c1a35ebc7bdca481cbcc7d6d5d8a455980a76d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c1e375dc04d1212c2606c37f5f27d530c07f373a5ac2fc43f9630f7d9581a82d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b5f28610b43a5e901943a22b7879d6d9535ca452c06db4f22be9c6df335a6523"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "edfcc18cc0de0bb2b6a120755bc72907223d7cedfe9a24d069c02253ddb032eb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "73ba62d34efe48656d9567201c0da4595021f28df62065a1c039f633507f73e8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "1a0238813f3f31a2fee28e9aaf4229df915d456c690a5cfd17585f38dd96bf89"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "34b64ef9a926d85e42773d9263dc638d83b4f233e4c90c194f96a93b9c90c8ad"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "85790f45865e17dc2f44bba2b4d413b2ba079300b5bcc7578b132418e47e0aec"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "39437bf28015b3c271d1d33b17b846e7abb57a86172a273139f569f1ad3bd4fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "106eaa288b5cc5e1c2d5cd6c52e8fe0b96ca3f5ad2c22c5b00ab63b62ed7475f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b90f7be669b33c69ce415934a8b5a2c2b9f7e7a97d5c5b7d6b0b6e996ab74b20"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b366a6c9ef837e3b5e3bcd24d06aafa2af7e2e63022abb43c3a2a98789157189"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "153584fec6e0eafd0097452cfd8fda8e48d3cd2446d54788c8e69f651bde8dbe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "cc6e6de66e6c7ca111bc75289b15d2d0e41e42a29965a59a90c8aa3541b676fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "f442b126dd7608e5696197de2041ef8fcf0f4caa30b0bd57e5ba36cc44c0856c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "bb35f80e089354aa45be1fcf30405792848995e19063e338a9ca30671b8571ed"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "016996dc815b4b3c4d98c26cf659809818de9585293f1510927b1a1c1fd194f2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "1bec036b593e696ce03d6bbfeb53712a39d45c96b32d61cbc44a30c523312ee3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a4611ba177ee12c616624163fe5bf01ab648331986b8dc041a84cbf8f32f591d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2aadab0f747098b860dafb7ce4b94308da2f18a164f39664bd6f80b564d0873d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9e44027ed22a827cd345b3d9ab92a99a37e21515ec3c3a7c16fad384e6c15325"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "01c02ef2094dedc85bacd1924c50e2816feb00ed9e83fda69da11712feabbf30"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "56b0ee2041c017f9d7c35d6e7830ad00bdbe2898d443e5b0cf0119cd87f11307"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9c1ca9299c2c897df7e9c89935f1e0548079ea361e380c857650e4aefc8922cd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "420e2ea5472bbdd98c8ccb6b1c7c1f55c850b45de0a27ddcbaf6484f03fd9f9e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1806f4d112060e8a010e1047aa1f331eae50b5096178f5ebb7a7eb6dc3b44c20"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f14569153e0b52aa274f8bcb521449e52334addb3bcbf777e3ad49ba012e8eb2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c7ac7fd89545fc85a74e18c8369f10b68d9a33154282e08f564a20fd0af0b91e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "46ba1ce7828787e4d7ffd92d686a13ac3bb34ae17b916bfb2542cfaaf7c93ffb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8d1f8c393b9abd07b787eecbc7be5479ad0d1860f09dbf98e3df2cc5b4687f21"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2829acb6616f3b8a0d8955a1e63fa2a584b55bef4d551dd7ca0e15baacda4d26"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7eb50e9c8e2c34edac84c0f9b2d1130389a440ed12cb3837a2cd7b1f3baa5775"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e615e56e7dbf1dfff84c3a56262182b8d8ac46a2774273aab4edd6eb2456b8b1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fffe46f0bc6916f1152f4c876ec93d0e9f71672e6fb73f8e14f6dfa52582f659"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "10f28fb0ad82967e3173dd8993d94168251731f2549bf85d293fa14d3797e73c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1572e551ff91b4d8f62166bb071daff60a5d13a95aa0975f9a17d2b525e3f1fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "ce0ba8bb9709688ee808bc91a682976efbd3588944abfeac892be37a1157fa5f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "5f0eca43db7395fbed6fed2e7f7bf0f61e1eb634676d8025a7d70edb16fa3c56"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "d7f7fa372db0aa5fe63623592a371b1bade5ff1c0db59d1d2de758a037c6cfcf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "45611b8e64421f017ada4c3d0537cdc1f8bbf622a258679b973789462aa5c8b4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "4262b2c9b5c942cf1e4d1fc4010e324961e45070e166aca34c8617adb47e70c8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "ef39283b66c6c86446b49d77710dd8cbf3b299c7d586f97e9ae76f67de729140"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "99b10e08cb66610f39f81b4efce7fcfc9e9e25d72bc34a2fdf027925201896a0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d01d442e6f34252cc95c36a9bab94cbfebd902ef0195f723f93bf4e23f6e8aad"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "fd86798ba8ef3c7383be422cd39d57ddcdbeedc41794071f6a733271bb7490a3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "6cc6a49f331f3d8364d53d1e64119f2c99c533101f5f97a3adadc746cf8d7ec5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "d816907c58a12ccda981036b3e205378d09e1843812e493084b5a72959e1f777"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "f20dac94007f2883607d877dedc880230627dbb0de1bab8a81b72a80c26d5ebe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "c9486e1dfda1c4aa6d1603cac4578fd23d68bcac1d837e785c8fa6df5f232ea9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "832938aba7dd02f6c01beceab7852716eba32c0a70e86d968eeaef4edc51a4d3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6af8a5e5cbd3757a51f960daec4d8be68c1d94d38317ec2f340d0faac28208af"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "90aa501b26299db8481a8702c95533a036416ffde29f2facc0114aabdb2d5c68"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "aa668eb4e8b92eb02d4b5d9faf4dd6b273ee7c6364390d93fb15c560e5312651"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cb697664b554022fe4fc138a7bdcfe3c11c641774b4790dd7fbe65e5a3c3315e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "39bc3764517d7a7bae4b14334a93a81885d0abf342934c5c67da6d64bb6cdade"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d5fe584690b3f14426ba25801277e86aae37584756492d1f087f17e0c4b4931c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "abc6f67c22d88048a0fbb53f056b1795b72af0a6b1789716ca22bda8edcf5425"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c133f8e9ca69f6c00f56c7f3f5c1baabd5978418c2d6202538e0239668d76cc8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ff6c177a8d5ba89aadcc181d147c2fd741d44e5b04e6d93ff167a01c1f5d4d6d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "27bea40395c2de92bbb7bdc8aaa9f069e4a0da964c581018b15ffb3a01c5f019"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "68e259f52b837be338674cd59e6c12657fda60a0470cd34d2a64f23e8724c4d5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "36ba40b984b788c358259ab23370d2612ef2b763b3a73c62318ed310d34a2898"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "bef79f1d851f4e2984c42442bd1f3c12c41f10d9baf43de61f966d1ae0e4588a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "13a22f434c238e6b24bfcb6384d8a33062e7ffa19caa24cc29f34ee16492c4c5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2347b978d117ed6d8186b3c09f1c4110385c7342dce7cd747c2721bed1b018f8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c89d82b84db8a650720d1ff327186bc6922d66aa1dbe94d1f5c1e9d35679f3fc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e2584ecb04c5ffe07ef975ed761e22db6fa94808a57831d166baa1ba2773ccb1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "81745004dcdc6810439b3e13e92bc0e6368138450e0818bb04edb6ab67b40c37"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7394afe1c1fe83ac49c0f589d27a64b9bf4e7c7303335aba485d3574648125fc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "285969688c4dcbc957b763f39845de0d36be1106f60d2c0f36a3ceb331094ac4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "dac058ce58d7e8ca31953433c2950ba1d805b188dce797c3e50c6f1479eed060"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "aa5fcf40724454916abbb5b99dac5b7dce51c10572addcc30510ea569a0a19ea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d831bcb1c24b6ef47227b98369261d44677b9e9cc34e1fde388229ce198dcbb9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0dd83fcb1e510c97b8229e1ecda26aec7decf2dd53d6406a4d2beb55597f51e7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "062b09676bc23ea3f2924634be57d0e26b9c9cecf219d6ff1da5d9b48c9b530c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ad1cb0666e918270eafd42903e3ef1b07c77bb3f46a3b1a7f0189fc166341208"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6618c7eab68ce330c6759ba686c0c2a9a4c24d6348bc1f47cdcb5031b9a810df"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4bf4710f3a21147c1c0ee29f612351dc6f282b2a1c22aaa7d3ca2fa8af41bb71"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "acc5cc0846f0192123223bb3f86a235290071e01d53ee40df71bd33f8d1c96d5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0cca81e04c21b82274469893171b47f44135e973e5aa59b0124bb9f3aceacc2c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3e092bd8f975995f0fe9b3a3f24dce128ee6c15236c34652ea972f25538df76f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "11ffb41dbbb741adfb1c85ef553064295dd62a541c1218e39dcc32a69082f38e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c5dc0a91991d793108ec198690cb48821abbaf18fd62b516cdd5f2dbc9ac0822"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "81022fe7e79574f39909a50f9545519fed4b57874ac0c5de25edcae8fc0ce548"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8648c5063461e861a8df0108801963700e5d0a67743322db1c2a752f636b2cdc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "aa00f058b523591bc089b0f892059335e78a66b8ea1d6de46ea988b1c2256c51"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b030c33d83f7a645745c67dc920b500f05204a86d0aaf81d5fd9c8c1d4b3b406"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "22897d6815cde0821f43b305fce535171116aaaa2b83b6617443e552ccda0e72"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b93cd7247220a2845608c3b8c810cf5bed0a765a9f92d7183430eb9ee88e127e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "139b8ea5fd7ef8fa188d12e9e69f9616116ab1f91f3303fc6e322df9c75ee03b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6d9af334934ef9115df81e6292c8ec5c7594b6f74e9383003781a6edfe096c64"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a2d3d7e83de607d44c82bb16f0ca8fa153bb34c1daa857ca3046b5f271b0d7aa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0994dc81fdf414f4a06cf8e4ab1387b4893a9b06a67962e97330215e98c4c710"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7d2fb098752c10f6de75fc660a57f9c70f961b7db2fff6b8819066757bed9d7e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4fb6e97ab1bc08256b5a35cca941303bef4bad27e7ef428169fbdac524d82019"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "aac0a1a63d0f6ec64c3513a3029e0a2b1c866ebe7ce7e2ac15e4630d92e7b350"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "6b1efd9d10e0a8b21e0c68526c0866f8fc03e2c2346b5cd170639d24b38d21d6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "c4d5c198f4fff523cf5bd657b4ea335d1fe5c4e7c91e9fada5fe1e74343bbc8a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "770b964aa23a0f5dc054678209c4af79241ee1b2f540a7158df06c045945fc27"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "dce0257282751bfc92424563fe776f98af0dc26e071a994078d24c3e502334c2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "67354e588cd1ade11664ae5bbb741d0a8d633d59bc2f5c184bd7778ed3257109"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "2bd63039f12d0c9751fc245cdb4926e9dbf38ccb0d044193bf23d1e542fcd3e8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "47fa3fea3b014b5f161ea504cc3930ddbec1010cafb2ac10eda36138c0915614"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4e58c50377151f0f281274c8c8c19f618289b244cdc67a997d375371c1f1a93e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "4d8cd3d9ed91331b6e013f7c125d368e84efce20105c94040460b4457f841321"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "ec99d72b041b376a3daf5dfcabf35bd01b17cb3ac014f45db80f98453a51be0d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "49c17891c0e8d908d4dd493c5c8eeffd7ff23daa56039b6b305d7080508488fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "3d9e693f22536ee27d8c46d89bc38ccad0f8976cc4cb64ed0b15b08642458b40"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "a1d3d52387ee42d25f4d15efc468962ff65c039b62317a58939b59bdd0dffb68"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "84ddf414b4bd5cf01a5edf10bb5338a452310dfcfbd935f268fbeb0ba812cd77"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "edc8250998f130ec5862d1138ff496043b919a8d1becfc6c058392ff2d8af7d4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e207c55cf26acd58445a6cfa058da4367260584c28242f3d02b40a684035841c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c553d8a38ac1277e31cba6327d945b97d6843c2a7857804f9f2ce0707bae36a7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7178a8af7f55a8ee04fea26f81c4b5074e404ad11d325a3bd269fcca668b6796"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2038dcd59d6de496230daa9b38e698366f20907f52f112f0619550a059f3d8f4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9c3b97394c32622df22ff42f4da9ea163a6eb3d75ed95fb64bd5584621d7cc6c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "98e1de1e077549084d24ad29c053159a0d66d93fe7871d1a7f94913879650325"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c5fa9809848656ed775da9786b148e163c98b708ea01eea968a387d8d83948fc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d815b4f526b238d185a816d22571815b901f6424ea92359d6574faa87b540e22"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e45c9d3b226f2ec424e67891ef8b2d3858e09e0afcf8f13c6148155e03830b5f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7babbda701dac411920943d4da738fd8d25f6f5bae411e19c6427bee82e3e5fc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "032054c854017eabe9c86425b1d922462afa831abfb279982e2075f11b5e6265"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c2a58071de9cbfff0df466a2d806caba4c73a726d83289e2c99b4a4bd7a8e929"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "11b6a07fbfd13423a7e281cda9b6eda95cd32c8d8a8eb6963048ee4eaa2f23b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ce871f413c2fcc1dd979b1988dd79d49c645d013e08e40c2059a875a4c11ebbd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1c228dcb4086945524ae3065faadbfddc34972dcba78a677bc24efc45fad7e45"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "77cfee7adcc55e20a1d68d7df392d0b703ba213854b411d0cc416b461de538fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "34102a32fa1112aac4f7be9815067afb4964c9077208633d7ed651164f80dedd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "257d9b163ce31687c6c6458e02d624a0ccd9cd9e4eaa1699d5fc1d41da4df0a4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "e98ad5dcf4c96f6d80546b6e6e29052fdc67383eb40b8b5c86dbaad212729c43"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "2aba694d89e0d7c747f326ce375b9aeadfdb440863f238d9df69a465965bf51a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "ad359c910bef75249a886f994b5e40871f84b6fc2624e1f55d706e389dcd3735"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "ec4827dfdb61f55488e2f3be25ff23fc353d076e7f5d3abfdd1a20edeb6feaf7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "0e91ce94d46d4dd45c12a6dabe882dee90b3a75722e8c9f953c17b643bbb725a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "a4d0f98604e5d1db2039138f4a3e81bcddb2704a589aa8afda599131da3b2f46"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "084d17f00c93ca3ea1e2ddfa450330444519f909295290242b9cb982242743f5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "5719cde0d566943d609e4629ce46b9c122937a29f47b97edbd38bc0243b0272a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "e6f901dd807e63f9dfc4f695a9fd1d4492e90dacfe5e51b2052d7d985ba23b99"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "88d781db8562f37c7dd63b376f27fa726903a5c523d028b17e2794cca352a218"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ff007144a1b213a67acf0a3ce022f61df77398eee089052eb7a43da304eebd64"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6ef0fe1c369afbadce1ca6203ec07f997f05f249b6e015e237a6587f19252885"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "68c28f59ebd60a607a4b99087e1e3a765d3da16166b1073ae7df5caa7d384e48"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8d1bf6f8729c6bc4ce4b8274a51895424bb330df4bc3a090f28cc09083c7d25d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "514d2a02d1116ba7d4b3ebd34a5f61530ddc18779399e33de81ed16c982f25c5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "f6b9989e4574cfcfe47faeedca54041b3f12722c19d3e4a770c54f7566d35d7c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "59fe78023547bc4c403abc87ce96ada4860b244d88a5102cc4e8a53530de766a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "6480eebf2798408296b34a8f1d0c8fc10bc3bb1ea42e520b83cc7e6a7e752d1c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "c2e7e935bf3aa2f8f4cf2e8db7814b3b0568ade870c694dd0abe32b233a6756f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "61e811a0510f2ec357ba0931b67e320646ba0db172e61bd98ccf0e0c726b44f7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "6490d3638abe6ad873245a6ce17a72fb0bbe1df3b42fe7a2cfa6ad707156a86c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5882a08537b3b5de3e4544834d429d75538f74744a9c0c972b0e7bcfec60d503"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "726e73a3ba3091d452dd599a943028c2b07fcc17a333410dd118c526dd428a40"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "e053b4f46676757814cc79ca60803c32a6cb9e77e59ce37e72ccaa092bbdabed"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "21043e43f0d3ffadf91907ff07261967dde5db0e8bc7b8393fa2cd2ad7b56a01"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "7a2b1b95e7d4462109537fee1f3a9ee975e37eecc6c2c2c15ec134e318c53e67"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "f04e3514744caaa507a16e8bf05cc71649f900cbdc28b5109c74a585f07bab64"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "f4635230aafe6a25e107b44f664e9c4ee39bee82d85c3be5b01e055b8bd179ee"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "4a3d1cce56eeb3283ed3efe8f6f110f4e62becc53247a91ed4024dedf6b90ebb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0a3911eb8362e1002a920a987aa7bd37ebcd66900342c7765c90d23f1f43274d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8f13822a1224dc7809bc2d60ddc08168211f6b984dbc42f90fe35da9418633f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "83047c1e4b38cc3f95cdeb8841c2872d2840803ea12d1411ccce0735b2690b96"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a1835524366dd0b664f599222144a5b5dca65ad4e7de05269fac43aca33a6b1c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ab0297e5fac94bf095e7dfa48363707c9317f0197755b210a164414647d21657"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "96c32ca0532ca307e5501fe0d36ac7e97d49120302650e66ffa7044597bee727"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c869199653fa5c60dd31c212c9ddf6f9b1c2a954c9b4838a1c3c6f23807b44d4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a2ced6d9bcc3435e31835fed69767a08ecbc28f83e0e028af70ee5d9b473b1a2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9ebef247da3d3631889f2ec2e9b9e89b92f7e89e78f4ce8df2386fa6f94fed46"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "28e13029a6a0e8280f971a4c22730de76dd2d4f9d21a1c856c0570c35d3ad775"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ebd2715b34fa286bab4e31c5ef0a0300c7fe625f296e92001f0796053761b3f2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d3b99b60af4ffd7f0728909ca66fe13d7fef7bd5ce766a99f3f80f04b715b611"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6812b4a460aac5b04c80f6bbe23a10fd0bc82028198be1f89137ee7a531a7175"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "36ff778c1e11f06c423c67d3668c6f37871293d1dc8f26b168b0704d4ca64683"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0350f89f99afe279308356667f46d6b06425e1be516cdf4aa81993da76dda122"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "48b9c2dab6bb086b22709ce91fc9a11f60d9bc54679f6a03da22f486e191319a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e08ba2742a9cc8eba09d2dfcd648231feb60ca5d8bd5a50a5659b88629110a39"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "aa568bfe5840dbc6f66bfaf56bcadd928f69e2871a0cf6508070e9690f7a535c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "806b5e811cce65fb3308efc15f90ebc812c05b774e011faef5a1584609e78389"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "d16520b471d2c0f30fded5f80d32789bbdb9be176b07c9461445a884d53fe2cb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "97490e09b40005aa57c057020cb96714334bbd0e0e561f504b217fb01d589dd0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "2eadb0a2cbec3118bd64921189230ac1149a2ebd519d3d42599654688791c69b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "af9cf9791c00bef4f902b12b23bcb2bb2c0546499577dbb0813ab73b90538001"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "4106e1ced00aabd85a6b1c5ed94408033d41a7a6e1e1b11987982308d1fc5fa8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "eebb8d8b789d53ef63772a82036311623029c08e57e635d67e99edd9116d110d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3e07516daf015ff514a5f0a808687f4a7a01c5f070c941a9591b6b970dbfc5ff"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "a3ce6843fe46fb09767c7e860e00cf337007cd47066901e568d1cc9e8c5cc029"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "d4ebbd2d5e6807961762fbd2e16a729e7edd57d8c89471656dca2e68fdfc3955"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "40ce492e9eebae1398c7c268b17ff9954c2030db420e223dec93f6d1e09ab4ff"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "62db6588d21c34e6203bd3b1bf4efbfc6b4f4ab7bf4fc542375ef6c2b1d42a32"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "6d6038d59227a4cb46f0c753b10848253f52d62aa96c85760d574a2c5a9cd0f6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "cdcff7bf58fb1c2039a346de092252487ba32b1204bd8c3272f8c32e8fd8b5db"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "91d33a878087648bd282821e9ec9fb6c0053b1415bd575f7508256568463e5f0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c878e93fe6f361cf5e22f8a4f6098bc244a7317f74519395f3d8514e34573023"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "829116ac94783c2875ee54099bb61e1ac465b16e1305f1ccbef86ce37db06e9d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "40ab008f32dd1d80393b07374007c08c4d1d3adde584a40da989bacc0aced83f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7ebef3d3e22b12fd8d2ee0d5f0fb102a68bc17dad41f667c83738b69086801e0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "cb83c3172b0d541470116ed859a5ff0137c06de38467c7c700dd52c52a994954"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0218a9dd64f588d6c73b5b3f86a0214bcc970e46f3e642ad132871f0df394362"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b5b7a2d24c1f02d8811e7c2613534aa8ad3dc3200022133747e85d55dda10459"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fd740dcdab7248fcb24482b20ef7fa24925186b54b399ba8ebce4c888799a69a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "416086a7b01461385aead1905cb2f68293b192a1fd1d0fad0abc23f10ed14cb4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3a72402ea0814e4968bd85c7d86acfcf17b5ff86a534a677b5e1772eb43312b3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0b57b2d6a3c5e10b5df47e6bfe26c0ab92bca13238fb564c1cec31ebd619a9be"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9e0aba1f816663edcfb149c96668e5be8b7bfe5c7c008574187402b031ec5d54"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fc52e31c5b6fb83e55f05c6281d7626845df0c984c70d23bfd10332177aee5fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d429e99d7eb4ed77b2d4ed586eb9d5d702a049b1d1c2160b85295d18ecf959eb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f13a944d752619ad2012d4a922bec7abf40d77081d22ef552bffb73f6553adc0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0f4c6bc29497417afc979b2958b6353b677815777187accb4b3437e2c7090f12"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "80bee3f8fed607c1b3f15bab98f4e6754ac8fc4b02652c7969a02300dab7f82a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1d9c77ff391fba81658ed5877ef1f0209f25c500649290b092c8e37f20c7b406"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "68c5cc0ac2aee3a48a0f5dbb10e87dbe112008289da238adaf67890dc408a055"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9577b32f2ef1c7b2d93d8067c3164afc6cc42ff273ad122d751c4f51760b7338"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6e2006041cfba705b6a0080b6d32774bcff7bcbd2b2261708759e59ee010a279"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c2e670ab6163466912d8eb932053bf66a26950bc3635336775609eeb49de15e4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b62dbad9485eedb352d22514eaa666fa41dd14e096ed74e5f358c33ecea63117"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ef09edc1967c1fc28578b90894287dd9e6cfd0861da3f810aff39beba77beab4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "dd49f92d08a0a79de791f2c36c5412f57ca67660b05bc14975ea7c406bd42028"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "63358b0d1e723b7efdf45abf725c51694271727dff8d2881bed9f1a89e31d9d8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "de1b416768eedbd52f0cb90d089a7aaf73e2bd92d002327570e5ed822ff9b2fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8cdbd3b34f41134f12920c1d5fe779c6419d9af7ca967f0ab4904efd6a7064f5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e3e7ac09f81f07605bf0b096ce22f261abce187fb572c59c80a9b41d39a346f2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6c3ad7468d245254d6ee7745f3a89d334f3132debda4126453caeea591990d0a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5562d6f4e6dfacb987d022e7b4511b7a6ee60ef7493ed3c305367441804e8d8e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "15f93495a402d7cd4f6ab2396756b0e43a9a289b5152618295c8417ae7681da4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "96c2150d46206f4077622a68d7bc2626c775520b85da6b722189c92afc20e348"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "41f213c6b9b483befa7c55b934987d5b21fb8b0e64d77056e382038c81878d16"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c32a14eceac8729bebfcf9e877869929603916c873300d7f9f5fdcd8a4a92001"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "37d2cdb5ec2b8664a294f37bad2794db8d6a240bfb6f7b6bc8234c48525404da"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2ce1ee99fc37deedb60e51021a7983d998464166a8f73a23200ed246ed0f064b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4914563bc47e004070303e1c966ee9b4a8b96d4965bd64f84d97d2a0ff5c18c6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "049fc08bb9f6efcc0733a7e6da326da67493ad1f8f5810ca63b8a818389ea5ab"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "70eb5e883a26dee6f78a2b0b996cafc3c938f953ce0f83f65d2804adca69c304"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ffca3f48d9cb667a2883d9a900424b2d2fbcf3af2a6164874416a3ca00d74fb7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4153c7e5bd917c14fa385838aec389911c6052d1e09b06b6521b3c814f2060ee"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1c300ebb82873782a9b59df79babff60e3fd7dea2ef88dbc5d5dd4f82287550a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "746408c665fbe4c855c084cafabf1c6a2ac54d46f29fe82fd72b4f88752532a0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b76255aa21326198ac91ec8a257944572e117d73b3f895bd451a0a58550eaf45"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "3e71a60be984fd07cc12a32b7a8bb9613ee006a25dd74033fb927c1a2a1e340b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "805f8e6d77f18824c02e57324378b73e7983d3ee4f9ec2e91bf647131fd54ab1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "e6c8b0c296058bfca45b39b8b73493cc08c5cfbac4ec5637f8bf46b47ce3df03"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "d6006fca1ec560bf6cbf703bc9067886f1efe1b419bca34384acc46754c7f937"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "d7a709569c5d40185db83e90676eb6fb09dde186717c03d4ff7d680c3d48f62f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "fa6c39b68b0c599517842656731d8e8660a4cbe5f25bccc502f399d983ff86be"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2d0459c6d1c89934fd0657d491a91ea99b045151a6354d0504d102661d1160fc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4370807abfad292c657ed2ee85143d081ade83ed40950f1e33ba8d98d8dedb57"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "000a48e94d5862535e4a906f649caf63121037558ca623a7c38117408d136e9e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "4a287fc2f2074fdac60fac8aceff4aa9f5c6f029321c471409fde8ddb3738e97"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "ccc85306baf3e7a4209ff9a1b490f5fb6830073a7bb4636b055b8b744dc23b23"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "403022c2ff7a6494289072cb26caa3bf3e00874536a3614db99d6fd70be63090"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "132d2b3fc2cafac0ad52111303f4946e5d546561ac1fe03d3a2018cdbbff0a79"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "15ec3bfe5b2eaa427380bcad987a94869eb05cf0ac8b9648e084eea025508a76"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "adae582ddf5760162cf8976c044a77bcf6e62cfeda0cc99516afbed845dd713e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7e4f206f8a8ebe3a79b8883480b0fa3da3be782ca0d2b3e6f250cc982534d969"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "98fd078cc19a7a87246b2a03e1cea85537311d9a5d448d2691a6ee3dcae33484"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d0c0892d7f35aefa8606467fda83f8e11abe069189ad8dbb7c07d18fbaaeb780"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "78d97ac06b85ece3e6be8558a402eb4f87d2a0b27b0f967f708ce68baa16fa26"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "299986c446bd73530d1182a9548cd791ad9fd87b59ca82ebde0b2ba045873f08"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "18b2ddaf803616f6a96269bcdd4e205a157ad7c91871cb757f39c83727921827"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b07129a5e2e62bf97f8e027cc0077bc69a3597b3b9b6df6a9bfc9dc6c4cbfce6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f048180ea11efbcefe66c484de8fd34588c97c9ec5ec35a15bd4a9137ca49ebd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "34c9293c39bd852f427b4a6f14d17ebfa52960c1b9943327adfcbbf51fa1f02f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fd6ce632c8a8cee7b6b5ebef80e34df3edeecceed89f50b73ed3d0fee3a95b0f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0a937dd4f9c20f5ab81dda8eafd5f6b61ec9eaee564a21a7e59a9497b368ebb5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "794a0e9b5334701cc30b68638959c596872bf0988354b94e3b060e0f761fc5f8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "626999013f3d3a26bee203174e61673e6dab81e652374ccbc825b191c0554638"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "223c072136812db7b68612a55269b278c3aa59394e15e36dce8d3451012dd555"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b812bacc6024bd9628c3721f477b2c5d1abed90fe44780579a33080bf02c35ed"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6ad04731fd08c694b8857b74a8b611768a221e0afe0384c6cecbce9bfc3e67ad"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "17b0172b9ff2793bb25d007a2c9f31374ae7391bfe5073a052e99846b1d4e418"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "ec02b74c22ec897ed3edc5f2bc7d10a439055eef3ab4defdc17011ec112c81ab"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "f442397154e2aebb7fc291fd7a5d0b28b0ac3d099c21844ce66130d1293490c9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "a27bb35ba10d0e8c681c39d34986ea4b4727a4cc286bfdffa7bf97f6d39ec6aa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "7809d057f63911ce1c6bcbd0e92b6ae0f0df8544ba8e5d9cb3f91a20aa216b7a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "88ebe14c3d3fea51a0fa54473e48bd1872cce05ea4f3cbe3952e5d98e49c207c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "9b679d9bd43b6071b688532f59fe7363e7c2a47c081e6ec6febaf8b37ea2986f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "0d576c16fb2979316b1eece01bcd5ee33a76e950b0669a305bf777eeccdaabf8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "98e02c01f731cc9271f55003ea3e5b641676b151ea461e3a3dbfc6fe2028bc8a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "be3d5fb94daef459f876bcf445190f8a92ce01811992b17840516b38e08bd02c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "12888908368a79b809564c5363997ba3135bc04a79be9f60fb96b194d8bd9170"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "32bbf7b74720fc452ad2068e2b0087a15be07b0b1a58b17774e61b03148a5ed0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b81b6e008e154f583319afc010bd23384aa114185fcbd1e316adaa3b5b66adcc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4c26ed41cc0d182e779124a940f2ebaa5cac0ed95ffda2b828b40626ff465eaa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "500eb82a5840ee6620f00db5f5e80d1a4ab4e5386263ff297e7ed273eccf7e89"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e8870b6fbf5b37078ac5cb1cb403379c889fb809fd00f022dc56624b5410d7de"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "af32415b2a3f111323d633221a301ab8959e053f081a620a1f088582bd92c66c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "2a1771935db2243fa2ed8249c80471dfeda2bfe4f6b48455877c9d3b8c12e855"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "d97d92e8f4b87a5e839e0975878cb9ce23004f098477cea58f2d39b669c16253"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "48c76cda5b6ab6199fcad875bcb916df8d0de24022ccec5be8e7158dc7bda24c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "ae0086dac3399b7176f10212cbbd570416562b94037465b7ffe4d6ee58c2cd19"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "11503e3bbaf24812be3204a8a8563ce2e74e42883d4bd729f0a95eeec165b20b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "7e4a4d179533ccb2b9f53bc7a9e6d340f6a65e5e15cef88fde5aed7feb9dd034"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0cd60b621b74348010b64e863a194f8b8dd085b94fc354393b51e48004204582"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "928c3168fc3cf9c2485e2f1c8bfdf44a91c7bcc5c7a79dcccb5a297cbcf53ea5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "40b065e0a9195dc326d3586e8056db4dbac7f9d240323700e32f03161d5d8303"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "8bd2411d7e8daa5c26b0f3db6daf1365b776ed3c14fcca2e3db6739e8f89ae94"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "53b24128322ca68165d29e8d8fd226441bb29e7c845260b5fb6b279ab7863194"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "c4f450639adc650805fd21185c07e9975ce0c831a8184f7acafef05eeb966648"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "fe1b41e5661351184d1aad3a86e6f3e7b75500f14ef207abf7ba52afd114001d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "1817aac90292b6c2aecbb29d8ce1bfbbdeec80a6797feb48e9523ee0cce67597"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9cc87337f3846f5ff5267f45805edbc764c4cc7f1b3a8acaa7274911fd4a5593"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 84608, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c9a820381bf2813b3bc1d5843e1730071e6f6fd89ed41c0cf8c3eedc09dec66a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 84240, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4b7300ab08af377fa7b90278440aa01d32e8f4a05d7cd433c512c242e9f2ec3e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3ef51f0e0a4dbc1da4f794aac30c23899360e583e453824f93d4180bebf9f7cc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "077a9bff06a3283b078beeba8384c90ea7eb2cd8625e54b68b481e26f6b722d6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fed11c103c25933fefb922e433b546f381413dfe50e84482751ca7569ee2c7d8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7aa3f015fd2ba0c3f230fc6746e6e31c6f7552998238ab670b56a6d66b986c60"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0f5e6a9ab4ab0b8f044ad149ef85bc759dd4bb2e4e626b205751061d15dde6f0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "867412344d6dfd414809ed25192593361fe688edd74f77da283c962f4fcf5920"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "aacc4d37e82cfdb2d4da49785b1a514206945f6904da983d6b6ff1bf8b0f8b32"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "75e31aaed0364850a7d005945a0af110d8985fb9eb47b97ac45fc3c68886dfcb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e52baaacf887d14ac47ad987e761f28544f79c286ad1beddb2bbaa29581dbeac"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "54840298790ec55f6eeb8681bcf0cde905f65b3fdb5a2d3402cfd90f470f559a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 85376, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "592ea3f660518008a31c4a6a17e021b9cf70e8f761cedc33db951efc9a34c3f3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 85264, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e002100d24f28c4c979b98ad807de5eadb387a0c857ad7f325dd621d6547f935"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 85392, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "40c2db1505ee5281fce0c0c68358740a3ff66e75b56ea76c31cc9659d8bd7586"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 85280, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0edf4654faf84ab7ff48d9d30ac9bae93dd0100306aa933f6960202b7e6047ad"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 84608, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ef3f90eaf34573d2b2a924e01c4b45637b7273b1fccb8d19fec242845a52967a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 86848, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "ea923e91c3a624b20fa6badf21c13ae1247770a497581ee9a157af5729bef35e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 86736, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "a6dffe0a1dfc952dd23c1082584f7791d6a713ee9fa8da500f593842db06f35b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 89728, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "6b144b9a0b100321c62e828e37a557568b50e18e4e1f678902a9baf5a29ac6d9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 89616, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "8e1299f4097a0cf7a5f84691afd7031ee812904c43d47311c6216fa9096052b4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 87424, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "3eab29e43ae414514e34c1bdd3ea3893015087e276d98d5598aa613c97ac38bc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 87312, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "ae15fc655096d7e7a7794dd96f2f3d734ac5f09e94bb8c0ae5b951d85897ce17"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 84240, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5a06b4833e03f3726bd0b01b3da4b3f6ed2f282d2dbf992785b8caeefeb43462"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 84624, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2045bf680f2f382f9036f34ce0d2a3da0b1ed5f2749993c234190589481b50dc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 86864, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "2e417dfd92335d66ba2aabfdfafe141a1439c574744453f9d533449b32022d23"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 86752, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "b73481c76015be1a72d13b0f0de521c2c73147e79d81d2badba33628f5885898"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 89744, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "58b53cd687e492defb6e35172a9ce00566cd2e09c9dd1071c7a71d520a534b98"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 89632, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "a1233e6ce773c84b5ccca318fbf7fe3abbf3866bb9d1cdc8135218f36cc8fe1e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 87440, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "ff2a39910637f804665e27b15ff92e6430de8ecc3a24762fea7b10cc5b46431b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 87328, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "50973db602fb4e645b4fcbb19794d86d4fab8b3c84689c31151c86e7d6511033"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 84256, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "586199f8b8ab8f049888f609d6db6508377d8c0346d625a4bcdbc2078ae5cef0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5636fc239d1ff844d39be90c41545fcd04d36ef0b716885d18ff5e5c9cf7929d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c56d95556559335b60c0677b6390b80d54b7e0bd9f053df9278f0546b1f6aa7e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d3705b57c6fa51d4f3fb5e63c85153c6bd094ed5ded39449599234c7ab575167"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b4b0919b8b9bb8d38e915eaedc234b1b3ef9750fd161dc86ae8779b91de1da73"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3483f7d16a5b14c8b4b6ec7cbdf71d1390637008806e6fb5cc5c2459139aa8cc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "87a2b4d447937977a7bac91ea1f3208b7ec63252c7c48642e573e01359f12c15"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "951062c8974b8c379903ff7b8735037c0208afe7b64c91401c806eabdca0e0ea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "06fb4f33e4ee57be685158e9288f813305130634b98378ff49dac5303384fab8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0e1a6b02054f379a02858dc751f5e5e9f126257d71081a4c8de60d97fd6d512d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b8fce2372d7c44e8c10ab6029b88c0fe1db586f126a0139d7e315615dc782284"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f7e7d443aa024e366039b0b0e8e5df1b2b8852b733aaa7f118669a0e2609aa4f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e60257195137e7e4dad72fa379409224d2e9bd48fd4d07c830f323efb45fd179"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b886cf416810e06d5ddefea46fd9e041950452f87029e40933d3f99814818282"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5cb7167f726f77dc2c599b26d78646018ae31a407768eb31b5a33eb294b5000f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ee3005bc907bd6953857164f1ecedef7e3523947ee22cc630deda77150cb6fee"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c457639a0b8ad8b296acfd838c2b05161135977d549f8e34f6e493e69b303c7f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "67d76840f655acb578160efa2098956f3b10b308367e957f331480bdaa0dfaf6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2535ae5d575489a07f227c6f3f5c9431fb1e77a983b983c349adfaf72b81a8a6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "343a1257b8ffefb84801276bcaaeab96c301eba320a15f44a07a56e0d08d5bcb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ccc05701ac3d605be4d807c2a33ea6cc224a55d57d247c4254a8b2a408d785ff"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "91671fb134947c51344d4c6f5d16da8f8da1887d1763d966a033f7563fab8158"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9b33aad46aad9406643765243bd36ffb533607d1900edc54018b79c9a8fffb4f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2cfb9e173d0116319c9d9f2f973f302313f5fb12dec9b84a0f86404c907c71e0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c4f43c01aa93b52ca74e777c544b93d5761700133740893523c7f338dcde7984"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d771b3c9c55600ed7f768bea775839c17495c8a45d4beed51b2450fb195c4928"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0da0d254d4cb7dc0e15625f6e27a2e2f618c954ad7f58938debdbe4243e551f4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "74c158f6c80b06c434d77a049ea6c11bc80289ce0f3d9fe3e521edead2aa6564"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3b8766331cef3a140af23f2567b4d278b5da923768471cb7485c608214bc11f6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9328339ea9d78c82e3cc9f85257e730f0eaf393fcd30650a675d81a4e4b0999f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "88ea46a26c4798a9735ffb4e693104262ce50f2708756c6bac89cb9232add793"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2ff9a67087487936a55e3cb0c76c50f874fd0ca3fbde9f5e841f3e53dd0cfe21"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1b9edbb40d0ec160653efc062beb219688ffc56bdc92f99af0b122d12358e7b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8696306dcbb248c011c3fcddb60e87f9b13842db026c4ae6076fd622995d52f3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "790128cb04c498b2ffed3ca327f1570db71816621455a41f66c97930c8214d6d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f989b0e1f32b627c3d7222e474a551797e21111bc3a6c90c9a1ce435c3900692"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "848345f17102c8c47ec13c2eac0e6127d697ad8945db85091ea1c0042207b3f8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "29da4b5f17e2a695eb0637b289438de5f45d27d599a1f7d00eca7c2507a831b3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e6d9193975dc910ecf54ae36b40df1fb49d48acbb3d5abd293feaaef2f57b8cf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f6b243eb83c567f37a737d7d771b843ee8a8984cbd058d103202be0e71c6d95d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4c8c92b4c325ffe4f50a2aaee1801d6079f4058a97655590787a5c84cd4831ef"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d955944d6164aa1962acafdd65097af6aa614574de3c760699687042d7ad82f7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f48bf95941187c8f10da0d81ffecd5c2904ac68679f346ada964f8bc0698ff0c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "bc647107072be7051225b82d3c93c4e3bb65be4f5aee484622300c1fad0b30cb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "bb13a114bca560aa02ffee6c53a7968cedb8df0304cd0ed18cd9edd98267c08b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "489d88dd7043ca818954ef19174cc42c6f591a7cc39697fcf6c3e21562fcb52e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "bca61b1561c4d583d08bd8d627fab687a7079ff75325c3046af12e88835f4487"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "c56f850c1c13f532991104465335372b17dda7e87d629a2ed22d6da8e89e3f92"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "9fc70f765677aba8d0ace70146e15db5342f363291f049af9df492e01c0dcd8b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "8b08f2c9bd3bf10d6cf7d09996a8e8f49a1f5e8e724fd6ed7b00622bb8033138"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "76a5384bb2d64255ae48239ca31fea9adcded14333a3a81289b1efd3c0769ea7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "829e62bf98609252896beb35b100e3a5bd6a3e081c95d85dd9a9ced9f68eaf79"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b9d739382313dcc74894d068f3f65aeb4d70a2eca9d0b739a5fc18a29dc41e7a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cff652398c983e7fe1459e0617e397ffc8bcbb6f8849517e7f36f768ce69a781"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "9e170759fc08cedd19ac53738ee6d2d7113523db87ae11c9834c4984696fed69"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "ed524c4c448cd15173932df0707cf9fdd9aac5dade82139a6ea1a57e08264598"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "1bfe1be86c16210ae2bb36230f56263f055260ccd0908f51bc89e3d825631f08"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "0f8aad1216f40f6985f8054a1d85e8d87f85a2e610d337f496b728a2e2eacc91"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "0962c01924f795ea239232f3de1b09ff0e22d9b2aa883e554728c50389121718"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "a109fefb810f28fa6c5099350a73ad0c2980311694b72f1e756a14513af7d7a2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9a4bab459053a5097e46a219ef8b3f4580ddedc7ad0ff88424cde985e2003151"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3d844b964d6156d8fda1895419658ee2ef661d7a4cd1b521fd1988f68b2a0781"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "35f60b64cb1d0925d8431ecd2b9553d8f7680ee4002210828d5b3d633207d07a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7d913a3e76ed3f2d0f262979ab6e576f6b826360e45b51659e8ea07ff7b4b0e3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e84ee259e2ce98f95d7a42fd79eb39e1f9806bb7de5fe881638f87eabf0777d0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "809c9202f630d61e27a391c8854523ce5187110f2a3fd01a137d152abfd1fd02"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4895b2758eccd4e7f12aaec8472937cf277861da5b269f862ffdacea9b08fc64"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "001ff5b6b977cad0769c63f81ed129224724d156f1d99114d6be0091b0c0b5b0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c79adde79e9ce1d7e49df899c55086aacaba394de20c814a5e0a311ec3a1e078"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "212ff388625de8518a37bf5ef517bd7b5c0a23d142b55fb6062a05628ed5064a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1a129ec0901fa3316a3df56f10fec11aa2d63a5278bc94a1fa476fd9fbb52919"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "652ccc3581c795b1964a592566c66b9d726c40e49c843bdbbcdb2b507de1b9d0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "484b6819f98bbfbf510bb41c6e01287bc9dbe384cccf272a20b23a96f71869b3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0eb2ffe83210c05043c03cf08e80158a5f0b99edfa2bcb173464038afe278cbe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "df63f755582cfe468627f76d60a1eab1109a3e5311298b86a473f52e109ebe90"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2fb9933a1ec366c1fc9ca0c1cd7adba3f576f3583e9b7a2d3fdb86dbe0fa6a98"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "eb90ec4b92d5d04ee369d33f7021967dcdc5c1ab11908a8cad8137c2ccb6868c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "58c91e2ebbe9703f2b01d6a2ba1221f28ed1f6ad576f827cf86135f2ca3a8f54"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "02ab85595c74f9751059bb630434cdd0b6f4f53f18310d75ac9f85938ee3dd53"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "c0c0af8099f55cbff17ae3be0b77bc2da3362c1996b6301b2ad30b44b318dfae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "ef46b08e8244174964e6ef7fcb0f91f3d9976f7f1f0da18a380b512d6b7ae8c1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "ed2d33652cdd5a991fb3889a351a28693b9da790b2dae515019d153088c0e7e8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "bbee5a628f4eca5f3777dc8943279b855779b55ff661a8c9b5a3767e790f197c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "57effdada0e19be4a930aeddea8497d034b5af432e3be730daf04002ba9e822c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "838788acf6f3c8318b42fd0125767cd1d0abb781f66190d4556a846848ac6f9f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "b3af34fe674f017ee8a241f799b6415159a81aae2725b60f1892d7e45cda14c0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "e0239228f7baf0d68328dba70075522e0651ce44ca9bea876050d0469d5aba13"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "37b1eb4a9c039581e86326b1a2e82426f50268512538e2939ed8a8051c8ccbcd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm103aKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "5784b18999da3f7437ce18c6247149075918ab54569765e0d7e4e116c676c917"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d8179588db4c90361246e863602d4b59a10c4ac1df979bc292d8c7d341d4f0cf"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "36c582e4c99a1ee57e081139581d28b545ccb797e7a2ddab5d88c21a652b6a09"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c78c22172cb4c8ce9eb59a239ffc8b982aa7ae9b2ae02272c49069d1ab25a165"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ca36db79b96be28b066e9d309de4f1b4091ce9b04110872e35b4c9e2bbe0a519"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "137e1e3dcfafc6dab320fa6df8f5b3ca41ce2cb2173dff407af48433100fb264"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2723c33e960cc4350105dd2c2002bb1a9b5841ed1564080b809cd969753b833d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "aca6c0da4e0abadc3480e4a4cac0c17ec58b892b4801639dc9f0b746b290e217"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "abe4fe2f1a11f6f5d8cdff22f8f72a275605c32c8f1ae352f9f537c36a188d62"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 166528, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bf9a4e03050908cb19161119a39479ab8397dfaf0f9af212e4522ff8a241be18"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 166160, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bfd4390c43c5e71cdd6edd12466e1de10ed067241b6feeb07fbe9dcb71389ac5"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "95b09facc50f124027c17dd8160cad2d8db17a05982570eea05891ed05870467"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0a2253106307d8aef719bb223cf38aa7d3e195753468cf16dd4eaba859dca340"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e25f8e83a830e71057875193a85f8ebbc4a06f27cb52c431169098392c2d8376"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d472836931af853f29fbd9bd85d82d0038f8afee41e76860f3b3231689595481"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2aab01d94c5d6ea6bf8e128f359c21d478352f1c6cf8084c9244f6078231b412"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f15efad0de24404a845519dce0ec370ea29249d3297cab97159fa1728eba8a0b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "59ddb0138810219c772fd8c811787181701d0ecbc4c436bb3ab5b20672d10128"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1d3288287d0f6cfd03aa9909242a29f5e0829aae16ff172543ba300cb89ba4fe"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7e68432212275a6f69762c38b2ed2353014b3d92b97f82580cad0e6cc8e3f3b5"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d4b288803a689c6d784e44cedca260a95a287b5cdda8174e613187aeed1fb99f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 167296, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "794de023ee1e53dd6a23e1bac76b7d27482ea265cdee28eae105a10866d5c4a6"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 167184, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6ea011043e4b752aedefb20c6f78f6f0a7e4c4f5bf4724b5df286cadfc0cfaf1"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 167312, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9e73edcdaa236d7462bdbb9068d07b401e50b054c297d5f6b8da930d2a2afbca"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 167200, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a45c952e4d9802b81d180f29bccbe36bae8d11e6f79069efe17c8883803d45d8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 166528, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "07a907a261622594c6c530fa40e5a4fd5f452c1b82b604d93103fe27e9c82d77"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 166160, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0bc78b389a72ccda30d777c9a3b956f4796191ae8c99ddf9990c1e616ef36632"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 166544, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7189762cabfe6053ddba5d42f5fb2d93ab6ef62bf18cd1954b39fea37cb9b83c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 166176, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "64adbaca38c746be429d8edc4e12d60389cb6aa2c2ea7c4b5a763579980a932b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "de701b3e5d7fde04c614b0d82946bf7846a14c5a17bf7c9721a48bfe056b34b8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1e29fa3ab719794f876abb72ba8fdc59aa98c0cd66ff175a1ef21b0bf46ad5c6"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3d3a3c5d565e40d6ee54f6f9fc3240571eff4150457fcf96982454cd78d7b39a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3c943f3b61fb42ce59605d1a24b30d310b47b1b31c1ce3546684493530f579de"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "17812015004e69a14d14137ff9f546593187de7a6b96d7c2e7e7d062fc0d566e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6bca3b04c1329ead2a4f8e13cc6c23b567e15085e9beb01b01b66087f1a5ac59"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "aadc83de4c8bf644204aa63a6aeba4d0462ca3ec6be972694f4c3592b751e820"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "28f5e8b82154ad39a4b00103dbe45c9061f5ad75a2ad144895ea5eec158b9652"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e01f9f77143b177d0ed7046ebdfb2f823b39d4fd0a772dccc398cdfc9396398a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b09ef1af31915c8eb9baa0999bcc310e90086a1b4065fa4e3630b7535fc02989"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b777f8984743146cd0c78d3d1b4a18752f50e45b3c83218c87049af524a4f97b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b31e464dd92d6d720f6f04476f3a563e3265c7ee7c5ee377664cd1df6ffe84a5"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6589cbac82a2a11cd4a1b74da43c2b9e06b00c88251bbdca0434b99945d5507c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ccc013cce4b3376c55ea118cb4ae787a9f73e3581926d611272fc3524f4a15b6"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f11fb2ef291792aa1c379184ae10303296352eaedf037b2cca6600cc4bf1ae90"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c9a4dee3f499d118c55cc3e95eafa9dddf30b92e2d2871cd0ebf315f83e3a6f0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "784a3caee7465e0a8779c9f28c2f6dd658a6e5415b9d1eee5819432d86eb43bd"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "24d55e88e91c43644b1e0137302874f2e2b35027e368d12d93bcb419d68e2673"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3e3ce22bad227612df5e46c2bc44690c7a15621be65fe49e93352392a02ef588"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "638281e17318dd36ff4b3967edd5e931b2070b23a7d049e4c018a5b20a5a0277"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "46d7d2f38fc04e0db622c7fc87be2379134bc0f8f79016fd01df54d522d780b3"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3ab4d1b4b5e526f2cef9cedc1993fc317fd33b020913ee048b3404d1714897a3"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0f38e4796c90084b93ad14969deb9757d5ab6e18f64724882efe878172757f5a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9f3c4beb3f63ab94c3305106623959d6f80e060b82289b2ccb1fce2aeba35fbb"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8123269f46dd9c8301fc7a13eece5c2a62ddea324eaf68488b399ccf4db90e72"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8059b2caf1d2a570b27837a1082f471b4fac58785b207dcf9f8555c777650b62"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ce3e47cac7592acb05b383e1b979968df8544ae46f66eedc937388050a53e8f7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "efba45ce815f35592e91974a637a258ca76690cf6415df96252591471e1a1aad"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4bf4da320817dd2d4b9b340463db15d2fa54a401eb8f4966ab7527bcc56fe03e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5d51fcb716375e4e6093b6923b735f11fdea3ce0dde0850e0bb974a9323c7801"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e3154ae705c8b07b6fe906243b12d8f9681e30b683a015227a45a06c0ab23dce"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "58ee530dcf2cfda0720a6951ab326f0107770c965657880db27ae31462cb1d37"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6ab449b77bdd372920fa059b82abfad848c6cea023074e2473904d42efca365a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "cb872b030866e007715435359555d4fbf542c22ec06d319c649e3b26423050a7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d1e6c5145de83e3cb5df0e372b2f5c01880a29c30df7d7bd429c9831fa6f28d3"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ce0498edc11dacca760cbfdfa954dbd6cc9656510f8ed795976636c917250b3e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c7e43292fdf72f836d6437b1963a2a4eb38b1e6f8cab09b768fb6094057956ad"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "335ed7f9615b7062420bf35e16e0bde70b0c8559f96283fd5bd3ffbc251837c5"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2768106ad734c4134a0643ad4b9c384cb5829b4f39a02415b993215ede35f264"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6de79e85b7249b6ea31579d09c92d9ad8e0ea57f7a0c918c8d947aaf968e8214"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "95ea6d1af0a0fde0aa471d5f1c8f35206d4549b4cd6d4bae7c4ca2dbcef5fd50"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d62a7fa923fabd4ada89191ca5e890b812e204f376369cc929713530d9e77e2c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "298aaa2291d07459450adaebf42eb83390a69851ee0249530a24889375f3c831"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fbbcb7f3229fc8ef855ec6fa0f99f3a487e07785e22b962a4c921bbf13f39884"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c9164613915e9eaf14918b50a5479f3cdc0e03666a0dc16ba584c3f0cb8049be"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "273ed2c1b50419af6fc53f1062d37a23a081a222cd5fe053b42486d367047451"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0af5d4d73a8c37c174d4af4004d6ade7dc062b1c89d362a005860c51b9d58713"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "990409428597242fb981d8506f5edcdf63e3d4e9a212ac7375a881d4285fa40f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1428c0872ca3d8713f9ace1025cb58d2923c5d82e226b8a72dbcc5280e0d9a4d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "180f3376754e5486eea7c45ae303bdb09535d237da986d4002bbc5726a9a0b3c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "581bc6f1bac6fef747ae1e1bad267de9db8bc27ba0b9bf756625c2997c378b6f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5133b792a3bd189b96a58b41a13532c4c797547248645cf6801879e5854d2c7b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "536aedbd5ab46d464a5ab4fe234ae981a2501b1cb282aac330881cb101da757d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "712d399ffc108220bd168f1085952b06ba81012204cf4d95cfd237c8ce25dfa5"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ff5cbadcc10e8ebda33fbec1f8342b5392c1dea710f4eb3a71d296cb445e7d19"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e7305bfffe8186b428fdd3c75ee0e1c96fbb0ea7d5b57412c15496833491692d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "295dd071d6bc7ea09e66ca89e8f719703d9cec8a8e170b39d7b07263b504b517"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "87918e5b49e5aeee6f2bef110b6d41be2c44a59133ecb41a647fb6ee3ce77710"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e78276100cfb48db727836e556a73f466449083031a6accd6d8993e58c7d4e57"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "11b32ce09a13254ac35d52e85f00545e20947f3d195dceac65b07f79bf5a9d14"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5305663225554a5a003b4c71586df50a2f52bff1dac3495260c46a8b84f5bcd4"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3046bd578350b8bc5d57156b1ea3b3f2911e0011edef85a92c4bae5f117a769d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "06e55f3a364f39a1c569beb7093f6c2d0b7da0a47030f80e0956291f5d25f682"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e956a09eb538bdd4c2c67394366b97837036b4536d29bb352c7f2ec176e4570a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a04876ba15d81a8cab5d76d9a578447aeb5454e01e3500141fde93e8788bf9fe"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3795ccdd152980fb3bf2e39295a89801ab380a02d27e39064db32cd743cd213a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "32f97b9c72cadde2a31b2ca7b5351439a0c930259e71c013349853dbd3cc7e22"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5e33990d07f7e10b5ce3ba54afa1f04ac1c8c15e3bdb2ca8b853bdb31a4b6078"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e3457b9600306ee4f79684c65edee35514b62e70e7a0f20a5d1c9f1f014624cd"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "cf4d304d3f3fc184e8a741078cfdd7988027e9246a0167169a4da0c772379ab6"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f9f4b3f42e8522b0845dcba9a11d39c5b443e70791f93886920b6b297b565d79"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "53eea56a2c85e4fad722264f9d872503f81b44d0a4b735c2c7a4c4756e491fc9"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "04006e966749f4acec124e0f8d7adcd19dc000be25ebab24921b0f2cc8461945"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5999ac5066e077ac9fbf72ef88d216dcc76dc8b7270b73b62063c1ce0501209a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "14a0a00323e9f345c53b911371bb45e1e9fe52f13baae641f2207487ed03e426"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "39971318b6ce9d127087f59626786d8a186ace1601154ad4dae916a0116a0528"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "43be189e4b8031092f601cc369acb049d7d4fe8991f47f826094e6153f6e7579"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b920f451286edf3aaa5844091197f376b6eb68c64d9a20ec2dd5adb382ee0146"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2a241b492974d27886b4b47c4bdb55341293238b5ece0f0cb2fa374a420ff19e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "637470070db81ad0f8d6fc2eace2d5060ebdc246a07fdbb82e83609b69a39313"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c82359d4339642d54c44dd1c561968bbf6daa967df5a7147214a121b41ece309"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "eef0ae6e9034eb05de14da766f84899ec95a8932b0618abed3dc3fa3cf936a80"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3f7b3a3e108170de57f7fafed93136028aa84cc6fac37218326d8ecc5632e2db"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "170bc3b0a6b7b4dbc188b1b32a13519778e8a2975abca911bec6191e38c97aeb"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3e6be5370e62bcc7430da0eae276dd3369dc79711a62f9eadde22bcde8964bef"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1f268db54143d54d9497ea5f0d810f88f17d1cbb732a0f797d3da91771e000d8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "96d4df679e9461bdaef83aaa19e1696cfed167b87a6d3c6e585b7f62b58e9f47"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_103, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm103aKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7069f8e48cd49789a721a5308cab89d2d9ea4ccd079c934c3157dd84ce8bd73c"}, #endif // EXCLUDE_SM_103 #ifndef EXCLUDE_SM_107 #endif // EXCLUDE_SM_107 #ifndef EXCLUDE_SM_100F -{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 115328, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f0704c30376c3e55d9f67a75fa9ecb81c523bb10c11beafb1939b287e7b84d4c"}, -{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 114960, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "95a415e36c51c3773e972f871d6dd7650e9f93ae903b0791508636b650ea8240"}, -{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 115344, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4ffc3f78772f53625781e5c3bb972f4cc1b7821ec438dcbb3852b54972f86bff"}, -{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 114976, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8bd24b8d9996ac62e355c0b7fd0b1bd0e96945d3fd26874091a413590397870c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7c58254b9a03786a67866364b9f8d53a5f72034cf87f49d642479a0c1699e40f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6e05a2fec2704963fbceeae54ff708fb244332faced4c78acfbf61a2d083a2fa"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d0ed52039542b80c748dc84dccad49b5c4614f1133455be6a6a49681952ae529"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e071d0558f7d373f6cb782d8d4589889f7c9b46e3849919f6ac45e62d5828022"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "30157e1b43d6b1f0b36a69746bd541d2c0039e68fdb4a55023ba0be7805672db"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3bd65b9c11dfdf30afdc68dba1ffa24571d5b34598207d930dd3a47de9ca461c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0ce9bd688effe38ced9806889ed993be54bc846d4eec272576f0fc3de2b5411a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fee06e199cdab0777cbd35411833559053daba8e8c901c77104775d4342aa21f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "66bb463af106c9f9b754aa662e6739764f5a867525afb66b8e9f0213bf23c04a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3b8fea9bbc2c6b22e9c366d9abee83f144729252d78733b05af48afd4da5e828"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "184ce29aff0ee0f785d680ecca6722c20478a8093b880615cc195da52cfa697b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c186691915979cc15c1b5542a2a6f9309619eebaac37f7b9332c8506244236c5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "571e1d2a801483fccedef8930e7673e073c0a154110f12275c97c7c588941208"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4b5a39aa8df81d5ce860eb7af0ea92c0dca954c27f78b4192241d130c06a8f0a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "56a8bd9e9c6d47fabb6db8cac04a404612525631ce6dcc42217226c24f7929bc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148800, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "20585aaf5104795958ce7b4b68b8a9bddda78d3e618b4bbdc69f814ecb517bb4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "aae464f9f3e86277437ea5d879d78c8575c6cf514f962314049b88b725aa7353"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8f43eb6eb5afcff8e7c29c9910b1c946bdbdea4f84ac4b93afdcaed89385b385"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "eedcffd750d43989be75f885a7be643425ee6a4d81d6788ed2efb3536244d3f6"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1411af9a31cd9446ff4a9442a800276792d1e14db480d763a9125de983bb6835"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f616469d97ba34b7e9a128e54b27462b57a5ac5e8ffbacd161549ea02180875e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9dc92a661a0e99ce65e9359d9cca7966a10f2aa8e2a2a11709656147f3ea4d35"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "389939baecd11d38c4ea9dece8fc01130c9f5d4642faeed6f80ac9fab32e6a2e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5047a74786e968a5307808b54e43db6fb82fb709a2e24da3d6e76f7dffdb2fff"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "bfb1ebfa52f65ce204e2fc62832ce75d36e92a4add0afa840eab07e26aed7a6b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 148752, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2817d1c3b8fce8bded676cccddc41ac0588a79d6ef45068e2d88a7701782e062"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bdc66c9468eadd16b146306d8db9e88a8afa9df4e1c6a1685d1e385c7dddd0e4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e9675b8224153ec4fffac4fc820bf9b90517c482e22752b49aa19c20655fe163"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "746235825084f1549e3e793c8165fa995a3f2222bd4ea2827c76138f71843960"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "642821f83d9d0b2a5a48405efcc521937a450cfff9140c433d992ff4296df13f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d64bb8ca68af5dae479c5e54a978c68f46cf31fca5586c0cbb19d59cc2af615c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1e60b7c6af25db33485c0d63d963dc6569adcdeb8f10900925ceb826471ba8ad"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "993d5ac5eac07d6198ee322fff80b475f948b963359f17f741ebe0d2d0753bfe"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f2e163e58d38f84c5ada45427651170dd49bbf2adacd77ed30dd5e1c5b50bf82"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "43040745813af9d7fe8ec49f7824612bc6f8c2f4fbb581d4ebf4182d655680c9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6f3c77067193ccda6343636c5ee68ebd38be1920c4ca3e99515758879419858c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "19bd1184bdbecb7d1bf560090b6a3293da5f3bb262acd6d01fad0720fb50f43a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ff48605169e277699aa2b396e2a7c0d19d8084445542dfc5ab0ff24e885d6907"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6ddf9051e9f87ad40421d92834ff6074eb7bfb8631ef424378b1b96a26447450"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "23cd0b81e2feb42c61390e6b856a1fe6caccc8f2c447de07d4042d7993f748d0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5c253f1d1a1bd0e0e05c4a0a8aeb54232949692f9abbeb05eeb8de1d44f8040c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "182752c56c80ce1a226dba65cbdcd6a2b9d79e9cb51da89b5fcfaf61f4d76b28"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b9108d8a59a08d38032454e9c886612643cce4ddd0ee5e11b81a8e30cc6d8181"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "06b3fddb30d8766da3a28e976a8fa98031487e51ff0fe4f644f495f48a2b3d40"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3ec29768ce7aad01017e482a78008a3296f626969d2dc501ace9b4377080c99f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e5eb03726574eb3b103f0619aefa341e00eeb2a0fbabddf0c45a3fa689d397eb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6330191735ad17b672e6b7506e5c77b9321fe5cef41c39c38503568c4dde95ae"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148800, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "54b63154a30844da55a103b6afda7a2d410476c021d8944544490ee0de83d3ad"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "09640e1e13fd51f6c1ade0f82e3de4f595dcf6f77dcab832bdceff9e249be3c8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3bbea552f5da37acf1cdd14a0459e0655629f62bd84bae9d8728d93c49f93ae5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "40c8992f886daca18c847e0298541064fdd17304a6f44a19b35e7752b89374f3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3304d4eebb183f71af8af0dc166959ad0ff257943bdc4d621a5b28d46c8678f2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3c22d48ce092c0267c821aeb6be7ee5c299b0a4d46f7929a4acfa18cb1b5f684"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6a2913bf0f9201d59d155740b23d64faa49fafbebb3bdcd8686f7fb8c03abb11"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9f2e4d7c9b81f1dd28a3d9a730a733fe5e3a419d690e1673f9cc974e74a43b5b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4f7042aa28233ec876f5dce817032c06c2d586f7c113df6e992be8c8c372abe9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5589c6a487175f8983281fd050449e62f880e9c4df49cb041ddcf14b8c7d18d4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 148752, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ca7ba27435d453cc5704ba39e1400406f4000d045151728253662a4e1ae32702"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a9f8099026f80b6cd1baf0bb02a94e23eb27b10e75df892e4737633f72561424"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "cc8fc66d1145dd1248bf06b7fa1b77cc230a4390804dc6f2f7b7e990376bafa9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c1bc485e7da6f804f4bbd7ee13430e3348282378ad1f9f3364f272bba2fe23e5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e91e524a608e12cae913e2bff52ae1bf715f6970acfdc6e3347f5e1706d38601"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a089a50b7a5d4aef06380faf57e33a0036327bf5508c478c98d5638cfd51e262"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "38b44b9a57b83c1717603d5966ca68027e55dfc29de4c13847907d16cfe1a47c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b54697bf65683fa3eb95c2ae677c0766fb2be2725a9d914ce228d8ede4362bfd"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2ee3d5f7dfa3de0ca5a69c5779a5d8b49fb001e8594870e42a5ee94d9f5ec9b7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "53d0532b149af48426a5b3fe3c932223667d72fa022a7f16d76c28b4d958f2d5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9ef27b8718a521cda6c14a5f83b7c93924158f7b5ced0ca95c80bdf4be2a9753"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fbc369a615351583c2dd4f6cad3ea08a108eadcf44966914ea72724f0d9525c1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7ab70b75f16a0264cd9fdb368958ec9c1f095778f02b836af677243283a366bd"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "867fcfbb9c26258b08143e70ba7953915c944bf8af91664a5e69193c02fcc656"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c582a4b8cf135b977a15f09a67539763a19064c2606dff64b7bd661c6190691a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "44c28b02750b901c104460978227e532ba61669a3537116ccfc1e16389adbbfc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f4c65fea9096bd718ffe81aa7c5a49230249943e0a607166fe4ff5498dfd10e2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "865416c47598e1f3c02f0dfb757afef1813cf734940af943ea86be74aa113f14"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "55b21a64ad49fd060812b535250bdc5876ac54afe418dd0dfd392b9a08b46b5e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9b16c7a949f3a8dd23a2f9d33cd7aa75882efdd90c933f19bff558510459ff72"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2804df190fb6b29742b7d090b8a7c2342738938e03f25f7bbc712f1ed37554e9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "06c0400e1e159a7dc6f0890384a3114125b10397d90e9f1800918cad51f464a6"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b4737d0944a7e9ed161b551ccb179f94367abf5a8752187ccbab894419a450a0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "be2071b7782c1ca7ad5d2c32efbda0108d6388d25a4ad5eeb0380981e30d608a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ef38c733fe6605219905da469796d5d3b525720996a1fcc55cd1a002b06a405c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7b5801fcaa8b8f864d50bac9f3d525355e2206bbeaba9e48f802838baa280ea1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1515fa9b1e086008b15414fb4609f08bb49adeafdab0821fbf34291113c74492"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9e95521e298ee525d7f3fdc690f28bef8f344e42aa386188553a8b4cdef4b79b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f4bb764b1cee32ca14e5d2fe3344560b149ddf3bfcc1bb8931964401e89df227"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6801ea1e421df296facb1a5cdffe9055d309d8839cb58e78fc89d5408ccd8b96"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197952, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a450344f0e184be569bf3bb0a5beca3d429f052188c97f401ead68b868a6011f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165184, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "fc5c8d91c7de9949674986e695d5c616eb8df7d79c758db2aeaf49d71cd731ba"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4bba824bd0f0fe13e1ba9d2fc992237bd25cbf9ad428accd8e149e10dbc87a89"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "98d52ca635a0b448418fa932a16e6fe8c79e447a49a03774f1cd1dd820cf8bab"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ccd02265b645bcc66900e8883d0eba3bf7779928725215f88463ef37b77a9a3d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8d3158a5317a0c7e7a7eba40cee13b04afca09fe5c921bb6b6a8e8ac2a4c0a74"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "07517a72e842d61871ca0f4f619d3ffd97abc324500f2b464a378f150d5ed67a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7ab01f58c6ed6d50e06e074ae65cb53a2618bc3c135f08d92d6099d09d935998"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4f0e2842b8b633440c942cc67309b1058bcad0adb563e258c549032f04344d0b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f606e4595b92cb96262c013c3ef1eb945d472d5e8e708472bba7d99670442ebd"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "63e3be32fd7e8b7fb94bbfd1e598ce2cce43c08f352ae6a0960f46bbde50b2a3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 165136, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a276485bd725efae7d5170fd68d5a02cd725a312fd29d117d81968c3f3c8944d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c6b05584fe83d19b269bed9972c6ad419421077b50396546d7d2a7609b81af42"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "172f26553080980644e9998b27bc59942fb6708c20a3e6b28390ee0eec2ec1e7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9c231e8636c13185f99f56f1b4198caeed3464da0a0db73385e7b13c9872ad13"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8c48a9b93052fad2f0ef219e99363255214fb0db2d017ef5b4e3ca5e60cf88a8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e8ad55021071de558a4635b1d36a847c12069c87c8c8f6c33c943f0a89a37cc5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9b03ad7810aa2b256436af86bb00b432d112bcaba02ac28ae8b7d39e34615ad8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5936a25f2045bcef2da4412644180bd0acb31378907deaf40f15dba84985f967"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4022b4751d65bb9afcffc3b9bb1c8c4a262034c9fe1f5f1dadbca4f6eb5fa0b7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "da1950c80455565d729d28cbfb93bd645ca5da4e1a574c6d46a6f897ea1a9f7c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "acc675da2583c35b2abfdaaababa9df664be62f8ec7755ad35ea197f31f9f2e1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8e67e97799851fe9b33eede139a3510c12c5f421994f3f06fae96ed4663490a2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "bfcbd02dcc77313e3a36350c4cc5fa5cd60b560e658b19baddc26ef38bba43b9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197952, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "358c5684c74cd0f8ba7420088a1328ab175532d59b9e7a7f987ab6bda3ec8d90"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165184, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b9e3b4b62a436ae5f97fd87b5ba2017ed175b79569ba093886224c9e0bab2d00"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c33d97a9752cc7c2d76e325c427a829843ae52fcd20a51f9abe6800d4479a5b8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "818b59213eaf59ded2182681fec06ab21a29e3dd9b017a119e8bae97f140a4ac"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3c3b3984371a03a9005ea160a0f63ae5b884f6fb725ca46337d36f20252a94cc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c01b65b358dedce0b6bca06897f0fd6a85a50979b536b1fdfc53d6ecdb62d9c0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1fb0fbddc619be78dd00a7ddc45a565979d12765c9aad1638b4bb83242cbd127"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e1862af9572cf8f26bb1e542196a5db089e495840b014d60107b4f63a3fb4305"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9371e1836c8922b81756fb1275286ab9b0d26eaf307a5228aee1a805fd5092ca"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e0e3366300ab3d3de8b4c6f1a3a60a79371090432e0d92d428b8314cdd85f445"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4b3d3d19409e77d68fd76e2d382df4a9244adbcac9474b878b212ddc02c2bfa8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 165136, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d038b9f3668dc383047c5461238dd75247821501708c73f0106800a74cf31920"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6a8d0d5fb1f44d1f5830d139cc95d99877f5b20d9542bda5bea05def7bc19761"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7d0bb0b614e227e84102d2b9cf779f572c467930f53f644e8db4ac79cc563857"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6d159017308139cb20b85be954e235dcc7bfb0219e019cd2da6b0ae1e246518d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ee6a1347e9d8c704f1fee654743a32329a102cbbebbd05384153b7c0274d3471"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "cee730c0fe53fcc854ec4f02e179798f05710e805a8647050767d07fa97591a8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f2ead8b15d54efbdec78060a2b49c55d8ad32b33fe00597c7bb1758b0fa5ca52"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a80aef6512588be828aaea7c40426c17b6d382c3533639848c20084512bf0c1c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a655e47aed0accee37a0d8cc8fce3b9d4ee5762e0734606053c0b530d9261d10"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fa78a954c1c2345aea313f7878e12fd854966ec3d3270f5b8d6f5cbbc17a6a06"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b783fceb98a43719498baf01a981923e157e3cc5f714a0a1b498bb76b486839f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "64026ae18355d759a7738898ca21032b918d4de008332387b6bd14592da9c55c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e5b4be61ca5adc864a7f98b2b11533493eec9f0bc39d7d946bc9204e71b6d108"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0c9caaf3d870672daacd2a2ced4f906beb36b8199d3407e0b40d3afa06e7f014"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0192935794168d4d064c7dabc6c0a0fff0720bf3ec1666de2d2a632beaebbe0b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 196888, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "f64d9e61ca8813d3b0484e8b8b2f2df6118feca38d77a83ca8f34ff613b4b330"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 222232, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "7343cb229ffb790d36d1dce4ab9dac40a417787270ca788a8f907af729ed2969"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 184216, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "de134d3fd732b2991689df8039803b10a87dcbd6f44b9eff29b3f0c990c8c4a8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "ee1c1e1d31f4d59eb61dc6839b0211d7c139527bdf673a929a7aeb7717da1686"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "69df326b6441cf4ca362e9dcad305fbc9fe27c0a55810221c4a1bc8ec48958d7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "4bb1f910ab23624e8ef898bf74ea822386df563937abefafa6f67c3862251b13"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "922f3963b4fe906a08d969e383888b4e7f4b67c41954138cc7c01a5b11ce3f66"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 167280, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "6c9936270e5775f40361d3112b89b85a8f122c7d78c851938a44ba7c004d5f07"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "f5da832638b25425af3167dda47c676b35f174bbc18043a658e5885b860a73a5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 196720, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "cad888f53f3db9390e1b16181a81b05be3537027a4aa70cc8ae995495fad8b0d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "ca86f7c865e0fc963c985733af5387f9c7fdc6fe6fa9782db8e654edf27a4323"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203376, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "c546cdc08b4c8ef7990f8488093f79d14d3ae3a1b7292527e00e8410b58b4a2b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "1ce11beef5952828ff8ece6a1d073cfc33c09943749b1ac89fa7a796ab4e2447"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 152560, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "8c1fb9de1642877dd342e40f5002b68c0140fbdcb3b898b4b1ebd2fbbc358fbe"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "c93f99da6bfad49dbc8f3db18d1ce433373376bb6a856b19ebbead185d067acc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6372e47580909564e8a5ca8e89779166de1fa70789747408299528052cd2bc7f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "481a161abed0e4d0393f449aafb09d86b321a6074c877ba208b1ef7af1bdd192"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "68744dce8e639605ced93b4545d416fbc622be6c57a18fb41dd65eee5f8b67cb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "334961499d4f54c1b5f963d9dd7cd6a5e02a3f3ce1f6b51b6c50b761b66da047"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c3cf70f109f3f5fc45c098a7e042448f4f34c1a2b7c9f21267a55ce3e27b8f03"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c68814b398f361c5e265f5029922773eda4a326b433a4772b7d76f4dbfa7e5a0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cb5416b0d9812e8ace1cd8d75dbf42cf47320fce682d897cf3f9aa98c104f460"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f59b5f345f34a2715f743b5f2e902450fbdbdd5601ee7c5a56653373b81de774"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "284e4488d80c171591c65ee958a0c51e6606de05dfc16faf2da03f09b791d436"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a820136fc92e9345c086c7c15a9d5dc32200a51647d61c503afd5980ef43f9d9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197680, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "38846a041491a8bb5d5e6dc04bdb57d2ce5ada60f16dfaee3026762417af52e6"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "34983b25203e0a74114ddfd0825eb2773858756c62e0828154d9eeaa908d7f1e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "922d2d8f7aa4c1cbb4f5405e3ed7bb67658ac870c9be5c88086299efbafd40a5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f452b403e803da2c8c3d178df14b119b1b16d500d5a00ee1c3b1b528b9750fd5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "00ee491daeea9fe1d6f0b99c64da62afda519fca5e05703aa5c95f3687bde075"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197648, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "783802348177d093fdc78d5018e7d7929e15c0e33b810c4a9997712ba2d38204"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6aaebd35c24644454ef0646580290d0193ca18f3bcb07879532e8dcb34e019cc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1894b62c8d13c52b84262d220a8731ec363a2b487d436c93577db0e8a5d25317"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "53a84cb7ffe9d1e1ac955677334348cc6e38603c03234b7b7638756762de6266"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7f156a3d5ab5669df57bcb4bd439051d2fc4046bf41091c64209b6b930a9e1f3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e9972c23b902638196ce6699106cf6399f8eedf3dc29da62d5f602fb562f36fc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d68f6485270d865e21d5666a93a5244a6d7a9bb9cd3da201e744ebd08e226993"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1fa6a70302e89f8039cb964bd8dd94d43fe4e9524a0f34bf047be0dec3232f12"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c43e47e4200de2c35b2266dcc08622e77c39b846655c5b0f9ba408c067d79f5f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 196376, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "ab5f2c764e7b0b24649fbc632fee03dc9a71cee9adb9f8a1712172a7564bec47"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 221720, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "8d1fbd0311ac527f16c7f833c8141797602068fb9d81344842e3b6ff84437ec1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 183704, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "85f7076a0793bad65558774d52002d6232f4d4bfca26230b722516fef0755beb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 221224, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, false, false, false, "e0b991c54e57d90935fcfb67373cd0eb944aac48963baeef889bb58e6bd356fb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "c6aed048412db8d37c1bf2303958b46ab4292455d5d77a6adf3f22333da0c12b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "0b53b7334c274e3794ecf7790d90099a26bda0adcfd5cbc050c86690b7f1118d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "443b8e5fbc2b5598a762aae0bd6a41bd5205f1645749abcb9b5f51d406b51174"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "e75ddd013bfaf83386bb94d1f63084ce230eca2e42a0a6fa8c2e065876c9b710"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 167280, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "9afc97c0ef35f853e413f848bd386473399ec4a22a051ac412f169872494fdb6"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "75d1788c5c25cdf3fd833531b41d57d329ca55b6d5d5854e822641e49a24ec18"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 196720, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "4ffedd7b42a73c141fe43214c50fea3a7a60b77ae4e4ca2eeff08c227fc5ac8b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "47a982e31cd9d605c3368f97fe3801ae5169c796192564440d1e4d4e7c0f4897"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 221304, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, false, false, false, "478475f4e3dc241f3420511db9a083d4316385ded0d2c52c0b47c5e5506811f3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203376, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "81dacabff8959c480562debb06af33fe17ee1b081675adf04c151c32a52959ed"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen", 221208, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, false, false, false, "77c33e937ac06f8c4110e02027d23dff5dfa74f478addfea5c945a9609f4c3f8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "1aa7c83ef725fa994e13d62f97cac0fab64ede1069917c2a8982d2ccb0eabf0e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 152560, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "484c2bf97d760665e85ce67dea0ed9241ef8613d141fbfd906d75010c4e0dbc1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "f8291b42b587059206ca0312c61940be39db5d19da4cb2fcf73e975577e897f5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "144ddc8c2e56ac1019cf6ee0cd09f2bb7ecc610694b892bb854608ba1155fc94"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "65702a07a8d484a4ab52dc445c164062aa61d18c075cae69ef208c31b1d55964"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f2f79e67ad1d0208fd8b98821d11b794c7e0db5b1b9ca8b05812e9bd068ec87b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "171248acba11a1c8d39d3fff7d9c40adff9ce929bf96a77fda6e75d67a4461c1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "39bddeda491e1d7adff06f050255dc51e6b426f44d9cef1cce2913762dc3e70e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d9ace599827f51a15e5aae5355a0180a1c1856bbbc4747e532d5eca24e647869"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197680, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a11f9d18deb5995a2f238278f3f96e9209473a6b21e5efec6dcadf5a3461e871"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "233f1feb141f9692f693cf716d66b034401373433753d3254f20cf468758346b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "764d5255c937008123544a4a1ff4829fd16167411aecd9696521ea0d1b8fab8a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197648, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3d0edfd30d33c39fee2e14cec9aeae38f67762fa07e96b1f99b0afe2ef2f8ef9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "39861df19322926ba336c31f41d48fd533bb72703a925dd87459df174f90ae13"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "43fd901582019866091d2d9ba7cdea4f8e478f7dc34b0da74343c469d53bed17"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8fb0f24cf22e5ec33e31ee8fbde3ccad26c3747240c1bcc4096ee5575385abd1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e0264b9216dcce6013e8bb4e3a5b230ee1b345f130fce846ba7ec4259b4ecbb9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "bee3db791670c5ca873040f785c6561ade829a6423fe5586787dd9a1bf558386"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "580a288eebeb400d999a257e8a0855a2c3787518ad675aa0b31ddac7dcf48f3d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 196120, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "867ff63ac778ae4dca37b2dda13839da120af2f9638c5e04cf8c063905343860"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 183448, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "4dbb1b2ee792ae211618ecc554c89b29c6b312592ffd889f26a15ab96950cb24"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "e8f13600d9a713844ab602164f89c3139395789343292f9cee190cc6022f8985"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "e81824bd3c965084db06c027429d527f979fa5b876e7e994a9b3accbc9f7a0af"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "962f8356723ed6288eaf7890b12691c65e193f2b07701415709c376739425041"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "c80344a02e5eebf4937070d356693adb65258c504f35b1f8aa9c330f9a685326"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 167280, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "adad4535d871ee40d4c0214653dcecfb9b3411dcee6e5f1c8c8e709dc487e5a3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "b4c66f5087731d31217992d1231ded0f53eb651eb14c5e71c13e2ac2884bf211"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 196720, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "eb5cf42ee0a4e4028b88de0826155ca09940b7f7d6d3518f53935c6f03681ffe"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "3e500a39800c985bb17cbe76ba6c3c46556c10c3168646bd186fd916b2f5b52f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203376, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "c6ddb0c912a9bbb4363e6c90527a54b131eb142ae36bd3f037708106e5c85823"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "92a9716e896dc56b2402f71c1a31dd07266569ec0490e98be9398cdc6bebe019"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 152560, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "e5c39a6516c883edbe2f6895630d9d43a6626bd01de279a414fc5432e349756d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "e5f2ff0b5672b0ec209fe4439905561f4c3b6b6b2275edcfd5ead0953f255d41"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1198215a8020153c07a8186eb2bbf4917282a862ff28347e8ff539e2ce3349e7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7fe3e5c2712fc59a5fc8385eb7c8c5c196b3a7a1ba3e21ae9e3b7531012a3caf"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ec50a04367c59425f4d48b786d310c4bf219670c51b64a4e0461abada143fe9c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b29ff55d0f966d0f0d2a3580e1c09b81ccea992342705d8adb4cb55545c00ca0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "eaf0fe0f0827cbc14d6452e71b5c4b83d9807aba4994542913149b8ad879bb95"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7fdade09cdae4a8c416b45f6ee9ad7a534719091a0a8929b216c9f413b371003"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bbc78d51eb44d03d140f2c60e0c4299fab97e9b2a8f782e99ebe9f6094c00771"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "010845b5f5008df6664b46790a5ecbb4688b60408e4f577ee24ceb913af3e821"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "03e447b20d8890aed47d7c89f5809ee7495fb22bbcf396ea0c7615ec77b2fe66"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "404362c3b6b79f6b4e5b16ed5e491b667bf6eb5ee820cc2ff09ab4a621884685"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f5fc605362bf67abb88b1b83068c0d78ba1098bd03bb49001f19b11237a58375"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f24164c3ee34e2346ae354e1b28ec18d9d604bf3459538a97d88bee61102ef8b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "796c571978ae0218721047b80342ac30013b2587a6f7328fb9d9319b99b25359"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e60f839fee39f015a0bc24e48a27fccc07941702044cc7ca1e6b91f6eb6c67b2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7ffd457a4b67f6a910eb227d748de7cec37bb3e7bcd03347e7edae8d53182020"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "fadafb10a2ca7d946f2d8f0d0a343d6c06b8bc5f4cebe7e6810080964f66b18b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b86ea4d2fc42055a8c2e6c74e9ba6df89e512452b6524e4b38518d2d0f43bac3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "516ff30e48411d62ff10eb6854b1a04b57c80aa95cf2e9c713f8331005fdd217"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b03b5fa4cbca31124666b2e17d565b52c609068765c899ed2257e37d28b8596e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1869a159d650d7d9b5573c8c82605c3f3b843158cf255f5b3aa2ca71334d9cbd"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7e232c0300fd93539d64c6634ddefd89b0eeb3e159456cf27e42323d1b775a3b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4dd184c8b3062a9a3a55c4241a4a18a0b00d1802aeb4d02b417c65881999ab87"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8eca69affa5e92a05b2d69d2456eea7ff9189d22cf96b8182c7bbdaf3625192f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "698713ef2b4d0df84521bb21a2e2f3383e72796e3c7f3f680cd65f240d99a60a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "731750469bc744f3809619bac609f8f80e58d28d90cefec9f4214fb3c9cd04c0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b192f60e8150cf53f77a670dfca8fd9ab3e12ebfe04a2aeae529fa0e13b275c3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "20fe547141d9bdf0c88cabeb4ef68857b8f1eb4e96980d005164b13a80154120"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "720fbe28bf8f60863f148012e2e3c12091f59ce5ae0b09b86fdb707f28fd4a2b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a4b8bd8d1b3057b9908cc4498313d0616af5e152986e9c9c883a43478b035f30"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e1d38483290550d191a6db1615baf32346f14394b418667347550872dde6fc97"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4c005a16046419e4f04135826021a6ae7e4d572fc61edb96f21b40470fabaf9d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "be3377e4d8bd74ecbc8f05f4916f2d878e63b0e1304c680da26f243b84edc9a5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "70ac8cb4b42d67926b1f7b640ae2c917fc59b920230043b0db2395ceaf2715b2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ad7bf958c285a6eede7d043c2c4baf4183e8dd4a002d5a180db2803b354c9c44"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d95104a3246ec8a533bafa81810a1759d1d04850ea719a88f2e4628a8ff38aa4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "beec0abdfac9a67ea6ceddbe8ee8249d4878394580046e53ec6ef836534d26b4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "01d0072b86c1af4a028ca91c7bc9e97cb64486a7bf1987edc7eac4355344d4d5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b0b39afc7e5c3522b214f525e3691fb0cfd1b463a06508b2640431e8ca02a6b5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "920c926e25f6bf1f5fa3c4ced3aec7f84a897f1f2a9d534ce740706859c86829"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "66840efbc35e3cc8e305aeb9bfc991038ad6d9f7d8e1e7d23813a5fbb03fdaa6"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1a70bca2b049348a8fa9325aed00b239b4f121f2499dc3656bdccbd15d8ade68"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4a0abcc0dfdffa8a02723e28634fffe84a88c247bff9821cfa4036e09d49cf28"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5968da87d756d225b503d58eddcad56a1d660420d1fdbd9b3910a4c3faf6cf9a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ea5f0e0c5b15a03dc9a7d4131e0224eda2b0b94047abd6fedbea7d75f6d14d52"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "86c298c8ef51c2f460ff349d0a7e7b43e8e30a7bc51382588b7e8b41646f0e1d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a661802ade91265f6b9f7c663507404e65f2b030cdf4dff5ab801e6a6cf4f46b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b167368c14a3c714b9e78db7aa1d4f5166fb967c634bf6c5cefcd01584b6137a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "fc4b0ddf323cd254caaf0f81a07ceecc87e2acd23d3ae6aacb8c4bfbe3fa0d9b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "63edbd204e4f09abbfb041237a351cc2014454efc57f8be40b40a53da83f1d94"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a3d93d725cc7229f57516c4e7c59558ea61f12c4ca8ee887fcb90fe8b303e1ef"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8cad27bf3daa2c3e474ac8d898fe2b3f526d69a2aad0fd84dfc2f2de973765a1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "191877b188bcf48966e3649a2d1b090eadd8880cb52e900c73b0104408238e36"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "41123f891c04702c7d996d325933ecd6ac52f898f20a304aadd385951770edf9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dd685db1cde9b675744035b173a19dfb6931447d7cc0269b554933634bafd6e0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "51a99deb25e942cdee9cb91d74197489cad899cc92ed824634efd1f0c39c396b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "41d9276566b14befcda708d47f7d0e3fb170d325705dc207257de38dbbe0cf79"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1cf090ac142c8bd37394f8123b6d76b6fdb5f4001089f392413ed0e51bd500ce"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c4498df27219c3d3f76d940c318b1a24c565cc4da4a91f7ce6343977ce6610c6"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b9c0e1d6a8fbe706bfb4cd7922efb24e5d9f368389e0a463dc4cbd4f1832ec01"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7412a1db628cb080d95a5b97a1911ceeedf4bf9b35344e6ae0bc76d8189b1ad3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0429e041f2e9dffcf29d5ce32f3f18e37c4fa4074d03642549e3fea6afd9bace"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ef049983e7730d8536360ad0b3d61dce0354117fc86cd224eb7de0443870c99a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4d936d47502793b28127353fe24bc6f5038339a7eed6a9ee221b57c1125143cb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e68e1613c49d94f1ea0d189f97f119e15bfee6b6cc2f95135b309a847a058b5c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d87658db1fe89e2a0dc157b6d50d187dc4c543c4b95c3b7f7ab641b8107bc790"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "17e273967d3b5852b617991e96a2a0c066b46533cf1550585ebcbe2c1ddd9a44"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "901ccdd2345fe4f39e7694401c6b43f522323a9af394e01b27fe9097ba390244"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4c835431c9b4882edc61e6e40c4b6c79a4afd11ca734a0900457b147e76083cb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7ad445c273ef9b6ebc632cc58147d2cfcdd04a96e0e14e70930a6605836e265e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5d45482522fbd2a10e7320f1c80627007f2febf183288f7df32293e5279aed01"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9c7864aa5734a2c633e58deed09fcb3690ad12ebd9fd5041b1344ef5b77189f1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e7301dfaf1b68bd766ecfb52225c2e2dabf6a237aafa85d4c3b0862f55cc8094"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8bce369cfc9e41d23b1bd0eb7cf5e61134e895235c4a539391d767e838328e78"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5d653cbabf83f8978a8f98c7f3a4a2747e76490a15cd19672b498264899c4059"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "77442b4b7c56a2cd6d17db6547340b2416035b8325c2a6d79ab266e46bac5fb1"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4e813a7aca992fe3940a2c428c63d4633a80133aae8c829dd0352e251705cd53"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8fbdb11209554af20891ef66766d51cb0a3245198c5919081c31927313efafb7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "48cfe6d8f8375e392395407ad08b7ce0cd2063ff829845bb169f4e7402d435af"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b7003ce42ab4ecb91d8a7de7048b537cd92c77ade0054b2416173b823755321c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f5a050680f0db5636af531ec2ee67e50be2590bc8ceff9e6c029b5e023626242"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0cfd39bec9eb02804435d2f1377c46321b9e87bb490470c5d9594a3842aeb306"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "43e92f8843a90e1027de040a636718caaf16faad85aa4bf438cd1e98561070d8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1bf69967363ecc6c20ff48439d47d408884224a1048f6d5ab6a380850aa417f5"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "49a940c5c0a92bfc5cad253fbc54b89eac248d68ac189b29ae9aa190acaf40cb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "43f0a99483181df0dea6db22cd1db9f636f0566d35364bdaf4f4b88edef0207c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3ce7d85e82ffcbde7bf7629dc94412a5920aa15ce5dfcd69926f7092043a1edd"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6bcd41cbadbfdcfb7d361e99f9caf2a02227fe94884a053f75bec9627b80b901"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7531f639aed54c332eef72498dad80097c2fe3146bfc8d0697f8ead37781ab4e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bab0c141234bf8590f102056a582a580788ec403b2b53535e880864758c39236"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9ba5c8e4dffeabc0a201c8849c88f96d6211032c0344aeafc9166fd202699377"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d59f0892345f7bdbaa0ca6543d097d5bc23f56a39b71d0261078a9f1fe8ea45d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5d7173f2670b36ad191559738c3ec60420118324f66114b9946f4c3e98cac113"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ca4dfa41ab0f20f23bf0d2c14d9fbb696aef8f386fc53adb1659e1a07f3b95a4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e2067e3318f32256c80ee2fb23a4a3e7ede7ab8b1fa426dbce58be7256bf7475"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4bc54480a46eb7e6b1caed3bcfda2ab21a77aa50164a31db9d6cedb742ff154e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ceaffc7ab6ce5d6b25ec2dbeea6e93045e61e8df2b72deb7d9a90c0f9015b735"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0a5044846985540dd3509d423567bc17969f8572fdec95e4a24efb740a33bdbc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 197248, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "300b793fdf2f471290d55fdc4c79b266db4731f84a6009c6db6ae920c08d879d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 196880, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "787b0947384cd599b0a2d15733ae768da5da921fe9c5ac05f597607fe1aaa3ea"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197264, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cb84bf85f446c326f10489e62ecc37c24f6e5da4666ec8b078198d36aac2ff2b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c133b2bbc99293fc56fe6bba0072def73f000c90761a2654c19666b4988ce6df"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 197248, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8c59a8fbff859d99f19d233497abbf666cfdb938bb515af7453e24a132144b9e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 196880, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "84b8e74d1d19262ff32c62de56ff185dcfccf6c6d9a1e1da1c580f32614704a7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197264, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a734acec54de61b852a0a014d1c3b9a0de17ff257f2746d0f72de123cc6bcc8d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fca475f0d7f3b831168cd8c16ea8dd14850c4b58cf54829185a33bd56e1f5dd8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f5ee9a8bbfa73a505e01be718419b02d7ab4ad2702c2e8b021cd95e5ee9dbedb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 173360, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "06e39722c472b022426cfa09455322a23b04b59d4d65459d94e42b8f1ccda2eb"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "18522798e2c430f48c704ea8ef9a7060718af19a22d5699d18689106755a89ed"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173376, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "21c6f700e957a0de8ff42bb085dc767c69ea901330963d1eba8bb5e6158f9146"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9cdca41e99526cfc3775a1d1ed52ff3bad4cdac32c183408e8dbbbda9ae7e872"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 173344, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b5e7736bb865f8b6c815aa02f558c346c8078720c970cda1edf9e66fbb85605d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "ca05b068e8ddb3155cafa56edc9185488aebb37594880ac0ad547574ca74a3d8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173360, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "0bc59ffb3cf018e21a4eaca750d2514c5e05bb067c58eb0acc2369c5fe6bd93a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5e8728e3688982d92c59a85a7dea332fe779a9e7bd64ab7d66f9028a91079e15"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e84c4df494b97a949e16dee4ce3dfef7f119d00cd0b9b6bc3758f7c00f0438f9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 173440, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "b2a31516ae353dba79ec827464617c0167e17ba45ffc3efafb8845d90c5d7149"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 173328, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f27577d80277cb3aae075155257b06ed4bad4ba277c3d80f983e58c0c6e5fcfe"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "514c3a2a7213cdd6a99561b21b7bd796126131dc51f00f7d1d05925af37836ff"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "0e8814c5c737fbb7f07e1e9ce38ccd65b515b0f3fdfd6ce9a5c888541c8bb745"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 173456, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "82d2d58a95d9602327bd8c141f75b3ee4dc937ae9917fabdef2addca4c2413b4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173344, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "863dfc622f56077a4a628149f0c168dce8d58f0df118c80b933c915f1129b6b0"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "38e1dfb109e8659c89a6f1594f398808a00d9d53eed39a1abf2977be51c9e0da"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 173360, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "33f0abaf94d994af94e93268dcf60994c2d4301122d63115c741e95053da7d45"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "5c436aa39093502dd942550b91a00b85cbdba29475525ca7e42c95d74e64bba8"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173376, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "168cc741a55370b63235cd10b6578acb88b889740d49f4b49cc2a376e83369dc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5d781e741385c03c795d51c0512fa805057f01fad3d9338d9dfc16f6c2755e37"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 173344, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "64c4f408235d080c8c48a5f524eae98952e2aa002de306159efb994df3c6217c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "8691b77690d140528fb88a3486785a74a55e00feac79f74eae2066755501367b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173360, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "53cee71b9671bf485e3c18b76a1fe0bec38a5dab4f2be0cb29cd351a06c65a89"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "50e1d21dba4d04a64c975edb50eaa2dd80b675a1a25387777b689fae0a54cdea"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5311d4c91f50ab9e3a0591289cd9b48ab874eaf3a15e9c612a2d4c509a91758c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 173440, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "73a8e71700b6e3a2edfea3e93678fa8a66a332ab979af6991277dcc2f1129470"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 173328, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "df0cba4d8a1f1be5c5991ef24882323532845399322b64ed8f56b2832568e70f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "7846dd507c01beab8c17dfe8ffa028e1e55301a1b4b853a457c7dc30a4673a01"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "141ce2f8bf67d1f7db45cfbf571117a4a0ba9c0bf5e64a031e9d1471d62b6643"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 173456, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "a6c647a7f864d277bb5889aaccbc5572cbe14284e5749dd10361f6cb5182e1e9"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173344, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "726bdbc667b198fb593a7656c74fd43d1dc9a0381453f0a2d7082d36b2217dbe"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f02893c58ca71d268f68dd07a61e1004bf4f903e763643c68319413d94aac141"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206128, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "ad9ed8794303e04764458b2e2a7d4d544d72dc706b312bc0e04b1db0b0e1e6f7"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 206208, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "1a410d3222b6b52b541950d78f9ecce8efb102fa09b93985514e810f05fbb80f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 206096, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7608517863fb8b471020ae81fdf43147115c162118a96f226f7860de2d1b2398"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 206224, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "6b3120b85e2678637bb146c378fb8f8be9e5c6c27575018506de7e2373d7bc4b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "40213b32987ee2a170c196d9a803a5785dde736b4221dd7e83c7fdc3d4254bfa"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 211488, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "cafe0b8a814b878361c04d53ad2cfc2ed11686fd577f27f12d9b07eaff9bd901"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 211584, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "51c895888ea7fc94227986c408ab8e180dd8e7b1357a6ce11d86b2d714d2a46f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 211472, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "c467b6d0a5eae1f0a43cbb01ac86fc2120135459c6df1be26095a632bbe2e6d3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 224040, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, false, false, false, "e030fd82869dd05caf10d38f787b7ccb98adade41933dfb0c81a90129a5c2dea"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "14cef9b4abf284fab23e7f40f1c7d663643e14e995f4ff25d043e67246e3341b"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen", 224056, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, false, false, false, "38b1158047c942da7f3225e3d2a902cc14278ece0bb2e1fb2e64fbb17f462e49"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206128, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "8f542a84cca9eef69debb730e2901ba65a0f045e9270140e8f84faa1b993b206"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 224136, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, false, false, false, "7aee7eafa17e19fbb94a4013831c63fa9e1edd169349a11f87e279d07e72f12f"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 206208, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2cd8e097abe88069f8cda1e3ff2d513462919b71c5642855f100deeeeae5ce5e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen", 224024, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, false, false, false, "a741eb2ab17d449c7ccd98fb41b826634a762a09a91c450829e8d2274adcf470"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 206096, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "de50952361abd4ea6148c27fb651ffce1864ab0e18c32eccd65bbdd42eb0324a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen", 224152, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, false, false, false, "991a7128db6767ef32ef5da43856a221e03d21c499d4d0c65b4e54ee009aaf82"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 206224, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "7e776f052e8002a41260c25505b04e497670016dd72e9e8d903033535402fae2"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen", 224040, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, false, false, false, "19eef4d8721a1112c1b4842ba513ccc73d02f098ba061f27bc11a5a5c131633e"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "93c0353e955d2b87cd8b44e70af4365687edeee9de372d85bed88d4746a672db"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 229416, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, false, false, false, "30a5391b2a2d4242c8a64febd37cd857e55abf2210edbb1ae3030ff0be7b925d"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 211488, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "0099a6e9ad900dcf9f9e058656ec00cfe1e5d2d5ea3117d0d0b320f06b7ec154"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 229512, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, false, false, false, "2c0a6463566307b0ab45225db1b15b09c6da5f03f21a273d1c090f88383d93e4"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 211584, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "0cf4f1876dddaaf6eeeff3b911a784d57d078b14793fff5306da7ad2f6bb9696"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen", 229400, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, false, false, false, "132e7c7246b7e4f54cb3fec450670c129ed2093b08b02f7aae665b53f2144c22"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 211472, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "12dc610b5d13613c7ccfd43a23db03e36b5f9a324d8880c4ee664022aa8615ac"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "0174eb4d5f6d67feb22f1158e30f0fcc747149da0c232c9e9ae3f1b4237a51fc"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206128, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "61253847656d105ba8c4505d2246a27c98a2572dbfd29fa4647b88d5cb28a9d3"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 206208, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "38476f51a7fa20946e9c2857539a828a7aecc8c56b104e3ba56340f17f903062"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 206096, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "954ba71184df69ef776954c000b9a2e73df9d867382757f153073f8c38b5aeae"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 206224, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "e036e90bc6cd4927c58734fcc06302103ae79db28bc7e8d6598cf4325356b348"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "6a7cab90b7515e065cf3c57d246194bee8fdeff5d31551c646b5224ae44a2722"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 211488, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "41ab3caf1766f8294257c7b743df41e41ad7db2d35c5144695ced9b47c195f2c"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 211584, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "4b172910c44c6be754095c926b5d1aa26647c1f65211beecdb44cdd181183e2a"}, -{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 211472, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "30795515b2017363c590fb9c2628abad5d69adb8d6774d3bd9411005d1fad8cf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "05b8f0f4623b4c7a9f1f762b2ca417533a98d5ddcddd26de511e23b93219fd87"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "04bb7dd864e85da193f16942bd0fb2dc1dc839c1a45b4c47e857e9a5e0f81531"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2e9f7b881727d6f72478d33105c608991f9ee0ade928bbb8fc928680934470b3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0adbfaea7c934a2ce943188186e78db25f2d5eb68ce4c54b6dfcf3736f8e1243"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d20b6d7c9a53c32e8b378c19d230c34b24e312d4818cfe8797ce1c4f0a144ded"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "38aafa3b6329ad6daaae9cef906f4407523d4325f0d58b868f1903afd6dc05ea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "9c25f51e2c8162be9e43f4ffc0b29f9733ee09dcd54064ecd178f5dfa3c3c9a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "ff4655130cb316805f8ac41fd37845fbb45e4b32f62b7d7f9ebb779a890aea02"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8770536638b0c23fc4ddfb3c66dde03ee8bd3cba10131edb71552a492f469027"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "299720f247e3b163df65ae48227e9cde6d47ec6256b83f2f7dc4797df0e75f4b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "1b374d04a1be87afdf5b58111927f3c67be62609899964d92f9922063bd765d6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "168d2e4b4889c2bb44ba973fcfe1f33449825acb69209321b89c5668664959f2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "947ed32b52a6be8e779e3817722976a917d6d891d979cbbd29bb66391c66d6ce"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "690649405eaabc4f0bcea04bbb250f06003e45ed08e61b115b5817f5cd14ab4e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f3764c47bd306d75b04a730c135c1fb3f937e3099c9138f7e2b3b5fd282d77f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "965932615935f295add7e98d0ada061d941bc860940d344ff3af0569bfb5ba31"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f2f036527c5719b30dc9024d0c6e758e95bfcd032b387304f5c203f46ae2f0ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "22af2d0fef273860bec73f9ae3ae6993fd1cf90922ad5185a788837c892fea60"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8e3a120cdde478ca2270b72a7b06493ff36b081943afe67163ab81e6594bf9a4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7533d9cfab9b6c820c965ae4c5ae8c60ac3822d5dcf4f59d30f523abe8f3b445"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e7745a93b25412c5f4db2323a93a436510477ae12c86e96f9a8d2875550b85a9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "09a57e24e11485043717b1aeebf0d9e47db846bd93aa826012166845afe13467"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fba8c47e275872a16e4680d79e9b232a2d0ce41ee68169a16cbd748695d87efd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f90130595158146ccf1747893725cab47246489d25bf7d33b969ea50992f0785"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "aed358a02c33fb009f97f15b2db8958c5ddacd42986354c69abd806991547b36"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a2bcd583eadb37c38d8e2b16ebe804f3cf93a857f91556fcf848fb5214328d6f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c063adc4e06cab40293a1cc8aee15d357a4303e0de61976902a1c47c157f6806"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8466496d70f6d3068afd0e9c68eed7a6a566035ea421d82f4a4a857ec20d835d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7d6c03988ebf50d2dd9631c8198ac1151f0a6367739a029b160bbd69a2274a88"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8be57d092e79d244081ffa68bd052545b8fb00a753e6cc04b631e526008d9b6a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7c99c708ec31420469ef6c42597c5676e6b316f189e811e503b7a5c3d5e491e4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b85e9174e6082afa711fee3fb63a9ee500da8830892b08f55a7002b6ee71fd9f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "53331cc0ed31e90bb58353eb625109d7071279313fd1f1ab17c2cca0c556d3ea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4601113f1c02b0e91ad1987e190aacf59710faef29f47b10dec479180f6869dc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5ca826cda653a1b3e537ac9e54f8938122a5f62279dc03237bff54cbdb151d54"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5d60ac1db8b5562530959f723f3a9b745c039b95f46877d39f2fd08879f00de0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f0deaf3738e697a70dfeff8466985d367238f2b7a398281fb53bc52d304da844"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "86fbb43a8a1d2c9175984d714cc35c1a6ddeaa8ff2c5abc369ca1cb7074098e6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f5a4980b8bddc96193fb41532a15eafd292d4aa1fa78a08d704861afb9cd7235"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "70b33e8117dc304a556606ea8ee0ceb0e8ff591d38b1576e628de2e6283fc96d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a38bc180b5d3fcf2a9998b9e9a121f0342effde9bd0f12b1ab35d53954369088"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1aa9afd1a7317e49c370f6150da56d30ae6cdd08d86f355b90b77a0da978d390"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f5ce24557e80dd4c72974aa03aa0ffa79ee880a229c9f230d843e8dcac98ddbe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3de8b4d185c7a5f8bfdb582b32280a6f6c38635cc78e8c60ca27fdebdf66096e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a665593e6853bb6f9bb1f3023c7faa7155398cf43b53fea9f49590be3daeaa84"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b4ef2a611009b3bf613f48fb8265f481a15dc59cf377ac338f3750d28fef2bee"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d483c80512d76307842eee3a62f4c670b473926aefcd4ad99496f1c152fdbc88"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "2b4e6186737ff366fec3e99991f462229717c31881a7795b81efa4d472bbb768"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a7c17a2b6181cd44c75523afb5dbaab269077369beca13b1ca5749e1f2b70c22"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "27e0f7516e5c200610a932673fcb87a5813f049248ecd570a983e78ade8a80f3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "493b5b6f761265c78cadbc940df1fb73c0cc8fb9d9ea7bf249e2f9008c1ed121"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "bc4a218487ad652a433160e3ed29e007a69e5a4910a07167ae37d98702aeb821"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a8d1ad244706bf15398c862ce02a8ca235ac3cc5540b5db4a39b7cb7918c5b5b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9d2e77701741138962a639c7b3dfbfc7c51624dd14be1c1d3f6ae4ce5f2e027f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5ff00e8c19190d691067fc383d6d305c31a7d14b6f29f1ae74e97de809c761f1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7f00c12e1adaf5423773b215ce9d0f35e04de1aeb41bc761dbe4e4329f44aec7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "41a24c5d76372043b7bb2aa297d9c531769ebe8a30b792f935a3ddfeb458a545"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "303a6666daf3d086ce2db824c468a4add33b191ab8b306669691a5424c42bdbe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5e11fd2dca1b7833b10ee1a1b4a46d5b1d3e3578836b3d626af422aa1ce93740"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e5b60116470a8b7316cc806318117d18748cd15b243a847dca9897bf1cbb85bc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f082ad2d720c240ba347e91da7cf7772852e949fed372343e883a1290c99ad63"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "de6f1ceb3b86c54d65299adef3344fa11bc696c48f12869c5276fd632b39648f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7843b771a0b73c69ab74cb1a9d027469a5406a06538339f04511aa1a86b501db"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "239ed4bfcf1469dd4902ab4f9de32105b67e8116b3f9fd8ffdc0efb522d93b38"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fe0a28b772b72f0058f146800e97ab80a6d63153c36a0efa20264217e1d7a23b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d6aba5c5d46c6f72cdd99ce273d827df6390cab2c23d38526f9f23e699205c0b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "924bcad6e0c513a7086b5679b10d9f3899d2acb916998eb9dfef26356638bcd1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "53b484ee29e70b733b85e63000586ae66bd0f740ff683bcb161c35cc69d271e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5bf2a1040568e0361de1de83396e4c2e163f0ec22a3aeb2af6f3ce7db40d966f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "05da2c0479f97675a26fbdb0213a2b7202cf393046365537d4e568d06ab68949"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9c0884038d9d7e2861b645435dc5397d5f88368e0490a1088dc8bef0d52a0037"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "fee342fe8963a2ae6b010dfa82b1351104f22b6b31590de292ae3e76a3a4c180"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f924616a332a511e41b14522bedfaf4f5a52f3ae452f51d46ffbf2ec4c9b0a0e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "483551a113c45360ab40630eec3801702deecfb378b468e1b54640fc7c8cc789"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "05fee7f5eab676f0da407b238dbbe5b65a76d2c7554a9b50b887bd3c091517bb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "190942e5b54eb818ad0c11c98155072f9affb45a77ed8bfaa3993c24d943a460"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0dce7f028741faaef8b292db90bb45cc68dfbcfcf3f5a33957b0942b9f5c67a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "63961a7dff248d2322a99bdb7d7f93a6202b2a27f78b2c5804fd964dc42e2ad4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1789f301e844e82365889ee7e81e8b72a1c0b3d8933f7079882a67f8c57d2cda"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4d48e55b4ea3a0ae588c383f2c144f1c4ce9b727ba98960dad795aed8d213643"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "91417fba211c085909fc913a52160b88cf7ea9ca6923663301047a66db1bf84e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "41a60ef9d1430d0f8fcc0530b5271e703d0e79eea3356dc66156047cd082f857"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8ba805e8e059c4a277765d30ae9b9edf192b7dcd4b28de0225b0a5774d2e07fc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "da2b0a53f3a89ccc1ab214bb42f6e5cdaa4f0feb2a4f59514d516f0cc5626919"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "ac33718a363c131c8eb770ca46946a486fd772344b46a5d8218afd480265c282"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "753702a5688022ec9d9c89494a6a49bb7b537616773fd7cfc6a9a49641188b31"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "0e13b250ea9edf93aeffb448ad11f15fc16d9feb01a27ef11eef3a442f29bcec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "895c2940932de7896a98cdecdec220ba10ecea9d5d8e8667a110d96db4a88780"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5199d4fdac45506b4da94c10f1452b0900097066d644650bb1095ee25cbc3eb1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6003795b1b8db19a6ee04f0d76c2f987bd1e92f8624850fb6d5d078092e79e40"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "7b2b9ac796108f4b6d5a446c2b6b4182569cee398813882323883c1bd0d429e1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f96e6dd10f1ad4fb612c72fec3d47b4f84c1db8eec8c3c846c746b6151479edb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c4a438aa8cd1b00fb2366fe0b004b296d82e42333f096b49b3ca8493420eb674"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "5cf4f42e2a650b9b558f7b84acf2f5402a8948fe9d1c71ca4446840eb65265b9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "ac1ecf1467bbc22fedde4f6949d30da49d84bba874730b826aee87b411139730"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "da5cf1a0f62d5dcdcbb9a336a5a4b6b97d29b69ce0e2f9a6b401fffeee87f02c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9f3d08f1439a5db1ef9c316ed59b2a1d1b75338a718c64533928a774561fd819"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "007f64df439e3558af1d21f0966a6e6555d914d58332c9e6e7c57d0e67fd3f72"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "925f98f8a545eaa7770d2076fe9cbeafffb5538359ad87dbc0ac55087536dfe5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a0bf3a670fb2ecdcebf800a02f8f6cde7296f8a45b46c4423dceaa26255e2055"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7375f8ed2538b47b58f5e8fbf5d386a31ba0960a048c970ab0538a6e22c07609"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4d68c5f64d4f3bc518d85e46558745a9720c96e550435e0e5aa6c2ac36c31d28"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9650048af22be66c384e78a52b1917bf8a43b06e0d16f7343bb3af22fa7d4a60"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4566fdc4addea63713c5bc40a9fff86c4a75a7caa5ee7f9d901d940bae159b4e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d2e2d7634560938ab6f3a04d8fb8275749ed0ebe61fde6b8740112642ced52d3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9a9488c49a3648e2b05978b05bfeb66fcfdce90e78fce76125f7730ea3a59c0f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fabb519c67f83d56d7c2aca271b63e6a7a71a7598f7f2b7e448854e42cc41f2a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f719221e86a18d7a7b0de0c82441d818453f14b90f4d48d744cd78c86570a07c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7171de90eed14dc8910bf0bfffbacb2b3907a12cf3f1979e6145424df2b27e8f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "bcdb60daa77d63b5ede693807144957cbf8f16cb3acd7b02946fad00618c3a19"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2d1e3e0fa542ae2a4521328621ca8e966d544cefd09f7cb4c9a98d7c125a2b14"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "37f6ecb03847cf797c4f78a49bd2c9fcf833b4f2562bccef6725e7ddf2203e3c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "42e87d8938a6bb04aa0bd12d1fd482e922a60fab30600b04674d5eca27c20017"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7964de04c0be1366c765a5283c58175118901b09939f669443e2b4b99034fb43"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "40cee0fe4c0e54f004b9090a465a8d64227054b1b2250aaea05405f97cec2539"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ea60cc1c592dac662d54d78b5e481b24381fff581a878e7e484a99e53880a70e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e52bf0af78ca6ed85ea1fae9dadad8487b8af2e41fc995c32de9dcd5611d19bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "441871f204724cfc71ccdea71f6353522efb421163c95e7b35586750bfeace1f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8e675a307c65bc9aab4d6e0d14a96401f7df6a37c0d9597addcae15dd485bdd8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "28f32b63d0e99ee1110c7588d8b28a9243b274e4f1361256df9940ad3e38cd90"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a64a484eb20dd392d66b3dc5b977133434c6331949cc2bb78790491878b98aa7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "71542f18538c9f9e1b491f46cc91a8beafc7edf3f396dfcc52e9ce6734882a50"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ed48530246a2c2be2e019a30dba233d7e8c1dc76c6b7c228ca3d720398113d99"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4d333f627917d3a9cf4eb2f072120ed977f3c1b10832bc47fb543d542607ef10"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "650373fbbd8d9da8c30fc286dce1dd17837811e9c19a4861024ac9c2b929cbe4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6d5cef90fecb079d8b9ee94f8720841a26bc7c29eec261eeea3853bdea97dbfb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "68616227109d043e2c976b418a73e21f4d540a55b04698073d7b4017585cd031"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8a223bc5c100017b14f955eb1afd6fc4b269f320f6fc3204fa4e483e37bcf34c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3de2dccddb1686f756db18f435a09359efb827dd1d144643045ce63ca4be3660"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "64d0eb5d0fcfca284575ae9e5ef6b8770ca2206e71e6b3f5fa81aef8574e11c0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9b3c54547fde6745282c046be75d1e8444032b14631c04e33489a6790db0c081"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e757ed1553f2a62e553d125525f1f8e1e486977878a43627507301929ad055b2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7d283a4b2160511993378ef66af0a76d986ebed7ab8c4a0d13945ad9ad01ad49"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3b0de86b50230ce5f4cd5e050895b716f9c131163ead90342abc1550457d0bfa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9c86f2b133e33efc090d4eb0440c26bf9d4f59b3791128b456737f172a81536b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4360856706c27c60908af3e26c80aaa5ae9f9502cb38c45975b784c09f88c282"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8824f4829c7f58c06512c4c4df6c02e1c8dc01e4fae4f70c520f6e5491007ae6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "fdad575a3b6b58bbb00dae82d3410031c6ce25b35626bbd2c5781f9e1a8f6c73"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dc5217ff1804b9f37309a0b5bc5a8636f973d002cd98da59a400174f1f0a8559"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "82d838c46e27247a17ec3e7e6210d2d0ec91e19705d92264bf8ce719a9631e17"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1b882bf0fc46abf0e5a7c1a03b574d0496cfba917e915f68475c752827466c41"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2e9679f264ebfd76e5a4929c83ebfbd79afb885aeb87d66006ebf80e23bf3b0b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4fdf477d0e9a7e88c868c9e7242d06c152b201104cb79e02f601a6963c3183c5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "62d9279dbeb5f132187e99e41b49509f828abc2fc9e1434dedf4c200155e5510"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "879c6fdb0b7c68f1a8dd65929b986d2872fc13021591cd888c4d8f204a873e2a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4bebc778f6eef2a7ab0b855ee6b7f12b563df0d7c357f1993bddc5da9f929abb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b2b30a5b3632bd057e12d520a3bb08c269c774a3a8bd91a63cb6e25d31442406"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5f074df02e5fb8759acc43fe4d67cab5b2aadc80b94de36f450c1916682910d7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "cb73bf917a78860d5028047a978febfe2ecd4d2e0ae12155d22f5845b3366d2c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3d994ea86d965f829f2c762611c9819380b05b452347f45ea888793addcb1c9c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9962fe960e9040e25eb0b47c7d57d079ced9feb8c0574031f1dd9b260a3f5c49"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a7b6bc945dc0d31cb8b195c7457d3982dde1300574393f27acf283b761c07e1c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "218abada607945267e545bd3e609a92a6b9324199e747725663021310e7b1763"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d0911a1ec5ed5666d6b26fc3d08c4c9317eeab17eae89b0f673d4c692096db87"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9a588f481f97d2d546faf22510e46ee0d2346a2d869097729ca616c51d961f80"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a59bb4af9f079878c6e4cf41eb8a60f90aa3e91769d7ecdab0426d2278841457"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "38bd5a88187664be9a003225631e1e90373a24a6baa834b29f247e35ee41ce9b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "54ae66f099c90f8e1d4d0f6bf36d66b7320cd025d964ed423fd460adab7179e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a82fc8f6903df06d9aff0116f133e1caf054543f7a66303b520a6feabf929fc9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d604553e32110b9a7a8605f2df83d0f9a76612b22546c59dd2bcbe22cabc71df"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fd9d9bc346ff4c6570014fd7a3bac39fca1d1b26e88ef8368c9e56dc6129333e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 209176, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "a8d4126337b67e6c85f3577bf765d13f0c1502d8235d4c66bcfcd68863f1d109"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 214040, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "de6c5b090936b152ae31f6fafeac6f41fa1901fd365824d8c7228f60c3c87e54"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 198552, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "a83fbd5fdcd908df507225391f12de4e68be0d3af2044726def91b4798e9fac2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "3a7d205a760c1c6cad0e230b0a7a6e9e1d9ce5f6e8ca17b94ca4c90898a0d799"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "a21444ecbcda95fb3354363e4abb5bf0206b3c2eaedc1ca0fd84a512c8007d4b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "d9afe3d29fc1c7bfeb55dc4e4537695ad0c9f6ab81c954499caa8ec5e967a6c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "b9c731efb60b79ad75fbeb35100a61f1474b427af74cde13bee1d57e1a351c37"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 179568, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "567320f500af5c8b18c86dacb7ced3e8d2a40dc140f25a68fd58c54a50e4067d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "c26858e7d99dbe3d0f35beb87f7c07e9a8e18941ed3cf84e88036d6a316b7dcf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 204912, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "4597de39dbb5b2aadebd7464dcb5137505197fb9a0978591b1880faeb747ef0f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "bb0c752dde67fcef94404e8f94580e7efd65b1382ccd8b0fc3f5754a937f5957"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203376, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "222fbaecff224e5b7db8b32973947a235ede2a89daff5ba5f1288bb7da0640e0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "9b2a651a0c96d5992fe5f1b8887b4f6bb0ff681e198c66d0dcf6f8c6b8b819e0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 166896, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "28068c210ae640a7c7f685103a1ead0800cbd91672a1637728e8bd984ca117c4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "b163d83722b20221590d15defa3a82fa6e288b659d3fe2ceeb509279b1501512"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1f82d1a12a537a967a4704cef4000e46e85b0aaf1d130dc77fe1c6a96aeda490"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9ed85cb91dfa2dce742a94ca117ad6dd4c8cb73a003d46c298aaa1aa2193dfeb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "839028474b5cda7d063702c99e4b77d416d6a03a836d524b1d05b00d50074ed4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2d9845706a344834f615ee757d5bdba442b2d0ea812545a09a584abf064dc1fb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9838ee92d0b0d28eea095e6807064464be5d0da0da7f5aa40d0d1f8348b2339c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ff1798c5b590cb995699021e484d32a54e9a93ba9d4a3b83e881687b4e6e7729"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2ec094a45f6a1c8c01b5974b9c6b1702b9b28e90de0fe7e32fc90f3c6c744eec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4bc27048903fb6fcae525333c9da683b0d83de20128526a3871e82132ea0daa6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0bd80fe4ee255e523057bbfd326a31666a08977cade9cac788876ed5763f945b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ddfa4fc0d0a84f9cea2b7360fd57d3cc02f612149313f115d49a0851aa3d5a89"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f52f71a9502ad4ebde295ffd5dc0659c4871e51c3465e7a87b5f182ba50ff003"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "52adf1161594658ec6d0bbae2a7a8d6088e332b3b036d5d72d97276c33c8e987"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dc7aca46e1c75e93dcc92aac71e87daf28e15abcbb1d7e8f8dca5749d1b0d402"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "fc915493e708897490fdff728f689aae6a070c39387cf2e6c3d5347a359a7ee2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d74d6fccdd2e5f9ef63f8b48ffbf626464bc6db9a7d8f616b967897f5bf6cdf3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5afea040f6bc3a52a5207ed3cd230f616d2c08f6687f2308eafdfbc2d8c50585"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b74cea83d3bae0de1563ea43d21079e7cd759b281ca13eeb923d9f3974c55567"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a320ab7a90391384126fdf1338e07cacf4591521ba5642ff7f3fa97b41c4927e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "87f4e98816052aca6c86f42cad3d6e2e1c00da5eb96ae24f6a0afc2e76a3f28e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4c65dcac6e28e13e3470799d0d9b85d7595ce9c37a08f536cbb74ecfa3eb5633"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f135b1f69fc7dfb6635099b78adb6c70c3f9c80bb61c1b3cc0d2605665b4181d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "51a17752d99b2d6506eb6cd9b98eb3caf4d6d0e3bf0a5e2b94f24fb11f8e34a9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5df5b4bb5eb5bab14604a6a9c23acce5905e2270a9db8697339df0392c1e36f8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "bd3b5623f8ece693918f1bb1ff238ab5f8abb7ab0fe39c0718daf75aece28372"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 208664, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "733e2a00f5846b1b146f29c0cede01b054e5bbe8653180b8df7f72b178d8e0b7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 213528, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "8db3544f9359b2c7496b3cbc67a1586f51ae54bd30d1c1da13a2b7322ac38b0d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 198040, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "5ce2c9e4a1bfc839c0c7f13862b5b36672cdb2a0c7ed6f1d01f734421523db0b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 188456, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, false, false, false, "de92ed323b6b334b7112390e4d38a9c4185b050c05c66a22e14c373d7292a95f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "b8b3967bb6f1ca7a9e449f894818e10bfe03afb00b829f22928ab06930901d5f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "8feeffff79b4993ed7ba30083e4df70b0d8797a2ba0556a418505ff501ea94b6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "6220305ffdb8eee14c5b61c200b3c73a310c44eb6dbb813c101177ec6f4e41a4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "13845d88092c8c2a1463d0a6d4a1107aedb4077334962ff13e31844a8d4a7dbe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 179568, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "7517d31a7b27d64e6a74b5236ee991efa95df6c37089299a4a1b1bd024fdf5cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "d707353e33b6c7c42cdb610da0c5ae4bd3fe4427c548dba2d6dca6d751aaa7b9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 204912, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "a3b078160de4ce340073729074eecd6dcb035c2c47bf6e15504b54bb94fd9fe7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "d62e629d79eec08b7b3423f861490803531701d6cb50de2e5a974d38fe9a8a82"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 188536, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, false, false, false, "d177840fc9c1db9509761e70f9958a02427534fb526d30634fc9be024ec7d661"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203376, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "fb4790a72e04e20b5cc53f8b7d73db584cbad656d29bb7d8df4dd1c60de857bb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen", 188440, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, false, false, false, "3e1cb8e3728550f0a573ae807e1ec7339fc89a8f67349138e2222227a0cf9b1b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "bbbfc136072b4b53181210f3135eaed376dacd7103a4d33bd8a4b0e75eae17dc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 166896, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "c63ca9ce5c7b5326b73816d7c60938fed622fe7fe25118a6bc8c532b75643b5f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "3d023d36bb1b828702a5d2eb7d203049ae8ff9d7960a2ff4597112dadb9c1ec5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6f52a113f20e5f0f07b085047ea8b9aec4601f44fd6fd7b37a23067cb3ab370b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4ac2a625afb36702a384939dc7f00a56c0ad96798a28bf9a4acfa78918fe52b6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8d9fd662ccebf04bf963d79b2f1ea5ce68582a8ee48befd27f5700be28d7a1f4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2c37ae95a774a6e9659a4875f96cb842436e41a622abd5b4b5d046029e94b65b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a6964abb75557d9693b05a757c1b024bf864f99fb804bdecdf6ce1ffd52dac00"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "adaa1af084be9e0424ce8a0682f46efc627a62e09e810fff455bcfa7966452c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3d3c5b3d03402c074219d03301d626b1f93b1ed7ebbfdd37a9d78d71502a9aed"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "743ff387fa876094cd131556d032aa8a5df034dfe9173abe8516c7eb9e92daab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "131627610b81b0d763d8729f011a38c75aa5a8192c456001a1942b80f7aebf53"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4be0127aaa33cd66f7675d6fdfeba8b2e47b8af519f4cd6c7e7d38e3a2edc8ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e53cfe1b095673c3a392cde992c4a81b49acc5c52e2db485d86817261c7e2745"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "cadd5d6000e745c5b81040cc5f4bc56525645ec699fda84cd9057351b1f41079"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3c8119683a7cf41aaa93543ce7d6e84e010dcbd3cc8a606e84e48c3bc208c2b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "686c52c3d90d94377582838caee5950c11a2be03cf87d4d0451ea723d188e0c9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "268729ccaa79651debc7d4bc062d8d503233d595c856b5ce543b47d09c5b6ef5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3459b1c9fb6806328f7d264832711743658e6f841e79db82966538f782e29ef1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 208408, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "b6dc0116488c192357f89d095513d523aca3aa7dc21043482e13304a64ed9152"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 197784, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "147691e6afcf98c1b1fb9bfe239aaa06a949b8a293f0bd502ba96d51d3786727"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "edc0d726c954057cc0225b6c0e227420adb8d9726a95e0bd440e8fe5fe7a9391"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "2edf2cf0fd6004c7ca49484a950da2074520a1af50629315c202bc5f24f57b44"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "845475ae75a85b9d80b66a338dc05bde69fa7430bb01b43667270c0362df0866"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "79c6d42b13b395a9681b7255c12cd839d7d0d07e811e41634761007fd73ac3b5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 179568, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "9529f2164ca56702844cb93adb784d5fb32c21d0d9f5b89ca0a64fa093d22b02"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "e0c41b5de2f993340f8bc8fbc0f40a08785737856bffddb5a35e2dd5ac4930b1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 204912, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "faac1554103e5125bfab3f81ac958413384d4b262ce8a06546a721e1c9fc9e18"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "e48e1fd141515a1f0cf77429944d792ab6f0c060625d1047ca9f3f5c4f95e323"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203376, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "737974e40a554e088ba46fb0cf3f06d83c785ec237aa4cf66211fd89e866f054"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "a4aea110aca414f7aa910a0d5df302a916b8a3d627701b8b54698d759665d8da"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 166896, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "fbd063d9d97a85119654a18396513b5112548dcd277631965527c40295ce9eb1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, false, false, false, "b80fda6863c3b4bff040e1ac386d761ef14e42a041b5660c0dca2cf55d0857b1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b27bfc247671a0f5e33a7d85df6888095b8bb3511a0055d5efd2ecc155ce8b20"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7e93198577a825ef25c71dcb69af353a1919cbf8bae8551e12cf7d3bca7e272c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "97d32d0cdf4cacb8f0e573ec0e978d305913fc490ef1781fff44beaf87df7b71"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fdabc447773af2116137b00683253ed5c3124af17f190547f1817be9ddcaccb3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "276ee9972c0704a7049a51e8e8cdc40de37bfeeb32d1e710c7e7b6acee3e598b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "88f628213dd2200120f0b6f544c545624b0d083c8d0680c5504a24d38f0408d1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "51ad48e187d4731b06d732d2dcc8b7ff0a3e527f14f1b7a6993656c48fb286ff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "0f7f28e0cdc65fbbe12e99c2a589ab12f12885a96083236840f89858b332c952"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "7f7f441b3390c8581e4c99387ea16b8f67b3c5c46274549970586764cc464ef7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "39be861106c5485dd180c142bf56cf87148dc45b79739e7711b97d91fc5bb686"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "96ef9d75f88fa1b89e5544a57c42ea639de567dfe91f788f7eba74ac9aa1e054"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8d74b4cc8d04872df61ce3c8bcfb4eeec1d1250b14c44ce72ad82fdea2be5cdf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "749d6905611a782961651a1cbdfb60673027c511d886070554bcf334dffd37b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "80af07e86236949dc410d30073ae52e144b24c0a9c7c1ada885907aee550e10a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3e063230d985613d325f04a2d065e776cdc87dbd42b3454737482e96187824b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "fc81c75aa11624f1e4fad5dc2c73c1fb6f6503b9127f224319ad2a5750b32a14"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "eddbac469507a4bd1338a75b8efb648fac2bb20eb0a3f4e3bf68cb06546f4ac5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8dd87c715d343ca5dedabd7c1c3aa0f0ad87047118f002f320e17d09729ed3b5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f1f1b705218717055fe5950a0e33801371e5a58e28d6c4b162ca8b2ad2e64b8b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c7e75e784d7c412223a8d7b8c312b75a3870e9e42a1081ff56e03df28e2bab8c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "39391d290a58fbdd3afb9ba573b310df7c05bb576dcd2b1c7ec40964643e9174"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bef70485e0ab610ae4d30a88dc9c7f0026d0cf727ff83a0d7c5ab2c9567cd933"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bdb41c93877911f6094692a3a88e3f87d37183856ac9727113e7a84949436040"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3f760c8c10523edcb427b9285d9c58a8059cdb43edb3c804df98b5d15658bd78"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4cb8fb7f0e9d0f5e84f01133223e031655adeb930899e55f2560194528288e85"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "08577dd9fe2bfa5e7e8931ec6eda9c1b0dacf4d8402a19f543a83e23f2c74e54"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "873d0f2200e76dcc37a1546efbe39f59983ffa5c8f744a209848846693e84764"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "19755124e59a0edd2de810e0533fd1e4897b7cfa4fabb6bea3cc96e921c011a9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d5b4828bf9097ab9eeb298b92fe42f8a284d7a096564bbb91e598d13f3843ef6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dedc9eb698aba1aafe5ef0bdec76c94541e46974e9a2dd4a18639363d58fccce"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0ef2d405796105d06f56222ca5699e544fe17c205e533d96f24f1b280228b580"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "735f8697bd9ebe34b7f061de259afa14ffd7b8dd395bdfabf3b32dd202a99f6e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "76a668aee4e4e6e6ddb0111cb1f329e231a8209eb7a453bdd11dacd6609e8b25"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2b7aa6661b25f7e2f4ae653ef368a3536bcf8e5b3cdef7e97724fb9c53c233df"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d084bfec8d62070b931f237f14dff549263fb1ac39844bbbc2a776bd2355c2e6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "086120eb448375ffb66b6009ed5696bb28012472fc5f97bc048811523c1c174b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e8187a16cd30001420a765255bac2011a82222b00359b688ce77a296da9f0dbc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "18208017056ca4f72a27eb3f8aeb587c3b3c283b59d6e8d98043389743dfddf2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0d3f8854b1fc67a8320be83a8cd57f210909f5439f74b3a12e37f92836f37508"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4a3c0d8e833c95c691966280ac492844b6f7d39bc3504cd2474f22bf7fd0f446"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7d228cf03e5a6e0fb5a642390c9a359c0c0fb767b02735877d099f5b31441a22"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3ce7ca81debe4863938631dd0c85c53e79f7832a0e2f3ec2f34e774b3c64fe10"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "952ffb22f490c75262549cf22ba25c8c8b63f6450d79144b7e13681c83f2dbcf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d241f9c4be5d98b24f3e87796cd1852f071320d41880ce0e77bd23e02c4034fe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "451a5fede25e8556b6fd59017c18c80f3a598377084be65dc076f1a93c490cfc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "26d990bc14ba3104d7b484f9f85a9e66272965c26f8afa8631c70a6bd1e89ad5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "72ab51134ad847c2bb0af13eb03b67c761b8f10736e9ceadf4d8270215b0ef7b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ca0f3d19057c3156a2224903cf446854b47c3c5d604e0e5fc122f861aee278fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "de0c08a3091b5a0889f0fb036075650c81570d832f9ae3c66cc2191138b62e2d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "23c9c9de5a6dc8c0c0e53d481939e38a18718278b652a543392801de0b0dcdfb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "02a359f4bbe4b8f29da673ec5d5560f6e1e134f8972d97cab05a2deda03ca345"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "bf415a63276381f16e728a7c731f94dafd7acc257e89bb6c46e5d94fa0c5f9a3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ef314aaaab7d62a8f62e13f551636dd9777ff9a984f7916ba8be1e786d9e682a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f038709b76b5f50aedb10e33860fde8b31c0d3fec694247a4c21f2b6d24277e0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "96c2e88dbe4e28bc63591b42721294e79cff3e0b53403c7181f9513f0e132c9e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ec4189e11d0943a57f8ff9befa6aec37dc515f06b86386a81ecb1836addc966b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a63f4111a443ef5a2561e0cc423fab393c4edfce020f82f54e5ddaf03b8ab2ca"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "40bd45c44cc0e53aa3fe73b712a82fe54fd9910ac70816f58c3a24c7546f0c2d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6536f6b1186c8e5f459c69889b37540fdb8692b91d0da084035f5757a8b4b0fe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ce85ca11f7bc6bd1f4067a6caa172a949d7894429fd76feb51be7da1921ee371"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "99ca1f74433cf56ebf3d43a1c05537b02ae5be848199609e39cdb677826bd0c9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "afe01af44be448da61cb5af2fa27d4b893e7a1093ae056003501cd5f7385b606"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "33d04a6c3ba7fc47acdfc4794f2b05bd503a81911c39ee4966b51f091260aa15"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "629339fab7f1ad960993db6d754aca59f6c29645650c48405a3ada9e874df94f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6ff9fa101bd309f6eed3892cf9fd185c12c64064923f752bb19be3658a2b0f73"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8787b9597b8eb26bbcba87df58fd53cf7f12270c38addcb41f93e37c60887de1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "10d8ce2d3f84d47f83dcfcbd90482ef9a12b5fcda5b137f45ae9d4a530fa93d7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "873bd47627b799705169a664fabf3d70dba50cd95f6a250ca1c55fe5997155aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7933c0973a185a064e77653b80854190e1c59a25983bbecfa02c60ad96ba694c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4238e5470e45a43ec6f4ab0f8ce9ad6c88abc381147daf213641c5006f9f9528"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ecd2c217c4905130bfc6468daccdf3d0827073da491a3b4c7bb5632ea1dd6302"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "de9ff4d306f14e1fc9b462f0b84e29e92d269c20edde0e823845e17ea9a632ff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3d64efb40648fb6c64777476eb97ce65918e9d43bd3357f7b71393d9ca416051"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d971e762b9d7f42d545c63b067f7f757c24b11fe40d476adf630028b747f64ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "fb2b473c084e2eec34e68b07ad9270c5eb5ed9f6cece555903091fb423fec6d3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "036bbcb8ac8413caa27543f7dda66ca156bc0151dd52fc006dce786f9b715594"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "884be65877b4863c1f04d6fed693c4c72f4b7beace3ddf09f36436f713dc15a1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "20f61e50d9315bf3bfe24897539f73feeab2d796529638a1392ccfde939f849b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "defeb1e40269e12d134f6220cf516048c4e7b9718a5792179738f4b99e60dc26"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "abbaa44e627fc764b59cd345812e46ac49e57816e722c00832aad19f74fd69bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "86ac9f97c61118d220f2d5cb0959d80eaab177301a406caa84da3f6359391ff1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "3b78e20f2efcfa944f87621363b9ac5492e0ebd5c4bb848d47e5211e3d6de588"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "bf46d0707c966c98766f1e8ea3d21d31bfa7f6547eee2bdb63698f0e41af74d8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "32859c0abfefddcaf09b276281099188aa1b1f0a5494d46c17a5e1131f32cb71"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "32094154396c84e63e7a8b08543441367049a69ca5f4ef7ddc866ba2b9fd0a04"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "1d1b62b42e666f13df6180d8307516684b6c230ecd746b72b96baf60a309f6d0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2e99ad1c0c0229fe7aed471a1166edb25df2e4c2f9719e509ba9f32705f3dc7b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "5c03a4bed123aee439683349462a5dd1a144b8508b47d2f661dbbbd5cab3554e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "93438a5a3022805c7c2ea9c11631c97f785d465c2f719d37840a76e8f0e127de"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "755c3ae3deb7729dd0fdefc8b311ae6fe88cf2b1bf5bdf831c62f71faf064c89"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8270a9f68faf20618931c16c41c4f7bcb49f71847c5308b7a8a6a91af3b999aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b7425fa4357e238c4accd75e055a51332e371e48ffd1cd5914d6f65322f1c85b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "737d91f6ea398db8b982d35232116f00633320d0c7c10173a434788e38eb834f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "66d73c75c26a74def5bfc6d12a22b49aaf0e1cff52a3094785cfedb6f74cc6fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d5d845602d525c24611deeb33f8586e916a0523ce9ff4a3e38a834920d4fa671"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2f83441723f31169d486149bb4db7efd9a37aa0b2028d73b0ef57dbdc6485bb6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 128, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk128HV64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d97cd13b763cf28d0d97885b8bfd951f18571cf7df138cc3d1e8e16224419de7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 115328, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fea01b2429fd64a9f36124114e87c136f7c36816e6f856b676320e8a6513d2c6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 114960, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2f88b236009e7f5c2264a97e14a1c6371d094c5eab511a730f1b0d0b84753952"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 115344, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "10f15104bf8bc938ff66a3441bdbb51b84845263cfbb3a27585358ec80bc8d1d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 114976, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bc1df23f5701a7ae3ca5c62ca12f485f035d79f43c171c04e02c3e9c0486e2e2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 115328, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ad2f818ec381ecba4613356e85ea1f041507cf12d23ca165ac7fff0722114a8e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 114960, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "58c122794122f97a7d1137f4642c19478fff49cd6f84a2d254b1cfd71f276af2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 115344, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ba5d39da6e07431a93f6dde5e36defc641d95eb4b5649a7f3cc3d39243725402"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 114976, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f22be0f94b623d2bd80de9ea584d038a05eca2db4d8a8088d0f95a3a88a8caae"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 230704, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "2282aeca324424c61bfa5fb9a7ac04b464dbdd4c5edf50f27e76efc2299f89fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "523cf02c058ed20b18265992b4114bc8fe8646038ef4a5ea56a34d343a4ae6c4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230720, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "209174774f206be5972d68ee2eb80b7000e6949a45ca476783ace21bdb67e138"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "bfc6edb0e0537b533fae0bf61452e9bca781fbcf65186c4352a2ac3bf440ee37"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 230688, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3963791d9274372e7d0fb55be9783db065d99483b91b58e7fdea8c16dd402234"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a3133a5b9a77317925bd97c4d7de52bc498bcad3d4cd260a533424a2a80a56fb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230704, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "f0df1e23402027348a3d4a890a66ac9497f4bf0b41614c5b78e4fb7c2bc44652"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "45ec80cc80a043f486ece5286cd88632a54bbaa84c825c998feefc36233fb49a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen", 230784, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "88b9f42d3a08671e58f104a6cc1cdd5740685de970e16172eb4a3213c87d1309"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen", 230672, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "20e46451657f20b40b58b12155d8c893903d0a1c2141e57bef832e4e40be73d9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "ca219ba7c17ee1d561af4a3d27ddb91b5ddfec020ad317c62277a05ebabb7494"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "75f79f2746323eda03e75b9aabaea7ea8d0e5add853d1d7f46ee0de69a91b5f6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 230800, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "3ff8cdd64e96d4c34ab2021b689d566ff738740c5ee28f33dbf2b8d1e1402da3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230688, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "deaee395a4d415fdb9d9d8fefd8d1962f170c207327ac5424c3e4d3ea071801a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "3a95810286864f67b6dfbf71c63d79057331c80c6ae2d696e9b0bd80cae9adbf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "ffc87a52af91b7d627d20ce2ec7b7a94cb167be11f6183848f5df7eabbe52157"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 230704, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "691c81804bb4be947d1c9e19ec339c7d641759c9dd4180c8074dc108dfa6d9cc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "43e7cbf87e0c481dedc3b41d681882ee50e5f8af2fbb206e5e247498ddcc2f75"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230720, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "e080494f813651997a7a49fee98f577d74a0f332c31e4ea9219d791a8bcef60b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "a9b216b21ba67aeebe35ad622eb6a3e46a85ff493d3393cc87c2bfa4d90f19e6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 230688, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "5baa7bf353e4cbab9dd003573c788e6d2564d680a082aa8f5c74b119fdec57b6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "74ae308a568bb0fdd0a0f63ca80cea4402fee88613080b8bb17d604517a2172b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230704, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "74c1dcbbc23470f4fff537bba89e44ee359f8556447cbfb33bb3c15fa3699576"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "6938ebda855f5f5c209969d7eb7bc0556ef3361b26139cf0a4537420ff987a2d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 230784, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "cfe0ecc5438a1540f076471985bc2badddedfb0ce3a1d37b828a1e062af13d0b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 230672, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "e5287a3cd9fefdd69a7c1796655b79ca2384ef5ebc11a84b508da35ae64755f9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "17c74e47c1fb7ccf1f07b682c564b21bccd57ef2042e15e5d5b60cda3fded8ac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "082d7b94c954fe363248182a43da84828c2539e61885374c527caa6f16acb963"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 230800, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "b5c7e4cf329e846aebe7b5528cfeaa1b503616f0196dda64680734424634d237"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230688, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "2d05404866e245ff8bf78ef2146e34a59963183a3c7dd9f775e50c7ade2bdbe0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "8cfcfaf7bc191d832bc0b9639ff9779f2beacd2e13fc438d2e4e43db61e0852c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "5726c126035b0aec9f4a47e8384983f71721370c87407d6733beed2377e9483e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "99a36242fba85d6762d73f8ac21a2839c420d108bb9c0e860ad545c7d4e6bc4c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "a09dfa12137d7270778c433f6f09ae09ab86ee1711b4803953dd3a926ed20571"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "03100b2ce81443d7b6757a280f8879c111ab128e7180174579dd3bd95ac2a34f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "7365c723d88528d1e1eb95094e6b178ff759eeecf3cc476716bbf2bda5202bdb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "839011cf60fd4512a8b013a39cc4578623efa236c47b5c8787734849f1a4b8c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "f5373640acb371af029ea1c0d415fae4613c4dbd395f26f090f021b9c3f7d0c4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 219680, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "1b0c394c83f54f47167d01b2625a450e1d549f3046d273970de5c17db6d82045"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 219776, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "bc85a37761de2f4a98eff39c4585ebde7e629ba4864a427f22f8cd87858356c8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 219664, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "858c7bd904aa0b1550f747239a616178ee4b37e34b119b410421649e1b941516"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 207656, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, false, false, false, "98cdc92f9bbe98435e64081edd78230e049c87cb68e7517cbaf5d9d70c949ef1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "a124843ed4328df618c192ff5b5883996c56e51e33cd75452ffb476b0a3290f2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen", 207672, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, false, false, false, "f39aea8b9e66747016e2bae4120d39d9935d1e87da03dfdbe554e3a603dac238"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "0c3a70252970fbf1366c7004717ed23816778304e960c8bf54f54ba2411d7ced"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 207752, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, false, false, false, "0fab5e32f1e603e1f285198de1821a19911fe7e16277e5565fcf7faa1c4c9b83"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "8687122459ee369c9d5a01268436f7878a126e16a920b8bd83d1288d41b495bb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen", 207640, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, false, false, false, "dc4caaf2b28941fd81758054bebdc4037ff9d739ae6bded372ac2cdbc3c7fe1a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "694c08fd79652f6f2bbd04daa40feecfe5a67eed4fac09b5b1530a3e888c832e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen", 207768, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, false, false, false, "57ca661be8b3462e86ce4a665b92e954dd5f78d57a262585d4dba95a0a3d1574"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "91b6dc28990fe206a9336bb904d8c87fec7e6d58e638f863ea2b289aae93ab68"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen", 207656, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, false, false, false, "2ae9a96f12ce0807d59bfc0204aa90bab426e91120dca7d54f7db7804359a350"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "08cf5c2f31482dd7d7f4062e07e51855d10ea4039b201a89ff9509618895f0f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 213032, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, false, false, false, "b76eec1c40d22cbfd8d67559c32245bf28051aab1d3f36eb44425e01dda33700"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 219680, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "f59220b229c7bc2d72fa68e559d573c5a1634a0fafb9c96c3b2cb1302b2de0bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 213128, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, false, false, false, "e3cd1506f6fd64c46c20457b3a43df58ac9e411215b9953790d216dabd39d336"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 219776, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "52944aa36352a3da5ca132ad531445043f273e5438deea3691d33e36b92510df"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen", 213016, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, false, false, false, "4380df2469a985a9a95fb8a60c24e42d124aa9e97e89007c2bc0b63d9310139a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 219664, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "a4964362a5b283270bfd8c7eb04c8ba8a8f2ddaa29f649819f202e2f4e18a13b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "9501ecaddd5fc43f8bd1cde1c89f907c0f89f68b6bbacdeba735d37a15fc25ac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "947f2267cd0e11d892b7f47667f562d2ac91d2033c2d3afc5d39b515e442dba0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "f194ccdcffb52315b2a1c9b668af18933c13ed42221eb2c1951fb5ab7b8f9da9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, false, false, false, "3446e061903f1ba54cc40921dd9740423f8f85d8cb2bed613848dece76f91478"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "0fc220e7d5688fc9a5ed259dd765ab7fb241ae0d595ae510a1570e5e19ea2fb1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, false, false, false, "dbd04ef6de4381681594b74f435327cb8e56ac4cf0daf7db4dacfb36f2fd464d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 219680, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "8a84db932bc65e1f200833cb85ac05d80255ab5480b2ec1998f10667a5f5a3bf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 219776, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "bad5e24654fbcdfe315333a2333c95c51724d93577de7828dc4a15b620137465"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 219664, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, false, false, false, "8cd56b6f731da6559c64e3fa0816057a600d616e5c13d0409ecc4517a4a56cc9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e616e3cd2cba87bf4d46c4b3d945770189210bf85895c73d2cdd351e434fef97"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "46c301085aae042eb4109dcd95bcc19e037ff331a1e37d2bb8d678a31a2feace"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fa0b1ce623be794f37048a396c2ea02df6e930392e2a529afe81eee6e6190318"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "09a8d495dcae35ad2ecea9fed00625fbf01711a19d3a8a4e7462b595a035aed2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "39445362d0659a8abf2561e69baee7ca6c5f403e7d6f269b1ce7aefb3b811526"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "2f37421d5d2723d745611caf977564083fb2e3a7104d058c255660996b6cfd95"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "7445d4decc27e66268ff91aaff45fed34affd66c39a3b0fd202ba91e5692e3cd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e6c3efb7d5b8052022fd87b76d1d1fff57bc599b826b782517bc95c7b61b27aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "891454d7ad7b3203ab929e155db233a1c1f6ec7c2fc670e8abad6feb5fdbd67a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "acb298bf4fef5d7cd65fec000278dac3b6204de0f43cba9c93b756f0b1b5e975"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "de91fa3c72c9fa4d1ec952f83a4b05d806ab20e5a2bb22986d8fb2f591234792"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "09ea1cb5868af354a9b8ee541faca349c0cc6a765a48e68da60b19f57a4e614c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4d8f8dfa2dd28b40a2032a6fc340785d5c01c47e264fba9f4ede97962db88b06"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "56404c5cac870bd64daaf8909447e939f49e4fe7ab561d959746f310e096b9ed"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2abf12c9cdad70c7e67d369a1569177a0dfa89a938daa1c9d91180893441780e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "bcd434633ffee264a98c41d639862c2a3526f7193322bc1df103b48e41080d99"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c36f2c6887bb05ca56d7e0e3b0c243a9ec9b76dbabdd813f3afa3ad39a710e32"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "22e0208f80ec03c394f19e733c05a11c58232c740a486ac5bd6bbeb58d240063"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d3fd1e95c4b5b50523119542c488d7ae756036912a04414d98b407b2cc665cac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9d4f39bbe57cfc68a20c63aa3b13988c89996077ea70183975fe67a4d08c8f5d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "17a59db153eb28c6e20ea851062dbd0c4cba258fe51ac3003d53c9404f9d0b22"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bceac3e5baa46cb3a86e6adb0edd1d8758117c21a98cdc33f69b0e829f156130"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "94e9854a866be5803a3864c0e57b5428ea080f69c38377c76f8052fe3c0ecfcb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6cbab2d27006edcfaa51bd7c56e51317f3f0ed5387d8d4654dcbdb34b85ed18a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9ee2dfdbdfba20808f7bc9369f87b7d1c28c22d0d6311af0b540c97d312307ad"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6bfa8f124c94034a61ac7d7b225f21367e2f0185277a250a826c1758b403919c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "45b39c4d3859583aae2f65de5b13a8b306cf2063c6f4c12e8cf13bd98d8a6ea0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "880c34e3cb870819bae2b43c52d3d351726de4fa6205091fda0cac1177c5ab6c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "46d6f8f09136c503cae3008b58abb94e4fbd42ddb6a14378365cce8a4beafe54"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c05ed9c08bb967a32b675c58cf8cfeb05521f14d6ae3045e4f7abfffeac76c70"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e1d50b5be36955b53173e86021a3c387c525cb323431aff12c0bdcc92a1d0064"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "62e3708270689e0dc1ae2c27c5b8af2185b7e738d616e4de42974af33c90d228"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "67eaa18e9ee899cb7a32b8361c67a773af0eeffc941fe9054ea640e1456ece5a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "03cc6320381294b63cc6851e94bd3446a722c048f7e2496d79da1d4ca9e6c82b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e9a360ae039d9ee3660f30da4e0b9c493be5fc87f354249cb3b8b72937f75d3a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "aafd06822c950d6d67b5f734c733eea397cc052ea91faede1795a61bdd59b25c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1869620485d98ca81ab56d35b7c4a6335eb837ecee23d13a3042dbe91b6a208b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "366cdecfdb1c1c8139665f99b4d3493422eb58c3d696088d9cb1d9d90300421d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1d0e2427a4798fae906c60a66d0c1da74cb7241b7d78fbf683117e11d89bd9ec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e81d1d6c976d4628f938b000c1371cb30902c5e34f3d9a37defd7bca1a9d3948"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fbfd3c06bd343435f65ab8e5a0709476b47918747c3b3aac02586d2a4f0350b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c53693de94644dca5b05a70441e946e8864c0b7b6f86de6b1deb9d853b58a89f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1b40fdc50832634c5df0619c5d9ed257dc4c8033783c0c1ce52cb4d4c3ae4861"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ece20b8d13621bfc46a39be84abe1cb2964c2a39c0f37367b038890ebbb8d29b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "70af0a082184dff7f2ba29e80a27594de64632d7c826ecb380d13af5c8098a81"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3828c6985c3fc0b0215ea233df7b152b6f8848682393e955e471281fa63707a1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "bf137596e298ff4a7ff6f1873dd598a2ee11c08e86d3621a95b0a93462030881"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9dbda275ce219573575d492b0ba02e391182cf7c9e149485051f9490c1cfe24c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "91c320605950c0d13bc702bc633f62c468d881db19e967efcdb6a68d8bb19ef2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a45e9b500a97bd68dcda0c0658fdaaa65fc37158a6da10d84913537dd1ece708"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "260612485ef3b5df819967b5316064be72ecf3e280f54fadfa540add98e5950b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5704606a232e6c2696ffcdbac19d66f6abd686567c7f40e85fddd640dc9fff16"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5e99879d3c70a03c2ad314be67c09296c83426b3d8613b983da7fa7aa35d61ae"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a4101f08a9a668ca8fcc8bac3fb1522d67d8a49a340968ea0e8c6610a772cfcd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7cfb6a0df0c3d5eb4af8906380d5ee94ce640d8747fea6fd53bb2f6a87c0c0b6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5c83236269386998505c694bd540488e1f3f2f3513dc4f59573b027a42a586cc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6085c4b2c9c515afb603647059658519e3bf8ff6cbcd3e6c0092da56fafeffd6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e3c06b0ced7c0a4c90fb051fd024fc456d1bd50380100168aad1a1a4f11d0d71"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9003b56d70bc347dfb21b86dc112749f4764448dcd6ab65f23f54d703c3a9812"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1c9ef59122b5944bb122d3c6ba191eed0af3898888982ea392bca3c9b14bba8d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6a0fdd0c70cca8bea1c8cdf898fa3f9cc0598e8d3e4776d8e63c4d151503bb89"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ff6349ef311025f7898043e5195ff1ecfff3746d2fedec2f40c4b43cf2cb4847"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "509fe7de2340db0fe3b08bbeb585d7a17b7839ff8867f490e72c12f24e53d3e8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "808305f0425b74fba2a507b990362398ca88925a11673c3f17be2e300253d2fc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a49b5c792d4d0ee4c25e6e9d770ddedece83bec27337b544e6b2af714567e316"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "88ce730516cd769c79449fd6cdc1c363ff9054b43089543870bb50c19bdfd409"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bb14c06fee0f8cd56d0734c95cc048dd1d2732ed106ca908dd6a25e050dd928d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "872bba01eee5218ec41c621dcf50801d6b17078f85dd4473996da1132bea10c1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c79dc7a34dbd2674ae28db6f98afbf52055d6a4f99bc3ab50d87504410e7dac5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f4a33294f595ee2667cb6abbf6ed8a46a98e0a5b31b1d07921eded2c635fc5d9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "51809fab141e67a82ba7f7fe2d7eebb1c20cbc3103d78786358c9939183132d0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7aee6cad5ca3d7fa79b71d7f8d9b596cc486dab05c73a6280d295745a11b5852"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5c49271f28b15df646936dacc4ece6aea19f751f4b749c60ca53481c997c1c05"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "74ad569a410f67b919b1bcb2e78ea02dcc5c4bfebdf645ac97426294ce08cddc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3d848ea97cf91be90b2eab17b1c4c5deb4ce709a1638a6b606c25fcc56adadb4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "69ba8fb9494ce9ab40508cd44ec00e918887a0c2840204f0cf3678205a094bd7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f33f5fa428e8b9e9b17ee7b30180deb02e1dd37692385f207f5a72f5eadce3cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "cbc918eacd64b6a0f3acae419f3be9e7c029c053c7665c60a03704d060ca4791"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "764e6e62aad479abbb4b39293f6e97a87eb48dfd18f5bdf3cc4cfb6dd7ed4d55"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "6ea4f61c37bdf00d3a79f5621b3ffcc19c142fb62836875ea88120e70342fd99"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "b8a546d6548d709ec517ca300e588a92dd06bb0434ce74ecf8eebbf8abeb6fed"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5b52d57265b796ffe8588a1bf6c7f673af98f978828ad8d0e3d854271c5989b5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "219e01e3f040e250bc5ad9730d7e1e7e9e26302ef4854be272714285a3e32d40"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "4243f6d8417b395341598c695d19d705adfd3fc3768b4bdd2a7943edac89eba5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f1fd1f42759fb97592699dc8ca9640b7fe8dd6097616ca132202fba82917667f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f433ce24925f66b81ddb319f0c867ea74fc8051ec2b8501b5331a479823d3bc0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "328c83546adc0cfeda610ccb8709c5cf68752459795da53ff7c5f1e465efed94"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "7fcac4760c7265313697e4d46467f63fe2fe66096d8df8b321c31daaa4203ea7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c4704f24cddbc326f2a214d47e4e1867ca46257e6ac521fe7b635baffb6600d8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "5f0307f591f4039dec197ddd9bdfde0d4902c8cba42808710924514c3e52fee1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2fa163d9cbb90a4d13556b67b8709072752602bf3b397437baae006cd3aeda88"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "db78c3f898e1bd385ec08c5fb77e75917440a33fdbfcb9c68f67c86fc5f12e58"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "262efc66d1931caf30b83e190955dc4b65ac0a75fca06492c553e28abc2f429c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dca77cdf9b0e1d06a9536e5007e6fc8a235dc77cfa571e68046b41de1d7a80a4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "95f0907f01be58b4ee86998682a91dc45e80dc412950ce08a138b084e9871b9a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b4aa35654144e5c7f5ba66bc0bb842caa022317d1cbe8a2c8bc3d6f9bbdad337"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "eec8a13c6630244ae0da7bfe51dfde8a3b63dcea50963c96d896428feafbf8f7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b9754f82e1b0660d66e4f061317b5bd4979b80dc5c2c877ea18339d838b265f3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f5875ff8b3a604f726de583e3e4593ced15a3720c1306811dbdd7a50731d9cb1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "73ff2b6d77c3451b2dbbae56905074a7816fd3c3a7190c290bb9402f07e860a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5bad4f517177d829997a63b0aff38dee45329e0d2e804afeb3cd3e856069bf6f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0bee2a9f4268575e3fbddacc04ec86cfca5e915393fe5e1cbe9d98c3cc3eabb9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "cf9a4258b8154eeb34acce00a24f591e18101e2947a2fa5cbbbe85f363ee0026"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8f4a34dea6b8c9cdf39e3d854405672a9a69b15067888a8d8b970c950f425261"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c576c40208dab33daa7a90db08b89cd69ccf9cc01e2ff45ea267e5dde1f49d0e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f8e0ccef41bd83db1905f46408592e502a84dcc1c7c9fb48f9ab9c832d8772c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "64c314f9ed371e112c137fb80b00f5072cc851119bca11a9075a0a881ee39a1d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1835e18713224600f08735cca0cbd94c3f0f380322d8dad695fe854cc974f772"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a99099cc3bdfe4faf8f5e684603cf4882f8541fc231138c72788d6124e0feb22"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "622a61aa87914a37764f1d6dff01ed52e6f95a3fa12bf7d00798a6aab30fad38"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ac248496af28f2e1e8ceac16727003cb5dd36cc6bd5bc3177840997a2ba04fce"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "60c7e18d347647d32818c24e28bc0a127a9b5b7a3aee2b44f4cd406de1bf9244"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "356c7f6f295b4acf5058fb41886eeaab1b259c8b489a40ceef29f580ac7b8ebf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ca6dae889098c2686ba84df86f8a63650fd5c1d5e06647db3006221f5a994811"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c83c3b7d2b48d1c9364bd212792e30894aa6c52fbab4543c73dd471e1e2235b8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0c389269db50f0195541ccc0d6b42636d41c28492d12861cd14ea2e49e3f101c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9514fc2d88bee006df1e753029e4a7fbde370963586498f04b310711b4c4e1cf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "96fd03be660e3d2a1388a1ab79b758433c94d6bd5daace41dbd258ae1f500315"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0627a605d6d69f66cdc9e7a5bdfe5d65b1de645b47e6acc18ca9ec9803c32cfd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fd60d4bfe4667eef058a823ceeef8ec4fba502ff7f059e84a7be85600384d740"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6d7ff11b92511d60c3637769c250b08b6f2d2bf7571c4c77de0290f6d0cce6d6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "895aa2033a92b870b7cf2485e425536d55675c84d2dfbd4a490731b798877370"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e7f36270b7b10257e16ddeaad5f258d8276400d7d796186e78a43d53d1d03591"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d3754b324cefb2db0591b0377a51b22c6c48c331aa18dea221a889ac6ba0c47b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "52f812ab095dd74383dbe25ffcc0a612210f492bd6fe48d4820ea678b7e73b41"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4d631d8726fe6618f52e3de4aee0b2aed539dc9fdf6ef7d10973b87fdb63e4d1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "026342e0f7276e5dc6287a3206bd76a77bfa2d5d814effe38325ccd69c1000cf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "60cc762ebaf199af127549bd21b4d0de0450e1a868b215148ec16476df406be5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "94460b64c315565be46ebb748f2cce808c7d3f4cfa82095f8863d900bc6cf125"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "afc0179200301111d4a32c249bd769a37ffba31188f5da9fda2ba642a335186a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "af3cd815bc62a5f266b7e487cd97fa2b2fafd0915039bdc290436b223f4a53e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "24275505a7723e65143b295d9a0c81a0179d635e5f0b446020a9533b5aa4ccbe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "2dbfd8d8459a215509e2d9c03e66a8abbb35adc2cfd708ca295d4025682d17f9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7141079f5373955dad1979eeece09e36fad07c38e604162128c0afd4fbb8d000"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a7e889d27cd82be821b071781ecf47e4bf8e8d0aa14d909c39bac1595c14b14f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "90ca8abac3843619f962806164ba3110b65f21733f3d30c2d9c86b2f721a6f64"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f060a6f0558d27f9a9963f9c9ad4ebf603a01f9838bd44484ccfb7b31fc909dd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5ec8f37f528fd35d500c9c031e0b30c93934d3a4f57de6c201420306f5ccb4d1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "68699538361832ed9f1f7193e1d490199b841c2fc092938c5de5baa4bdc49c1a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c7abf61614b28aaff1faab2b033aa63c65f7c29d20e6feef62f563012482b2ea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9c49ebaa7bba29359ed530fed9589a7a3c7d15d5b78f01d05228616b3fae4a51"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "750693671395600b3202688b4d97df7bacb5db807667d69e0ccdfa3f15162e18"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "563c7328cdbf15c63f95775620ca3dc9f62d53c2a52315b6f0b333b0e267856d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1feee601f2b0eeda2bb6fb0afc942b5570c66f1d8fc97a735019183789114138"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5d0ee6def6d19e92836cef3085b75b0c0ef3169ccd12c7c6463728dec0b3b13d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b1f510605ae81b607f85883d528e91b62c38d2a81c3e3487126e99cccaeec46d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "54dd50082673f9fadd82beb4578e3562edc203a5ff906508224e74be2575d085"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e55d881edbaea2e06a3a7d8eaa81132c169d827d4a5172d64069d8b01f6cd181"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ee9dd7783428e03af1b5a2f76a5eeda18461c1797341b42a5f92a669b4599d73"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e4730e680bf3f2f01922d38f7f71ca2a965f97e6c868de4a1521daaaf0905e8e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6f75909181113996aad1aee2249ee7f0c3c5df5c5868047f3f33d2940de37182"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "161d42c74c1beadf47c12c45ded8aba6fc03aca0c32554f8d1c899824b6215af"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "642b83740eceed517f335e4c5e2c1f181c3e9c48b7594ab474270682ae4f102a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d050b01594f2a0cef664f20f88a5c37de3fc0a8c7a26baeb6ee50b6d8d363765"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bea1070c85d075e8c4a859480a5829dc873653e0875c5d662f801b00ed586956"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5bbc7f9f594bad0ce9b4ce71b1814d2a55a06dbadb4bc7381675fa212162c711"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f04a480bd36b7f866f6b94997cb4d3c6b18d5a344e2b7ead2dbc98813846d3f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3c527de52db7fdc335f93ddc678213a6c4d37b9870dbfe3edc448a9d880cf3e7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ac8a20da8a3a4843ff303b7cb9598c693103c0ea802d24d2133047e7a9f93613"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0c7122d4b07e94343990914018269741b2308f20f2f5849b8f2b2f78d412da1d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "641b0c19fb4ebac84f73707e083b0f3efdd833a5329cdb3a7a1bc0d2d85198bc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "18e3a422dea752aaa9c99caa3b3cb7e1e999aa438a596e8cb19d54a826722628"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4de6ecaeb598dbfb212c25d0b14c6499bd7610e721d6257f83e126114f4edf7b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b76baab331322cef77d02fd4a62c29fbe925fd46e1fd4c2959d225f5c06d0d33"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "22420e2b46d05acdba18ff68e3ae7eff44da7e0dffe7576e3f2d11d7fd2fa463"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2a050c31eaaa6492336f8180bd723309cf513e2fdf724c44bc58987edb0701f6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3dcaf0da05986e0307b891d06bf0a7f1efcdc00f980ac16e786100e764420102"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b0e9caa2117332652176e74fc1a0fc5d79fc97c541498bb429567d412b632fe6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a6b79d7200e1ed2a7ad27f24c493cc23bc14fdb02419909052392c18f2ea53a4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "68a0018bfc957af0057f8544cfa84f00899290839de8522e5310e14e2e0b452f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9fc0d8d587cd6769f7cec5c9496155b0d713ba74257b6d411cd7fcc684605f10"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6edc4c4d1eee8bbbfe6860401662aa182e3aa443a50f9ed7bca81bdcd0babefa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "90e654644ba2f4fe06360c9d5cb7c2cde675f65e35bae32c6c7199a154f3f01b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d1ad3f6a3aa717d75236b18d654a01407561330e67d03a237c861e82d7197372"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "933a1651518512d2b601878c7262e78af012b7b5a78615f09518d576695363ac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "79790f810d298b7c94c4c64c118d57e4dce3e18c24d9ffe210c13e683e36a13f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "652438f78abed8b272a51ba65a5fab20f19b4f4574cf59664bba7cb238ef0619"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "10ddf61e42e31893cc7908c02da6cec876b519cb84e0761d51ba5bdf9da98615"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "cc91cc51e0f9eafa1eaa43b893202a80fc3a0506f56e19707c2d52a04bc5444b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a0f1a07aed27e21308f12f5e81af9a8be529146a12afbea6fa0e27baa926697a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e51d99e964525629ed92ea7c14d4dc409ce4f92a1b289f3af46b43061fdcfc3a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7d67f94ba0b231ffd16b104b02e4cbcbdfdc5bf9b8ccdf86fee2254f8153ad8c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9455a2d61077cf40d9f67f6257e4512c273b313c7ab1a64909ece8692929baf5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "518e2fb9d21fd096a997f9cfbd9914cdbdec610225a0071b6e8aefc321447738"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "866ae503b24b6093de2282b7c963bbe811afd4e9850dc6558c01d20d34e12bf5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5133da22d30b5e8ebbeb255be156b6c92f552e566836f9caa68c8b2302c92831"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9bdc9755fa1987ce3d3db9c1ca6b82ba8b6a20dade59294a1e9e1a0bea23b5ae"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "fb61782ab285673dcc40bf99deb8f1f597751d402a73628208b97b293a78e400"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "231f9d8c3b28706b7573f3707b7369743c2af2b6571e1eb2ad61ba049827c7e4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "248bf4cdb83f4bdd138a1c7ddea0f1ef8c59aa44171da36d133bcb83469cbc3a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bf667b8d8643066cc4dd25c515330e904edf8db68607c4ecbe714063e44dd9d2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a8f423d0764b150277508def060998c37d88f802b680840ee4f37b639684163a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "55dc616f69f103a848505539718f09b650cb990f0dbd38fbab7332e35e67b9f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "af1ad0263282d43a149e9e60a2608e8b73242488a754695dbc1b6216cc41c93c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "75989abba39549c79b0d78d832fe83de7c98283c09a0ddd2500b9ffa6603df0c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "025f27ef2847deff54e1b865c8fe628489eb877bfe2788d463f5429caee6dbba"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8256eb760191004f70d62c0bb6f730f18b57e5cfed86c7328e2beee397e0d767"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a55259bd3fbdca6dfbd4bedc564f85a6f72951c1eacc2cf82d40b259acae8620"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "dc993887eebcf1b92a4be248f0c01cf0740c8eb5813ba3f4a1718b4c7b79fddf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "eb9f5fea3e6077445149bc36687e5e3c2a88c9ab585ecd858f955500ff447ac6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "35357f86358d2e38c515ff0a21f4bb3e9ae424a66935e70c4e1d6e9b87df4fa2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7eae84b3905bb183e37e64d7e3b338e570d51b081ae9f807bff31d5a39ec07bd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8a5f94d354958aaa2a81743ba5af0340db03896339e83e6a07729f6ba14cbd2d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2918697fabb6cb1b6681be1a866760643c444c12764a2e914214c66f7af19f2e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c4b1c4f97c5958d4b79728f77b9a1127bf5c57b47a3e9eab2d1c16fbbbc2c084"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "6b3300334212a47adac8019f82e845357b89ed517a39ff83d9cc7756e6e31134"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "5def6180b7fe1c159badf817c66b93038597ea4a1b0871d93ae2a22bfd4b46c0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "84dc0943ac99f5d1217b804096adc4abd9bf42592a26907f4524837298f76585"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0aba6305a62b49535264fd650df08efa4252fbc52dedf4266ee9e2f6690c0475"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "323af3fbd1acad9370a1d2bfeb5547aa1a5bc1e00a3a87d7dc69036c3a78fe98"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "763bb3af54efadba868cf491467d95c36b8797a567aac06590cdc7bde77c4f8c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d9ce3f382406443163a8209e486bc246ac74cf9f0b2a9988907fc41a5995ebec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3efd0afd70bd25ee97bc30ab732c6a81fbf0c7aea07c763f4eb76b5a08c9401e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5c435ca96fb3cd76b1c2848fb8a147dd5f1389b43b7758661c17de83df12771a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4804f2fbe9f9a79f1ef714bbdb3264fe1b545edad996460544b54ad34cd950b2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d65455d7fba855ac156ba160945caf2168b9c396052871c82c25114386c2d370"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "893acf2df5d2374dbb97e50ca9ab693fd0662fdd47a27d3d554f0560def90cf7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7965f29391e9755816ce5bfaab58d66df92b7584e0a3830452145fe9bdf2fa5f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "badf772bcab24e951462ba5374c585bf5d14d39307aa451ab7a9f78eda51182b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8e8e62b3fad9a1e923a661d2fd7b1cb5ecc47b17384a5b5b1d17c004fc2b3d68"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "583c1e8a427b48e06845b19423d4e778ece645d218c8de62ad93f69e094262c2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "928f7c4ac2700ce9dada084272ed9108c4893c52b42ddd21a5acc33e661e64ce"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "da0a173bc7cf618f368386a56c6834b0b6da6d3f5a728cc3c396c94e2fe18138"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3b7f2b2d5f13ce729c69db31ca5bae603cc8d45c85a26618882c71547606f9f2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f2260d48d30ada80acc93b21953750b55e0961477b5ab3697329c3de4baa03e0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2c45bdb4d11ee1d8e7cfd342d00285660ebc35a924cb5e24a431e066c66ed1bb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e5c2b47eb7d86f664d02473fee19e1761ea236be8bc9ceb2034351cce6e060d6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ca692ae3a26d34eef2eebf498a0249be8bf1aab266961a59b5f52c63bba5dfd4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1b976f2aaa18c55f27502a431c0db85778c0e28de7e06391cd3ccf037b928620"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bcfb83c9012ac7cad39c7e7e21e9c0a9381fdd166a1710ad62926146fb1695da"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "37ab3edf8bba689191e759969a2bb2675c8f834c4b36b8443f2397c1ed7b1d8c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f1b3454df31aac1bff8b89037670c49258028053bd63e821c2bd2ab240b38611"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "bd446aa01f2745c5ef1f6fb629c15b954af5c85a661934716cc75c12b0939ab5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e17fbdf2b130f5021f093e526c431160b90cc2f8209f8f81074b56babbf95dcb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "60ddf47bb188bca9e5076b140b3dfaeeffb8284ece156f29abadd770ebe9b578"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "fb321ebb18765ae1f998ce713e9715331c598e7a038eb271e6c4a61307697abe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "35d294c801202cdaa97a2f1f7d4f6ceb83c44dd70cd6b5511b0fdc14a0a26756"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a6fd7e29e263188e5e7e54fee5af282d0b1fc8d122c035c5dfe8d4a4f3f1c758"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ab509520af2d363a15ed4334728a4915861bb83859821111cdce082cd4eb1681"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "540767289f0c6f4eb4eb3f904e6c4443e4127640b86cd8c028576fd622d11c47"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a1800295cb0980f3be9cd0984a8ab1c99f3b057573211797ab97c98b7f521989"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dbf9852a2620efed7e176cb7fc195ea203e3bdca6e57d7d99d36e061437252e1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3368a1d179d5fa30e3cf185a17a61bc30457d7070d05bf3bcf0687c4f4033f49"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "868cb8c7bfa5bdd2322efe4c4b08fbff1e1e93a20f8054cdd48a60e7598ce8e0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0b2c5c7edfdf9f75187335f53f0a00e2c32851bf13913d441d02b7b6f59a65f4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "da09ce82ae0834a85d973935df3dbc8fb565c39121d67a376a695e8749312452"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "882e0b60060cba2991c8fcff5232d3ea149b386f14542e55ce187723324edb50"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6ddc364aea9733f9d3f156ce4aaae3bac68b3528f6f143d569adb8332daa00d4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "838a2ffc0223625cdc9e6a4367e8f53086e1459541a0b8f987f18d9a1b7a2f0f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "23cf9f3e957fab976582069263ee624dec46f1295af1b60ba036176c3eff7be6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "85d64eb1df16d0b2037542d0b2e2eae57c4e9491b583614bfbcaab0c1b148516"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9e819ce7d607e0afe230487ece5b15d41878a7b6709eb8e0327df8a7f572e743"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "da62209dbf025bddaf25b1b590b9de195d5bd763af059d688b30de0e41248dc1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5a4a043bba3613dd45a92b9d42d360a40def4ee9fd78edfb90da0af6aa907c0b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0905bf5d24b805bacda85b1fdfa78b8ee36783381ed086e71c00ac3996b84b75"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "53db18fd54f87516506e624473e902445c0d0baf3fb5ca4e9a082f4601a41b7b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0a469bd3406fda575b08901c46435148e42a03ad77998b5560ce351c502f878b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e0008c35fab2bb88a45ebe11f066648ca45afdd9320ddee21e4eeeeccc3b9ce4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c8239187892f0ca6c701d41b95c1fd29b833e7e85c174e7a9113cdd9a7d1416b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "006d62f5a4de461a4f1bcb4ea38d1c4244f79ea8db7d69c698603bddd2c3733a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d425911e719834181a8251ece15ddcb95f87cd6aa29a4000813859adee6d0334"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1ccd083f388c7c9091078feb8c3459742aaff1a0fed28f7a9e2afe523e57e491"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "fd9e8faeb6a57a3cfea6b4a2d739522b046001453a4f29dea13831d780fbb4aa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "09c8423f142047add0c4623ef148bbd9171bab96ea00d0b880653674bab28847"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "361e330ef33acb5f0bba00f626897ba047176bf370069c2794e6af94c6a62330"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "636a3667062b470a22bdd716e28f2de0c6688c17ea6e9bfbff2027b0a4d675fe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "294bff605681e8c7fb87a63581c2ed54c420d3830765c0777b164eca716bbac4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f6d67050364aea5ae02da8764102bf3d17d73353e6efb4db9c400ee622de4693"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "bcd4d39fc22886014fc2f2772b105a70ff3af04723420d05de975008fec49f42"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "7b88c90bc6c7a3e6ea84e7a9b0b1be705b2f04190d1d04f2a08ad179025de5be"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e6d9e4bbc775e7796efc0c8fa823ade2c1bc0aae0452a33a4853f504bcfaeb9b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "671cd97198f2a21bcdad62d069d406f03be7ce8f4f37f5f3dfd9670ce3b6869a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "20800b9946399a1f820db6d4f3cba0cd49fc52f21bf12a4781e9245c27c05af6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "9586796d1e4513af3dfb1c0d8f1a0e0456989af3285178bb28315b4afd59eb2f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "97ce40030bbf472d848c816778b3ea65ae794238cf4d6f1e184baa901289d3b9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f75844c62cd340114ee2b352ce3252244a9c2eb7de77a338a2edd8575f9d84fc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "0e19f07e551f89e93ab8d5b91dba74448654765be044508e15bd0b83a233ad40"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "99664abaafcff7945e31cc206eaadaeadca124624a32ab42ea528ed7b5bbcb8b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b21fbc193e34bb967cbc8924b1773a9edee1fd238597ef0240c36e98eb2d6aeb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8caef873c7c5f575f4c03fb64e00801d73c0896aaef3615fd4d27b679151858c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ce9fdbfd7ace464c86628110426e751559cc8ef81f1863cb24d630dee5463c4b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d3beb9625b7efcb56509cadd3b511c5213a8b684bc44a1f7a8c4adbcbd4b0c7c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "23f4bb2e3d68fc5d5ad00354a7535eca44a761d4b142875ab751875aab9d7f8b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "0219e81b507a43c96dd7c1a62694749b8b36bbc854039b354bd9e5dc6d4802f9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "584d6d49175319b0f39ca11acbcecb97729146e9658f90dd5c990404dad20405"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5de3f37f50c0b02c7f39c4df1497707a251d25d15ff978a3084d135a4a20a80b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "d52bf299ca6a704a9ee292e63b7d6307009e6b2c53035f61d952f4d0bcc320b1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a1148ab561105d7aa1378124f8ab563741b536408d156b66b1552857341bb29c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "c742e5ad54d5f5e645b8cdb134b1786b295842fcfa87b9104a3836c450d3144e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b873f37b0b982c10b8f288678986d45bfffd4e86175e26cea60a9ae4582d39dc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f577ef4161533b3c155f58376b1990ac2f5fdbc8454caa119cb33006852a6e32"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "221c3f719d30b355ef63a001f4ab24fd6fbc105a63720c5163c970310faeddee"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "44d48e3b1d778455de4cac6b475212dd107992cb5bdf4b2e610913a2c007867c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "1111b2e6a4bdfae37437a4077737d391dcdd35b93cffdbce32273d865248448b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "4e60c213f28fb61f82cba412cc923f8385dd4202e393160a7fe04b25af758cb4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "3beb7db608a472c4427d19cd5df72c4b94fbecba9f370bad11b8fbf8130a90a5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "755c7b3bfa49269f76fd59f355d07e299fee8ceed01da11b84e447834149ecbf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c672a2469eca99f9c16ddca799866c114132c33fb37a202a91430f4b93cea452"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "688652cb71b12ef250028697812348a62d89ed706feeaf669f7675cb47f39251"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5e1a29b37b4216b0481237167075349ef24bf8b8c69af8192092788a67ca2c56"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1004f17b51864bcda692a002a9fdcc0ee9cc3ea2f08c489441cd91e3be8c143b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4bc82e0f6608bc8892954eeceff1fe9b04a8b3bfa72041a64a543acf89b23d97"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9448e22bb701443ea2c95485b0a969ebaf8b6bec8987de2b11a29f14513a412b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "094f1a54aed921a633b0ade1dab1c427e62a99c49de8e537404d7bf4bfb1dfd2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d92a98c26ce8cf8ddf1299072e405cfcff5078aa6631d77435d77159a703a704"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b5e0168654bc4201a129abdf53ac1d79a20bb9030a500e3fb6484461a5582b5e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b787f5f1acb75b6d9e75b1d9d7c5db24634ba9ac269eff360b860bd3dd899b62"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "aa5dcb590e5d3c9b01b07e0b589d72df8229acc83dbc6ed8a3f75d50708db3ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "19f80dc75f880778d8f2294f801b96d441ebc34ad54327a4ec9322493d6dc9d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e9e50d6cede1c2fb05375c22317e50910ebdb8487e6901804248ebd45232c186"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6866aec1a0f4a1e1242a871cde653001fbb7d413527334f2e004c1a5b1bce52b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "635b31cf59aa2bc4360dc99f06335da539314fd58564f44b9da7eec8dfc256ac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3cf79be30d355fce1265569692d0f25f79c22f12408c3a87b1ac2a7e4e938f39"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "89fc622af70c62329c9f03ed9917c9d06760a92c634735838aedd3d3f757502a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ae9be700082e4e3b8292fb5a89ecc4e6d839396a577c58869168cd6d51544449"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5944970a1876ad98ed79a8b051ea0b7e7b5d314c6343c1c7df5e6a750fc94115"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5bd13ec42700dedf373f5f282a3234c3e5dbaae03eb6548d74b755f413216f46"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5f73fd37df814db1a086974ba308ffe9aa184f2a4ab05561c83e160d6da953b7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6010cc1787abd7656ff5e1b1cf7fc26bc4906d35bac146c27bbed53dc474ee42"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "47c76913193ce3028b2472879afdecd57e68ee9e6295539c41a2e918fb35a65f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9e9492f65270f0956cf14bfc002678a9034fa2e1071d88f58da95c4e42517709"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e7daa6fc827ff090a4af48320f2be36da2ddd96c78d342c1989b69e6dbfb4baa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5362310025e1a143b3b534e87ceb5dc9ad8d9777457202117f2d3143ad4a4e4f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "93ff178b5e4149e0d109cb432d0821a71be139d717b6a1d0a5c397fdd4059cd1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "39579106278c67e9de04737d211fcd5271ec10366ee8b0f41d03ce5c3716de1f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "604d66ed6a3e6da3c0927118aedf7166e9f1cc50d6448d698e7cc8dd4b3d65ac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5729373da5bde1f471b68921f868b9bc7e3c850caab7c65031ce11a3af7db9a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b840b62c0e99ccbaaf95deef2f24460563050c5a078cca57b1e62b09dd9f0e57"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7881d1a7b5d436153af5cf9dc531b18fcdcc4717b8707bf87e17a6f688afe897"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "eda23a0503ccb8f8bf13e9a8439474b9a70187687cbace3aceb7916df49fbe9d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "edf9ec633216f4ee4f8718aa24ff03546e1dc112f15a3ff5d68e7f09392f230d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9aa59cbc1c1e79b184fb377cdddec2a1c1d0d4a2d88e0e922b0381a5c6837ed1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bc53b45d4be1ee009ed4a4c3ff00612088b9e9f97b1b8aa90c8d4b8704afeff3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0cbfbf77456fe4b5ee0918c63b630fbdb9cce44b6c7999b06f1f1bc9d0227814"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "135ad1658c2519dac07a1d59ba641fca527a8b3705c3b893077cfa0e3df35330"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "fc6eed3a40d4f71a776e7b6c3f3b78fbeb7e2d98079488b645f328c1aff235ef"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "19875025253fd4ff8d3606c3cfaef552ed6d1fd8957e43911b67521b16d44f7b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5580e76f46beec05b180309561890f742c40397aae376aeacf4d6010376d2127"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "050960571f1969ba0dfbc043d0466622a7cc3764fc3e609be353fdf3378c439c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d259ee4de07beb56071e7574c83314f1cdd1188c4a284b6a02a84868626323f0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c3dabc955f51c43aefef170a144d3293cbec7760d5de7752144e12e52c026be2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "2de64d804f2b115f76e745a8d0330595688fa9a679e712be747acf7dc10d23ca"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "35c7ece295edc3a176abfd8463bb43e934e08167c08040ebe52208cd93f1e08d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6c6371b6657941e7d6bc331e93ae2480f8bd131d62c4ac54e9fd2d0c81405a6e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8e8ef06b2f4f346552df18d35ad884d819f4a84309aebb43ca3dcea926b631c7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e9e6ba20c7c3ad2f8515848da6fe224af46bb4c0ce6434bc33e832bf504ccd8c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3ced9e0a6dc77a4d429b4df9cb1b87e52adb3b1fa3080d19e977dcbc5987a261"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "62e815499878916bc960873244521aca3610f19e871c85272f9867bb826844a3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "deeacf0e55fc11073cd01604eaf23d260b326c07a4f71cc2a8f5d21d7f79446b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0e755cd14abcce3de19deb433aa942a045cacb40be2f114b26d33390ff9b4eb1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "35984ee9b3f1f0099068dd54026b301e4715f67e3d866a07e2c2b6bc2966440a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "2ddcf1fa170f8cceee32eb5256fd5d2358579f436579d140a87aaa009d3b6369"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "321e191f29e2514a72ab69ce92a244df53fd1c7abe539cfdd471017f0f6924a3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e486f0729320cd4365ae4aa4271474017dd85273d529eccacb3b55ac5da955a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ef76a0179302e14cf387d6a04a636878fc79b18246cd1d50b8c495edb82bae1d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "276b66737a952652fb141ec0d75f081d906e77877cc5b0fb91160411091edd0a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "72a52710c33d08a4c85f4e1ee49360d8eb8302f0d75988a2e048bfd9e3d1c7f6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "856b13a5d7e079bda02714618a124efcda1a51c7665733333a13f73581923988"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "0d4e28ecf71419ac9cb11bb7a828b664410a62a714aee38c633cef6eee25b2af"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5e74d2d4820fab5a0162a0c8e39e79db83ef9388b9308867988b21086fd12877"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e39f0af4fed2cec871fea9c641d70b6fd632adfe4cdf4a698d9ea0eacc3b5a87"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e9c25d1b7de0eb2d05d40ebd0516262233bb0395afc300121bf93707fca2e87b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8f8a64eeb6e00a86ff2fd72a75d9661cec6d0440e5f55eca540f85792b45ca4a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "476a51525a57c6220268a6e739228b392167633159e086367588f17321386dd7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "7d22d016d1f91c684d3b475ab5fca7b95165d8cbad322e0d42e5d4a8d98234eb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "6d6d4efe94a014621346b5b9fcc0224f8c537f0e06ab670611600bc85b0027a5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "0ef7e467f5342ec39924e85da563edefc1dc76241c0d60251e73725140b909e7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d9cea852bc81985db5f0f155f0a0bca755b3a0678bb11858c58655167ef55480"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "0b8e7d3e6bc82b8a878f560330f17d89a36218e5fa7bc3b830bba187e86a7383"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "7f1d1bc03831a3dccbafab0f9623fdd42579e9bb043799ea189c291b72969a95"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5147f58c15d58713ada9bf64be3faa44ecdd0f30b8ac8be846675d1cdda54fff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e3734e7f58c5287d4ecfa2e22c21051a2423a6a814423e76fa7d52120dc6a6e8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0fb0c992f82a38d9b51c83e9d2b059c0e5d15bc971378468bd12c838fa18090c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b0c49f59666a9dc35e969c8aab03be25e47dcb08a694e20065c1c6858a1a1d50"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ceccb4cd05ca6e8915938b0ea4ecd91380d8c9d2844d8fd462789dfc771ba130"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "04f016b667165b53df1ef35eb0985f71e69aab1a00480fbda74a8944d8f3a49e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ac372eb841f77fda99d3f36158fa8754011552266c7710defa7d12cb38419c06"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6f84a14a7474491f9899be33b60500c93b42411aca2edab687dcddfbe87584ee"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9573e9d0a3ec834b79468db44cf12c6ff47a542000c610678b461041663200ac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "12bb448ee75625ccb41d6289355cde5405cb1d4af815e43b3fc53f81dcf70334"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "7fe5fdf18c4205cc815b4e753961a92df1cc00fb815d0e317a596a56ba9a2031"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4ad03e063436072814ddf1c502f66dc74f62c652904585c2b144e8919310d30f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "39ce84acf42367f0ce63e2578624f02eee2bb8271878a587c27d07f2f2efebdf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "08b7e3c68b33aae21cd609c77fa2b57777f3eaa90873b50333573ffe8ce122f3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3a308d61c018d156c2091982eaf17d1e75a6d396558415e3c6ee628d73be899e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "06eeb54ce52714da5d31a86c894c64c4ffa21e1489504feaba1c11de7af7e14a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f83efeac6a61a9a02278e6868c4540097d37e70d5566b4953040d0ae3fb83010"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b051318bb454c43fb2ba3761172a0ef4e18ff4e76caae6a81e5de29b1a597558"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e1505a7991cd81b76182750fb7f1229f4d7e8dd429452426c1b740d26e78aa2f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d54057c32cc7e5df089935acd91c4466187159a8b23acb3be6e9ed46057a0aa7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f526c55d58866d1d75bed46640b4f7705a9ef77bfa9903e7e4f7c886c9922046"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a47e96422ca22f80a434f388cda5bf8040d2da2d5307c9bb85855a6b49fd811c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1e2fb078bb2af84feb28e8f1ca3252eb48f04f97de1857fe542550a4074cef58"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c7b39973d6c56f362436e393c4a4e4d4b7adb3191f1cdb3bef55a95fd7add3d4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "709cc4e1c3fcf3f842fef4842fb75ec8b594896b9209fa5357e0eceb18d5f515"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "400c88380227a0419d8b15e4a4888bc0cd5a01ead0e42928dae3cb22c92470a2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5aa777b26adfd9927985cdce4d7048964ebfe9701658dafd9088278ae85a311c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0936632d9375e47b1078419444504f060d6cf979bf365bc892942a2ea12a1d7a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4f12a605affda17a352885cbc6f590f3c4f25c615dd74d2377643dc88f24d898"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2431ab590e8d30ff4c18fa4542bd3d768216fceb4d02b90907c0cec880434fdf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1320033d274ad57e08e7a0f2791ac9dd273a6675cc0635d9985c104e3efb2ed8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c45677f14e3ecc636a2a8ee0540058740f53eca5ed3160c62216e7c948eb05e9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3f85664b77ddee31cf93932c165f97de9eb5193e13277e420c9e1561cd8bc032"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "27960e30ac60630af02e99a2b8899949065dbb97d5c2c311f5703213f3aa9386"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e02a1d93a7bf3f673d7ba8e497ecb146100ba57a58420eb770cbb111d8fe92ca"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b69598946609cd5e335e55762f0fb1338bf1380bd57a70027018b8f1503b173e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "531700e627f3f29c5e726e5ba3c035c7b65eb9aff3c89b6dfb515c0510fcf085"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3da063b91900cfbea969000a8a731a8d00fb327fa0cf05abea2e682a2ea38dd8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "fbafae078b4a46dacd2667c315e9f17c45abf40a031e284488909e25d8dcda2b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "016174f6cd8becdd17a1f9524310c2fd8fb959005362c82c578709e9a6465cf5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e3c8303317143afbc963d2b0c1d91f482180511a30383ce2c8704dce91b33dfc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dad96e1b3772cb341e42871d0859a527549767a9f8778489bf94566923488ae5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ece2404329b143c01f60b46463f0231d58c60c2916c2f0111c598cf8f5f6333f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "fcc14d85c1e185601446c21b4179530f7304d4bc09e15e056a391d98802d99ea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "86968ab7b2cea287ee5f5a882f27002884306f0f765543a23a3b04b9effd0bb5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "438d29288fe98a592a1d540340f6546b369d44912faf88ceec46c656ea661db1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6b1e8ec79027a79d5afb778aacf4faad44a22de8ffb54530f0c456d0acb3eb9c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "705746f0241bc5c00046a102070fa86a9553497219d058f4d7534fb3e03d067f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5aabc6ec362e345416a2f6e2e65fd34c1b2e0d088b09cb285bf6c1787bde267c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f544c8a4ffa957705187173a88495d9ef79e39e3a9fa1c9661d5b8b71faf9db3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b4f72b8d9f9180f88cd087177feff3dcb851ce95d8a354b3acf49100322686f1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ab418cb0d2e9dcbf4136016dbf5af50a2378bc82b5009c681423d39a726fd397"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "56648a6856249c95f5186ffec2108ffa1a46c36ab78e0508d1e5664fd9c7e4fe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "24ca04b3872343d2f6267020f3d6cff3f5db2edc96cf9bd7860a4983fb9d2b9c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5b49cc9e8e5c2da5077a3ee775dd7c63b33e792b8f1b1640e18ada513cb4e224"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a89fb1a7466fd5b21f3a400c6e05e51a71574502a790a9b2a0d25d3556a39a6c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9a5f4b611480ed683d73e602e40d7f09f3f75635ccdf0c2135ab8ab3ae9f6348"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b632a3ce3953f672d514c6f83e9dd7749d509ec96ba980b5ff93f05f83315952"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "31dd8220fb5efb4aa83dc70fab51b252912da18ff54d4b1c4e8621733011ad81"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9a21636b826796ee6b20e9472a6b960e17b76e1a43621f54860321a6f787741f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bb7904049cdcfce88f20834a4b2b548bdc6d2f9e71c8c11999717b27552a3c8f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "149c3a64f4c44e94f12ffd3820733e1053147932267fae119fa3647b4f936bea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "023a5966d0b8b253f24223b5e034d3d1ccbcb135e5deb364f0bab824aaa13bb6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "68eb373b02d60dc12f0e797dc83d98ea0f63df50b0ba945bff9b7007f5255772"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b4f294bc752bed0b17daa94191368f2819813bbd3231574f48607f594c562aef"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0bdc8eaf912e27d3f079df3d64d8eb1540eeb49404e225684f238bb6d4386aff"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4ec8d580d5687717d61bc3b354d3f5a600e9c7465a67ce46d2deef36a44da470"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9388bf0c44104d005dd04d75c117d5b70833d2c1b638db5d45515faf9b60d587"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e2c27a017aaac0ef3ddddd09b6d1e0da4e9f6e3cbc9906ad5d244d48bf112e42"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0ba3ff42abed612db2426e06f46115410e81872a0da06d7ab19e5773e741bdc9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8957f6d956637f791f9c8a1b7769a1cc542b85240caeb3b3250b768b027bb9f9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8f219c60826f3d892f361ffafa35ed778f6bef7ac7e11ddefcb1a07ba626cd49"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "62aead51024e29c821b98e151d5ca4a11245b9072bd74f426f5283f33a9cfbfc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "18bb28dd08eb5a4a97f7e7ba9cf67b9006401a922ac86e58468a305894f213a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ebea824bcf524aee1613403e9f59de8539422fa32ecfa173e8a6d142df718198"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b3985858b32649bdcfbc499f61a8dedba6b3dda5f22bcd849704c36089982f6a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "950641f2ff8509970ba970cf5bbaf015995e63d68e427188f4e6e563cff8fe1c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7e8abbdb65d466df301c77a8b0e54cf615ba685245f991194636a827e90d2d7c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "da968de6bc9e2c842740b4e049644f14f7d0e019911d66dbea894056d228c71c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "40bbd8ffe14ccf651bec1d7e17d20434470e442532ec74b4499acddf6fec7fbc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a43f40413c123c0949ad8ce7e9230d97c0d8006a48da4ee2af0d891cc684c66f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGenRopeQuant_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGenRopeQuant_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGenRopeQuant", 188536, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, false, false, true, "76c763e5b002ca028192a60bf81997a079041878f159d8fffa71212e0f61d3ed"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGenRopeQuant_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGenRopeQuant_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGenRopeQuant", 188440, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, false, false, true, "a21f5e3500474b06e65184ab4e66a0642e2c3de737bd86036bdc68228aa80c4b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "93bb2fb767272cb7be2241359daf71a1ee2bc8b48fb16d5da09b5b5878e2db74"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8d79de3db2e4b222caabb3435972d7ba997edac0c750b95e66c853c3f9527693"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "21381144ca179f95684d12190580282b400badbd5f56c82ba5b8dafbd7f711cf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "63e88e765685e805c687d3042a316a397a0bf667a18aec226c290847d33712cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "208bf610d174e85c33c0c4bf316753b995e73480d62fade48effb37996304b05"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "725417d08f4865f2055cd320e3fb4b1787c753a8c7d037a32b4aef5f078eb28a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "20f199d9871a4c218c3264173bd30f0fae59cb4270edccd65b2aeea26fbe7c94"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "51013689186149a42adeb6d17e51d6789985ded8cb04b5b2bd7e17140f20831e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "727892a9b46d471fe672fb619aef600e297ddaa3796b6de449b29800a76ffb8f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "47fc45024faee7b16af547b12955992943086ba30feaffee773770cfd7397981"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0209b954629975f0a7c84d1c1d0ca8ed070a9d7907ad6d957f4d2051af1f0e06"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "cbe1bf99c330b010fa178830fab98b19587362987f05802212b43533f0d4bcd1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6d3f302e4ed442461638a14fe2e4228df1db5b9f41872ac1cfd9e8ba844c544e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "004af9ad5650b1694b08cc0fc40f941c9491313e9010e15acaaea2b312ad1358"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "30f01e1bcf7827c4c7461ef0a8f563a1046503e7369fc15dd3ac94c3b5c948d3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5ea9bb62cf68b79669c82c68126ea1c296bfed86b0bbfbedc34cc0227c37d9c9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "141f19258f37daaaa0464ce43f2988675e1cda3fd042de913048624297a4c48d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "78851c84bdcfb6b66bd60eac1b01b7c2f405dfa0f3fd3f1e9b5067daf9396856"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "093ff201db87519037fa57bd4fcba2382c8abc02b83d32f1b1e025557109b501"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ed60b0eaca30d1323daeb1243c3648ba5dc0b2be1f140a2841eaad64faad46ee"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "df5b8cdc694ed3c2a8ff348052c3d1e958568f08894a46be50189c15d3019e02"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "60284ea59719ba93e0cc19d67de8aeffe993c27eece5ebebc7c97ead98a48847"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "36be814d7222d23034b19a84f2c98f5f4ebb18b66df91544826dfc2e4acc30d7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "13b4bbfe47b4a49b7304d747053e1f0bfd96094d919724d22c8fa837bee9dd29"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "021557bdc396bd86207b1ced0406296d008b63ceaae1dd8e7c048727f61bf2d1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "899d89440914f8c980b60765cefb50fd4065d32fd21db01279f07d1cdc240f7d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "9fe19c1cc192bda8e63f33aec6705cee1912e8d22224ef4f5b60b2702d2d0e6a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fcac751bd81e15bddf873be3b9dcbd13b0128949720cee817eb29443e10f9e45"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2ae7a9dd4e86855611312d99f5b2ca848905293555d20dee7044e13f82bdb471"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "ddfeebe1955091095f88e19534af5ac50eaecdfcea32152cd0f2588b7f83df40"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "c88a22ed7a57fd10be2061ee6308bed3f2a9412c47f86e5cf20b14049daa9bf3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "e65fed0ef09f3b22c68a0e5a97ccea20b16b0b12203c1b55bfe8dd298ccc8e5e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "94227325c820bee9a4b1cf3185c8255987d9c01c0d449afec620281275b63b5b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "29829d749b4b770ea7469fc6eb59a1d4fe99c501a97e4c80bb1d1c78a3150a5c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "d4a498e28681762efdaf3de865440403101d06af49c31cc43e4ac23104cd4cd5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f26e1e0554aa230a2e7fadd2e80bb52f2cd0273b32a0547cdea8ec13f0627da8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "52df93a4818d3047d851f2dfd99b70c17217929393d8a573d7d5b034a6e14b2f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "50c688d0a2fcdceeff54ce372da20471008c90923be8707b3a6926b45eec3b8e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1a5c26ca7dc8475ff5e6625f1447fd4196f02f81e44627ff52b7f79d4e111c3a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "eecd66e100d91d89b4f6c8f66292a66de31b7410eb7a103ff0d04cd245b25e39"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "22847c604c6b953f628424aa1beb789d4ecc51c161533f983900cff18ba5baa8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ef6230d2f480af325f88628219b73701965104bb8e40bf027f428c2afbe84ba7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d67f90eb425cb01bc6ffa3f80097aadb9f44cab9b49f811022cfc2a3d25f9917"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9ce473ad2752ff0b6b38e9acbd09399a2ec1cb5bfbca000f5f7f257e77932467"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ceb2358ab56c155e73e9b1b0ea7fbd4cc4de68778a6c92b4a6887e771bbd3c41"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4b109fb2bd5e8a0c51ee6e719211b18fd7131a3e04defb220b486a9d44b9d3d3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "503699bd3e5e544ac8277d224bae9d2b60d8e7f3db95632af10308fa137a1a28"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "be8215fce3873c23f1e1e02cfe18800026e85931c528a4fff764b64bcc864292"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c28ca14d8a50a27f26245fbd75d4ba3a5a6a764aee5daadbb4d0641d4dd256fb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "34ef1334b85b5bac5c771ea94344cb2109c1df8f05fac4b72d36b6ba9fd645c3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4d034aec548fca0e93176e31640c9785be51bbba206e59fb23898e071aedb753"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "66473e34ca1bc5266a5fbf59b672a79d02f0da34594e903e13e413114af4b9ba"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "34ec5f0d2b982815d79f94b2a1712db0038a506e9712616fede278abe2dbac86"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dda92592d29d8ae105f1481a24bfed27958691650ef44a67ec125a1a8e960b66"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e016919ef1828e873a44f9cc1623ad02a8531df5d5e2774023f7341b86eb5c30"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e759970d5e5455a27d1e92526ecb87f0538697fcf1bf01d51c22f2ccd0fc8fb5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e74c7ded7b65b08ce9ed39f4fbca88c4ba23324a684f8011c8c02c1483caadf4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b26d5c3c26dab3d6371c7b5acf7883fb7de40cdac975b3a291f0d9cc1fa2a4ef"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4146c9dc2b9ce470edc3d2990c05cd6230f195af6fac95590790fe8fa11d3575"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "07f9b99324e0c28c0428cef1ae4c75bf0d36e06625b60b8ea51816d296f347a9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "34074c4056757a55ecdbf5c27f437f5ca24e7af6383777c30003cf9fdf708fde"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "be3dc15fd16244ba3f5780be66b9754470599dcc1ff2c47ec8bf586e6b1a07a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "39a66d5e25eee8e275adf3d487248522dbd71d8967558a6219404c24cb5e1f02"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8554ed5919068ded3b9e0f11f484afe599de1e07e081f1d1e790c1cae1c62c33"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8102c4eedc0686044cfbc9c199287602d3425f5030112d5f8ba3667e9feb6be6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8173d1834f0cae10580ca9494b4eb33605095ae2a91d60da6250e4fd8d1c10f2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b1af32ceae799ac0934957bf6ec12e93b0ffdc3b4d8f3fa37c78a6c6aba61f1e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ea8e21812e2cf0ccd6770d0f1f20fa53831603c69661dce2b98c7b0e4e8eeaac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3cc15abfd9ba9dd82e56e5afd47848af9b1cce0ddae40d6a2cef03179d84847e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3869235fa90d58c42b58b9d29292879fe8f8782b55eb8a46d541fd68fd70817d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dbe2ab82c9a5a76ebc83a3b59787e70f4e200b07f1ee3e01d1d5e8efe0ee37cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "02f0df210503542456821ef83f2dfa78fb706cb10015d13fec47d67685fbf922"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "40a44495db8a56a099b2849c5cafd4e8d359a46ab4a6c4f59a0ee4bd11331d0f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3e589c8cea67581918dd887542b4b7e45f937710b88d079928d409bcc9b3f0c0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4c8f8cb7c245d6ab9ee686a9732e4fee4285d0111474d2acd547ebf650e4e90b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "aeb1fa4e24bbbb18b573846ba852e5b8b96ab2e005a217344aaf48b2418d65ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5ec2ded5af835b696596b3c55c8de79157eed9e7d7cf8df5259038d05decde58"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f206bc7abf944ad909129b62974a14bde53abc591b57ae6f6db4f460548f586e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9e7478b1f6d8f847491452129957f7b1a4fca12a32e20c1d51b17957525875a5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "028dbf26965ab815cbe7c3ba36ea5787b28f37fa880721a23d313cbfad5998dc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6bcd4ba4ecdb57ed95a2e4b5f6b0b55044e40691fdb60d3fe2fed5a91795f08f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "aa4d271df0bccbbd39a3f45505cb80e40d05152cca6ab9fccba171527da865b8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8cbc319d8b176857eff05558b5d9934ad6eeb861a4e0abd632fbf16a9c47d32f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "254c9ce83a70c96f7d2acf65f1450709576425c78949bb757dd1a6af89b9a53f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ae344d9d53b57c2b4ee8fc105ca1107676b451d36108de66e1cb27cc2d36546a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7de0a79a1592f85f2f49dd9248b5f5f637e8010353e11a3b7fc55d555844d2b6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c03e128b8f18fe676c7d9ff57b90682b0938bf805c7b6b6ff17952fa1168c1d3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e0f00d114cc2424ddf15e19fccf8355fab17f7e6534047bd59d5283dc680476e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8241c5ceb58586b355b304b7a9274d905c56aa7740902947de5f1c2fcf742cec"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0049f2a54adb48a31d5cb38e608eefd266372bf9c60fa1ebab680a8bc58c270a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ec9f91e728a282d8ec78d2785a4c14eb027a133c6b24a4cbed0f8c5a4abd789f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "187c6cd154e8b471ad812cde0b478c882a54f6c5ded3dbc484535d43ca28f765"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "00a13a0b869b02bbd1fc528297246246ae92cdbd36d52170277a3d1e4586a702"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "0e4c1138d6e163993e9e2855a0d342e85b0dcac566c33726b406bd42a64d2033"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f62ad36b939f3a89937e86c0c612f7848fd45a647da93508ff9aa936b417a3d5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "504d4b252b0fd690515d7737f38d639a3bf80d1640f3a74cf7ceda065b7e2dc7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "ab5c7c79e6a75a6c5583a663c3680ee5f83a5e61b2a1beaf30792fddc276d22f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "64aa7aac280bba49b503821ba4a39519d39c290bcec8285408235e71f62127b1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "dc426e516f5ef3b274616536ec9d40a13110f532c728054105cecda1cfa789ac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "8fad77aaaf2cc956008ac694c36f840a2029d0addf8885fa89503c2ff732b537"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "e003bfc54ff3c1419b642c3fcb3a88826b790a98d976923dc72e7d5861243f41"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "74a79eb652fa2b725d79f059cddd2a91f928806648bd4a90289740699d9a9019"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "80bf0b0b39abe6af1b7b3f1c547138eaf8ea51df585362d02a314f9cd963a01b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "cf50b3b6aab997e7b71c0d91e2478ef568928d7325bc5baff6d32936ba23a8fc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "a5bdeb7d818faa755e536e667e019aa87c212e9dda8671e942ce70f9deaa08eb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6fd32a22565ddc852bc7b45c15184eea8afdf753754f9e85e16cd65d99f7dfd0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3fd5f2b1b6d71af80d647c25ee499cf82ec36c7e4af5a2a641ac3ba05cc2834e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "cf9e7cd07a6fa110c3437a929067173ba84ff88b9fce9f6af9a5427cccc3a0cd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dcedc3962a848b4281292f894dbb68eee93563c4163bf3237b5e58a693ecb6f9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "915b0915b42a762ab734421c29b9e94c726bbf18fb665207431ceef39402cd58"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "72fe9d9876f9db51db569da1086f8821b10e996898dc57b0c2ba9d074e52219a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "3bdd60c8f3b360fb702798a6d3dc3b0567941e86c0acd3de95b399a0dc5c9f7a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "683bb2bcd600a75c529e6a0bf719abd12fcbc1a90eb3b051734839eed71dcb53"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f979f1f925e802f62b6e9ec97605ee6fcbb67ea3ec42f71f99c15e4010e07ca4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "53e620b4bceca8c61408aa6a526aa07836c41c3aed0bcb66d55e3ca8fb868e23"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "56644ba9def18e7885ce4f1c1e83b0db155bea079857cd403eb19a55f147f7f8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a776712cff414ce67f5e00cc87f77825db8de999796eb87078aee24e5f230ea0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9eeb4a7b8d22c5f6f1bd7c86d3706aef4de2350a73053ccb8ece5e49dac71c0c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "4fc67b429dd894ee8f64a695d937cf76f57e4259b9bcfb8ec65beca9f9c5aa59"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "dc2fd3e81131f67925595d444b022fd9d8475116c5d228136bffc07c93b5da69"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2eeefff677cd48dca4efe1f94ab348dcc800726982a3b869522e1e5544f619e3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "abfbdfcbf52e7edf7adc0e4284b1d2fa67ab7d4eccdb1341426534ac820233a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "b2523f8990851b1007f3e342a30947ef7e1b794d3ee9b86a9e70bd280ba3f312"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f22821f266aacb60fcf2381a07ba2d3a4c164387489a6cd6382efe7501d8645d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2d8a22c83d12bfe2f25949c8a30ea894d38c5bf5f16642f4ca52c137aa9c4e77"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f3ab2765ae21f03eb3560cdba059a8ded131c0febced82ee8a9f158fb3291343"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "48b8c1e96f479eb898f83bd2f596277665b2982dc9748d09328625d123f4c432"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c2bd50c19036da2f5a8716f7f27af4366f40e3d5b5cefbe8429f73eef62b416a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "367dac997787f09687bb6fe2ba5de0bfa7599097e37683c5fc091552ff3b6fea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d86d88de2f42a67607238cc8fbedc0659cedaf8d5ce8fedc058f9a30b4047d47"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c9eee4a29de6dc115a0e7ecea27e3a7103d6c4627c80f5b7b07f499aa4137fbc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b6ce5888286a92a7494756b769574a9507d4439956c0203b447d5bf65b964605"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c81b62b7ab60c86f341ac7b9250cc315d473ac5bd599941db8b121cedbf14d4e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "219989cb982a32a25ec76bcad15afebb05236b6678e68751b9fac9fae6d81416"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b48fa423bb8ba46b2e98e5e0d3d60c9828bea341c7e25e521a874e7a50a07041"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ac23282954785bb7b07f1a836975723e4634f25c305a68afe4884ec2ecb9d774"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d21eaeb613a7d3a0091fcd0f0c468ee07f156ce12ab077774a1ed9625a84a860"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ccdb4cda88834b5ca7bf6ac4805e474feb47225d73a90180bc92fa6bf4189f70"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "258e2c722d92e8f3cddff634242b15e397413c0f335e3737bc2e6b9b511302c5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ccb63518a10cafe81963a5c91b4f3685607f775cb7e5dd1df8ec960eb6ab639f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "536e3c297f8f7b7b81d387f91aa2f7653d57099d10cc54cd3e94dbf6455b84ab"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ae942c89a3b8e1579de8bac2f8c4ce4615120b926faf937669d905b45a97ce88"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a5a7110d7b4748f7bd4027781e11791573cf93dd0ea781aca13d7cadb2d81f56"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3663cbf3c460b9148536d2a5840525b20d59c6ab7b99b7b886f8d8800cbff220"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "86cc41ea00ca366a2735b1ef5e8d1cdee904ef34e5edc1131839c98403f8b12f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "06810341ff3c9aa43e00db93aa9571604320237340e8e48b6f7ad507b97a01b0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "03ea365e091a0e9190acb5d909c2be8c1fceec3221bfb5d824b11d29b061e02c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "02e597f9ff27d01602d1773e1ab4f706fa9725378f709359fa1d5e08ea789633"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "23ce34562e454e6932f22a430f809d9b5653367ed00b6c1c6715e6e65d622cf3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "baac07413dbce80e96db0970052626370910684dfb76c0e74af897f24f00e381"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "854e10d15382fc5f32fae02f7250b0356ef524872ff8fdeb52618c4150a77eb0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "65b56070bc6c1b66f78bba73d48c77e94f6fd4bde93925e0dae85f35fc48a99e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "99ec8687548f71478cae78b52da001dbdce3166f43b85d2417066e6946330f13"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a77480528d60e4a0161998fda170076d1157aefb9b0baff305d1b22594c178eb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dca8774a021b5423be82910d8cb006f2a514b7521fd073b967bd807c670dfdd7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "86f9d6bc372628deabd169470b821dfd4da8032877d63fea650b6a4ba8c24c11"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "99a97f3d6f18623941baad915e4e65b5da8cc2446f09edc070a8d0287f4cda98"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3a566575eb01733756530260e8fdc2bacd7d5eb67205a422b8c68ce5298ddc07"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3c4e8b5741242e0f240a634d016e4f1d0b39dc1cb0b2b9a9693340085315e274"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fda4600b48051a472dd734acaab189bec63d9c664f14184483db64e60e332019"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4720d6cd7362d9fd09078ad230ce3335ca58bd78b4c5df1b46162c35a5b574b6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "67e2152ba4f6ce2ab59de40b675a136db9ec456a3e029742db664d69529c8de8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5932368f16a85f06b1d2222e43a78b81ccb5ff3e0dded6de5c9d825f45ee86cb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6361b244a01301057ecae383a75d160b09421ab190ec77b6db046245b13e8ae8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "74b12a6600671e1f1b9e9d2c996b4ef5ca87dcaf1db09d081dc4306eb99a0f4d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "95d62797e79c3e4ab68565ed33c455f4d8818e6ef65073ae131ce2c1ca1e6360"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7024f7a93d6e6922a67b999fb641c1d6b4b2a5fa21b643ee0f15b6e5511ec98a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "659519440ba4306d15e1cbe56a5dbdd786704c68a1d2feaf6fcac0dfb340be85"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7c35d75acdb9a9b279523ee969e05ea6d5a9dd2ffab6112f3f1e77c35613d18c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "48cb3a315b6c4622e525ca348db6548d5766d980a9f357b746ac6aeeb4cec5e4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1f19b3dff169fc7c9000b0bfc9ca5b7768f44479d1139f1f4846adca0793af50"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4ad7ac12a1d218c68c16e84b7aea4f90bd3c6ce9367d69bb0a6d52d656b97234"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "edd3271c747378c64a426451ee79302b94ab1fac727ca05af992dddf21fd8f2f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3efe482c170615458987d7aff126cd3414a08eede9ee13069557cb7d682a5ad0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "eb74e0283cc3d934f76c117d98711e393245336ab68b2b6329d5f99a91d7a51b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7090460d15cf7fb3d56c6fdc32763b7da52e1496705bda60790ea55b23113f88"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "18ee06358946d3f90e2a909ade049f98740bb16d0e88f1dfcd40c197bda3d185"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "83cee5fb48aaad73f40ace240234a4585e59056cb83d3e5b140e9f5d58a54142"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0b74e6678d8df77cfc1a79e059a4a6ff10bf24431a66adbd741400aabf5feae1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9feab5e19335eb3f73a2b9be1be39f54f7aab7f4b7239e2661ce29d550ca7ac0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "f10b6ca7a91f0738f3e10849ad689c32c0a5f06fc1c3ba340d7b427cb0eaa0ed"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4de698b58cb9355497e50e19f10dd1020d0cc2ed29528f2e4643741cc30e0a4f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6909fbd71448266c39c81b13a5ff710c9f7d78ad456ffc16003230da364d7ce9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a7827afe178492bf88d96fb7ac9d441b804107de66965c6cd102a977ce47ef68"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "7694dcb8fb88ff9395d8be33d52f297e27f1ee829849207fc5272a94720b5122"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "60fb42324ba0aaf6ec605d794a1c11a251d567bc88e9b40d8537ea86e5534c91"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "e4d55896e0902942e85b2d08d5cfb4483938372cabf14ef492f201fd083c5f83"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "ca40782f80aa787f8ef725195a21be98165db65f8a20085331c951bdffaf65e5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "f80f22fb65f1154e92b83d0a656a4e86438eb2df089965652f781803df51c41b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4dc4d8959175915f973de20a7782c218bb0f8ea0697563936d180f2705819dcc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8f78317dc81d2a53fa4e36c37443de5903bd443a209d42ac13ee4129949a1ed1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2fe546ea82ff5f3404e2ba7679b5ae9f94c14d574e61472eb20f1d13319fb6f5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "076052b2df685d31589799ca39134eb68a7c3a9ff35788dff35739c4aedbde5f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "94d2e47aa84cbaaf0ba1d7fefbf96a24e7dc16c8dc3f1d0dc8d600907ddec260"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "fc928a18fc6fd8187c25ffb930cbca4020418b1317485a706224d5e7ab60b8fd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "f74544e26cc1e43a089e73235784f57e858086b64ac7109c1b2366853b6ffb72"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "4aa0c100a9c163da780c5e7b9dd4e5ba9c4a2a3825aaa1838bba03ba53b01cdd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8443360d8ee8ad695882fff439b213c50b285c19722174871a8cac6fa0a99df9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fde09ca10c5ffd22ca0687ea0f6085ce74c623dfa654eb52a08f46d552d2326d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e3573d42229de3228ed5c8b722e6be2ede691b8f6076bf02d2e5cdb3a93f07e1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c4e8a53a241a7511c35be7e851769f45144f941395132f3d0a0c8e649cecdf3d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b8ae0f85ac8ec74b0ac05132ae33fd81cd03abecadb2d361e2216c3db0346fac"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e8252b1e7fb8887c7d0b8b8123264f769f05aa0363520cb2c5813906ead0def0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "59aa0a7ca991f2400e2eefbf809a8fa5fb9d890ddb2c137908548b80f5f2fd51"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "62673ae9342a9cf2e19bdb41fe82f0ff7c281e7edd407c9ead1aefd9443e7ede"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b6575df51422be5b0d742c49e20fd76b5460dde0d756000b11d855a828eb2b1e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ee565b1d7c39d0c1bc064c1d6bd876cd708a9764a5223b69d0a5bf3fef856a5a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6ed3d5c109948a05f8987ae29a698ec15b16adc31fe2860249a6d255dfbb0e4a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "88ec9ba2112679e36d9fade1f846bdcb7b6708f3ab46bb8ced136a245ac9e730"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2311d7ecd185343cd91865a2fcf200385f6fd6089b4bd1045c5c7380d49d37a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1153b0a689e68feb81d6a22ddc2647a4b996ddc04ed29ad9b66c09a6ecb5eb71"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5320107b2806ccedbb27730197ccaddf20033b5b47d5ac3b0c3cab9b9f2865f7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "191445f2648bf2533d0ecc31738cbcdcf7595a7c457587afaa3ec95dac9dff25"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "86f49ce5c90d4c374fe38d608527bab5d51e9c5cedc015239b14328f35b14ab9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0e15a9a2bdc9700a908bd7799e87cfff1b95a86e54a98568c375f1da8f1fad52"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "855cf84b5005d9b31268ecb97904755415e0f8d7b3068930f47f285a88247be0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "90c638f27582475806d0211c2a60a2bb807dc44c94521663adb3e16f3588c230"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e12e192a0491091866eb4ba620706f3f30634666984cf93ce2c05ce52b97d5c7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bbcf1aec5c2ba957f96a13f04430f16741914e9f043ea96b9d0d21103b3ce999"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9b0379c16e41e0821e8979f13f05d4b8d4e472eae0dcbd9a92fe79e8c24056da"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b714c2d46f27118c7af98ff86c445b80128c5555bbe7237778386f02c58e7407"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "795d4d4c465b5c87c7720ad328027e6dfde4e3217f487cd0142b9cc96657c77d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "79d848ac9b1162f46a6581444d01442d3ca2648c269f597247d058bf3f500f99"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4b5ad411a0ad64670c0d8e783fac08fb06a2f0c4ed64c80f3b6a21227230a954"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3721c70ce6b6c54b4cef9a0671f9e42aa71487a75a758d45e21437c5884e33bb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "06668bbf07e9041b67e19ebff12c55525034ef2458a24ebe75af1c9318de0ca6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a29a3af4492e504ab8d32806ff2bddbfe9250c8c99c657a7c57e275d6703677e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6c61edf1147851949291f9c96d670e8fa55da6da6777bfba278c4cef07523e47"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5b1db312937b10f31d598add0f2f1c26596ea7381f325f73f3742e1099fbd08d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "179ca99b71e737af3a0519026c3fa42be43c26287f6e8ae84504e20c20cc6d67"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1813d319ec6d8b49d01edafa8f7539380906f7a57bb67b341a61a975df377d95"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e189579356621d3e03398d1b3cbd58ba27b9df55a283d7e2073e1714e6c37da7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1df73e6ba3d4fc797b0fc0b8a2ac657634b57d229835458415b47522a93e715b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ec4c7b1e5a4410e974719eb811936a38da721da09054f08140f26280d6599587"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5df52f5cc8ea0865fcf10aff3d77ea032f75a429495299d753d62a4185d994e4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dca65bc050ba6c5d89251788b526e645c5e6b4b737968cf41a5447725c06df15"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b011b3275434610c4c10453631b8426cb05ad6b58c6019f4847f9215c39b5d00"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4def7d5a610c1bd22a3163ed38890013d837f3af6ae0eb89119f52087880b3c0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "29452e06356b6d2c952fb62bb418a34085996db2d36ba61c5f9e11db7026f8e4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "786ac0e5a7928f9655fd72eaad3d271a488fc2dd9cfdb897361a43e1b567c27e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c0d5477d4e21efe11715cb98f8b3e032b309b7a3a37d0ea731b5b905a341586f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0c6a277f175c2eea899b83798a24a287bcee961b0128c5983e7d22857dac0b73"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a89aab06768c1a7beeb1aff0449eb164574234d779484b3187832eebd28a35d0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4064e1476c96a95406098a4a88f476464783a7c91e2d9f4109b34d394fa8c9c0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "706015e8b73462f88d6267d3658ecf5cff7fa096f9a0256116a6b661503790fa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6dd5317ff1aaa8bfafdc2ddacf0b1971100b8c6ff0009fcf57dbcd1f859dfbaf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4a470d478a3944120df187376c83eb729f53defc53faeca07067445beb854966"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "683bf9baaad86cc1f97a782f68a0a01ca0fb2d2de42455f9496513dfd002e258"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "66fec99623c995823398a411d5ae05635dc1ee116cfaa0b00c8dff5c0feddfe4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6fa9a7721b7beefa5946e4ed7d809b7b3f2d367f49e288a24abb5a0da4021376"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "002ff1413f8f4d0d53ddc5dad3c6336e1aaec7371de377982b18be2734068ad2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5c020d21655a43c262eb836d8c2956cee59503934d9efa56db19c48f2640f281"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8a4835b6023f5061f96dfc9d736e6c4607bdfe2e534866265fbea2ddb6963f04"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9674c5eada86e1ccf63f98d67586e93554b0d64aeaf913a069bd20fe114a7eaf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "768da251d29979dfe9aa6ebdfe4a3cf9d81377dcbe73204a90af52f3b52889a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f99a2dedf9aaee1fea4b0735bb63ed4dd6aabc3045b1440c286df419d2f09b5d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ca4f1b3a48460754965a0e8ce1ae66e135ea2469399aec84993b22bbf34000cf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "99a468a4180fb524f7ad045a05873ac8c47a2bf09839e74d806170c0e78e28ae"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8d576c03f5a597e0394df0fe250c0efe10d3ecb2535188a354ba07a6d1040851"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3316de1b65fa7fab6dc13aff714e8a41020820c28b494a0abfd68b74a2f9b2c0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "380cc753caec6ae0573f31133d0bc20905d167989bf4afff923ec646da5f328b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "55c3ff0bd7f1c08c73ae5290a192b2dda6b3a60fe7a0e05f9d2e8a457f96a596"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b0d90d6a1eb613fb36c7121ef2273bbcba673c041c579a5249fa8d8aa841b54f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "697b7b10267abc2749e7a600d3960a7bed4479b359ff976d3aaa56d3aa8fd192"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "784851a0fe4c3981951d16a6fefdcc750d8dbedb863d026ebc61f20310e3bf37"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8a9654cde82d4186ca69e40a7bc2e86986833643e73ef347b6d2d9465415d243"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6fcefa34ef984947699083aabe4a3d6dc205f4b90473c686cb74bbfaa20084a6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "868ad07fab08c0fe0d293001aad7a196e6bb1c9d966c2ce0f7ce2cf42876dcb3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "dacf8831a0b9226afc87bdd16e74ec2016600d8340e0d702fc107d7e199d5e41"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a4500c80cad3d2c0b0792b135ee151fc6de5856a0200a49fffbd2bc8c73214f8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "166ea72aab02e6d366bb514e853e4aa956544cb81ceeb61fb8aa1b91462332ba"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a45160b90eddb81203f9aa0e6914d956b984b65a4a8a0c5a5f59edd2ddbcf9cf"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "efb65e0ede61d3a42aa1c5fc3f601c3a9178bc00f34d84aa7c4e5822b053feea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3f5520fe87cbcb480f60b9fba3f5a83de1f2e4093f3c0c2659b231a076eccc80"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a9e273c6186b1273071035e171b1d129ca13aaf8cabeccfabbc8066bcffe1e50"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "833da88c5e871ed115b8f6cbbd43b84cce58b2a664e034d1214db711f15782ba"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "def8db290aede7a3474c05a12763b69b649946278cf5b5af6b2987a0a7921f05"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "53a3758b557abfb04f1551944d2017aeb8b1455a7e4f3cab50bbc85f6e09952f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "00114e719798366b5ec3feec3d35eb19a84555a94ec6228b803c31f230dc98cd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "54568f1bb1a8fd61855c73babf8462eda7d2f026911019db28b6e6370b29177b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4de523fd244ac164675a492fb51f6e40ed9e06b8b9bcba2afc41c7fa82dfd9d7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1d423b0a694fd08f9c543e791abd27ab2a4767ca4e4ba85dd1adc1bd2fb7a4a5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "58d4fb9a52787191a2f2d73fd487f848323270a918837f6846dfed4b79851ddb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f6aad287ff7246e5eb5d4905ae5b37e608184cd8046dd557e101a4d01ea98a19"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "13e4c542470b7692661dc2023d2f10b0ab84048f282e2083ffe90236d007b22f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a16e4540b346e18e5e7c9fbbb68d6af8dfe04fccfdda7e9b24dc9989a1ce320f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5bb575c7adf9aa847803390d00e321124caf4829cf1bead868b4a6500b5ff7d4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "95686600dcdf9e598e17d8ee4c17efb752420aa35d7f6b8b126cf1f69859b32f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "15560e3c22b6bf6e39cbb8c5b202c4a32cac77f33e09398af465d63a396e0413"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ff314841cbae92eddd423a3e72404072400abd375b193e5bd2e4d73e3a13929e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "434902fe7e752d25adde39850be630c6f54c277898a7a799f3614f0c1d248b30"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "68c2ff79834ec9e0c36f6a94b58a41a31f1ab025a7d703ddac4fe450d8bb958c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a5076d4863553743a0b0aea486f4b34f5a6a55c5ebd2ff9be3c178433da3b3db"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "2abb4ff9103e8f3616cd53f0c474bcb647f9fe456dad7e3c8d96c5d6fe6bd1d8"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "cdde5f24295cd37b34d9bf69eb25add278d55cf378cd766c65cfdc66bba9e25a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "01fcdc4964bd66430ce9a2831b7b78b708744bb085616ee5d0e4aa51349d000a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c316249aaea3bde251a4208d380bd1e361af8f57c1e899f450160309588d8e7e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4aed3893a843d13eeb84efff474d4498be3a2fa5f415c0c8acdcb4d0805950a4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "064d61fc3fe734768014b29bbbccdfe08f657488ea6dcebb0f547377420f1544"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "5563e75350e9c790241dcee60eb8cc8a244a540df35ded93438ca54fae4f4807"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "a654c22674f9b4b11216a71491cba478dc0903ecef988817a4bff51dd7773fea"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "b28c4e381104183d8edf2fe5752b2e3ebe0e39d7c8c99cedf2810ba046c886b9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "82738c2729aa344bd136eb604b6da3eb190d15c5cd9fd9d1d71d6a2b82c9c827"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "c9f89e0ff6afec98ef9b05020b35eb89f70a6bdae0e07954c7cd685a2fe20ca5"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "ae0f46949bef4be703827775d501a2d28d391748fc8a4b88314b91affa368586"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "206c2883fb4066413df639ab6d7db334e4a63abf132781d70a614b81170aa2fb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "13f670f805706103d41ef1771b17f1eb65135cfa49e3e4235d45c3ffafe7fb68"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "6c94f59d3016ef4c471f336edc06118463d9d9e9b9c97d2d2317d3afce25cd87"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "7a4c80ad18679020fa90570895e337911fb1d96070d3352209e018aca944ae6d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "1e8ca5a9e8db4efd91c9f6d16ad497c60536a091063a630e55862157bdbbd5cd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "692392adb4b4e8319873803bc5825d2f40ed1c9eb7158aca8dfceb1a23b110a1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "9735065aa012d131d98d1b416fac7d6e84065925b17588994c3557f18029e39b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "82c86f52fb9ddffd605ca0954727f67faa833b0d78d805deefe83403923516b2"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "31669bdca80050b27d01e8336997b4cd95cd5f3b797892826e703812913da8a0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a2eaf51726cef44ca33af90298d312d90ee50505fdb3507ed87291d045c8303c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a4c4cac95fcf4cea2ad863927aa909e33b9e9a65fe00407f368b76582951c503"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9d80b8dced244577b85e5a88a0160d8c40f3d33c910b20a1dce9f142866a9872"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f991de040c88d8087f763993fa35133ca8bf9674c1d8f10e897b622e18dd069f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "da749c1cf6493b366676a1641f7673b64b6aaa81763a7eda2af7d29bb9892c73"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "afd0ede277b3017c7388a27be9c075160baee7bb150cb629422e266caed89268"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b9e1ce901c3a036df8c06ffac033ab789fd0b599dc8294fe81dbeb97170c23a0"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3792d21eac507a4dd3874a6bd623397d59615d77e2ee1853c831b6fa7dbacf5a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "042638ffdcf82c727283b1c7da45b274cf09f6b4c21c455a3702e730853fe450"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "735e1d7214ffd50a9815877fe9e6a0069b6a7c1c839d8af71a8052d179fe0357"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b9b4d3fa216b9f33bc6bb4e6552988a2e7d781c45e25711ff1c86544af6c6aa1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4b57ae35c9d2bd2d86b193f95823b629f4c2b6fb56f64e6e37db619b1068509a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0deed42571c0b381e04052d8ad89baa271752aaf21f6a643b6cbb50cd8e5492f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9e997a239514a6f5e348a892bf70edbb66d6ca6af4f94a5798aaeee5bbee52fe"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f66229d95c50f9a6684326c62b86de8d87fc1f379a39f2063c7146ed8d7d718e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "28d75e376db0c7a25c643b680f50a6ff79dafd07fecb9683d9458e91a46d9bfd"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "cdbbca58c7a27178e66accceaa656b93d3ebf33a95f77dd8dd8bb272c072b269"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d8cdf72b8efb295ec2be5d007118c43db58010417bb078085dfa20827892501a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a86864d5461f43acee9bd294aa35e11e26451cd1709cd125cc5338cdd1c3c12f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9dcb9f0681bc91081a4c369e4d96514c86ae0f642d76622823947b4eb79adfda"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "365b24eb9f70c521c42947dcdf260290293af1d861ed8586349a744db2badb9d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "158ea5ac91842a9c535169dac25dea7d6e472959a36157616e33070e3af06305"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "06426913e6f3ccbe6a8ecc91d1ac60147bf82c5f450084a0cd54ce25cb449c88"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3bc51a49a8d01e24412c3ad2baa4ceb84f6f1c59d73d503f1d838c0371a6136b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a96b35bc096b322296a05b87f47826106eb8cfe4bcbe2a745964e15d74297a8e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5e31d42cd387055fd9c9b0d2e64347b1a34ec553586d5233a18ed6f04c1119e4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b22de1202d684a9827c49e3d870d056d1e23fedfd3e2eb472527f5d36434a5b3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4a98405e10a4f8cce21a5c3ad5d378056f7ec71094efb17eb886c0e0bc6e578c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0a59b920e723a14c5e35aff74c456a0bcc79a9c23582ae309b778ac2030a694f"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "bb8578577f4db41b470d2539f31bd650a613a604f53fff307ada28366b17f74a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ebc69a11c415f1d9916e3118166e977d80aefdd7be93859ea8f74060b77bca1e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "b7c8fcd2c5f25a11426ef2809bb0175944bd406f0a0155867b882f39b561b9f7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "cbc09ffc319c713bd9f1516163688c1f46b43053e6adc49a1be4b98ab815e7de"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8a7214b31dc2ec2b10897e4e0fa6eabf223736ac6c693155ffafe4590986b419"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5ce6d0efc93f85fa7e4f932d873d7bb1e8fa53b3a26a7b52b5d790a0da0fe9c9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0f6fb17124bbe5f9c303bca10d89799aefeb99b58f1beef0b4a16055bddddc34"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2edbdb2443239c4cdcc547e24c9ba61feed9bc55d732abd51c92d576ee1dd52c"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "92c9eb0004360f7b3a83cf8523d5c0e9ff3aec3f0d745f98ba300e767f22e5f9"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "174515783852d31a63b0bd24d11edae3ae6436d564d1fcbcf1148520a6fce6a7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8ba7f789f92117a289b156df9bd4a9f4c7a0112352a15c48711084d314cb63fa"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e170ea29645a610d72ff68bccc7b4adff206f9f85766e6d1e4c9a99c13043048"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "552734b0c4b67de4190b7eb1f53776e599fef154e54959db858cb7780659a11d"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "2d336a8b76ff2dacefcecf7555e3a5652aad0ff5cea3faf6e52b2be0b0b089d3"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e45c26147391ba7a69e41f4d6ffcf715123da6879181b0efdeff9b26bce61949"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c46df192a06d201b56cddb99b1c8bab18ba8b0a257f651c05df53cf15eb23745"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3c08afb37d4931421bb003d6ac3bb13ff232d092a96e202c12b9b7ab85a41284"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "4cacf212062a88705dca49a1c8b2f4c378a9c89f3a6992afb7cc1c5ca21b64bc"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f039e8d5da687cd4d89bd4cb91dcd5483ada981e9cd10f2e2a37442330e4732b"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "7cbc3856120651270f264210448ce852507cf0c3dccca5fd84c1e42c4629c621"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "37bdac7a0aa2ada24c8883f65a6635b589f2906e9a01c7448f8d333dfe63c393"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "776d9307180fd2d145a757a4d2869039511a143fce2b5bcea0cc7034f9ef83de"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "74f1531632ffb6ee0d32116217534a820ffe8cd03d3e178bc26559dcb6a20bf7"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3772b81d1dc59d91e0cf869cd270d21b9dd1686cabb13e4b12e5cbbf16eeedd1"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e7557ff9f5d740bec87a3e690ea7e8d4c8825a9ec43ebd44d1bad8d362c9b666"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "03a78d69f3182321f764492fd56ec82b853fb130e67281a714ff5217cf2ebe59"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5ce582e665358e83312d9c93e3fb172f2c602a0cade199035ecfca6acb07b30e"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "256780dfc750fd08830b57cac824c69089b0cc346aedb4525780af99c1508712"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "36cf7eb76b5feb1269c62633b362f1ff81be22262be459c47f2931a94ff9a908"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "6999fe518076383cdd041cd98f537b00bd1b08b1fa0afcf7dd07105212e4d937"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "8e98d6abc574aaa9ef8706889035b6aa074ed9bc8928ad54e87b57930bd333df"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "d51a10bc803f6e0f781b992965dc1c8796930c3a8a22911b81b88de6b698918a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "6d52c97a99d5650a5b3ba7eda75f694928d74e9c104dfc929df499ce57007e7a"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, false, false, false, "4e847ded9e3315f77ade6d65aa368d0284d057538c0d11be9aabe2d90826b429"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "82129712f7d89c08bf2ad6ad3a3a8207ab2e14401091b4bbc048fb1164323dd6"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "b89c495ebd3d45b31a4127ca68355a657802280ccd48bfe829aa10128184a294"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "103584282fea62b4ee409336e575da6afa3484fffe54dde042e689cee0f3b7bb"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "2ac94e700ffab0304c90d3f43988086e5b3bce5655625963589e6cfea97a5af4"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "feec0df4af922191110f7b76306439e0d91afc61a0faa8d4fb934bb721907a62"}, -{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, false, false, false, "9cd8447578a419b9e027d8002fb7395fd11b3c51afcad73326e8a6f84cf5d356"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3c71f7a49bf17ba80e30840e01fcc0906c96cb21317338df038fcd3aa2aa31c8"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e385515cd1c2ccd1f472163d4f1ea66c009400307bbc35d86757c6e743d122ce"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1bf3fb47aea861e9292983be302ae4e57b84ce04c53569bb8c7e405a7ee39a2c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0370fb9b485adec725bcfdaef1cb5bc5c00833dde0a4802c804223f3c8f89852"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "403a09e9ddb8027d90d2432829436fdc7490781caa64e2d32d6cc21b1b6bb4be"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a6568ae23203a00e9458aa56259af97462d0a760399b6a5967e5a79f5054940c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "dd331a69b66e4d58e25336f30eb3851d19d4a6eb097c683f0f8995fbcc704f52"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "296cba7b2b3487c6500fbe93bbb8daa46a18b588dba4bf1094df7b6132c299a5"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "510653114629c20840ca2259e2b0132051e6d6460dff5e346a818743526377db"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b952f49649f8c265d24889358ddd641204a6374729ec1513b263731085598d94"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5a169b4390bec64adabbbc6b4c1cbd8808fb9195c27a1a3cde3a9b53dfaf5a58"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "af7e4aef35bc12a3baae7c2b50a3cef037eb5d3d215ed78f17eee268ff9e986f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ce44f8a00cec5afed4c18fab93fcf1b869c1a9479b4c8e6cf605eedf888a0810"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "198f4de864e36d905154e608fda96c34a609ad39d88665a0e1c0fa53d552b59d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0042417b7f8d25a611db61d0a52661ae4eecd24eec2ddb12b39494b885bfa6e3"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148800, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "630ee24de61aec42c5ea05c1ca92b12112db6495141c4fcebebed7dabc73c413"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f51e8c6769b4f10cbf6fa1446be753e63a59a6a36f17ffb0b138482e8b00dfbe"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "5d56c8c7c4c6ac73bc3c0dd36f1a28e9e75cdfbff5042b3bfc70b1555c98aaeb"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "742c76ae061cd37a5199f1fac0e478f4cd2fbed0269c59dca55bdebab696429e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "215f0b2d34033e47b578b40b60851822a0e4efa4fdab83b20e5efdb261e59b25"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "13dbcafafd5ced0f688a60bc2f6876a18ce657c8ef80f5aa92627e34dc5c0a4c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "bd20fefc86cbc3d7531903523a326a601210c1c34f8b0960c0f1bd82b71a8700"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e4bf44d0bb2e20443ec4f36c2cb4e91ef94989c6d505860a6bf0d15a57d2f4cd"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "cedaaf8918ec898e9a95034b87c1220c3c180a4f930ec2b7f9a8c66d8f8b2b2a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "8df40709cfcbfaf43c24e03e601748b2c715e4d898010de153c739a85c5227c6"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 148752, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "08d596c1578d0a86dd06da43ca77393b7efb362a720f93d266144dc496e0eabd"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2388c3dee20b2172ae5276e48581d61baa69f8a293bf36b2d54cbdac4b4e9c9f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "02f4f1b54019a36d23d591f4f99fd8a70463fd6178130520039a419a95dc4a16"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d9e9e34464c7bfb6566b3db53a77e638bce2a452e0e6c70edb3c55096a5f1897"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "26a6f02d0eee4fa778bbcc14fbae9e58147ca41e71d24c6f414db091ff94a77d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ececaedfd067f61e60e8f3c6f65ad669b801b98c45e3acabdf313c30db97cafc"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3b32773b37ccae86e1bd7165bef9f1fcb929eddbcaeeaa9ec3964372e3f0627f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "afae9eec9aaeb9cec4334b3a890c9925d5326d530125267743d93efd3978f825"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "846081dffdea40374049fd82fbab19e8aa39690c7598ef87dc0deaf8f9d20709"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ddfb1fd21c6d35ca3acf75d962120a77dd902a402f4f91057a11f32e6c9ae3d2"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0f552209ede77db8a19b7a195c32fe88009b6ca0061d8c6819deab5c37544301"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "638bd9f72f199e38918bd4b311ded987faceb177ed9ceb6a57aa756c6c663bac"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b252e87ff458188d53933d3d90ba990f51aa724defaf057da440f32feaeda137"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "ec9610194f1a80b8b864a1e10312f1092a76d352c85d85736cae8e4dbca69b07"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9f97e6910efae2adb20ef380bd622279ff6a3d6951995793bf5d409ecfe8472b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "cea48c52e4d2250bdeab086edec1e00aa912bff34ef12622d7b74f4bcfbfe389"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "88530527d07a12631d9a67c497d95b4da404b78f2fda165eff875ac72f11b2b6"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3297bbf1768aff5eafcfc513774766cee10a54ef3d0d4221342c148a34bff345"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1c54ebf7c7a05bd07332d095ed3dd1ef79783fa2c76da2d30aec5e2916ceae92"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "f50667f1adf72779a15f8b9c5fe7c613497cd989cf539bcf00c64cfaad57a7da"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4988a10b9c4dca21b3d5061eca715f9f1cc349825a4b31a6ae08851a4721eb59"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c246bd6a009435eecda6dc661ddc68c3ed5757449053c8ac5df05efbfa20a695"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148800, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5acb1d37370d7217cbe3773e86448f5c66c369200dad36b914d73422717ad859"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "283578b3f92cbca4ceb77b6a598ab7cf4580e909d0a294f8354161a758d2627e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "18d9bde9f23995ac846b8ee013243df7d71ee06055ff46828c0f6baed45f6b14"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "70dbe45f87f376d1882bac95f3abafde71c19b9becf02ac861a53924939023d9"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e5f566bf0c931320a6c72cf0988445bf652e3250ec091dadc61b67a93bc17bc2"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "5c09956e86b937bcd4de0a15492bdb963af374ed75065960bfd0b8a4ab0c9ad4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "93857a15e8ebd8707a9d8d8d4a731b4eb7e3255a078dbde74ca3039091935564"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "2e7792ea0d99250a1adf369338a6bbbc60c84b8bb8e95151d906fb73e59e0468"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0ac8855f83625aa67243029107346f35bc2dd0213e7f5cd11ee4566a87b0c7d8"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "56cd88eeaf5c44186fc1dea0c9a84fc58c55ef37b65734a8ed06568a931f2cec"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 148752, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "55b1e07f1421002294ed06e158cb10056b972d5ada6918c9904ec6726ee20829"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "90a871a38657c1434ca44b9fe943e21609cf91fade34fd7153850712fb3b6c20"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7b31d2f8edee191975074b5d3ceb3d7072d5949303b6c7f69eb5adef429cb84e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "16afde2429bbbb9357d5b9ecc4dfa4c14d8baeb89f05ec8dcd63bc63c1232f6b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8ea3f87c1ed62f55850a978580055ce35f3c4bec975d18e5392fd6ca6225075b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9f17353188c699cd07fe27a6eb99e7e52f5b76cb09b12f2513c6cd1529d2f4d6"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8fb8200dbdf14e6497f4a062ed4723cef5bef4e9aa2320051163733127fd6f34"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ca5849a476380a72f0c3c4852c868eb56677676079df31358fd812b541b8424b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "3b5464906df2a6be3c695d33ad5e6cedaca59bc147fdbcea01e50c3f24fe31b7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8f0186d80b557a25a05bce279d8783c6bf903e70c8ffd24e5ac320afe5363786"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4f7fc7d0a6251764b337bd2125e1237c0e9ef0222632f26d8830325d9d3600aa"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "007aa1a372680260b52cacda4a93c1d3569da8023ffcdda18f58e82756bf1201"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0fac884927d84bc8f85a874898f92632f458a305b4212bae8d6ab1b98f112478"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e038e4f56a894268e41ec0aa2fe1054faefc433f78b9d86cc60f3c9d73941a12"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6e2c63b4eb9bfbc8434598c1cbe75c5d5dbb69a6138fd4ccefb594d00b382528"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d6c9514bb07fad05c9ad201f0d55b826ec78e83fcdd6b68ed0590fcd7818af0b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b0e1fdd4b89d9d138d2412215f583643239c3d2abde24a9ad73bfe170d10d89b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d1e8324c5565736eb9b0d5936a4cd3ba50929a4df2b25f9628b3a682a8ff7687"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "a3764bf3846eb700079dcd07042f9ac6a962dc750657529b3a851b7fa9914c46"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5253f7df6b3ccd857fe892b18d97d94a43f5f35f4d3d6b5101be751c269e672d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "135b0d5fd1e7cf3020c2885a930b1eda1446a3adc4627e8faaec1a2a0db80b66"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "e30d6d981f02deba3c5591eb1f1aef599d5f1ae4abf43ee461d043fe4090dd1f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "140da7c6aa927b07b4792eb5a6967695c1a99722c7f51f689af2dd7ab08cdd5c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "56c82fd71b1de3a903eea09694a6a864a02d0262abefe9a4e548400f15bc85ff"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c1b1d6d561a00592fba8e589a11afd2dd9b0639d025b90b87bec401a05189344"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "de18ea5a90d8dc98c8a234ab483e7b6f89e319af5eac6bf3d87f2965ae6d2dad"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197952, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d5e76915597131e0b746748527d6bec48066131a6ce104f13a9d03aa75fc048a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165184, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6632e34288f2b71f206294a99299742c98a7df3a7b7d7f3b87a5913fbabf6e20"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "65dd75017c844f3cc9ed7f18585d935c73df4e2718a316dd217b6df0d88b8825"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "62b554f56ce12831f7ee7dbad2a8e3bab9c4d9193aa22a40b26e49278c741a0d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7c76db08a227abb189af6b46565dec704d15b670adbb934e4bd1a87e41179ed6"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "19b765ea65601780e55e0ea33dfdc6123009476d361dd8e6ce2942fade7bedc8"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "36a8903cb3aecd2a593c377172c19c5a142bdcd411c9ea09ad9655812823d063"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1dcd85700e996eaae34d4aa266ee4ad244d78be491367c5bda024f4bb4bd813c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "1a52f6a4cf1982ed402411c8aff3c7db6f14084d658767aa38d3895ad775fcef"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4db415abaf409e18836ff6456ab1b91ea7815e698dd4ced5cdc2a045d1c58c19"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "094d9f1804011a005746ec88c060313fa3c33360d850a385a0424fbd68118967"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 165136, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "103fd3658938434fdc1ed0537fb6b7a0ac5a5fd2ce5c2f3ff750cab097d6d461"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "865e75cbefe17de196cdc2b3dfc1fe37319017d815cf8de915d3b5d5707d85b7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7b11eda8cfdd4583595d4f69c2dddd65e6133a6871cdd11a4e7f7eb34a342853"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f46763594c37f88d92fc915abcee3a989421cadc08bbecf8ceb5fa10db03e69f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "edd7d63817d7df526bfd26959528348393088cf4cc38bf2d15cda4d132f9ad6e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5aba33f87a7e93848d261ef6e9258c1254e60541633fd5d38af88e4d9ec4e26f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c248a5bfd1514aac87da3fff5dd4df3f67b3f7d88872c96f3b77634def495c4e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "35795587109cbac73578852e5a41dbba377dc98ac1ae55151a7439021d011084"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8262de00f6c2267e4edfdf5a13a3b631d8909d5965c0b77d997452055282386d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "ed6273badf5bee730cf6a6d8036fa798240620649c4ef8901e64c4415703e464"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "627e32f31d4f08ba11f8f9715dc7857bf7cf15c8b49affb0edc0f719def148a5"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dc35d1178d4e694602c15fc6d9822906e8055c70dd62863de4bdde00922f07b4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e3b812d44b84ad3ef5c6166ddbd90b3f0ca6e648c31de72c6f30b35b1dfc8bf0"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197952, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "89dc8b2fd3253e3f59a6566101e3532bb17fa321bbebc178361c929d7c96e538"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165184, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c2e66720c6c1f58e3d924ea84c7f22928ca5394ca1b06ee04f2ec57ac7c55a29"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e9899dc1a30408e7f670b3db7e4de0472898e2a8322de3cc8cc39afd3f2c74d3"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "4ef7a8684f81ccb6bb086af1b27acd8551b883377b8771791874a452a9be97ed"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "c95c5fbc0115daaee50a6340013848a38d455e2fc1384480eb1e68c9f2c9bdb0"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1e701b12bc525f337753d52ad5e21258ada92629d33c1f9690aecbca1e6ad8d5"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "685f0e1fe7eac86143fbc790f61708ef2444d084ce8b1fdca9020a8ca1bf60f1"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d8715dcaf6e7a7b63ccf0fd014c5e600556f4033ca40e3f28903ac3b1f450a93"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "440d2d9d2e28b398af91d5463ad7c7672d04cfec691af6aed2c183e2199456a0"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0bc43eb24a63ceafef7926735d911b544e87dc00d277b354b7e44b7ef70d668e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "1a9e9df2c60e2facd13f96848e083cb2413841e8d92dc5e72c5e4703cfa92c1b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 165136, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ded11d0189ba2a899c11ca6cbdc9eff87cf0a452cbd967863a453c37bc1f849b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "95c0020542c6d2cfb530de14bcd60b48d04fc4e5a178572e1af753ad919e4f3a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "4d1e5d8c8de655763eb0bb3cbfa4f4edf1408311849c42a3142867eb688bfb1a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6bc328fa86560bb56303ee0fe77eddb15aeb8d858a96dd73e328d14fb9d2113d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "49524b2b71e621c6ded36fea16904c89414904d2bfcf8f99b5dabffb36c5cbf4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "775b04244c9d3497b4f6eaf8a04e43603b28c84a3c5af7867810818beb878203"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a15fff149e83ef7fa7792926e2a903873b97b628f31fcc697fdd779731bf5ea5"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d763c203fbe8cf77404ea87720971601f825ebc9d6abb0e425e30d3f450756c3"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "22da3adc3c0ccd95843e00f3cce61894c710e894abf2ce058b97c60f95f4201e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e73c404d1f3617a58489dea5f92029cf014517a596494987e3526f4e0cbd07b4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2c41997d0f38e50da9403c7ba1d5b876d071ce314cf974caf5fc5d5c187a8fe0"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "a6c5e3b57bfe6da29e5ac802f55b2651b9ee073b0dfbac1e14f76675e8719c51"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fb243b3081c409f3278c0653ca620d89144e5eecd770b5c6e7e1f45843f3e9b9"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f8c9d41d9d2d443b7c4192e881fd1e5d883034b5470890758ea009d8c556bf4c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "5f69aff033793221a54aa5aacbf4835c2ca559cc39ee7ae5c3cb7081b1b56c94"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2c244e530dad918cf47dc57222cd915a6babfc3af38686a69b2a9df888a4e983"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ce7edd66a2444d4e55c7e176e41d6dbd8d1e2048fd096e44f8fa59bae53e87a7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197680, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "09ab5569a23831ac0cd33f73c917623f7cedb5f734788575ecbd4a618666ce9e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "52cd6382d0b6a0bab5f544138fefb094f85c6070c37cb03254bdfbbe3b106726"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f586d33c4005e7196a3cdad813a3fc27d33f845c1b94a92874795944e0d6cb93"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "291881d31f6958a01c2304f1c52ded01a95ce8e172d1afd4870909dea0c97e0c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0a6be59532e298ccda079026330b07ba00fe5ef53824ac00acd60cdfc99d3a10"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197648, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0811e23e48d422d2fdb9b452cff605da399f4905f3891d4576973caf07838811"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a83526d6527123c1749c8e4d4dee661b847775a09ff632d2bd0c4a2e9bc8f3d9"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c78ff60cb6d3d6b77fa8a58cdc7923b7693401a5eab643ad04b97176208b9689"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4cc81cb4e1094b1bb3c028022540a4300575e97cb8185b109c019b7b41138b0b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "aa8c47a23581053048635aaf0ce81089eebeb3861d4ec4a1a21f5953ab0c0a55"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "1a887d25ecdb9adf184d5a2aab877c18a893e857b291909312d82fbd53214217"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d28a82f6f79728ad2c3e9f295d287ed401091b71f9768f3ac68ba11d8182af17"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "0f788f59346a35bbd0116b0fabc817e8dd5685ae65737bd9c90ce6df90804eac"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "e187ad79c50f9d741913e96f878741e0fcffb8cdb2209a773e804c0f864d4c34"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "402d201ee80f71269285725061c87473f564e40b2be510a6f39aae2126204d62"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f866dc54582a62a950011151d91d54fd1295422400a71605d11343c84cd83988"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "fe39ca00c4768e1ea6d8ac188157b81c14414042787ed7b3ff9e5a4dacf80cfb"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b66907fc650133c51da6765c1fae45c30f1ccc5d7f14cbd3f06f3905328e7c22"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "e3465b23c79ba86363d9ac00cac2a3964832e6267d0d03cec47ae17470f38346"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "42dde8966796b071056832d21c2070a438e9ed2745f6dc918c671820628e8082"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197680, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "8cc486545c3df186514d387933060836dc693b9f152c1243754ada9f98229c9f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "167e72ac244589fa7a5dc702d70870aad84c85b7598b4bf6593e8a4a37741755"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "dfba6bc27199c77a2f7c72ac4bafc11f75f055f3426fd041a22daaeae2dbe69e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197648, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "229fd6a840b1499615bc219abdc30c0aad8bea1e54317f8cef2f80b9c8017c74"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "fcd6490a06f970843f4fa14f558362a55dac98459e8c5352595df25c1309fdc4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6b2a8a6b3b921f402d9098831cd3f4d09628fe857bd8da173633317dfd16f565"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "d148c7998d1e2ee6f9b0c63c4fe12c1253e325293e3a112399a689dcbcc83682"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "93387fa095ef84c60ee8a439dde9848abc6c278264082e4724e2e7957a693080"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "83b54cb83e99a334306414a1dc1db98870e35fc4f0b218149d1592373fd2e38c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "11b4a6ee19faf988e76e632cb660a6e6b83c2c0a1f46d4bce658d9613208f2b7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "23bc00612f702de79bec52fbee5dfc429d10ecea903f2da1d50e5d23d9a900a1"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "6783da4a5e49f58330575427483152b76194ac79998f62c95fcef926760e47aa"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "6104fa394477fa417c45f735cb302fcc42bc72e9b9874129d2f9997d19e01538"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "554241691ff97f2824e53abde57c2c63709932a78debb15e62b012e6cc3b5abd"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "54eb71c5b2572e5339c57e0c4e53ecf527b094de8ef90f4fb32f145b6e720056"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c3242915efb87bd1490b389faa4d5bf73c9056e360d417709926b7ae4b23aa65"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "0453855733202fbe6cf59983d07ecc3112f6590b7170e8c1659bd886ab02ff3c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "545fe85d7be22fdb4db0fc92dc60e3d6d125cb324574b5e3a9e74877d23f4f59"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9fa9af8265edf7a3e77c17049809f4c874a6c30d66160a32eab7cdcda0d91bad"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "8b5be29d4b93df69dc046d74bbdc2823054dec1b15a6e3e2ae47adf5afa4351d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "8ecf1d1f2cffcfb10d8c97076ed6dda4687d9098a26f014fc981e3304b167285"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "eb8315064d0bc1da3f9c26d185e5408769a01b1dac01669e8079ae39b25d4842"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "87ab8d702603e63ebbb9b735f5ce76aba63a3e5c570f470bb056965e641f10c5"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "95f23092f950e9550937b656b579cc7573975c266946e4c1147efcf6acd32ad2"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "5fa0fb7040e5e1e85a9b73d981dad2b512c7d4ba281bda4e9dc78e7b5f2e6d1b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "89b0bad8ed25b2ebf98fc6c8b331fd04bfd5e6c282cfffc2e7e6037250cabc98"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "46c623584ca7f7fa823fa8eac189d1e1a7241b4114e743114ebf7cdaa594eb29"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "9d21dafa23ce66cf06a7665a6e403a0dabafd984520b2b46d8e583d3bc3d9140"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "51047d49348c257cab45cfa6b4bcf76ec100598ab0ae47e211679404976fc0ce"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "a8e512c33c5506c70f1859fcf1eea88d90ff1b68b400ee768fc89b52a97f5cc4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "9f0900137a49c299e911c51460720624211d6a36f1ad6f1dd280b90336e163bf"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "005487b5093734a88a39bab65316e1d334bda8400786a72a521ebc841bf2b312"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "30d274355a5f897a9dad0587947c1cc3f472b849455f0a72c080a0b60d512bac"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "2379bd4e5d4bb1ca22b8cc86f41d59c370cef92cc56b4f6d311c3c5a7ca4bdc2"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "ff7bd5e244b52ffc9b8a825ef7a58ef9fb0d0a13417138c203c41aa594ee64df"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "3769f29c7c60e69003ec9f22e15949e07896596b23123663eb3b6c27207eaf35"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "4b16aa45398aa74627010c4e19418e67bafcd9e1aeb42254c847ce244915c860"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "76fbe1c4df139f01bf6c5418ec51a4fbba408a92bcc8f6454e6152a08da53011"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2712c8e23c4bdd1413e428031ababb914e0ee42d1c459a132dcb29c4b2d09fc4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "318232d080d1500fff46e471c3e54947632d17442e9a2b0d2b6a4b6376faf7a5"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "05f668f6fbc9edceb7594c529b05953b5f4c4aabdc6a506b5f6847b8d4e747ad"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "690111d5a0b1e9cf9730cafe0601dbeddc19d0b60c1e8c062c2f95b69ef8e534"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b8076938dfc4105efbc9697048ce4b34592c92ec0b47a3df1022c84631784abf"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "66544e50021e5fbf1cb0dca96bea28b254af81bee3e529aea32b3ae6aebc9b47"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d943124896834ba0fe2501fb0588935574e5f01606f318d28c2547769a2e8e25"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "010460dc07589e69da57d4f23e36fd460e78a654e8daf75d3f166804e9bb2180"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "b3eb7f6ab946bc64bb5d754f9e52cdb30a0f6c5451652ba776a1ad4791353ead"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "38e1c255bb84ec8781f387fb4b4da142a288ad5d18e1eadcb0980d21aab8452a"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "3901090aca8b411bf295cef4068e375c9774c25f7ecba4a6d5d7ea2c390b67fc"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "2f31d6a17560ab0b40dbadc76ae3afb93b9ca505b6b60fd08323ae2e2a124568"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f8729a42de3b9b0dbd089787314735e2bed519c7bd4f76e6ed16f64456587e03"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "d6d4d753a9ddac8ead7c4cee15cca40f9f9dcef9684709da704f259adde65274"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "2bf70fa6a7adda6ac048320fe3165cb09704fd713e1466adcf53dcb815f6dd17"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "32e30d2138fcfa69fe2a37d7019c93b4cab59fb7c9d57ff6101c0f37bb08dc98"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c228d801c2aa001b09811d6a09f27c67a4f8ab5c10a26437f8f3546de1ab30e2"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "6fc2750ce52be14ad0dd28ab767a561d19705c94153c1db3ed0b2b65546b7d7f"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "1ad984abb54a2141d7384e18961424f409daa6a5490f891145ef6f7c42c4344b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "448ee6977bff35327c42a1e12883c6ecfe8e0aea9b3e564cab38b2d391b1589c"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "0cc678c306b6560251591a5dbfa7b616555381d42cb0e3bf8bed2686c3bc7617"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "367fce625e7bd3a147c8b49e5496594047bcd268968162eae2c4e1efc70f8f96"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "9912daf8a6957fc4f5c68a80921ec522a8a84db4e0c4bb1aa89dcbcf800f5b24"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "65ca419aff813b77d8783aa56e43e9f9a41e06608b7f45a1db5bdbfa9ae27282"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "006904cd89cacf2fc60948c6b0c1a51ccbb0e102bf6cbdf48041358b64517982"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "a5735da6b1734e55634c2fe283f894a14ff0bf0c3b06ace5754d63a2f62f0d41"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "fd2231a602bb6f4b2982dc7a1566861f9e58d788901969fa63c7d78120983075"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "d22dd68e1b11fa4a1a48e1c917a0a818bd357c6258a74008d2c863f19df61d18"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "c9558d3c12ebfe6c70d53e743e73c8311b7b57f496078e1a7c68896debd05939"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "24552a840e08ec9ae2487c429baad2cf81ee6451f9f8dcb0cad4a77743ce938e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bc394bcc4b035cb2d90f1517b31709bfdb7964806b13a7315c7eb7ae51d36e83"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "6931db28c828ba21640f30418d96db9d3a21dfe5dfaf69b474d6382a8f3da2d7"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "e1f03e5fb1209e1c6d83e36946a527a0d5a4370cf66fe274cc91b25ed7e3e473"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "7afe74d9d5a5db63114ef72a1ba4500fb278ccc7fcd8bf69f36f40816f95424b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "93be500562a954b7292e0cc77e6e7156f960093482ab82c2479cf0bda221984b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, false, false, false, "21ef0e342d56ee6bad52d366ebf0527ba0d68e3aa8746c11e2b0a3859f2c52fd"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingWindowCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 4, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, false, false, false, "61bb70ab40798d0f024b1996507dafe125a56f9b63106c541d7355ad6d09bb39"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b96830f94b3a2610c14e9ab55c208447af67269c3e0e34d23fc14902c71c99e3"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "87b87acd20977468abcbf18e170c92908b2299e534c6f290924a847acb5d5f26"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7ccfa44d8651fbffec632af7257a7f454ad275d6a75d5011c2f9802831541664"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "b5d28df6719eb1513118da7ee9bcd6cd1850821f74d3de5f58ad5fc692e111ce"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "71f7db659ecdc1b8e915f33b926eac475e14a1afcfb1a6344a7c065850cea21d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "bb41494344df81ece64f889984783c613a02d84a5d274e90d40c966c3ca4bdea"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bb966209d95720a0acd4ff30466c0c8c7d0f5d3296cdcb78e930dc85188cf3f9"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "c34418d7b8ea0787c1bbff0a0527cc7fb4be3022c402283beb63c1c6a2ee97bb"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "39cea29c77e493a21c78e1d8b2e10ddfdfca47e4380fd720f625622ebf1b65a3"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "c52e142f48282aacb759a1cee7508a3a1e842f065b9f6c17408b5fd8e2ab6d4e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "7a0caa41e0daf9f1a7b68437225e19e768860e59b08c8e3c0928a4ea8a73cf93"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "d8d7e0c6a534f7faa1e816af5a7c7ef294c5a660d1f64e4bad65b4aaa5a5c0ee"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "14d12b516f3e1915de727c36d063d12f542b39824ea5fa74535e24a2add35bd2"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "0186ac4890a92e52e0eb842aace85c8fefc63180f374131dd38c6dacd0836a9d"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "3d2b8aacfafa927959480c8bdca33b254b8f2528b46ac90c8bb569fe28fcd513"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "542cfb6bcc74e3e9384970be51763e1bea236eb3235f26aeb8132c683940d99b"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "854312eeda613c35fb3ee3b30359251fe88cea0bcf45577a067d2f2b20b34427"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "ee96a4bf140d762cdcc3da9bc90573099a0f67d50533c89bce3fb727a86def82"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "bbdaef50b654b79cd689755f1a09518298145d37fdf65f736671ce9575f2c727"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "f9776e930844b5eb47f8b5c0e1a486667ee478d3e1be94fee8607e3b951b92b4"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "b968e7277205a86cf840cf2cae277a67ef64e5509b04f50e06c6cc5548f1b91e"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, false, false, false, "f9aced23bf5ec543809b46f2a113e98ee33be9ab9cb01a457d26ce28466756b6"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "33878e0f6cb0682c2da313dd5698bcdd2fe82667a4a151f5ec73b5daeb2bf284"}, -{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, false, false, false, "9e933c3b6f959ed74c987c7079510741c39ceacd457fdc6d56d49a0ea631539e"}, +{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 115328, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "68f3e93004b7b94483b416ee00be948c447459698ce9ccdcd6bb7f36d4e79db3"}, +{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 114960, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "315b425f3501a7a415166548eb48b0c434dacde3670f72123abfcd94f1281707"}, +{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 115344, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8c05b01fd6d90f5ffc1ecd5d9eb421f20d0895cce1a3ad41b5c26b7e2f2fa01d"}, +{ DATA_TYPE_BF16, DATA_TYPE_UNKNOWN, DATA_TYPE_BF16, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkBfloat16VE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 114976, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "af6f9aabf5bdaaf678774c200e9229487bf96f8eee512f8414845f9bbb8c8fec"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bdf58bd8c758ae08d2eccbc191693afeadb23bf4dcfa41eec81e8167edfccb61"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d7b8e405a60301b2264816b53f8227496520afe414a9a7660ad4a4f601acab3e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5afd35ea347a154eb949e2ed42c405ebaaa99830fc6b0c7bc7a781ca7cf4298f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "745868896d4d30328e29ea69bd4d1fbb6d5dc9cac3ee84813b62f62aff7dfb9e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "dce46aab1d415405937139e3b3d22fa4949f26f6f5d890470edbf022921c1c97"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a953adae603b91f79f2c020a54389a06556fbb2f4684aa816e36c53810d9a054"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8b3100cfbba1a75de1a772b95a1172cbf7b273fab473625dacad2ef43e54e27f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f9540a505f453e2c459ddfc3e9f33870b1a15cf374354ff4648a10e094da52cf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "878570721aaef6e13d066e9f2d51ee6b1b58cece2bfacb58c6ccc17b11805136"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "36e52a647a02c9b20f157822b8bc1215ddb077441f58484a816166b4d0e99071"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "524c364f4cc0a7155a0d52b4dfac1b9e8fbd4b4c0bcf86336fd472f77046ebad"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6958f68d45606cdaa370e981ee9a78e5d40f86eca388129e54dfcf030ec0200d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9b8c883753b6cc091844cd1a0c97ddb5364036374f34dc3a48529c79ff7f41fc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6360e054d633fa967da7b5102961094c0c6da78b1924c3986cd47d6c3c44594a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "81d8610469bf7fea85aef96f28491d4c7926894ebc2364751de9ab522e03276e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148800, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f9f029696bad53b2aa98548c02fca00142cc61fe0e2b053ccc8fb5757d36f382"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7b4ae0312106d8b28d8c2f7ee411720ba6f5d2823ba4b162ac59d7e9997d0591"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "52608ed5eb712ab52e19a510192db01383f6367d9b7283d5d714a48ee8279320"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "46b99a7c9ba34261caad710fc89893106d83e3e7a65afd36129b7041e4408936"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dd9e575685f120e7b1881f8adbcb48b72960ca8a88955f5c7fc9671db74c17a4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4d29fee99ac3a3cd9c208e189b6d2f24e98bb1297126a2b70627a8c7f501db48"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "45476ffdae8e87319b1cc74089d7d62f5da841b437cd668dc164f7cdf37a0cf9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "552415b83be3e5d9fe0a30932cc2018c84d0bda488045e4697d15ceaffd97314"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "895b7ceace153b85d8252379cf8aa51a17c98891e0c52d10e936b56bf1aa03f5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c5befbdaab121de81bf9c38a22a1b9f64337aa8c96d1c0bc58719c2e34748452"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 148752, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "63af5d07ceffbff2d0e097cec6f90d36c44f85c127207078aca7f0a080dc5d5d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8aab39e5b666bab82f2e6e405e0527a65e639dbf0e454c97d633e0584d283a36"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e49d3d004aef4efd613c053478e2d984b114edab2bdd6f36c98aae85c1f873c8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "de504d2a3022d7878f54fb9440cbcd6196d46f628f71065984e754a8b3b8d894"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "35bb593e489d4bddfbc45aa52e68856d0a145af8d418d8a4c35e04eacdae03e4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b86f8c50ccd37404139cafd3946c9694ba0fc7655a8216d571186d7e23499c53"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "528a207b216eacadef28eae4ccca6e45479403919682a30815af7146bb6fcff5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "2db1824115effc106908d5a0fe38c25d3ceb744f72bf1de45b89a6c398ce7cad"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e56c9930fdeadeb73d7c248c02b306b3eeda91ef08cf91b85be04b87b040f678"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a3258f5969d7d1398a04c193d18bf59bc67e94ca90d6a9752fafc3d198441b40"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "03c0cca05be8a53821864383cd0c66f40f729a4700aa03cc86f1a960a6397aa2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "cc161e1e8b774ede8a05e310bf1376ebaf92009d7c6ce8b20c4609100a922c7e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7dd551e01bba7cb8907474709ffc2c3f1d1723ab611a47b60db86466aca25d59"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3110f319c05b33a15a4708d838e178f9413c3e994aef5c4da4624ba6246b0efc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0df82601193afaf40d86e94540f56962382d067e680488065ed244a9ba2a9a79"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c6e12e0b80daf559e230f3e425d0f956b179af6f7280abeca67a65f835365dea"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "23fa2600995de6ae6e45ba4ab9d240746c9494806f390064cd7cd27461d429f9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a141f26613226424515f29d052994efac251d276f4260505d1986ca6da551272"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "601f86a24cc648b0ba375bb15c81aab5a43cefdf4eda164dade7108c4232d078"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e02fb641c6c8ff418c225496f3be0f2c768051c428844be6658a8f519c19010e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "129a24ab116620f42e10222f4db4c7ccd9c9c6e2e63d0752fa026a3b6b829db0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1644aa34c66965916849a088eb2af7b2a3fa43ef385ca21908f64b52d1624333"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148800, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a5ad66283619c98a117c1b1d240fc21aad3febe8d1f5dff9087251f7d5521948"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d4167aca9bac6779ba18f71fcb03bbe2fba468b269f5f28a0aa38e54ae29a220"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c3232e21b93a467acae877f01aa9565fd7b987aa2a3e47f0b3cecd0d8026ca49"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0a92a543ed63e3140b20339d07407202aa000d088e13930f250170ca5c1a601f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5b197f0e662f347691dc708f5a8ab060e34d0e42e1aaf53aea6d0ee522aa6d56"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f71ff207565b0fc67ebae546ed3a3b2b20eae85ca9d4ac8da92951ba401ca58a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ef4617985616c8e80d049dcedbdc8d383d832ebca42e6a359dc75751f6742beb"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6d6b864491fcae40f75e64f6e0970a5ae102be931d043c457c13e234d02fa78d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "2b468236c8d72a87f03c805677cd951d96bd953ec21bee83503609d0d01f7291"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9cdaf4f9041b0c3117202f233d4573512143308a9934b5ebdd7f3ec230e518be"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 148752, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "197338366b407b4a4c2e8ca128b866b21900d607fb3bf7071a932e279febc3c7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1145bc05570b03e0c12250b13e3d17776de3950a1ec9eff105f80b6e703ea4d9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "77a273f12eacfda83cb6c86859166cd80f0950e297924217dfe38364e701349f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e76b99ccc03174a9d2c5db07a57d9e4c5b15cab95e1cf9a41a6a00dd571e1795"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "073b7fd6267b7f24de6116c852856dd3e9ba0f5598d61fe95f80375220ae7796"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3cfb84601594b63e88b9a7762604ff4afa8675612bdf8fcdfffed0e9d03ec4da"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9f3a7de0c2b688371f36b3bf3d30d100f43a44ac76858349cd75dd5f36bb93d5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9989f708c1b37400acd0b0be7f57f3e21ca51a709f193d72e7c4ef9fe00d5f4f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bf025e30e4b763d044ff67c550ae597e93c5b3ecf4aad6aca2bb596e2bad34b5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0ec6f730329f96ff6d8653f82b7ae3e046dcce7c71c604d12cb7cda28d0412f2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4063ce59973dea4009cae0521726ca81449b6bd70983987dd4abff855d12922b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "24a0cb10bfc33ff51c9e0854d498ff155eb2329b87402c479def663d23889919"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a0eff03ac7cf2f50016df80e9577d7f11a455593df24e7d7390252404a9fa8d4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b05694ff9c62aefaaf937fe7a0191da3f56c2bb7cc74afc344b024d92d1602b8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c4a45268a37e0e0a9f1ef1af421f85e6163591ee9a7e3b77984969d19bcb8d97"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "617cdd56019632c599e248276a395abf8366d1618c10e1a20dacdb0775493f1f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c9c3a843239237a8b55757af412f84815c9bb25b7454a75157bf6c27e024314f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f5453ef246374f0c98d684c4143c993fc7b7b8bf4b8e30e0099e7061df7f529f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3c6ddee13b83898083e23c5b9c283f57e7ca29668ec5484a440a144a3a6b247c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2aaa1446dfcd6dea4b302ca37b9b9d57770f4cf2cc43dfd2f3b715fbe7e0b518"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "67a7fcb6a7549a768e3c927c6440a9677d5dbd2aff844456c965e47f129adf60"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1d3a86f72ca224e5e669a4bf7aef3b5f8de50366a3f7997a47eee8cd025aefa5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a4d473a06b1cf71b94b57f5abc9b50104f2befa0a53bbfc5d37bca8d538ac833"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7d853cdb1b9c7c60249bd8e2bc9b70d96719b7279a19b3399fcb087ee75b83a9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d44ac499329c0aa8cb91cbb67db0421c5ac08924f0e1af11ba8f174a2afe084f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f57e1f2a7aabd25e251ca15d49cc08db5d417a048ea82e6a23e820b93eaa161f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b603bdd54528c39dee808620e2309f865a5ce76375ffc86ff5f818faa942b38b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "df6c3360f802127452f4ce0b0fbbc49c9da2219d99a78d6fa28da35b86cb718b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a25c6f225a510a1f867310808038090eecd64a1377030a32260b54c831958ed4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197952, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6e18928cf06585b11336cd256367d37c10cf6f3b0d4a09839446cb30e1a3e0c1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165184, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "abc67b6396c789e7584cf4e781831de38de96eed6b9190e1b4695ece2e1bb40a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "10c8242d4986cf646dfe5882c7626cf8cbd1e6187949911d45f9f6b66c63845e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7aefbbcd98a3f2da4c743977706440c49b1ac14f7ba0ad9ad93261e00b425e92"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3d5b9a829412a268ce280c7eb263bee87acbb570a174c13b77d781a2f2de35cf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1dba6ff6355e8c5f7024eb53a45f65186555871caef80dc3fa5d9053d52b0dd1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "12d335c98f6a79443f8ad8e0cd96c72507e6582828b51e1237308e719ffdf193"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5ce2f0c22e955646398c080c103fb298fda210d200f38852487ac40934d4a4dc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9084ce2ec49b568c9b37588e8a6fd8d08171d65e1973292c0f1993a6bbe9797f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "956f6bc58fd5cddc6127e0b65fc686bd59f0026a61056d275654fba0162578dd"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "99740ab4e0ea30e482e7c507bd8ab5db62126d5996c3c8628f0fd1c150d34fc3"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 165136, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "43bc90a9d0cd15350a46e82b09e92a2052b03d3289ad84171ebe95549e11d84e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b22ec0d8577945c97bec9bf13db51358621089bc1a4bff0d6063c28e39a79280"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5eaf2d20dbdede8e71cef9e63ee2542e4605c9c5e85db6018a56e2294b64097f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "90ffb00f7f8f893a23eee866ca80c6e0a5a26f680b0810543e23967582554636"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "11ce929ece95cbad1d5cbffd1acce43f538e8a1eb70822c461c3c704d6e77f99"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "86856f531197467f0dd233e19246d727bc6dafe3106a2b4c612f37018d248477"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "aab8244d7c47b33b0e55a4fd438903b3f0ea7ced0589b76cd586ab20cba444c4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "37f71a4e95272f8d4d79cc0cdc273af98a47ecb71eab94869c4abce7962e771f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7ff5777d30412abc87c07a6586faff03694e173224c2fe1aa31108a8b797d4fe"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "77797e2cf84b32dd54a70ad4304e10b800681a9cf20821a5d063d9da68794364"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "aa4593c93e4ae16d2580cf363539c6f841a1b0ca1a3fbbe9b89a70632797ebe0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5d03cdbb5b7f78e201beada8766f7197d4ffd72c41e6b44540b7464fc43228fc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "168bb81cad156018f1c935abecb84a70649fbefef42b7cd7f8c4093be2a3ed4a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197952, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6680f324784b210aa88f4bbf1ad74f8063431c3ee217aba6ef3d2105b3fc148f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165184, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "18dff4d35dd0b6ccb7803661ceed90b80ebf4f99d568af6d7aca30a2d57eee6c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "92e471a3aa5c8d299ef6f4aa0cfe65166109be540d8351e83cd71c8b5f62dd1a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "104f31009c062abab2b1b0af86b1ff61c035a03be040a8aafed6d87b8acae59a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "69ad48ea794d542f37717255a81d74fae7e97f73aec0bc5da11264c586b8ac22"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4ada0c3e8278577e5b0ef7ad1f7534d646e2e51daaa35075a908c8767dc83c31"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "eaf19b9f64886bef0f7e4036c3f384197ddd09f651fdb81e93db8c28324cc0a3"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ed3898df0b466a0250c0c5fbd9d7a33ff7922b2e15de2b1a5e095fd089e298c4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b4d773c9d9d5cc122f8878cf515c03a86b7045ee6c9cf5ed169883a2b81b9960"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e57184432ec69156825122cbf16adea806f070e1a14e7fac26766b14936b9247"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5d158eb8eaa192ee6f923dade56384c290c35c1123efd8567cba68bc56cfe880"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 165136, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c0e882bf617aa211474e421d06fcb1e19041de2f3ec4c1351e8ba39d4ca447c7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9da704e4eaa7ea1b8132bcabd970f305df1a281e39edfd7b8d6a6f9aa301c816"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3e46f81c9ba3f25a5de1716ac1037f15333d669141e8657f6f18a0e880d0bef2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a623e5a35f0be8a053ef1228e7a8d65834d60ab2ca96f06494163d7ea13d32ed"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ef1f3fe0fcb3ea363221e3a2d0779c02d73ecec157716ce2ce6a6b63e7b81524"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5a3889a7c377d4062510ef55f9e1559b4454add0d06c9f1d342994c3a4907aa3"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4f437f3020abbe36d3414f96ae904d8ee45528100f979d6fa2674bc14ba5d709"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a662cd6562e1a0bfb754d43c5089334849e10ca8df5abc534475e5cc962b09de"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f9bd0b6a306cb7780de0a9ef59024acad21d84ea0ce293be19b191773e5209a0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "aa9ab394fc4c819d606899fb43f50f4062bb97b533b91c433868d76b670a5320"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6ab953846f228747d66929c39e4e99ef6e48b88fab2dc374a0c292be0f187684"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a0e3f89bfae8a468508fcaa4303d26673dce7892141c3e37e7d4f17b64c39159"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "723f92316efa1b7ac15eb6d85bfa0bf7fc118c7ae26ca011ee655edc79af086e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "39f28ea38850dac2f2fcc3035b1e8afe651c322e21616ec75089aa808469ff20"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d7eb919d6bf02ed82ebad5bb60c8fa2852c1fc443b842a6668815f74b58d4bae"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 196888, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "9c013193f824c59603816e6b103f66bd88211fc1ac71a4aac827891082b70619"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 222232, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "3106d5b7bb5d345f308725bd99d5dc68543403da44ed1c84816161693262819d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 184216, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "ef18dc64772e9d31d997e36479253466975db430afa440d1a02940957d921206"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "d8c589300b782c9b38d29f96c194c4853534206eba531905d2b55b455a31ab09"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "42bd542277df74e1d7f675e34145332c0fc88d5950519faa28151994c1132878"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "e14acd7a15cb671dee739cb2ebd8bd2cb969140c2645d805bd2ea1bcb5454010"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "9d984e1077677576b93656353586dec02057ed2d4a5d24d8acf961fff3490ad0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 167296, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "ec3ee20b4aa5d5eb96d90ecbf3668ec2bc17f4d690a3159b956c107c9db95169"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "7563cdf61bf7630985d676543df77799c129d37a2454a695fe8258738fe99316"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 196736, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "8410c67e4a5006869f7a542422999c83cda128faa3cf16c8644a673e8431dac7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "474ce302afcd3f859dae554c901cf1f71d0ba1a20188684cc2ea8166ff3c88b2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203392, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "d31fe770bb7a54fae1a546fb0993e47c7f5a3e1dcd51f0b6321645f7252e54f7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "e15472cc0120672c96529c6493926490c557e8d73aad6a21eaed8fc69ecdbb99"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 152576, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "77fe4efbf3e5b53b1ac3e339e717aacdd5cc11599e62a1d73acf6cc28eea4041"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "11f0ef8a110e45f603777c10f1168cb23182093c96ac380b43d7ac39fffa8c90"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fc8e5e681fe5d1faac4f7dfd667509db32a9b130a9ec5a4b373cf9878e7087ba"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "eff6a00a325e61b805fbf4384dd618eace3cbce19a18068a784401aec350c7bf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e954aca652591b3e15efdf78da60c9bb887e3ad795df4679a5f34f65f363aaba"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a34757c465c7693f9070f7cdd0f27cce17f1b354070194111ea8bd2e9d992ae6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c5600fb5bc53a1febfb5d5fc53507456761d3cf7556eb65d496f229e7a1b73ac"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4ec01ee2d5360ce73ecc586d65f5ea4aa0559accb053b5bb6853111025a5a830"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "37c28f442ec2f9e08cec4de08b91845757f44951f05244b6a00f3caecf2f4ea8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "995e5ca1c201077bc23828166e4ffa6c70bc7f0bffe90864a2f7024a1ecec148"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "179a79e73caf6fd8702cd8534b9675db11dcbfd88dcdeece15c12fc34bd75dad"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "89673066c5d1ef6d74b06b955de3f46ca7909cf23becdf4d1c909dc1919768f1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197680, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5afd84d0db245268bb57b0cf07004dd974b48fda50304285bf5d559e92c28fa7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dda1f7a7d70c240647080e881f0352dbe37dfff87acb5cf9442b2ce641c47fb5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "38d81d7e084d3fa5efbf5bf51c056cf5f0094839ef46fccdc21d62197a13fc65"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1240642f32b9e854cfd31242e0982926b33f64bff57ecf4f3948f08f62558ed5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d7896599316b94a084a4fba878a00e837b79f5f465eb47717e17bccec159ecb9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197648, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8c549f4fad4386255fd5c5f4da97265796699c41415fc74d83199f4beeae1a48"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c5a6d74724ed2a2f3a10932c0ad097e754717319a001234e1ac3f8be1b016de4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7d47ea20263622e4c766e1f24d861aed970c580d16ed65d1ef3a22db056255f4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ea5e25c1a9f3a24a83158d56f30796e0c6cd4e9db21e35e952794f3f7e2a4e2a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f1b7d81e338f8ef6f9f4e8ef9771c0fe7af387c2701cd261ab159766c03bdc41"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "accc9627d3359dc37105515c747ec9d41dad1102c497031a04fb944e93ce9c7e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f33329a0fea4bb93e690496ba672c0bbe319f0fd5c3bd25bd3474da2a00a3494"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ef07c0a7e482b2750cf0ec8fe08e081876023dcbbb81c8da49671e6c3120881c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "90c77998f0c13c19769075b176add5ebf9a491774339dbfda1ad9e3f4af18b75"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 196376, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "848e37c69fa1ca3f84afd2938f636be0331d7715e96bbcaf3fa1ffa13bcdf68b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 221720, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "4e40fe6aa85be370907a5c4f00a4c385715eddd7b25ee1b85897026bbc5bac04"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 183704, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "df89efe455f3ce9a86f694ef3870f3f63c5eb971e7a952de43e1dfd12edc4845"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 221224, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, "8edf7725102cf36521383e42c80f3e7674a7bb1311e56af5c516a87984cf094f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "1bfe774f717a143e6ebb7bf4151ff6cb13750e2abddbb1806a5f2341cd923647"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "8e0063bbd85125f95181c9576e5428d8ac184cdd278e4d17095608b691c88946"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "da784bd01fd31390ec51cc5183aaa872b86d33830aeffd912edfacc27ab12b37"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "ae5f8d0faa086e2015233486ae330a71b752b6226c8c75c379c09e848c70b59d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 167296, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "234308a53d49b2c0239df23135146302abf25a63c5cfbc0985f909b7eb367b40"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "dee044201398edc582e4c17a726f544a3cab3b78139998831b420c4646c9797e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 196736, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "bdb2434bc9c054fa432e4b6e061ea5d524d1d702954a21435003db177763b310"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "afb855c9a854d81f0bd86cbf87801cc4632a6688fd4abd79059dffe2020cc518"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 221320, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, "6a58f492ff36295b137ce99ac862d990ac367f7fa9803aadb8256b9fa753f288"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203392, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "c65f47a387975350db686d152ebd5d0cdef42ea4867c823422bd15f26a5f8450"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen", 221208, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, "5942f368b4fd3054bedf323791eb82889752e1646add9807329826ce4079c78a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "475ee39270105f290070f834ce19ce135b4804112425ee1b73a89848363ec260"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 152576, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "55682f7526e1e8944294c908dbde551413631119ad96cd4d5d0e511c6e115b03"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "02f9f8e140868d0359cab45b13362c93d16b527d926337575046ef668d4545bc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d8671276213bd7155fb3887797cefbf37abf97a9be540d3a6e8d71de61f12fbb"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "018409ea0bed3a1c5d6ba792749e11953827e9d3f365e96ff001f1913805a70c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1488eb00eeb6f85ab0ff55495f110b2afb0c4a590ba139d2612cf97e1e96f0f8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3343b0cc94de35fe138976a7f1d40f0e98fc3587b3b1832221601f008f32b392"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e898f9590c06afa6fd6f4c34d43c44f57df9974b142c263b557f9ca72d0d4f1b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "60c6b8beb62f2a75a2d552f5ad1ce1a209f9b29926e86236ad8cb38e4eb4bc25"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197680, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "61468eb8b2e3bab1ed4bee931480c4c0b78e41d287ad4d8d3bf2e6871569f602"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1eafadd6fbefc46c325f45a9856e9914910856dfca8886325585097213cfdfc6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "27fa61a703b17cea3592af467bc4253be95d4982edf1149ff207639ecbf02e7d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197648, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "dd3011b84da27fbc2ba3435f7b19c24854e434b1e2f4577382c98b31d79b543e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9ef81de4864c8221e70f56660644b89673c30f05cb4944dfdf1e5d4ea5ded8b8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b5149690cc82c835129d677de548b82682e4d52dcd236b6476686073c85e68a4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e5e646f128ec1188d589ba550ea646800ee66a23570f2793be42713ed319a1e0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a863bbddbe3f09f01ee49ccf5f2bebaec7b901a824c993b09589ddde039361b2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e3daf5f9f4f6a91e3b432077aa23867f43f9eafb692fe81465e77ca51f1d8144"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "45b3a23df0a62aa3389d2d05eec6751ae35fe45d96b7bb863a75d0eb1e3af434"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 196120, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "625d96ba29516b5cac24e44e37c433586ba48e158fd45d393cc4a9ba00c8a104"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 183448, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "85eabc04987d70a40b73956c770cc92b2f9964c616602d9b560a1a54f0ea2320"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "2b2c386fad6ec39e706e8875020383205fd92d99405301659058ea80054edf79"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "a4eafbe72c891e6415bd82181f6be57dc63a6253b6dbd1969e2756da13b450a8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "72bc3d87e509a4d8e27f13a6f0dd82b07b072234553eb53f9c2bb0202aaeb492"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "3c6c10c5ec2c37741aba3bff01c716b08686696d93864c6edab018023c494e36"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 167296, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "14906e935e1cabf80c0f6240bf8cbc55b4fbd08eb8901980091dc3eece512b2d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 163088, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "6c7b578146471f8b76230b71dbe8e0617ef8108c84d5128f055b32a2fff7336e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 196736, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "5aa7e4e3914f201ea5ef37ca92ecdde64560e7497a55c3fc207411739e69a948"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 188432, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "18d85fe3e6f7e617df5bfa030fc0e7573626146159a12c29748b0a4475a6ba91"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203392, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "fafa397bcb6e7632993dd2125758b8249543a3e20b60bf7f02c7defdd5ae5f0d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "5fbf2368b8403227f0912a256fbfc6b40d12b5393db2fe02f4859f78773c98bf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 152576, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "f1bd7f837804262ba285b0004d061e2d86ca02262872d031256924172b682521"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 150416, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "5b340d909bdbd3a6ebadf20c9c25bc6a9bacdb09f8a6eb4fe8660238bd48a7d6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bacb84c7750c2efe9d44252d4caec11b7f5d3f71a5e69e1e73ae3d544b8f3ba6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "337c867887de7a6ad4d3a27a0c2e7e2227312376f6ecdae29ced7697144333e8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2c8abed589f1f7fe0efdd0143810575685f5da04a7e2e493fd3251439a265e0c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4f0c85932aabd36b03d78bbbf510b8190afd177904aa02c387a5909b373ec66d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0b28e1ea50fa4492cad12cea6d779472730012f7ed492f708d360450343b6784"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3f5951a6ed1a80cb2eef4d6fa22ed540855d65ab64ab932c5f437d0fe2444b43"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "295387a63774a675596f43008bc3d4eae494e8524678c08b1425599711787d0c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a73fa43dd23b2c6bd077f4c4d115443800f765e8c234d4c0629cc5e15a42f796"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "703ada93f9caa4133a0ebde877fd1f28a2b3bf047e39b0bdf00cccafedfde950"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "96eb61b173314001d93c961c38b9f7d2ef073f1aa6f5000e4573d692020179f7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "89c1a999f89d5722668933cdc1a9e8dbeb37badd1bb52a66b70fecc8dc30dc86"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a4eb4efad22b91e3371e56bb1f3021788f3979e0071fa4abeaf3d8e859d3ae55"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3e247f4bbeded5ac21562236b6031c2d3d27edee5cfa3702e5ee174ace9c0f3d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "93fa9679ac5386417e30cabfa86400727c0ac57bce59adca991af7835cbd8672"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8b27e90abf75379e6497800ea03d49610629857d986d3518aeb70a858a83a468"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3971039fd81b21e41586131385b2d04027a8035ba628629a41d8019db3375149"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1145103769d352bdbfe1ff16eaeb807854bfe93a1e32f0241ef9257b9350e463"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9d5ac1f778f47582202d3042cd98764c5d574405ff6a9a07152a734953e57bac"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5e5e3ff55ce8fcdc5b541d2a1a5711529bd32f662c5e4a8062b28aa3dfe1576b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c12a6c4bed0dafa2487f329681f8504cf2606dda26de3d1d5e2732994cd5b0ec"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "176b6a159e3cf4c72ac8ee57fdbf748a019a277746283cc10fc3d810d008a525"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a90690ca090660e007e114344ced671c1c11816b1b496cb8ce204182e423ebf7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "dd09bdc46c2ea1b26df9805e9650f5594bf87408959e46964ab37c4bbab2acd8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3112bf3fe5db1c535a9be2d314ba2cbe364c30ca8ae92e3d793cec0f4d15006f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "db2f0ec9a184afc6990f5e10af9724ad7dc64c175d0543bbf4ed1e8ed6a93593"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0e140fb81fde81dc254c398f80b0a1bf64cb1e7b52e82001de665db8398b26ec"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7f8c21f9f2284cee3fcce071992b5870924351f1c0d9b58214291146c94fc299"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b04fd1d535d90c00577fe17d57ee701c80f5ec3b33d2fc840f613fae4082a9e0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "af97c12f16d100fb9d2a3cea8eae2adb62e419c92694403acf6ff6d214c33777"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a757907eeed9b1352b3519254818aa48fcc9a4295f918c436c4f4d2f21f38610"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f6e1c72c665002df1778b3dd389586a2d4af83d7a76c5150b5afcce1a863a4fb"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1c3304956c6df82791dffb9d5581c433113a79a9655fa219646d3ff96a621750"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f0803c89431ee43bfb0a9507721ca0c35b1c2dcc5f23584ad9a86cf8ffa1bfc8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1a9fbf1afbb0f870693a3f19c8d1f43ed1702bbc60fbcf051bf6e4fb4cf069b5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d92c4316a04e00eb49fedcb9c72a7061b4dba493ca0e06e5df71c870c8033da1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f3d0e002da2de3ca734754f27f2abdc41632ce08b631158299a4bea78ebcb08c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1faa56743fba565d6f466534ea2babe693f86af8e0d8004c13f8e70d9df517c2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3430923748964305401a95574777868a8a3fae2d0d9211cf041f9fc315abd6e2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9e7b23940f64d9906f9bbd593008c770c2564637754b1ed611e1409a768bd0c7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3c313238d9c7a4b8310dc8692690374f9bceb65295a73626050e4e9a3f7b0faf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "216d8ce0cb7f8fa1a4f947f0ac6a1ce9763da1bd61bdd796943a05914dad161f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "817cbda92925db0facbf58db33a3a756b9593c2102471bc6adfc949d0f19ab8e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0262c247ea59a8329e80fef90ea1394e573f95224e2dbbe90b4eedca5f21f2ea"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e40e08779c92fb9b7686fd4e89b074456f6ce9040d820183eacd3df6360c06ed"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "06c3a2d9cc329656c90dcb54d3037ca897fe85be39504de99504056a2434feb9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1fccbc9dff4d25c7cca18c8bbd6c234bcc7470ed9a72eec84da7b43c4d604032"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "84d3cb99d053d5d8e53b038f0d3f475264d312f2f05362146562490d225b1f61"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0ae2b69961f24c61631e3945d0016e8e6331445cfe75a5e1f8422ab9fdd2d397"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ad36075d18800efe9fbf1f9c7d398c434daf23ae42e109cb15ef9f1a6310ab16"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9a4f8c177033c8634433d01339ffea09e3e42486579499f2d3cb89b5dc5d8e46"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "373eaab3b26763a972290abb74cd6a634628be98c5ede4f81dc2427734475395"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b785f127863c53afcab75dbc212fa9462dd0a68c3803ec99b36d72e301c49940"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ca3fba52cf84efedf9b28f7ffcf0b39dae0b84104c23b3c4e0ff209555f61b03"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "fd02efabf52c87c50e1bf71cc15faa3f1d12340abee239c3118a4b530700b8b5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "47d3194ab1b4389563729ec58b2c537f1aeda11dab85f8a07b856bb09ff6e3cf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f4fccbb468f69f21ac0b149e9590fbb63ac4dc5e27153601445496d57886d0f5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "30ee5cb6b82ce42bb9e65deb6e7834dc5e9b74f0e0d6be4013841f591c7aa815"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e13b321f9e33adcf1ef1e7c3dd4c629c9baaf65314740ec2fe13d30c79786145"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "337a8ccf35f17927d87b3d5de26a7107eeb25999513e00a80c84b6398a27f550"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ae9cdf25cbf000f5ca490af6bd227c4990a2c62cbc36b02dc43c4cd675fe6554"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "44c488101031a9656d6e8f34b6346e107e33e64353822098194a18d10ad8d768"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a8638a28b97cb030292cb32e85314eb09f4296497981769413020ea4439a18ba"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9b7bf3f753c834eb1c2a36012dfe695a4b80f763cd0214ab35c2f68e8c89eb65"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "da211ebcb3648f28d6f6da2b19877cbb85f0a6839e1d69419a2cb0db37439a06"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "17f61912d554b2e4e8267a195bd5121d64a2bacedc9cde73826389273241084b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "881ec4afd9f2485a6bf48271fbb8167cae731c95a49cec3d294115421693acb2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "aa3f752d28f042fbea83c3d06e6cdeb7ccc5e795784f8966ecb550cd014f162f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "81db46646af846f9cd9aeebe11a64578a6668b0f8d2895cfcb92c19d569875b1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "43a2b369f322cb30774a91fb490175d4db89a470acbf7efaa3c27df0a8a67fd4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "527e89029fcd8301648c6352f4df8f97c4a9a389535bf15735325ebc585ef167"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "10462c1177ea160c81e48088f46e989124e76f3724bbb9c83fecf2eb51217a02"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "561a2d50cb73a473df7b4bc0bff3b6d878ba43843c7ee73d402bf46ee7c44a86"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c400649701418b06ba07e1aee67869f3f5799864487f087d717c90c715497aa2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "37be3ef8d09143d67cfd368d232124b35ce330f9fe8e7d6d7517bba0a009faaf"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "118d4bc62564f30a0a293347332680c14718429d923cbc26390d9d2a0e8a347a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8ca88e56ed9fb902a7500d69a6f32758785f08cd781d45e3ca32152bf8bcff5e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c1ca6361f3d318232919a903fd1ce43bfd2ea1e896f461423e09c25abb2c9500"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8bd6091a489d7a5ed33d4497bb5d364a54f6950a4853f38e07f72404343522d0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ab7a6301d672e3444c35d5e7777ec14ad2f6fab659dd00a28e8262501ad23aa2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "aa38706ef66a2316a3b1d210e39bcc9bcaf2d8d6f8ed2b9bb7849f7400db9994"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5593f20d250fedcc2c60f2e3c71cf49ddf4bcff511e2dbd8c2de73ee5ec5c4e9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "57c57059c39698a391508f96428c0997f0bfda2054bae1365b8e8bd37e26a14c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "712286fc68c3329500b7369e4b9f68061441d078387b92cc99dca4633e36bd22"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b33308848fd55a89a97c44c525ae6db65adf8a09a3effeb0cb42893228523eba"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "00d3a5e1f1f800bb141d71ff0830fdebd83cded9e5193a05c41207cd12e90802"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2955aa6ccc167d2f7aefa775ff457bb09bb4c22d200fb3f8815b5b3e751ef631"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "05ef9a6d4cab71d4a9aa9f55ad0c159c338b47c34365e7dcd0903f39c2c50fb0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "14033995bde18bace8e421be6db17cfe5edfb47b936916d8299225f29aea1e4d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 197248, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ac1e28bc7586d17199ebca68e7406e99641b4637164835401f028cf6acb48fbc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 196880, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ff65ecb67209dee29303162be64beacaad7cf2b15e6af651619771523bd4bca1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197264, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e2a4e2ff860255065f2baac378d925295eb7cca6a7605b71fb4c4af2914731b3"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "48eb9b16a8ebbd69fc49db8997e2ba5ff49b455eaddc005dc54c3222de18e436"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 197248, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b39bdc5a45866bf5df39b7d2f95c23afe6a81f5f57429749949c0ba85c6125b5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 196880, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4f7bc1f31e6a9b0d08887569fb8a32cd6d7c44d3f55ff67c37850e005c59bee6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197264, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5476ad6ab48d4b4c778e9ac4353deb9213686e7180b4b4aa2a3f2a489c8e4290"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4f22930cb99794028ba71a604f303938e6640eabd4a345286546cd3a7f81ed2c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "739ca7d147d9760ec12fbd4767c80c8152bb5a014df885425fcc1759ad3c6224"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 173360, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "426977e16feb57558a34cf3f3db490c1d52b2bd768d4ac16e81d5aef8199caca"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "41837db3d4bb457f47861e5d43199156e9e22c2057f8db5ae4cc420cb76e427e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173376, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "1010a148b4e008e9d9c04305e1fed836389e9aec68e608b72855b15287d90129"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "5510ba6cf272c7c168eb9d48d0de7399476257c29bd7e0d16bbd45b167919bd1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 173344, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b8ac5a822c3dd6a48b762710b4ee61bebd0e9e55d8d1ddd6645716df2f70c89f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "187870f18edcababb97fef99048b7a4cf22e118c8ea73723794cc64851011290"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173360, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "3abab5ac59558abf5cb7fd884c7f1a50ddac27828b3b3723bffad7d45c2313d0"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "da5aea4579ce0a2ceef88e7146f90a064f4ae9199815f8d50d167201ff863645"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "94388e784869878bb4c03f07c3e76b38ed96b786ebf0125dc1958a6b69e597d4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 173440, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "efe1df204f615aca44d1111f2db8e0eccae7e91baf6344e8d3ee7f2d919a937d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 173328, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "0d0b285a024909ee0f18e8c926a277975495024618890a74fc32d0f7f13bf3a1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "85066133f5ad3e3d97a334eef40b27d97ab69737ebd4bddfcb7d96478c32f402"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "3c43216cdbe39169951eefb77b52dce891b6917cfaed36f8d70de4f4d9eda0a9"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 173456, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "6dbe9e15d444d5772b109fbebaa2427ccbb2862045f93b65ca6ee6a86e6fc7d6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173344, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "8c0ed40b72d4c4306897e4c4e6d9e2319cc1c1d4ba5b53177cef0a1f0ba7d6c4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ca05ff8982ef438e28d71c4d9573a0d748ff299e51bfd1915b4af2b11eb5bcad"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 173360, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "25c5794161671ff7bafe7e6f0fca36aa56381c87dd8f580d3311646938b59196"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "d7d7209808db70921ea14a3f8211d2f67201ecb3129a1459bb0af29322962b25"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173376, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "8a48de259d42ed32e290aa93e9330d292bb266a3527362f80c34d9772f8dc2d6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b17023ca142ca0e2139bd35d176fc79ff48c9b9f262742ad7aa8d00352d5ff7c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 173344, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "80f6a907975d65407874150fa1b94b286a53cc1645761674c51c86d799beb175"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "d77d8fe202ae30a865ddd34952ba2c99d50dcd0de85f7fcaa346b1366b7d190f"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173360, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "81f261cd684120468f8fce9be6763d077acec51c274e089845542f1783709cc1"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "633e785e47e281a2666b3129327d2bfb52db0c71c13380b81fbb7301fbda6671"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "b3d482aba7d0dd8723494feb217891b5db81b822b5593af93baa4cd4fa328446"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 173440, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "0c85f88e121595e2f43d03bf08a857dd5c858d07cc12f802c54b12044f8e9b39"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 173328, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "5dadbd55fc5e7ef2386d43b1fa2bb0ccb2060d0c153938945718999ec94748c2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "5cfd608c65e7fa0d2cac3cfb66ec9001157da5cd4bd4b7761989861696d54993"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "0a4bce2addb4bb1748e8a21a08566979fda902a21f5dc51caeef7ac27f8df567"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 173456, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "44c0adcc63c9d75bf82788c04c72c178d28a6ed5b189be97fa989d6b580d4eb3"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 173344, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "4e06ea3b40d9f18268b33e62eefb2214b3b3cd3fb49e48e9e2630717335dead4"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "11268de55521214c4520d8329d5c7a3c31b71fc13a037af16b1d55cf5f0f42d7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206128, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "72fce66f6916fb96b9f370271dfc7a8549923efb0ae3ef8f069e7dcf06c6f6c8"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 206208, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "3263122899817b6887eff077e54ebf8cd04bd09e242d05b55f2d52bd88693979"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 206096, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "a1c3b4afaade05ca607bf621806aab64382e15f912ddaf455c00313d20727d81"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 206224, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "c20a6cb5ff7c5577cce6339790fe1904115e9d95ebcaa662ce3b13443621dafb"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "f1ae28e72a60d0b1e4f0b4563b2c2b2049c345adc9fe3dacbcd29a82226ff3ef"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 211488, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "8070970e370e0d64068d921e62a408311db0ec59a539bfde0f2d62278750da17"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 211584, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "c0fc66e11b73f8842c3033f025d6da4e54a29cd84f633fc0f18926e9542c3dc7"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 211472, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "d6a577fd0b6951ddd9ef9aff01e25bab566d5d7c63d0fbbac03935d804867730"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 224040, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, "073a1844b1229dd48bbc0dcb67d064dfd3d4758e1342f38dfbfbd19ab6d1f721"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "c34dbc29d7990a2a765b583874185f62511111044fa25afc660bc0777f9692ff"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen", 224056, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, "b8609fbeae5c75e38fb9906ac5f1b47db8856364627469f5a49dee793c025ed6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206128, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "87c1fbda0127f631390da54a9bebd1f76bc4ba2accb60aac9bfe932c36d491cc"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 224136, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, "b9369043852e3101f8c949c36175ba1179cb4f86f99da035667ffea3b7e5e2b6"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 206208, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "46d35a0de8f73c94124046bf11ef301b6a597f8182d3f76b1d1450613142f70a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen", 224024, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, "21f9b0d9af04009e265043e9340fc6d8092c2b63662a2ab26cebd97e6ce4c6d5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 206096, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "52498e12b2c0afa5d126dc631257a700f8eb6bf5f66c15933700701f707e87fe"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen", 224152, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, "c2e21121973e0647ed77e69b0209e9028711ef8495bf7f4d2f2d36a2af1e35fd"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 206224, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "dd10242096e26ee27eff47cb321088a30d75076af2eed5a9aed8e57017092461"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen", 224040, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, "a3dbe9c426686e09a1220f24576062ebd0893bb2f63d1d298a4a34a9425d8422"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "a1a741a703b4c00cc4f3888bdd3642aad88b84ed26f47de7622212aa3ce15e4d"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 229416, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, "ef118219de1f1c2c6038fe18fc40d3ac43955607a557612fa2c357cf4756607a"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 211488, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "05dfedd2f92120249070d97089d2b3002e933591f584be0de37f36b1f85dbcc5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 229512, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, "f9049b6bc09efcc3b6f5d70cdf8a1cee5efe7d802c4315d68ab9481c3757c627"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 211584, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "748577e1c9029790e7ea29087621766d8164c5b621d169bbbaa5483f1d67567e"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen", 229400, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, "25f354b553ecda916267c254058739711b0e165fd8e76381ce9b37ddde01abe2"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 211472, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "c80d592fdfd8d3f05bb0360f1d438ff684cffd4378a2fc22e417bef0bb11f759"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "bcc7db4e82085ce9681438149e9c124ddf5b1324df762a4db7c250a723bfd884"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206128, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "564232562731c905f7bea2b73b079d21758e9eb215fa220e16eb54faa55ecada"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 206208, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "f6951f3c1b893f0dc9bdaf8e38794872bb9f7ae95ce7283e12ed8c6a8fef4b6c"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 206096, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "cdd1e7666703974c29e0328433f17f2914863f1b0c1f23fa55b2129b5360c321"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 206224, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "ad38088aec520d81111f4e0560ded1255c24c42edf9d2527e064068fe7ae96d5"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 206112, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "cab991981fe02e96d5634aaf1771784b492e74b51454f47fc5e1d096be6bec7b"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 211488, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "96e5b21c66b7110d98b9f6a7ee68179dd35e073072f9f3fbdd9473bd60183202"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 211584, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "8d5409db19b66d729a363d107489c0afbe491e3c0560efd5678839968e40f096"}, +{ DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 211472, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "9917fc9fc2ebb14b615008f11ad4e94aeb11ad24f3fd9ec49ae394d92303b694"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c23df1fcafd8747952adb941e2e38a68b0e4f78a8c657d6924988540bfe79d22"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4828c9ba9b72e1cebcc8e90393e6db48988ef12ca99bb1d2948e89dee2e283a2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ffcba98f213e86f14efcd13f0fb18c9bc7894395037d9666b2f54ab1f17e6a07"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f82976b39e093c7cb661a08b2ff934ea1f2a4ebdf3a914a149d4176e8651c02a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a6ebb9a0b7c2ff1a36375a40d18587af87143b735d51e19e7f98bcd67cc2ab7f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "842e8eed837fb6d7280aaf0ce81d973baef63c41583f295ca9a02b09078655b0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "17cc898f232a25a119237ddf72619ba1d3070193248785c26dab8a0a9adbdb71"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "07779218a9ce7fa6309dbd6bf7c4d8406ffb803b514483628e2caa92cb8d519d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "a014c46778fb37b575e2e144f1ef8422369ae9152352b920d6641c1101dcec42"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "1dca0eadc2b763f86592d3f33b098357d0aa172594dd178f33dade6243cf7e1b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "44faaf0a640c358708974cf886a049fc0c94d6aa46f9f49c33bc3bc819ca4631"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "cbf3b704ac7e0b4c9a8bcaa5686cfe8d4cb944eff78be753de105e68f1d79008"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "acd9c03b11624e1a6e01049619e70e48dda55b89d8de391005f16bc45cae7e90"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "b66f7dcd608c057fb6fab33043df84d85c4a652c8e736c0ae37bac47570b86eb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "f5f511f58f927ed8c417e3f1e8c0a77719cd02b2f859865ff212d4652e7edd72"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "fbe8e3104d30b4c2cc5562404e0e12943ef2b2c7baefde3d83c02c5ba7b4cfb0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "35a548cb2adc3bd67b108bab9217e6706145e871e5f0c1bb336a567cd37aebb8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "c42dbb793d9e96948c49ab8d9c3bcf0a21d157fe2a62fdce220cf95cab41214f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "c62000041bfe879312200ae2ef06287a7b26682e50c3baf8d4f4fc1301bb5a19"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "164ae3138aff9010006846f26c93860a76e8750117e09b4bd60da7ad180e03c5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b6121f8c0aeceea898d22271a3ee122ad20298c6bcc981a5f964806d98134e06"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "83d06c2a2a9a82ca989095f7d7542fa4a2afe621d8cd67836830600c995e611d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ee6e500b781fac6c024329033a686f572255cef5adbdc63f01a44e743203a669"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4956e29688f692961642eadf8d445c17a847a87d40347a22e4a4067a63d4adfd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e67056d9c016424ff53ae3926edf0b0a345314a45d16b1547eb61024349a9070"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f49c40d2ab166ec9aa249835d06803f91c43f92e8792841b5b32eb31c3284785"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7d14620365f661251cbe36356bbb390afda1b7e2317f6741d67a7c2abb1f1632"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "15fdd828b45bfa482610e4421c35905a4ff218a8537962c929c0bc73fddb74bb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d8991106f12b5e4afbd5c870d42ffda9965093a7ca1e7260bcc3d47d3318f77a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8409a296cb30957094acf89bd5b599172d3c5164b76fe8189819c9781128bd2f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dc13996dc08551243064376ac065f58921ebcb014cbf69f7e86d0a8e38162524"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6bf8055115c681aa6b9d2db6964235c57a5420b572ffd39b8f19e3bfe81b9126"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b4264db069de49c11b949933bcb847de4d4f46f4e75881f1d71a11b1e12a4b53"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "771966d4c665a5ff93de367d9f15382bb36f7e27bdd911042dcd6c89009787fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9ec462e8e1d0f25ea4fed1c12853da30c0f6659db1fa7270bdf4dd5fbe1e9df0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3d6bb6eb50c0ee476ac11548faa1e13ed5bae8ba94499d23769984b8eec8f76b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4b5ae32dd034f461b80dbae6e3384a07a6b2dbea6dc21c59509a66ccb492453d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "845466e90d609fcbc8aae74d0e11c5e791ed9ab9976af5c54141d9e5a1584955"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6707aafe2e1dded86295ca8fc0c806d49873fbb0e117afc80ee634683afe5e0a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d9d3f07e11ff7bf44dd807183176123ac8f8ffe9482c176389d9aeaa28292fbd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "59181359b6e3db26a678eb6447d668ea93de1a2e3e922fcfc142c3b5063ee57f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "631416ebfbc756dc1dbac994cb3337fb4669d15104959f7e2eee731df1e1444c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c05e91a40e22b5de778b5fdbab8434f38d7beebb46b4e4e1250c1a257ae5f056"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "296bdfa5890394c629ae18451d0b873e7bbd267a30ecdef61a027a43060df247"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "23c29c2de96251f7f5ff7d9cc06fe18d3bd51b264cc98f379a7e6cabfc45d866"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c477465694e1368abe192829609ed65b58b86bd501c582b6a7b0e7714c288fa7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3483f3486ca743f795f170f8b9512258e84fdc464138fd473ccb4de6d431d599"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6688ed7c7cc8123990d9ef0f8159e904b488d0e89736d08cb31b2c3b3f61dc54"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "92c2bf4d75c7ffe9a4302efa970fbdcef9190dc5dc864e47ba935caeeba8bd50"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6858e694250be96138ccb34dc25afa562005fed2e41c4de4e8da84592a547b7b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3c04eea152efaed8eb59ddac4e323d69a314bb49a067553e26364bc1a530e84d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d36bf4a8ca2a3aedf3625ddc8149361f1893f2055159801fe608fcbcd92317b3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "25d6ef3f201e7696c752ae14ac9af5c6f2ebe354ad88ee6176137cb68472db2b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8375473b42970b3528df9acc33f5a4b298a92cb0daa3d4e3b0794e69db800277"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "96476d036d860f7bf7bd970f1a32b210ef8a3e843b102e2d7a073a9dc835d342"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6676521b0406a4fa95ac623cb53f5908037193f070a37f5bc33c694c6641156e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e3c2e481e5ddbf74abeea6011495a58bc0ab45b302dc9ed15f192f1667d38c77"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "216dfd542ea711d96473e25991db8fa84f438be34dc36cabe6bcdefe6c657620"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ec3eb0e7cff4234a51a555d01890fee3c0b7255daa6dbeeb060179bec7f805c8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c89b9db1dba782e871ff6840a89af741f1075b411b598515a5b8abcb9d24a328"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "bc7b022635c3049096fce265fb03f063a450bd5b6cd2824799298efb4f003cec"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "952ca3f69a611ffca363bdaee46733caa8d63ebb4ebc20efb3487783542d15f5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "af29e0fc9f56f89f88b53c005dc55b7fb1e930f0feb05538260e44b2ec6b5a6a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "644adbf4275023d9d95e12b944b3650eb23a1f427af39708053de916ef679d44"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a29b23b145ffb1c479fef05616759d08f03752b632dd904e610dae7d9a9b6ae3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "cdb1a403eb08498a6f0118973dd02714bc7a7d33651a49894870da49e8f56dc6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "574dc10ffa8ab58b8186c173b01f390332dc747803067717f60d75cf2194604f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "80c6076cc2802ee8ab3e4da37ad58c66b6f770d7cfb5e01ca63b9167b9b1c531"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d11a1eaee3b205bd5ed148ccc26df3c7540bd559cdc1a7ac757ca5533dc0a3de"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "626a105edb17f30038719d001a8e205421123be02df57e3716f4603008596d9b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "bf7975753a212dabe9a73e77d0f5e27db47babf43e9d021b21448c7e1d9dd77c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "de0667bfb8d5c0059f443581615dffa36a091f16ddde81dc0c85bb42d9522ba8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "df0caf000d86cef9f60cb98120a97da4640086f838626d6ef86891134b093af8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "12ced1a262baca24a41dce4094d68410a7b7c900f1444e7f0c86171794a81a36"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "13902b73b034f476a97312ebf44c2ed56abe2886462a12c17c208bf165b58679"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "daace6f98dacb64793f6636ba75f2ceb56d02da7c8182dde3b42ac7543ee5fd2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e4ada2309d800a82a022baf29ebaa88a9aca1ce030c04cb94334fab54a62cc73"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d41ebca21f486e2f89874f72cbe673e7f4dd9b58d75ce5e55ccc27915028be66"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2112728c2fd1bb42585258e1dbdcd01dcb0066ebf29b3d5c16b2c9224dac80e9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "af6abb99cec20142208551a3cb1908d0acf63a1b3219a09fb316a83b6fac7082"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7d4719d17371e44320f019ed26b6594b73022642af87ff611c2c33bb17881460"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "f8d723b42eb386b4b7b1dfeb749d64679765a3dffb77ab8214aab68af0953c18"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "14b8a503c41dee58c4a56c6fb66e7618808a4fc74541cec813d12f56b8dd6414"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "d3ea4ba5b77d983ec318f8a49000cddcc6a99fb5c76e1ae25ebb0f10c33ed81e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "f02c83bdbfe9db72c659ed582078cb365c2c67f30ef51b5772ef0e956651f740"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "733c8eadb8ff1b1c4779bb9b632d1eda4dce73d8f463881567a515d243f6a672"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "835231d4e97be0a61ba9cc5c39573e3cb161330216028cae8cc2ea78c7fc7d4c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0b3664fb9a6ab6f87e11e8204fdf45a0d0fa7cba0f719bb3e0b75d5d5e3b24d2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8d9d53ea036d7886453932aa97e850ceaa27b917c909ce6c673b6501c6126df7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "3d064135ccc15aa37f3a767dfee7c5fe0d35a172f3260644b6aec240aa110df1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "e8dd511881132d65badb648f55cbe2609b7bc5b01633139c9ad022a98f4309ae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "6dd59b2da7804d22116ef911453c19c94710ef3c468553a071bfd29d35baf7c8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "b5cd534d0ea54d8ca29ef558f95fd31047127477e71ef44e1521a6b351aef6de"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "cdb4986b3fd7d920f711cc5fca3efa57ef46bc2cc5573538f54f99728f90d149"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "aac1e66deea630befc23f0f6c51722f71dd4236f51ae1672c6d69db7c7e8b01d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7dcbee0dd968ad7d80af97193e061071a05e3ba14778208a2e1714ab71a605d3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e9cc4919c985490fb14b8f498c8fcf07a80ee48d9fb04ee2cd38dd3c61445971"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d8cb532c686e28e03b12479eb8c292ee7ba638cca1c7cfc490b62a39e53600a9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8b86d9166b15a69aa406cb2e60352ae8b5c6e1093fba2598e919dd5a558dc9c3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "88ff5100f6b0e00bb4b3c626b8e19bd047fd5a3222c7a042ff3648e7ed94680b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bf6852e925037815893661f7d7e3f7f79aab5e60e514c7fa158740880133dc0c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1ebd33183ee100f96deaa4e587996b454c4e4514cbcdc3f5f32159886f2435ec"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cab49fd4c6e3e60d30bb44ffb617f05e58d23bc43138695b8b3d1fb4e449cc50"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "669da8e74f609b58e914e6d6c6316a2eeae0c23b4982b95391e416cd80134e03"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "78f40495e8747af9f60bcdc68a6aa7fa84ba5b7ccabc3c5cd46e76946b812743"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c276f5c5f4ef83c54c2abd9798f5bd894c937aac194143b572a58558c1795547"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e8f05b15a755aa3ba7a1f89f6f5e125454761e574307c6882a59113f5448477c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "22405f78292261e02f6d2b239ed277c76b91937731026efe97384fda97a95273"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "eb283d8efff8a314f7cae4eeae76a3394237753a0ddd625be6c04145d3673f5c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c9b93cd5650232c3d6b5e280d5069581141d6f1c8c523b85a8862726d105ff32"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3f8c977e7d29485c7e909387584185eb0f18b01ae95e1ea81b176a069d2aa831"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4c4dada51a0bc9a66ab2b7ca35f6a1d70efaeee631b38a5c46fc158ed80f0c59"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "cc9d5c6a3fe3d360a47239e8ead6d4c84fdac26eff211fc603854de2976b4d91"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "141d07d3388dac906bfdc51dbda4af5ec3369bd13810c622ed75eaf234e6b1a2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "57cf406ee6a3ee86379946bc57914adea55278c6914d825977ea744e5449214e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "259ffe58aae589a30b5b2a6f180fc0c5ddc3351a4cf72232141eaf6ed7aff069"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f63461f9a048e44cecc68b4450d3426c3ab928f76c34bff9dc6b7f746462cc97"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6ea6ccb975f23e7e9755e596320a8683b38e4554cd8d0a3451aa4d25488a1692"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8219475fc3ec4c3c890d1aa1f7a9c08ecae06edcbc9da966a04e4ca1a5954fdb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f0a86cdccd8566b50f814df1f0a3e557f146d016ea71ae757e75312729155c70"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0bb12e7607d783ac7a6c4a29074fed7376ff8b04fc6068bae050d014c3d03c92"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e3b1e24d146f6b326ba5735c89b3a53c65982d407c8a5b5f813848c3a072e62a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f6319b7dca3ef996daba0470a0a33798023f75c978f945fecd12bf57ff93d2cd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "faf2034930f380ee11cfdf91ccf207ec9ef7e960fb71d1d473a735d562e17252"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "673b5bfc9b2254efcd5073f10c7679cedcc85966cb535dfc41128523c43b29d4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d7f236e2c503751786c9a3b10003c07d50e9a95434f7cb0e027de8f07501fbfd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a9544faca77149996f63a8973bdef2b52429325ef463b78c691f731a248c7660"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5a72d0fe729db66d3a3301a1d7d97304768d4fede841372e1348217726707ae0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f3c61723660fc05dd11dd0c4fb9154bb3bf1a9cc59f28c254dc890295fdd3f04"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d79a56e967ca64a7dd99c8792033abb7ae4b09fed01efef5dc1f193c54f8aed5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cfa6c89b20342f06c7fd854dd1e2a5a4939f4a744ed9a3cc689f74fce647063c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d107e6af2491d12787748d25411ce28a1fb7dd56ed199bebbe96a0c63503dff0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b1d7f1a4486361deacb0ea716b66d2cc0cf44a249826e3241622f6c612bd18b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "70f31570ff930699185deff26bb86ab46ddd8b1f905ecc8e002949f2e927ff5e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "2bc197ba5cc8a4889cce4e886aa79e60fe669452e16c0a54288fad618008558f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1d6b45ac5938162141eaf99dc6359b2db03eb2e89c33640fb6fdeffd7ebadd16"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "20c1d7edf8810056df94a0fb7f8139b09581df8849d4509832d26997962f19cc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c23c2c9d0476771bf2a3d70dfa8be0826fcccb43bb06d4c697e2529c6fa1ab22"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b7bf82d97524228ecb616f4aa05e3390854a91a8978759f24838d04aa6f62eba"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "91fc26c7c0da6a3e56e855ff945ed1dfb10ce095d04886e7ba4831a13123519d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f0c6e81e8eab3250c0ae74865c8022305eaddf733941bbfdeaae95b39e9a8885"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8da28677a778c2ec188ea425e0201098ff27c812c47aa4d1cc3164a7296f9a31"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "cea787d207dcd98e6ac0ce6b46b31a6b32e4697a202c13de8f27e74be0ccf1dd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f6b7eb47f6ecc3adf249842047bbcfdaebeda4a1666efd30d374bf158e58f231"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3c20044a29b0181b4eaa7d7bb21a5b1d3dc33fa160e6e18afb8e3d5f335010bd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "65871dc4184b44bddb3ac6bc326c919025809228563564989c4dd4bff81141bf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f08552079ebd3a1b3b6a55ebcb48ad0e067ce659809970d8c48e6dd852825ee2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "32ea0e2795725411eeaf6883d57edf0f2d2d4de183bce0123a7529fa981b6541"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6f659871c03d085f95431549736de1d8ce185f93891d7ba097430860b3a27d8b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7655fd3cdf4bd4ead94daccb34585e4baff19090c1efb5aa419b0e6c3a335829"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "22c5624655998f6824fdbb9d0f82139107c2a75bd33bf17b5afd2762343fee82"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "36a6c9807c7c7c987eaf6dc9e348800ba1f73dd79f8c6eaebdc916fd7c20ecd8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "169170e3b4e76d7a7f9f60200d32cfa65327f493adc801235e38b6a98d982ebb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d718e692f9f134a4a0b1aa0de4612c7e9bf7cb92eb75cb22029b3aafdbdc0504"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5de8977a7a3aa1a84cc5db3ee8fc0640a9b10581b2fa4ea2fd29ac5e05bc379a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0e510fd9c02e73d79c2c332a35a594bdd545e0f995d546ce0664d4627b77d3f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "dfbb580738a25ad22eef6509da00f9a134f9e26ff138a1fc6f99f55d5851a635"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0cb60a0d4867979782a3f4511f1ef7632bd44741604538d9f04424b1ae1c2806"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "be90ae0cadcd8d6854d18962f2dc339c3bc1c6e571ef30586c724824f488c1b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H256SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e3e26e6462c45c870067d856550cb781b1116d8ac29dd0121e00c63e294bd2f4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 209176, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "43452e8dba06adf38c105af3b1283757628e306508834b59f954c33d529d48b8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 214040, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "811c30935092ff2a47f1239b7447abafbb33ba32e0127efed574167ac89401c0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 198552, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "c64799bd39026758bb725687a7d83047c27f7c64b84edb5baaad852ebfd34a14"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "e6cebda6f454815d909fb8a89c82d87ed12b4a942f4a8f18b1792350f0976240"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "8ea7a1159d4fb7775ab031508260b4dfc44bd84415393bd4f01fae36f7ee8c9d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "916e07a5d3dca071726c9fa71fa7e550f1f23597cc97ab000fa8fccfcb17f4b0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "7c564d6654a1bca2a4785da520b57bf1b5f32756054caa9402f10d7431d5e91f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 179584, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "24aaf936aca4d1895a3a35ddcc24ff421f35fa63f64cfb0f38368c0d54f6c581"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "3ad87f97193bc0535f7c23e89fb450fe25998d1d8c7b6dc9b349e7f8b5c6a5e8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 204928, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "4c2075e75a9ee1cfd7f91df30ed49ea104ce88538969a14fc56de2df28749b39"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "ca85cac89537b0217809210af2183fc3975bf5f9a5a9843712acd93c4b33a4dc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203392, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "22b5134697408989ccb21d04129b94ba3cd5ea9e62ecf09ccfc2759e09b92e7e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "58d2e44aad8b76e04e277372d4c6430774c064af868cab90aec545dba76ccf57"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 166912, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "4bdb6b050ceb80d65c30ceb7d602439bd7072fc93fddd0b54c19bfdfa72c3286"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 128, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta128PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "40218e7194e6c9d5155ddea7e270f940c685715bfa5bb868b769200c7e3e035b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "80df7972e52eb13dc7e99ff0f0c9d2e6a0aec9ab0a32f09acbfddc77f2f430e0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "024bc69b09483806a6c804aa3e140dee4f305e0015ac8c1eb28d9832833ec1f6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "22dd820af2252eaeee47fdc7e4cdad6ca25f449b57adee8b352e45dc269944a4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6cd0d1a1b5b1fdd8269f13ffd37c847fb661a9a0be022c73298a82bfff1c4e2a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5ce6ee7bb095af3edfb04e2c5b5d08cdef5c3dc9d58ffee240c57915a87258ee"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "977fd0e824875438e5d3b32b075a915b7881cc511de1d175574c06b91a730386"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6d1f91e7718f0065f094c8f41a64c8d3658a61de5038acb8edb4d39a073fbbcc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a3654d91f518ca9474d4f5db91e05b58c423ac38e1abea4d42e49384cd76584a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "02093b807f3bb870d53550dabb9547e1c518a03f76e4b72d4d90e1e10c9095fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "cc54f8256143ca27be153e96be2a519a9e40530923b5638319868e748ff7779a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "59487e1f947fff615bf429f047f2914a3313f66a84f03555f8a068161d95d426"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dd3fe40eb2a45a308f8eff1b3c014f5d2b54eef483333b369b46e155eaec2894"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e8d0e58a19eb122bc922663542f0807e021e93eea6218f3910158b43e83756f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1428899a525f7d04c1d379c53451b90da2ebb6945e42f3b0c21b7662feda9a44"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b2ff69c1d37fe54679c3a392e5aecb232751214e99499467f1b85cf1b2c654f7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6816e99e0d3df3c6b88eedab8362d3046f44850ccf62097d47e675bdfa3381fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "974303530276cf0977cffbb29cee47bbb09a34338580669dae67d73e3cee5889"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "858bb4d34d2d25043e33a9dc93c39a965d16c08df41a3cee2d10a4349a5b3258"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "aee37db355465ea78bae8e8539730986fa62d778bd4545e67e2cfb0c3b5371d5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4cf823b6ea56b6f530c46dddfa1027d23536943030ec6c2babfa9c7f5436382f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2374b457cc8113243a78823d3694ca40f65ffb59cb458ad60495183bdb09a4e1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1d9d7094c8807d76996549c6bbc5b5b2dc62deab5899c09d35aafa6475112bd1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4ba57a282e28f0ea3160586877c947c6df729dfd270527588ccbf142ed3dc98b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "413dd3e94ea58b41c23d1321a5253c0959becdd5b59d313d9fe9a908f2b244e3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 208664, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "235f5e737274103aca438e9a573f73791ecc8dde1e576b109eb5a18c3f5eea69"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ32Kv128StaticSwapsAbForGen", 213528, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "2506360b63509d1c99f986da13e9dab9af7470087088dd922c99e52bee597dfa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 198040, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "7e59684a7b621683bcffc78bdb9526260a244c8df4c455940a5b34abdcd46714"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 188456, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, "e61ef47ac485097042a442afcbaa61d3d2ddc9dd05ead70efe19f977505ecc92"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "87de606c77a58d13c9125652e0b47f073bea57f3856d294058e39492719fe1bc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "d364ddaed1d0c05c1113361253d1113e7176865426a6794a1519d12aaa2fb922"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "030e398f1272191d34044eac466883c36cbcbeda8d2a14155c1b5ef93ed7c0fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "eb512337d228d2075f2eb6a30284e0bbc5df343160caf5871c94dcd645ac6b4d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 179584, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "52436869bae8d6fc59a69ffb1f55957607cde1dd8fc6ffe1520ae0c89b47045e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "b2605464106e01221d03b5e72f0e710f1468b40cd3e3cd3a40c1cbcb720aa492"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 204928, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "3726aeabf81e235343164f072f1ec00f16bff3d265a9218b999f3f6f96d9e2f6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "bb92e512168cea4ee89bc4d2fb3dc868d38e01396a21386ad8d7f155e178a1d0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 188552, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, "4be4a7aa8cce288b4910f826615a4f41075120522f159ae255a69d9c2bb64d0a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203392, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "32c798d3e8e8cfe2473be9b7e62de7fff98b4734c2671e28725acdf82999dc12"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen", 188440, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 2, false, false, false, "7317050ae3639c4bd5b3df32742bcdae64d578af9ead019fbd26f5edbc6ec288"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "55c97f1817bc1c24e5095b5e4ac0a6e45ddddacbd3a86c3639582bb8657dd17b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 166912, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "156b61a6a6d1ea0622519a8cfde8341fd96ca1c74792464b8b8eb56f03f7fa72"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "b1be32f52b72b848df99213f3e6795ed1a31e54a479ac3edad655f35d498c20e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "47fa5a2baf8ad13c2da6979c6f78f53f01a5fc271ef8cb33af2466f6c0235750"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3bf53a78e06f25ad1979be6f72b2f6d081d01ff4335ff53dd1c4063534074c1e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "502101967995c5e58957a864456cb3e362a76e42e24220710abe1dfa0925d9df"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7672949f7441c42c4a2e88cc3a2ccd1bf36954f917a6b665696a7283e76f11c7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "533844c7acb13d2f4d16bf9d742220bffa0a9cd26733dbac179cecbf4eb98ebb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a212ca936b9082bcddb417aa27d2ea168a506cf51af65799e8a9d66b513d6b14"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6fa8de591e0e65162e185bf915041ae077143883b2954408737ec286aab63ce8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0234840ab728b87e64a6c536f2e1b2c833585b5829dbcd8c0ca8d8d754c179f2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "471f68c81960f388e68b43570ed5dabbb9261f834a3a7d56792c5844b0eaaf47"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ec14003a8f1bdcd07db50837f891a73a11a23635f22cd72228bc21073b65e0dc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ad1b2b5ea4a125b7026af20c4e19514306a5be0b70de0f0860653f16a3918d10"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "59b27986bda943a4a7c0297b13956d1f572cfb410e5f096bea597133db8c586c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7b5bbd889b8cbcea1f760c16c5db9c503368b9d2cff138602a28dbc0057d1c83"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "39382d2f2bb732e3a5c78ea2daaba5f53f153110809dd5e0471a92df7a0ce29d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b9f0dd638765158f4e86e3a03ddae36d0c66edb744616e0ac3b1eb998cd0c31b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ae20ed03522c3dad1b162764d1d25d1faa94f3aef0ca4d348c8b574dd9c74ae7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ16Kv128StaticSwapsAbForGen", 208408, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "72e68435f3dc8c0b31052b5adf054ebe0cf691429bfd1509d4a11338d46f5374"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen", 197784, 512, 2, 1, 0, 2, 0, 3, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "8e642cd7d98eb749eebb79e5f17daa7139b25a1a40d9c1c0dfb328bd655a2346"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 203296, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "da64f6c92810f6e89be22c50a15caea8c936637a2f1b504fa8f068b6e1bbd34a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "531fb6bb502878f060c042e5a725b3ac9a50316b7e2aa9d8659d84651dfb8410"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "e89d978fa371633c7f1286db33e3cd5962eb7a8300ddda4dfec6468cbe1b2726"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1MultiCtasKvVarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 1, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "aa1992b504425de969b9f62d618ee0a4b2d7808b2a453135ccfcb7de50df7d49"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128PersistentSwapsAbForGen", 179584, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "640138a7d101df747f60e7e7a8c225722372452f182bee327f7846ee0ed6b7f0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 16, 128, 16, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ16Kv128StaticSwapsAbForGen", 175376, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "b15b5dfe93c1381f2857696d98055ee2316d52c75ec67e25bcb569b6de2e47af"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128PersistentSwapsAbForGen", 204928, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "5526ffd0708a0ce0086e180cdc51ffcc3d451d748808e3dbed99447b3cf44341"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 32, 128, 32, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ32Kv128StaticSwapsAbForGen", 196624, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "912f4f73b1ab0b8aa46c979a3a34173a15277df59b11d91a440ef09dc805a0cf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 203392, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "93dc3e16f739a13a9bab7a119000ecd302169e3b799bd9593b5f15d814eee671"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 203280, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "5f3e2a6daf65ed2b8f1dc057bee9ffaf1c11ec2fab93219e3cd8fad33b67d2e6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128PersistentSwapsAbForGen", 166912, 512, 2, 1, 0, 2, 1, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "58567be113011fc291050f69967f73286ca09e0856334f90c6ea93ed1c7728c8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 8, 128, 8, 128, 512, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H512PagedKvDenseDynamicTokenSparseP1VarSeqQ8Kv128StaticSwapsAbForGen", 164752, 512, 2, 1, 0, 2, 0, 0, 0, 0, 0, 0, true, false, false, false, 2, false, false, false, "5b2b807fabe2605c29a82d0a6fc5eba5407cfd2884ffdca0d2ce013814108589"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "45bcdecc204a23c35e76488dbb41ab1e7bb84bf6b529adbdfd22211e5edb4c21"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "17d39a4f7c140e8adb68d18a93f483a4308aacc7056f7f3f2a96dada78a12552"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2028ebcd09c313668918667cacfe08d5b68d8a691c6a5da034532cc592507423"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "bcaa023a3b925bd5108a3f5d8e3ad18102bccfa885356f6aa5f5c46f20342edd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6e9ef8756bd9a1e008117cf62eec4c69f15a60a3146d220a0edbdab2646225db"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "bce4cc959a274d2d9556b8af8fb5a12d6aad08b3cf2c8038830aa93bf3b30ce3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "0d2d85eba5b87e65b55f2d0c1cd06be59c90ac75786c97a8f5d6162b74672105"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "78962ee5030dbe2ef25a0cde558e17be77a389ced5b79178403a31a54ddb0f42"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "0e8c84253c8c48dffab3d4f47c093a1bba64c2dab96068b8e113ba7844051176"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "2b24dbbb8d36b0f16458a5e2f26162e75b02900f7df85261024e67ac63dff46c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "762306175a093f8a71e073eea246d895d32673ffe7decec01dac37d903cbf19e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "702d2d2dcbb76889052f7a49b6df33806dfc320303566444e3c55a26a0e22668"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "19da82b557ca5134491644943664020b13d3b8a52682bd592a18a43b39c21646"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "7f31530a1d78986615f599b673c69fc636f2c6e46d0bd870291611ea3bc51abe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "f1a3a97a0c4ec2904d41d01e11f1e1fdaac789eaa56c9422c56fa925c10d23f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "9b7674f87c79d62d551bc271e1116a7da6ba6766121959358b14304bce5bb9c1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "e03e11b8dd64f9118f4b8169add7c09e9b04ebdaaf9415bd85f68a856ac1d39e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "ac92601347690f1ac1bf58326db8630cad3abc3fc5ec3558c0c4bcd8e5c0dddb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "fc1a2b23459d3b516324933c5d98923ef7dfa3b5c3fee64fff6dc5cd733bdb83"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7b5aead1dc6ffc8b8520eb505e623b9669b5d6b84e3aa46f521e5f355ab19507"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "572d011071adcbfe86cce5f9f42c0080897e004a0e4ce2684cc6664460706026"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2c03b7d1fd3c7c2fccae461bcbe1b783649bf0e38057de1a52c1cfed8b157d80"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "87e7210e74a6d43e6fe6aa28fdff1abe77181acafc315328ba41ae5016bfd18e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "87b514bce97119aa4c0a40286d548211dc49bdf908366d3a0e1bf78f2d35fca1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4e09619f885928e27af62d5f00ab731db97824de62318d6db63bb270e6fecc97"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8d92350598542454d88aaaeab7dbb7f635a35a3ba3ef875b23be2966a18e9e1f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1b038d5a1f0b00ca89674719b61e40a32e14de598563f3b2e2f2f5e8c9da97ac"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4f83c8f34c567d05681cba784d75f2c860b201ae605c734978a843ae1cb14e37"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "cab4a0061c343d0df6cc3df3da5c9bcb75ad77e4c6aa72226113389d8458080f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8bd50f8a298bb635cacfcb14c9b504cce6efe1e95738df91710a64c82b06624a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1767f5318a5535165405874b64bde50546db5f2a117075743d4119d1c154a5c6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e4116e2dd85d4ab03b22472c22ee82fb0de065ab505a0e427bc0d45be28993ca"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6fd0306c468d94bc1c1f5ce4b6f02279bc3263be75573c4b5ccdd5840856ab92"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "96d0c9df72a0a0306e147dc36a8270e94906cb8b25c8fcf8e6b97fbddd96f133"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7d72a6152e2d92a6f150e4c455409a70fe67957ca2d643a4ceed76e5378999af"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5ca9781a1ae0543ce537e1a4905b0193df7702ac6f64160f1ce092ee3f557183"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "83656f5204de2b0e464d3c937e782ebe3702edb37db5517825136b5c56533907"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0b525878c7d54560f96e47b311aac86388b42e0d8e27b352bed24cddd8d9904c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "91d7723cdcad22cf1945acf4e705fecca30ffea9483703552027f9ec7dd55ff4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d73c050a9bbbac377849d042a919042a8480cfca3c124bc5464278ae9176dba6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "39d9843ee816571f2ca1d6557e1766f92b7f2bc5ae2f5c1714605101ef55124d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "df94ff0f0275abfbaba7fb3288214c907b693e9e8c29b456378b52cb31e792c5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ccd67488aa4f4a9891a335c068716ad6b5ea07d3d7f6b84a2746ba4dbb13f271"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3e028e008ddb08c95be1f03f2a9ea278b2cb9156db0c7b52387112ae53e58cea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "23ae3776b77ac881bf05132de2c7c165fa1c8d462db6460184c015ed8cbda721"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "85fcc0244f033b90edaf2ed558a32e10caec9ab5117bd8476776f43491342f7f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3abf1a2e3b0838605bba0ad917d86a8a472b68dddd63578f7155a2f3984c2773"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1eef678286d2e39cb924f9b66534e4ef8be6c5b8c3a3e540747f5f82adda7124"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e52451481bdffee2e4c79192559128c6c38b5ea5d81251e72e52b7ed7f818aca"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "02cb68acb51ab2c4cfa6d0c04965c2b84eb37e3d702567e60fdc1287b37013e4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4b246a84690d8de454941b530446db8f2e96a19daa1f2596bf3f1eab1a592c1f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9c866008e05b667dbe8784cb223e52ce39ff16026f3df3042444a596ccf18854"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "82a0c804619e58c026635a5122b92d24fee5353bf97c3c0c47efe70e3749991a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f36c5bead2137bc74e74038e087b093d1f322a18455fdc53ca0645637a62fd23"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fa19a3c261481ac97cc3d52547c529fb82f38875e0e362da72796ac4862c8487"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3db64370108b2ccb7544806bd3959756fc20ab9962b874b8c9beeca49e5195c0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "51438f9faf13f4a3b4bc93958b8babe6e3df20209986a0fb808b179e53c97ca7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6d95c1fd748443cffadcf4bc4d6caf4df9fa7ca6b450e104757988d56f8354ac"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "213c461c2a2d594a3bd3aaaf6491d320c9daa47f1ff69574ded7dfc3ef7df3fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c0cbb3aaabba53e5a2bb5a3c5d5be12f925f699289ea81fee5a01a79083838d6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ea3991a5c2d4a8df9b6b272f1142bf081d0e22b3ba40695fb44dae5bb2b54969"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4762c04d7ec2e78763fd3a611dab6036c125a9f4af82c08f7668109d5ae08ab7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4211e337f1e3999d0f71c53cbc64b24e19a188d6b90c43b18fc91820bf7ca7ad"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5d5a3ed9428bc8c077dfdf9f0c1c1d23c8d471b709ab37f5ab6e47508a7fda8c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "cdbd6ed269140d71eed5ffa7d21e0bbaa4eabdeb854e0ad2d00e9dd2acabc185"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "58a0e6d2eee0748821cb79ca465dd29822528072f899dec42f7896dd98ec81f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5ef196a371d2ed3f58d7380c4704a030d64fc6fe5d111a243517810572b3a56b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8646b917572bf061ed83cf1a9c83d50a2fcf4061c2e7edbad6af886f51d26ad4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3e318171c93f56341356f05603e63cb3c86184b79361aa42308931b6681ebcd3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "37f2983b49a51c2d71c4d8c8521367bfe9c58693ac07b08af39b7a7d2ffbf137"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7c4e960a7e9a4b8a9c56cd2d4a4dc4524ad23594017d7a9381346358f4df5603"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3bdef7edbe187048378a442c38cb96138240b7afa4ce9f802344d14b6dd39533"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f00db251be55e3ed06d20bbac1fe3aea87474ddceadfbd73b898709ffbfd7535"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4052ecf1d07a07e39f25b1859bb074719beb5a0e7765d2694d034b5510f19f65"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "adc851772379c73c8aa341b5e3fc724583e5700783ce7e9ec92fc7029cff0069"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f962246863fa36b522f8c3bae120f7c1025a50d96571cf6c284ea42c6770fe1b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "ad3fcb257c0ed69e318c2abb5cd6a25c837c628f72aa37dce6b861adb23b312e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "ea671fa63a2944c1d3fde2279f9a079d9c9831cf61402f7b1a09243dced135be"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "128930cac12bd92157e88e43de20853b855ad4e34eb57c6de2769a98ab990527"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "aa8183ebea711fbaf4adb6e5b4baf73c6e9dd488e2d8bca598811d549df76973"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "b0b022a684422ec0f0bd9b83981fec14eead772029596c24e876e0d7d0733872"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "29805c0b6e767a7d774a83990609be0187d3bfbe991ce4d8988b602d9d80e6c1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "127056a533ce013f670519670b48ebacb3a0e27f359e831289c8fef6413bcc67"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "526cf1578868d7af7fccfcef04dee60e4fc14a1f4e2e5f5ec814c4221e6601e9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "a9a984662244e0c278b754bfae19699956855592f1b4276108673df23730411d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "6a250840312157f06d34b34a11377a23955975416be191c29782338f8f2adb54"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "38b0b81dab3f16db20dfa1b9377e6262e29536d759bc0e99642fbc5a89837558"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "e2c2bb874f07c69d40d2082dde7682de6ea60a8bef9b447da0a9232bec3d9eb5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128PersistentContext", 115328, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "03ba3db9574c896a1e9c8d8d7c26892ee6bd8ccb0ad3f29ef5b0ddf490036d54"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", 114960, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "473fbc917bd306914e10604b5a22be9e2ccc25f112e46b4ea6f0da81c4fa2b19"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 115344, 512, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "53f43af5844d997cbce6b74aae58990008b486ca5993faa249a229ded949cc0b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 114976, 512, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6996ccc471118e641d54c59eef081befe0dbcafd51d4c274dbaaf2249e7c4757"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 115328, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ac89ac9e5d1637fb26f163a7ca069bb9a952312d50fa8996fc1ba70b7f85416e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 114960, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "58f9d9f55d79b248984bc691f4043c5b0f22f5ff5f1f515cb52588392b283ef1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 115344, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "63ce48ec271a232f430cae2c608eacc1a6fff03c66662dc4a9fd5e032ca511b6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 256, 128, 128, 192, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 114976, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3783f2a41326334d266936a34e354f2ea15cf9bdaacbc3a34cb0e0546f965fde"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 230704, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "f424ad4d2e384334cf845ee1fc2744697848b89201ea658c950161e59866f2b3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "caa8ca93037e0a5e4493742a58d156255a973c570f9ebb124367111b162c5fed"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230720, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "ab05ef4ef34efe365e15ed6cf8164dd7803e5b1a184597075bf8602a9762652f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 0, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "20f6edccb714e89d396603074d8504304a85c18ccd0eba81d8c2dc2c28aa44e4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 230688, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "a0e498331bed01264f5602beba9d29dfe3f90f8f92608d92531b7c5c1a260cf2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "d96b68c17c42777f2b225e80389ad2d4dab3b08f5f80556a337ca8d50aee7198"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230704, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "07b46ba98d270535850f8c999cb1d6e7d65f9635f894fc1eae35059d516c7789"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 0, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "a839fdd15899be72ed3299df6dc1a02bcb9b751eba4aa27420fb45971d9976fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128PersistentKeepsAbForGen", 230784, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "1d171aa591589edaefd9bc19c349866081df7d2df9d03e7a648b061d3714c393"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ128Kv128StaticKeepsAbForGen", 230672, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "c955fe37e50a83f2f91c9354d6e2413e32ed084b6f97ac66cb5b1ac5ecf05558"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "483a5d4cd86652225160910e156e59f7480a2842d9f626a3957499bb94f48640"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "492fffcaf9d11b5093d97264f10246070b7ca8e1884f750c99ffb0c6cbdcab73"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 230800, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "754d319b8271c7c9282083f7a542886798550a01a310174fe30aa8b6a8341ad3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230688, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "1cdb537900f430e1c385100128cef8b3d77510036cafdacd6137facc524bf023"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "2a6f5a4ddc61469e28c19d82ad71ec6c2c9f789c90644e687464c89598205f13"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "a1d28bfe57fad4dc12e6dd1779ec469b3e17a10c08c0c26f8a946221e945a543"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 230704, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "7b598e076a756aabeb26b5769fc7933a14eb4d04efab4a4433eebcc7216833a3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "2ae39d6bf48689cd9c878d3234778418fd5aba3c201c6c54610cb7e482450ced"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230720, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "a4eceb5a819d83844075bc072649ef310493585420f439e9dfb1468b7d84ff34"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "c31101b7b14099016f6fc5e6b0b3e5f1031802bd22a080a60f8c541ade6bca98"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 230688, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "4dcef4c35fc0b97e0bb0d501c11a2a98963e25ed0649fde4d34ee3cfedd6f9e6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "dc3c8f26eddf3af007365ff77ab5c2f28eb248e093ecba38d1620d0ee5bc957e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230704, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "245ac6f11d56d02c737bc85911e40c074ff3452e8727f1ade87107249c271f35"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "56c94bb6d4e62978cd3324b7f3e130221db952387498f7e7337662c9348857b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 230784, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ff07f32350a873250a4f0c275c3ac49d5a893dcf228ea46f787a4897af99f7bb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 230672, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "ebda1172ba8c65ae3961fcd7e8aaa00adb19b1982a5c7f9d00f0cd75c87236ea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "e8b836812b9321102021bce658b8362c25918506b8cfbad371d43874ddfdb61d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "4e3b0adab2c175db912d176e409a37b640296852c179caa9e7b06a93080935aa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 230800, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "b1d874a6cb0812d8e849ce1dc461ff4af9dec1e7cd6cb3fc3de37168710de373"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 128, 128, 128, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 230688, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "692b7814f6484a1992b5d4654a786f1c00f8d24f05a7d176ae8d0ed1038f6259"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "3b7a2cb00a4553500cfaa55cd4083f19cc6e535570fb1edb227eee56511b90ff"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 320, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk320HV256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "28e776c211623d8643933531154004f418851512cb2ee33ac18ec0f08cb0bbcc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "0985ce11cf56f72a6dc118d184b8397c29a62c7323972f5b294499102c978027"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "69549a9c3346525040f64a9c1b883b759acb77a1ec02e50f4b1cfebf2052d091"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "37a56467683c1c2fa647adeada6d60d4b14dec1cd773beb939a40adae602bb9a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "2721c881dc996a0cd3067c4d450b4a3e987ce993117349198c65d08fc023e40d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "a2a6d1e34cd5c823cf2d94e523dc2d2a1b3f0a7f5fe36e78eeedcc4a23d73b37"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "af580331c39763324ec71c42f13e15ec7ba4c8cd43b87f46be4015f32a0aee89"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 219680, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "5c5f55707425bb20c1f7eb4afb5db924a0c4d40ab2106c4b3d4bb2133948234a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 219776, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "c0b4bd6f6151d1e7bab0e515eeebc75a449a4e6dae4a73db2b6ded16e2aeb21f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 128, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta128PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 219664, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "795be65bfdd49deaf53fa197a4344ca1c6298c4a5cf949c20a32a001ce041a12"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 207656, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, "3c1948b6836954b6e5b9d93110cbf6cd796034678c273e4a4b65f0f51a6bef6d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "0b1e1c70d3258ae26ac0b652a4f16bfa54286e89f492af358001ac2e16de496a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen", 207672, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, "eb355754a1a652ea28e1eb3bfda347a3d2e0b9b918b3d922ef51f146ff71c0d0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "6607efe438b24a1baa6a4c0de6fd63b8f7039b1f6a3c5eaafcf0ec6ad23eb5be"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 207752, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, "59f278a59ca32df56582d0eec584c2aaaa67ae42b38a6aaedfdaa3ed5e612b53"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "20235490d83f1f43e5311b5dab68cd19bcf67b6649e52942876add78befb96a0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128Static2CtaKeepsAbForGen", 207640, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 0, false, false, false, "e1814bf3cd5fc31beff43fa8833651b44be902acbe104e1e321a7f3d6d9b909e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "4652f6c38008571da3f70a59f4739f904acd6c46ddd05bd9e271d08e5dfa2ab9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Persistent2CtaKeepsAbForGen", 207768, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, "fc145568a92f4e6056e7feeae3215986fdaa7753ed51290a9c99bf12c8ee473e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "8d3f9d0206be6bd331dff36d1a3986a4f26d571acd2529c7139549371690e5e4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128Static2CtaKeepsAbForGen", 207656, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 0, true, false, false, "08af5b101bab39c81027a43b00841ecc61aee85aeb45ab326680cc029bd1f7a1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "2e4830b0e5166529adec521a016899fc7b338ef3cf8661a47ebbec30bc91d9ce"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128Static2CtaKeepsAbForGen", 213032, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, "8f09825766073a875f33b98306aba1cf7a54001eeb781b15be19d1aaa5244da7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 219680, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "7af6141ed29985f17a78464f8500f5c3f2e93288bd02592e96379e57a036329a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Persistent2CtaKeepsAbForGen", 213128, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, "467ea000330af6673c5e3d5766968f0d22a0e03f0a5e00d757c37b5e356e4175"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 219776, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "7e98115bd664e3bf9ba729137a53372fb790df1855583bddba7106aa7199744f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128Static2CtaKeepsAbForGen", 213016, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, true, 1, false, false, false, "ff353d2382444c62ab992168c9a0e68e31822af9c2543866ffb53a8222b57c0a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 256, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512HVPerCta256PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 219664, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "e21eca5ff603c28024fe8ae2927bf0cbf4ebe1d8966601f52f92d6d0e470c778"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "6418ba030045e2fd2864b51e44da769b8c8539d10e8cb7097a8b7bbd50dab410"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "4feadb6d50d2a25c55bc0aa31a87fbad0408086272f2b9b47e9e3430d6d4aa34"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "43831036cb9f576ba22e9b2a0c56e32e4287e577aec4e39dcad8e3107d4fb077"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, false, false, false, "59be2070d0f05cbe72039b99ca8e0851967bde426a606deda38c00eb4632e685"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "5d86aac67e0329f30c2b0cc64afd4cde9ce0cc9d1ccfe77568be14c1c2ec7d2f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 0, true, false, false, "0eae82251aa1a63d189815b07b88d17324b4fd0c81d22378e44f4af29d0ad1fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 219680, 512, 2, 1, 0, 3, 0, 2, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "378cd05f7dcb755e662ba7d2282137b62161209e0bd3a9974d066c11ac291ffb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128PersistentKeepsAbForGen", 219776, 512, 2, 1, 0, 3, 1, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "87f0d53c7621334a41c29f0dfb0383ebeef32da0a0d1a95cf1a3b535eb18b091"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_BF16, 64, 128, 64, 128, 512, 576, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OBfloat16HQk576HV512PagedKvDenseStaticTokenSparseP1VarSeqQ64Kv128StaticKeepsAbForGen", 219664, 512, 2, 1, 0, 3, 0, 0, 0, 0, 0, 0, true, false, false, false, 1, false, false, false, "bf5e4da34949cae4303ffe2e80497f8987bf794bd9009ad3061b4666dae0bf6f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "307d882fecad4eb380da3fb2a70953757513d297da5c52c339c169e222c7dc8b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2e8666607868e075eeff0dab23abd3962e03d220f1e1dd1a0c5da98eeb8d9958"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "beddc6b90c640545b9935aa69d7398e2e067301f1789769b8c2b61353561e6c2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f9159eeb0b79714b95d6fbac13d1e3f9521bf859b779cff2c36e1ae5f5207652"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a80338ef79791f0c312676b2c26683cc82f9a0dde033b5a9ad9c1cc2ba11fc77"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "76c402146f72b8c88589aea768cfd582b9986832bb18d652cd3550f176b9c7b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "6488f3778d4d5eb10a107152ac15d43ab1552b408b6e64484f7442cf55248e57"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "8bc3e6cfc6fbba0b0a93c7f816c291f090aa8ce1a21e6f63e7b558e64d9af1a7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "f6c48eb4f47b3922f905e2c263b7a982af5332eeb4ad810efbd196d8d85bcbd6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "47c0822ea2649c03ccc67af387eaff7b25c6f3f5c09877ed63a6baa57abf6fa2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "8791a85dbd02851f9bd6c6d068ee6c59469896f3ed0d6092f16966bab3aba8ef"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a5129fa90a271abdfb762434624fd3982bece02632b6f27c9582535c3cedb172"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c21c5f1400e60a7fd8232de8aba8ac318ca462eb07c2450c1952fffd239a9569"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "f402ff5346f6d88740c7b8ff6c7d6b3c166e2da87a1c57b077ed307badaefee6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "e439a1ddf9b7330d6fc669b85cd442771bf8a5c2a3dafc0466ce7aadf57ce837"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "93ee6eac1bcbbf2fff3cf69536286400b1bbe9f36e7afabc0436735921e71bf3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "82e20b241feefc9df910a7d0c0b2e24b0d12a9ebe6bd04d0d08da44aa90a8dcf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "7c40644bb494697354095411b6a193584b8ae4a4433dc6fab85fd27bc699b332"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "09cfb7d78aa147f36741fb53578881940a99f799b546478354c9930739eb7b0a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "588e3d075f5161cad3f1e01c280546e66a092f618cb4c6007f4b9fddb7c6317e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e75476318f528f892a490b62b99f0937108845d414ba9e3dac00e99148bd89fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "73e02c8e846e6b1d2b3778715615660f55047dd30c13bb58a9c488e3ab8f7786"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f6a141db8ade80cd9623bd5d167ef7509b5528e94976493579d847535e1d01b0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fd5baa0b75d3384c9c7694b4bf21bd913fc2911d1363bdb95ea363abe213c965"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7e4880d622a8cfc8b171824e4b63f92d43d85145699f810ec47994599dc2e267"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f030e0fb481e9aea8f0a24363e2dc751d13a93b22cc1d902d7eb6057f8c9f703"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "37192223f2b8ae81565751aa21b838773668628cafe1cd70b272c086295f24ed"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9942ed2c147398fde8ffd16882c333997339c27bd49c3f432fc99655e338b83a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "650c511df53536f35fdb3942b1d6bb9350620d18510cc888f871ba904b69ba30"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "130793e10ef109e5e6d6a506611eeb098f106580dd8fea460716ca2474fba886"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "000c981f40dc9023ef1cab0507e5c35d729900545e054b96efb83678b7000b4f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1dc1da2b5e28cfc02a46b702b52221e064517cea8a96c5eae7299fb3715e3de1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a0f2c137c1bec963eed3fc356426f46598b94953fde590070f0a1b8174dc8ec0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "dede58d45dc4e416615f0e4c9e05ea2bdfcfd9568aa9db43fd8650f3f6f8f6a3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b1c200de84d6b1c01b691aa37fca6d2aa1ff33e01737b51c6e9b65407c0bdb49"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7c92ed25093e1c7ad1d5865dc89067c5d5aefb6f33fd703fafac548a2b2707db"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9df20f09ec1a7c4237d4020a57ff1759d16966d2772a9510c8137ca4d2469e2c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4a2b8968343034bbe3cdd20f81dfa19f7fe75785a4ce7eb323517ec46850878d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "269e69aef3062f797f0ef2ebfa863aacfee7f1a2eaa64d8dfb5489b3591298e5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "84cec897c2e9e27b9cfbadd7f7240a4e9f4c48313bb6f240257a170df0e06f2a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "804b567cabbbae44f6f3625de978ba13328e11a1b952f99cbd07d7b2da80e17a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ba4a3632c3e70f1f645d486086c0a6b8c3766af8cb2e5e10a5f75e1ce5879993"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ea5047d3ecb5170b1cd300db4551bad410211a3d955a40e7c9214327d5560315"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "02576e9b7adfb7b3987e07a516ce5afde2ae9a3a0a84c2c58f504d0d1a606608"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "51844bcea97bd425515e34c520ff92c4a4589f2f7a7c39e28d3e43b976759431"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d8e7318a8a822914ca39dac16d24cf9ee07e77d9f3a370d22e293444ae5a0397"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "112104b3274a8a4b04f46b38612d86d22ba896bba0df3582389e7acb4052ffa4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0363112d29686860c6691685f34ff362b7f7fbcb4ba48a222d446c1b2b3b4e7b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e2571407e7f1fad10926bada891a6dfaaa5d51231290c340af13ddf51a6a7957"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7666919df55f9b899d37fc0c826f7ea996bf3d2cddc4d5a6547defde938e6326"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5fa0062888585c12a6fc9ea4a95d174c6e7fe665250965c6e8705c0f6245ec6b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4109342dfc073e5d9686034d9497e4a4b3f33a9aa407c254bf812ba396a03514"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1c6a1755299afaef47d7ef0126831ed69f33666d3b86e9d2257cb7f49a50ee4b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "404463f79fa7f0f8a73b97650668085eba61f19efa16635ea81843fb394211af"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fd7a1d427a6fb17e3add15ee59034d42964b748c47458d1b51bf924f4db2afb1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e13b25301d62e22bc36f6ec8363f22bb93277e90708890e8a337caab01799be5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "be6d486aa56e91777cd913c56f985f16e67afeae767feaf36629b533b9243f0b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c4257940e474cf3a29454011f2eb4813b5c9015d680a63297e8a389ebd84cb38"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "732a799011c26dfc8190b2ccca1593a728e8ebabe6b7a24bf6d66160fe65ddcf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e0e9bd7288146fbf27a8cd09cbcb79821d51e990e5f36584ffaf09327d269d4e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e735ebb83586ed6d6821a478215adfd155d16d1c149ce055123e1b3ebb9cc62a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1e1b819dd8e7e51485f43706e6c84219947d533eb8c46a263df407553b00dd2a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "cc89ba89c8fc48f5acf9c7b6defd336744d88cd84326142d8c11195ca40abc24"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dd503e619a135c2830865b77295a2011fbabfd1ba7c237fad9bad226dc0c13cb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "402fe34754a1a70d278fc7528fc8188f709a5df002d8005f14853699bb3b3ae7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6edde130b289c370263dc434fe0dbf5f49cbdcd3dfc7fa6512d2be67b28e388f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7f3271cea4d675d2cea130b36ccccf3d42e46f006d652b3b67341c09ad315bb5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6b4b0fbb72799016f5b52050e5d02568be1b2545ab493b137ecf4791455d7c86"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6fe7fcfd11a03f63c5b7957ae92c139832b81b7fd5ab291f00e67322e235ea71"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a2f81f03631d908a81f30927c597797040363bad7094d616f6e200b5ece1e3f6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "98973e941cddf04353d11fbdc6f037d24a05f5187c6eb8c0dafabad760a4fc0b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "28dab1991f2afd7e20b2ff0cbae0888a1ccbfc285b6cf07b9c8238a03266469e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "04aae596f4ae4b21195712610e4630a725b0f09e279c6122ec7596889399cb9d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "2a2e12d211dc12a9a35070a15720813b47e17eafb655ea5fd0d9b8587554dac3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d0eaf10e30d3175bc265aa40136664d19be98f03ea7185de360dce87532c71f2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "db3046413189c143d0cf6e6dacc087b6bab2ba4e8dde94210745a695e98bf320"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3fadee13bbb7c666ad23f12cc2466b9b76197cc4c4be201b49219efd8669911a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "3491b807b06090f1d01456f3aa11fc90818a657f76aa782b6e05a460bec10382"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "53474a05ce81bfd6ac23672baa824d209fbb925c1cab867093cbf88ebf7f1d43"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "e50842f99106cc8c66fb415bc9b45e79988f385839c1c956f57a87f1d2d29ea2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "88a6145a802c5422f19e75fa9cec2f88b61e3d293bf56980d68127cee5854f38"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "62267cc571a30e4552f495532b97af461327a7d23aa5a14557c9637f384a5c32"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "1ab22c880cf6974eb3f73a0d5c27bf92373de1224c42cfc3bf600fa5b8d9a4f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c05e905b47384818f64fb9130f9bee0c177fc1183790ed87ebb010994e4c7a15"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "15e1544451f8199f33c9e557d32a660b97dcaa6343232b4ff3e640ded96c5bae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "5f6ddd404e55663f2cc957a9f8dc7a7fd9b1290be87965a457d996aa65c6a1ad"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "808b06a8557a9c71bfb2124fd711417e262acf6f8234d5a93cc8604bf2100030"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "98eab98c065365e545bd86f320836cf3563589c34bca30a20c8fb140838b0bae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "7cddf2d7799a8828fac38900a902cacba019898cace527283e86b9d9f9ecb279"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "41ac2ff4fa337c1e077cfa1ea70b9801ea66b105fe84a5c17b9c0e093d75d48b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "270f6568af2d074ade846ac389955450a32663caa5faad4f5fafd383928cd674"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0dfed98584fe32f1ee0dffa51b30828cdbe7bb955ad31e12c158c26af45684b1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "591532e688bb70b8b04ca3b03e068f33e0ea8dbc16bf32f30bdfea086e0c73c7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4b80c8f2de0b507d7cb75d655469b669da9f19d0ac4a31d5672769398f2c66a1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b1404c447ec65668a8b7facc265c04df8adfa7688005e40f23ecea98fbcc7337"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2d0cdba8b00f99780724230d8edd2514e3b0d81622ef5c9b7dcd37d35bbef401"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c21f6f75d2840acf6eb57e1764d9dc1702b719156a28e7fe5ed6c9635e77e838"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7f816d620af0aa7110d0a9b509f7e18dd6dcbd322e098366e714c9dd8ef28024"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "905c09f653c6e6e4c7721a9bff851138d73c7d2f8b6c539f0cc1da7ad90728c7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f9487e13393053dd97e6ef0534a6c4b8f7f153741f0be19ffde09210707e708e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a387690f34e5821c878f65922a9a394132c09ac77b270cace94d51ea5ff68cac"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "54b11ed84ddcf910a75485f5388ec5f91e68f4e12aa4f7cc014037526a7dbc04"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0f9b5fc14c06deb8bf340478faec33382b7b373830bd0d3860f8725cc31c52eb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6e935c1ed853b1c42290a7b4b0bfdafa944161d3c0c8178b5264a456b1494ea0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "11bb3de651061a301c3512c7d758096ab6d1b2d2f8d48a71974abdcf59deaa22"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "14a3d889acfd23f06a19e1a376bc30519df39b8a09f2ae4076e80108140a09d9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "64c87e460fc502312ba889919e2a097ba8e78abc316328f2b85289814bda87de"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "fda13f6559634246dce2d8aa568dec3f37c743308617d8c327b1da32fb19818b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "aac8850fb1c22154ada591ea964a0a204b12700c7d44ef6e4001a2b86bb1cc2b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d3f352c9c6dd681ffad9a083d88f76b10384f5b00dab4c5518f483939f94bfe3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ae6ea8c2fac768316b44f79716097cdd8dc8794bef28d33e582322fa9d5245cb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "97377149f44f4b09073d54d6edaa8d4b3e85bd32198deca165d8b454a395b6ec"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "118488a3acd3856f3f25a5f3a3e1b4826576788cc166b5120a2b644553a86865"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "93472c711430b9ac140cd62dd1ec39c1493047efd9718adb068247cf3e1f0da4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "55eca546687cd709911cff2e53534f71884c65639fb070c9f8260e1ce65a851b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9ba982f24104c37db6029ea664865ac77c244e1eb227ed016b192df27c9b676a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7a82e7e3fea9ad696068353135457a9f3eebf080efd0e661c3a5c2a08682a8c3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3a334abba626d788b8558699f1e98b2645844dbd3dd5f1cff5589c0d5abecaaa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "31b7484813c54ff7d084ca94e1267fb42e6f41709ee961964a256f81583a3459"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d158a4641b3fbea023ff09286822553d573ee0ae13c6380ae1686b76753268d5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "81548d8bc5e95bbd52c7291b7d026e17e1f9b7b28291a8cb470f469b387b16a4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f8e584bb2f19e9d934d2c4239b68d980422f0844d284edf51a420c2aecceaf2e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1af507dd708220d73090455bc6d8ae67035503715ca02f625ff234b6cd629f11"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "276e65f3bee81b0f58b79ea6932c7b20c9b0d9fff58d9529b9eedb28154d0338"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b1edde5470b6eeca2b8f670937e740018674fecda669e9e24dab3e09d343690b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b43c6ce70f57f7d753633cb9cb2b5e586f270b727934fbc6b4c61cb86179c33c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c5541ba10f06241f39f82e6e0151ae4041f65d078d6e68296e3ab3d24de38a29"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cdc952c625cd6e67c75e5f7a029cea094b30a9814cee2fd7b1f95876ee40e35c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f70891d730b1a5d5128d5a7adbf30279c3092bab26aaf78580a2c2c0e18fa046"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5beee2b25688707e04f790d86b4d3cb40da47c52accd0587856968f56d3f568c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1d913557f28583e59dd5073a69b4eb531164af57fe26e3964814782a74c83f76"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d0c6fcde501dbc4210a19de4950fc80c44374e914efbdc0c94d2d8e549ee2d98"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "dbdc91a13facce65c62890c82931178b39f68d721628985e78633a2f6b4b9b97"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "412bb241f443b4c2d7131814f0ee12cc0f572fbedb11d4ac0ccb99b314fbdf1b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "cccdae3139cc77fe7ce8af87a9795dcec9b9b5ddafabeeb4cac6b892307354ab"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "89af4c96b9d4a0ac9464ef6fcacc2fd46ba4367d799256ab6678e67a9ed72e95"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b25ad02ce8a178896e3c0adfc704feeb9a043264e2929da287e5f28c0c8466d8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ee57d335f6db226871e12b3deb15ba4912eb9781f05cf28256995f43272b35e4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "90aa6b883128c4f2e8fda8107484c5c307504fbd8f44780658cc6f6420eefcd7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ab8efff5420d7237856a82e94c0df5dcb5cc9c31993faa8a42e1bdfddb38fda6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a2e2e2c0ffb968395c6ee95244c5c3cfedc4ab4e45b3d4b3fa964510c586a159"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c76a9d03fdca74dd1924742eea637e1e353d2967337944c3af1a410acd9d2beb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "47d2ebeeb491cc82a8e3fb2367f3de0d43ae0f288f3b10317091342da91c4c82"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4f74963f03b2010ad2876dc7e96b3d98c166c23641aa858c779911d85c0745f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f5ff94e6896c3cc690f255d197172201cbc6289fb75bc468b58d21244ea35b2e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d6168b3a513d84808a7860ba9d839290afb3c0c13d35eeb7a63adb492a1e863e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dedb84f2924334137e993e16fdb0a30ded9632d931507f71b00dd859ddb9ae8d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5705bed7250027fdbe19606e2ee0214dd07ecf2586db2affb43722d09f59df0c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a2935a92de9cc95f5bda96f3008a31a42bbebbfcfe7730fb73779db50b4ffb5d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2b3e51def15506aa2be1ad8e41439e2aebe1d5fb0994e60368c59816024ca21f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b3328623b484cd93da764a92fe1711f32ab1159fee19e465d91a5440ea312758"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c4be26d5809e8da21eb4ba0f1730dcf24d3c369e4d25d2fa484a355cccefcc19"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7d384a9c46a816dc88d6d2f1f29d97c5b55aac556cc2321842ba20ef30f52370"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "51dc49c8a583a36991c7a4a35d8e2a18f63c0d36a4b034f71a2c501b0d74c2f3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c9a6170c18d30123a1b58d9600de1c619598a76381bb781e31ff0dcbcf949a73"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6d18e23649ccf6fa184d47ba75e5c4840cb7a9f40e47905d7f92064d205cb15a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e3f314b880d4dc5ca1fbb60910e66f97473097fbdf4675eb44fdad92e93c2d6f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "376cae64c1f0280266ce7d2977de8b5eea2d7b8020032077825834b865a9f1b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ab44168767e262b508fa497f3ccac18898a33fd9561da51800497eb6bf6df300"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d89cd255df073c03ac116bac098090a12d62c18484c0d94c089ab540e28d1e20"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "070981e34283d383949fbc6b0a532991af05a656714ab5988eeaab22b78aa480"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "40026594112948abcf6ac0d4b55a77a6768f800836d36c7401664bbbc456891b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c9e791fc3dc7320c1fc224c9fc5f8a0c9638ff5e61c416cafdfeb5f45e05b2bb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9bbcae063440c3c584546684f99fb002071f201739870c8b0095fe8b2485d528"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9e462bf749844c3fb0cb546c96c4d9c9db81bd471e30176a9e2cc51f323f9cfc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c088cce1381ec5c668599ccef9a2001ddf789d7e9a5d42b3ca78847e5f8343d1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fa5b841699ddfb3e70ba38af0c45bcd50df93e8a949ca348faae36847d5783df"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "64138589db27f1596ad562e163e4a450cab5867e518d9b4598fdffca93302e85"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f5a567dc551fb9195273c39b25a6713466611e8b6aacd53df31a4e5be2f60e62"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "37e0ea489d989845a7b25abd08168b29eb0201803e39391b1cc35b00248da541"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5464bcb1de767ddd36408b985bf388a4fb89faf72d06b846bc31db8e2fdc6ab3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e458ce4a9c9dd76f40712249739473f0562626241b7422f5bfbda2a6e6d77483"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3abf70179d2e5b1ec69bc58cc11831a3dca36a03a9ff3f0940ca9e6a6afcfec5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7766fa40fa14e3f33e4938a0354d3a13c631d843441f58e211a68863ae278c9b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fff505c539b08a8dbc88da30990f2ec93f5651588bfd84be68ca574e4d7db020"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ba8474c4efddbe4f8ad30a15021d6d5256010df85051a52220c8ec37067608a0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ac696aaea563ec0a712668f168a1e9992ace204abb456dfdd1468ff87d8c491e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "eb97ac5cecfc59fa635db26e5e81af181dee43f4a1d8bfaab6835fb48fd6e39c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "2dca7fc9a36a5c51fa6e06a32705215ed55b4f67036fa091f2dd2803735ad821"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "cf1c2ce6c6394730b913294f84e724595697819f19950e3717811cfad225b2b5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4b0682e513154603de5d53611cbb2b72d00694bb09dc5283b223fcce00ce55c2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c0bcef07df131b143215605b23725d2262942a45753d9a822f0583fc73fbe82d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b852496e0c9afba04083666174187654bd43962e696733d69b18817f8cbb3d67"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d17a17093c225a49d67521e5e42b6b4ed50fa995c6f6fc1e03693e37bb830875"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "58e92b04883b0cdb153275052821adafd6e4a8af3a4d691328c3350314e19e48"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ff0300e0682382780a1326aeb59194d621f5aa9a1f4ce35559e3b0e09c9243a1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "23c6e8aeaad574fd956668bd60bbc108b74338398cda66f0831382e366a99b15"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "551f21857be651913b3a0f009e342434b1b032852ac639b8b693658cad64cd3f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "67b9d2612850acba286d7edc3e3c2945f136053f66b7b534a81850e39b55a34e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "77c6ce8b1a09d898d77eb4ec4f53f8f30f06e9b8335961e2d06939ab8b4c92c9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "557f1a2752778c30c1e1891df6160c16966d02382679398bc9053c67a7591fea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "806ba79537a8b238c3841f07af0d2e4422d0b80c2385e2382ec12930b9e0b991"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3025a0970d3f02bbd8db08b532ad50be750b609ec20328805591c5afae475d6b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "4d7dcf9fe94dc4c43899062f27aa4108ad1c51f3365deb21c97dc7e345a981e9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "7b9bfd67c5a7141a1876af79b9778d3a8ea1c0d132fd110710b49433f7bdcd8c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "8bd78d3235a70d641e8e96c4a517d1edbaab020b9c067df3085cc6fa44b51731"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "04d4218766813a1da0e279b784804f3f94756b9385de41b3a320fa27d8367b55"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "e90282ed8404070e08a0991182744f791e557615130332bc03a61f9451bb663c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "9234169076a4759e58ea975ed1c8ee857f18730bc68c920730764c5937d51411"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "04c1069a9ca0be22b4c712d5d6559ec479c0e680d7d52b87bbdf1c028e548a66"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4e559a657ae6b4e86ceab117ac21eec083a6330a6ae297cc545ea53f09716bef"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "e12593311261534da421d353de55e9862f0cfa03b6f9129033bc9147f4219592"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "8b05f541cdee33fd5f857d09faf800107243872198c149a17c86dc8c0b94cb40"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "a41bbbc3575fa2212bd96059663ed19ee15414fa8097ef26b0dbce915349bc5a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "702b4faae251a3fcd8539597a5ccbe39585d11e8a5602d5c744ed1495c91b106"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "572c2080e554ca1886cf2b410c0575d289e245fa3c33f4ee8570d55e0cbe0d12"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "cfd5263a509633305efed984c2c61241d73b749c4fe269a0d6a19cf2d54227ce"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3380b401ecf1fc1ab025d788bd23b92815a633f95aabb2f4fab6d790c21da344"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f84f663cc88a4257b22285bae73b71353fb5aad263b2c72ef8e977e7d483292d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "29c211094ef530e7eb2c5eb47add5008cd0d6b70d2df4e9c977a1a9d9fadbbda"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3f00c2480af294429b9aaa806e3a5f225148ab25bbe849aa07e989f2e14d789e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a69326edc64a1be8d75c385f36306332123e1a683127cdcfe91fce777f64155e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b21cf576c6b8a3139650706d0a589a7974cf8154872086903143f9e46ad3ebbb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8cba8b88c5a091fd6f77a7c678f6e5dd8ac3e24b3d0de33f086727d63e7ec118"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5ea4abb234e396f506a9d317dfc4cdd11ee7683aab314277b64da2c4516dfb45"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3bbd3bd469b1abbf9989b4957359d62f491974c8279ed3d2724fafb4818d9177"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d9315e4c37a695667ce3f3cc816395b1af8d8e8190e45f58518b1b58a47e3e4e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b94514adc070e8b04929e8c4fa789f03ee3495e1a6d77a5fd30db121b4391bc1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c37ca0aa7e9186004f4446861978d6a64614fadcb48f0fd3c6bb0b38ccc45c3e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "194508d0e78215189d0d921d3a638c4cbd9b121557ef0276b103b6b4081d386a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "dae697cd7555dc6ddfe786c3c2a1cb26181db47c3b7cb8f6790ac67ea245b8bd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9e0423fb52d8f9df8da244e61255cf3db649ea7dc0f23d81115782a657b439e6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d523ea496baeddbe3c887c9b9c807e4e908f275f968f4794d270f218cdce1230"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "acb14772b69cb9d900d7f610d1c277e02213fd42091d6ec6b816b278c1460c55"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d21c4bb9f5f85ddac3fdc552e103909d44236de14da617b477115457eb837d01"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "455cc69ab81db9fcba91ab6138e35e6af0cfff5cec511faf82c2a975f22c7b03"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7a247b5f211893192a3d4394ce15c9272e00e6e8a7f8cb3dc48157fa9a1ea364"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "02ab3792220547eaa85e33db6bf1eea627c976af958894d20ee722ca91df2481"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "33b6ee47da0093324c8ab315a12a23897873d40aa44d3ef98b5685ef3b6366a3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a840c1c53fa867435f96a9023d67e2ef5dc2e21df562a40910cfbe0faf42ea83"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8d5f5aecc75614000671ae0977fe7eac5541fe3ff720530e1fa6d6dc4ea8874a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "92020d52ebde76035ebff7a042c7f8c55614422ea3a6075bf662763b45a7f417"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7749fafc7a976de21be6731899d6f7ffbff3739ccf579ed31f87087e75ebb654"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a8dbf0ac9482b0f02689b9abbd7264ea63eee5beb6ffa426153b033823b02140"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d903d1ce5523145d5e12f8c294ddccb6b9b1cbab96ab73a1992af248f77f41b8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "742fabd71c1f45278ddebd556f33efeb73ee4873dab5af4dc1758e4856fc643b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "2cfdda48f953a73a4b433ef0b87b23a39cad727c9070307786a2311ea6b8537c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a722b8eae583e2221cbed14fdcd6eb0020409536b278cea15927b6e168c1ed37"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f371213fa0866ebe63432fb3f615389ecc1f2342a26b9bfe5f8b0272e6a8421f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d43adde99d099355237a88268f5e8c783a7d1ef8f8447222d04de2866c6c9d43"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6d0f22247052132ceeaad7f753598f55628b356bfa380b003ac981c680d143fc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9db8f286a8490427bbf07442d7428c9032781f59fa18345c054e1332c22963d0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ffef3d0af072c9538904c5d597f4c3a05e5f21972b9df92f782f53ff4c71f5a8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1d55eba7955875cc439f5d52cfe5582fbcc22b3cf667f97988633fae8f3cb296"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e7bfa0ddc8078b68e931ee26e3329ebd58b837c9a2468f8b12a4fe093833d25d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0c503099b4cdd5ce74540588eb431bd1de6a7e10cc9ebc16f3be8182c14c5cfe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "2eb2e9cef5af0f5825bb1e92ff14d3cf33dc7558511bc20166f92cc80bc7b0b1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e0abc5f74bc9bd73e56b5569be3971a3d2eb0366a070977d3e57e4a12c54d0a7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "df91aeecb59d9c9a5698f0c4457cf650edea52b48d81d6d0de68f790fbed46d9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7f3273e71768f76e7d1217a36a2eca87a8cd51c73939128fd0a3b71ce92f4e82"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8c5d8490c58b53729cf94713367b4f60aff1bfc74972823e382c46ab0b5d6682"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b84ed2f3c07cd6ca3becb394aa3acffedfa1eeffb01e58b1559f75d691319a6b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "bbd3d5255a7b3de7db092af773de0d4478f7913679bd36b00629e553fa4c670d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "16c2f7e38e0ea1da35d716297c2ec7b1e97f02930e9d7d4cf8938feb9e4b5fc6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4190fca3de2a32edc135bd73459b6fa730ea63a68959645f333a3ef8a98bcb6e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "972a958d20188983b90a53697a252fe2cec7576e1f6aeee1238820346b7e722d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7d8c16db18d618e2f88d879604e4e6a7a041c2f881f67cb0a6632b4b3f961ae8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "cfa1642abbeeecb36b6a840f9a1ac0c26f3d98ae37fb6905b78a2a2c704f27f3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8628fae3464cfaee7e3dda118a4219831632930e87b34e5117ab255c3d542c7d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7db0ce3bd6b8729502c5feb2b752f046fca19645fb41795ebded2b84bdfebe0d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ba3bc2aa0bd5149cd417f57801915d1fd42cf419aa3d45605fb809d72ed8a2e1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ec10233c25f58a5315020e9118b163dd17b44269af4aa83dd6b66a4e8a58decc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f1361d9b86a04e69bb7a856b9c6bd93fc0989804006286002ba97688aa44d429"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3ee5c53eb8d41b07674d27e84b9965b7565b4609a90ca9bb8efc6f4d109875f7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "14ecbe042c5cbcd6b883e990fa22da0b2806e6df2a31649f6fc413b1a0cf370e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "c8a21ac0c302c997d2702c5fbb98ad3d6e0f0e96f2e242fc263c3ec7479c48b5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "ba1572d5ccceeac0626864b758d4862f2339603fb4c567d35aa8e25900760588"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "3a289858cde144534530f313b5b10af067d469d59e02eb7e1928bfb5c35a4129"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "6db3e7bbe4adc19fbbb756e6cfda5e00f3631c5cbb2f576a7e6f3654b66c5b13"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "a2761a85a2576976aa17990c932582cc0800e1fc77c93809a4fb6d463fdcc735"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "74c3a07012a40a749046bf51022ef46b39ad7739579fc2be7e50f1eed8281b76"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "1530a104b144b9206713ae771971816e24ae7af89a926a2c4311319f277ad46c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "5de24305fdb9c43faed5fe783fcbeb589d17b55508ee40b3ff4ee12f8d7267e7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "ae0e0386f880e57456d0a8ffad6bf6714240e87c3783541b1a1c3b89a206502f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "2bfa64905cfb3ac9457b03a8acd2554d8c0bdecb0ce2f344740f2dad598ace82"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E2M1, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE2m1H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "2c595e0218f39d75eb4f3d92b505ae794bc353cbbdd85e95e04359c55ed35084"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a236d7bc1ac1eca658a65fe3b2129e81308def3ddb94fc36f7c8d5c1d36a27b0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1592da0bc8085b67b13834bbfa1c2c394bb13ea2ecb9cfd25d630422a725c0e4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ce4b44f797acb5bdee9d44b6874ee403635e2b8da305bada1622edfaa9d9ec90"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3a5b6b5bec9c85af265947a7f7b92a14e2555715b9232bae17e8e95f459610c9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1c8c6d9221406dc0278a43525d5c98a2a6636166a9400188d04adee97454b35b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "362f1754669be1d40104949130d3fb943c83b9104d76aa85a7610858d5097b66"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "f62b0c8d5c0b5b03c06a474b3b2f61a2da73f05559c66478b5a8c3be3d52627e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "aaf1071c2b338cddeb5ce60344ddfee7ea85ddd7f1afb7ad3a1c7ecc2e191fcf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "1f2e8e77f7f121caad371365f2e9a1ff35236fdfcd7b47abe709b35501cb53db"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "f0d6296d52e8e2cd50baf87b7046e0737437bfcc350f48ce92dc1a5c4648c4e2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "bed0d43c1c3e6977afcd90447cd8a6ca313e8fa46268c54141c1c598190a4448"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1defbee4e3f41cc7728de396e42a0b4e555c112c2118e6d13ed53f727fa15cd6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d9e43498c56c37189bcc53bd873db2045dc962a06478d5ca15583832cc389c89"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "fd23002b4d51659fda04735a81819b89fe1f00095589846789270b2d6ab703b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "cb02ee2ac5ed93488b0121b013481fd9d7f55c7e63eba0a88e8b05b1f028633e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "a117d76546d23d50622ca8df586ffc117a9755580c1a40ebcaa07fe287e579a2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "c5fbc81c868b38bad47c02efc47ce36b7af7da238e413c6e10544110b54dcc18"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "ef5795feeec19136a0b4f0a048c7edb70f85f25589e711b4c29ecc129b96e5be"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "4141807b4d4a7446a8d3c7f9cdf100fc67c701d3dfb33654f0d5f31a41c3a33a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b6a1dd2d255b0e087d8b2386913dcf8c7f5f31def5a44d229d3dec51c037993d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7434acd7840e36e27bacd0bc1058af34b3fffe3eccd58d396956e77f68cf4106"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "51b4182db698945997088226a75c97846e1cdb94482d4baddd932f3fd4e502f0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b28e785408fd06658bc1e181507727b35c739b7e75225089a3842d63707eb180"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "df0bf3d7d8508b0efc373d3f2477aac41de686e91c1acd0649f4afaf68c86a93"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "edf112d2b39aeb483e2ac27b1233fe907d035f82227c1cc12d6da455c03fde80"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f358d0db59d963f573d36941e66224d70717299f8204a89adc2c3bb66f0402e9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5eaaadd792701b290b1b962bfcd689ec2e64853bb0b817f731074fab45907973"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "bc709da4b2824d1ec8c6967c2520ffec652aef86cec1bda743b37209b0f6e5a0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d721e571c898765ff2f5467646b5262016ea6bf9411d51e4ca6a8902c8762ecd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a57e89a57b309d07c0856fc4d4d4993a0061882b3f073cfccf5157d2100027ee"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "260446fa11a6c6bde1b2d49b8af4c87b4abe7baa28fe565775a7fc4f448ec00f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "39c5a8a14ed43d46854ea807daef8b1eb7e983afcf32847bf427c4e07b9e0d83"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a83bb482666049dcb6d32b4e50bb0a691d45209a83233d42606b51d11cd1d188"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c9dac90e6b86fc4a9a1f0b54cb5f74e504b211c9aff40730b5f25f79eb26484b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "349d311d5c67a063b7d68dfcb740eeb87b7060e0620ececca411b328d1da204b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "550925723de527935e7b8478f57ff94601cba6bbea046f362a55706dfa494b40"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "60949cdef56f5e50290f014638559a87669ded30adf9a6b4dadffb739b940e4d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f1d9ecc029df3ec051849d3fe8df7c89df91b55d1e13c7b50cf3d8b72cc17e71"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "eee885d343245763cd9ba711dad7fe042c6c0f1af70f8ef4325c740b65b81f1a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "994831493b76d814ef9b18eb60302070877839ed7631547cbb499c48bc40952a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7d1b6f0ef5c3d533c6df66e957ff31d3c1bd7b6aff53d57823a8ba5e10427b89"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4495a04467b417816cab8d30a5d178f204285bb652e6e27e4b9fe41f5fbbc9b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a5f75a9b00f84265141680ea3c48d9be3e4c1165e94b50c66ac3a5dfb5dd6d8f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "45ee41c52167101a59fa6f3e94df839376241fe99dc90244a00e46e3ceef339d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c07897536a58281176c385dbce72b6402b493d16646499bb1891c6e1f506d8f1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "042e618854b28c53189919c0d7f2010aec15aec078db07f9bc2da0f141de350b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f76f5d1a8540f9351998f3b28b9064ebef59c4772cb8ce4c5d2a29b7174c8d76"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "eb6ca14210f526d72e0c77df31dea0c2c52dda08319e1d1cd489a90678787ce2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0122d66fbf796f1698f4a9ffd02b420236eb4e2c32bb87221a9d0843546871bf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b7715c9c5a2d3bd62024fd7b9a21f81794544d7a3540f3d65bee616182b7c27c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1eef08ce24d80a8d156d05caa7ab10c398059b98da18ae4a08b858ac355ac96b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "00ae6cb01f2d19fc6b8939b7e6e7bbb9fcdbd7ae69c8f85f21484c6cf6d12ff1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6abe6cf9df98a5610b58ef974ee4b5e24acf87a8dc14781fe01c90e146ab0a50"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "fdf86ac918d36514fa27cdb8e0cc101c85b334c0b9af33077a704ef8df2c883e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "897316fa4839a8124091b3b1a616ca5285cbe5b052684fb809bf029f58cb564b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c32737c48afbb541c77bb66c36333491f4246ba01ddf4c3688b81b85d2a03344"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8c3c3547df75f9169ccdb6c85da93f5882980fb17667443cee7b56078c2677ee"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "20339def554170dc7e15baf9386388247c8833f58b50121d1dd10ebbb1eececc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "01f672df7066600450c2cb29ea5430b311c02dda514c6ed0feaf57523827a9d4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "fec09a6a55a177c4ec9f1e9f9f033c962425f3beb8d150a44231a1cbc639a497"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c6960b764db0ebda2fb1678712523938f57ac3c7fe1ba8aa58f6c2fc3da0e21f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ff8a606fcb4f09f1c8d92119f225ba0f2dfcd0750ca31da6301b10dabb1b14f3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ec2f24087d122f2463be858d0cc05e393fe968d16fab8c55079eed8a5235e4c1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b06636733cf4bd5a4f46be039e9f52a11a8c96d7b6610ff5220d2b1582f10a80"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "73a053981fc3848a6b95a42c747afb4c5d74c6069b053cce6a96f0472c5512ae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b8c3f7310ec3d69bea2db3beaf2546e8c84582a2e3e20840bd8feaeb344575b4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e35e8a24b6b614954939e3ba348fd7b4c5ff7d58bd39a01edf31096901696a09"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f355a9913c61ae22caa46d9ef056f7acefb4d8ea309e961861e4af3acc307ccb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0e83611822684004582b0fd45b3acefb4120c88ae4fbf5d30ab7ebefdb563d57"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "698b84db50f96b3a1d516cc72f7d516b519c3899e4e0384a266883f8a9254c4a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "040d1a4693eb2976c10662681ae4c39d2733b25ef2bf86df026f09574dbc9eb6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "32fb9731b367100a026eebff407b547248a22a4ed8ad471a9e9ff2161e75df35"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2583d60a61b98fd03c60a19c306b420dfba297df5fd24acbc61103522456a9f0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "fdb7ee71776c077cc4cc891e38c93d76f838af5106f49fd915f4bed5b76023af"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "45d293d0ff294c07df45da99a978869eaa881606fe6e14d3abc050d1011f2f63"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "efff4ad0bce481f27fd33d1fbee1ed06115e5eb87330543d422e52a7b68281d6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e6a4a24ccce0ff68b550bbe255dc304a6bc1a59d0c4809978f107f8d8ede9d23"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "51daf54e6b1636871473d4f8f0e9919aa4f4fe940ada12300b9f981d81cd03a3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "613bd5cbfd8d23349f265cd7903379cf390fee41ca491d834cf883fca852266f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "1cfed09e283155d21414bd034ffed4e6807f54a026dd58bb98479dfff3b1ed34"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "d656654d831c1f74deb803eb8874c89f2adf4ef86d874b3fb1a93423240521bb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "0bb36f048b0417a1535c4ed3d26d4f170c639f1fbbf0e45251187dd902522323"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "4019b61ba5410942135c344a7db24ced1d1b852155e8f322b9726df91e44e956"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "27fbe6d7a0149a264418540534d85bcc4f0d0163a5fadf0d1a1491909ca07d10"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "296c245315abc819913bcb3195adff9abc47ed374b415641bcca2dea4b2f534d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "5e145b3c9e373d7e60e9af56e8cfe041eebebca360ffee69a47c5a8d6406eef6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "418c1073cdc43c9a8de68e29b15884e656897b7dd173d65c33fdac5173a6863e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "476199c931d381c1e774f994432d08745f0bf20cfe76f5751833e8c91519fe3a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "db7caedb670081e6420191be98647a6ed43b9858bee7dd79d8f33847cebbc367"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "35443a008777d704a2d3663c0d356e81590bcec7d775ee95f54c1e962a321659"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "4e87c2d6a0b0bf537cde94fe67cde527cbbad88aa43bc19d1f843eb4e629a809"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3fb1b2d488495e435f28a741f29039cf9fb241dbda8ff683942eee0bee79c0b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b0b0c102fb8c02ea377720e925540a44981db9823cd2ce4515bd65b99563e435"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2cae184314176d182cc5a20f4149afa2e103cc292a0ce1b6f4d431275e5f4676"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a6b54c9dcd330eca3776467a2db804585adc0099a44bfa54c3cd0fc2fef9611c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e4e3940c15d6ed95e4dfe0ad5a03ef1d685681d57085fed23f42d9eb2225147f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "64106c5e59ab0be29c2109d18acf6ff10146e1fd7ce9749e3ce4f2498cb2d36b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "94fcda395bd8ae0ce9a28a3e9d4ecc55291161c99eca5568f5877f9735c1574d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6bf05ed4a6c8e491922ec5c8e8bde46882b0f0c673a0ffb50165f9243ce8d6dc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5dc57060b1dfed715e5a3a355aa6e074488dadfd5f0582e9d838b1f98e9b0853"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "463ab6fb7db3e435fb7467d4905e6fbba4a3bfc80bc28a9d6151e828779c9895"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ab4e917685785b83fb192e83f6e795f80fa3b88761e18a74a2311ae04becf28d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d7e8c374b7f184c6adcca44fb4b60d74d854d0a0d848447d0af2efe3d97cc052"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "032df4887ca639d92cf111a4bc1305539b35da061bda310f1b14541e45695acb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0da5451c3460d7191d9dd0ec17fdc16a7721385c7d28636645214f99e29f6d4d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4a66be6f9f2b440e913d9a8c9f4968a952fa6acd92ce0036e1e04adc1e4b5366"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dcd65b6a11b9442600c2e9584beab33a0b6d43861d160dc1d6d73e4f891069fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b30248a49d02c6aad34705bcada216ac2265e728d1c51f547977b5b3377b3da3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "779920182ba546f26da1a295fed1aa3aab49a67b6f2f6f27ee21da724314adc3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "2eca133549cdcbd504ec819f4195f41821156a13fe8c4abb9a4ca645c0e25722"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9905b3d8bd381fe96fa20d262dd83fb911e9d77627d01d5aec622d214a46ce27"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8564b88f656f8d5c2c7580555f22d4132bb47906fca283ff356ff395a3e8c839"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3817962e80deb1570e60e375d4686e5de88411b7e83667d7fcf35ec7c91d9a2b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1fe065edbc871a7184fa6142e2ae891562c909591dc3720418ce844e87510cd3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d566cd7347053127a4c593c3e8318b46d6cd270dc78e4ae9b53a81c219b0ab40"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "bc67b17b1e0127c2f1490d3e3f5c4fbdffad76a7f06b7ce869b0ade138aaa08a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "fedf469a3e05505a976ae4adf8ffcdf2c8e333b20cc68f8f5e72f6800ce91079"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "08de67859738a0a0c3b784835d46240a6919212af173d40d4d3970dfbbeca6b4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8b6ba01537944f8a154178f06956978e3ee7cacbe7e6091822dc8009b0f3fd8f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ebb5b6992e6383ad9ad3f758e708bb0d0c40a2feb83eb7ac036e8158d8122244"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "54df2a941947f192f48ae4d002bac23be460cea0171e4277d62540614f086730"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "03a3285c63922b08a3826363ec6e4016e318c1501be6e9865a794f4672a41731"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "064bb15917bfc245144665e30d24e07534cff895d191b495fae6cbaaf854a397"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "84ae877a60400356919869627ca5fe4f6c9c74d41a5205fc7bd6b3fd02a18343"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "839296f1f214cc310c0fd8e0de940bcbf4f0caed26cc139cc967d94fa0ebd582"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8662a32781cce1fe48368ea67df720e019c1e54b66e4db11e2eab790c0960034"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "897bce207312001b400723fd55e92aab71d34062c2f9bc297c3d8b16abe4d08b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "299d3d941204b0595b28bce1b3a54f5ac3de0ae07852d1a4bb90dd95fdec9c9a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "facf9095999ce1600db8a16d8b4a8a4f2f9ff3f41db9b2d268efa051ea8b69cd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "099de91b221e4b10dc3fa5cd7b6beec7991bf452c6a2c3fef4f860769fa9877f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c181e19d1d98ddd9f53e81b180b03120295851b96c216070a1ad16ff5367ad70"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "fbd60b21ea9a908755692fd5a7a89364360e8eee492ab5233a004610c2d1b61b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1a5cf3af3c37c534562fcf337c84bfb3695db6bcb78bec8873db129a3ac20fd6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8be8943df06da9a8eb44743a574f650b315086bad3558b1dc19debfc1d42c0b8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "505d02281786270234123907fba605e25ca5c7734468d75d322fb43359bf6695"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8650a10943020401a7033251279f8d2fb3f8594dbb0ab3654681651a4e04a7bc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "150c7097aa2acd6455ddf25e9751262ee5bfcc8cda3963f0a9aa5848d6faba05"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a007a33af796111a9e6630c65c7f8adb4ca6c68fecb1626da4bc3df2c4d78f85"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "94fae099297357be51945ea38a74a42151b810ad4873021a6661d8b3577f0bdc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "17ae5f4f0747b7a6c84bdf8b0578808f972d27a5e8fe2e3689e211ef4e873ef1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "277734abb964bddc73e118e1c74f0d95fb30b5dcf35e4ca4138459d6eec23dd8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6b2d770f1c2300ba95911192c249398539c53d08b7d4dce4505e7dc553a14a10"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "71895247e292639a5784dd1d556cc089070c9adbb628f316393ca781b75a1dde"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "597cbbc91ebe2389b1b7976f6af360328a96c0a6d1aa193f4f6773012d05cdf7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9c4311f1bdd313a80e7293a84ca847199f79f432c0f89d6b3f9b07bb5b9d89d0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "213dae5d26f8944fdce00eb019f3909859ce35eb2aef8e789e2747fa7ec0034a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d7da81440033ee2439e69c90b86a8b98be22de31c40f0b2c0f29845424c5c1ae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8956658374903a61b4c19e02dea3680b14a1862bd438f6a9cc3aa4b45adc375b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "04a955039c045493124d7bb410f1784f69fc17e505263e13415fc786211d7c63"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b24410fd73fe0418812a3974d5f383c08b94a25a94b7ad84de39aa0f0cd2d4cc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3aaf3231f0974b89c1781ad9efe4b9aa9ca3b356ce605803db0609a50366e934"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c370d8d73c9c1fbea2a8f3749fca94f44d1a7391d7e7111b68f35eabb18d184c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "577571433746f09c52f979e82d191cf06a1a8ccf5047e060f9dd98fb7926d5f4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1352c0abb6658f843e63ac2db0261a0b19744cb802b44044849410e00c8cce20"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5b18259419b6923c7e74ca0c8578ba5557c56c3a979c63d70e15efac9247048b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "08e1cf1da036c1f20cfd60cdf85a6d6b1d4958962eded92a51013825ef0d02ea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7247bfdd90d6317562c063155376f9d10fb50de3e0aa10dd2d4b8037817b30bb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "61ea970365d9b4cb023949b24ae4306fe1c172f927713de92aead147ee8ce32f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5b6e7c83ca9c4cb6a5d57bf64ed16af5ab6907e4ce409e71fcd3fafea2841e3e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a6106f46cadb75b1a432c00727864454cb725829580f4a990f5dd0fc76587dce"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ea2c3cc2afcc55d9f5ad709b75d7b9b6d74932e047ffd413c8dcc5d9427dbca8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1a7ed0a8ee0c0c973a680d2306781ef09b22036f7e0c104adb6d4ea1a284e97f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "92a51ec8d63208ef5ab34de49e4a40af6e9f6d7944f53110f1a44cc6827bcac5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0b9cc2e27cc3784ef5d3e85bbef4149d95a21f3fab62335a931b4779d99d9019"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f42697dc368d1bb093e4321b3c568baafa1cf6f4257b0db4e92c015c14509ce5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ed78577a04f01d67712b1186d26d281ce03dfa7ba33eb4756d7f54c150867cda"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f7daa650dca52bca82ee0c2a5fd01dc4a3799cd4164e7c5b74496ff121fadc3f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ecbeedc561ec8c8806519c6349cf520618fc07609979be91b3123082237581c6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "44b89c6716ee60e543998d22f433fc520d45e4ab44489f3a8e0ec61b51022e9c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "045c56b5783788ba549a5d4d2ac721b831301c4e78766238267ffee69fc631dd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c0d66d395d16723f0956a98613a01ab08c2cfb2d805e8036a7becad4f74138c8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c3318a3f78154d1bef4a0a4408430c38581d4628914d7a7a3fcddb3f2a9f599a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4f1252cfc5ba9ace29f5bf5e8b0e9e145fd0a575455821f2db4bf9c56dd2d058"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c5e578c6ffd5087cbfd95ac4c0a7906765ab8df56b80968c239a69e101ef6295"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "09913b0ac8bae801a4fec0ab8f45c1613393bc0a633550a2e3e8cad3da58af14"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "358d03ca2e559d3d2f562bbf75951dda61b799fc3152358ddb7c03b25c9174f0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "87c291f1eff42eb6c9e9f039997c799533c413837086ccab5ae8651afa4abf75"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0c7558d213f485fe0fb13da7db6fe47d46e96b8c50b4ecd74721fdf0f55b47bb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "bb28be2e1b1696cafd45806ba07edef943bd0e0ba847365a0b8798b371ac7936"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "25dc1af69e1b51ddf72b348638c11c275766e02c2ef67fb9f0cd5f07c19bb05e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4925875fd2ae852c4ac0a58def61744f9d571bab010e35dec15e9ab3069e7009"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ccd00cb8d78ba6a1b75bf3af1de4e64767729cc4716132c66ee7e8a304bcdb20"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "91b998012d12b4aca12e339f604a699e2190889d94d8bc637189c548d3020441"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "517d8d086392666c5a0b2996c4b96a540b5393f2fc7c8f5a5a442326bfa19c98"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e863d40294be967ff75162096897797af0d04c0e97ba525e190b81e0dac6bb65"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6ebb9c7ecacb5c4f2424abc0ab8de4ba4524e17edffb710c80b1bfaa04a30f60"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "45991e11d5c1b1a73d5adfef2ef6764228930b2b12acca66f8f6976c4b7b0858"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4606a200b1b5c744f53dafbf37abde8f2a864428294a04a9d4b187d75fcc99b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2a34c731f760409df0d0331048aab1f2530be630c306a770bd6e4cfff227b0eb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "64d8a3bbe5d6d5d726708e373b1c88d2e43c58f6cdb0fcea2b829f913fe82ff8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "daae1a4ce4c5332ee13432e26d2b36fe416bcec19a8138b244bbe68ccb894466"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7be3dbdfde04e06fe905b8885a6a45f0eefad3c16a77b513339be5f2a102a82f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b4dca64bac71f56ba75b55b1ff09a718ff137e70e4d67b89b38db2b891ddc8fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "b10910fe3fad4c48e0810d45e6c0b8e8ca170fa877cac1e7d537785f622e36e4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "1c6e7a1cef2839ac08156a7ade01f2a8aefffe7c3ff11bcf84d02e78d7f42232"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "6607fb64332c38ac7c129686ad8f70b86f97547903083a7e3896235c266ef609"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "df28b84d43c221929df68b53a93a2c25a4b4d57a3bba7973d4a13dfb85edcfb1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "7244a7a4c2bb2d2d32424b9f7a2cef72aaf9333b91711ae683604b9b170a9993"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "be5300073aeebf5483a78509a06fb6a7c0a3a3a0233d973546e9d509cfccc6d1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6335eabca31840a94a38790da369f11f3d44c0f26e4e44c93e251aa21f027fba"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "868424b5dad716363c37c64eda51dffa4af4d25ca204aaa3dbfec7ca9ec07810"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "c4233f248d1ee2b373663b318ad6014cebf48fbca24036fb1e84b7fd75b8ecdd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "8997ad53f41b43f1674ca01df5aff7b963a7acdf7db17465856037df6599587a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "916ad6449bd24107f8cdae22369b19c13cf3bed53906ec5f7c36579a78e9ddd6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "fd6d0e07e0b41ee08c4506b2f8f0a2f403741e6aff5ea36d00dfdfe2322ae248"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "c00feed8fb36e8dd4476698161e251a865063b6214d6b0eb9762d3ab9bdd3723"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "931b3e5fd04f62596bf7d32265d987d487d42b0542acb760431f6768d27c0ec0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "12d93b06e78e135de58f0702fda3414f9011c55af5e32098eee84e510a51e947"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b273ce61925ab21b5a5698e48374b87c2e8372e766de9bf060ba755242e6cfe0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ddb319279079171aa97317bd0e9086f1edf62891bb56ec8ba226db02d337ac17"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ab598e7b1b19003c6b1e9228ad917226ff63dbde3886102877d54c2b25eeafd4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f917d81d4876c96447f3042a81b4ce47249be60599eb02eebc2f83067d9df59f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5c860fc257f6ae4e1aeaf05e9b0cdd94e810adf4e419387f8f3b632118a6e3f3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4cf5615b5fc288572625f1f39681e00f399a67e74a62b87f3704d5f38269e344"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f22d01a6520a722040f4c470a52b9bae2855d6fcccc1b4f1c4d7c8b8528ec2d6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "743afebb0f46660ab1853b02f366584acd7167f0b597d8b40d8af76ca0c348eb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7abfef5b9ab55fa74d35b891a6cfa4ae719e2cd09a22c94d80fdf539ff3d0d3d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "65b36af92bfdcd163132ea728fd84fb3cf3bd324d5604e51fb72e7cb0c0c54c3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "31cf19343cb8a51772abe8ecdb248cc9f33a2e5ddae58d7c51896d52070f3173"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e54dd5dd6132e34f6815dbdbab8572a1ab22de3943cdc93d47d290b027bff965"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "24ae470f7b91f1e7e67ecfbb779e944ff84b115ffd5d125aac169051687d894f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "552dc609e01f3376f33c5209fb01bca1e9c456e7ec443817e31eaeabaeb2eef5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "cecd3ef5501719fd72468465c57b0757d415d527e4710475c7b0fc436907dab9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ec9c2882a69cbf746dc2e00398c620ca8c92e61fc021904fce6d0e05d38564ea"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "456336dd534cb9d1408f01bd8c494b18e803ff03fc7d134274e4db341c13d762"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3cc93f532e5e211ff064f7abd564c6bf1e15fbb09bbcb5ea9a996fc641728846"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "22d1e0f91477a1371692990b11be0cba754af50b2e5a1a6f2ddea311fba8d7f1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "403218191c4ccfb118f9b7fe81922f4425a1b8a393545bb81b6a18d812d4f428"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3f288cd3c04571f23573e31644dc53b0a4b237d1666aa6a13b70f5703df3caa6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a94bec18812dd7532827585b45c867c6fd9cde0b37660eb9724ef06e24e982f7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3722ad0fcc77afda4752a72b3d4351ba73ba447b536944c4413b0ee7c71fae84"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "17daed6fd248c555d26814f3659ea0a4e6aa1784428537a85b128164200d59b2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e781064b884d7295a03d790c801a407a2665e88c3193d094dcead13ce67f6c61"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "334f8792c6b0ed63aa05773dd8b9fa0a95a7b670cf80766b7ab72bcb8be0cdd6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8a6113236b700b9298389017107ea9683b35c299e485fb1fdf6c79572cbb5c1e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "fb61b48ac85cfc883c312b1983a6c7d7d738266bb8210b19f6403e8cff522123"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "933f8b812ee886acd0b38932dae016cb06698ce927e8ab582b1d657ff333372e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f5c30868cf492f3b5159329b5fca37a55a259bb0f9fc670f611eeba7ace931c2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "471ee7d64ceabe613559cefadc49ed0610913a573a27489bdab869e2def38495"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d00fe44d725328426ba74e3b93b5e51b0a710aefaa0ccf14c7182fd82cd5b968"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3de8b1ff016b5193d6f03362701db54c43466de332f98e93f9eea48d5e0e7762"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9abf40c245f5e7715d641e1307214c473d3cc2360c1e73d2235a7b203c930d61"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "dc3a26a29c40ff04b41b015dee4d43eb37d92a8dcf19cf31c01cc5232425abfc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f14c610beaa57617535e8d3298c0b98ef178e2459ee3221fa88a3a5dc561766f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ee7c9a0db0d6cb4eb3da7ee170d8660d304b2dbfe64848830e30349a1c633053"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "18eecad9e630fefa32019ecf5636b91d5ba6c950132e38d778d5b94c648831c5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b589baa20c2fe35b642e0e81355867e83701daf7cd3ddc09992556ecc6095a22"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4bcf4ae0a5e7d00d4cdfa72321c8472e5a942ac717b20d2f8e73d110311c86d1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "edbb0ee95b4bf4bb8827646a4cd44d9d7650d69b95af73da8f7011e677faa940"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "183c629d5fb1f4f799ba9952c82fd138a40a36ded1d9a33b6d377ae5586a1fdc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "35dc67d9f28896ec44f4ae6c92e46d5440aa92f4016d9d45493f5bd89ea8e985"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b78b9263879c55af1ded843dded2cc81db7c752965ca4671a1f78495ce57e94b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "52beea980deb377de627e09eec3442827fe1aa762372f7d0827332c5e3dff0f9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "33995dc78ea85219fc17371fd55cabe57990e29432611e9bdce46afc71647667"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9ded1086e07bcd9b1169d4950a0308b4c5a1f6682ff7538d6a973e7efef8cdf8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e888df8307cb74ab7695f7b707f808a724802ce0a28395b910761d4d12647e18"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ab6d863f8b42640c662104a3a29e119a248b6bc99179f281a883f0279a47673b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "22c2c53ef66bf73a21dd3023a1e32090109986963643c1caa96a772efc82a935"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e3499f1eed8eeeec270efca123efc98d27cdba7311838b26f0be15338d49de57"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7333a393a93f730c1d2614dd403a2178b63fe0f7f7125d5d334d4b9e48c44f33"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9878b20d85bed4fa9abf63a7d8959e6852e8244eb879a5779efa418baf2ec755"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4c750c8d29ad63d95faf62799bb1e9b5a2ac197f603ad994533939760b741895"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "24ee43434af7587b87ebe023a11b90e3b42b9bea252e51625a8c73e808c1692a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "937f82ecf64087c04fb502f41a947d056bd35872e6f4d4436be0ec2b8353c856"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "05e31abb2a2ecc5fdfe72427968fba8a4633ebcd0e98072d477daf68e0ed95c6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "b6ef6291f1d2c220a9da4869a4b3e50dfb4c190b6b56d688c3dc9dd242592383"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "6320c91a0123b9e87616148fc46fd5e91b4221785f98a632b496f9b2eb242ebb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "273dd52739b15cb26e49554a459e05bfb10e2ca36a8e49eb573cf2eaeb11cc3e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "b481dd2e9cbb4b3cc04aede3a04b75fb03d9b58c20a7e9c2f6819da61fb8e491"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "a5b6a372d9fc110b9ab73281bba8c0dad2f81cfbb7e6725d12e8a5175b9b3573"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "4fec6b576691bd426399f643d17149e9c5cca5e07d2fa1e953c2388fcfd47f3e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "6e897792c6f977475c9a04bb6e4ae7e26ba4c469e5c94996e529261e04bfe468"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "f7062252c97140a3827492dfce26c92b83556b9c1940214933abc340200340fc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "eab98593869052fa463198e02860751a2125fcf9f3e749c700a31b77cdaa3abc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "ab467dfbc11b06c8e43f672944fc0d33c8f7252de33ad40491495f926e45417f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OE4m3H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "2671ba4375be07fcab4de89c626b99e171c6a85c8394697c723cf484025e11a2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "12b8ca48c0a01b07097fe8a958c382e4dc7d3f24221876f294f86dc53695e54a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3b32304b2cb83925262943793fdebc6443851c0f7a242aeb0d846613479c2ff0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "73e9d0b86a729809b4a6f684b9fa9134dc9233826a5fd62154f4d70bf9710543"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3ab643120f3d638686db4f8c0a338806d92c77026dc48733b2a54a3739e796c1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3a73f5d297ee2a7076d88b67f88944c6dc5a2d388973248a8792c35fdd8a15d3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "6384efe183b792223d5557b871b9e6fa65a93c0d823c09175a184123e29da8c3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "5eeb854a5bff6b297b14507c37033842d89926001037a8ddae988480871fc7bd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "0a1a7dd80f02a06580a4a6a1a93c7cad3f4ed345b2975dacfd8046757229615a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "90f01c366aedfc053af97ca5f1121e48dbcd6be261a6534bef97dbdc01b2d287"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "8e6d169f634d714abf17608ba92eef2c0c73898b119ded5b39fd16e08bf60078"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "749dfcd37a7debd2d6f1acf006afd99c8ec0db5a0e68f489fc872975df98e320"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "080c291f08f21edbbdeb5cb610705d8b521b53d8f9a85d4ffb419304d7976b1b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e899b21833d2906b5cb3a87055ab68bcbf9e4e3a8a61ef5920ef24516493f712"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "a2bdde0462d89ac398c7a936ae540ca67c5bf745a07d4d0707a7ab98195fc3b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "37aa7f7f280540ebf3b06471cc109462cc346f011f46d14450812f57646795a6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "bf362428ae5b2e2ef227c1004f9467c8668116200b24731d6b1eec4978839f87"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "e32bfed4226e6e9e8daec25bacafb807d08a39393c7137714ffe8f3fc80f4b83"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "7ab447ac84901b1dc5692db8e473fb1b137a3460ee8caf1c5a35c766bbf130dc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "24cf8a505f8155e9906d5641584cf90e6a264fbf098416a982fd33b5cbed66f0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "eec916fdee6a7173c29ab4c21f43478fed60be50c88af7794c13bf7ebb7d72b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e555865a0a341eaf1d6db8d2f817fb5da3e19bd1324f667ec39bab2fc81809b7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "03526db4f54b416b787319e14966aa2fd2fb34325301c4c615175a182d436a5d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "3c9d5fdcfd233c61797688dd996e64aba1c1666eb3144b0c19bf236acd8d9ae7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7cf897bd2f21a38c66f30558cdd5e7614527cdb0e6812e14ec7d01397f924883"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "cb3235a1744e89df02b85267d0df9c02d2c2862e3cf7273ce74be090ea449637"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9667d0188f60509450ab10fefd2f00a52baf51562aaecd2135f0044a3982362a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7d826e2e19c92de15f84b12d4113ca81da6a2b9e630cfe04215a2e413c97e20e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5b7f5f18a664331df9ef26a662f71338b232f104cae688958a23e67265f4acc7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e1d53daa75d2df35e40344262f1d4f49ee020a1034fbb2b0c6615df099342b68"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "50dd5f5a72aaa91eee7662c63cc4def61ed81686f81a743e2440009f2e6db136"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "841497ebac89606d11ac5b5873876d5e738230b7d7b833046166d3e1b51354a8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "23c4604c23072647e6626246e5be41e7986631245bfb7efa85373a6b02cc29e7"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2d97d30278db0ce7021dc76def01e34123b50105eb555fff5d6ad2fe7999c03f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1f5f6b33e5c85e532965527351a60518f5f44a768d64616d7f4fe2ff230e8c1c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9d92741e4bd356c0d6af57ddb47bd74ef9ff1d643a3d79a9999bc2d14976cde3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3d0477040dc28c904eb9e02dd7c83b742dc19dc0fea98fb970b8af7927533f67"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e727ba25a392a021b86cd494a33f61f2921b287fd1469bf2020baa3170c98d1c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a19704b649bd9c822c291f72583ccabd46b9f0e1d9b401a8540fed1b5fc6f3c3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "90035d3d32cc54a63b2c0c703566cc416b8708e55cd741d4cee36d4290651783"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ea3c0bb89ee9a927eec76b8fdabd3c546c4973911fc6f74f19283eeb7a8fe9a0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f8e44980078d15e6ea7a27eeafff02a1b7712ea78487f48a2c1646458c8fb912"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "01e11930f31ae0a738c95b237de886a26ab6abcd8778b6e8590f321e2e7fde47"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "405814dd62a0a4c643c63199dc201c9061bb750abe0d535ae9fa466512cf19c6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "1831135f8b12410f90698d94aa92728c57135b8d7be6f0d3c5d12fe1708b08e2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c2e37dd93354430eb330d2ec247786c47a46760196a849ab9b2be5981fe99a8c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "378e86eecdabc59f965ff7a3c451750bf68e69f7cbcf0927072bb78856488d50"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8c7e3d827072c9aed647b9920add929b1a553777adfb3e03335e06146ad21908"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0f59de246a7d66a01c20680d1af35fdd75008392dc98c33b26f5839f363476b4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5c8e4bf02ae456121bc600d6923448c9cbc6cabc869b820ffb0eda33c5aeadab"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "de35624b57ab42c0dc0b2980bb3ede6596861fc11b1ca6e849166bacc9eaf7df"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8ec704236f167cd1cba27a09a1bb99a31fe765e5bb2bb506104bbb704578f09a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ac2b26af8d6347820e68d2e055ae2e7671c37c714185191bf0f16b42a89bd82d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9326e2abcf17290cb49bfbb37c78752e831cca05fb09436773eaacb6940d713c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "38cfb87f40adf8ea842f2872b05c87124e0a2e9516198c0333f20d3a030447a5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "679e82ff9e524e47cd7186f067e28f49fb7cc260598b2a6df3ce1b792675eb04"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e4335e3b198c56a08553665e23cc80ce8a7959da3a6c8d0d2e114bcf672beba5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "301a57704772ae051a10b14548fffadc821703f07142fdb290da3d762cbfdc64"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8462a00c84a0fc11fa836ca34068ed196dfbf0d9faa50d5321615fe047da0f10"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "bee385b75f67865ade30d58f7336fe305a5369018f897134998c39d8ee301ca1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9d44f7532cb6672dbe5d341fb8ff2f54e02af78b287134085c102929dfd877d0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "32a919f9ce2e733da4cf5a0f677e61d11ab4df4a93c0af289be1a340ba11a3e9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a1dbce260c4d697e135f4c04f53f117c18c89c8541e485df2cba076ed9b8132e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b522a4b4b6ab5eeeb624e33cb70cce3ee936444bfdfbbff06a36a86d53bbbd1a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4716684a510ebfad2c31eb5718b09dd4189fc8965bc9d02ba24bd60f258a7d17"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2fe27442f9360ee059a6c057045742ca6daeb25ec5cf056ef397fc8bfabe5047"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3470f50c1c8a4024c3c50bf93ade7385e698ab82b159adaa640063f09162b52c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "620b2e76b3e3475f753fae7c19cd6c08002089f5d15d7c5f3614c5ec755dfbdc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "82d87de111fd082c8d89cb1b1300362ca9e04555a5c4ae4cfc1cf32996878a66"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f88de0847d526fa470ab7e307e58ffca134eae9b783797fa7d73e62e80e9a37a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c2b726b1911b222299cc14b6b92f7f36fb251861430c826e78eef908e427b627"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e2e0a939aaf629c7189f5accc44c3bf143d82e69208d6dcd37e8e7b957a8baef"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e462f0f61fea78bfef5def0bcc609b56ad4191e0171e4aa8b04e103218341777"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cff01731ae5fd8951bc72ca6de1148f9cf6386c75081e5de32d94e75c921e6d4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a0d54a41ad95f85ebd5cd33a0fdaef1cf3aaf27dd6f6708fefb7f790f85d0e7a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4e1567f8ff2b83466d3fa8f38cc0710e9981be22fe1125ad927e762587249cb6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9d56aa0af811c9207c2a53e52ecbb142c330a711dd95148a0f094f5da9d28648"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5d22a1805116aa44fc98583b837b8637d60456de7619e4eb5c9f92fc081cfb93"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 84800, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "160cf7dedeaa88b36f722b67e23032e3076b990542ba31f7cd2d9f9e9a833c54"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 84688, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "3707bb8e3a986bc0626bf7ebfbd4c7318886757b7411c9797bcd7ca03226ddfe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 87680, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "9d206672e7d88b0ee05d2b20ed8f84db79cc8b2c5da3888899a1b206045a624c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 87568, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "a265152dec6802fa7ec88b76a719b381012e41fbd64de0c97835c1da22d2a676"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 85376, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "70ae9c9a275af22968a943ad951be3051d38d09a6e747ffb48997023fcd9393b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 85264, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "a426663e92fa9cd71afce73af3ff53cbdd222d0feeb4870dc51ac68cbd624b2f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9de514f1de88080b9aa67437b334ee385db4058da6583bcb1efd6743463ee481"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6ae92b1f5a6a6b9fd57aad8a7ef6a398de9027aedcbb51f20b5d3eafe8f32bb0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 84816, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "2689320015f437531229ac1b57badc2ca2e143ef41a340b8999409ca88f1da77"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 84704, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "3190ebaeffc1531ca731db24ca3d98407f6f1756d9cdf6ebe277a1997f0954fb"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 87696, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "9f6ce993891268fc10238f4ce66ce7e4c893f925b03b33b14e30ede0d8baba45"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 87584, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "bce1d92255ec96df38554ca82855e881fe5f5103bafaa8e7630a52b3471f40db"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 85392, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "a38b3fff91216572fc277575a0e98f0dd07f1498c649a66f8c79e3bff896e4ec"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 85280, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "8cba3fc9a8671845cbab17f9e10b4953034df5be48410139540cdcfcb57a1744"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6f233dbd6c04b5cd3049db429ca2a6a3df450335aecbb16961280587be2a5e08"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "73ca12aa0fecf707ad4a8ffd7385e60d2394bcbab03ca5c11d88da3a572c52fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2d598aa4626573de6114ea4d510bd0827238be1ce331bf5c10deba1c8cd2faab"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "df924669b3c0f4e4e4d2e40ad3417e391367fb62670741a159b4918e029e01fa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1dd9ae795f909f33c3fa756930c6782d561eab2ba575e2ac1bec20b101e6c71a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1d556e750587428e102bd2855d9802ae58a3cb21253f05dd858c282374d9a064"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "05db01ad125474ec35df8038ef9ef4e3c6250ccf01cfb28d95400e0f5c49edc0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1af51f8464d54a38c6c43b85a3aceaebde6780ae09167bb17e3fa8a315aef19e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a91bafa868cce0a05bd80e8854d788cd52b5a9b747b10e2735573362ede63c32"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3870246d17898de523172f6167024cdd0f2d0dff57768685f00266f0ebf3485b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 213520, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e0046ec841501e0fafba0791e0bd0f670c830ec5fb4c022004007657ae877da0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "15971c486497e26e88a077222be6b43a1da5c88c44be20d4b9ba41a1b960ffd2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213536, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1ae93e35606aa16a22c50013cb4551506c3b49ecfa8b912b534f1ff273dd831d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e07cc62f3c445f2353c46ce48d3cc6d1fc0faba7eb805b2e80cae11022e97108"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ca6dda80658c1f16ae20d269a7f4b01a99e9c637034fae6dd5034942f30b8b4d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a4daae7d12dd8dbd26639842621ae1c2c1253bc31e8f8ea33a537a0259dd15f2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "2ff7f98de636d6e69bd206a4063ae2a9ebdd31837380b38b89ecd9ce2c3d091b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "49d38fd1ee384b1122d4a2512e4a8417b0e3f9a7502eca3326187264dcb8c95f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3d57d221e0365f5dc0f195d18bc21bd8342cd7a3c7e6f769570fc53cca72a0d9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "64e873882a3739f57ed36958d53d4a961951b099e791d8f12b58c1569fff588a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "998cd5b9ccdf67612d7c8b3aa0e6024d219d05e782475c43705692da14c572f2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c83f7aa856fea5366cbc774fb537a6212a2644a07f4a34d1a51a434ffa5cb09a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6ff637a77a6e4c87ed3d51269a132abe0c5d630951978db78a3dd252785502de"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6252bf8670ab6ef92b0123f8f7baa093f7c9a9d33148561a14e6ac2cf1859438"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "112c42f7cc33d8b3688ae05bc08b216fe658a142d0cb283605338092b88a2958"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "179e8097acaf188cfbe6be6d68153de5db908e86eb9132f36b3c0ff450b545b9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "abc28e7c7f424259067e271b8336b3b793034743cac3bf5d6666be9897f2eaff"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "52235fe090b399547381756ed0f8765efc5657cd4224d35a1739aa6ec11e742f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9044d6812ab0f2cf58e5b5a9eb179c8728635a35174be4903981ff80c935e29f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "85d8ae0e0cba617107306cabe04e7845d2ad0238900a051a2b93805196732dbc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "22cb1dd6e7b0f20fd7c926e1b0a4254411e99e8ec4470dcb7a65af91c7d4a9ae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c397b2cc5f612e788f147e0fa55e8755a633e470150bbaa392be83da4af0b54d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6c1ae3e49d41b7a004ff25395f6288c448a65032b6e20fadf64bdb5c5f15c5ba"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "afb72570c514224e8f869f9ffcad2b5e10a834194775b6aac49b595d80797625"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "171989e8dfee861ac5f2b207b9c483b9ab6df828a505bb70151eb9f485760a2e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "97a03dcb1fab20a27b3cda32b1dfcb49da7b3df53e3b0df2978b35e13806830b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e03c9890eba723252ea2a993cafb15dfdc4084d929ee9e46326adcf8663e70f5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a768c50cdaded76603becf178c09271ec919735da63bd9a763fe8c2e1554368b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "45a2378a85e4d7471ce84f87e2211bcd3fc5675c6cc6077ded669a93114c3596"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214336, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0737ce7a96a1b5ce324b0609ebe80183fb7b6ab6cb0f857459374d20cab4bcb4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181568, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7b631ce96e074c6f91e2e8f2f241372585fcb3802513aa0fc4d4cc170f21e40c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "94dd4bcad4bdebdc6421450f0dfae4d95499d13e07886698451fba34d32e2fd2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "644e9a141190971591fde2eefdd2c4d684839485a11ee2e6832eca4a3093fb80"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "521f549bebc5e605299c342503b355dd4c422a6db22d5e17edbfd9bfc057adef"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181552, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "728f545c18fc7e344f90acfe29d329193dfde7a63b55d22981786c04621611aa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "de55a08dbe860036cf9ecdc3088dc2ebc7e196adc4f8a30584056ecfb39b940a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d39560f1bcf6e87236fdb4cec0dc97dcd424e55e8c1249c56e36419757b224f4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8668b220f61e93dac920b08096caebd6e31aa89a7097d39198b925bccdbe1d5b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8304a83c3801f56c70be021542582b125787e50429fba7a863bd6b688cb496cf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 181632, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4ac2e468b94d8882a727c4e0f0494a08df292500a9afa1d9249b72fd95582cfd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 181520, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "89f58ee2484963b318c06644cb41b9bc33955bf75351624e711507bb5d304f93"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "50d2f8ba40555e9a7e910693006f61830497ea00107934db4df4dc74484a0f04"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e873bce82b3a281d0e34cb3893f489ac6a58df9173ab51f53fec778feafeebae"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b1b3565ca5abff377bd391acb6959e0cf59a95f91cc6b046a6d0698f4d6b0a58"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e227ed35a6f9c8c223e79a84a0d6424cd43701ed8812b0e0feb3e158af23b652"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 181648, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a9f4e66e5e9f3f3971dbd1bc51a0bafdea5c9e7658fe674431c30cac626dad13"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 181536, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "af909807750f0a361b3f684047c09905485e1ea7262fc2ec861f11e38eb52e51"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "18c8cc614bbdd9884eaa8ee0765afa275a4be1ac44a256cc2b77e862be50c2fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e4a5639adcf5fd59b3ac117649772a8d9985dfa84a0feb18393a72bb6acbcf9e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2610a4409d29b474a8739356f2bbd7e957e03e7bded8b9fb1e4fda8e01bf1587"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2cef6cce70c5721cd58494dfea88c214df0f062946e49b90f19f3849fce28a18"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 213632, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "49f74502a55038781e5f8484d62c1df7608325f3b26147ed8b339d79d352cf09"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 213264, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9dcc444bd329c57af379ae8f1d28eea7ee4c8d609058cd847ba97b3d00ed409b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 213648, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b00746e1238a75839a0720356d60fffe67f4e2c31edb9c5443eda58455d3db96"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 213280, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "811386821aba2580ced08ec4ed6dc48fa7dfa8a229968ab401c60d9b2ddc9970"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "70cf564a38b8383e4914c85eff1bf6abff0bea6d3c3714832e1c0c0ee71adda8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6874f58e51418ea2fdfffebd3035a1bdc2274309c12aec29d1d59a6ce0d44aaf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "aef57ebf2dc787de3cd478c8deb440edca26b45be9127836701d5fce104fd75a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "2d9e601a0b27042663d8687da9a177a5932ed6e0e674ec90c20f17c98d229e3a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a719e531bff6fc2b5832ff3932a7dc104a2f270e10180905969fb75c7f4b49e5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3b30df31bbc59f1b4582f79fe3dc01a4e74e96a8b5d9714a74f530b220919bef"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0609caff457553c90d859d4d49468cd75e304d809a54050d228bd945f9436b6e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e17abb3b3ae75bb45a3d01a7782b30f03b04ecce3de4ef1c6bf61496a9d828bd"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9fcdc79c15f5d253622c4394b6ad421723d0a1826a7c647a4b99ef3152e3d6ac"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "13d9c0afcd700c8dcd6368f040a95c19b3bd5e17fd1eadb8526140d2290184d8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "18e4ed5e5550e38fd07a1f2cffd8ba6c0155661c61dace8447aee71430a31079"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f0a8222d8ed51cf7d4f4383c321462dfbbcd66dbecc732c8758089d3d5ad5dd1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e5de0d8fcceef9a40e01a966f4c1c260d55665129e90d7c17d5f4ba7e28ed2bf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "fecb58bdda04b3eda5cae6d0eca175aa39621a038fe156a026626576c6696208"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4ba0ce9e52eb1462c691937645b53378f2293b7ab973f157e82606227f55f3ed"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a25fe9989a805cd153ac69508ac2a3166f321f5c4e9d4f753cd4d52d3261c2d8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 214400, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7cb8483d9e315541d64d9dc31523fae4f1eded72baf63713d9672362febae392"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 214288, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "3721415ba10974002e342afe5740459e9495ca071e2526700037b4ea3a3e1a0a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 214416, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e070e5a0e587eb3e549d1df591470112075d4862344086aa9edbb4c75242ba10"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 214304, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "08e6a90b0472ecd1eeb3a45ad4892ea696a7251b99177f1905a340dc55437149"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8ffe4b6e309dccbaf36479482c6947d84f941d1b901bcbdd1352c66b0555279e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f162783d40f116fca3d5c0e1b2a70fcc4a9ae16ba67bb77a69aad1e4b1dc065f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "20ebdedd2788381386a7e8631afd861c75a93751e8b5c9cbb09438aab32b0706"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214320, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "af14074ace541c3c9475459f4f3791252d91c7351bddf6fa871d402838e52000"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e0d858fb1f4ea2d4414add3df9d82270491886bbc684d6a1f66d35dd10f85f0a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f4ba2fe689ea1c179b3e95742a677ec0b00ed65c8ef5334bb011163ecc989b1d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 214400, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1dba5c22074afa80ccc2c610b6968333ab1b7c2af55b12d50d981dd341d307a4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 214288, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b031646de4355ef5e11217899f0cbcc9f300a21bda7349b2aaab51120382eb39"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9237dd17e295a6765246eab96ca9c9a87dfa09ceae3b102ce6d5634f4b5396c3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4ca0905bb07b31dd4c36a5498a8e4c1649b7999a0761fbca6149f0471eccdc71"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 214416, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f22acb31704f103c3bb8d0137edf026071fb383fe9297628fbd621438cc7289a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 214304, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "42d49beb7ae5e1472af2aafd7de9df40682870e1a3d839f1d451372aa20b4995"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b7f64c3c5686b6f0dbaebb13bebb78d8554705d4821f318c19eeed182fde6e10"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a4d02aa60e2f48e69d1deea2e58b5d33d655ccc1e055d0e2eb6307ffd2857b82"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "471ca1b1bdd8da80870860662fb02217284c679537175ee6365c88afa8ec92d9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4cb68dfd962f1f5ec1478a5352e08d3e503876b5451553597fac6360841f60ad"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9041d2012807717d329d0259e52fcd4a571bf252e012e4b5fe2ec09f0eafaaf4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "b8b054cd7b449c0d2ddf0847535982c37e08d5938fcb790f0d31812c04ff778e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "b409ebbbe63363aa6d6c7fb6e249368538a33b517bbbe6ad347a2ae5b02d7240"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "a07111270ea89263789c4f5ca2de62c751b9678ac24ec4ab1a751c92d034131f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "79e038ab86c0a8e77570025f613d6b6d086f229515b486858a5ff5bb4fa9f895"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "95c358f9b2c8369766c871e2b0757d974babdf986041032e7f50ee18f0b8791f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "0d05a7f34ae9f8e7f20c6c3735ff5110176623b30f94db5ad7f62b9b164285f8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4d501b3831367398b29e24bd373b9d7d13b26423b38046ba70b2b75ce3c39101"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "775f79898691b3d391d2718e0083f862783dd025fec42180745cbc7f36dd286c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 1, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "e70d1776b5cbb71520479bc3698dc479355030196ba038f21d1e424a4e27299b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 1, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "e94fa984c1dba8c4820e522ead24df94fcb63c34142e21876a300295f004237c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "614872027af0b1a013fa4b73c019fa0e31a8c1402eabf218d34ad3f4b8c6f226"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "69251c0e8d9c3e057edb7b49686e780aedd00d2c8175290b1458bb589d99c962"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 1, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "b9f2bd48e283fd01e57c33b7198a769577b07492fec458cd2393a632c76cd6ba"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 1, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "c3b06fee041d2045d66e848ff43b691b3f6d77e1b6fd352eed4c312ae0eac25f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c1bfff0ba6fb41aa310b1e755771067b916f8f1612f56d2f5e53ddabc18c0d3e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 41600, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "eb82507ac4d86c0dcd0374ab803bd680b514571a9a18404d8c6d444ad094f2a4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 41232, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "57e15b775c3bb1e4a6a106f95fbc960a403bae8baf722076b4bc698493da99f6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 41616, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ba2735261fbfeaf2ae8cdcfd815c1eb5961786022e4b220d7376276fec662eb0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 41248, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ac31dd2edfe0f96c8660e96f245cdfc0f9cbec604e832cdf90323176c7902b9d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "389882fcad40258d2cc733947d2af7416ae5025b86459230534977d40d843b7e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "67a662169b25da3bb2f804eb8f3aba8f4938295cd08bb92c17b5462e64485d89"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d4bcf852acb42b81b360a6b7899e3958f3d8380d8213ae5cde759f1dfc8091aa"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "81863830109cf2bdc47617be9b6fa8ca264fbb8b00f3381bda99b02e22bb93a6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "84d064239eda256c8c941a8c50d042c4bee18e4c35c154b01a6b831291bc5b5a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "beff1923b523b1189a08d065b83aeef08619e388fcd0bb36ef04045ba8393709"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e821ab4f4734359b4d7a03b5d9f1fc58387eee3606ec3bf01dfe6abf3ee4fc39"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "3986cf21d9a66836cee41b6d0757a85ee6706d0040912d04676faddf025fe242"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8c0875b4a133268eca6915ed04a45ed9b9ed98bcb574a35ba3a4480e007ee95f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "74aaaa7de9526f30cbb5eab92c697d6926d7327cf8a30dd0c3b1f4dfaee63f17"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "18d1c18df5243da7e7fce97797b2d42dd9dd70e0b05a6359e9d38416548b93a9"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c93db845aaa95e4228a78da83a75fa0d67e4fab8d8987f2ef6fab2a3170a1bba"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5cb0855b549bf85acaf184cefbda66716291b7e18d7cabf164f079d7c4143775"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7efe1711fede7e44dd0a731f0674b66032fdc1c7ab8d104eeeee8fa928eaeeca"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c87f6447d9a19a43e4a463f4b1a02bf5d336a347d8f5eac75846c6eb48cbe6dc"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "61dc57fece8ba688de985576196ed1fb047c718a5d3268ddfd565de44c7a17d8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7a65dae5d643b09c5f403ef6f9add2be0ea2399446541b73c8c0c4b30a8d26be"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6f61f6cda737b1d8fa49b2c5c9b2bfc006fcb09efadab5cd01cbda65e49dc61a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "cb2453d57c304d831bee3c8b5cef9a044df91a4ec388655ddea9425f2d998c69"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d6f363eed9b6376ec0ff6840b5e8c2658b57aaa45ad93c01353f249c832e3b2c"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "2dee68132cd37d72800edcb51985371737754e45df2b31eadf2ea52ba10efba8"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f9eb4f39bf5a70bf1ee6075f75fc8a5aa9f41c4f1292bcaf8b1f5294ef4f6b2f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "aca23af87610e22d0e680b53da25bbf6bbb94ead785c48e9f96343f285bc62f4"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8ed173e55d628118034876bd42d6d10e5ae6ac15b715ba34cee3bfc287b359e6"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f6427011ab3e819aca50531582d0598046203217c7630b109747e73b2f2b9166"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "fb5e0ef8c94f18c62575c6317d8df7228a39b6af88466c642e0ebcfd4e684ecf"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ebf1df90e29ed30baf2d39db34bb44e0dd0a3ffd834c67ff77d83686b070b41d"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "25b5517abc6c825c5d660ee3adc1a51934a7d25f869b7c68977e7d852347053e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "133470c5689e0094a2e6b59140ff7849b5742c98fa84d0620933b38e3ecfd68f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ef527373437e472f03c19ab8dab7618f972bf749653dcfd31dda86c2562df54b"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "7aed22d58fc0213fabf26db7a1bcec082f9bbf154b5bcf4bb35425c0e5655ad2"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c4d78733d1fd28086e0a033f88f7b7d652465d414eb04d4e422814e34f861fd3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "68421355b2b599b6859f265cfbfccf2b739e3c85bef6c31afb0cd91f807e9919"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "beaf23887cb932cdfb7a87304f9793e391a2b973e38854c48bfc738f42ade596"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157248, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "71d140835d0f36e2d02656837a249ff7f1f7df6acf9a30a2b1bd6dc5f5cddae3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153152, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6ef1fa5ca435a93e937a98ed5378c7a43534f6837a11c98f07755be84a76d247"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9f5c17f19ad227bef61bfc9f0d6e9bffeec10ea7c274c0dcb5e829f393819f45"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c31c48120d352d5cb4fe8cfdafa985c16fb84662f06836281d6f0f058e18813f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157232, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6532b18c2ecd5b8a96e4e1a9369dcb265700e688d5cfbe8e5553e7ca16dadd54"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153136, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "54dae9efa204943cf2ac63ee0324da457a42333d8af2a48cd42a921855b46f2a"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 42368, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9e767a96d1442adaebd54160c0a812ce12fb5b43627721d9007d4f370e45cade"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 165504, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d7d1318fd8e86265b1378ceacb53ffb0c93dcd55c45a54e17dffa697e21a347f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 42256, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "94502e30fe227dae26f515bf722e192b0b284bba61df24554a4ba1d887c88802"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 157200, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "fe247a7a24d530fc6f2faeac0e3876cf94cda83a09bb3f4c2eca780847274e3e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 157312, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "3326198835026679cdca6a85840aad278fba2df54a1aea3f5eca37929d234774"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 153104, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "fec615e152e13b2597ae5bc915532639e46c76551bcff28b13cda6639a50aa36"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 42384, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "109f114d95b74f7b8ee68169507b0dc55a49ea9f8c38d307e2d955c4a2bb44c3"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 165520, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "86960233b2a6895025575ccffafb5001849b528e7569fb06bf4a5494c1e352ac"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 42272, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "599b0cdc290c2e4446366e28933e1e7103eae9e50480de3cbe384d49347735fe"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 157216, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "691744bc5b2daebc50076ec6f6a7242288e2b20f997b5ca0ee7cdd3188fbcf46"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 157328, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9302c28a750d23b83104d925b4775fe6c2f4fbf98727f9abc42e4576c23b55d5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 153120, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "464cb678b4a842a518ad67e617c0edae3fbba517e7a0245f555cac0efb20f201"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1PersistentContext", 43328, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "4291f1aa90fc07040d0aed4a4374aeb4d33c7a8017060a3f02bb638c4d633725"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK16SageV1StaticContext", 43216, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, false, false, false, "4f29f1738781c23dd36a3e5a73370be28d9c0ac2372bc003d982c9079bf8a419"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1PersistentContext", 46208, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "dbc73bdef82298e3b14eef7118a3661571a133b3ced4cb9d8a641b2c05ef0e7e"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK1SageV1StaticContext", 46096, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, false, false, false, "addb5eb583c1d1d4d0b1e5cb850c5640f22eb1779566efbdf6c895a0a855ac4f"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1PersistentContext", 43904, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "ebbdd2cdcca589a61af4367a5251860d76a0da2d5dab9ab9eea7f62a6a5458db"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqQ128Kv128SageQ1SageK4SageV1StaticContext", 43792, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, false, false, false, "3b13e1d4a17365882b95147635ea6ad912aaf5305d57d40daf64ae96b3eecfc5"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1PersistentContext", 43344, 512, 0, 0, 0, 0, 1, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "f66ba9e19108e311fe17501d039553c4ddbecaee2d21dcd0c701876cdfd2b024"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK16SageV1StaticContext", 43232, 512, 0, 0, 0, 0, 0, 0, 1, 16, 0, 1, false, false, false, false, 0, true, false, false, "a8d13b94f869cba282e1f16c7f7edc26256cf049d1923aeeb3b2e50c48592bf1"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1PersistentContext", 46224, 512, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "c27d144ba2f59fadc6cdf5b0bb15659cad6f8ca9aed479eb4f59eef8b86cd0e0"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK1SageV1StaticContext", 46112, 512, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, false, false, false, false, 0, true, false, false, "a009b5f8d5a65cc463234156154b259aff0472ea88598c29dd11049df0c276da"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1PersistentContext", 43920, 512, 0, 0, 0, 0, 1, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "886c09584083f9ca60cdfd0831028b7b40214a0195ef9afcc2ee5cd1c2c4b951"}, +{ DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_E4M3, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin, FmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext_cubin_len, "fmhaSm100fKernel_QkvE4m3OFp16H64SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128SageQ1SageK4SageV1StaticContext", 43808, 512, 0, 0, 0, 0, 0, 0, 1, 4, 0, 1, false, false, false, false, 0, true, false, false, "f896fe48868fbbb684b428f0fc06973d1332ee5361b12f6d4f2fe868cf8d1672"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "0b285cc1b9c0b424fa1476a930cc66ef4b6493c29528e426f61cf578a4f27cd9"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a7f68e612712580c3752bf054e66e4c22ee8be35b38f07dce6da27cd453f8159"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "d56ceba80c260bc442dc21ad41f937938b427d7c1ea41a3b13e06619e1b04e27"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "18d0d48641ff44850c1bc6cfb1a3ee2965f9cb8a2774ab5c0dc1b3e5e9450c39"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9a27d2fc6f8979381d7bc973db71646a5c5c06db5e099b932b12e2feb23c970d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "688e4e857f89267b5b6ac83aa9a61ad3fe23ae52264b99d8033d0ce615c3693d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "db0b1527c63af5ec0d00a92911706fd3cf05df70f8fd1843c5e8f19802f75ac0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0db3c30ca18b916d87944754bcd78445043781fafa55b53b3a46de8934bf8516"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4f62ca5be765e9e811b9ee906d00eb4384ad683299067c62dbb38d11b0cd2b58"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1ab3598e0ac68ba2d81f8956142b02ac5c593f8dd74ba5ec0933181e8d439c57"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "61012c2bbc22eda75ced1b934798a59d509cb62933d9a876423e0fa05ebfb0aa"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "31d02e2398acb15263e9294c440c4d81b287fb94655a61e158d633e85f6d7fd1"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b648b07da3f9a9b6680a3f90b2bd958e01fffe5930252ea0bad9474aff9dc1e7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e98cfc9ecbe18f2a228da8740bad204c50439d947c1059e0639c9f424381e45b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "758d1efaf3b078d5d4aba384295994606883f3021cc9009a89d571593ffe9b25"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148800, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "294fcdcd310c2a2982a2eb97b3ddce4ea695f68d8ff703c83d2b234ca7834415"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4798fc0ad3d07d4b309a66cffb49fa8d5e847c4aaa504eb39ddffc9a5ec5cb27"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5ebf1c793cdefbdfac7606892eef97a8427873b6e1fc6808f58f255f50208f1d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f0bdd1c9c69b93242e6819ccf16b57a7c5ffcd7e22c9b9ee7719f89d34f26966"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9e00dd334d3a2cb310048d60b354ae19608702b381e37b2f1baba2e54b443eec"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "953260d3279578e471d17b4f1b4a2056893d47d14c0160e8b09aaba2755d1057"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "cc4d42a979bfe6077cc38c679197a7f0a2b1b02448a90623c708edac21e6ef11"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d133e3877509824117342420264e2185bd1c0f71c2e91a0e93c77cfffe8acaf7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a4cef903b912a64e77cc2d50a026ae6286ef11a1184316574f43c191837284eb"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "6a45d94a31c93a4e1af783a3bd4a84b4fe20e21d9c3d3a88d943f10ed4e92da0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 148752, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d64539339a9168bdd4ec45f7143bd11e5af3f07323e2a042b4244a0f52a3f6cf"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a9a89dbd336598924c72d8373833e855544c174f7d6474f9d814eb67486b42c1"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "69736e0f46a59c5ec6e5ef842c3c63b9609c7b6f916e576ebbc96a86a6773b76"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6cc8d7f165ff3340abfff7ccdf3bd518379a7cc3098a2a4536785bbcd897d2c0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0ad808d55d1ddbf2fd424b39861bb0ef0d496911f66367870f5f9fe1b7cbef9d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "08b56030226d921c571e7a38abd7d8b20814e21952dd483f30506140b9cb6901"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "257e399b07667cb8dc342c59db8002fd1dc3aed809a6f25325a97352c1bec3c2"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c01419ae16c19c02e2d844ec8f239f56d37cca338a4469df544dccf480d57b2c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8fb0f1ed8294089f96f50d25214e446011e87ef657a241f45c989bbcd65fdc56"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ab3b785b40ecaa53c195aa4f76fd912dd19090590e174f75aea871e40bf9149b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9d9a82606f80423f670e47be44c427b2e91b11907ea21b0be84f5c9b086611e7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4df1addce24b4c430834e6aac66bd79db1e35fc1a2710f0e1bf93820c89e0ab4"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0023508d6f2d6d331333ca8f3fb7014b14b0bcb8a398a0e8bdc446d7d44ebe1a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4427312f91f478dfba98f8ef563767c8f837e25d8975bf886ceb9cc955645846"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d526c33799bad894f451c0ecffecc36e7853ea2013e6d944fd18d03e3acaa8ca"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "d702f43a2764509d4fb4efebf2376b2a9ef3bc4df68bc3188005a126aae5aab7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2e0ba41e97464356e11e749da03a511391e7e404a29b7cb23df9abab73072ad2"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "21beae6dc58a8ddf3b5a684e753b5804d355e4b2d9ea64a4efaf72fd45d794e3"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "54544b0e4c3375ce33bed637bd8f34a14215ec623f2cb5356be7f59a6d0bf74a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "da007c3467b4db17eef586a4617d5bd1a73a7e05d5315c3622942f4f205e3f9e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1d6b84f0704967c6a303226be1bf8fa3178b1bb95449816a047bb14fe55b714d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6cc14afae6dde0ceed5e53374ee3d0b7b40710e592f048856d0e7d1ac1a180d9"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148800, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a50415912aad76986cf068a0d7de536aa5efeaff24984763a221503bd6d6d22d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "51c637fa322feb32ac79b3df3b3f9d0682474c9c3a1e8b1320a2875bf52df01a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "672f8bd58011a73c80a5ea87195b5b115f95be488d7645c15f8fc176165d7f98"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "fbb5fae3a8d079ea4bf7df917b9c9c9872b2e3da563c979be91b5359bb21ffc9"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148784, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "eefd5cd4f1b02c1759c883656ca7752ec0136b0f1edea2db9da7c92e5a364a61"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "db971d1064d2d28df3973c560b3319424e9752e9a40bbaab431227a725ea4a13"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "59ad2aecd3a1cb1c8978328dfcfd809b418b46e2deb7329d3aca802157c503e1"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "e6fea6cc9a5b688d78e453afc0255cfa581ef70ec76589b55dfa1e0d0eb3e429"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "10079a259c18ea340c32ba2cb0a4556cbb6b6e1ba64e119e6b6cbcb0150efe0b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "8f1a8772dcd8d9bbaa72ac2f40d79e33a9a63dbcf72344ea9ad7bf957de2bd7b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 148752, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "e88dc77e50ec4bbfa888b309ef42d7e0f4a1310b5b03e9448f250445c32b3366"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "f678a1cbbfeff8a9e5fc21c402a77bd2756d15b37d4e080a2a400ef9bcb14706"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dc597a8b0c15451e7f2f2132d8c2607d4b6f00a361880d400a49ab7299f09404"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "910b817f23f13663e0bc89993889d24423c98fec8bd21d94683225fa1acf01c1"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "f9d7fede6d099f30c3863f179ac86cf9f21f9d28d366132397a07100ac0012e0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "43dd13eab1b925c78eefded81a67625b247a093483770bb59789b0626dababae"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 148768, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "84304050be0a3c6bf6feb1b4156bdf49b4fdd0514a476767493ea1add82df500"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b2015af75f1f91bfdb219eb296c76207db1301ce2463a56236001da91c5ea109"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1c076ed7b2554fc41489975e8587f85386f4b24c1ae6b1f4388d2461e404d85e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0b32b54c4bfe459f99b9bd3d4dad15fdefa02c8adf6ca0b1fcac25d2c678a5ac"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 128, 128, 128, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H128SeparateQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "556edd2dd7c2785d90a1e8ecbd48b3b28f3c3ef9980ccea182f0ebd0a9b5232f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "b8f845518f984fe958a0ac01208a2f0a9210e88d5725b3c1115b7596185125b3"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4d20e351fbedc038cc23e42c42733a26db89b3c744500a6850b3cbdb47610609"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9314ada2a8a18620106424ea5cd4020c694baa7822b5cc96ce4ce5d16ec540c5"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "9fb7fd2bf2a37143800df6d6f3073fdc18b4fffe810f2dae950f3b241c64d27e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "95432d413bc6fbeb2faa8a75fd5c811fb185b3f1b383692324055b5953208296"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a3c1ca4ba2970a00c9baa38a718163f2f421dcca0d8135f56f1d02543e4c59c8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "27b9f304b48ced528979b91a87608c92f5e344aef43d257392dc815d9b69be92"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c6132c6c36cc556a5ded9ba5d55ed02cf34830d2149937d49c4e5459d1b3181e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7388fa47a58933ad953882d03d042d5a4d09a0162e8c24c8d31780e9027f6b39"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8d6963d3db6cdc0ece85172cdca5808458a813ecb1e927577637f59b01a0ffd9"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0f719b3bda017e7a1d713d7dcdcf8d635a14812af692d0d0cf6d592af1aa561c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "5862a8c56e11086246ba0486dcb99c28e74f3df41ed6404eb321900edb3d0c03"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "502f95db396c5e9db63991055cade18525c7aa5708b70e57d2075c876ff7e95e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f4a59168fce4bc479f55f77d8cc44ddbe10539516bef7f7b66c38824478f7a07"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197952, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "bffbdf1dc5bbbc94da7906827ba3c22561b37f20d523f03986f75d9200ac21fe"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165184, 384, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4ba1a64f2f087f760ca2e92c5125cb694cba4a6f6991be81e89041da536593eb"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "233c212d417b1ac997ca88f61ce9ffb883239c9f2947e1d1f536d4fcf6d436c0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d705b1aa117b6693e87401e91d8fa7bc197c781717b885743ad9dd02c5c04130"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "580c963e87ec15c5eae45b3939cd94b1374791213b90a42e5d2187a1d0360cfb"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "68d379dd7723903140eb785355290c9fe0cdea1e88e14a99165f49911a938c31"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8fb2595b2de47dd8198a7943481118bc34bf6c7a0685f118cb73c68347f3888a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "54c461a6256c85200583919dac9fc9ece1dbc4a12c0f4407f2dcb1ed0c0b4e9f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "acc85f7ad87abab0c1d7a3b2920775e41bf9d32054012e3fdcaad54b22dc5ad8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c12e473f03bdb48c57f28cf5584d0b3cfd42da28b80f74731e493d5ca5539052"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "bcebb99dafc6c140ce5a1844ef7a91324aa7994f24642ed241060b1a3a4ffa48"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 165136, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ec64dea34428395100a56443a7ddc33489fd8890ca9267df992b1855c234d36b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4e25118a686ab1d7434cf225b216f19b465151b0c8011a5d5708c7f5986f0bf5"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "91e54d279b806f8dd8e615e783d3214ab135dfa0a15ebaef8d20d2530fb781c6"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "88b09b48d484f4b07ce7cb898626fc3455ec6fc6f867b4cf7a2a608df463fc8a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "82c11ee6f78157a78dd9513fe41138d6450d2f271b27a5e9929ecf1ed8ad21c9"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e023ef30dc02ea9e3c3dc4a99cc04631a1cf1da4fc99ff3d4a4ba398c311e6a3"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0c2ba36094129f299282f9b742f186c4fa5643e4cadea26836c99427073ccea2"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "855df1a09a2c0a7462bea0954b9bd8c12de3252a806122097bd017a491be0196"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "866c7340a197570838fc88fc3e671ce06fda8c2c9c1942af95c73bac396a7965"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "20e5a575507c0f90aebba1b4a64f7095ad67ad237737363ef1372545b6a68791"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "549f36cc792151a4ac115bdc9526ccc4166fd54f8c5a6fae3e6ea0edff106b5f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c8d54e5e71c988205d1e91b1083e1e21202f0c023ff950021af9d864f0f6b7e3"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d54b81d01a9230a43d3d2d555f6285cc11075884cbbf906d4eae83c57be14faf"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197952, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "60c002d4232b3502aa22069cdb83ec2385688cb51ee3155b638f410948271df8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165184, 384, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4f48782f31e069eccb79d69583f00602586f61687c0fce6294e950079863259c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b19851431cc29622910d7de989e1c5de5c2e5dfa6826a4586bb3a24e12e10067"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "96b853811abf3dfa4ea324646d31a182907e700b4fbae933647fd88cd70d61a8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "882faa2a44f555058f1c1ed113ca8ac6d3a66884b0a0e6a318b8a936f7c21234"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165168, 384, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c8c392648372f9dd6c3032d9d78fe902f33d5b566f35ec3c23064726c56795f9"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "45e2715bdbe153659f1fa2bc537ad71183582994f06f284d06726c7319a22110"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7f362529e83cddf91699a6a99e9cd2549e929795aa266360b0989a9fa76c8640"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 197904, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "135c75359ebaece689af885bc3936ff6fbaff9112d0a26f94a094699564cae2f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "2b88287e3de69a33cfe9de0eb25fd186b4534ff967bf5d8e3e531b41a101f380"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7d4a5050b5170e69f4d2aaba96d109fb6049f8e90e56ff2c1889f07b305d3203"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 165136, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "92febfbe489f78e136a4788519589944ba4c08bc62341d3573272e454714ef73"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c125598bd6ea898f74d8f61785f4ddc1b157189e5c6b5928fb9326e68171f649"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "123470674f5d5e2d7a8f28cd96f16471d534d0347250345ba6ea87534e3aa3a0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197920, 384, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "634ecfcbafb3a05f292365ca111acf427e0d4d7008cfbcf9ae8e5f58354d9855"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b9ac4f1ecba12bbca08a522887ab8f54a08c79fe81690ba68adfa41d70b7b495"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "e73fe526ef684cc3a3515adba95101d2897438b663c39bbc949b4e4e90762ff9"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 256, 256, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 165152, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c06506064c4488f2e28c98b10469b4e7a63aab8e02ffe7c03bf7c71e24bb8466"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "277e01a75dbe2deb24fd8836a7d3fa57bf335a7c30d76e034bd0276c37931903"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c488b8557b8ce0f6e7a578f10487fbdcbf41be8472f9bf762d04ebf0eb086481"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2bdb3a1c70160f17ae47879762151ddcf005f02747e80d8936cce141c210d43f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1084401b86cddf4c635a2a95e3c2890a8da54280afde6a703b276d65527fa0da"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128PersistentContext", 196992, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5bb7050e89d242bff126aad3c1eb861c724b14ac532f788c9a879949bf5b370e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqQ128Kv128StaticContext", 196880, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2a13d781706d1bda22d6164898b602d90ab892d6074a25e77f9d1d90819523b4"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 197008, 384, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "950e13422887e0388518027968d0b25b44ad5c19cf68ea3cacfeb3bd850a0377"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 196896, 384, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2f3e8ecbad7326148888d2f45f9bc37f7305229fb789304e0c0508e90ce90be2"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "43abf94fc4dd48082457d5d5d5acf7f6f59592f79b7bda9b78b448617ca36fea"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5facd2b38eb9885124932c92dbd1613c00a65706c18576ad270e3674b690d02e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197680, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7f25d92b94a4bdefbc0c7387ea4be0064b29e06c06b01e0205f2b0a23977bf44"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 1, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "333a5ffbd8cfd6363c389d89b75bcd495ba34fe52767c925c0d8e48893a6778a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f2a5da83cb155a9e42a2745ceb2678628ffed913b1cd1a61bcd1381160cf0a3d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "96c854c95abca0ac7dacb6c8b14eb3b04ad8665096257aed14411572cfe682f5"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1b8f8c1c6d54bd29035ed7a683a4d9a1b2edac23a8a9e7d8d37b408ac3770221"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197648, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "86e42fd567f5ec8b7b30968aa3840ede2e7739013f6a784cfaa1c154297eea9d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "be8e1b028655984e2d2869177bee6af9673c9f5d2fa4291e3d5d07974757da38"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0f58dc3260c1eed6cc05359d301fbb45097025b9a6f983dd1f0a77eb3028c469"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "14ccddcd1f45a20ed680bf163ef44b5106f38e668375ca2fa22d77cf183b89d4"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "01fb75ab82e200a20f3f349898215eb803202171bc3cf3ce94b0cdd27c08b33e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "c76c7d16c9fd42280bed0f9acca3eacffc596d43d162446e96d572342974d7fd"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8ba4c70d311a5d579ea2233b4e505c87405588f61377dc39790aed7b0225b1a8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "73f4374bda8efdfbc0dcce3f64596269b15fd7640a8be2e0cc1a8283e0455267"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "ef743808e9b02e49b36a0171c34e7dcab359636860a1d3944aacddcb2edd9b5d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 198016, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "761311b4b9969138d9be01b2938fa18ebbd46488e62c7491723122c8961ce071"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqQ128Kv128StaticContext", 197648, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "23611a526307f8e395a7d9abb94b55d738bb1accd8191321cd17acab82322649"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 198032, 384, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1771d518a34d1acab51b1dfff8b6fdb071f34ddca2f1896c80bba1c40fcc2b79"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 197664, 384, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1a38945abf43d78a75ecba55648d0ddf0f4aef725116df4f82b2156f72e94660"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b74e3a46745d41e2515f683873a155fcac9f1dd80f3daa410d583dea143d7196"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c688eccf9a24e5830108d6807a1b8247ba735cf9a23b882601ce0149b7070af0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197680, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5d53d533df1c6fd961ef468858fa0137a79b36afccd84e0ce7283e260b4a28fb"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32MultiCtasKvGmemSepVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197936, 384, 2, 32, 2, 3, 0, 2, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dd40b749df6df97221cfe01a14615ec0a1f718d3ebe3ad8d985557894913971b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5125480b80f6d59784aa20069e58c758a8a1e001ef018b865d79fdebcb078cce"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 197648, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b642ac5a772c8605c241b50704433bec4e99b3b91973c6aa0f6a715309854dbc"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 198016, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "70b7652c6d68223faf725ffc5fb9e0a681573f41d815a1cede85daee827688b2"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 197904, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b54c1998a6a81258ee34f3f4f0968af16b84da093a72f0f397beebe6e6f8f318"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7465246575561cb23b9ef3a542ee508567f26765c9e183738caa524678522437"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 197664, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "0c6083538ae154586c80790d3c0c837f50c269efae2f37ef965279112c59c43a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 198032, 384, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "a881e62af27e785995918f5d0c9d7222bc843ed257dbc1213d94ef2fed1bf47a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 128, 256, 512, 512, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H512HVPerCta256PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 197920, 384, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "376b28b0799a5407a1fbc8dad4468083f66f6d27d6acb080ddb926579eed997f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "859b4ff753266db76290760d12b87b68e40cf8afbcbc6f0c4b6f98b3f8650eee"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "2ff72adf96c044d19594e58731b26db7bf190eedee87fce4d8e1e6b231a6d377"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "60d4ba5c8bb4b271ce9bf9f841a3896ddaf630e3e7dfd9de5c75e4a93a5ed5e2"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "44a938697533c3958467ea182790ea6d9448abc34f190502873586b1bf532cc2"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "69f3d145e622d8d361da869b78b6f0900d3730440203c2807e9743e2336dd268"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "5f1210b338fe08fedbdb08c32609d884ad9a05d7af45d0fd757d26b281418c91"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8a901493bf0d159199f08b6e34d0ec28b4f45430627e2610966f22efce623974"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cca12391c605a44642d9f07c9ac3faf70f371f44f2aa3b857339cfb269acf1d8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 82560, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4fc0f2b528eba40066acaf2df685d6974c63796a7b209117674c086be106d6dd"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 82192, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "09704dd351e313398cab628cc9000d04ebb3b3e0ef2c0b6129bd56dd91c3f709"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 82576, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "e906288554f3e35743a8050d27a41ae0784e13a8eaf958debbbfad697b653090"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 82208, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "6d3d835804c90155eb37bc6e72f80fedad08286a0a6f6d0d6be87a75764c03d8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "5d5591c796065efee8164b346d799d8ddc7e8b199aae15c8bebde5440d6e3058"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "2eca0548c291e1fdd85fac8e64d30d65b7bc2055f27626220986511a37e9b10f"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "7297fdb1f12e0ee199080fb74fe3d2573c34dd46b600e96a8a98cbd3428a736c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 1, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "c69862327c980980349425f11480ece298379e7a2ba215107325a9a428d22e30"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1b31cb08d3128b31012c8353a10e4c577210483f039a5f7cc57706b268ef04fc"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "a9655ef2fc25f6ccdf85dd1deba3caa5f361468ce1efd5383b3c14decfacde0e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "9b43b4bdb347a225f4036fc0e1384b40def360118275c1c5697ea0c12bc81257"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 1, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dd9bbe3dd85a83c2e8fcdebcb853398cd70b960e963ca337db7e01924f4ce263"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "4e542f52b7177a3f19fc2bfd1451467dc48637bceadebc7fb470a33a750a84e0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "b06a4b8206c083d97cea3cbcd1be9cc66c051e175785f97bb8e5e7b3989e047e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1206d75845602785e85760fa45231da7f9bddbd5689c4e79f400cd024865ff15"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "7e0b1ad6d49bd0f98bbe1ce9148d98c8ae779adae665d33004a4ad4f85bab7a8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c739edcc10d14e7514a838582d0678104f0d8fb5a038937fc0c54c7cf9a542ab"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "f77c36e20215ee2ddd04cdca73cb8b68d657fe4b37fff48721cc010578b0ac80"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ebc98e04747e6513b6fc0aec22789d265525c858f1272de84da3523803e88a5d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "cfb088c226b37f9e367215bac8fd60369bce923cbae0b5f0c744c46a64087413"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1d1658d78262826a4cd7a207794c39a82fc8ce94ca9d19b506585b8b05b923fb"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "8f81e08bd91ab593b22ba47a5643077046221a0dbd8584e3f855dbcd2764dc96"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 1, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "130e223fa2a273f815882ea5f96333694fd944d748d1aa229ef8287ff1b3c705"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 1, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "d0ca646ea0a25dd3fb080f836d6266a9089c7991be768aa23ac2b8419c81ae65"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c17eabc4725690b184a34aed0a32913ca7420bfc0367b150b292598e2a81f984"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 3, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "630990372445505ae67e8d547559f773f1905823843fefb6eeb14bd68587f5fa"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "681093b1c60de471924e386d4f494ea6e0439f24b9eeddc4f2c7873effa6a06e"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 3, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "6118314157265c4b9adfa5a68e6817e5fbe1ec33d5c07be8b260b0fe375d1056"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "ff541a1f5b8ee193ecb6afaa229338b3efb85813608c5209df3021d6c904497b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "1a378806462fa46837186d4981c6a1445a4732f0745002adcd25cad3f041035c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 3, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "bcab86852fba45372838b7b6538ef591611ac69a6966de65c9f9d624f7ac5962"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvCustomP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 3, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "5bf198930ae9ec285073fc32834300b6ddfa60f8ce403e9aabf703363ef28459"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "c838e3fdbf36d8a779409e8208dccaca4861d0ec2dc18bb101b98347398de685"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "8497bb3b3630614b05604ce532d66ad936068a532afcc943336cfd8a5d522451"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "33001782313221d01b341c134789d134c35aa524093c43b4fb95eb96c8f227dc"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "bb10fb09cfab5fa7d5e7276c4cdffb6370e6361d3ee4037600bdc4be84bc67d6"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "4db960789b1e6d0f32eccbd8d0469491287269a04e017e491b1f2ff48cc60d48"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "9f9b7b399bfa363ac25d02caee87b30ea77f3c70852366182d54f3585a20210b"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165184, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "628e53aa23df3ac7aeb5de13aa3c517800c70b68babe0502d23d845e89ca534a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156992, 512, 2, 32, 2, 3, 0, 3, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "125a8935f3b9649bc92ce79e7035faf8d620e590bca0804e1b2aeb0bb2a33d57"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "d068a7972ebaf89018035e87222eb0c21624dafa8d490474643952e421a7be66"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "c24558428440fb0fdb4b3866780c69dbaef0c9debfac09f9dbedb75a1d1d4e16"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165168, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4d366c6e4333b31c5500c8a58d79e3d628226d410e82c032c6229e262c7e2561"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvVarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156976, 512, 2, 32, 2, 3, 0, 1, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "4832ee691fd7c1582e1e2f66dc1baef20e8bbd515133d2453e2e2cd803a49343"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 83328, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "6bdad5cffee335fdec6dd2e87fe1b3cf57ceed5dd5f4eb5b7051ebba63717efa"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentKeepsAbForGen", 181632, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "0fe892848c5f6c768aa0abd99ba3f324fe6b86aba1c85239e3e706ff235190bc"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 83216, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "a0ca4ec81b9daf9874eba5e47afd2899fdd6a0c58d562f2938e27d1cc9b8acea"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticKeepsAbForGen", 165136, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "740255033c35a4943ff5e38f563fe056ac59de228d76ad550b4fcda8edbad5ca"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128PersistentKeepsAbForGen", 165248, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "16aefde2aa1f083bfa162bc4ba1e1f27af57ef66f9b1f075e407b51c9f8520d0"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqQ64Kv128StaticKeepsAbForGen", 156944, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, false, false, false, "04c2ae2aeb8d4a87d69acc0e82c473dbc96fe1ed790cdb65b3c9671640e4e6f7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 83344, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "ec21f93e17642cd0faab604b4dd163c0dee7edf659dd2af06444aa751cab6523"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentKeepsAbForGen", 181648, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "acc8281e40aae4e41e3507a30100d8d3ea3873d891db9e28a9acb7b61d399c9d"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 83232, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "1000166ee0623664a50c99ac15b3ab9941e75cf45cb3fd255b78c2b7211033d6"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 128, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticKeepsAbForGen", 165152, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b038aa92e5076151d45ff2ad54f2fc0de10dc351f09fe9ccf3412237aeba692a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128PersistentKeepsAbForGen", 165264, 512, 2, 32, 2, 3, 1, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "b350a51af2b44015e361252e12896f7e2aae5e61a5d57440ca0be6f7c015e06c"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 64, 128, 64, 256, 64, 64, 64, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin, FmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H64PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ64Kv128StaticKeepsAbForGen", 156960, 512, 2, 32, 2, 3, 0, 0, 0, 0, 0, 0, true, true, false, false, 0, true, false, false, "dc555f51aa6284e35f8058ac08a1c873f45973f41d58302a0d86c8abbb24d355"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1b6ba31c874fb178ca9446d2b3fcc88c5a1daded9b8eeeb35a0fdb7489d5bbbd"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "64d3500a594d08a16e73eae7291088b72bec254da02c60f4d0e65c30e0562915"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b451a711e17d1ed3cf255f8341ff4c08886eaa5854732fa742d5e73856094913"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "0b8323df55f4e484ea26ae05fe26985742f385b6a61bfde9f8ac05f3e5a478e6"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "9daa5e21e0441fecf08144fe7be0073378f096aac447489b7d1bc82cadc8b0c2"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7947bc3a9750713d3b70d5ce74d50a02c257c12aa73580c96d78453ce6079dd7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b727acb5ac8dc63cc2feb0d784d5cc0bed32b8b1c884e1c05888e8ca9b34f963"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvDenseVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "cf84612735ea96dee9f01a116a34b0b56f33f4cb8285a928c7b186bcd41ad27a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128PersistentContext", 164480, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ce4fc4e7eeadb3eaccd8da4fa6605e1183a325933650dbce245ec55c3523d659"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext", 164112, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "77a8e985b6b2b433cb711dda107ac395131512cbcb75fc028b965858bf04d8d1"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128PersistentContext", 164496, 512, 1, 0, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "beb17e3afb41000b2610164c59d34ad7782ac11742fb66ab35776414bc4b6736"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PackedQkvSlidingOrChunkedCausalVarSeqSkipsSoftmaxQ128Kv128StaticContext", 164128, 512, 1, 0, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "fdafe1f3b62c705d72976457d8b0634a93ef90e5df031b69469930e5989029aa"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "1ab058950d8b922b232198dbcb85eff262e7e78c437b1663200b34c666d0d6b8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ffbf4b6dc13977a93a21f2b8432662272a51380447db3a63a70d5522cdd89ad8"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 1, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "8c95fb3c436e74e6d1a12cef0d29e636a011f4634a5ea4e3cf8e983bdc6537f7"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 1, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "4460b1a8d796ce9944762bb9f563b1322a44c8b38ec03d67d04e802f4ecb8346"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "f3d467204fcc18fce6febe0de56b0887839b840eb3c81913461f08b538fb2114"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "7442f310fc6c48b35260223d3ab137ae700fe62db2dec8195a7720906d27800a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 0, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a11dac373e3e8ff041e4e20625d3260017c57cfe5583c2ee3f8a8ea65b5b498a"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvDenseP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 0, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "2816fbe7f18c6d26e6144625f6a4052aea4570e72e5ebc42a91122dcfc89c361"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128PersistentContext", 165248, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "830d0a337939ba4f53862ad4bdff079b540a13fc40676e29eb1d175ca3586080"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqQ128Kv128StaticContext", 165136, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, false, false, false, "ed3371d674936f10e3e45dbba2fcd6a61b2d3dad9113ceaed283c4fb3912a8f3"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128PersistentContext", 165264, 512, 2, 32, 2, 0, 1, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "a65332239fbb9f2485306302cebe4c079bb0bb0109dd40f0c8b0cb0bdb01c954"}, +{ DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, DATA_TYPE_FP16, 128, 128, 256, 128, 80, 80, 80, kSM_100f, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin, FmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext_cubin_len, "fmhaSm100fKernel_QkvFp16OFp16H80PagedKvSlidingOrChunkedCausalP32VarSeqSkipsSoftmaxQ128Kv128StaticContext", 165152, 512, 2, 32, 2, 0, 0, 0, 0, 0, 0, 0, false, false, false, false, 0, true, false, false, "b649ae5affcd8f7c0dcd0e1abd1b46e3da593709315a223e72e2fc0f75803873"}, #endif // EXCLUDE_SM_100F }; // clang-format on diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaKernels.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaKernels.h index 8bcc29e0dbdb..dd4e7b0eff4d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaKernels.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaKernels.h @@ -496,11 +496,7 @@ class TllmGenFmhaKernel tg::CudaRunner::Grid grid{numCtasX, numCtasY, numCtasZ}; // Prepare custom mask for spec-decoding generation kernels if needed. - bool const prepareSpecDecTreeMask = params.mIsSpecDecTree - && (params.mForcePrepareSpecDecTreeMask || params.mLayerIdx == 0 - || (params.mSpecDecodingTargetMaxGenLen > 0 - && params.mMaxSeqLenQ != params.mSpecDecodingTargetMaxGenLen)); - if (prepareSpecDecTreeMask) + if (params.mLayerIdx == 0 && params.mIsSpecDecTree) { int32_t stepQ = options.mTileSizeQ * options.mNumInstsQ; int32_t stepKv = options.mTileSizeKv * options.mNumInstsKv; @@ -561,8 +557,7 @@ class TllmGenFmhaKernel fmhaData.mInputBuffers.qBasePtr, fmhaData.mInputBuffers.kBasePtr, fmhaData.mInputBuffers.vBasePtr, fmhaData.mScales.kSfBasePtr, fmhaData.mScales.vSfBasePtr, fmhaData.mInputBuffers.slidingWindowKvPoolBasePtr, fmhaData.mMetaData.kvPageIdxD, - fmhaData.mScales.outputScaleD, fmhaData.mInputBuffers.dsv4InvRopeCosSinCacheD, - fmhaData.mScales.dsv4OScaleFp32D, fmhaData.mScales.scaleSoftmaxLog2D, fmhaData.mScales.kvSfScaleD, + fmhaData.mScales.outputScaleD, fmhaData.mScales.scaleSoftmaxLog2D, fmhaData.mScales.kvSfScaleD, fmhaData.mScales.oSfScaleD, fmhaData.mInputBuffers.customMaskPtrD, fmhaData.mInputBuffers.customMaskOffsetsPtrD, fmhaData.mMetaData.firstSparseMaskOffsetsKvPtrD, fmhaData.mMetaData.sparseMlaTopKLensPtrD, fmhaData.mScales.sageAttnSfsQPtrD, @@ -584,8 +579,7 @@ class TllmGenFmhaKernel private: inline uint64_t hashID(int qkvLayout, int maskType, int kernelType, int scheduler, int multiCtasKvMode, int headDimPerCtaV, int headDimQk, int headDimV, int tileSizeQ, int tileSizeKv, int numTokensPerPage, - bool reuseSmemKForV, bool uses2CtaMma, int sparseAttention, bool skipsSoftmax, - bool fusesDsv4InvRopeFp8Quant) const + bool reuseSmemKForV, bool uses2CtaMma, int sparseAttention, bool skipsSoftmax) const { TLLM_CHECK_WITH_INFO((headDimPerCtaV >= 32) && (headDimQk >= 32) && (headDimV >= 32) && (headDimPerCtaV <= 1024) && (headDimQk <= 1024) && (headDimV <= 1024), @@ -615,7 +609,6 @@ class TllmGenFmhaKernel // Bit 54 - 54: uses2CtaMma. // Bit 55 - 56: sparseAttention. // Bit 57 - 57: skipsSoftmax. - // Bit 58 - 58: fusesDsv4InvRopeFp8Quant. return (static_cast<uint64_t>(qkvLayout) << 0) | (static_cast<uint64_t>(maskType) << 4) | (static_cast<uint64_t>(kernelType) << 8) | (static_cast<uint64_t>(scheduler) << 12) | (static_cast<uint64_t>(multiCtasKvMode) << 16) | (static_cast<uint64_t>(headDimPerCtaV >> 3) << 18) @@ -624,7 +617,7 @@ class TllmGenFmhaKernel | (static_cast<uint64_t>(numTokensPerPage > 0 ? static_cast<int>(log2(numTokensPerPage)) : 0) << 44) | (static_cast<uint64_t>(log2(tileSizeQ)) << 49) | (static_cast<uint64_t>(reuseSmemKForV) << 53) | (static_cast<uint64_t>(uses2CtaMma) << 54) | (static_cast<uint64_t>(sparseAttention) << 55) - | (static_cast<uint64_t>(skipsSoftmax) << 57) | (static_cast<uint64_t>(fusesDsv4InvRopeFp8Quant) << 58); + | (static_cast<uint64_t>(skipsSoftmax) << 57); } uint64_t hashID(KernelMeta const& kernelMeta) const @@ -632,8 +625,7 @@ class TllmGenFmhaKernel return hashID(kernelMeta.mQkvLayout, kernelMeta.mMaskType, kernelMeta.mKernelType, kernelMeta.mTileScheduler, kernelMeta.mMultiCtasKvMode, kernelMeta.mHeadDimPerCtaV, kernelMeta.mHeadDimQk, kernelMeta.mHeadDimV, kernelMeta.mTileSizeQ, kernelMeta.mTileSizeKv, kernelMeta.mNumTokensPerPage, kernelMeta.mReuseSmemKForV, - kernelMeta.m2CtaMma, kernelMeta.mSparseAttn, kernelMeta.mSkipsSoftmaxWhenPossible, - kernelMeta.mFusesDsv4InvRopeFp8Quant); + kernelMeta.m2CtaMma, kernelMeta.mSparseAttn, kernelMeta.mSkipsSoftmaxWhenPossible); } std::pair<uint64_t, std::string> hashFromFmhaOptions(FmhaOptions const& options) const @@ -661,8 +653,7 @@ class TllmGenFmhaKernel + std::to_string(options.mNumTokensPerPage) + ", reuseSmemKForV=" + std::to_string(options.mReuseSmemKForV) + ", uses2CtaMma=" + std::to_string(uses2CtaMma) + ", sparseType=" + std::to_string(static_cast<int>(options.mSparseType)) - + ", skipsSoftmax=" + std::to_string(options.mSkipsSoftmaxWhenPossible) - + ", fusesDsv4InvRopeFp8Quant=" + std::to_string(options.mFusesDsv4InvRopeFp8Quant); + + ", skipsSoftmax=" + std::to_string(options.mSkipsSoftmaxWhenPossible); TLLM_LOG_DEBUG("Searching for kernel traits: " + info); return std::make_pair(hashID(static_cast<int>(options.mQkvLayout), static_cast<int>(options.mMaskType), @@ -671,8 +662,7 @@ class TllmGenFmhaKernel static_cast<int>(options.mHeadDimQk), static_cast<int>(options.mHeadDimV), static_cast<int>(options.mTileSizeQ), static_cast<int>(options.mTileSizeKv), static_cast<int>(options.mNumTokensPerPage), options.mReuseSmemKForV, uses2CtaMma, - static_cast<int>(options.mSparseType), options.mSkipsSoftmaxWhenPossible, - options.mFusesDsv4InvRopeFp8Quant), + static_cast<int>(options.mSparseType), options.mSkipsSoftmaxWhenPossible), info); } @@ -867,11 +857,6 @@ class TllmGenFmhaKernel fmhaData.mScales.outputScaleD = params.outputScalePtr; fmhaData.mScales.kvSfScaleD = params.kvSfScalePtr; fmhaData.mScales.oSfScaleD = params.oSfScalePtr; - if (params.mDsv4EpilogueFusion.enabled) - { - fmhaData.mInputBuffers.dsv4InvRopeCosSinCacheD = params.mDsv4EpilogueFusion.cosSinCache; - fmhaData.mScales.dsv4OScaleFp32D = static_cast<float*>(params.oSfPtr); - } // Sage Attention scaling factors fmhaData.mScales.sageAttnSfsQPtrD = params.sageAttnSfsQPtr; fmhaData.mScales.sageAttnSfsKPtrD = params.sageAttnSfsKPtr; @@ -1013,12 +998,6 @@ class TllmGenFmhaKernel { options.mSupportsDiffSeqLensForQAndKv = true; } - if (params.mDsv4EpilogueFusion.enabled) - { - options.mFusesDsv4InvRopeFp8Quant = true; - options.mDtypeOut = tg::Dtype::E4m3; - options.mDsv4ScaleBufM = params.mDsv4EpilogueFusion.scaleBufM; - } // Enables the optimization to skip the correction step when possible. options.mSkipsCorrWhenPossible = true; @@ -1222,8 +1201,7 @@ class TllmGenFmhaKernel + ", reuseSmemKForV=" + std::to_string(selectKernelParams.mReuseSmemKForV) + ", uses2CtaMma=" + std::to_string(selectKernelParams.mUses2CtaMma) + ", sparseAttention=" + std::to_string(static_cast<int>(params.mSparseAttention)) - + ", skipsSoftmax=" + std::to_string(selectKernelParams.mSkipsSoftmaxWhenPossible) - + ", fusesDsv4InvRopeFp8Quant=" + std::to_string(params.mDsv4EpilogueFusion.enabled); + + ", skipsSoftmax=" + std::to_string(selectKernelParams.mSkipsSoftmaxWhenPossible); TLLM_LOG_DEBUG("Searching for kernel traits: " + info); @@ -1234,7 +1212,7 @@ class TllmGenFmhaKernel params.mHeadDimQk, params.mHeadDimV, selectKernelParams.mTileSizeQ, selectKernelParams.mTileSizeKv, selectKernelParams.mNumTokensPerPage, selectKernelParams.mReuseSmemKForV, selectKernelParams.mUses2CtaMma, static_cast<int>(params.mSparseAttention), - selectKernelParams.mSkipsSoftmaxWhenPossible, params.mDsv4EpilogueFusion.enabled), + selectKernelParams.mSkipsSoftmaxWhenPossible), info); } @@ -1302,8 +1280,7 @@ class TllmFmhaKernelFactory KernelType* getKernels(const typename KernelType::KernelMeta* pKernelList, unsigned int nbKernels, Data_type dtypeQ, Data_type dtypeK, Data_type dtypeV, Data_type dtypeOut, unsigned int sm, int numEltsPerSageAttnBlkQ = 0, - int numEltsPerSageAttnBlkK = 0, int numEltsPerSageAttnBlkP = 0, int numEltsPerSageAttnBlkV = 0, - bool fusesDsv4InvRopeFp8Quant = false) + int numEltsPerSageAttnBlkK = 0, int numEltsPerSageAttnBlkP = 0, int numEltsPerSageAttnBlkV = 0) { static std::mutex s_mutex; std::lock_guard<std::mutex> lg(s_mutex); @@ -1312,7 +1289,7 @@ class TllmFmhaKernelFactory "SageAttention allows numEltsPerSageAttnBlk up to 64."); auto const id = hashID(dtypeQ, dtypeK, dtypeV, dtypeOut, sm, numEltsPerSageAttnBlkQ, numEltsPerSageAttnBlkK, - numEltsPerSageAttnBlkP, numEltsPerSageAttnBlkV, fusesDsv4InvRopeFp8Quant); + numEltsPerSageAttnBlkP, numEltsPerSageAttnBlkV); auto const findIter = mKernels.find(id); if (findIter == mKernels.end()) { @@ -1341,8 +1318,8 @@ class TllmFmhaKernelFactory TllmFmhaKernelFactory() = default; inline uint64_t hashID(Data_type dtypeQ, Data_type dtypeK, Data_type dtypeV, Data_type dtypeOut, unsigned int sm, - int numEltsPerSageAttnBlkQ, int numEltsPerSageAttnBlkK, int numEltsPerSageAttnBlkP, int numEltsPerSageAttnBlkV, - bool fusesDsv4InvRopeFp8Quant) const + int numEltsPerSageAttnBlkQ, int numEltsPerSageAttnBlkK, int numEltsPerSageAttnBlkP, + int numEltsPerSageAttnBlkV) const { auto const computeLog2BlockSizePlus1 = [](int blockSize) -> int { @@ -1363,14 +1340,12 @@ class TllmFmhaKernelFactory // Bit 35 - 37: log2NumEltsPerSageAttnBlkK + 1 -- 0 for non-sage, max numEltsPerSageAttnBlkK is 64. // Bit 38 - 40: log2NumEltsPerSageAttnBlkP + 1 -- 0 for non-sage, max numEltsPerSageAttnBlkP is 64. // Bit 41 - 43: log2NumEltsPerSageAttnBlkV + 1 -- 0 for non-sage, max numEltsPerSageAttnBlkV is 64. - // Bit 44 - 44: fusesDsv4InvRopeFp8Quant. return static_cast<uint64_t>(sm) | static_cast<uint64_t>(dtypeQ) << 16 | static_cast<uint64_t>(dtypeK) << 20 | static_cast<uint64_t>(dtypeV) << 24 | static_cast<uint64_t>(dtypeOut) << 28 | (static_cast<uint64_t>(computeLog2BlockSizePlus1(numEltsPerSageAttnBlkQ)) << 32) | (static_cast<uint64_t>(computeLog2BlockSizePlus1(numEltsPerSageAttnBlkK)) << 35) | (static_cast<uint64_t>(computeLog2BlockSizePlus1(numEltsPerSageAttnBlkP)) << 38) - | (static_cast<uint64_t>(computeLog2BlockSizePlus1(numEltsPerSageAttnBlkV)) << 41) - | (static_cast<uint64_t>(fusesDsv4InvRopeFp8Quant) << 44); + | (static_cast<uint64_t>(computeLog2BlockSizePlus1(numEltsPerSageAttnBlkV)) << 41); } std::unordered_map<uint64_t, const std::unique_ptr<KernelType>> mKernels; @@ -1378,14 +1353,13 @@ class TllmFmhaKernelFactory inline TllmGenFmhaKernel* getTllmFmhaKernels(Data_type dtypeQ, Data_type dtypeK, Data_type dtypeV, Data_type dtypeOut, unsigned int sm, int numEltsPerSageAttnBlkQ = 0, int numEltsPerSageAttnBlkK = 0, int numEltsPerSageAttnBlkP = 0, - int numEltsPerSageAttnBlkV = 0, bool fusesDsv4InvRopeFp8Quant = false) + int numEltsPerSageAttnBlkV = 0) { #ifndef EXCLUDE_SM_100F return TllmFmhaKernelFactory::Get().getKernels(sTllmGenFmhaKernelMetaInfos, sizeof(sTllmGenFmhaKernelMetaInfos) / sizeof(sTllmGenFmhaKernelMetaInfos[0]), dtypeQ, dtypeK, dtypeV, dtypeOut, - sm, numEltsPerSageAttnBlkQ, numEltsPerSageAttnBlkK, numEltsPerSageAttnBlkP, numEltsPerSageAttnBlkV, - fusesDsv4InvRopeFp8Quant); + sm, numEltsPerSageAttnBlkQ, numEltsPerSageAttnBlkK, numEltsPerSageAttnBlkP, numEltsPerSageAttnBlkV); #else return nullptr; #endif // EXCLUDE_SM_100F diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunner.cpp b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunner.cpp index dacae09aa588..aad8525d103d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunner.cpp +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunner.cpp @@ -30,8 +30,7 @@ namespace kernels //////////////////////////////////////////////////////////////////////////////////////////////////// TllmGenFmhaRunner::TllmGenFmhaRunner(Data_type dtypeQ, Data_type dtypeK, Data_type dtypeV, Data_type dtypeOut, - int numEltsPerSageAttnBlkQ, int numEltsPerSageAttnBlkK, int numEltsPerSageAttnBlkP, int numEltsPerSageAttnBlkV, - bool fusesDsv4InvRopeFp8Quant) + int numEltsPerSageAttnBlkQ, int numEltsPerSageAttnBlkK, int numEltsPerSageAttnBlkP, int numEltsPerSageAttnBlkV) : mSM(tensorrt_llm::common::getSMVersion()) , mDtypeQ(dtypeQ) , mDtypeK(dtypeK) @@ -41,7 +40,6 @@ TllmGenFmhaRunner::TllmGenFmhaRunner(Data_type dtypeQ, Data_type dtypeK, Data_ty , mNumEltsPerSageAttnBlkK(numEltsPerSageAttnBlkK) , mNumEltsPerSageAttnBlkP(numEltsPerSageAttnBlkP) , mNumEltsPerSageAttnBlkV(numEltsPerSageAttnBlkV) - , mFusesDsv4InvRopeFp8Quant(fusesDsv4InvRopeFp8Quant) { TLLM_CHECK_WITH_INFO(mSM == kSM_100 || mSM == kSM_103, "Unsupported architecture"); TLLM_CHECK_WITH_INFO(mDtypeQ == DATA_TYPE_E4M3 || mDtypeQ == DATA_TYPE_FP16 || mDtypeQ == DATA_TYPE_BF16 @@ -60,7 +58,7 @@ TllmGenFmhaRunner::TllmGenFmhaRunner(Data_type dtypeQ, Data_type dtypeK, Data_ty mTotalDeviceMemory = totalMemory; TLLM_CHECK_WITH_INFO(mTotalDeviceMemory > 0, "Total device memory is invalid"); mKernel = getTllmFmhaKernels(mDtypeQ, mDtypeK, mDtypeV, mDtypeOut, mSM, numEltsPerSageAttnBlkQ, - numEltsPerSageAttnBlkK, numEltsPerSageAttnBlkP, numEltsPerSageAttnBlkV, mFusesDsv4InvRopeFp8Quant); + numEltsPerSageAttnBlkK, numEltsPerSageAttnBlkP, numEltsPerSageAttnBlkV); } //////////////////////////////////////////////////////////////////////////////////////////////////// diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunner.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunner.h index bb84c45e6ee2..ee1ab71eab1a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunner.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunner.h @@ -34,7 +34,7 @@ class TllmGenFmhaRunner // Constructor. explicit TllmGenFmhaRunner(Data_type dtypeQ, Data_type dtypeK, Data_type dtypeV, Data_type dtypeOut, int numEltsPerSageAttnBlkQ = 0, int numEltsPerSageAttnBlkK = 0, int numEltsPerSageAttnBlkP = 0, - int numEltsPerSageAttnBlkV = 0, bool fusesDsv4InvRopeFp8Quant = false); + int numEltsPerSageAttnBlkV = 0); TllmGenFmhaRunner() = default; @@ -64,7 +64,6 @@ class TllmGenFmhaRunner int mNumEltsPerSageAttnBlkK; int mNumEltsPerSageAttnBlkP; int mNumEltsPerSageAttnBlkV; - bool mFusesDsv4InvRopeFp8Quant{false}; }; } // namespace kernels diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunnerParams.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunnerParams.h index 9252650ac67b..369e634fb61c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunnerParams.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunnerParams.h @@ -38,9 +38,7 @@ enum class TrtllmGenAttentionMaskType // Sliding window or chunked causal mask. SlidingOrChunkedCausal, // Custom mask. - Custom, - // Sliding window mask combined with custom packed mask. - SlidingWindowCustom + Custom }; //////////////////////////////////////////////////////////////////////////////////////////////////// @@ -56,22 +54,10 @@ enum class TrtllmGenAttentionMaskType ATTENTION_MASK_TYPE_FUNCTION(Dense) ATTENTION_MASK_TYPE_FUNCTION(Causal) ATTENTION_MASK_TYPE_FUNCTION(SlidingOrChunkedCausal) -ATTENTION_MASK_TYPE_FUNCTION(SlidingWindowCustom) +ATTENTION_MASK_TYPE_FUNCTION(Custom) #undef ATTENTION_MASK_TYPE_FUNCTION -inline bool isCustomMask(TrtllmGenAttentionMaskType maskType) -{ - return maskType == TrtllmGenAttentionMaskType::Custom - || maskType == TrtllmGenAttentionMaskType::SlidingWindowCustom; -} - -inline bool usesSlidingWindowMask(TrtllmGenAttentionMaskType maskType) -{ - return maskType == TrtllmGenAttentionMaskType::SlidingOrChunkedCausal - || maskType == TrtllmGenAttentionMaskType::SlidingWindowCustom; -} - //////////////////////////////////////////////////////////////////////////////////////////////////// enum class FmhaKernelType @@ -224,16 +210,6 @@ MULTI_CTAS_KV_MODE_FUNCTION(CgaSmemReduction) struct TllmGenFmhaRunnerParams { - struct Dsv4EpilogueFusionParams - { - // Enable DSv4 inverse-RoPE + FP8 quant epilogue fusion. - bool enabled{false}; - // The cos/sin cache used by the fused inverse-RoPE epilogue. - float const* cosSinCache{nullptr}; - // The physical token stride of the FP32 output scale tensor. - int32_t scaleBufM{0}; - }; - // Input layout. QkvLayout mQkvLayout; // Attention mask type. @@ -298,8 +274,6 @@ struct TllmGenFmhaRunnerParams void* oPtr; // The output scaling factor buffer. void* oSfPtr; - // Optional DSv4 fused inverse-RoPE + FP8 quant epilogue parameters. - Dsv4EpilogueFusionParams mDsv4EpilogueFusion; // SageAttention scaling factors for Q, K, P and V. float const* sageAttnSfsQPtr = nullptr; float const* sageAttnSfsKPtr = nullptr; @@ -369,7 +343,6 @@ struct TllmGenFmhaRunnerParams // row stride ceilDiv(mPackedMaskMaxSeqLenQ, 32) rather than ceilDiv(seqLenQ, 32). int32_t mPackedMaskMaxSeqLenQ = 0; int32_t mSpecDecodingTargetMaxGenLen = 0; - bool mForcePrepareSpecDecTreeMask = false; // set the attention mask type TllmGenFmhaRunnerParams& setAttentionMaskType(std::int8_t maskType) @@ -387,7 +360,7 @@ struct TllmGenFmhaRunnerParams case 2: // tensorrt_llm::kernels::ContextAttentionMaskType::SLIDING_OR_CHUNKED_CAUSAL mMaskType = TrtllmGenAttentionMaskType::SlidingOrChunkedCausal; break; - case 4: // tensorrt_llm::kernels::ContextAttentionMaskType::CUSTOM_MASK + case 3: // tensorrt_llm::kernels::ContextAttentionMaskType::CUSTOM_MASK mMaskType = TrtllmGenAttentionMaskType::Custom; break; default: diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/aarch64-linux-gnu/libTrtLlmGen.a b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/aarch64-linux-gnu/libTrtLlmGen.a index b00b8dabaa81..2b801e7fc405 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/aarch64-linux-gnu/libTrtLlmGen.a +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/aarch64-linux-gnu/libTrtLlmGen.a @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7057231cff1dc01ca4249ca8da35869df4fc6403d1cd629f657c3dcf08c6440d -size 7299448 +oid sha256:aa57847e1873d669e313860298833aec78450e3baf9a1a01f078eb46534afafa +size 7291080 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/aarch64-linux-gnu/libTrtLlmGenFmhaLib.a b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/aarch64-linux-gnu/libTrtLlmGenFmhaLib.a index f2bcb9d72da7..c5ded85192c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/aarch64-linux-gnu/libTrtLlmGenFmhaLib.a +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/aarch64-linux-gnu/libTrtLlmGenFmhaLib.a @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:32243e9163f943f6c8908a55cfe287255e5268a545d04b794b764e374f15ea0d -size 5075466 +oid sha256:fa34826c99c3f55b4b56576af678b0c6302067aac1cedc055ab578b245a55e81 +size 4071348 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/x86_64-linux-gnu/libTrtLlmGen.a b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/x86_64-linux-gnu/libTrtLlmGen.a index 0f82d902a00b..55c427774b5c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/x86_64-linux-gnu/libTrtLlmGen.a +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/x86_64-linux-gnu/libTrtLlmGen.a @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37db982312fe782dd0ad8282e695b6c8a83428d306dbe2f2c94b6a578f5bc7f6 -size 7352462 +oid sha256:22bc6b33e9003048bee99c18e8a6ba46c7811f299c2a5470889213d77e5b8fe3 +size 7346268 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/x86_64-linux-gnu/libTrtLlmGenFmhaLib.a b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/x86_64-linux-gnu/libTrtLlmGenFmhaLib.a index bf41da145bdd..e2caadd1a725 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/x86_64-linux-gnu/libTrtLlmGenFmhaLib.a +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/lib/x86_64-linux-gnu/libTrtLlmGenFmhaLib.a @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50cb4f541392084afda1d4c809ff387134b0cedebb35a9bcea683926b42afee0 -size 5146356 +oid sha256:18a8d7fcff1c041a41b2f5ee60274287106a71f606065ef01008dd09057321d1 +size 4139500 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/Dsv4Constants.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/Dsv4Constants.h deleted file mode 100644 index 0e111654cd5b..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/Dsv4Constants.h +++ /dev/null @@ -1,69 +0,0 @@ -/* - * Copyright (c) 2011-2026, NVIDIA CORPORATION. All rights reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#pragma once - -#include <cstdint> - -namespace fmha { - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -// DSv4 fused inverse-RoPE + FP8 quant ABI constants. -// -// Logical FMHA tensors: -// Q: [sumOfSeqLensQ, numHeadsQ, headDimQk] -// K: [sumOfSeqLensKv, numHeadsKv, headDimQk] -// V: [sumOfSeqLensKv, numHeadsKv, headDimV] -// logical O: [sumOfSeqLensQ, numHeadsQ, headDimV] -// -// Fused epilogue outputs: -// FP8 O: [numHeadsQ / headsPerGroup, sumOfSeqLensQ, headsPerGroup, headDimV] -// FP32 scale: [numHeadsQ / headsPerGroup, -// headsPerGroup * headDimV / quantGroupSize, -// scaleBufM] -// -// Inverse-RoPE inputs: -// positionIds: [sumOfSeqLensQ] -// cosSin cache: [maxPosition, cosSinStride], with each row laid out as -// [cos(ropeHalf), sin(ropeHalf)]. -// -// Dimension mapping: -// headDimQk is the last dimension of Q/K. -// headDimV is the last dimension of V, logical O, and FP8 O. -// headDimPerCtaV is the V/O head-dimension slice per CTA; clusterDimX=2 covers headDimV. -// headsPerGroup is FP8 O dim2 and the head-in-group factor in FP32 scale dim1. -// quantGroupSize is the 128-column headDimV block; FP32 scale has one value per block. -// scaleBufM is FP32 scale dim2, the physical token stride for the scale tensor. -// ropeStart/ropeHalf cover headDimV[448:512); ropeOffsetInBlock is the offset inside that block. - -inline constexpr int32_t kDsv4HeadDimQk = 512; -inline constexpr int32_t kDsv4HeadDimV = 512; -inline constexpr int32_t kDsv4HeadDimPerCtaV = 256; -inline constexpr int32_t kDsv4HeadsPerGroup = 8; - -inline constexpr int32_t kDsv4QuantGroupSize = 128; -inline constexpr int32_t kDsv4Log2QuantGroupSize = 7; // log2(128) -inline constexpr int32_t kDsv4RopeStart = 448; -inline constexpr int32_t kDsv4RopeHalf = 32; -inline constexpr int32_t kDsv4RopeOffsetInBlock = kDsv4RopeStart % kDsv4QuantGroupSize; -inline constexpr int32_t kDsv4CosSinStride = kDsv4RopeHalf * 2; - -static_assert(kDsv4HeadDimV % kDsv4QuantGroupSize == 0); -static_assert((kDsv4QuantGroupSize & (kDsv4QuantGroupSize - 1)) == 0); -static_assert(kDsv4RopeOffsetInBlock + kDsv4RopeHalf * 2 == kDsv4QuantGroupSize); - -} // namespace fmha diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaAutoTuner.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaAutoTuner.h index 00d1270ffdf4..8d8f92f419b0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaAutoTuner.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaAutoTuner.h @@ -66,15 +66,7 @@ class FmhaAutoTuner { // Select the MLA generation kernel. void selectMlaGenerationKernel(); - // Select the grouped Q64 MLA generation kernel when the capability predicate matches. - void selectGroupedMlaGenerationKernel(); - private: - // computeNumCtas uses the same cluster occupancy helper as enableCgaReduction. - friend std::tuple<int32_t, int32_t, int32_t> computeNumCtas(FmhaOptions& options, - int32_t multiProcessorCount, - bool enablesLogging); - // Enables the cgaReduction if all clusters can be launched in one wave. void enableCgaReduction(int32_t numCtasX, int32_t numCtasY, int32_t numCtasZ); @@ -86,7 +78,7 @@ class FmhaAutoTuner { // Get the maximum number of active clusters for a given cluster size which considers the // floorsweeping configurations. - static int32_t getMaxNumActiveClusters(int32_t clusterSize); + int32_t getMaxNumActiveClusters(int32_t clusterSize); // Selects the tileSizeQ for GQA generation kernels. void selectTileSizeQForGqaGeneration(); diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaInterface.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaInterface.h index 3ed9a276e2c4..a89be228dc9d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaInterface.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaInterface.h @@ -69,9 +69,6 @@ struct FmhaData { }; struct Scales { - // DSv4 inverse-RoPE + 1x128 FP8 quant fusion output scale tensor. - float* dsv4OScaleFp32D{nullptr}; - // FP4 scaling factors for KV cache void const* kSfBasePtr; void const* vSfBasePtr; @@ -108,9 +105,6 @@ struct FmhaData { void const* kBasePtr; void const* vBasePtr; - // DSv4 inverse-RoPE metadata inputs. - float const* dsv4InvRopeCosSinCacheD{nullptr}; - // Base pointer for the DSv4 sparse MLA sliding-window KV pool. void const* slidingWindowKvPoolBasePtr{nullptr}; @@ -159,7 +153,6 @@ struct FmhaData { class FmhaInterface { public: using ModuleCache = std::unordered_map<std::string, std::tuple<CUmodule, CUfunction>>; - enum class KernelCacheStatus { Unknown, CacheHit, CacheMiss }; FmhaInterface(bool exportsCubin = false, int32_t numRotations = 1, bool verbose = false) : mExportsCubin(exportsCubin) @@ -171,7 +164,7 @@ class FmhaInterface { // Set the verbosity level for logging. When false, TLLM_LOG_INFO messages are suppressed. void setVerbose(bool verbose); - KernelCacheStatus generateAndCompileKernel(FmhaConfig& fmhaConfig) const; + void generateAndCompileKernel(FmhaConfig& fmhaConfig) const; std::string getKernelNameFromConfigs(FmhaConfig const& fmhaConfig) const; diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaOptions.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaOptions.h index 33f3a7304753..4852adc90b74 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaOptions.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/FmhaOptions.h @@ -16,7 +16,6 @@ #pragma once -#include "Dsv4Constants.h" #include "KernelTraits.h" #include <trtllm/gen/CudaArchDecl.h> #include <trtllm/gen/CudaRunner.h> @@ -46,12 +45,8 @@ struct FmhaOptions : public KernelConfigBase { int mChunkedAttentionSize{0}; // Dry-run: print a log but does not actually generate anything bool mDryRun{false}; - // Token dimension reserved by the DSv4 FP32 scale tensor. This host-side value may be padded. - int32_t mDsv4ScaleBufM{0}; // Enable the auto tuner. bool mEnablesAutoTuner{false}; - // Select the grouped MLA generation kernel in the auto tuner. - bool mSelectsGroupedMla{false}; // Enable the BF16Q+FP8KV K-only transform path. Disabled by default. bool mEnablesBf16QFp8KvKOnlyTransform{false}; // Whether is exporting cubin. @@ -133,9 +128,7 @@ struct FmhaOptions : public KernelConfigBase { TO_JSON(mChecksResults); TO_JSON(mChunkedAttentionSize); TO_JSON(mDryRun); - TO_JSON(mDsv4ScaleBufM); TO_JSON(mEnablesAutoTuner); - TO_JSON(mSelectsGroupedMla); TO_JSON(mEnablesBf16QFp8KvKOnlyTransform); TO_JSON(mIsExportingCubin); TO_JSON(mIsTracing); @@ -201,8 +194,6 @@ struct FmhaOptionsFromArgs { bool mIsNumPrefetchedFmasSet{false}; // Relative error tolerance. bool mIsRtolSet{false}; - // Whether to use separate transformed K/V resources. - bool mIsSeparateTransformedKvSet{false}; // Tile scheduler type. bool mIsTileSchedulerSet{false}; // Whether to use an ordered sequence between softmax0 and softmax1. @@ -241,13 +232,6 @@ struct FmhaConfig { //////////////////////////////////////////////////////////////////////////////////////////////////// -// Whether the output dtype produces per-block scale factors. -inline bool hasOutputSfs(tg::Dtype dtype) { - return dtype == tg::Dtype::E2m1 || dtype == tg::Dtype::MxE4m3; -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - // Check if the options are valid or not. inline void checkFmhaOptions(FmhaOptions const& options, FmhaOptionsFromArgs const& optionsFromArgs) { @@ -258,32 +242,6 @@ inline void checkFmhaOptions(FmhaOptions const& options, // mNumInstsKv == 1 are supported. TLLM_CHECK_ERROR(((options.mNumInstsQ * options.mNumInstsKv) <= 2), "Only two tile instances are supported"); - if (isBf16QFp8KvFullTransformGeneration(options)) { - TLLM_CHECK_ERROR(options.mNumInstsQ == 1 && options.mNumInstsKv == 1, - "BF16Q+FP8KV full-transform kernels require numInstsQ == 1 and " - "numInstsKv == 1."); - } - - if (options.mFusesDsv4InvRopeFp8Quant) { - bool const isSpecDecTree = - options.mIsCustomSpecDecodingGen && options.mSpecDecodingTargetMaxGenLen > 0; - bool const isSupportedDsv4FusionConfig = - options.mIsMlaGen && options.mSparseType == SparseType::DynamicTokenSparse && - options.mFuseEpilogueIntoCorr && isKeepsMmaAbForGenerationKernel(options.mFmhaKernelType) && - options.mQkvLayout == QkvLayout::PagedKv && options.mDtypeQ == tg::Dtype::E4m3 && - options.mDtypeK == tg::Dtype::E4m3 && options.mDtypeV == tg::Dtype::E4m3 && - options.mDtypeOut == tg::Dtype::E4m3 && options.mHeadDimQk == kDsv4HeadDimQk && - options.mHeadDimV == kDsv4HeadDimV && options.mHeadDimPerCtaV == kDsv4HeadDimPerCtaV && - options.mHeadDimPerCtaV * options.mClusterDimX == options.mHeadDimV && - options.mClusterDimX == 2 && options.mTileSizeQ == 64 && options.mTileSizeKv == 128 && - options.mNumInstsQ == 1 && options.mNumInstsKv == 1 && !options.mSwapsMmaAb && - options.mHeadDimPerStageKv == 0 && options.mMultiCtasKvMode == MultiCtasKvMode::Disabled && - !options.mUseBlockSparseAttention && !isSpecDecTree; - TLLM_CHECK_ERROR(isSupportedDsv4FusionConfig, - "DSv4 inverse-RoPE FP8 quant fusion only supports the fixed DSv4 sparse MLA " - "generation keep-AB paged-KV E4M3 configuration with standard non-tree " - "causal generation/context position semantics."); - } // The number of instances for Q and Kv must be set together. TLLM_CHECK_ERROR(optionsFromArgs.mIsNumInstsQSet == optionsFromArgs.mIsNumInstsKvSet, @@ -327,11 +285,8 @@ inline void checkFmhaOptions(FmhaOptions const& options, "Only headDimQk > headDimV MLA kernels have been verified for Hopper"); } else { if (isContextKernel(options.mFmhaKernelType)) { - TLLM_CHECK_ERROR((headDimQk == 192 && headDimV == 128) || - (headDimQk == 128 && headDimV == 64), - "Only headDimQk = 192, headDimV = 128 (DeepSeek context MLA) or " - "headDimQk = 128, headDimV = 64 (Mistral Small 4 context MLA) kernels " - "have been verified"); + TLLM_CHECK_ERROR(headDimQk == 192 && headDimV == 128, + "Only headDimQk = 192, headDimV = 128 MLA kernels have been verified"); } else { TLLM_CHECK_ERROR(options.mIsMlaGen && ((headDimQk == 576 && headDimV == 512) || (headDimQk == 320 && headDimV == 256)), @@ -400,18 +355,21 @@ inline void checkFmhaOptions(FmhaOptions const& options, "Consider using smaller tileSizeKv."); } +#ifdef TLLM_PUBLIC_RELEASE + if (options.mDtypeKv == tg::Dtype::E2m1 && !options.mIsTrtllmLayout) { + TLLM_CHECK_ERROR(false, "E2m1 KV cache is not supported with public compiler."); + } +#endif // TLLM_PUBLIC_RELEASE + // PackedQkv layout does not support supportsDiffSeqLensForQAndKv. TLLM_CHECK_ERROR(!(isPackedQkv(options.mQkvLayout) && options.mSupportsDiffSeqLensForQAndKv), "PackedQkv layout does not support supportsDiffSeqLensForQAndKv"); // Q does not support E2m1 dtype. TLLM_CHECK_ERROR(options.mDtypeQ != tg::Dtype::E2m1, "Q does not suppot E2m1 dtype"); // Make sure correct attention window size is set. - TLLM_CHECK_ERROR(!usesSlidingWindowMask(options.mMaskType) || options.mAttentionWindowSize > 0 || - options.mChunkedAttentionSize > 0, + TLLM_CHECK_ERROR(!isSlidingOrChunkedCausalMask(options.mMaskType) || + options.mAttentionWindowSize > 0 || options.mChunkedAttentionSize > 0, "Please set correct sliding attention window size or chunked attention size"); - TLLM_CHECK_ERROR(!isSlidingWindowCustomMask(options.mMaskType) || - (options.mAttentionWindowSize > 0 && options.mChunkedAttentionSize == 0), - "SlidingWindowCustom requires attentionWindowSize without chunked attention"); if (options.mChunkedAttentionSize > 0) { TLLM_CHECK_ERROR(options.mAttentionWindowSize >= options.mMaxSeqLenKv, "sliding attention window size must be greater than or equal to maxSeqLenKv"); @@ -422,20 +380,17 @@ inline void checkFmhaOptions(FmhaOptions const& options, "Chunked attention size must be power of 2"); } - // Special options for block-scaled outputs. - if (fmha::hasOutputSfs(options.mDtypeOut)) { + // Special options for FP4. + if (options.mDtypeOut == tg::Dtype::E2m1) { + // FP4 output only supports fuseEpilogueIntoCorr. TLLM_CHECK_ERROR(options.mFuseEpilogueIntoCorr, - "E2m1 / MxE4m3 output only supports fuseEpilogueIntoCorr"); - - // Make sure the number of SFs per row can be divided by 4, required for interleaved SF layout. - int32_t numEltsPerSfO = tg::dtypeNumEltsPerSf(options.mDtypeOut); - int32_t hiddenDim = options.mNumHeadsQ * options.mHeadDimV; - TLLM_CHECK_ERROR(options.mHeadDimV % numEltsPerSfO == 0, - "headDimV must be divisible by the output SF group size"); - TLLM_CHECK_ERROR(hiddenDim % numEltsPerSfO == 0, - "hiddenDim must be divisible by the output SF group size"); - TLLM_CHECK_ERROR((hiddenDim / numEltsPerSfO) % 4 == 0, - "Current hiddenDim is not compatible with interleaved SF layout"); + "FP4 output only supports fuseEpilogueIntoCorr"); + // Make sure the number of sf per row can be divided by 4, required for interleaved SF layout. + // Details can be seen in DtypeUtils.h: E2m1Utils::getSfOffset. + int32_t hiddenDim = options.mNumHeadsQ * headDimQk; + auto kernelTraits = getKernelTraitsFromOptions(options); + TLLM_CHECK_ERROR((hiddenDim / kernelTraits.mNumEltsPerSf) % 4 == 0, + "Current hiddenDim is not supported for FP4 output"); } // If we decide to use Sage Attention, the number of elements per block must be a power-of-two. @@ -532,13 +487,12 @@ inline void checkFmhaOptions(FmhaOptions const& options, } } - // groupsTokensHeadsQ is allowed for GQA gen and explicitly selected MLA gen kernels. + // The mGroupsTokensHeadsQ only works with GQA generation kernels. if (options.mGroupsTokensHeadsQ) { TLLM_CHECK_ERROR(!isContextKernel(options.mFmhaKernelType), "mGroupsTokensHeadsQ should only be enabled for generation kernels."); - TLLM_CHECK_ERROR(!options.mIsMlaGen - || options.mSelectsGroupedMla, - "MLA generation with mGroupsTokensHeadsQ requires mSelectsGroupedMla."); + TLLM_CHECK_ERROR(!options.mIsMlaGen, + "MLA gen kernels haven't supported mGroupsTokensHeadsQ yet."); } @@ -551,17 +505,7 @@ inline void checkFmhaOptions(FmhaOptions const& options, if (options.mEnablesBf16QFp8KvKOnlyTransform) { TLLM_CHECK_ERROR(usesKOnlyTransformPipeline(options), "BF16Q+FP8KV K-only transform is only supported for non-MLA Blackwell " - "generation kernels with BF16 Q, E4M3 K/V, and H64/H128/H256."); - TLLM_CHECK_ERROR(!options.mSeparateTransformedKv, - "BF16Q+FP8KV K-only transform cannot be combined with separateTransformedKv."); - } - if (options.mSeparateTransformedKv) { - TLLM_CHECK_ERROR(!usesKOnlyTransformPipeline(options), - "BF16Q+FP8KV K-only transform cannot be combined with separateTransformedKv."); - TLLM_CHECK_ERROR(supportsSeparateTransformedKv(options), - "separateTransformedKv is only supported by BF16Q+E4M3KV full-transform " - "generation kernels on Blackwell with numInstsQ=1, numInstsKv=1, and equal " - "H64/H128/H256 K/V heads."); + "generation kernels with BF16 Q, E4M3 K/V, and H64/H128."); } if (options.mMmaOrder == MmaOrder::Qk0_Qk1_Pv0_Pv1) { @@ -648,26 +592,4 @@ inline void updateFmhaOptions(FmhaOptions& options, FmhaOptionsFromArgs const& o //////////////////////////////////////////////////////////////////////////////////////////////////// -// Update the runtime DSv4 inverse-RoPE + FP8 quant layout. This must run after the sample sequence -// lengths have been generated because the compact benchmark layout depends on mSumOfSeqLensQ. -inline void updateDsv4InvRopeFp8QuantOptions(FmhaOptions& options) { - if (!options.mFusesDsv4InvRopeFp8Quant) { - return; - } - - TLLM_CHECK_ERROR(options.mNumHeadsQ % kDsv4HeadsPerGroup == 0, - "numHeadsQ must be divisible by the DSv4 packed output head group size."); - - if (options.mDsv4ScaleBufM == 0) { - int32_t constexpr scaleTokenAlignment = 4; - int32_t const numPackedTokens = options.mSumOfSeqLensQ; - int32_t const paddedScaleBufM = - (numPackedTokens + scaleTokenAlignment - 1) / scaleTokenAlignment * scaleTokenAlignment; - options.mDsv4ScaleBufM = paddedScaleBufM; - } - TLLM_CHECK_ERROR(options.mDsv4ScaleBufM > 0, "Dsv4ScaleBufM must be initialized."); -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - } // namespace fmha diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelConfigBase.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelConfigBase.h index 8e0af884eb21..f457449a7a63 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelConfigBase.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelConfigBase.h @@ -118,9 +118,7 @@ enum class AttentionMaskType { // Sliding window causal mask or chunked attention causal mask. SlidingOrChunkedCausal, // Custom mask. - Custom, - // Sliding window mask combined with custom packed mask. - SlidingWindowCustom + Custom }; //////////////////////////////////////////////////////////////////////////////////////////////////// @@ -135,20 +133,10 @@ enum class AttentionMaskType { ATTENTION_MASK_TYPE_FUNCTION(Dense) ATTENTION_MASK_TYPE_FUNCTION(Causal) ATTENTION_MASK_TYPE_FUNCTION(SlidingOrChunkedCausal) -ATTENTION_MASK_TYPE_FUNCTION(SlidingWindowCustom) +ATTENTION_MASK_TYPE_FUNCTION(Custom) #undef ATTENTION_MASK_TYPE_FUNCTION -__host__ __device__ inline bool isCustomMask(AttentionMaskType maskType) { - return maskType == AttentionMaskType::Custom || - maskType == AttentionMaskType::SlidingWindowCustom; -} - -__host__ __device__ inline bool usesSlidingWindowMask(AttentionMaskType maskType) { - return maskType == AttentionMaskType::SlidingOrChunkedCausal || - maskType == AttentionMaskType::SlidingWindowCustom; -} - //////////////////////////////////////////////////////////////////////////////////////////////////// enum class FmhaKernelType { @@ -399,8 +387,6 @@ template <> inline std::string toString(AttentionMaskType e) { return "SlidingOrChunkedCausal"; case AttentionMaskType::Custom: return "Custom"; - case AttentionMaskType::SlidingWindowCustom: - return "SlidingWindowCustom"; default: TLLM_LOG_ERROR("Unsupported enum."); return ""; @@ -502,6 +488,8 @@ template <> inline std::string toString(MmaOrder e) { X(tg::Dtype, mDtypeOut, tg::Dtype::E4m3, uint32_t) \ /* Whether to use dynamic numTokensPerPage. */ \ X(bool, mDynamicNumTokensPerPage, false, bool) \ + /* Whether to use fp16 softmax. */ \ + X(bool, mEnablesFp16Softmax, false, bool) \ /* Do we enable max value inflation? */ \ X(bool, mEnablesInflateMax, false, bool) \ /* Whether 2 instances of the softmax task could be merged ? */ \ @@ -523,12 +511,8 @@ template <> inline std::string toString(MmaOrder e) { /* Store tensor to gmem directly in the end of the correction task. */ \ /* True: vectorized store. False: TMA store using dedicated warp. */ \ X(bool, mFuseEpilogueIntoCorr, true, bool) \ - /* Fuse DSv4 inverse RoPE + 1x128 E4M3 quantization into the correction epilogue. */ \ - X(bool, mFusesDsv4InvRopeFp8Quant, false, bool) \ /* Whether to transform K/V in the correction task. */ \ X(bool, mFuseTransformKvIntoCorr, true, bool) \ - /* Whether to allocate separate transformed-K/V resources with independent pipelines. */ \ - X(bool, mSeparateTransformedKv, false, bool) \ /* Whether to group the headsQ into one CTA. */ \ X(bool, mGroupsHeadsQ, false, bool) \ /* Whether to group both tokensQ and headsQ into one CTA. */ \ @@ -657,10 +641,11 @@ template <> inline std::string toString(MmaOrder e) { //////////////////////////////////////////////////////////////////////////////////////////////////// -#define KERNEL_CONFIG_BASE_FIELDS(X) KERNEL_CONFIG_BASE_FIELDS_BASE(X) - -//////////////////////////////////////////////////////////////////////////////////////////////////// +#define KERNEL_CONFIG_BASE_FIELDS_EXTRA(X) +#define KERNEL_CONFIG_BASE_FIELDS(X) \ + KERNEL_CONFIG_BASE_FIELDS_BASE(X) \ + KERNEL_CONFIG_BASE_FIELDS_EXTRA(X) //////////////////////////////////////////////////////////////////////////////////////////////////// @@ -727,3 +712,4 @@ template <> struct hash<fmha::KernelConfigBase> { #undef KERNEL_CONFIG_BASE_FIELDS #undef KERNEL_CONFIG_BASE_FIELDS_BASE +#undef KERNEL_CONFIG_BASE_FIELDS_EXTRA diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelParams.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelParams.h index 5660ca7eb716..4364f3e938f3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelParams.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelParams.h @@ -347,9 +347,9 @@ template <class FmhaOptions> static auto makeStrideKv(FmhaOptions const& options strideKeysVals = hiddenDimQkv; } else if (isContiguousKv(options.mQkvLayout)) { strideKeysVals = paddedHeadDimKv; - } else if (isSeparateQkv(options.mQkvLayout) && !isK && - options.mHeadDimQk != options.mHeadDimV && options.mDtypeKv != tg::Dtype::E4m3) { - // Non-FP8 context MLA (DeepSeek 192/128, Mistral 128/64, ...): V is not head-contiguous. + } else if (isSeparateQkv(options.mQkvLayout) && !isK && options.mHeadDimQk == 192 && + options.mDtypeKv != tg::Dtype::E4m3) { + // Non-FP8 context MLA: tensor V is not contiguous. strideKeysVals = options.mNumHeadsKv * (options.mHeadDimQk - 64 + options.mHeadDimV); } @@ -525,8 +525,6 @@ static KernelParams updateKernelParams(FmhaOptions_ const& options, slidingWindowKvPoolBasePtr, params.ptrPageIdxKv, params.ptrOutputScale, - params.ptrDsv4InvRopeCosSinCache, - params.ptrDsv4OScaleFp32, params.ptrScaleSoftmaxLog2, params.ptrScaleSfKv, params.ptrScaleSfO, @@ -573,8 +571,6 @@ static KernelParams setKernelParams(FmhaOptions_ const& options, void const* slidingWindowKvPoolBasePtr, int const* kvPageIdxD, float const* outputScaleD, - float const* dsv4InvRopeCosSinCacheD, - float* dsv4OScaleFp32D, float const* scaleSoftmaxLog2D, float const* kvSfScaleD, float const* oSfScaleD, @@ -807,8 +803,6 @@ static KernelParams setKernelParams(FmhaOptions_ const& options, // TRT-LLM restrictions: the quantization scales must be on the device. It will only be loaded // when -loadsScalesFromGmem true -dtypeElt e4m3 are specified. params.ptrOutputScale = outputScaleD; - params.ptrDsv4InvRopeCosSinCache = dsv4InvRopeCosSinCacheD; - params.ptrDsv4OScaleFp32 = dsv4OScaleFp32D; // The partial buffers' pointers when the multiCtasKv mode is enabled. params.ptrMultiCtasKvCounter = multiCtasKvCounterPtrD; @@ -891,7 +885,6 @@ static KernelParams setKernelParams(FmhaOptions_ const& options, params.mNumHiddenEltsO = options.mNumHeadsQ * options.mHeadDimV; params.mNumTokensPerCtaQ = numTokensPerCtaQ; params.mOutputScale = options.mOutputScale; - params.mDsv4ScaleBufM = options.mDsv4ScaleBufM; params.mScaleSoftmaxLog2 = softmaxScale; params.mScaleSfKv = kvSfScale; params.mScaleSfO = oSfScale; @@ -940,8 +933,6 @@ static KernelParams setKernelParams(FmhaOptions_ const&, int const*, float const*, float const*, - float*, - float const*, float const*, float const*, uint32_t const*, diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelParamsDecl.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelParamsDecl.h index 074c72995c22..4928050b65b6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelParamsDecl.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelParamsDecl.h @@ -52,7 +52,7 @@ struct KernelParams { // The output pointer (used by STG for last tile). void* ptrO; - // The output SF pointer (used for block-scaled output). + // The output SF pointer (used for FP4 output). void* ptrSfO; // The attention sinks pointer (additional value per head in the denominator of the softmax). float const* ptrAttentionSinks; @@ -66,10 +66,6 @@ struct KernelParams { int64_t const* ptrCustomMaskOffsets; // The debug output matrix O float* ptrDebugO; - // DSv4 inverse-RoPE + FP8 quant fusion metadata and output scale tensor, only for epilogue fusion - float const* ptrDsv4InvRopeCosSinCache; - // Dsv4 output block-scaled tensor, only for epilogue fusion - float* ptrDsv4OScaleFp32; // The first sparseMask offsets in the Kv sequence dimension. int32_t const* ptrFirstSparseMaskOffsetsKv; // The counter for the multiCtasKv mode. @@ -97,7 +93,8 @@ struct KernelParams { // The SF scale for Kv on device. Only needed by trt-llm kernels as the scales have to be on the // device currently. float const* ptrScaleSfKv; - // The scalar scale for block-scaled output O on device. + // The SF scale for O on device. Only needed by trt-llm kernels as the scales have to be on the + // device currently. float const* ptrScaleSfO; // The sequence lengths for K/V. Required by pagedKv kernels to avoid unnecessary computation // based on (ptrCumSeqLensKv[batchIdx + 1] - ptrCumSeqLensKv[batchIdx]). @@ -119,8 +116,6 @@ struct KernelParams { int32_t mBatchSize; // The chunked attention size in log2. int32_t mChunkedAttentionSizeLog2; - // Padded token dimension for the DSv4 fused FP32 scale layout. - int64_t mDsv4ScaleBufM; // The factor to add to the maximum value to increase the probability // of skip correction during next iterations. float mInflateMax; @@ -164,14 +159,14 @@ struct KernelParams { float mScaleSoftmaxLog2; // The SF scale for Kv. float mScaleSfKv; - // The scalar SF scale for output O. + // The SF scale for O. float mScaleSfO; // Threshold to decide whether warp skips softmax ops float mSkipSoftmaxThresholdScaleFactor; // The sparse attention topK value. int32_t mSparseAttnTopK; - // The start token index in the output SF tensor. Used for block-scaled output SF offset - // calculation in generation phase kernels when inflight batching is enabled in TRT-LLM. + // The start token index in SF tensor. Used for FP4 SF offset calculation in generation phase + // kernel when inflight batching is enabled in TRT-LLM. int32_t mStartTokenIdxSfO; // The sum of sequence lengths for Q and K/V. int32_t mSumOfSeqLensQ, mSumOfSeqLensKv; diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelTraits.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelTraits.h index 4a9c801a7230..7bfa8e896249 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelTraits.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/KernelTraits.h @@ -26,7 +26,6 @@ #include <trtllm/gen/Expr.h> #include <trtllm/gen/Kernel.h> #endif // TLLM_FMHA_TRTLLM_COMPAT -#include "Dsv4Constants.h" #include <nlohmann/json.hpp> #include <cassert> #include <numeric> @@ -58,45 +57,14 @@ inline int32_t getPaddedHeadDimForSmem(int32_t headDim) { // Whether the kernel uses the opt-in K-only BF16Q+FP8KV transform pipeline. template <typename FmhaOptions_> inline bool usesKOnlyTransformPipeline(FmhaOptions_ const& options) { - bool const isSupportedHeadDim = - options.mHeadDimQk == options.mHeadDimV && - (options.mHeadDimQk == 64 || options.mHeadDimQk == 128 || options.mHeadDimQk == 256); + bool const isSupportedHeadDim = options.mHeadDimQk == options.mHeadDimV && + (options.mHeadDimQk == 64 || options.mHeadDimQk == 128); return options.mEnablesBf16QFp8KvKOnlyTransform && !isContextKernel(options.mFmhaKernelType) && !options.mIsMlaGen && options.mDtypeQ != options.mDtypeK && isSupportedHeadDim && options.mDtypeQ == tg::Dtype::Bfloat16 && options.mDtypeK == tg::Dtype::E4m3 && options.mDtypeV == tg::Dtype::E4m3 && tg::isArchBlackwell(options.mCudaArch); } -// Whether the kernel is a Blackwell BF16Q+FP8KV generation kernel. -template <typename FmhaOptions_> inline bool isBf16QFp8KvGeneration(FmhaOptions_ const& options) { - return !isContextKernel(options.mFmhaKernelType) && options.mDtypeQ == tg::Dtype::Bfloat16 && - options.mDtypeK == tg::Dtype::E4m3 && options.mDtypeV == tg::Dtype::E4m3 && - tg::isArchBlackwell(options.mCudaArch); -} - -// Whether the kernel uses the full BF16Q+FP8KV transform pipeline. -template <typename FmhaOptions_> -inline bool isBf16QFp8KvFullTransformGeneration(FmhaOptions_ const& options) { - return isBf16QFp8KvGeneration(options) && !usesKOnlyTransformPipeline(options); -} - -// Whether internal builds should use the E4M3->BF16 placeholder plus SASS patch. -template <typename FmhaOptions_> -inline bool usesE4m3ToBfloat16SassPatch(FmhaOptions_ const& options) { - return tg::isArchBlackwell(options.mCudaArch) && options.mDtypeQ == tg::Dtype::Bfloat16 && - options.mDtypeK == tg::Dtype::E4m3; -} - -// Whether separate transformed K/V resources are supported for this kernel. -template <typename FmhaOptions_> -inline bool supportsSeparateTransformedKv(FmhaOptions_ const& options) { - bool const isSupportedHeadDim = - options.mHeadDimQk == options.mHeadDimV && - (options.mHeadDimQk == 64 || options.mHeadDimQk == 128 || options.mHeadDimQk == 256); - return isBf16QFp8KvFullTransformGeneration(options) && !options.mIsMlaGen && - options.mNumInstsQ == 1 && options.mNumInstsKv == 1 && isSupportedHeadDim; -} - //////////////////////////////////////////////////////////////////////////////////////////////////// struct KernelConfig : public KernelConfigBase { @@ -169,6 +137,10 @@ struct KernelConfig : public KernelConfigBase { } // The data type of softmax computation. + if (options.mEnablesFp16Softmax) { + // E4m3 kernels will also use Fp16 for softmax computation. + mDtypeSoftmax = (mDtypeQ == tg::Dtype::Bfloat16) ? tg::Dtype::Bfloat16 : tg::Dtype::Fp16; + } // The maximum headDim for K and V. mMaxHeadDimKv = std::max(mHeadDimQk, mHeadDimV); @@ -207,14 +179,8 @@ struct KernelConfig : public KernelConfigBase { // Set numStagesQ for headDim > 128 kernels. if (mNumInstsQ * mNumInstsKv == 1) { - // Whether the kernel is a generation kernel that skips softmax when possible. - bool isGenerationSkipsSoftmax = - options.mSkipsSoftmaxWhenPossible && !isContextKernel(options.mFmhaKernelType); - // Skip the check for skipsSoftmax generation kernels because this is intended. - if (!isGenerationSkipsSoftmax) { - TLLM_CHECK_INFO(mTileSizeQ == 64 || (mHeadDimQk > 128 && mHeadDimV > 128), - "Consider using numInstsQ = 2 for better performance."); - } + TLLM_CHECK_INFO(mTileSizeQ == 64 || (mHeadDimQk > 128 && mHeadDimV > 128), + "Consider using numInstsQ = 2 for better performance."); // There is no enough shared memory for 2 stages when the headDim is not split into multiple // stages. if (mHeadDimPerStageKv == 0 && keepsMmaAbForDsMlaGen) { @@ -598,11 +564,8 @@ struct MmaTraits { mAtomPvM = options.mSwapsMmaAb ? std::min(128, paddedHeadDimV) : options.mTileSizeQ; // Keep BMM2's MMA width on the logical V width when keep-AB is used. SMEM can still be padded // independently through mPaddedHeadDimV for shared K/V storage alignment. - // headDimPerStageKv is the SMEM/TMEM staging width (e.g. 128 for Blackwell context MLA) and - // may exceed headDimV (e.g. Mistral 128/64). BMM2-N must stay on the logical V width. - auto headDimPvN = options.mHeadDimPerStageKv != 0 - ? std::min(headDimPerStageKv, options.mHeadDimV) * options.mClusterDimX - : options.mHeadDimV; + auto headDimPvN = options.mHeadDimPerStageKv != 0 ? headDimPerStageKv * options.mClusterDimX + : options.mHeadDimV; // AtomPvN is limited to 256 (UTCMMA-N). mAtomPvN = options.mSwapsMmaAb ? options.mTileSizeQ : std::min(headDimPvN, 256); @@ -754,8 +717,8 @@ struct KernelTraits : public KernelConfig, public MmaTraits { mSeparateSmemKv = true; } - // Use the largest power-of-two tile up to 64 that exactly divides the correction width. - mCorrTileSize = std::gcd(mValidTilePvN, 64); + // The tile size for the correction step. + mCorrTileSize = std::min(mValidTilePvN, 64); // The number of keys per tile. mNumKeysPerTile = std::min(mNumTokensPerPage, mTileSizeKv); @@ -844,15 +807,6 @@ struct KernelTraits : public KernelConfig, public MmaTraits { // The number of transform stages for SmemTransformedKv. mNumSmemTransformStages = 2; - if (isBf16QFp8KvGeneration(options)) { - // BF16Q+FP8KV transform kernels use one conversion chunk so the SMEM budget goes to raw KV - // stages that overlap the E4M3->BF16 conversion. - mNumSmemTransformStages = 1; - } - if (options.mSeparateTransformedKv) { - // Separate transformed K/V is about producer/consumer ordering, not deeper buffering. - mNumStagesTransformedKv = 1; - } // Note: mInterleaveSfV and mUsesSharedPagedKvIdx are inherited from KernelConfigBase. @@ -1092,7 +1046,7 @@ inline KernelTraits getKernelTraitsFromOptions(FmhaOptions_ const& options) { // [...................................FullTmem.....................................................] // [.......TmemS0........][.......TmemS1........][TmemP0][TmemP1][.....TmemO0.....][.....TmemO1.....] // [TmemStat0] [TmemStat1] -// +// // If mSeparateTmemColsForSAndP and mSeparateTmemColsForSAndStats are both true: // [...................................FullTmem.....................................................] // [..TmemS0..][..TmemS1..][TmemStat0][TmemStat1][..TmemP0..][..TmemP1..][...TmemO0...][...TmemO1...] @@ -1257,9 +1211,9 @@ inline int32_t getTmemAllocationTransformedKv(KernelTraits traits) { // different rows (across the rows is only supported when tileSizeQ = 64). // clang-format off // Layout example: -// stage0: [row0, col0-127] -// stage1: [row0, col128-255] -// stage2: [row16, col0-127] +// stage0: [row0, col0-127] +// stage1: [row0, col128-255] +// stage2: [row16, col0-127] // stage3: [row16, col128-255] // clang-format on diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassPipeline.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassPipeline.h index efb0a8584ff8..59ee6d7a4461 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassPipeline.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassPipeline.h @@ -527,10 +527,14 @@ template <int SequenceDepth, int SequenceLength> class CutlassOrderedSequenceBar } // Signal completion of stage and move to the next stage - inline __device__ void arrive() { mPipeline.arrive(); } + inline __device__ void arrive() { + mPipeline.arrive(); + } // Wait on a stage to be unlocked - inline __device__ void wait() { mPipeline.wait(); } + inline __device__ void wait() { + mPipeline.wait(); + } private: Params mParams; @@ -1019,7 +1023,9 @@ class CutlassTmaAsyncPipeline { } // Get pipeline - [[nodiscard]] inline __device__ Pipeline& get_pipeline() { return mPipeline; } + [[nodiscard]] inline __device__ Pipeline& get_pipeline() { + return mPipeline; + } private: // The pipeline. @@ -1270,7 +1276,9 @@ class CutlassTmaMultiUmmaAsyncPipeline { } // Get pipeline - [[nodiscard]] inline __device__ Pipeline& get_pipeline() { return mPipeline; } + [[nodiscard]] inline __device__ Pipeline& get_pipeline() { + return mPipeline; + } private: // The pipeline. diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm100Pipeline.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm100Pipeline.h index c6ea2b145233..b5dfd71b5967 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm100Pipeline.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm100Pipeline.h @@ -22,7 +22,6 @@ #include "CutlassSm90Pipeline.h" #include <cutlass/pipeline/sm100_pipeline.hpp> - //////////////////////////////////////////////////////////////////////////////////////////////////// namespace trtllm::dev { @@ -408,12 +407,16 @@ class PipelineTmaTransformAsync { } CUTLASS_DEVICE - void producer_commit(PipelineState state, uint32_t bytes) { impl_.producer_commit(state, bytes); } + void producer_commit(PipelineState state, uint32_t bytes) { + impl_.producer_commit(state, bytes); + } // Prevents early exit of producer blocks in Cluster. // This should be called once before kernel exits. CUTLASS_DEVICE - void producer_tail(PipelineState state) { impl_.producer_tail(state); } + void producer_tail(PipelineState state) { + impl_.producer_tail(state); + } CUTLASS_DEVICE ProducerBarrierType* producer_get_barrier(PipelineState state) { @@ -434,7 +437,9 @@ class PipelineTmaTransformAsync { } CUTLASS_DEVICE - void consumer_wait(PipelineState state) { impl_.consumer_wait(state); } + void consumer_wait(PipelineState state) { + impl_.consumer_wait(state); + } CUTLASS_DEVICE void consumer_wait(PipelineState state, cutlass::ConsumerToken barrier_token) { diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm100TileScheduler.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm100TileScheduler.h index f096893b19a0..a2ac23477b9d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm100TileScheduler.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm100TileScheduler.h @@ -31,7 +31,6 @@ #include "cutlass/conv/convnd_problem_shape.hpp" #include "cutlass/conv/detail.hpp" - //////////////////////////////////////////////////////////////////////////////////////////////////// namespace trtllm::dev { @@ -338,7 +337,9 @@ class PersistentTileSchedulerSm100 { cuda_adapter); } - static bool can_implement(Arguments const& args) { return true; } + static bool can_implement(Arguments const& args) { + return true; + } // // Constructors @@ -524,10 +525,14 @@ class PersistentTileSchedulerSm100 { // Returns whether the block assigned this work should compute the epilogue for the corresponding // output tile. For the basic tile scheduler, this is always true. CUTLASS_HOST_DEVICE - static bool compute_epilogue(WorkTileInfo const&, Params const&) { return true; } + static bool compute_epilogue(WorkTileInfo const&, Params const&) { + return true; + } CUTLASS_HOST_DEVICE - static bool compute_epilogue(WorkTileInfo const&) { return true; } + static bool compute_epilogue(WorkTileInfo const&) { + return true; + } // Returns whether fixup is needed for `work_tile_info`. None of the work units returned by // this scheduler require fixup, since none of the work units partition the reduction extent. @@ -562,7 +567,9 @@ class PersistentTileSchedulerSm100 { // this scheduler only schedules work in units of single, full output tiles, the WorkTileInfo // passed in should not be used after having been processed. CUTLASS_DEVICE - static bool continue_current_work(WorkTileInfo&) { return false; } + static bool continue_current_work(WorkTileInfo&) { + return false; + } // // Implementation Helpers @@ -620,17 +627,25 @@ class PersistentTileSchedulerSm100 { } CUTLASS_DEVICE - static CLCResponse make_invalid_response() { return CLCResponse{}; } + static CLCResponse make_invalid_response() { + return CLCResponse{}; + } // Set data SMEM ptr CUTLASS_DEVICE - void set_data_ptr(CLCResponse* clc_response_ptr) { clc_response_ptr_ = clc_response_ptr; } + void set_data_ptr(CLCResponse* clc_response_ptr) { + clc_response_ptr_ = clc_response_ptr; + } CUTLASS_DEVICE - static bool valid_warpgroup_in_work_tile(WorkTileInfo const& work_tile_info) { return true; } + static bool valid_warpgroup_in_work_tile(WorkTileInfo const& work_tile_info) { + return true; + } CUTLASS_DEVICE - static bool requires_separate_reduction(Params const& params) { return false; } + static bool requires_separate_reduction(Params const& params) { + return false; + } template <class FrgTensorC> CUTLASS_DEVICE static void fixup(Params const&, @@ -823,7 +838,7 @@ class PersistentTileSchedulerSm100 { // Data Members // CLCResponse* clc_response_ptr_ = nullptr; - Params params_; + Params const& params_; dim3 block_id_in_cluster_ = {0, 0, 0}; }; diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm90Pipeline.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm90Pipeline.h index bc2bd63f18d5..9339e488aec5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm90Pipeline.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassSm90Pipeline.h @@ -30,7 +30,6 @@ #include "cutlass/detail/dependent_false.hpp" #include <cutlass/pipeline/sm90_pipeline.hpp> - //////////////////////////////////////////////////////////////////////////////////////////////////// namespace trtllm::dev { @@ -180,7 +179,9 @@ template <int Stages_> class PipelineTmaAsync { } CUTLASS_DEVICE - void producer_acquire(PipelineState state) { producer_acquire(state.index(), state.phase()); } + void producer_acquire(PipelineState state) { + producer_acquire(state.index(), state.phase()); + } CUTLASS_DEVICE void producer_acquire(PipelineState state, cutlass::ProducerToken barrier_token) { @@ -234,7 +235,9 @@ template <int Stages_> class PipelineTmaAsync { } CUTLASS_DEVICE - void consumer_wait(PipelineState state) { consumer_wait(state.index(), state.phase()); } + void consumer_wait(PipelineState state) { + consumer_wait(state.index(), state.phase()); + } CUTLASS_DEVICE void consumer_wait(PipelineState state, cutlass::ConsumerToken barrier_token) { @@ -242,7 +245,9 @@ template <int Stages_> class PipelineTmaAsync { } CUTLASS_DEVICE - void consumer_release(PipelineState state) { consumer_release(state.index()); } + void consumer_release(PipelineState state) { + consumer_release(state.index()); + } private: uint32_t dst_blockid_ = 0; @@ -707,7 +712,9 @@ template <int SequenceDepth_, int SequenceLength_> class OrderedSequenceBarrier // Wait on a stage to be unlocked CUTLASS_DEVICE - void wait() { get_barrier_for_current_stage(params_.group_id).wait(stage_.phase()); } + void wait() { + get_barrier_for_current_stage(params_.group_id).wait(stage_.phase()); + } // Signal completion of Stage and move to the next stage // (group_id) signals to (group_id+1) @@ -719,7 +726,9 @@ template <int SequenceDepth_, int SequenceLength_> class OrderedSequenceBarrier } CUTLASS_DEVICE - void advance() { ++stage_; } + void advance() { + ++stage_; + } private: CUTLASS_DEVICE diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassUtils.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassUtils.h index 436e571e62d3..eab9fdb2a6f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassUtils.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/CutlassUtils.h @@ -20,8 +20,6 @@ #include <cute/tensor.hpp> #include <cutlass/numeric_conversion.h> -#include <cstdint> - namespace trtllm { namespace dev { @@ -183,33 +181,6 @@ inline __device__ cutlass::uint128_t convertE4m3ToBfloat16(uint64_t src) { //////////////////////////////////////////////////////////////////////////////////////////////////// -inline __device__ cutlass::uint128_t convertE4m3ToBfloat16WithSassPatch(uint64_t src) { -#ifndef TLLM_PUBLIC_RELEASE - cutlass::uint128_t dst; - auto srcB16 = reinterpret_cast<uint16_t const*>(&src); - auto srcB32 = reinterpret_cast<uint32_t const*>(&src); - auto dstB32 = reinterpret_cast<uint32_t*>(&dst); - -#pragma unroll - for (int ii = 0; ii < 4; ++ii) { - uint32_t srcPair = srcB16[ii]; - // The second source keeps the placeholder instruction shape stable for the SASS patcher; the - // patched direct unpack uses only srcPair. - uint32_t dummy = srcB32[(ii >> 1) ^ 1]; - - // Placeholder for SASS patching. Replaced with: - // F2FP.BF16.E4M3.UNPACK_B dst, srcPair, 4.5736980577097704378e-41.H0 - asm volatile("lop3.b32 %0, %1, %1, %2, 0x77;" : "=r"(dstB32[ii]) : "r"(srcPair), "r"(dummy)); - } - - return dst; -#else - return convertE4m3ToBfloat16(src); -#endif // TLLM_PUBLIC_RELEASE -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - inline __device__ cutlass::uint128_t convertE4m3ToFp16(uint64_t src) { auto srcArray{castToArray<cutlass::float_e4m3_t, 8>(src)}; return castFromArray<cutlass::uint128_t>(castArray<cutlass::half_t>(srcArray)); diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Fp4Utils.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Fp4Utils.h index ba3263115443..00eaa16dddae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Fp4Utils.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Fp4Utils.h @@ -79,10 +79,10 @@ inline __device__ cute::uint128_t e2m1ToFp16(uint32_t src) { //////////////////////////////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////////////////////////////// -inline __device__ void computeNvFp4SfAndOutputScale(float& outputScale, - cutlass::float_e4m3_t& sfOut, - float const& amax, - float const& sfScale) { +inline __device__ void computeSfAndOutputScale(float& outputScale, + cutlass::float_e4m3_t& sfOut, + float const& amax, + float const& sfScale) { // The reciprocal of E2M1_MAX. float constexpr reciprocalOfE2m1Max = 1.f / 6.f; // The FP32 sf. @@ -100,11 +100,11 @@ inline __device__ void computeNvFp4SfAndOutputScale(float& outputScale, //////////////////////////////////////////////////////////////////////////////////////////////////// -inline __device__ void computeNvFp4SfAndOutputScale(float& outputScale, - cutlass::float_e4m3_t& sfOut, - float const& amax, - float const& sfScale, - float const& sfScaleInv) { +inline __device__ void computeSfAndOutputScale(float& outputScale, + cutlass::float_e4m3_t& sfOut, + float const& amax, + float const& sfScale, + float const& sfScaleInv) { // The reciprocal of E2M1_MAX. float constexpr reciprocalOfE2m1Max = 1.f / 6.f; // The FP32 sf. @@ -126,7 +126,7 @@ inline __device__ void computeNvFp4SfAndOutputScale(float& outputScale, template <int NumEltsPerThread, typename OutT> inline __device__ void convertFp16ToE2m1(OutT& out, cutlass::float_e4m3_t& sfOut, - cutlass::half_t const (&in)[NumEltsPerThread], + uint32_t const (&in)[NumEltsPerThread / 2], float sfScale) { // This function converts fp16 elements to E2m1. @@ -165,7 +165,7 @@ inline __device__ void convertFp16ToE2m1(OutT& out, // Compute the sf and output scale. float outputScale; - computeNvFp4SfAndOutputScale(outputScale, sfOut, vecAmax, sfScale); + computeSfAndOutputScale(outputScale, sfOut, vecAmax, sfScale); // Apply the output scale. cutlass::Array<float, NumEltsPerThread> scaled; @@ -204,30 +204,27 @@ inline __device__ void convertFloatToE2m1(OutT& out, // The number of threads per vector of 16 elements. int32_t constexpr NumThreadsPerVec = 16 / NumEltsPerThread; - // Find the local amax. + // Find the loacl amax. float localAmax = 0; #pragma unroll - for (int ii = 0; ii < NumEltsPerThread; ii++) { + for (int ii = 0; ii < 16; ii++) { localAmax = fmaxf(localAmax, fabsf(in[ii])); } // Get the absolute maximum among all 16 values. if constexpr (NumThreadsPerVec > 1) { - static_assert(NumThreadsPerVec == 2 || NumThreadsPerVec == 4, "Not supported."); -#pragma unroll - for (int32_t step = 1; step < NumThreadsPerVec; step *= 2) { - localAmax = fmaxf(__shfl_xor_sync(uint32_t(-1), localAmax, step), localAmax); - } + static_assert(NumThreadsPerVec == 2, "Not supported."); + localAmax = fmaxf(__shfl_xor_sync(uint32_t(-1), localAmax, 1), localAmax); } // Compute the sf and output scale. float outputScale; - computeNvFp4SfAndOutputScale(outputScale, sfOut, localAmax, sfScale); + computeSfAndOutputScale(outputScale, sfOut, localAmax, sfScale); // Apply the output scale. cutlass::Array<float, NumEltsPerThread> scaled; #pragma unroll - for (int ii = 0; ii < NumEltsPerThread; ii++) { + for (int ii = 0; ii < 16; ii++) { scaled[ii] = in[ii] * outputScale; } diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Fp8Utils.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Fp8Utils.h deleted file mode 100644 index 1c54dd0a0467..000000000000 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Fp8Utils.h +++ /dev/null @@ -1,278 +0,0 @@ -/* - * Copyright (c) 2011-2026, NVIDIA CORPORATION. All rights reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#pragma once - -#include "CutlassUtils.h" - -namespace trtllm { -namespace dev { - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -inline __device__ void computeMxE4m3SfAndOutputScale(float& outputScale, - cutlass::float_ue8m0_t& sfOut, - float amax, - float const& sfScale) { - float const amaxPow2 = trunc_abs_float_to_pow2(amax); - float const sfVal = amaxPow2 * (1.f / 256.f) * sfScale; - cutlass::Array<float, 1> sfArrayFp32; - sfArrayFp32[0] = sfVal; - sfOut = castArray<cutlass::float_ue8m0_t>(sfArrayFp32)[0]; - outputScale = sfVal != 0.f ? scale_rcp_exp_only(sfVal) : 0.f; -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -inline __device__ void computeMxE4m3SfAndOutputScale(float& outputScale, - cutlass::float_ue8m0_t& sfOut, - float amax, - float const& sfScale, - float const& sfScaleInv) { - float const amaxPow2 = trunc_abs_float_to_pow2(amax); - float const sfVal = amaxPow2 * (1.f / 256.f) * sfScale; - cutlass::Array<float, 1> sfArrayFp32; - sfArrayFp32[0] = sfVal; - sfOut = castArray<cutlass::float_ue8m0_t>(sfArrayFp32)[0]; - outputScale = sfVal != 0.f ? scale_rcp_exp_only(sfVal * sfScaleInv) : 0.f; -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -template <int32_t NumEltsPerThread, typename OutT> -inline __device__ void convertFloatToMxE4m3(OutT& out, - cutlass::float_ue8m0_t& sfOut, - float const (&input)[NumEltsPerThread], - float sfScale) { - // MxE4m3 uses one UE8M0 scale for each group of 32 E4M3 elements. - int32_t constexpr NumEltsPerSf = 32; - int32_t constexpr NumThreadsPerVec = NumEltsPerSf / NumEltsPerThread; - static_assert(NumEltsPerSf % NumEltsPerThread == 0 && NumEltsPerThread % 4 == 0, - "NumEltsPerThread not supported."); - static_assert(sizeof(OutT) == NumEltsPerThread, - "Output type not supported."); // 1 byte per element. - - float localAmax = 0.f; -#pragma unroll - for (int32_t i = 0; i < NumEltsPerThread; ++i) { - localAmax = fmaxf(localAmax, fabsf(input[i])); - } - -#pragma unroll - for (int32_t step = 1; step < NumThreadsPerVec; step *= 2) { - localAmax = fmaxf(__shfl_xor_sync(uint32_t(-1), localAmax, step), localAmax); - } - - float outputScale; - computeMxE4m3SfAndOutputScale(outputScale, sfOut, localAmax, sfScale); - - cutlass::Array<float, NumEltsPerThread> scaled; -#pragma unroll - for (int32_t i = 0; i < NumEltsPerThread; ++i) { - scaled[i] = input[i] * outputScale; - } - - using OutVec = cutlass::Array<cutlass::float_e4m3_t, NumEltsPerThread>; - reinterpret_cast<OutVec&>(out) = castArray<cutlass::float_e4m3_t>(scaled); -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -template <int32_t NumEltsPerThread, typename OutT> -inline __device__ void convertFp16ToMxE4m3(OutT& out, - cutlass::float_ue8m0_t& sfOut, - cutlass::half_t const (&in)[NumEltsPerThread], - float sfScale) { - // MxE4m3 uses one UE8M0 scale for each group of 32 E4M3 elements. - int32_t constexpr NumEltsPerSf = 32; - int32_t constexpr NumThreadsPerVec = NumEltsPerSf / NumEltsPerThread; - static_assert(NumEltsPerSf % NumEltsPerThread == 0 && NumEltsPerThread % 4 == 0, - "NumEltsPerThread not supported."); - static_assert(sizeof(OutT) == NumEltsPerThread, - "Output type not supported."); // 1 byte per element. - - auto inH2 = reinterpret_cast<half2 const*>(&in[0]); - auto localAmaxH2 = __habs2(inH2[0]); -#pragma unroll - for (int32_t i = 0; i < NumEltsPerThread / 2; ++i) { - localAmaxH2 = __hmax2(localAmaxH2, __habs2(inH2[i])); - } - - // Perform warp-level reduction to achieve the amax of the vector of 16 elements. - if constexpr (NumThreadsPerVec > 1) { - static_assert(NumThreadsPerVec == 2 || NumThreadsPerVec == 4, "Not supported."); - for (int32_t step = 1; step < NumThreadsPerVec; step *= 2) { - localAmaxH2 = __hmax2(__shfl_xor_sync(uint32_t(-1), localAmaxH2, step), localAmaxH2); - } - } - - float localAmax = float(__hmax(localAmaxH2.x, localAmaxH2.y)); - float outputScale; - computeMxE4m3SfAndOutputScale(outputScale, sfOut, localAmax, sfScale); - - cutlass::Array<float, NumEltsPerThread> scaled; -#pragma unroll - for (int32_t i = 0; i < NumEltsPerThread / 2; ++i) { - float2 tmp = __half22float2(inH2[i]); - scaled[i * 2 + 0] = tmp.x * outputScale; - scaled[i * 2 + 1] = tmp.y * outputScale; - } - - using OutVec = cutlass::Array<cutlass::float_e4m3_t, NumEltsPerThread>; - reinterpret_cast<OutVec&>(out) = castArray<cutlass::float_e4m3_t>(scaled); -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -template <int32_t NumReduceVals, int32_t NumVals> -__device__ __forceinline__ float reduceMaxAbs(cutlass::Array<float, NumVals> const& vals, - float init = 1.0e-12f) { - static_assert(NumReduceVals % 4 == 0, "Number of reduced values must be divisible by 4."); - static_assert(NumReduceVals <= NumVals, "Reduced value range exceeds value array."); - - float max0 = init; - float max1 = init; - float max2 = init; - float max3 = init; -#pragma unroll - for (int32_t i = 0; i < NumReduceVals / 4; ++i) { - int32_t const ii = i * 4; - max0 = fmaxf(max0, fabsf(vals[ii + 0])); - max1 = fmaxf(max1, fabsf(vals[ii + 1])); - max2 = fmaxf(max2, fabsf(vals[ii + 2])); - max3 = fmaxf(max3, fabsf(vals[ii + 3])); - } - return fmaxf(fmaxf(max0, max1), fmaxf(max2, max3)); -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -// Write the FP32 dequant scale for one output block and pack the already-finalized values to E4M3. -// The stored scale is amax / 448, so dequantization is `fp8_value * scalePtr[scaleOffset]`. -template <int32_t NumVals, int32_t NumPackedRegs, typename OutRegs> -inline __device__ void e4m3PackEpilogue(OutRegs& out, - cutlass::Array<float, NumVals> const& vals, - float* scalePtr, - int64_t scaleOffset, - float amax) { - static_assert(NumVals == NumPackedRegs * 4, "One packed register stores four E4M3 values."); - - // E4M3's largest finite value is 448; store scale = amax / 448 and quantize with 448 / amax. - float constexpr fp8Max = 448.f; - float constexpr fp8MaxRcp = 1.f / fp8Max; - float const outScale = amax * fp8MaxRcp; - float const invScale = __fdividef(fp8Max, amax); - scalePtr[scaleOffset] = outScale; - - // Keep these multiplies scalar: fmul2 introduces extra array temporaries here and increases - // register spills in this epilogue. -#pragma unroll - for (int32_t regIdx = 0; regIdx < NumPackedRegs; ++regIdx) { - int32_t const ii = regIdx * 4; - out[regIdx] = convert_float4_to_e4m3(vals[ii + 0] * invScale, - vals[ii + 1] * invScale, - vals[ii + 2] * invScale, - vals[ii + 3] * invScale); - } -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -// Quantize one output block to packed E4M3 and write its FP32 dequant scale. The input values are -// expected to already be in the final output layout. -template <int32_t NumVals, int32_t NumPackedRegs, typename OutRegs> -inline __device__ void e4m3QuantEpilogue(OutRegs& out, - cutlass::Array<float, NumVals> const& vals, - float* scalePtr, - int64_t scaleOffset) { - static_assert(NumVals == NumPackedRegs * 4, "One packed register stores four E4M3 values."); - - float const amax = reduceMaxAbs<NumVals>(vals); - e4m3PackEpilogue<NumVals, NumPackedRegs>(out, vals, scalePtr, scaleOffset, amax); -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -// Apply the DSv4 inverse-RoPE transform to the final 64 dimensions of a 512-dim head when this -// 1x128 block covers that range, then quantize the block to packed E4M3 and write its FP32 dequant -// scale. Blocks outside the inverse-RoPE range follow the same path as e4m3QuantEpilogue. -template <int32_t NumVals, int32_t NumPackedRegs, typename OutRegs> -inline __device__ void dsv4InvRopeFp8QuantEpilogue(OutRegs& out, - cutlass::Array<float, NumVals>& vals, - float* scalePtr, - float const* cosSinCache, - int64_t scaleOffset, - int32_t position, - int32_t headDimOffset) { - static_assert(NumVals == NumPackedRegs * 4, "One packed register stores four E4M3 values."); - static_assert(NumVals == 128, "DSv4 fused epilogue processes one 1x128 quant group."); - - // DSv4 inverse-RoPE applies to the last 64 dimensions, [448, 512), of a 512-dim head. The - // helper processes one 128-value quant block, so the RoPE block starts at 384 and the RoPE - // values begin at offset 64 inside that block. DSv4 uses non-NeoX interleaved RoPE: adjacent - // element pairs share one cos/sin value. Each cos/sin cache row is laid out as - // [cos(32), sin(32)]. - int32_t constexpr ropeStart = 448; - int32_t constexpr ropeHalf = 32; - int32_t constexpr ropeBlockStart = ropeStart - ropeHalf * 2; - int32_t constexpr ropeOffset = ropeStart - ropeBlockStart; - int32_t constexpr cosSinStride = ropeHalf * 2; - - // TODO: use headDimOffset as template parameter to drop warp divergence. - if (headDimOffset != ropeBlockStart) { - e4m3QuantEpilogue<NumVals, NumPackedRegs>(out, vals, scalePtr, scaleOffset); - return; - } - - float const* csRow = cosSinCache + position * cosSinStride; - - static_assert(ropeOffset % 4 == 0, "The RoPE offset must be aligned to packed E4M3 registers."); - int32_t constexpr ropePackedRegStart = ropeOffset / 4; - float amax = reduceMaxAbs<ropeOffset>(vals); - - // Match the standalone TRT-LLM inverse-RoPE FP8 kernel's per-4-value structure: each packed - // register in the RoPE half contains two interleaved RoPE pairs. -#pragma unroll - for (int32_t regIdx = ropePackedRegStart; regIdx < NumPackedRegs; ++regIdx) { - int32_t const ii = regIdx * 4; - int32_t const csIdx = (regIdx - ropePackedRegStart) * 2; - // TODO: use float2 for cos, sin pair. Currently do not use because register spills. - float const cos0 = csRow[csIdx + 0]; - float const sin0 = csRow[ropeHalf + csIdx + 0]; - float const first0 = vals[ii + 0]; - float const second0 = vals[ii + 1]; - float const rotatedFirst0 = first0 * cos0 + second0 * sin0; - float const rotatedSecond0 = second0 * cos0 - first0 * sin0; - vals[ii + 0] = rotatedFirst0; - vals[ii + 1] = rotatedSecond0; - amax = fmaxf(amax, fmaxf(fabsf(rotatedFirst0), fabsf(rotatedSecond0))); - - float const cos1 = csRow[csIdx + 1]; - float const sin1 = csRow[ropeHalf + csIdx + 1]; - float const first1 = vals[ii + 2]; - float const second1 = vals[ii + 3]; - float const rotatedFirst1 = first1 * cos1 + second1 * sin1; - float const rotatedSecond1 = second1 * cos1 - first1 * sin1; - vals[ii + 2] = rotatedFirst1; - vals[ii + 3] = rotatedSecond1; - amax = fmaxf(amax, fmaxf(fabsf(rotatedFirst1), fabsf(rotatedSecond1))); - } - - e4m3PackEpilogue<NumVals, NumPackedRegs>(out, vals, scalePtr, scaleOffset, amax); -} - -} // namespace dev -} // namespace trtllm diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/ReduceMultiCtasKv.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/ReduceMultiCtasKv.h index aecb87412373..18042c7d2903 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/ReduceMultiCtasKv.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/ReduceMultiCtasKv.h @@ -22,7 +22,6 @@ #include <cuda/cmath> #include <cuda_bf16.h> #include <cuda_fp16.h> -#include <type_traits> #include "CutlassPipeline.h" #include "CutlassBarrier.h" #include "CutlassUtils.h" @@ -301,8 +300,7 @@ template <int32_t TileSizePerCtaQ, bool IsE4m3Bmm, bool UsesCgaReduction, typename DtypeO, - typename DtypePartialO, - typename DtypeSfO = cutlass::float_e4m3_t> + typename DtypePartialO> inline __device__ void reducePartialO(DtypeO* oPtr, DtypePartialO const* partialOPtr, float const* partialStatsPtr, @@ -317,7 +315,7 @@ inline __device__ void reducePartialO(DtypeO* oPtr, trtllm::dev::fast_mod_div numHeadsQPerKvDivisor, int32_t numValidRows, bool storesSoftmaxStats, - DtypeSfO* oSfPtr = nullptr, + cutlass::float_e4m3_t* oSfPtr = nullptr, float sfScale = 0.f, int32_t sfBaseRowIdx = 0) { @@ -488,14 +486,11 @@ inline __device__ void reducePartialO(DtypeO* oPtr, } // Convert the float values to DtypeO, and Store it to global memory. - constexpr bool IsMxE4m3Output = std::is_same_v<DtypeO, cutlass::float_e4m3_t> && - std::is_same_v<DtypeSfO, cutlass::float_ue8m0_t>; - constexpr bool IsE2m1Output = std::is_same_v<DtypeO, cutlass::float_e2m1_t>; - if constexpr (IsMxE4m3Output || IsE2m1Output) { - // The number of output elements packed in a byte. - int32_t constexpr NumEltsPerDstByte = IsE2m1Output ? 2 : 1; + if constexpr (std::is_same_v<DtypeO, cutlass::float_e2m1_t>) { + // The number of E2m1 elements packed in a byte. + int32_t constexpr NumE2m1EltsPerByte = 2; // The number of elements per sf. - int32_t constexpr NumEltsPerSf = IsE2m1Output ? 16 : 32; + int32_t constexpr NumEltsPerSf = 16; // The number of cols of SF per block. int32_t constexpr NumColsPerSfBlock = 4; // The size of each SF block. @@ -523,9 +518,8 @@ inline __device__ void reducePartialO(DtypeO* oPtr, storeGmemSfOffset = sfColIdx / NumColsPerSfBlock * NumBytesPerSfBlock + sfColIdx % NumColsPerSfBlock; } - convertAndStoreToGmem<DtypeO>( - reinterpret_cast<char*>(oPtr + gmemStoreOffset / NumEltsPerDstByte), + reinterpret_cast<char*>(oPtr + gmemStoreOffset / NumE2m1EltsPerByte), reinterpret_cast<char*>(oSfPtr) + storeGmemSfOffset, outputVals, sfScale, @@ -549,8 +543,7 @@ template <int32_t TileSizePerCtaQ, bool UsesCgaReduction, typename DtypeO, typename DtypePartialO, - typename Barrier, - typename DtypeSfO = cutlass::float_e4m3_t> + typename Barrier> inline __device__ void reducePartialO(DtypeO* oPtr, DtypePartialO const* partialOPtr, float const* partialStatsPtr, @@ -567,7 +560,7 @@ inline __device__ void reducePartialO(DtypeO* oPtr, trtllm::dev::fast_mod_div numHeadsQPerKvDivisor, int32_t numValidRows, bool storesSoftmaxStats, - DtypeSfO* oSfPtr = nullptr, + cutlass::float_e4m3_t* oSfPtr = nullptr, float sfScale = 0.f, int32_t sfBaseRowIdx = 0) { diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/ReduceMultiCtasKvUtils.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/ReduceMultiCtasKvUtils.h index 08b9c082e7ae..3574d30131a4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/ReduceMultiCtasKvUtils.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/ReduceMultiCtasKvUtils.h @@ -22,8 +22,6 @@ #include "CutlassUtils.h" #include <float.h> #include "Fp4Utils.h" -#include "Fp8Utils.h" -#include <type_traits> namespace trtllm { namespace dev { @@ -35,6 +33,15 @@ inline __device__ void convertAndStoreToGmem(char* gmemPtr, float (&input)[NumEl static_assert(sizeof(Dtype) == 0, "Not implemented."); } +template <typename Dtype, int32_t NumElts> +inline __device__ void convertAndStoreToGmem(char* gmemPtr, + char* oSfPtr, + float (&input)[NumElts], + float sfScale, + bool isValidRow) { + static_assert(sizeof(Dtype) == 0, "Not implemented."); +} + template <> inline __device__ void convertAndStoreToGmem<cutlass::half_t, 8>(char* gmemPtr, float (&input)[8]) { uint4 output; @@ -65,67 +72,23 @@ inline __device__ void convertAndStoreToGmem<cutlass::float_e4m3_t, 8>(char* gme *reinterpret_cast<uint2*>(gmemPtr) = output; } -//////////////////////////////////////////////////////////////////////////////////////////////////// - -// Block scaled output store helpers. -template <typename DtypeOut, typename DtypeIn, int32_t NumElts> -inline __device__ std::enable_if_t<std::is_same_v<DtypeOut, cutlass::float_e2m1_t>> -convertAndStoreToGmem(char* gmemPtr, - char* gmemSfPtr, - DtypeIn (&input)[NumElts], - float sfScale, - bool isValidRow) { - constexpr int NumThreadsPerVec = 16 / NumElts; - static_assert(NumThreadsPerVec == 1 || NumThreadsPerVec == 2 || NumThreadsPerVec == 4 || - NumThreadsPerVec == 8, - "NumElts Not supported."); - static_assert(std::is_same_v<DtypeIn, cutlass::half_t> || std::is_same_v<DtypeIn, float>, - "DtypeIn not supported."); +template <> +inline __device__ void convertAndStoreToGmem<cutlass::float_e2m1_t, 8>(char* gmemPtr, + char* gmemSfPtr, + float (&input)[8], + float sfScale, + bool isValidRow) { cutlass::float_e4m3_t sfOut; - using OutVec = cutlass::Array<cutlass::float_e2m1_t, NumElts>; - OutVec valOut; - if constexpr (std::is_same_v<DtypeIn, cutlass::half_t>) { - convertFp16ToE2m1<NumElts>(valOut, sfOut, input, sfScale); - } else { - convertFloatToE2m1<NumElts>(valOut, sfOut, input, sfScale); - } + uint32_t valOut; + convertFloatToE2m1<8>(valOut, sfOut, input, sfScale); if (isValidRow) { - *reinterpret_cast<OutVec*>(gmemPtr) = valOut; - if (threadIdx.x % NumThreadsPerVec == 0) { // store scale for every 16 elements (4 columns) + *reinterpret_cast<uint32_t*>(gmemPtr) = valOut; + if (threadIdx.x % 2 == 0) { *reinterpret_cast<cutlass::float_e4m3_t*>(gmemSfPtr) = sfOut; } } } -template <typename DtypeOut, typename DtypeIn, int32_t NumElts> -inline __device__ std::enable_if_t<std::is_same_v<DtypeOut, cutlass::float_e4m3_t>> -convertAndStoreToGmem(char* gmemPtr, - char* gmemSfPtr, - DtypeIn (&input)[NumElts], - float sfScale, - bool isValidRow) { - constexpr int NumThreadsPerVec = 32 / NumElts; - static_assert(NumThreadsPerVec == 1 || NumThreadsPerVec == 2 || NumThreadsPerVec == 4 || - NumThreadsPerVec == 8, - "NumElts Not supported."); - static_assert(std::is_same_v<DtypeIn, cutlass::half_t> || std::is_same_v<DtypeIn, float>, - "DtypeIn not supported."); - cutlass::float_ue8m0_t sfOut; - using OutVec = cutlass::Array<cutlass::float_e4m3_t, NumElts>; - OutVec valOut; - if constexpr (std::is_same_v<DtypeIn, cutlass::half_t>) { - convertFp16ToMxE4m3<NumElts>(valOut, sfOut, input, sfScale); - } else { - convertFloatToMxE4m3<NumElts>(valOut, sfOut, input, sfScale); - } - if (isValidRow) { - *reinterpret_cast<OutVec*>(gmemPtr) = valOut; - if (threadIdx.x % NumThreadsPerVec == 0) { // store scale for every 32 elements (4 columns) - *reinterpret_cast<cutlass::float_ue8m0_t*>(gmemSfPtr) = sfOut; - } - } -} - //////////////////////////////////////////////////////////////////////////////////////////////////// template <typename Dtype, int32_t NumElts> diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/StoreGmemO.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/StoreGmemO.h index 5b6ba596f702..5da3f02de5ca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/StoreGmemO.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/StoreGmemO.h @@ -22,30 +22,70 @@ #include "FastMath.h" #include "StoreSmemP.h" #include "Fp4Utils.h" -#include "Fp8Utils.h" -#include "ReduceMultiCtasKvUtils.h" #include <cuda_bf16.h> -#include <type_traits> namespace trtllm { namespace dev { //////////////////////////////////////////////////////////////////////////////////////////////////// -template <typename DtypeOut, int32_t NumRegs> -inline __device__ void copyFromSmemToGmemAndConvertToBlockScaled(char* src, - char* dst, - char* dstSf, - float sfScale, - bool isValidRow) { - static_assert(std::is_same_v<DtypeOut, cutlass::float_e2m1_t> || - std::is_same_v<DtypeOut, cutlass::float_e4m3_t>, - "DtypeOut Not implemented."); - static_assert(NumRegs == 2 || NumRegs == 4, "Only 2 or 4 registers are allowed."); +template <int32_t NumRegs> +inline __device__ void copyFromSmemToGmemAndConvertToE2m1(char const* src, + char* dst, + char* dstSf, + float sfScale, + bool isValidRow) { + static_assert(false, "Not implemented."); +} + +//////////////////////////////////////////////////////////////////////////////////////////////////// + +template <> +inline __device__ void copyFromSmemToGmemAndConvertToE2m1<2>(char const* src, + char* dst, + char* dstSf, + float sfScale, + bool isValidRow) { // Load from SMEM. - cutlass::half_t(&vals)[NumRegs * 2] = reinterpret_cast<cutlass::half_t(&)[NumRegs * 2]>(src[0]); + auto in = reinterpret_cast<uint64_t const*>(src)[0]; + + // Convert to E2m1. + cutlass::float_e4m3_t sfOut; + uint16_t valOut; + convertFp16ToE2m1<4>(valOut, sfOut, reinterpret_cast<uint32_t(&)[2]>(in), sfScale); + // Each group of 4 threads maps to the same SF. + if (isValidRow) { + // Store the output to GMEM. Each group of 4 threads maps to the same SF. + reinterpret_cast<uint16_t*>(dst)[0] = valOut; + if (threadIdx.x % 4 == 0) { + reinterpret_cast<cutlass::float_e4m3_t*>(dstSf)[0] = sfOut; + } + } +} + +//////////////////////////////////////////////////////////////////////////////////////////////////// - convertAndStoreToGmem<DtypeOut>(dst, dstSf, vals, sfScale, isValidRow); +template <> +inline __device__ void copyFromSmemToGmemAndConvertToE2m1<4>(char const* src, + char* dst, + char* dstSf, + float sfScale, + bool isValidRow) { + // Load from SMEM. + auto in = reinterpret_cast<cutlass::uint128_t const*>(src)[0]; + + // Convert to E2m1. + cutlass::float_e4m3_t sfOut; + uint32_t valOut; + convertFp16ToE2m1<8>(valOut, sfOut, reinterpret_cast<uint32_t(&)[4]>(in), sfScale); + // Each pair of threads maps to the same SF. + if (isValidRow) { + // Store the output to GMEM. Each pair of threads maps to the same SF. + reinterpret_cast<uint32_t*>(dst)[0] = valOut; + if (threadIdx.x % 2 == 0) { + reinterpret_cast<cutlass::float_e4m3_t*>(dstSf)[0] = sfOut; + } + } } //////////////////////////////////////////////////////////////////////////////////////////////////// @@ -101,7 +141,6 @@ template <int32_t NumRegs, int32_t RowStrideSf, int32_t NumSfPerHead, bool CastToE2m1, - bool CastToMxE4m3, bool StoreToRemoteSmem, bool MapRowToHeadTokenIdx> inline __device__ void copyFromSmemToDstMem(char* smemPtr, @@ -153,23 +192,20 @@ inline __device__ void copyFromSmemToDstMem(char* smemPtr, bool isValidStore = isValidRow && isValidCol; // Copy from shared memory to global memory. - if constexpr (CastToE2m1 || CastToMxE4m3) { - static_assert(!(CastToE2m1 && CastToMxE4m3), "Only one output conversion can be enabled."); + if constexpr (CastToE2m1) { // // Store the SFs in Layout128x4 (see trtllm/gen/DtypeUtils.h for details). - // The SMEM tile stores FP16 values. GMEM stores either packed E2m1 values with E4M3 SFs, or - // E4M3 values with UE8M0 SFs. + // All elements in the tile [numHeadsQPerKv (row), headDim (col)] are mapped to the col + // dimension of SFs. // - using DtypeBlockScaledOut = - std::conditional_t<CastToE2m1, cutlass::float_e2m1_t, cutlass::float_e4m3_t>; // Assume elements are stored in SMEM as FP16. int32_t constexpr NumBytesPerFp16Elt = 2; - // The number of output elements packed in a byte. - int32_t constexpr NumEltsPerDstByte = CastToE2m1 ? 2 : 1; + // The number of E2m1 elements packed in a byte. + int32_t constexpr NumE2m1EltsPerByte = 2; // The number of elements per sf. - int32_t constexpr NumEltsPerSf = CastToE2m1 ? 16 : 32; + int32_t constexpr NumEltsPerSf = 16; // The number of cols of SF per block. int32_t constexpr NumColsPerSfBlock = 4; // The size of each SF block. @@ -208,14 +244,13 @@ inline __device__ void copyFromSmemToDstMem(char* smemPtr, // Compute data destination offset. int64_t dstMemOffset{dstMemRowIdx * static_cast<int64_t>(dstMemRowStrideInBytes) + dstMemColOffset}; - dstMemOffset = dstMemOffset / NumBytesPerFp16Elt / NumEltsPerDstByte; - - copyFromSmemToGmemAndConvertToBlockScaled<DtypeBlockScaledOut, NumRegs>( - smemPtr + loadSmemOffset, - dstMemPtr + dstMemOffset, - dstMemSf + dstMemSfOffset, - sfScale, - isValidRow); + dstMemOffset = dstMemOffset / NumBytesPerFp16Elt / NumE2m1EltsPerByte; + + copyFromSmemToGmemAndConvertToE2m1<NumRegs>(smemPtr + loadSmemOffset, + dstMemPtr + dstMemOffset, + dstMemSf + dstMemSfOffset, + sfScale, + isValidRow); } else { // If it groups both headsQ and tokensQ into one CTA, we need to unpack the row index to the // valid range if values are stored to finalO. @@ -294,7 +329,6 @@ template <int32_t NumRows, class DtypeO, int NumRegs, bool CastToE2m1 = false, - bool CastToMxE4m3 = false, bool StoreToRemoteSmem = false, bool MapRowToHeadTokenIdx = false> inline __device__ void reorganizeInSmemAndStoreToDstMemImpl_( @@ -310,7 +344,7 @@ inline __device__ void reorganizeInSmemAndStoreToDstMemImpl_( int32_t numWarpGrpThreads, int32_t warpGrpThreadIdx, int32_t namedBarId, - void* dstMemPtrOSf = nullptr, + cutlass::float_e4m3_t* dstMemPtrOSf = nullptr, float sfScale = 0.f, int32_t sfBaseRowIdx = 0) { @@ -323,9 +357,6 @@ inline __device__ void reorganizeInSmemAndStoreToDstMemImpl_( "Not implemented."); static_assert(!CastToE2m1 || std::is_same_v<DtypeO, cutlass::half_t>, "DtypeO should be Fp16 if cast to E2M1."); - static_assert(!CastToMxE4m3 || std::is_same_v<DtypeO, cutlass::half_t>, - "DtypeO should be Fp16 if cast to MxE4m3."); - static_assert(!(CastToE2m1 && CastToMxE4m3), "Only one output conversion can be enabled."); // The number of rows and cols after transposing. int32_t constexpr NumTransRows{NumCols}; @@ -386,7 +417,6 @@ inline __device__ void reorganizeInSmemAndStoreToDstMemImpl_( RowStrideSf, NumSfPerHead, CastToE2m1, - CastToMxE4m3, StoreToRemoteSmem, MapRowToHeadTokenIdx>(baseSmemPtr, baseDstMemPtr, @@ -424,7 +454,6 @@ inline __device__ void reorganizeInSmemAndStoreToDstMem(DtypeO* smemPtrO, NumRegs, false, false, - false, MapRowToHeadTokenIdx>(smemPtrO, dstMemPtrO, nullptr, @@ -467,7 +496,6 @@ inline __device__ void reorganizeInSmemAndStoreToDstMem(DtypeO* smemPtrO, DtypeO, NumRegs, false, - false, true, MapRowToHeadTokenIdx>(smemPtrO, dstMemPtrO, @@ -503,57 +531,7 @@ inline __device__ void reorganizeInSmemAndStoreToDstMemAsE2m1( int32_t numWarpGrpThreads, int32_t warpGrpThreadIdx, int32_t namedBarId, - void* dstMemPtrOSf, - float sfScale, - int32_t sfBaseRowIdx = 0) { - - reorganizeInSmemAndStoreToDstMemImpl_<NumRows, - NumCols, - RowStrideSf, - NumSfPerHead, - DtypeO, - NumRegs, - true, - false, - false, - MapRowToHeadTokenIdx>(smemPtrO, - dstMemPtrO, - static_cast<uint64_t*>(nullptr), - arrayO, - dstMemRowStride, - numValidTokens, - numValidTokens, - numHeadsQ, - numHeadsQPerKv, - numWarpGrpThreads, - warpGrpThreadIdx, - namedBarId, - dstMemPtrOSf, - sfScale, - sfBaseRowIdx); -} - -//////////////////////////////////////////////////////////////////////////////////////////////////// - -template <int32_t NumRows, - int32_t NumCols, - int32_t RowStrideSf, - int32_t NumSfPerHead, - bool MapRowToHeadTokenIdx, - class DtypeO, - int NumRegs> -inline __device__ void reorganizeInSmemAndStoreToDstMemAsMxE4m3( - DtypeO* smemPtrO, - void* dstMemPtrO, - uint32_t (&arrayO)[NumRegs], - int32_t dstMemRowStride, - int32_t numValidTokens, - int32_t numHeadsQ, - trtllm::dev::fast_mod_div numHeadsQPerKv, - int32_t numWarpGrpThreads, - int32_t warpGrpThreadIdx, - int32_t namedBarId, - void* dstMemPtrOSf, + cutlass::float_e4m3_t* dstMemPtrOSf, float sfScale, int32_t sfBaseRowIdx = 0) { @@ -563,7 +541,6 @@ inline __device__ void reorganizeInSmemAndStoreToDstMemAsMxE4m3( NumSfPerHead, DtypeO, NumRegs, - false, true, false, MapRowToHeadTokenIdx>(smemPtrO, diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Utils.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Utils.h index b5eb47e012c0..c0edb65b628d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Utils.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllm/dev/Utils.h @@ -180,34 +180,36 @@ inline __device__ void cpAsyncPredicated(bool pred, //////////////////////////////////////////////////////////////////////////////////////////////////// -template <bool CxIsOne, bool CyIsOne, bool CzIsOne> inline __device__ dim3 getBlockIdInCluster() { +template <bool CxIsOne, bool CyIsOne, bool CzIsOne> +inline __device__ dim3 getBlockIdInCluster() { dim3 result; if constexpr (CxIsOne) { result.x = 0u; } else { - asm volatile("mov.u32 %0, %%cluster_ctaid.x;\n" : "=r"(result.x) :); + asm volatile("mov.u32 %0, %%cluster_ctaid.x;\n" : "=r"(result.x) : ); } if constexpr (CyIsOne) { result.y = 0u; } else { - asm volatile("mov.u32 %0, %%cluster_ctaid.y;\n" : "=r"(result.y) :); + asm volatile("mov.u32 %0, %%cluster_ctaid.y;\n" : "=r"(result.y) : ); } if constexpr (CzIsOne) { result.z = 0u; } else { - asm volatile("mov.u32 %0, %%cluster_ctaid.z;\n" : "=r"(result.z) :); + asm volatile("mov.u32 %0, %%cluster_ctaid.z;\n" : "=r"(result.z) : ); } return result; } //////////////////////////////////////////////////////////////////////////////////////////////////// -template <bool IsSingleBlock> inline __device__ uint32_t getBlockRankInCluster() { +template <bool IsSingleBlock> +inline __device__ uint32_t getBlockRankInCluster() { if constexpr (IsSingleBlock) { return 0u; } else { uint32_t rank; - asm volatile("mov.u32 %0, %%cluster_ctarank;\n" : "=r"(rank) :); + asm volatile("mov.u32 %0, %%cluster_ctarank;\n" : "=r"(rank) : ); return rank; } } diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllmGenExportCompat.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllmGenExportCompat.h index 750161fb0701..c12e789b9a40 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllmGenExportCompat.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/fmha/trtllmGen_fmha_export/trtllmGenExportCompat.h @@ -154,8 +154,6 @@ template <> inline std::string toString(AttentionMaskType e) { return "SlidingOrChunkedCausal"; case AttentionMaskType::Custom: return "Custom"; - case AttentionMaskType::SlidingWindowCustom: - return "SlidingWindowCustom"; default: return ""; } diff --git a/cpp/tensorrt_llm/kernels/unfusedAttentionKernels.h b/cpp/tensorrt_llm/kernels/unfusedAttentionKernels.h index 04bcdfbbe2ac..302c278f7a2a 100644 --- a/cpp/tensorrt_llm/kernels/unfusedAttentionKernels.h +++ b/cpp/tensorrt_llm/kernels/unfusedAttentionKernels.h @@ -17,7 +17,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/quantization.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/gptKernels.h" #include "tensorrt_llm/kernels/kvCacheUtils.h" #include "tensorrt_llm/kernels/mlaKernels.h" @@ -243,42 +242,42 @@ struct QKVPreprocessingParams { ss << "seq_lens: " << *(runtime::ITensor::wrap( - (void*) seq_lens, tensorrt_llm::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))); + (void*) seq_lens, nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))); } if (cache_seq_lens && batch_size > 0) { ss << "cache_seq_lens: " - << *(runtime::ITensor::wrap((void*) cache_seq_lens, tensorrt_llm::DataType::kINT32, - runtime::ITensor::makeShape({batch_size}))); + << *(runtime::ITensor::wrap( + (void*) cache_seq_lens, nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))); } if (encoder_seq_lens && batch_size > 0) { ss << "encoder_seq_lens: " - << *(runtime::ITensor::wrap((void*) encoder_seq_lens, tensorrt_llm::DataType::kINT32, - runtime::ITensor::makeShape({batch_size}))); + << *(runtime::ITensor::wrap( + (void*) encoder_seq_lens, nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))); } if (cu_seq_lens && batch_size > 0) { ss << "cu_seq_lens: " << *(runtime::ITensor::wrap( - (void*) cu_seq_lens, tensorrt_llm::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))); + (void*) cu_seq_lens, nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))); } if (cu_kv_seq_lens && batch_size > 0) { ss << "cu_kv_seq_lens: " - << *(runtime::ITensor::wrap((void*) cu_kv_seq_lens, tensorrt_llm::DataType::kINT32, - runtime::ITensor::makeShape({batch_size}))); + << *(runtime::ITensor::wrap( + (void*) cu_kv_seq_lens, nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({batch_size}))); } if (sparse_kv_offsets) { ss << "sparse_kv_offsets: " - << *(runtime::ITensor::wrap((void*) sparse_kv_offsets, tensorrt_llm::DataType::kINT32, + << *(runtime::ITensor::wrap((void*) sparse_kv_offsets, nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({batch_size + 1}))); } if (rotary_embedding_inv_freq && batch_size > 0 && rotary_embedding_dim > 0) { ss << "rotary_embedding_inv_freq: " - << *(runtime::ITensor::wrap((void*) rotary_embedding_inv_freq, tensorrt_llm::DataType::kFLOAT, + << *(runtime::ITensor::wrap((void*) rotary_embedding_inv_freq, nvinfer1::DataType::kFLOAT, runtime::ITensor::makeShape({batch_size, rotary_embedding_dim / 2}))); } ss << "rotary_coef_cache_buffer: " << rotary_coef_cache_buffer << std::endl; diff --git a/cpp/tensorrt_llm/kernels/userbuffers/ub_interface.cpp b/cpp/tensorrt_llm/kernels/userbuffers/ub_interface.cpp index d219572d2707..3e19f9ebe72a 100644 --- a/cpp/tensorrt_llm/kernels/userbuffers/ub_interface.cpp +++ b/cpp/tensorrt_llm/kernels/userbuffers/ub_interface.cpp @@ -16,7 +16,6 @@ #include "ub_interface.h" #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cudaDriverWrapper.h" -#include "tensorrt_llm/common/tllmDataType.h" #include <cuda_runtime.h> #include <cuda_runtime_api.h> @@ -81,13 +80,13 @@ namespace kernels::ub { void allreduce2_userbuff_inplace_launcher(int const handler, size_t const offset, size_t const elements, - tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream) + nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { allreduce2_userbuff_inplace_impl(handler, offset, elements, dataType, comm, stream); } int allgather2_userbuff_residual_launcher(int const handler, size_t const offset, size_t const elements, - int const hidden_size, void* residual, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream, + int const hidden_size, void* residual, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream, bool force_enable) { return allgather2_userbuff_residual_impl( @@ -96,7 +95,7 @@ int allgather2_userbuff_residual_launcher(int const handler, size_t const offset int allreduce2_userbuff_rmsnorm_launcher(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream) + void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { return allreduce2_userbuff_rmsnorm_impl(handler, offset, out_handler, out_offset, elements, hidden_size, beta, gamma, eps, residual_in, residual_out, dataType, comm, stream); @@ -104,7 +103,7 @@ int allreduce2_userbuff_rmsnorm_launcher(int const handler, size_t const offset, int allreduce2_userbuff_inplace_rmsnorm_quant_launcher(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - float* scalefactor, void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, + float* scalefactor, void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { return allreduce2_userbuff_inplace_rmsnorm_quant_impl(handler, offset, out_handler, out_offset, elements, @@ -114,7 +113,7 @@ int allreduce2_userbuff_inplace_rmsnorm_quant_launcher(int const handler, size_t int allreduce2_userbuff_inplace_rmsnorm_quant_fp4_launcher(int const handler, size_t const offset, int const out_handler, size_t const out_offset, int const scale_handler, size_t const scale_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, float* scalefactor, - void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream) + void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { return allreduce2_userbuff_inplace_rmsnorm_quant_fp4_impl(handler, offset, out_handler, out_offset, scale_handler, scale_offset, elements, hidden_size, beta, gamma, eps, scalefactor, residual_in, residual_out, dataType, comm, @@ -166,12 +165,12 @@ TRTLLM_NAMESPACE_BEGIN namespace kernels::ub { void allreduce2_userbuff_inplace_launcher(int const handler, size_t const offset, size_t const elements, - tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream) + nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { } int allgather2_userbuff_residual_launcher(int const handler, size_t const offset, size_t const elements, - int const hidden_size, void* residual, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream, + int const hidden_size, void* residual, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream, bool force_enable) { return 0; @@ -179,7 +178,7 @@ int allgather2_userbuff_residual_launcher(int const handler, size_t const offset int allreduce2_userbuff_inplace_rmsnorm_quant_launcher(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - float* scalefactor, void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, + float* scalefactor, void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { return 0; @@ -188,7 +187,7 @@ int allreduce2_userbuff_inplace_rmsnorm_quant_launcher(int const handler, size_t int allreduce2_userbuff_inplace_rmsnorm_quant_fp4_launcher(int const handler, size_t const offset, int const out_handler, size_t const out_offset, int const scale_handler, size_t const scale_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, float* scalefactor, - void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream) + void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { return 0; } diff --git a/cpp/tensorrt_llm/kernels/userbuffers/ub_interface.h b/cpp/tensorrt_llm/kernels/userbuffers/ub_interface.h index dc68154fb462..e8a48e2c680b 100644 --- a/cpp/tensorrt_llm/kernels/userbuffers/ub_interface.h +++ b/cpp/tensorrt_llm/kernels/userbuffers/ub_interface.h @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/dataType.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "ub_allocator.h" namespace tensorrt_llm::runtime::ub @@ -41,24 +40,24 @@ namespace kernels::ub using ::tensorrt_llm::runtime::ub::communicator; void allreduce2_userbuff_inplace_launcher(int const handler, size_t const offset, size_t const elements, - tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream = 0); + nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream = 0); int allgather2_userbuff_residual_launcher(int const handler, size_t const offset, size_t const elements, - int const hidden_size, void* residual, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream, + int const hidden_size, void* residual, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream, bool force_enable = false); int allreduce2_userbuff_rmsnorm_launcher(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream); + void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream); int allreduce2_userbuff_inplace_rmsnorm_quant_launcher(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - float* scalefactor, void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, + float* scalefactor, void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream); int allreduce2_userbuff_inplace_rmsnorm_quant_fp4_launcher(int const handler, size_t const offset, int const out_handler, size_t const out_offset, int const scale_handler, size_t const scale_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, float* scalefactor, - void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream); + void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream); } // namespace kernels::ub TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/userbuffers/userbuffers.cu b/cpp/tensorrt_llm/kernels/userbuffers/userbuffers.cu index a19059e9010c..8cb5814e0398 100644 --- a/cpp/tensorrt_llm/kernels/userbuffers/userbuffers.cu +++ b/cpp/tensorrt_llm/kernels/userbuffers/userbuffers.cu @@ -15,7 +15,6 @@ */ #include "tensorrt_llm/common/config.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/quantization.cuh" #include "userbuffers.h" #include "utils.h" @@ -1775,11 +1774,11 @@ int allgather2_userbuff_residual(int const handler, size_t const offset, size_t } void allreduce2_userbuff_inplace_impl(int const handler, size_t const offset, size_t const elements, - tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream) + nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { switch (dataType) { - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: { if (kDISABLE_FP32_ACCUMULATION) { @@ -1792,7 +1791,7 @@ void allreduce2_userbuff_inplace_impl(int const handler, size_t const offset, si break; } #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: { if (kDISABLE_FP32_ACCUMULATION) { @@ -1810,17 +1809,17 @@ void allreduce2_userbuff_inplace_impl(int const handler, size_t const offset, si } int allgather2_userbuff_residual_impl(int const handler, size_t const offset, size_t const elements, - int const hidden_size, void* residual, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream, + int const hidden_size, void* residual, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream, bool force_enable) { switch (dataType) { - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: return allgather2_userbuff_residual<half>( handler, offset, elements, hidden_size, residual, comm, stream, force_enable); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: return allgather2_userbuff_residual<__nv_bfloat16>( handler, offset, elements, hidden_size, residual, comm, stream, force_enable); break; @@ -1831,11 +1830,11 @@ int allgather2_userbuff_residual_impl(int const handler, size_t const offset, si int allreduce2_userbuff_rmsnorm_impl(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream) + void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { switch (dataType) { - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: { if (kDISABLE_FP32_ACCUMULATION) { @@ -1850,7 +1849,7 @@ int allreduce2_userbuff_rmsnorm_impl(int const handler, size_t const offset, int break; } #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: { if (kDISABLE_FP32_ACCUMULATION) { @@ -1871,12 +1870,12 @@ int allreduce2_userbuff_rmsnorm_impl(int const handler, size_t const offset, int int allreduce2_userbuff_inplace_rmsnorm_quant_impl(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - float* scalefactor, void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, + float* scalefactor, void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { switch (dataType) { - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: { if (kDISABLE_FP32_ACCUMULATION) { @@ -1891,7 +1890,7 @@ int allreduce2_userbuff_inplace_rmsnorm_quant_impl(int const handler, size_t con break; } #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: { if (kDISABLE_FP32_ACCUMULATION) { @@ -1915,11 +1914,11 @@ int allreduce2_userbuff_inplace_rmsnorm_quant_impl(int const handler, size_t con int allreduce2_userbuff_inplace_rmsnorm_quant_fp4_impl(int const handler, size_t const offset, int const out_handler, size_t const out_offset, int const scale_handler, size_t const scale_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, float* scalefactor, void* residual_in, - void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream) + void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream) { switch (dataType) { - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: { if (kDISABLE_FP32_ACCUMULATION) { @@ -1936,7 +1935,7 @@ int allreduce2_userbuff_inplace_rmsnorm_quant_fp4_impl(int const handler, size_t break; } #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: { if (kDISABLE_FP32_ACCUMULATION) { diff --git a/cpp/tensorrt_llm/kernels/userbuffers/userbuffers.h b/cpp/tensorrt_llm/kernels/userbuffers/userbuffers.h index 5d3ffe0cc950..96f21b748282 100644 --- a/cpp/tensorrt_llm/kernels/userbuffers/userbuffers.h +++ b/cpp/tensorrt_llm/kernels/userbuffers/userbuffers.h @@ -14,7 +14,6 @@ * limitations under the License. */ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" #include "tensorrt_llm/runtime/worldConfig.h" @@ -121,25 +120,25 @@ namespace kernels::ub { using namespace ::tensorrt_llm::runtime::ub; void allreduce2_userbuff_inplace_impl(int const handler, size_t const offset, size_t const elements, - tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream = 0); + nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream = 0); // for TP-parallelism, only single node is implemented int allgather2_userbuff_residual_impl(int const handler, size_t const offset, size_t const elements, - int const hidden_size, void* residual, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream, + int const hidden_size, void* residual, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream, bool force_enable); int allreduce2_userbuff_rmsnorm_impl(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream); + void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream); int allreduce2_userbuff_inplace_rmsnorm_quant_impl(int const handler, size_t const offset, int const out_handler, size_t const out_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, - float* scalefactor, void* residual_in, void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, + float* scalefactor, void* residual_in, void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream); int allreduce2_userbuff_inplace_rmsnorm_quant_fp4_impl(int const handler, size_t const offset, int const out_handler, size_t const out_offset, int const scale_handler, size_t const scale_offset, size_t const elements, int const hidden_size, void* beta, void* gamma, float eps, float* scalefactor, void* residual_in, - void* residual_out, tensorrt_llm::DataType dataType, communicator* comm, cudaStream_t stream); + void* residual_out, nvinfer1::DataType dataType, communicator* comm, cudaStream_t stream); } // namespace kernels::ub TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemm.h b/cpp/tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemm.h index c2bf35175391..eb939b57c2db 100644 --- a/cpp/tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemm.h +++ b/cpp/tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemm.h @@ -24,6 +24,8 @@ #include "tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h" #include "tensorrt_llm/runtime/common.h" +#include <NvInferRuntime.h> + #include <cassert> #include <cmath> #include <cstdint> diff --git a/cpp/tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemmNVFP4.h b/cpp/tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemmNVFP4.h index 6e901846ed25..616f9d25c2bf 100644 --- a/cpp/tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemmNVFP4.h +++ b/cpp/tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemmNVFP4.h @@ -24,6 +24,8 @@ #include "tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h" #include "tensorrt_llm/runtime/common.h" +#include <NvInferRuntime.h> + #include <cassert> #include <cmath> #include <cstdint> diff --git a/cpp/tensorrt_llm/kernels/xqaDispatcher.cpp b/cpp/tensorrt_llm/kernels/xqaDispatcher.cpp index 8a5eeee91c6b..35fd02e7f127 100644 --- a/cpp/tensorrt_llm/kernels/xqaDispatcher.cpp +++ b/cpp/tensorrt_llm/kernels/xqaDispatcher.cpp @@ -534,7 +534,6 @@ void XqaDispatcher::runImpl( tllmRunnerParams.generalPackedCustoMaskPtr = params.spec_decoding_packed_mask; tllmRunnerParams.mPackedMaskMaxSeqLenQ = params.spec_decoding_max_generation_length; tllmRunnerParams.mSpecDecodingTargetMaxGenLen = mFixedParams.specDecodingTargetMaxGenLen; - tllmRunnerParams.mForcePrepareSpecDecTreeMask = params.force_prepare_spec_dec_tree_mask; tllmRunnerParams.customMaskPtr = params.spec_decoding_bl_tree_mask; tllmRunnerParams.customMaskOffsetsPtr = params.spec_decoding_bl_tree_mask_offset; tllmRunnerParams.firstSparseMaskOffsetsKvPtr = params.spec_bl_tree_first_sparse_mask_offset_kv; diff --git a/cpp/tensorrt_llm/layers/decodingParams.h b/cpp/tensorrt_llm/layers/decodingParams.h index 76c5cedd637b..1e77b8919ca1 100644 --- a/cpp/tensorrt_llm/layers/decodingParams.h +++ b/cpp/tensorrt_llm/layers/decodingParams.h @@ -17,7 +17,6 @@ #pragma once #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/kernels/beamSearchKernels.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -193,9 +192,9 @@ class ExplicitDraftTokensSetupParams : public DecodingSetupParams public: OptVec<float> temperature; // [setupBatchSize] // Hack to init some data for the context phase in the setup. - TensorPtr randomDataSample; // [maxBatchSize], on gpu - TensorPtr temperatures; // [maxBatchSize], on gpu - tensorrt_llm::DataType dtype; // [1] + TensorPtr randomDataSample; // [maxBatchSize], on gpu + TensorPtr temperatures; // [maxBatchSize], on gpu + nvinfer1::DataType dtype; // [1] }; class EagleSetupParams : public DecodingSetupParams @@ -203,9 +202,9 @@ class EagleSetupParams : public DecodingSetupParams public: OptVec<float> temperature; // [setupBatchSize] // Hack to init some data for the context phase in the setup. - TensorPtr randomDataSample; // [maxBatchSize], on gpu - TensorPtr temperatures; // [maxBatchSize], on gpu - tensorrt_llm::DataType dtype; // [1] + TensorPtr randomDataSample; // [maxBatchSize], on gpu + TensorPtr temperatures; // [maxBatchSize], on gpu + nvinfer1::DataType dtype; // [1] }; class DynamicDecodeSetupParams : public BaseSetupParams diff --git a/cpp/tensorrt_llm/layers/explicitDraftTokensLayer.cpp b/cpp/tensorrt_llm/layers/explicitDraftTokensLayer.cpp index aedeb731574f..e014ee4535e5 100644 --- a/cpp/tensorrt_llm/layers/explicitDraftTokensLayer.cpp +++ b/cpp/tensorrt_llm/layers/explicitDraftTokensLayer.cpp @@ -16,7 +16,6 @@ #include "explicitDraftTokensLayer.h" #include "tensorrt_llm/common/nvtxUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/penaltyTypes.h" #include "tensorrt_llm/kernels/speculativeDecoding/common.h" #include "tensorrt_llm/kernels/speculativeDecoding/explicitDraftTokensKernels.h" @@ -94,15 +93,15 @@ void ExplicitDraftTokensLayer<T>::setup(SizeType32 batchSize, SizeType32 beamWid batchSlots, getLimitsPenalty(DecodingPenaltyType::Temperature), "temperature penalty"); // Dispatch context buffer fill - if (mDecoderDtype == tensorrt_llm::DataType::kFLOAT) + if (mDecoderDtype == nvinfer1::DataType::kFLOAT) { fillContextBuffers<float>(batchSize, batchSlots, *setupParams, workspace); } - else if (mDecoderDtype == tensorrt_llm::DataType::kHALF) + else if (mDecoderDtype == nvinfer1::DataType::kHALF) { fillContextBuffers<half>(batchSize, batchSlots, *setupParams, workspace); } - else if (mDecoderDtype == tensorrt_llm::DataType::kBF16) + else if (mDecoderDtype == nvinfer1::DataType::kBF16) { fillContextBuffers<__nv_bfloat16>(batchSize, batchSlots, *setupParams, workspace); } @@ -127,15 +126,15 @@ void ExplicitDraftTokensLayer<T>::forwardAsync(std::shared_ptr<BaseDecodingOutpu convertPackedMask(*outputs, *inputs, workspace); // Slice output ids, pos ids, next draft tokens. - if (mDecoderDtype == tensorrt_llm::DataType::kFLOAT) + if (mDecoderDtype == nvinfer1::DataType::kFLOAT) { splitInputDataToBatchSlots<float>(*outputs, *inputs, workspace); } - else if (mDecoderDtype == tensorrt_llm::DataType::kHALF) + else if (mDecoderDtype == nvinfer1::DataType::kHALF) { splitInputDataToBatchSlots<half>(*outputs, *inputs, workspace); } - else if (mDecoderDtype == tensorrt_llm::DataType::kBF16) + else if (mDecoderDtype == nvinfer1::DataType::kBF16) { splitInputDataToBatchSlots<__nv_bfloat16>(*outputs, *inputs, workspace); } diff --git a/cpp/tensorrt_llm/layers/explicitDraftTokensLayer.h b/cpp/tensorrt_llm/layers/explicitDraftTokensLayer.h index 17fca4513cf1..75883ded6e5a 100644 --- a/cpp/tensorrt_llm/layers/explicitDraftTokensLayer.h +++ b/cpp/tensorrt_llm/layers/explicitDraftTokensLayer.h @@ -16,7 +16,6 @@ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/layers/baseLayer.h" #include "tensorrt_llm/layers/decodingParams.h" #include "tensorrt_llm/runtime/common.h" @@ -84,7 +83,7 @@ class ExplicitDraftTokensLayer : public BaseLayer TensorPtr mTemperature; - std::optional<tensorrt_llm::DataType> mDecoderDtype{std::nullopt}; + std::optional<nvinfer1::DataType> mDecoderDtype{std::nullopt}; }; } // namespace tensorrt_llm::layers diff --git a/cpp/tensorrt_llm/layers/lookaheadAlgorithm.cpp b/cpp/tensorrt_llm/layers/lookaheadAlgorithm.cpp index 76da89dfec0d..09843fd7ce44 100644 --- a/cpp/tensorrt_llm/layers/lookaheadAlgorithm.cpp +++ b/cpp/tensorrt_llm/layers/lookaheadAlgorithm.cpp @@ -17,7 +17,6 @@ #include "tensorrt_llm/layers/lookaheadAlgorithm.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/layers/lookaheadDecodingUtils.h" #include "tensorrt_llm/runtime/common.h" @@ -36,14 +35,14 @@ LookaheadAlgorithm::LookaheadAlgorithm( runtime::SizeType32 maxW, runtime::SizeType32 maxN, runtime::SizeType32 maxG, runtime::SizeType32 id) : mPoolManager(maxG) , mPrefillsMax(runtime::BufferManager::cpu( - runtime::ITensor::makeShape({(maxN <= 1 ? 0 : maxN - 2)}), tensorrt_llm::DataType::kINT32)) + runtime::ITensor::makeShape({(maxN <= 1 ? 0 : maxN - 2)}), nvinfer1::DataType::kINT32)) , mPastTokensMax( - runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxW * (maxN - 1)}), tensorrt_llm::DataType::kINT32)) - , mKeyTokensMax(runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxW}), tensorrt_llm::DataType::kINT32)) + runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxW * (maxN - 1)}), nvinfer1::DataType::kINT32)) + , mKeyTokensMax(runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxW}), nvinfer1::DataType::kINT32)) , mGoldenTokensMax( - runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxN * 2 - 1}), tensorrt_llm::DataType::kINT32)) + runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxN * 2 - 1}), nvinfer1::DataType::kINT32)) , mGuessTokensMax( - runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxG * (maxN - 1)}), tensorrt_llm::DataType::kINT32)) + runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxG * (maxN - 1)}), nvinfer1::DataType::kINT32)) , mMaxW(maxW) , mMaxN(maxN) , mMaxG(maxG) @@ -53,13 +52,12 @@ LookaheadAlgorithm::LookaheadAlgorithm( std::tie(maxGeneratedLen, std::ignore, maxDraftLen, std::ignore) = executor::LookaheadDecodingConfig(maxW, maxN, maxG).calculateSpeculativeResource(); mAttentionMask = runtime::BufferManager::cpu( - runtime::ITensor::makeShape({maxDraftLen, maxDraftLen}), tensorrt_llm::DataType::kBOOL); + runtime::ITensor::makeShape({maxDraftLen, maxDraftLen}), nvinfer1::DataType::kBOOL); mDraftTokensMax - = runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxDraftLen}), tensorrt_llm::DataType::kINT32); + = runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxDraftLen}), nvinfer1::DataType::kINT32); mSampledTokensMax - = runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxGeneratedLen}), tensorrt_llm::DataType::kINT32); - mEncodeMapMax - = runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxDraftLen}), tensorrt_llm::DataType::kINT32); + = runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxGeneratedLen}), nvinfer1::DataType::kINT32); + mEncodeMapMax = runtime::BufferManager::cpu(runtime::ITensor::makeShape({maxDraftLen}), nvinfer1::DataType::kINT32); } void LookaheadAlgorithm::setup(TensorConstPtr const& prompt, SizeType32 w, SizeType32 n, SizeType32 g, uint64_t seed) diff --git a/cpp/tensorrt_llm/layers/lookaheadDecodingLayer.cpp b/cpp/tensorrt_llm/layers/lookaheadDecodingLayer.cpp index 986f0e0b978e..bf6e15080f3c 100644 --- a/cpp/tensorrt_llm/layers/lookaheadDecodingLayer.cpp +++ b/cpp/tensorrt_llm/layers/lookaheadDecodingLayer.cpp @@ -19,7 +19,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/nvtxUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/kernels/samplingTopKKernels.h" #include "tensorrt_llm/layers/decodingParams.h" @@ -65,42 +64,38 @@ LookaheadDecodingLayer<T>::CpuAlgorithmResources::CpuAlgorithmResources(DecoderD mPrompts.reserve(maxBatchSize); for (auto bi = 0; bi < maxBatchSize; bi++) { - mPrompts.emplace_back(BufferManager::cpu(ITensor::makeShape({0}), tensorrt_llm::DataType::kINT32)); + mPrompts.emplace_back(BufferManager::cpu(ITensor::makeShape({0}), nvinfer1::DataType::kINT32)); } auto const maxBatchShape1D = ITensor::makeShape({maxBatchSize}); - mBatchSlots = BufferManager::cpu(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::cpu(maxBatchShape1D, nvinfer1::DataType::kINT32); mTargetTokens - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep}), tensorrt_llm::DataType::kINT32); - mTokensPerStep = BufferManager::cpu(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - mEndIds = BufferManager::cpu(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep}), nvinfer1::DataType::kINT32); + mTokensPerStep = BufferManager::cpu(maxBatchShape1D, nvinfer1::DataType::kINT32); + mEndIds = BufferManager::cpu(maxBatchShape1D, nvinfer1::DataType::kINT32); - mOutputIds - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxNumNewTokens}), tensorrt_llm::DataType::kINT32); + mOutputIds = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxNumNewTokens}), nvinfer1::DataType::kINT32); mNewTokens = BufferManager::cpu( - ITensor::makeShape({maxTokensPerStep, maxBatchSize, beamWidth}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxTokensPerStep, maxBatchSize, beamWidth}), nvinfer1::DataType::kINT32); mPathsOffsets - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxAcceptedDraftLen}), tensorrt_llm::DataType::kINT32); + = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxAcceptedDraftLen}), nvinfer1::DataType::kINT32); mPathsOffsetsBatch - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxAcceptedDraftLen}), tensorrt_llm::DataType::kINT32); - mNumNewTokens = BufferManager::cpu(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - mNumNewTokensCumSum = BufferManager::cpu(ITensor::makeShape({maxBatchSize + 1}), tensorrt_llm::DataType::kINT32); - mNextDraftTokens - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxDraftLen}), tensorrt_llm::DataType::kINT32); - mNextDraftPosIds - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxDraftLen}), tensorrt_llm::DataType::kINT32); - mGenerationLengths = BufferManager::cpu(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxAcceptedDraftLen}), nvinfer1::DataType::kINT32); + mNumNewTokens = BufferManager::cpu(maxBatchShape1D, nvinfer1::DataType::kINT32); + mNumNewTokensCumSum = BufferManager::cpu(ITensor::makeShape({maxBatchSize + 1}), nvinfer1::DataType::kINT32); + mNextDraftTokens = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxDraftLen}), nvinfer1::DataType::kINT32); + mNextDraftPosIds = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxDraftLen}), nvinfer1::DataType::kINT32); + mGenerationLengths = BufferManager::cpu(maxBatchShape1D, nvinfer1::DataType::kINT32); mPositionOffsets - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep}), tensorrt_llm::DataType::kINT32); - mPositionIds - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep}), tensorrt_llm::DataType::kINT32); + = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep}), nvinfer1::DataType::kINT32); + mPositionIds = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep}), nvinfer1::DataType::kINT32); mAttentionMask - = BufferManager::cpu(ITensor::makeShape({maxTokensPerStep, maxTokensPerStep}), tensorrt_llm::DataType::kBOOL); + = BufferManager::cpu(ITensor::makeShape({maxTokensPerStep, maxTokensPerStep}), nvinfer1::DataType::kBOOL); mPackedMask = BufferManager::cpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep, static_cast<ITensor::DimType64>(divUp(maxTokensPerStep, 32))}), - tensorrt_llm::DataType::kINT32); - mNextDraftLengths = BufferManager::cpu(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - mSequenceLengths = BufferManager::cpu(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); + mNextDraftLengths = BufferManager::cpu(maxBatchShape1D, nvinfer1::DataType::kINT32); + mSequenceLengths = BufferManager::cpu(maxBatchShape1D, nvinfer1::DataType::kINT32); } template <typename T> @@ -122,12 +117,12 @@ LookaheadDecodingLayer<T>::LookaheadDecodingLayer( auto const maxBatchShape2D = ITensor::makeShape({maxBatchSize, maxTokensPerStep}); mWorkspaceSize = getTopKWorkspaceSize<T>(maxBatchSize, maxTokensPerStep, maxTopK, vocabSizePadded); - mTargetTokensDevice = mBufferManager->gpu(maxBatchShape2D, tensorrt_llm::DataType::kINT32); + mTargetTokensDevice = mBufferManager->gpu(maxBatchShape2D, nvinfer1::DataType::kINT32); mCurandStatesDevice - = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, sizeof(curandState_t)}), tensorrt_llm::DataType::kINT8); + = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, sizeof(curandState_t)}), nvinfer1::DataType::kINT8); mSetupWorkspaceSize = DecodingLayerWorkspace::calculateRequiredWorkspaceSize( - std::make_pair(maxBatchShape1D, tensorrt_llm::DataType::kINT64)); + std::make_pair(maxBatchShape1D, nvinfer1::DataType::kINT64)); TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); } diff --git a/cpp/tensorrt_llm/layers/lookaheadDecodingUtils.h b/cpp/tensorrt_llm/layers/lookaheadDecodingUtils.h index 8e3e8f6c590d..739cf65001ab 100644 --- a/cpp/tensorrt_llm/layers/lookaheadDecodingUtils.h +++ b/cpp/tensorrt_llm/layers/lookaheadDecodingUtils.h @@ -16,7 +16,6 @@ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -319,12 +318,12 @@ class DebugTensor { switch (mTensor.getDataType()) { - case tensorrt_llm::DataType::kBOOL: return values<bool>(); - case tensorrt_llm::DataType::kFLOAT: return values<float>(); - case tensorrt_llm::DataType::kINT8: return values<std::int8_t>(); - case tensorrt_llm::DataType::kINT32: return values<std::int32_t>(); - case tensorrt_llm::DataType::kINT64: return values<std::int64_t>(); - case tensorrt_llm::DataType::kUINT8: return values<std::uint8_t>(); + case nvinfer1::DataType::kBOOL: return values<bool>(); + case nvinfer1::DataType::kFLOAT: return values<float>(); + case nvinfer1::DataType::kINT8: return values<std::int8_t>(); + case nvinfer1::DataType::kINT32: return values<std::int32_t>(); + case nvinfer1::DataType::kINT64: return values<std::int64_t>(); + case nvinfer1::DataType::kUINT8: return values<std::uint8_t>(); default: return std::string(mName + ": Unsupported data type"); } } @@ -377,12 +376,12 @@ class DebugTensor { switch (mTensor.getDataType()) { - case tensorrt_llm::DataType::kBOOL: return randomize<bool>(3); - case tensorrt_llm::DataType::kFLOAT: return randomize<float>(3); - case tensorrt_llm::DataType::kINT8: return randomize<std::int8_t>(3); - case tensorrt_llm::DataType::kINT32: return randomize<std::int32_t>(3); - case tensorrt_llm::DataType::kINT64: return randomize<std::int64_t>(3); - case tensorrt_llm::DataType::kUINT8: return randomize<std::uint8_t>(3); + case nvinfer1::DataType::kBOOL: return randomize<bool>(3); + case nvinfer1::DataType::kFLOAT: return randomize<float>(3); + case nvinfer1::DataType::kINT8: return randomize<std::int8_t>(3); + case nvinfer1::DataType::kINT32: return randomize<std::int32_t>(3); + case nvinfer1::DataType::kINT64: return randomize<std::int64_t>(3); + case nvinfer1::DataType::kUINT8: return randomize<std::uint8_t>(3); default: return; } } @@ -392,12 +391,12 @@ class DebugTensor { switch (mTensor.getDataType()) { - case tensorrt_llm::DataType::kBOOL: return randomize<bool>(0); - case tensorrt_llm::DataType::kFLOAT: return randomize<float>(0); - case tensorrt_llm::DataType::kINT8: return randomize<std::int8_t>(0); - case tensorrt_llm::DataType::kINT32: return randomize<std::int32_t>(0); - case tensorrt_llm::DataType::kINT64: return randomize<std::int64_t>(0); - case tensorrt_llm::DataType::kUINT8: return randomize<std::uint8_t>(0); + case nvinfer1::DataType::kBOOL: return randomize<bool>(0); + case nvinfer1::DataType::kFLOAT: return randomize<float>(0); + case nvinfer1::DataType::kINT8: return randomize<std::int8_t>(0); + case nvinfer1::DataType::kINT32: return randomize<std::int32_t>(0); + case nvinfer1::DataType::kINT64: return randomize<std::int64_t>(0); + case nvinfer1::DataType::kUINT8: return randomize<std::uint8_t>(0); default: return; } } @@ -406,12 +405,12 @@ class DebugTensor { switch (mTensor.getDataType()) { - case tensorrt_llm::DataType::kBOOL: return randomize<bool>(1); - case tensorrt_llm::DataType::kFLOAT: return randomize<float>(1); - case tensorrt_llm::DataType::kINT8: return randomize<std::int8_t>(1); - case tensorrt_llm::DataType::kINT32: return randomize<std::int32_t>(1); - case tensorrt_llm::DataType::kINT64: return randomize<std::int64_t>(1); - case tensorrt_llm::DataType::kUINT8: return randomize<std::uint8_t>(1); + case nvinfer1::DataType::kBOOL: return randomize<bool>(1); + case nvinfer1::DataType::kFLOAT: return randomize<float>(1); + case nvinfer1::DataType::kINT8: return randomize<std::int8_t>(1); + case nvinfer1::DataType::kINT32: return randomize<std::int32_t>(1); + case nvinfer1::DataType::kINT64: return randomize<std::int64_t>(1); + case nvinfer1::DataType::kUINT8: return randomize<std::uint8_t>(1); default: return; } } diff --git a/cpp/tensorrt_llm/layers/lookaheadPoolManager.cpp b/cpp/tensorrt_llm/layers/lookaheadPoolManager.cpp index 397b4262226a..5954bc520ad0 100644 --- a/cpp/tensorrt_llm/layers/lookaheadPoolManager.cpp +++ b/cpp/tensorrt_llm/layers/lookaheadPoolManager.cpp @@ -16,7 +16,6 @@ #include "tensorrt_llm/layers/lookaheadPoolManager.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/layers/lookaheadDecodingUtils.h" #include <cstddef> @@ -68,7 +67,7 @@ void LookaheadPoolManager::accept(TensorConstPtr const& prompt, SizeType32 level for (SizeType32 ti = 0; ti + level - 1 < length; ti++) { auto key = promptRange[ti]; - TensorPtr ngram = BufferManager::cpu(ITensor::makeShape({level - 1}), tensorrt_llm::DataType::kINT32); + TensorPtr ngram = BufferManager::cpu(ITensor::makeShape({level - 1}), nvinfer1::DataType::kINT32); BufferRange<TokenIdType const> sourceRange(*ITensor::slice(prompt, ti + 1, level - 1)); BufferRange<TokenIdType> ngramRange(*ngram); std::copy(sourceRange.begin(), sourceRange.end(), ngramRange.begin()); @@ -108,7 +107,7 @@ void LookaheadPoolManager::update(TensorConstPtr const& keyTokens, TensorConstPt for (SizeType32 wi = 0; wi < window; wi++) { TensorConstPtr source = ITensor::at(ngramTokens, {wi}); - TensorPtr ngram = BufferManager::cpu(source->getShape(), tensorrt_llm::DataType::kINT32); + TensorPtr ngram = BufferManager::cpu(source->getShape(), nvinfer1::DataType::kINT32); BufferRange<TokenIdType const> sourceRange(*source); BufferRange<TokenIdType> ngramRange(*ngram); std::copy(sourceRange.begin(), sourceRange.end(), ngramRange.begin()); diff --git a/cpp/tensorrt_llm/layers/medusaDecodingLayer.cpp b/cpp/tensorrt_llm/layers/medusaDecodingLayer.cpp index 9e4098b34ebf..40eff62c17d6 100644 --- a/cpp/tensorrt_llm/layers/medusaDecodingLayer.cpp +++ b/cpp/tensorrt_llm/layers/medusaDecodingLayer.cpp @@ -16,7 +16,6 @@ #include "medusaDecodingLayer.h" #include "tensorrt_llm/common/nvtxUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/decodingCommon.h" #include "tensorrt_llm/kernels/samplingTopKKernels.h" #include "tensorrt_llm/kernels/speculativeDecoding/medusaDecodingKernels.h" @@ -89,10 +88,10 @@ void MedusaDecodingLayer<T>::allocateBuffer() mTiledBatchSlotsSetup = BufferManager::pinnedPool( ITensor::makeShape({static_cast<SizeType32>(mDecoderDomain.getBatchSize() * maxDraftPathLen)}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mTiledBatchSlotsForward = BufferManager::pinnedPool( ITensor::makeShape({static_cast<SizeType32>(mDecoderDomain.getBatchSize() * maxDraftPathLen)}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mMedusaInputLogitsPtrs = BufferManager::pinnedPool( ITensor::makeShape({static_cast<SizeType32>(mDecoderDomain.getBatchSize() * maxDraftPathLen)}), TRTDataType<T*>::value); diff --git a/cpp/tensorrt_llm/layers/penaltyLayer.cpp b/cpp/tensorrt_llm/layers/penaltyLayer.cpp index c72b8e463bc6..c6c57ca5034d 100644 --- a/cpp/tensorrt_llm/layers/penaltyLayer.cpp +++ b/cpp/tensorrt_llm/layers/penaltyLayer.cpp @@ -18,7 +18,6 @@ #include "penaltyLayer.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/nvtxUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/penaltyKernels.h" #include "tensorrt_llm/kernels/penaltyTypes.h" #include "tensorrt_llm/layers/defaultDecodingParams.h" @@ -85,11 +84,11 @@ void PenaltyLayer<T>::allocateWorkspace() auto const workspaceSize = mDecoderDomain.getBatchSize() * mDecoderDomain.getMaxDecodingTokens() * mConfiguredBeamWidth * mDecoderDomain.getVocabSize() * 2; - mPenaltyWorkspaceDevice = mBufferManager->gpu(workspaceSize, tensorrt_llm::DataType::kINT32); + mPenaltyWorkspaceDevice = mBufferManager->gpu(workspaceSize, nvinfer1::DataType::kINT32); if (mDecodingMode.isBeamSearch()) { - mPenaltyWorkspacePrevDevice = mBufferManager->gpu(workspaceSize, tensorrt_llm::DataType::kINT32); + mPenaltyWorkspacePrevDevice = mBufferManager->gpu(workspaceSize, nvinfer1::DataType::kINT32); } } @@ -112,27 +111,27 @@ void PenaltyLayer<T>::allocateBuffer() if (mDecodingMode.isUseTemperature()) { - mTemperatureDevice = mBufferManager->gpu(batchSizeShape, tensorrt_llm::DataType::kFLOAT); + mTemperatureDevice = mBufferManager->gpu(batchSizeShape, nvinfer1::DataType::kFLOAT); } if (mDecodingMode.isUseRepetitionPenalty()) { - mRepetitionPenaltyDevice = mBufferManager->gpu(batchSizeShape, tensorrt_llm::DataType::kFLOAT); + mRepetitionPenaltyDevice = mBufferManager->gpu(batchSizeShape, nvinfer1::DataType::kFLOAT); } if (mDecodingMode.isUsePresencePenalty()) { - mPresencePenaltyDevice = mBufferManager->gpu(batchSizeShape, tensorrt_llm::DataType::kFLOAT); + mPresencePenaltyDevice = mBufferManager->gpu(batchSizeShape, nvinfer1::DataType::kFLOAT); } if (mDecodingMode.isUseFrequencyPenalty()) { - mFrequencyPenaltyDevice = mBufferManager->gpu(batchSizeShape, tensorrt_llm::DataType::kFLOAT); + mFrequencyPenaltyDevice = mBufferManager->gpu(batchSizeShape, nvinfer1::DataType::kFLOAT); } if (mDecodingMode.isUseMinLength()) { - mMinLengthDevice = mBufferManager->gpu(batchSizeShape, tensorrt_llm::DataType::kINT32); + mMinLengthDevice = mBufferManager->gpu(batchSizeShape, nvinfer1::DataType::kINT32); } if (mDecodingMode.isUseOccurrencePenalty()) { - mPromptIgnoreLengthDevice = mBufferManager->gpu(batchSizeShape, tensorrt_llm::DataType::kINT32); + mPromptIgnoreLengthDevice = mBufferManager->gpu(batchSizeShape, nvinfer1::DataType::kINT32); } auto const logitsPtrDeviceDesc = std::make_pair(batchSizeShape, TRTDataType<T*>::value); diff --git a/cpp/tensorrt_llm/nanobind/CMakeLists.txt b/cpp/tensorrt_llm/nanobind/CMakeLists.txt index 4d6fbf9c2607..b523ae193871 100755 --- a/cpp/tensorrt_llm/nanobind/CMakeLists.txt +++ b/cpp/tensorrt_llm/nanobind/CMakeLists.txt @@ -14,6 +14,7 @@ set(SRCS batch_manager/llmRequest.cpp common/tllmExceptions.cpp executor/bindings.cpp + executor/executor.cpp executor/executorConfig.cpp executor/request.cpp process_group/bindings.cpp @@ -22,6 +23,7 @@ set(SRCS runtime/moeBindings.cpp suffixAutomaton/bindings.cpp testing/kvCacheManagerTestUtilBinding.cpp + testing/modelSpecBinding.cpp userbuffers/bindings.cpp thop/bindings.cpp ../runtime/ipcNvlsMemory.cu diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/algorithms.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/algorithms.cpp index c13466565342..4070811b2d72 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/algorithms.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/algorithms.cpp @@ -23,11 +23,11 @@ #include "tensorrt_llm/batch_manager/createNewDecoderRequests.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/logitsPostProcessor.h" #include "tensorrt_llm/batch_manager/medusaBuffers.h" #include "tensorrt_llm/batch_manager/microBatchScheduler.h" #include "tensorrt_llm/batch_manager/pauseRequests.h" #include "tensorrt_llm/batch_manager/peftCacheManager.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/nanobind/common/customCasters.h" #include "tensorrt_llm/runtime/decoderState.h" #include "tensorrt_llm/runtime/torch.h" @@ -129,6 +129,13 @@ void tensorrt_llm::nanobind::batch_manager::algorithms::initBindings(nb::module_ nb::call_guard<nb::gil_scoped_release>()) .def("name", [](AllocateKvCache const&) { return AllocateKvCache::name; }); + nb::class_<LogitsPostProcessor>(m, LogitsPostProcessor::name) + .def(nb::init<>()) + .def("__call__", &LogitsPostProcessor::operator(), nb::arg("decoder_input_buffers"), + nb::arg("replicate_logits_post_processor"), nb::arg("world_config"), nb::arg("stream"), + nb::arg("logits_post_processor_batched") = std::nullopt) + .def("name", [](LogitsPostProcessor const&) { return LogitsPostProcessor::name; }); + nb::class_<CreateNewDecoderRequests>(m, CreateNewDecoderRequests::name) .def(nb::init<bool, bool, bool>(), nb::arg("speculative_decoding_fast_logits"), nb::arg("is_leader_in_orch_mode"), nb::arg("is_normalize_log_probs")) @@ -136,7 +143,7 @@ void tensorrt_llm::nanobind::batch_manager::algorithms::initBindings(nb::module_ "__call__", [](CreateNewDecoderRequests& self, tr::ModelConfig const& modelConfig, tr::WorldConfig const& worldConfig, executor::DecodingConfig const& decodingConfig, RequestVector const& contextRequests, - tensorrt_llm::DataType logitsType, DecoderInputBuffers& inputBuffers, + nvinfer1::DataType logitsType, DecoderInputBuffers& inputBuffers, runtime::decoder::DecoderState& decoderState, tensorrt_llm::runtime::CudaStream const& runtimeStream, tensorrt_llm::runtime::CudaStream const& decoderStream, SizeType32 maxSequenceLength, SizeType32 beamWidth) diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp index 3c1823e2a94a..0846663dafad 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp @@ -24,7 +24,6 @@ #include "tensorrt_llm/batch_manager/peftCacheManager.h" #include "tensorrt_llm/batch_manager/rnnStateManager.h" #include "tensorrt_llm/batch_manager/sequenceSlotManager.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/nanobind/common/bindTypes.h" #include "tensorrt_llm/runtime/gptDecoderBatched.h" #include "tensorrt_llm/runtime/runtimeKernels.h" @@ -174,8 +173,6 @@ void initBindings(nb::module_& m) nb::arg("kv_tokens_per_block")) .def_prop_rw( "estimated_reusable_tokens", &GenLlmReq::getEstimatedReusableTokens, &GenLlmReq::setEstimatedReusableTokens) - .def_prop_rw( - "expect_snapshot_points", &GenLlmReq::getExpectedSnapshotPoints, &GenLlmReq::setExpectedSnapshotPoints) .def_prop_rw("guided_decoding_params", &GenLlmReq::getGuidedDecodingParams, &GenLlmReq::setGuidedDecodingParams) .def_prop_rw("context_phase_params", &GenLlmReq::getContextPhaseParams, &GenLlmReq::setContextPhaseParams) .def_prop_ro("is_context_only_request", &GenLlmReq::isContextOnlyRequest) @@ -196,8 +193,6 @@ void initBindings(nb::module_& m) .def_prop_ro("kv_cache_transfer_time_ms", &GenLlmReq::getKvCacheTransferTimeMS) .def_prop_ro("kv_cache_transfer_start", &GenLlmReq::getKvCacheTransferStart) .def_prop_ro("kv_cache_transfer_end", &GenLlmReq::getKvCacheTransferEnd) - .def("get_kv_cache_transfer_start", &GenLlmReq::getKvCacheTransferStart) - .def("get_kv_cache_transfer_end", &GenLlmReq::getKvCacheTransferEnd) .def_prop_ro("kv_cache_size", &GenLlmReq::getKvCacheSize) .def("set_kv_cache_transfer_start", &GenLlmReq::setKvCacheTransferStart, nb::arg("time")) .def("set_kv_cache_transfer_end", &GenLlmReq::setKvCacheTransferEnd, nb::arg("time")) @@ -481,11 +476,7 @@ void initBindings(nb::module_& m) .def("set_first_scheduled_time", &tb::LlmRequest::setFirstScheduledTime) .def("update_perf_metrics", &tb::LlmRequest::updatePerfMetrics, nb::arg("iter_counter")) .def("remove_lora_tensors", &tb::LlmRequest::removeLoraTensors) - // Bind to the single storage owned by libtensorrt_llm.so (reached through - // globalSteadyClockOffset()) instead of an inline-static member, so the - // offset is shared with the native library rather than living in this - // module's separate copy. - .def_rw_static("global_steady_clock_offset", &tb::globalSteadyClockOffset()); + .def_rw_static("global_steady_clock_offset", &tb::LlmRequest::sGlobalSteadyClockOffset); nb::class_<tb::SequenceSlotManager>(m, "SequenceSlotManager") .def(nb::init<tb::SequenceSlotManager::SlotIdType, uint64_t>(), nb::arg("max_num_slots"), @@ -500,7 +491,7 @@ void initBindings(nb::module_& m) nb::arg("max_num_sequences"), nb::arg("model_config"), nb::arg("world_config"), nb::arg("buffer_manager"), nb::call_guard<nb::gil_scoped_release>()) .def(nb::init<tr::SizeType32, tr::SizeType32, tr::SizeType32, tr::SizeType32, tr::SizeType32, tr::SizeType32, - tr::WorldConfig const&, int64_t, tensorrt_llm::DataType, tensorrt_llm::DataType, + tr::WorldConfig const&, int64_t, nvinfer1::DataType, nvinfer1::DataType, std::vector<tr::SizeType32> const&, tr::SizeType32>(), nb::arg("d_state"), nb::arg("d_conv"), nb::arg("num_heads"), nb::arg("n_groups"), nb::arg("head_dim"), nb::arg("max_batch_size"), nb::arg("world_config"), nb::arg("stream"), nb::arg("dtype"), diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/cacheTransceiver.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/cacheTransceiver.cpp index 2268e6f8ed03..5257c7adb1c9 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/cacheTransceiver.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/cacheTransceiver.cpp @@ -20,7 +20,6 @@ #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/batch_manager/rnnStateManager.h" #include "tensorrt_llm/common/bindingUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/nanobind/common/customCasters.h" #include <ATen/ATen.h> @@ -89,15 +88,8 @@ void tb::CacheTransceiverBindings::initBindings(nb::module_& m) { nb::class_<tb::BaseCacheTransceiver, PyCacheTransceiver>(m, "BaseCacheTransceiver") .def("respond_and_send_async", &BaseCacheTransceiver::respondAndSendAsync) - .def("request_and_receive_sync", &BaseCacheTransceiver::requestAndReceiveSync, - nb::call_guard<nb::gil_scoped_release>()) + .def("request_and_receive_sync", &BaseCacheTransceiver::requestAndReceiveSync) .def("request_and_receive_async", &BaseCacheTransceiver::requestAndReceiveAsync) - .def("get_serialized_data_transceiver_state", - [](tb::BaseCacheTransceiver& self) - { - auto serialized = self.getSerializedDataTransceiverState(); - return nb::bytes(serialized.data(), serialized.size()); - }) .def( "check_context_transfer_status", [](tb::BaseCacheTransceiver& self, std::optional<int> const& atLeastRequestNum, bool markComplete = false) @@ -118,8 +110,7 @@ void tb::CacheTransceiverBindings::initBindings(nb::module_& m) .def("check_gen_transfer_status", &BaseCacheTransceiver::checkGenTransferStatus, nb::call_guard<nb::gil_scoped_release>()) .def("check_gen_transfer_complete", &BaseCacheTransceiver::checkGenTransferComplete) - .def("cancel_request", &BaseCacheTransceiver::cancelRequest) - .def("has_poisoned_transfer_buffer", &BaseCacheTransceiver::hasPoisonedTransferBuffer); + .def("cancel_request", &BaseCacheTransceiver::cancelRequest); nb::enum_<executor::kv_cache::CacheState::AttentionType>(m, "AttentionType") .value("DEFAULT", executor::kv_cache::CacheState::AttentionType::kDEFAULT) @@ -127,13 +118,13 @@ void tb::CacheTransceiverBindings::initBindings(nb::module_& m) nb::class_<tb::CacheTransceiver, tb::BaseCacheTransceiver>(m, "CacheTransceiver") .def(nb::init<tb::kv_cache_manager::BaseKVCacheManager*, std::vector<SizeType32>, SizeType32, SizeType32, - runtime::WorldConfig, std::vector<SizeType32>, tensorrt_llm::DataType, + runtime::WorldConfig, std::vector<SizeType32>, nvinfer1::DataType, executor::kv_cache::CacheState::AttentionType, std::optional<executor::CacheTransceiverConfig>, - std::vector<SizeType32>>(), + tb::rnn_state_manager::RnnStateManager*, std::vector<SizeType32>>(), nb::arg("cache_manager"), nb::arg("num_kv_heads_per_layer"), nb::arg("size_per_head"), nb::arg("tokens_per_block"), nb::arg("world_config"), nb::arg("attention_layer_num_per_pp"), nb::arg("dtype"), nb::arg("attention_type"), nb::arg("cache_transceiver_config") = std::nullopt, - nb::arg("rnn_layer_num_per_pp") = std::vector<SizeType32>{}); + nb::arg("rnn_state_manager") = nullptr, nb::arg("rnn_layer_num_per_pp") = std::vector<SizeType32>{}); nb::class_<tb::CacheTransceiverComm>(m, "CacheTransceiverComm") .def( diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.cpp index b1c4391c0e26..1496ebbb883d 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.cpp @@ -18,7 +18,6 @@ #include "kvCacheManager.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/batch_manager/peftCacheManager.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/nanobind/common/bindTypes.h" #include "tensorrt_llm/nanobind/common/customCasters.h" #include "tensorrt_llm/runtime/torch.h" @@ -349,8 +348,8 @@ void tb::kv_cache_manager::KVCacheManagerBindings::initBindings(nb::module_& m) nb::class_<tbk::PoolConfiguration>(m, "PoolConfiguration") .def(nb::init<>()) - .def(nb::init<SizeType32, SizeType32, tensorrt_llm::DataType>(), nb::arg("window_size"), - nb::arg("size_per_head"), nb::arg("dtype")) + .def(nb::init<SizeType32, SizeType32, nvinfer1::DataType>(), nb::arg("window_size"), nb::arg("size_per_head"), + nb::arg("dtype")) .def_rw("window_size", &tbk::PoolConfiguration::windowSize) .def_rw("size_per_head", &tbk::PoolConfiguration::sizePerHead) .def_rw("dtype", &tbk::PoolConfiguration::dtype); @@ -662,8 +661,8 @@ void tb::kv_cache_manager::KVCacheManagerBindings::initBindings(nb::module_& m) nb::class_<tbk::KVCacheManager, tbk::BaseKVCacheManager>(m, "KVCacheManager") .def(nb::init<std::vector<SizeType32> const&, SizeType32, SizeType32, std::map<SizeType32, std::tuple<SizeType32, SizeType32>> const&, SizeType32, SizeType32, - std::vector<SizeType32> const&, tensorrt_llm::DataType, SizeType32, int64_t, SizeType32, SizeType32, - bool, tbk::CacheType, std::optional<tensorrt_llm::executor::RetentionPriority>, + std::vector<SizeType32> const&, nvinfer1::DataType, SizeType32, int64_t, SizeType32, SizeType32, bool, + tbk::CacheType, std::optional<tensorrt_llm::executor::RetentionPriority>, std::shared_ptr<tbk::KVCacheEventManager>, bool, bool, std::shared_ptr<tbc::KvCacheConnectorManager>, bool, SizeType32, SizeType32, bool, std::optional<tbk::LinearAttentionMetadata>, std::vector<tbk::PoolConfiguration> const&>(), @@ -694,9 +693,7 @@ void tb::kv_cache_manager::KVCacheManagerBindings::initBindings(nb::module_& m) .def("copy_linear_attention_block", &tbk::KVCacheManager::copyLinearAttentionBlock, nb::arg("llm_request"), nb::call_guard<nb::gil_scoped_release>()) .def("copy_linear_attention_block_batch", &tbk::KVCacheManager::copyLinearAttentionBlockBatch, - nb::arg("llm_requests"), nb::call_guard<nb::gil_scoped_release>()) - .def("get_memory_pool_block_indices", &tbk::KVCacheManager::getMemoryPoolBlockIndicesByBlockIds, - nb::arg("block_ids"), nb::arg("window_size"), nb::call_guard<nb::gil_scoped_release>()); + nb::arg("llm_requests"), nb::call_guard<nb::gil_scoped_release>()); } void tb::BasePeftCacheManagerBindings::initBindings(nb::module_& m) diff --git a/cpp/tensorrt_llm/nanobind/bindings.cpp b/cpp/tensorrt_llm/nanobind/bindings.cpp index 2c724adc279c..db263fa639f9 100644 --- a/cpp/tensorrt_llm/nanobind/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/bindings.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -32,7 +32,6 @@ #include "tensorrt_llm/batch_manager/peftCacheManagerConfig.h" #include "tensorrt_llm/common/quantization.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/nanobind/batch_manager/algorithms.h" #include "tensorrt_llm/nanobind/batch_manager/bindings.h" #include "tensorrt_llm/nanobind/batch_manager/buffers.h" @@ -47,6 +46,7 @@ #include "tensorrt_llm/nanobind/runtime/bindings.h" #include "tensorrt_llm/nanobind/suffixAutomaton/bindings.h" #include "tensorrt_llm/nanobind/testing/kvCacheManagerTestUtilBinding.h" +#include "tensorrt_llm/nanobind/testing/modelSpecBinding.h" #include "tensorrt_llm/nanobind/thop/bindings.h" #include "tensorrt_llm/nanobind/userbuffers/bindings.h" #include "tensorrt_llm/runtime/common.h" @@ -168,17 +168,17 @@ NB_MODULE(TRTLLM_NB_MODULE, m) .def_rw("host_cache_size", &tb::PeftCacheManagerConfig::hostCacheSize) .def_rw("lora_prefetch_dir", &tb::PeftCacheManagerConfig::loraPrefetchDir); - nb::enum_<tensorrt_llm::DataType>(m, "DataType") - .value("FLOAT", tensorrt_llm::DataType::kFLOAT) - .value("HALF", tensorrt_llm::DataType::kHALF) - .value("INT8", tensorrt_llm::DataType::kINT8) - .value("INT32", tensorrt_llm::DataType::kINT32) - .value("BOOL", tensorrt_llm::DataType::kBOOL) - .value("UINT8", tensorrt_llm::DataType::kUINT8) - .value("FP8", tensorrt_llm::DataType::kFP8) - .value("BF16", tensorrt_llm::DataType::kBF16) - .value("INT64", tensorrt_llm::DataType::kINT64) - .value("NVFP4", tensorrt_llm::DataType::kFP4) + nb::enum_<nvinfer1::DataType>(m, "DataType") + .value("FLOAT", nvinfer1::DataType::kFLOAT) + .value("HALF", nvinfer1::DataType::kHALF) + .value("INT8", nvinfer1::DataType::kINT8) + .value("INT32", nvinfer1::DataType::kINT32) + .value("BOOL", nvinfer1::DataType::kBOOL) + .value("UINT8", nvinfer1::DataType::kUINT8) + .value("FP8", nvinfer1::DataType::kFP8) + .value("BF16", nvinfer1::DataType::kBF16) + .value("INT64", nvinfer1::DataType::kINT64) + .value("NVFP4", nvinfer1::DataType::kFP4) .export_values(); nb::enum_<tr::ModelConfig::ModelVariant>(m, "GptModelVariant") @@ -295,7 +295,7 @@ NB_MODULE(TRTLLM_NB_MODULE, m) .def(nb::self != nb::self); nb::class_<tr::ModelConfig>(m, "ModelConfig") - .def(nb::init<SizeType32, SizeType32, SizeType32, SizeType32, SizeType32, SizeType32, tensorrt_llm::DataType>(), + .def(nb::init<SizeType32, SizeType32, SizeType32, SizeType32, SizeType32, SizeType32, nvinfer1::DataType>(), nb::arg("vocab_size"), nb::arg("num_layers"), nb::arg("num_attention_layers"), nb::arg("num_rnn_layers"), nb::arg("num_heads"), nb::arg("hidden_size"), nb::arg("data_type")) .def_prop_ro("vocab_size", &tr::ModelConfig::getVocabSize) @@ -512,6 +512,7 @@ NB_MODULE(TRTLLM_NB_MODULE, m) tensorrt_llm::nanobind::process_group::initBindings(mInternalProcessGroup); tpb::Buffers::initBindings(mInternalBatchManager); tensorrt_llm::nanobind::runtime::initBindings(mInternalRuntime); + tensorrt_llm::nanobind::testing::initBindings(mInternalTesting); tensorrt_llm::nanobind::testing::initKvCacheTestUtilBindings(mInternalTesting); tpb::initBindings(mInternalBatchManager); @@ -544,8 +545,4 @@ NB_MODULE(TRTLLM_NB_MODULE, m) m.def("ipc_nvls_supported", &tr::ipcNvlsSupported); m.def("steady_clock_now", []() { return std::chrono::steady_clock::now(); }); - // Global (offset-normalized) steady clock, matching what - // LlmRequest::setKvCacheTransferStart/End expect. Reads the process-global - // steady clock offset, set by PyExecutor at startup. - m.def("global_steady_clock_now", []() { return tb::LlmRequest::getSteadyClockNow(); }); } diff --git a/cpp/tensorrt_llm/nanobind/executor/bindings.cpp b/cpp/tensorrt_llm/nanobind/executor/bindings.cpp index a8d2301fa43d..b0ad31b7347e 100644 --- a/cpp/tensorrt_llm/nanobind/executor/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/executor/bindings.cpp @@ -16,6 +16,7 @@ */ #include "bindings.h" +#include "executor.h" #include "executorConfig.h" #include "request.h" #include "tensorrt_llm/executor/executor.h" @@ -286,6 +287,7 @@ void initBindings(nb::module_& m) tensorrt_llm::nanobind::executor::initRequestBindings(m); tensorrt_llm::nanobind::executor::initConfigBindings(m); + tensorrt_llm::nanobind::executor::Executor::initBindings(m); } } // namespace tensorrt_llm::nanobind::executor diff --git a/cpp/tensorrt_llm/nanobind/executor/executor.cpp b/cpp/tensorrt_llm/nanobind/executor/executor.cpp new file mode 100644 index 000000000000..34cc8182d1bb --- /dev/null +++ b/cpp/tensorrt_llm/nanobind/executor/executor.cpp @@ -0,0 +1,225 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "executor.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/tensor.h" +#include "tensorrt_llm/nanobind/common/customCasters.h" + +#include <nanobind/nanobind.h> +#include <nanobind/ndarray.h> +#include <nanobind/stl/chrono.h> +#include <nanobind/stl/filesystem.h> +#include <nanobind/stl/map.h> +#include <nanobind/stl/optional.h> +#include <nanobind/stl/shared_ptr.h> +#include <nanobind/stl/string.h> +#include <nanobind/stl/vector.h> +#include <torch/extension.h> + +namespace nb = nanobind; +namespace tle = tensorrt_llm::executor; + +namespace nanobind::detail +{ + +template <> +struct dtype_traits<half> +{ + static constexpr dlpack::dtype value{ + (uint8_t) dlpack::dtype_code::Float, // type code + 16, // size in bits + 1 // lanes (simd), usually set to 1 + }; + static constexpr auto name = const_name("float16"); +}; +} // namespace nanobind::detail + +namespace +{ +tle::Tensor numpyToTensor(nb::object const& object) +{ + std::string dtype_name = nb::cast<std::string>(object.attr("dtype").attr("name")); + nb::object metadata = object.attr("dtype").attr("metadata"); + + tle::DataType dtype; + if (dtype_name == "float16") + { + dtype = tle::DataType::kFP16; + } + else if (dtype_name == "float32") + { + dtype = tle::DataType::kFP32; + } + else if (dtype_name == "int8") + { + dtype = tle::DataType::kINT8; + } + else if (dtype_name == "int32") + { + dtype = tle::DataType::kINT32; + } + else if (dtype_name == "int64") + { + dtype = tle::DataType::kINT64; + } + else if (dtype_name == "void8" && !metadata.is_none() && nb::cast<std::string>(metadata["dtype"]) == "float8") + { + dtype = tle::DataType::kFP8; + } + else if (dtype_name == "void16" && !metadata.is_none() && nb::cast<std::string>(metadata["dtype"]) == "bfloat16") + { + dtype = tle::DataType::kBF16; + } + else + { + TLLM_THROW("Unsupported numpy dtype."); + } + + nb::object array_interface = object.attr("__array_interface__"); + nb::object shape_obj = array_interface["shape"]; + std::vector<int64_t> dims; + dims.reserve(nb::len(shape_obj)); + + for (size_t i = 0; i < nb::len(shape_obj); ++i) + { + dims.push_back(nb::cast<int64_t>(shape_obj[i])); + } + + nb::object data_obj = array_interface["data"]; + uintptr_t addr = nb::cast<uintptr_t>(data_obj[0]); + void* data_ptr = reinterpret_cast<void*>(addr); + tle::Shape shape(dims.data(), dims.size()); + return tle::Tensor::of(dtype, data_ptr, shape); +} + +} // namespace + +namespace tensorrt_llm::nanobind::executor +{ + +Executor::Executor( + std::filesystem::path const& modelPath, tle::ModelType modelType, tle::ExecutorConfig const& executorConfig) +{ + mExecutor = std::make_unique<tle::Executor>(modelPath, modelType, executorConfig); +} + +Executor::Executor(std::filesystem::path const& encoderModelPath, std::filesystem::path const& decoderModelPath, + tle::ModelType modelType, tle::ExecutorConfig const& executorConfig) +{ + mExecutor = std::make_unique<tle::Executor>(encoderModelPath, decoderModelPath, modelType, executorConfig); +} + +Executor::Executor(nb::bytes const& engineBuffer, std::string const& jsonConfigStr, tle::ModelType modelType, + tle::ExecutorConfig const& executorConfig, std::optional<nb::dict> managedWeights) +{ + uint8_t const* data = static_cast<uint8_t const*>(engineBuffer.data()); + size_t size = engineBuffer.size(); + std::optional<std::map<std::string, tle::Tensor>> managedWeightsMap = std::nullopt; + if (managedWeights.has_value() && !managedWeights.value().empty()) + { + managedWeightsMap = std::map<std::string, tle::Tensor>(); + for (auto const& [rawName, rawArray] : managedWeights.value()) + { + std::string name = nb::cast<std::string>(rawName); + nb::object array_obj = nb::cast<nb::object>(rawArray); + managedWeightsMap->emplace(name, numpyToTensor(array_obj)); + } + } + mExecutor = std::make_unique<tle::Executor>( + tle::BufferView(data, size), jsonConfigStr, modelType, executorConfig, managedWeightsMap); +} + +Executor::Executor(std::string const& encoderEngineBuffer, std::string const& encoderJsonConfigStr, + std::string const& decoderEngineBuffer, std::string const& decoderJsonConfigStr, tle::ModelType modelType, + tle::ExecutorConfig const& executorConfig) +{ + uint8_t const* encoderData = reinterpret_cast<uint8_t const*>(encoderEngineBuffer.data()); + size_t encoderSize = encoderEngineBuffer.size(); + uint8_t const* decoderData = reinterpret_cast<uint8_t const*>(decoderEngineBuffer.data()); + size_t decoderSize = decoderEngineBuffer.size(); + mExecutor = std::make_unique<tle::Executor>(tle::BufferView(encoderData, encoderSize), encoderJsonConfigStr, + tle::BufferView(decoderData, decoderSize), decoderJsonConfigStr, modelType, executorConfig); +} + +nb::object Executor::enter() +{ + TLLM_CHECK(static_cast<bool>(mExecutor)); + return nb::cast(this); +} + +void Executor::exit( + [[maybe_unused]] nb::handle type, [[maybe_unused]] nb::handle value, [[maybe_unused]] nb::handle traceback) +{ + shutdown(); + mExecutor = nullptr; +} + +void Executor::shutdown() +{ + // NOTE: we must release the GIL here. Executor has spawned a thread for the execution loop. That thread must be + // able to do forward progress for the shutdown process to succeed. It takes the GIL during its callbacks, so + // we release it now. Note that we shouldn't do anything related to python objects after that. + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + nb::gil_scoped_release release; + mExecutor->shutdown(); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void Executor::initBindings(nb::module_& m) +{ + nb::class_<Executor>(m, "Executor") + .def(nb::init<std::filesystem::path const&, tle::ModelType, tle::ExecutorConfig const&>(), + nb::arg("model_path"), nb::arg("model_type"), nb::arg("executor_config")) + .def(nb::init<std::filesystem::path const&, std::filesystem::path const&, tle::ModelType, + tle::ExecutorConfig const&>(), + nb::arg("encoder_model_path"), nb::arg("decoder_model_path"), nb::arg("model_type"), + nb::arg("executor_config")) + .def(nb::init<nb::bytes, std::string const&, tle::ModelType, tle::ExecutorConfig const&, nb::dict>(), + nb::arg("engine_buffer"), nb::arg("json_config_str"), nb::arg("model_type"), nb::arg("executor_config"), + nb::arg("managed_weights") = nb::dict()) + .def(nb::init<std::string const&, std::string const&, std::string const&, std::string const&, tle::ModelType, + tle::ExecutorConfig const&>(), + nb::arg("encoder_engine_buffer"), nb::arg("encoder_json_config_str"), nb::arg("decoder_engine_buffer"), + nb::arg("decoder_json_config_str"), nb::arg("model_type"), nb::arg("executor_config")) + .def("shutdown", &Executor::shutdown) + .def("__enter__", &Executor::enter) + .def("__exit__", &Executor::exit, nb::arg("type").none(), nb::arg("value").none(), nb::arg("traceback").none()) + .def("enqueue_request", &Executor::enqueueRequest, nb::arg("request")) + .def("enqueue_requests", &Executor::enqueueRequests, nb::arg("requests")) + .def("await_responses", + nb::overload_cast<std::optional<std::chrono::milliseconds> const&>(&Executor::awaitResponses), + nb::arg("timeout") = nb::none()) + .def("await_responses", + nb::overload_cast<tle::IdType const&, std::optional<std::chrono::milliseconds> const&>( + &Executor::awaitResponses), + nb::arg("id"), nb::arg("timeout") = nb::none()) + .def("await_responses", + nb::overload_cast<std::vector<tle::IdType> const&, std::optional<std::chrono::milliseconds> const&>( + &Executor::awaitResponses), + nb::arg("ids"), nb::arg("timeout") = nb::none()) + .def("get_num_responses_ready", &Executor::getNumResponsesReady, nb::arg("id") = nb::none()) + .def("cancel_request", &Executor::cancelRequest, nb::arg("id") = nb::none()) + .def("get_latest_iteration_stats", &Executor::getLatestIterationStats) + .def("get_latest_request_stats", &Executor::getLatestRequestStats) + .def("get_latest_debug_tensors", &Executor::getLatestDebugTensors) + .def("can_enqueue_requests", &Executor::canEnqueueRequests) + .def("get_kv_cache_event_manager", &Executor::getKVCacheEventManager); +} + +} // namespace tensorrt_llm::nanobind::executor diff --git a/cpp/tensorrt_llm/nanobind/executor/executor.h b/cpp/tensorrt_llm/nanobind/executor/executor.h new file mode 100644 index 000000000000..22c24abb4bfd --- /dev/null +++ b/cpp/tensorrt_llm/nanobind/executor/executor.h @@ -0,0 +1,129 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include <nanobind/nanobind.h> + +namespace nb = nanobind; +namespace tle = tensorrt_llm::executor; + +namespace tensorrt_llm::nanobind::executor +{ + +class Executor +{ +public: + Executor( + std::filesystem::path const& modelPath, tle::ModelType modelType, tle::ExecutorConfig const& executorConfig); + + Executor(std::filesystem::path const& encoderModelPath, std::filesystem::path const& decoderModelPath, + tle::ModelType modelType, tle::ExecutorConfig const& executorConfig); + + Executor(nb::bytes const& engineBuffer, std::string const& jsonConfigStr, tle::ModelType modelType, + tle::ExecutorConfig const& executorConfig, std::optional<nb::dict> managedWeights); + + Executor(std::string const& encoderEngineBuffer, std::string const& encoderJsonConfigStr, + std::string const& decoderEngineBuffer, std::string const& decoderJsonConfigStr, tle::ModelType modelType, + tle::ExecutorConfig const& executorConfig); + + nb::object enter(); + void exit( + [[maybe_unused]] nb::handle type, [[maybe_unused]] nb::handle value, [[maybe_unused]] nb::handle traceback); + void shutdown(); + + [[nodiscard]] tle::IdType enqueueRequest(tle::Request const& request) + { + return mExecutor->enqueueRequest(request); + } + + [[nodiscard]] std::vector<tle::IdType> enqueueRequests(std::vector<tle::Request> const& requests) + { + return mExecutor->enqueueRequests(requests); + } + + [[nodiscard]] std::vector<tle::Response> awaitResponses( + std::optional<std::chrono::milliseconds> const& timeout = std::nullopt) + { + // Await responses blocks until a response is received. Release GIL so that it can be ran in a background + // thread. + nb::gil_scoped_release release; + return mExecutor->awaitResponses(timeout); + } + + [[nodiscard]] std::vector<tle::Response> awaitResponses( + tle::IdType const& requestId, std::optional<std::chrono::milliseconds> const& timeout = std::nullopt) + { + // Await responses blocks until a response is received. Release GIL so that it can be ran in a background + // thread. + nb::gil_scoped_release release; + return mExecutor->awaitResponses(requestId, timeout); + } + + [[nodiscard]] std::vector<std::vector<tle::Response>> awaitResponses(std::vector<tle::IdType> const& requestIds, + std::optional<std::chrono::milliseconds> const& timeout = std::nullopt) + { + // Await responses blocks until a response is received. Release GIL so that it can be ran in a background + // thread. + nb::gil_scoped_release release; + return mExecutor->awaitResponses(requestIds, timeout); + } + + [[nodiscard]] tle::SizeType32 getNumResponsesReady(std::optional<tle::IdType> const& requestId = std::nullopt) const + { + return mExecutor->getNumResponsesReady(requestId); + } + + void cancelRequest(tle::IdType requestId) + { + mExecutor->cancelRequest(requestId); + } + + std::deque<tle::IterationStats> getLatestIterationStats() + { + return mExecutor->getLatestIterationStats(); + } + + std::deque<tle::RequestStatsPerIteration> getLatestRequestStats() + { + return mExecutor->getLatestRequestStats(); + } + + std::deque<tle::DebugTensorsPerIteration> getLatestDebugTensors() + { + return mExecutor->getLatestDebugTensors(); + } + + [[nodiscard]] bool canEnqueueRequests() const + { + return mExecutor->canEnqueueRequests(); + } + + [[nodiscard]] std::optional<std::shared_ptr<tle::KVCacheEventManager>> getKVCacheEventManager() const + { + return mExecutor->getKVCacheEventManager(); + } + + static void initBindings(nb::module_& m); + +private: + std::unique_ptr<tle::Executor> mExecutor; +}; + +} // namespace tensorrt_llm::nanobind::executor diff --git a/cpp/tensorrt_llm/nanobind/executor/executorConfig.cpp b/cpp/tensorrt_llm/nanobind/executor/executorConfig.cpp index acd33c0df769..830f30ab9c67 100644 --- a/cpp/tensorrt_llm/nanobind/executor/executorConfig.cpp +++ b/cpp/tensorrt_llm/nanobind/executor/executorConfig.cpp @@ -171,9 +171,6 @@ void initConfigBindings(nb::module_& m) .def("__getstate__", kvCacheConfigGetstate) .def("__setstate__", kvCacheConfigSetstate); - // Deprecated: orchestrator mode is non-functional (its executorWorker binary was - // removed with the TensorRT backend); binding kept for compatibility, removal is a - // follow-up pending API-stability review. nb::class_<tle::OrchestratorConfig>(m, "OrchestratorConfig") .def(nb::init<bool, std::string, std::shared_ptr<mpi::MpiComm>, bool>(), nb::arg("is_orchestrator") = true, nb::arg("worker_executable_path") = "", nb::arg("orch_leader_comm").none() = nullptr, diff --git a/cpp/tensorrt_llm/nanobind/process_group/bindings.cpp b/cpp/tensorrt_llm/nanobind/process_group/bindings.cpp index e713cfa25b49..235d64a8f88d 100644 --- a/cpp/tensorrt_llm/nanobind/process_group/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/process_group/bindings.cpp @@ -38,8 +38,6 @@ void initBindings(nb::module_& m) pg_utils::init_pg(common::get_intrusive_ptr<Pg, E>(world_pg_obj.ptr(), pybind11_abi), common::get_intrusive_ptr<Pg, E>(local_pg_obj.ptr(), pybind11_abi)); }); - - m.def("shutdown_pg", []() { pg_utils::shutdown_pg(); }); } } // namespace tensorrt_llm::nanobind::process_group diff --git a/cpp/tensorrt_llm/nanobind/runtime/bindings.cpp b/cpp/tensorrt_llm/nanobind/runtime/bindings.cpp index eec3cd79bac1..6d5d70aafb6b 100644 --- a/cpp/tensorrt_llm/nanobind/runtime/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/runtime/bindings.cpp @@ -18,7 +18,6 @@ #include "bindings.h" #include "hostfunc.h" #include "moeBindings.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/allReduceWorkspace.h" #include "tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.h" #include "tensorrt_llm/kernels/customAllReduceKernels.h" @@ -40,6 +39,7 @@ #include "tensorrt_llm/runtime/loraCache.h" #include "tensorrt_llm/runtime/mcastGPUBuffer.h" #include "tensorrt_llm/runtime/speculativeDecodingMode.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" #include "tensorrt_llm/runtime/torchView.h" #include "tensorrt_llm/runtime/virtualMemory.h" @@ -68,7 +68,7 @@ class PyIGptDecoder : public tr::IGptDecoder void setup(tr::SamplingConfig const& samplingConfig, size_t batchSize, tr::DecodingInput::TensorConstPtr const& batchSlots, std::optional<tr::DecodingOutput> const& output = std::nullopt, - std::optional<tensorrt_llm::DataType> explicitDraftTokensDType = std::nullopt, + std::optional<nvinfer1::DataType> explicitDraftTokensDType = std::nullopt, std::optional<std::vector<tr::ITensor::SharedConstPtr>> const& lookaheadPrompt = std::nullopt, std::optional<std::vector<te::LookaheadDecodingConfig>> const& lookaheadAlgoConfigs = std::nullopt) override { @@ -125,6 +125,47 @@ void initBindings(nb::module_& m) .def("materialize_with_tag", &tr::CudaVirtualMemoryManager::materializeWithTag, nb::arg("tag"), nb::call_guard<nb::gil_scoped_release>()); + nb::class_<tr::TllmRuntime>(m, "TllmRuntime") + .def( + "__init__", + [](tr::TllmRuntime* self, std::filesystem::path engine_path, float gpu_weights_percent = 1.0f, + bool use_shape_inference = true) + { + // Using default logger by passing nullptr + new (self) + tr::TllmRuntime(tr::RawEngine(engine_path), nullptr, gpu_weights_percent, use_shape_inference); + }, + nb::arg("engine_path"), nb::arg("gpu_weights_percent") = 1.0f, nb::arg("use_shape_inference") = true) + .def( + "__init__", + [](tr::TllmRuntime* self, nb::ndarray<nb::numpy, uint8_t> engine_buffer, float gpu_weights_percent = 1.0f, + bool use_shape_inference = true) + { + if (engine_buffer.ndim() != 1) + throw std::runtime_error("Expected 1-D array for engine buffer"); + new (self) tr::TllmRuntime(tr::RawEngine(engine_buffer.data(), engine_buffer.size()), nullptr, + gpu_weights_percent, use_shape_inference); + }, + nb::arg("engine_buffer"), nb::arg("gpu_weights_percent") = 1.0f, nb::arg("use_shape_inference") = true) + .def_prop_ro("num_contexts", &tr::TllmRuntime::getNbContexts) + .def_prop_ro("num_profiles", &tr::TllmRuntime::getNbProfiles) + .def("get_opt_profile_id", &tr::TllmRuntime::getOptProfileId, nb::arg("num_tokens"), nb::arg("split_points"), + nb::call_guard<nb::gil_scoped_release>()) + .def("clear_contexts", &tr::TllmRuntime::clearContexts, nb::call_guard<nb::gil_scoped_release>()) + .def("execute_context", &tr::TllmRuntime::executeContext, nb::arg("context_id"), + nb::call_guard<nb::gil_scoped_release>()) + .def_prop_ro("stream_ptr", &tr::TllmRuntime::getStreamPtr) + .def_prop_ro("buffer_manager", + static_cast<tr::BufferManager& (tr::TllmRuntime::*) ()>(&tr::TllmRuntime::getBufferManager)) + .def("set_layer_profiler", &tr::TllmRuntime::setLayerProfiler, nb::call_guard<nb::gil_scoped_release>()) + .def("has_layer_profiler", &tr::TllmRuntime::hasLayerProfiler, nb::arg("context_id"), + nb::call_guard<nb::gil_scoped_release>()) + .def_prop_ro("layer_profiler_info", &tr::TllmRuntime::getLayerProfileInfo) + .def("report_to_profiler", &tr::TllmRuntime::reportToProfiler, nb::arg("context_id"), + nb::call_guard<nb::gil_scoped_release>()) + .def_prop_ro("logits_dtype_from_engine", + [](tr::TllmRuntime& self) { return self.getEngine().getTensorDataType("logits"); }); + nb::class_<tr::LookaheadDecodingBuffers>(m, "LookaheadDecodingBuffers") .def(nb::init<tr::SizeType32, tr::SizeType32, tr::BufferManager const&>(), nb::arg("max_num_sequences"), nb::arg("max_tokens_per_step"), nb::arg("buffer_manager"), nb::call_guard<nb::gil_scoped_release>()) @@ -163,7 +204,7 @@ void initBindings(nb::module_& m) "setup", [](tr::IGptDecoder& self, tr::SamplingConfig const& samplingConfig, size_t batchSize, at::Tensor const& batchSlots, std::optional<tr::DecodingOutput> const& output = std::nullopt, - std::optional<tensorrt_llm::DataType> explicitDraftTokensDType = std::nullopt, + std::optional<nvinfer1::DataType> explicitDraftTokensDType = std::nullopt, std::optional<std::vector<tr::ITensor::SharedConstPtr>> const& lookaheadPrompt = std::nullopt, std::optional<std::vector<te::LookaheadDecodingConfig>> const& lookaheadAlgoConfigs = std::nullopt) { diff --git a/cpp/tensorrt_llm/nanobind/runtime/hostfunc.cpp b/cpp/tensorrt_llm/nanobind/runtime/hostfunc.cpp index 58b3faee0dd4..0ff16e582191 100644 --- a/cpp/tensorrt_llm/nanobind/runtime/hostfunc.cpp +++ b/cpp/tensorrt_llm/nanobind/runtime/hostfunc.cpp @@ -21,7 +21,6 @@ #include <cuda_runtime.h> #include <memory> -#include <mutex> #include <nanobind/nanobind.h> #include <nanobind/stl/function.h> #include <nanobind/stl/optional.h> @@ -74,33 +73,15 @@ static void cudaHostFuncTrampoline(void* userData) } } -std::optional<uintptr_t> launchHostFunc(uintptr_t streamPtr, bool freeUserData, bool lowLatency, - nb::callable pyHostFunc, nb::args pyArgs, nb::kwargs pyKwargs) +std::optional<uintptr_t> launchHostFunc( + uintptr_t streamPtr, bool freeUserData, nb::callable pyHostFunc, nb::args pyArgs, nb::kwargs pyKwargs) { auto const stream = reinterpret_cast<cudaStream_t>(streamPtr); auto hostFuncUserData = std::make_unique<HostFuncUserData>(freeUserData, pyHostFunc, nb::tuple(pyArgs), nb::dict(pyKwargs)); - cudaError_t err; -#if CUDART_VERSION >= 13020 - if (lowLatency) - { - err = cudaLaunchHostFunc_v2(stream, cudaHostFuncTrampoline, hostFuncUserData.get(), cudaHostTaskSpinWait); - } - else - { - err = cudaLaunchHostFunc(stream, cudaHostFuncTrampoline, hostFuncUserData.get()); - } -#else - if (lowLatency) - { - static std::once_flag sWarnOnce; - std::call_once(sWarnOnce, - []() { TLLM_LOG_WARNING("Low-latency host task dispatch requires CUDA 13.2+; falling back to default."); }); - } - err = cudaLaunchHostFunc(stream, cudaHostFuncTrampoline, hostFuncUserData.get()); -#endif + cudaError_t err = cudaLaunchHostFunc(stream, cudaHostFuncTrampoline, hostFuncUserData.get()); if (err != cudaSuccess) { throw std::runtime_error("Failed to launch host function."); @@ -124,9 +105,7 @@ void freeHostFuncUserData(uintptr_t userDataPtr) void initHostFuncBindings(nb::module_& m) { - m.def("launch_hostfunc", &launchHostFunc, - "Launch a Python host function to a CUDA stream. Set low_latency=True to use " - "spin-wait dispatch (requires CUDA 13.2+)."); + m.def("launch_hostfunc", &launchHostFunc, "Launch a Python host function to a CUDA stream"); m.def("free_hostfunc_user_data", &freeHostFuncUserData, "Free the user data for the Python host function"); } } // namespace tensorrt_llm::nanobind::runtime diff --git a/cpp/tensorrt_llm/nanobind/testing/modelSpecBinding.cpp b/cpp/tensorrt_llm/nanobind/testing/modelSpecBinding.cpp new file mode 100644 index 000000000000..caef94c5defd --- /dev/null +++ b/cpp/tensorrt_llm/nanobind/testing/modelSpecBinding.cpp @@ -0,0 +1,87 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "modelSpecBinding.h" +#include "tensorrt_llm/nanobind/common/customCasters.h" +#include "tensorrt_llm/testing/modelSpec.h" + +#include <nanobind/nanobind.h> + +namespace nb = nanobind; +using tensorrt_llm::testing::ModelSpec; +using tensorrt_llm::testing::KVCacheType; +using tensorrt_llm::testing::QuantMethod; +using tensorrt_llm::testing::OutputContentType; + +namespace tensorrt_llm::nanobind::testing +{ + +void initBindings(nb::module_& m) +{ + nb::enum_<QuantMethod>(m, "QuantMethod", nb::is_arithmetic(), "Quantization Method") + .value("NONE", QuantMethod::kNONE, "No Quantization") + .value("SMOOTH_QUANT", QuantMethod::kSMOOTH_QUANT, "Smooth Quantization"); + + nb::enum_<OutputContentType>(m, "OutputContentType", nb::is_arithmetic(), "Output Content Type") + .value("NONE", OutputContentType::kNONE, "No Output Content") + .value("CONTEXT_LOGITS", OutputContentType::kCONTEXT_LOGITS, "Context Logits") + .value("GENERATION_LOGITS", OutputContentType::kGENERATION_LOGITS, "Generation Logits") + .value("LOG_PROBS", OutputContentType::kLOG_PROBS, "Log Probs") + .value("CUM_LOG_PROBS", OutputContentType::kCUM_LOG_PROBS, "Cumulative Log"); + + nb::class_<ModelSpec>(m, "ModelSpec") + .def(nb::init<std::string const&, nvinfer1::DataType>()) + .def("use_gpt_plugin", &ModelSpec::useGptAttentionPlugin, nb::rv_policy::reference_internal) + .def("use_packed_input", &ModelSpec::usePackedInput, nb::rv_policy::reference_internal) + .def("set_kv_cache_type", &ModelSpec::setKVCacheType, nb::rv_policy::reference_internal) + .def("use_decoder_per_request", &ModelSpec::useDecoderPerRequest, nb::rv_policy::reference_internal) + .def("use_tensor_parallelism", &ModelSpec::useTensorParallelism, nb::rv_policy::reference_internal) + .def("use_pipeline_parallelism", &ModelSpec::usePipelineParallelism, nb::rv_policy::reference_internal) + .def("use_context_parallelism", &ModelSpec::useContextParallelism, nb::rv_policy::reference_internal) + .def("set_draft_tokens", &ModelSpec::setDraftTokens, nb::rv_policy::reference_internal) + .def("use_accept_by_logits", &ModelSpec::useAcceptByLogits, nb::rv_policy::reference_internal) + .def("use_mamba_plugin", &ModelSpec::useMambaPlugin, nb::rv_policy::reference_internal) + .def("gather_logits", &ModelSpec::gatherLogits, nb::rv_policy::reference_internal) + .def("replace_logits", &ModelSpec::replaceLogits, nb::rv_policy::reference_internal) + .def("return_log_probs", &ModelSpec::returnLogProbs, nb::rv_policy::reference_internal) + .def("smoke_test", &ModelSpec::smokeTest, nb::rv_policy::reference_internal) + .def("use_medusa", &ModelSpec::useMedusa, nb::rv_policy::reference_internal) + .def("use_eagle", &ModelSpec::useEagle, nb::rv_policy::reference_internal) + .def("use_lookahead_decoding", &ModelSpec::useLookaheadDecoding, nb::rv_policy::reference_internal) + .def("use_explicit_draft_tokens_decoding", &ModelSpec::useExplicitDraftTokensDecoding, + nb::rv_policy::reference_internal) + .def("use_draft_tokens_external_decoding", &ModelSpec::useDraftTokensExternalDecoding, + nb::rv_policy::reference_internal) + .def("use_logits", &ModelSpec::useLogits) + .def("use_multiple_profiles", &ModelSpec::useMultipleProfiles, nb::rv_policy::reference_internal) + .def("set_max_input_length", &ModelSpec::setMaxInputLength, nb::rv_policy::reference_internal) + .def("set_max_output_length", &ModelSpec::setMaxOutputLength, nb::rv_policy::reference_internal) + .def("set_quant_method", &ModelSpec::setQuantMethod, nb::rv_policy::reference_internal) + .def("use_lora_plugin", &ModelSpec::useLoraPlugin, nb::rv_policy::reference_internal) + .def("get_input_file", &ModelSpec::getInputFile) + .def("get_model_path", &ModelSpec::getModelPath) + .def("get_results_file", &ModelSpec::getResultsFile) + .def("get_generation_logits_file", &ModelSpec::getGenerationLogitsFile) + .def("get_context_logits_file", &ModelSpec::getContextLogitsFile) + .def("get_cum_log_probs_file", &ModelSpec::getCumLogProbsFile) + .def("get_log_probs_file", &ModelSpec::getLogProbsFile) + .def("enable_context_fmha_fp32_acc", &ModelSpec::enableContextFMHAFp32Acc, nb::rv_policy::reference_internal) + .def("get_enable_context_fmha_fp32_acc", &ModelSpec::getEnableContextFMHAFp32Acc) + .def("__copy__", [](ModelSpec const& self) { return ModelSpec(self); }); +} + +} // namespace tensorrt_llm::nanobind::testing diff --git a/cpp/tensorrt_llm/nanobind/testing/modelSpecBinding.h b/cpp/tensorrt_llm/nanobind/testing/modelSpecBinding.h new file mode 100644 index 000000000000..1aababc6ff89 --- /dev/null +++ b/cpp/tensorrt_llm/nanobind/testing/modelSpecBinding.h @@ -0,0 +1,29 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include <nanobind/nanobind.h> + +namespace nb = nanobind; + +namespace tensorrt_llm::nanobind::testing +{ + +void initBindings(nb::module_& m); + +} // namespace tensorrt_llm::nanobind::testing diff --git a/cpp/tensorrt_llm/nanobind/thop/bindings.cpp b/cpp/tensorrt_llm/nanobind/thop/bindings.cpp index dd1ff0db5410..b912ad09b5ce 100644 --- a/cpp/tensorrt_llm/nanobind/thop/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/thop/bindings.cpp @@ -85,15 +85,14 @@ nb::tuple trtllmGenGenerationPreprocessBinding(torch::Tensor qkv_input, torch::T std::optional<torch::Tensor> host_kv_cache_pool_pointers, std::optional<torch::Tensor> host_kv_cache_pool_mapping, std::optional<torch::Tensor> kv_scale_orig_quant, std::optional<torch::Tensor> kv_scale_quant_orig, std::optional<torch::Tensor> attention_output_orig_quant, std::optional<torch::Tensor> rotary_inv_freq, - std::optional<torch::Tensor> rotary_cos_sin, std::optional<torch::Tensor> mrope_position_deltas, int64_t layer_idx, - int64_t seq_offset, int64_t num_heads, int64_t num_kv_heads, int64_t head_size, int64_t tokens_per_block, - int64_t kv_cache_quant_mode, int64_t max_attention_window_size, int64_t cyclic_attention_window_size, - int64_t num_tokens, int64_t batch_beam, int64_t input_seq_length, int64_t max_past_kv_length, - int64_t rotary_embedding_dim, double rotary_embedding_base, int64_t rotary_embedding_scale_type, - double rotary_embedding_scale, int64_t rotary_embedding_max_positions, int64_t position_embedding_type, - double bmm1_scale, double bmm2_scale, bool fp8_context_fmha, int64_t predicted_tokens_per_seq, - int64_t attention_chunk_size, int64_t multi_processor_count, int64_t total_num_blocks, int64_t kv_factor, - bool need_build_kv_cache_metadata, bool cross_attention) + std::optional<torch::Tensor> rotary_cos_sin, int64_t layer_idx, int64_t seq_offset, int64_t num_heads, + int64_t num_kv_heads, int64_t head_size, int64_t tokens_per_block, int64_t kv_cache_quant_mode, + int64_t max_attention_window_size, int64_t cyclic_attention_window_size, int64_t num_tokens, int64_t batch_beam, + int64_t input_seq_length, int64_t max_past_kv_length, int64_t rotary_embedding_dim, double rotary_embedding_base, + int64_t rotary_embedding_scale_type, double rotary_embedding_scale, int64_t rotary_embedding_max_positions, + int64_t position_embedding_type, double bmm1_scale, double bmm2_scale, bool fp8_context_fmha, + int64_t predicted_tokens_per_seq, int64_t attention_chunk_size, int64_t multi_processor_count, + int64_t total_num_blocks, int64_t kv_factor, bool need_build_kv_cache_metadata, bool cross_attention) { auto result = [&]() { @@ -101,8 +100,8 @@ nb::tuple trtllmGenGenerationPreprocessBinding(torch::Tensor qkv_input, torch::T return torch_ext::trtllmGenGenerationPreprocess(qkv_input, workspace, sequence_lengths, spec_decoding_generation_lengths, spec_decoding_position_offsets, kv_cache_block_offsets, host_kv_cache_pool_pointers, host_kv_cache_pool_mapping, kv_scale_orig_quant, kv_scale_quant_orig, - attention_output_orig_quant, rotary_inv_freq, rotary_cos_sin, mrope_position_deltas, layer_idx, seq_offset, - num_heads, num_kv_heads, head_size, tokens_per_block, kv_cache_quant_mode, max_attention_window_size, + attention_output_orig_quant, rotary_inv_freq, rotary_cos_sin, layer_idx, seq_offset, num_heads, + num_kv_heads, head_size, tokens_per_block, kv_cache_quant_mode, max_attention_window_size, cyclic_attention_window_size, num_tokens, batch_beam, input_seq_length, max_past_kv_length, rotary_embedding_dim, rotary_embedding_base, rotary_embedding_scale_type, rotary_embedding_scale, rotary_embedding_max_positions, position_embedding_type, bmm1_scale, bmm2_scale, fp8_context_fmha, @@ -178,9 +177,7 @@ void initBindings(nb::module_& m) nb::arg("is_cross") = false, nb::arg("cross_kv") = std::nullopt, nb::arg("relative_attention_bias") = std::nullopt, nb::arg("relative_attention_max_distance") = 0, nb::arg("spec_decoding_target_max_draft_tokens") = std::nullopt, nb::arg("quant_scale_qkv") = std::nullopt, - nb::arg("dsv4_inv_rope_cos_sin_cache") = std::nullopt, nb::arg("enable_dsv4_epilogue_fusion") = false, - nb::arg("force_prepare_spec_dec_tree_mask") = false, "Multi-head attention operation", - nb::call_guard<nb::gil_scoped_release>()); + "Multi-head attention operation", nb::call_guard<nb::gil_scoped_release>()); m.def( "get_helix_workspace_size_per_rank", @@ -327,16 +324,15 @@ void initBindings(nb::module_& m) nb::arg("host_kv_cache_pool_pointers").none(), nb::arg("host_kv_cache_pool_mapping").none(), nb::arg("kv_scale_orig_quant").none(), nb::arg("kv_scale_quant_orig").none(), nb::arg("attention_output_orig_quant").none(), nb::arg("rotary_inv_freq").none(), - nb::arg("rotary_cos_sin").none(), nb::arg("mrope_position_deltas").none(), nb::arg("layer_idx"), - nb::arg("seq_offset"), nb::arg("num_heads"), nb::arg("num_kv_heads"), nb::arg("head_size"), - nb::arg("tokens_per_block"), nb::arg("kv_cache_quant_mode"), nb::arg("max_attention_window_size"), - nb::arg("cyclic_attention_window_size"), nb::arg("num_tokens"), nb::arg("batch_beam"), - nb::arg("input_seq_length"), nb::arg("max_past_kv_length"), nb::arg("rotary_embedding_dim"), - nb::arg("rotary_embedding_base"), nb::arg("rotary_embedding_scale_type"), nb::arg("rotary_embedding_scale"), - nb::arg("rotary_embedding_max_positions"), nb::arg("position_embedding_type"), nb::arg("bmm1_scale"), - nb::arg("bmm2_scale"), nb::arg("fp8_context_fmha"), nb::arg("predicted_tokens_per_seq"), - nb::arg("attention_chunk_size"), nb::arg("multi_processor_count"), nb::arg("total_num_blocks"), - nb::arg("kv_factor"), nb::arg("need_build_kv_cache_metadata") = true, nb::arg("cross_attention") = false, - "Fused nanobind generation preprocess for trtllm-gen attention."); + nb::arg("rotary_cos_sin").none(), nb::arg("layer_idx"), nb::arg("seq_offset"), nb::arg("num_heads"), + nb::arg("num_kv_heads"), nb::arg("head_size"), nb::arg("tokens_per_block"), nb::arg("kv_cache_quant_mode"), + nb::arg("max_attention_window_size"), nb::arg("cyclic_attention_window_size"), nb::arg("num_tokens"), + nb::arg("batch_beam"), nb::arg("input_seq_length"), nb::arg("max_past_kv_length"), + nb::arg("rotary_embedding_dim"), nb::arg("rotary_embedding_base"), nb::arg("rotary_embedding_scale_type"), + nb::arg("rotary_embedding_scale"), nb::arg("rotary_embedding_max_positions"), + nb::arg("position_embedding_type"), nb::arg("bmm1_scale"), nb::arg("bmm2_scale"), nb::arg("fp8_context_fmha"), + nb::arg("predicted_tokens_per_seq"), nb::arg("attention_chunk_size"), nb::arg("multi_processor_count"), + nb::arg("total_num_blocks"), nb::arg("kv_factor"), nb::arg("need_build_kv_cache_metadata") = true, + nb::arg("cross_attention") = false, "Fused nanobind generation preprocess for trtllm-gen attention."); } } // namespace tensorrt_llm::nanobind::thop diff --git a/cpp/tensorrt_llm/plugins/CMakeLists.txt b/cpp/tensorrt_llm/plugins/CMakeLists.txt new file mode 100755 index 000000000000..8b89cccdc813 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/CMakeLists.txt @@ -0,0 +1,183 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# + +set(PLUGIN_TARGET_NAME nvinfer_plugin_tensorrt_llm) +set(PLUGIN_SHARED_TARGET ${PLUGIN_TARGET_NAME}) + +set(TARGET_DIR ${CMAKE_CURRENT_SOURCE_DIR}) +set(PLUGIN_EXPORT_MAP ${TARGET_DIR}/exports.map) # Linux +set(PLUGIN_EXPORT_DEF ${TARGET_DIR}/exports.def) # Windows + +if(${CMAKE_BUILD_TYPE} MATCHES "Debug") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g") +endif() + +set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --Wno-deprecated-declarations") +set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} --diag-suppress 997") + +if(NOT WIN32) + # additional warnings + # + # Ignore overloaded-virtual warning. We intentionally change parameters of + # some methods in derived class. + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wno-overloaded-virtual") + if(WARNING_IS_ERROR) + message(STATUS "Treating warnings as errors in GCC compilation") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Werror") + endif() +else() # Windows + # warning level 4 + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /W4") +endif() + +set(PLUGIN_SOURCES) +set(PLUGIN_CU_SOURCES) + +set(PLUGIN_LISTS + bertAttentionPlugin + cpSplitPlugin + fusedLayernormPlugin + gptAttentionCommon + gptAttentionPlugin + identityPlugin + gemmPlugin + gemmSwigluPlugin + fp8RowwiseGemmPlugin + smoothQuantGemmPlugin + fp4GemmPlugin + quantizePerTokenPlugin + quantizeTensorPlugin + quantizeToFP4Plugin + layernormQuantizationPlugin + rmsnormQuantizationPlugin + weightOnlyGroupwiseQuantMatmulPlugin + weightOnlyQuantMatmulPlugin + lookupPlugin + loraPlugin + doraPlugin + mixtureOfExperts + selectiveScanPlugin + mambaConv1dPlugin + lruPlugin + cumsumLastDimPlugin + topkLastDimPlugin + lowLatencyGemmPlugin + eaglePlugin + lowLatencyGemmSwigluPlugin + qserveGemmPlugin + cudaStreamPlugin + gemmAllReducePlugin) + +foreach(PLUGIN_ITER ${PLUGIN_LISTS}) + include_directories(${PLUGIN_ITER}) + add_subdirectory(${PLUGIN_ITER}) +endforeach(PLUGIN_ITER) + +if(ENABLE_MULTI_DEVICE) + include_directories(ncclPlugin) + add_subdirectory(ncclPlugin) +endif() +include_directories(common) +add_subdirectory(common) + +# Set gencodes +list(APPEND PLUGIN_SOURCES "${PLUGIN_CU_SOURCES}") + +list(APPEND PLUGIN_SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/api/tllmPlugin.cpp") + +# ################################# SHARED LIBRARY +# ############################################################################## + +if(WIN32) + set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS 1) +endif() + +add_library(${PLUGIN_SHARED_TARGET} SHARED ${PLUGIN_SOURCES}) +add_cuda_architectures(${PLUGIN_SHARED_TARGET} 89) + +target_include_directories( + ${PLUGIN_SHARED_TARGET} + PUBLIC ${CUDA_INSTALL_DIR}/include + PUBLIC + $<TARGET_PROPERTY:${INTERNAL_CUTLASS_KERNELS_TARGET},INTERFACE_INCLUDE_DIRECTORIES> + PRIVATE ${TARGET_DIR}) + +if(USING_OSS_CUTLASS_FP4_GEMM) + target_compile_definitions(${PLUGIN_SHARED_TARGET} + PUBLIC USING_OSS_CUTLASS_FP4_GEMM) +endif() + +if(USING_OSS_CUTLASS_ALLREDUCE_GEMM) + target_compile_definitions(${PLUGIN_SHARED_TARGET} + PUBLIC USING_OSS_CUTLASS_ALLREDUCE_GEMM) +endif() + +if(USING_OSS_CUTLASS_MOE_GEMM) + target_compile_definitions(${PLUGIN_SHARED_TARGET} + PUBLIC USING_OSS_CUTLASS_MOE_GEMM) +endif() + +if(ENABLE_MULTI_DEVICE) + target_include_directories(${PLUGIN_SHARED_TARGET} + PUBLIC ${MPI_C_INCLUDE_DIRS}) +endif() + +if(CUDA_VERSION VERSION_LESS 11.0) + target_include_directories(${PLUGIN_SHARED_TARGET} PUBLIC ${CUB_ROOT_DIR}) +endif() + +set_target_properties( + ${PLUGIN_SHARED_TARGET} + PROPERTIES CXX_STANDARD "17" + CXX_STANDARD_REQUIRED "YES" + CXX_EXTENSIONS "NO" + ARCHIVE_OUTPUT_DIRECTORY "${TRT_OUT_DIR}" + LIBRARY_OUTPUT_DIRECTORY "${TRT_OUT_DIR}" + RUNTIME_OUTPUT_DIRECTORY "${TRT_OUT_DIR}") + +if(WIN32) + set_target_properties( + ${PLUGIN_SHARED_TARGET} + PROPERTIES LINK_FLAGS "/DEF:${PLUGIN_EXPORT_DEF} ${UNDEFINED_FLAG}") +else() + set_target_properties( + ${PLUGIN_SHARED_TARGET} + PROPERTIES + LINK_FLAGS + "-Wl,--exclude-libs,ALL -Wl,--version-script=${PLUGIN_EXPORT_MAP} -Wl,-rpath,'$ORIGIN' ${AS_NEEDED_FLAG} ${UNDEFINED_FLAG}" + ) +endif() + +set_property(TARGET ${PLUGIN_SHARED_TARGET} PROPERTY CUDA_STANDARD 17) + +target_link_libraries( + ${PLUGIN_SHARED_TARGET} + ${CUBLAS_LIB} + ${CUBLASLT_LIB} + ${TRT_LIB} + ${CUDA_DRV_LIB} + ${CUDA_RT_LIB} + ${CMAKE_DL_LIBS} + ${SHARED_TARGET}) + +if(WIN32) + target_link_libraries(${PLUGIN_SHARED_TARGET} context_attention_src) +endif() + +if(ENABLE_MULTI_DEVICE) + target_link_libraries(${PLUGIN_SHARED_TARGET} ${MPI_C_LIBRARIES} ${NCCL_LIB}) +endif() diff --git a/cpp/tensorrt_llm/plugins/api/tllmPlugin.cpp b/cpp/tensorrt_llm/plugins/api/tllmPlugin.cpp new file mode 100644 index 000000000000..f0dceb2f4a99 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/api/tllmPlugin.cpp @@ -0,0 +1,313 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/plugins/api/tllmPlugin.h" + +#include "tensorrt_llm/common/stringUtils.h" +#include "tensorrt_llm/runtime/tllmLogger.h" + +#include "tensorrt_llm/plugins/bertAttentionPlugin/bertAttentionPlugin.h" +#include "tensorrt_llm/plugins/doraPlugin/doraPlugin.h" +#include "tensorrt_llm/plugins/fp8RowwiseGemmPlugin/fp8RowwiseGemmPlugin.h" +#include "tensorrt_llm/plugins/fusedLayernormPlugin/fusedLayernormPlugin.h" +#include "tensorrt_llm/plugins/gemmPlugin/gemmPlugin.h" +#include "tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.h" +#include "tensorrt_llm/plugins/gptAttentionPlugin/gptAttentionPlugin.h" +#include "tensorrt_llm/plugins/identityPlugin/identityPlugin.h" +#include "tensorrt_llm/plugins/layernormQuantizationPlugin/layernormQuantizationPlugin.h" +#include "tensorrt_llm/plugins/lookupPlugin/lookupPlugin.h" +#include "tensorrt_llm/plugins/loraPlugin/loraPlugin.h" +#include "tensorrt_llm/plugins/lruPlugin/lruPlugin.h" +#include "tensorrt_llm/plugins/mambaConv1dPlugin/mambaConv1dPlugin.h" +#include "tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.h" +#include "tensorrt_llm/plugins/quantizeToFP4Plugin/quantizeToFP4Plugin.h" +#if ENABLE_MULTI_DEVICE +#include "tensorrt_llm/plugins/cpSplitPlugin/cpSplitPlugin.h" +#include "tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePlugin.h" +#include "tensorrt_llm/plugins/ncclPlugin/allgatherPlugin.h" +#include "tensorrt_llm/plugins/ncclPlugin/allreducePlugin.h" +#include "tensorrt_llm/plugins/ncclPlugin/recvPlugin.h" +#include "tensorrt_llm/plugins/ncclPlugin/reduceScatterPlugin.h" +#include "tensorrt_llm/plugins/ncclPlugin/sendPlugin.h" +#endif // ENABLE_MULTI_DEVICE +#include "tensorrt_llm/plugins/cudaStreamPlugin/cudaStreamPlugin.h" +#include "tensorrt_llm/plugins/cumsumLastDimPlugin/cumsumLastDimPlugin.h" +#include "tensorrt_llm/plugins/eaglePlugin/eagleDecodeDraftTokensPlugin.h" +#include "tensorrt_llm/plugins/eaglePlugin/eaglePrepareDrafterInputsPlugin.h" +#include "tensorrt_llm/plugins/eaglePlugin/eagleSampleAndAcceptDraftTokensPlugin.h" +#include "tensorrt_llm/plugins/fp4GemmPlugin/fp4GemmPlugin.h" +#include "tensorrt_llm/plugins/lowLatencyGemmPlugin/lowLatencyGemmPlugin.h" +#include "tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/lowLatencyGemmSwigluPlugin.h" +#include "tensorrt_llm/plugins/qserveGemmPlugin/qserveGemmPlugin.h" +#include "tensorrt_llm/plugins/quantizePerTokenPlugin/quantizePerTokenPlugin.h" +#include "tensorrt_llm/plugins/quantizeTensorPlugin/quantizeTensorPlugin.h" +#include "tensorrt_llm/plugins/rmsnormQuantizationPlugin/rmsnormQuantizationPlugin.h" +#include "tensorrt_llm/plugins/selectiveScanPlugin/selectiveScanPlugin.h" +#include "tensorrt_llm/plugins/smoothQuantGemmPlugin/smoothQuantGemmPlugin.h" +#include "tensorrt_llm/plugins/topkLastDimPlugin/topkLastDimPlugin.h" +#include "tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/weightOnlyGroupwiseQuantMatmulPlugin.h" +#include "tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/weightOnlyQuantMatmulPlugin.h" + +#include <array> +#include <cstdlib> + +#include <NvInferRuntime.h> + +namespace tc = tensorrt_llm::common; + +namespace +{ + +nvinfer1::IPluginCreator* creatorPtr(nvinfer1::IPluginCreator& creator) +{ + return &creator; +} + +nvinfer1::IPluginCreatorInterface* creatorInterfacePtr(nvinfer1::IPluginCreatorInterface& creator) +{ + return &creator; +} + +auto tllmLogger = tensorrt_llm::runtime::TllmLogger(); + +nvinfer1::ILogger* gLogger{&tllmLogger}; + +class GlobalLoggerFinder : public nvinfer1::ILoggerFinder +{ +public: + nvinfer1::ILogger* findLogger() override + { + return gLogger; + } +}; + +GlobalLoggerFinder gGlobalLoggerFinder{}; + +#if !defined(_MSC_VER) +[[maybe_unused]] __attribute__((constructor)) +#endif +void initOnLoad() +{ + auto constexpr kLoadPlugins = "TRT_LLM_LOAD_PLUGINS"; + auto const loadPlugins = std::getenv(kLoadPlugins); + if (loadPlugins && loadPlugins[0] == '1') + { + initTrtLlmPlugins(gLogger); + } +} + +bool pluginsInitialized = false; + +} // namespace + +namespace tensorrt_llm::plugins::api +{ + +LoggerManager& tensorrt_llm::plugins::api::LoggerManager::getInstance() noexcept +{ + static LoggerManager instance; + return instance; +} + +void LoggerManager::setLoggerFinder(nvinfer1::ILoggerFinder* finder) +{ + std::lock_guard<std::mutex> lk(mMutex); + if (mLoggerFinder == nullptr && finder != nullptr) + { + mLoggerFinder = finder; + } +} + +[[maybe_unused]] nvinfer1::ILogger* LoggerManager::logger() +{ + std::lock_guard<std::mutex> lk(mMutex); + if (mLoggerFinder != nullptr) + { + return mLoggerFinder->findLogger(); + } + return nullptr; +} + +nvinfer1::ILogger* LoggerManager::defaultLogger() noexcept +{ + return gLogger; +} +} // namespace tensorrt_llm::plugins::api + +// New Plugin APIs + +extern "C" +{ + bool initTrtLlmPlugins(void* logger, char const* libNamespace) + { + if (pluginsInitialized) + { + return true; + } + + if (logger) + { + gLogger = static_cast<nvinfer1::ILogger*>(logger); + } + setLoggerFinder(&gGlobalLoggerFinder); + + auto registry = getPluginRegistry(); + + { + std::int32_t nbCreators; + auto creators = getPluginCreators(nbCreators); + + for (std::int32_t i = 0; i < nbCreators; ++i) + { + auto const creator = creators[i]; + creator->setPluginNamespace(libNamespace); + registry->registerCreator(*creator, libNamespace); + if (gLogger) + { + auto const msg = tc::fmtstr("Registered plugin creator %s version %s in namespace %s", + creator->getPluginName(), creator->getPluginVersion(), libNamespace); + gLogger->log(nvinfer1::ILogger::Severity::kVERBOSE, msg.c_str()); + } + } + } + + { + std::int32_t nbCreators; + auto creators = getCreators(nbCreators); + + for (std::int32_t i = 0; i < nbCreators; ++i) + { + auto const creator = creators[i]; + registry->registerCreator(*creator, libNamespace); + } + } + + pluginsInitialized = true; + return true; + } + + [[maybe_unused]] void setLoggerFinder([[maybe_unused]] nvinfer1::ILoggerFinder* finder) + { + tensorrt_llm::plugins::api::LoggerManager::getInstance().setLoggerFinder(finder); + } + + [[maybe_unused]] nvinfer1::IPluginCreator* const* getPluginCreators(std::int32_t& nbCreators) + { + static tensorrt_llm::plugins::IdentityPluginCreator identityPluginCreator; + static tensorrt_llm::plugins::BertAttentionPluginCreator bertAttentionPluginCreator; + static tensorrt_llm::plugins::FusedLayernormPluginCreator fusedLayernormPluginCreator; + static tensorrt_llm::plugins::GPTAttentionPluginCreator gptAttentionPluginCreator; + static tensorrt_llm::plugins::GemmPluginCreator gemmPluginCreator; + static tensorrt_llm::plugins::GemmSwigluPluginCreator gemmSwigluPluginCreator; + static tensorrt_llm::plugins::Fp8RowwiseGemmPluginCreator fp8RowwiseGemmPluginCreator; + static tensorrt_llm::plugins::MixtureOfExpertsPluginCreator moePluginCreator; +#if ENABLE_MULTI_DEVICE + static tensorrt_llm::plugins::SendPluginCreator sendPluginCreator; + static tensorrt_llm::plugins::RecvPluginCreator recvPluginCreator; + static tensorrt_llm::plugins::AllreducePluginCreator allreducePluginCreator; + static tensorrt_llm::plugins::AllgatherPluginCreator allgatherPluginCreator; + static tensorrt_llm::plugins::ReduceScatterPluginCreator reduceScatterPluginCreator; + static tensorrt_llm::plugins::GemmAllReducePluginCreator gemmAllReducePluginCreator; +#endif // ENABLE_MULTI_DEVICE + static tensorrt_llm::plugins::SmoothQuantGemmPluginCreator smoothQuantGemmPluginCreator; + static tensorrt_llm::plugins::QServeGemmPluginCreator qserveGemmPluginCreator; + static tensorrt_llm::plugins::LayernormQuantizationPluginCreator layernormQuantizationPluginCreator; + static tensorrt_llm::plugins::QuantizeToFP4PluginCreator quantizeToFP4PluginCreator; + static tensorrt_llm::plugins::QuantizePerTokenPluginCreator quantizePerTokenPluginCreator; + static tensorrt_llm::plugins::QuantizeTensorPluginCreator quantizeTensorPluginCreator; + static tensorrt_llm::plugins::RmsnormQuantizationPluginCreator rmsnormQuantizationPluginCreator; + static tensorrt_llm::plugins::WeightOnlyGroupwiseQuantMatmulPluginCreator + weightOnlyGroupwiseQuantMatmulPluginCreator; + static tensorrt_llm::plugins::WeightOnlyQuantMatmulPluginCreator weightOnlyQuantMatmulPluginCreator; + static tensorrt_llm::plugins::LookupPluginCreator lookupPluginCreator; + static tensorrt_llm::plugins::LoraPluginCreator loraPluginCreator; + static tensorrt_llm::plugins::SelectiveScanPluginCreator selectiveScanPluginCreator; + static tensorrt_llm::plugins::Fp4GemmPluginCreator fp4GemmPluginCreator; + static tensorrt_llm::plugins::MambaConv1dPluginCreator mambaConv1DPluginCreator; + static tensorrt_llm::plugins::lruPluginCreator lruPluginCreator; + static tensorrt_llm::plugins::CumsumLastDimPluginCreator cumsumLastDimPluginCreator; + static tensorrt_llm::plugins::TopkLastDimPluginCreator topkLastDimPluginCreator; + static tensorrt_llm::plugins::LowLatencyGemmPluginCreator lowLatencyGemmPluginCreator; + static tensorrt_llm::plugins::LowLatencyGemmSwigluPluginCreator lowLatencyGemmSwigluPluginCreator; + static tensorrt_llm::plugins::EagleDecodeDraftTokensPluginCreator eagleDecodeDraftTokensPluginCreator; + static tensorrt_llm::plugins::EagleSampleAndAcceptDraftTokensPluginCreator + eagleSampleAndAcceptDraftTokensPluginCreator; + static tensorrt_llm::plugins::CudaStreamPluginCreator cudaStreamPluginCreator; + + static std::array pluginCreators + = { creatorPtr(identityPluginCreator), + creatorPtr(bertAttentionPluginCreator), + creatorPtr(gptAttentionPluginCreator), + creatorPtr(gemmPluginCreator), + creatorPtr(gemmSwigluPluginCreator), + creatorPtr(fp8RowwiseGemmPluginCreator), + creatorPtr(moePluginCreator), +#if ENABLE_MULTI_DEVICE + creatorPtr(sendPluginCreator), + creatorPtr(recvPluginCreator), + creatorPtr(allreducePluginCreator), + creatorPtr(allgatherPluginCreator), + creatorPtr(reduceScatterPluginCreator), + creatorPtr(gemmAllReducePluginCreator), +#endif // ENABLE_MULTI_DEVICE + creatorPtr(fusedLayernormPluginCreator), + creatorPtr(smoothQuantGemmPluginCreator), + creatorPtr(qserveGemmPluginCreator), + creatorPtr(layernormQuantizationPluginCreator), + creatorPtr(quantizeToFP4PluginCreator), + creatorPtr(quantizePerTokenPluginCreator), + creatorPtr(quantizeTensorPluginCreator), + creatorPtr(rmsnormQuantizationPluginCreator), + creatorPtr(weightOnlyGroupwiseQuantMatmulPluginCreator), + creatorPtr(weightOnlyQuantMatmulPluginCreator), + creatorPtr(lookupPluginCreator), + creatorPtr(loraPluginCreator), + creatorPtr(selectiveScanPluginCreator), + creatorPtr(fp4GemmPluginCreator), + creatorPtr(mambaConv1DPluginCreator), + creatorPtr(lruPluginCreator), + creatorPtr(cumsumLastDimPluginCreator), + creatorPtr(topkLastDimPluginCreator), + creatorPtr(lowLatencyGemmPluginCreator), + creatorPtr(eagleDecodeDraftTokensPluginCreator), + creatorPtr(eagleSampleAndAcceptDraftTokensPluginCreator), + creatorPtr(lowLatencyGemmSwigluPluginCreator), + creatorPtr(cudaStreamPluginCreator), + }; + nbCreators = pluginCreators.size(); + return pluginCreators.data(); + } + + [[maybe_unused]] nvinfer1::IPluginCreatorInterface* const* getCreators(std::int32_t& nbCreators) + { + static tensorrt_llm::plugins::EaglePrepareDrafterInputsPluginCreator eaglePrepareDrafterInputsPluginCreator; +#if ENABLE_MULTI_DEVICE + static tensorrt_llm::plugins::CpSplitPluginCreator cpSplitPluginCreator; +#endif // ENABLE_MULTI_DEVICE + + static tensorrt_llm::plugins::DoraPluginCreator doraPluginCreator; + + static std::array creators + = { creatorInterfacePtr(eaglePrepareDrafterInputsPluginCreator), +#if ENABLE_MULTI_DEVICE + creatorInterfacePtr(cpSplitPluginCreator), +#endif // ENABLE_MULTI_DEVICE + creatorInterfacePtr(doraPluginCreator) }; + + nbCreators = creators.size(); + return creators.data(); + } +} // extern "C" diff --git a/cpp/tensorrt_llm/plugins/bertAttentionPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/bertAttentionPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/bertAttentionPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/bertAttentionPlugin/bertAttentionPlugin.cpp b/cpp/tensorrt_llm/plugins/bertAttentionPlugin/bertAttentionPlugin.cpp new file mode 100644 index 000000000000..6acf0b3a9d25 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/bertAttentionPlugin/bertAttentionPlugin.cpp @@ -0,0 +1,1206 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2025 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "bertAttentionPlugin.h" +#include "tensorrt_llm/kernels/gptKernels.h" +#include "tensorrt_llm/kernels/recoverFromRingAtten.h" +#include "tensorrt_llm/kernels/sageAttentionKernels.h" +#include "tensorrt_llm/kernels/unfusedAttentionKernels.h" +#include "tensorrt_llm/runtime/iBuffer.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +namespace tc = tensorrt_llm::common; + +using tensorrt_llm::plugins::BertAttentionPluginCreator; +using tensorrt_llm::plugins::BertAttentionPlugin; + +static char const* BERT_ATTENTION_PLUGIN_VERSION{"1"}; +static char const* BERT_ATTENTION_PLUGIN_NAME{"BertAttention"}; +PluginFieldCollection BertAttentionPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> BertAttentionPluginCreator::mPluginAttributes; + +BertAttentionPlugin::BertAttentionPlugin(int num_heads, int head_size, float q_scaling, + ContextFMHAType context_fmha_type, nvinfer1::DataType type, bool do_relative_attention, int max_distance, + bool remove_padding, bool sage_attn, int sage_attn_q_block_size, int sage_attn_k_block_size, + int sage_attn_v_block_size, int cp_size, int cp_rank, std::set<int> cp_group) + : mNumHeads(num_heads) + , mHeadSize(head_size) + , mQScaling(q_scaling) + , mType(type) + , mRelativeAttention(do_relative_attention) + , mMaxDistance(max_distance) + , mRemovePadding(remove_padding) + , mEnableContextFMHA(context_fmha_type != ContextFMHAType::DISABLED) + , mFMHAForceFP32Acc(context_fmha_type == ContextFMHAType::ENABLED_WITH_FP32_ACC) + , mSageAttn(sage_attn) + , mCpSize(cp_size) + , mCpRank(cp_rank) + , mCpGroup(std::move(cp_group)) +{ + // pre-check whether FMHA is supported in order to save memory allocation + if (mEnableContextFMHA) + { + mEnableContextFMHA = false; + if (!(mType == DataType::kHALF || mType == DataType::kBF16)) + { + TLLM_LOG_WARNING("Fall back to unfused MHA because of unsupported data type."); + } + else if (mRelativeAttention) + { + TLLM_LOG_WARNING("Fall back to unfused MHA because of relative position embedding."); + } + else + { + mEnableContextFMHA = true; + } + } + + if (mSageAttn) + { + mSageAttnQBlockSize = sage_attn_q_block_size; + mSageAttnKBlockSize = sage_attn_k_block_size; + mSageAttnVBlockSize = sage_attn_v_block_size; + std::vector<int> blockSizeCombination + = {sage_attn_q_block_size, sage_attn_k_block_size, sage_attn_v_block_size}; + if (mSageAttnSupportedBlockSizes.find(blockSizeCombination) == mSageAttnSupportedBlockSizes.end() + || (head_size != 128 && head_size != 72 && head_size != 80)) + { + TLLM_LOG_WARNING(" Q, k ,v quant block size not support. disable sage attention"); + mSageAttn = false; + } + else + { + TLLM_LOG_INFO("SageAttnQBlockSize: %d, SageAttnKBlockSize: %d, SageAttnVBlockSize: %d", mSageAttnQBlockSize, + mSageAttnKBlockSize, mSageAttnVBlockSize); + } + } + + if (cp_group.size() > 1 && !mEnableContextFMHA) + { + TLLM_LOG_ERROR("Unfused MHA do not support context parallel now."); + } +} + +// Parameterized constructor +BertAttentionPlugin::BertAttentionPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mNumHeads); + read(d, mHeadSize); + read(d, mQScaling); + read(d, mQKHalfAccum); + read(d, mEnableContextFMHA); + read(d, mFMHAForceFP32Acc); + read(d, mType); + read(d, mRelativeAttention); + read(d, mMaxDistance); + read(d, mRemovePadding); + read(d, mSageAttn); + read(d, mSageAttnQBlockSize); + read(d, mSageAttnKBlockSize); + read(d, mSageAttnVBlockSize); + read(d, mCpSize); + read(d, mCpRank); + mCpGroup.clear(); + int groupItem = 0; + while (d != a + length) + { + read(d, groupItem); + mCpGroup.insert(groupItem); + } + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* BertAttentionPlugin::clone() const noexcept +{ + auto* plugin = new BertAttentionPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + plugin->initialize(); + return plugin; +} + +nvinfer1::DimsExprs BertAttentionPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + TLLM_CHECK(outputIndex == 0); + auto ret = inputs[0]; + ret.d[mRemovePadding ? 1 : 2] = exprBuilder.constant(ret.d[mRemovePadding ? 1 : 2]->getConstantValue() / 3); + return ret; +} + +bool BertAttentionPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + // inputs: [0] qkv, [1] input_lengths, [2] max_input_length (optional), [3] relative_attention_bias (optional) + // outputs: [X] hidden_states + if (nbInputs == 2) + { // BERT + if (pos == 1) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + if (nbInputs > 2) + { // Encoder in encoder-decoder + if (pos == 1 || pos == 2) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + + return false; +} + +void BertAttentionPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t BertAttentionPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + // if remove padding, inputs[0] "qkv_hidden_states" dim is [num_tokens, 3*hidden_dim] which doesn't have shape + // info should get max_batch_size and max_input_length from inputs[1] "input_lengths" and input[2] + // "max_input_length" + int const batch_size = mRemovePadding ? inputs[1].dims.d[0] : inputs[0].dims.d[0]; + int const input_seq_len = mRemovePadding ? inputs[2].dims.d[0] : inputs[0].dims.d[1]; + int const local_hidden_units_ = inputs[0].dims.d[mRemovePadding ? 1 : 2] / 3; + + auto const size = tensorrt_llm::runtime::BufferDataType(inputs[0].type).getSize(); + + size_t const attention_mask_size = mEnableContextFMHA ? 0 : size * batch_size * input_seq_len * input_seq_len; + size_t const cu_seqlens_size = sizeof(int) * (batch_size + 1); + size_t const q_buf_2_size = mEnableContextFMHA ? 0 : size * batch_size * input_seq_len * local_hidden_units_; + size_t const k_buf_2_size = mEnableContextFMHA ? 0 : size * batch_size * input_seq_len * local_hidden_units_; + size_t const v_buf_2_size = mEnableContextFMHA ? 0 : size * batch_size * input_seq_len * local_hidden_units_; + size_t const qk_buf_size = mEnableContextFMHA ? 0 : size * batch_size * mNumHeads * input_seq_len * input_seq_len; + size_t const qkv_buf_2_size = mEnableContextFMHA ? 0 : size * batch_size * input_seq_len * local_hidden_units_; + size_t const qk_buf_float_size + = mEnableContextFMHA ? 0 : sizeof(float) * batch_size * mNumHeads * input_seq_len * input_seq_len; + size_t const padding_offset_size = mEnableContextFMHA ? 0 : sizeof(int) * batch_size * input_seq_len; + size_t const fmha_scheduler_counter = mEnableContextFMHA ? sizeof(uint32_t) : 0; + int const paddedHeadSize = mSageAttn ? ((mHeadSize + 15) / 16) * 16 : mHeadSize; + const size_t quanted_qkv_size + = mSageAttn ? sizeof(__nv_fp8_e4m3) * batch_size * input_seq_len * mNumHeads * paddedHeadSize * 3 : 0; + const size_t q_scale_size = mSageAttn + ? sizeof(float) * batch_size * ((input_seq_len + mSageAttnQBlockSize - 1) / mSageAttnQBlockSize) * mNumHeads + : 0; + const size_t k_scale_size = mSageAttn + ? sizeof(float) * batch_size * ((input_seq_len + mSageAttnKBlockSize - 1) / mSageAttnKBlockSize) * mNumHeads + : 0; + const size_t v_scale_size = mSageAttn + ? sizeof(float) * batch_size * ((input_seq_len + mSageAttnVBlockSize - 1) / mSageAttnVBlockSize) * mNumHeads + : 0; + const size_t scale_bmm1_device_size = mSageAttn ? sizeof(float) * 2 : 0; + const size_t scale_bmm2_device_size = mSageAttn ? sizeof(float) : 0; + size_t sage_quant_space_size = mSageAttn ? sizeof(float) * batch_size * mNumHeads * mHeadSize : 0; + + if (paddedHeadSize != mHeadSize) + sage_quant_space_size + = sage_quant_space_size < (batch_size * input_seq_len * mNumHeads * paddedHeadSize * sizeof(__nv_bfloat16)) + ? (batch_size * input_seq_len * mNumHeads * paddedHeadSize * sizeof(__nv_bfloat16)) + : sage_quant_space_size; + + // workspace for RingAttention ping-pong buffer + bool const enableRingAttn = (mCpGroup.size() > 1); + const size_t ring_q_buf_size = enableRingAttn ? size * batch_size * input_seq_len * local_hidden_units_ : 0; + const size_t ring_kv_buf_size = enableRingAttn + ? 2 * size * batch_size * input_seq_len * local_hidden_units_ + sizeof(int) * (batch_size + 1) + : 0; + const size_t ring_softmax_stats_buf_size + = enableRingAttn ? 2 * sizeof(float) * batch_size * input_seq_len * mNumHeads : 0; + const size_t ring_softmax_stats_accu_buf_size + = enableRingAttn ? 2 * sizeof(float) * batch_size * input_seq_len * mNumHeads : 0; + const size_t ring_block_output_size = enableRingAttn ? size * batch_size * input_seq_len * local_hidden_units_ : 0; + + int const NUM_BUFFERS = 24; + + size_t workspaces[NUM_BUFFERS]; + workspaces[0] = CUBLAS_WORKSPACE_SIZE; + workspaces[1] = attention_mask_size; + workspaces[2] = cu_seqlens_size; + workspaces[3] = q_buf_2_size; + workspaces[4] = k_buf_2_size; + workspaces[5] = v_buf_2_size; + workspaces[6] = qk_buf_size; + workspaces[7] = qkv_buf_2_size; + workspaces[8] = qk_buf_float_size; + workspaces[9] = padding_offset_size; + workspaces[10] = fmha_scheduler_counter; + workspaces[11] = quanted_qkv_size; + workspaces[12] = q_scale_size; + workspaces[13] = v_scale_size; + workspaces[14] = k_scale_size; + workspaces[15] = scale_bmm1_device_size; + workspaces[16] = scale_bmm2_device_size; + workspaces[17] = sage_quant_space_size; + workspaces[18] = ring_q_buf_size; + workspaces[19] = ring_kv_buf_size; // kv1 + workspaces[20] = ring_kv_buf_size; // kv2 + workspaces[21] = ring_softmax_stats_buf_size; + workspaces[22] = ring_softmax_stats_accu_buf_size; + workspaces[23] = ring_block_output_size; + + return tc::calculateTotalWorkspaceSize(workspaces, NUM_BUFFERS); +} + +template <typename T> +int BertAttentionPlugin::enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) +{ + + // inputs + // input_tensor [batch_size, seq_len, local_hidden_size*3] or [num_tokens, local_hidden_size*3] + // input_lengths [batch_size] + // max_input_length [max_input_length] -- use shape dim to represent max value. If remove padding, this records + // the max input length among sequences; otherwise same as input_tensor's padded dim[1] relative_attention_bias + // [num_heads, num_buckets] (optional) + // outputs + // output_tensor [batch_size, seq_len, local_hidden_size] or [num_tokens, local_hidden_size] + + // if remove padding, inputs[0] dim is [num_tokens] which doesn't have workspace info + // should get max_batch_size from inputs[1] and max_input_length from plugin attribute + int const batch_size = mRemovePadding ? inputDesc[1].dims.d[0] : inputDesc[0].dims.d[0]; + int const input_seq_len = mRemovePadding ? inputDesc[2].dims.d[0] : inputDesc[0].dims.d[1]; + int const num_tokens = mRemovePadding ? inputDesc[0].dims.d[0] : batch_size * input_seq_len; + int const request_batch_size = batch_size; + int const request_seq_len = input_seq_len; + int const local_hidden_units_ = inputDesc[0].dims.d[mRemovePadding ? 1 : 2] / 3; + float const q_scaling = mQScaling; + + T const* attention_input = reinterpret_cast<T const*>(inputs[0]); + int const* input_lengths = reinterpret_cast<int const*>(inputs[1]); + T const* relative_attn_table = mRelativeAttention ? reinterpret_cast<T const*>(inputs[3]) : nullptr; + T* context_buf_ = (T*) (outputs[0]); + + auto cublasHandle = mCublasWrapper->getCublasHandle(); + TLLM_CUDA_CHECK(cublasSetStream(cublasHandle, stream)); + mCublasWrapper->setStream(stream); + mCublasWrapper->setWorkspace(workspace); + if (inputDesc[0].type == DataType::kHALF) + { + mCublasWrapper->setFP16GemmConfig(); + } + else if (inputDesc[0].type == DataType::kFLOAT) + { + mCublasWrapper->setFP32GemmConfig(); + } +#ifdef ENABLE_BF16 + else if constexpr (std::is_same_v<T, __nv_bfloat16>) + { + mCublasWrapper->setBF16GemmConfig(); + } +#endif + + size_t const attention_mask_size = mEnableContextFMHA ? 0 : sizeof(T) * batch_size * input_seq_len * input_seq_len; + size_t const cu_seqlens_size = sizeof(int) * (batch_size + 1); + size_t const q_buf_2_size = mEnableContextFMHA ? 0 : sizeof(T) * batch_size * input_seq_len * local_hidden_units_; + size_t const k_buf_2_size = mEnableContextFMHA ? 0 : sizeof(T) * batch_size * input_seq_len * local_hidden_units_; + size_t const v_buf_2_size = mEnableContextFMHA ? 0 : sizeof(T) * batch_size * input_seq_len * local_hidden_units_; + size_t const qk_buf_size + = mEnableContextFMHA ? 0 : sizeof(T) * batch_size * mNumHeads * input_seq_len * input_seq_len; + size_t const qkv_buf_2_size = mEnableContextFMHA ? 0 : sizeof(T) * batch_size * input_seq_len * local_hidden_units_; + size_t const qk_buf_float_size + = mEnableContextFMHA ? 0 : sizeof(float) * batch_size * mNumHeads * input_seq_len * input_seq_len; + size_t const padding_offset_size = mEnableContextFMHA ? 0 : sizeof(int) * batch_size * input_seq_len; + size_t const fmha_scheduler_counter = mEnableContextFMHA ? sizeof(uint32_t) : 0; + + int const paddedHeadSize = mSageAttn ? ((mHeadSize + 15) / 16) * 16 : mHeadSize; + const size_t quanted_qkv_size + = mSageAttn ? sizeof(__nv_fp8_e4m3) * batch_size * input_seq_len * mNumHeads * paddedHeadSize * 3 : 0; + const size_t q_scale_size = mSageAttn + ? sizeof(float) * batch_size * ((input_seq_len + mSageAttnQBlockSize - 1) / mSageAttnQBlockSize) * mNumHeads + : 0; + const size_t k_scale_size = mSageAttn + ? sizeof(float) * batch_size * ((input_seq_len + mSageAttnKBlockSize - 1) / mSageAttnKBlockSize) * mNumHeads + : 0; + const size_t v_scale_size = mSageAttn + ? sizeof(float) * batch_size * ((input_seq_len + mSageAttnVBlockSize - 1) / mSageAttnVBlockSize) * mNumHeads + : 0; + const size_t scale_bmm1_device_size = mSageAttn ? sizeof(float) * 2 : 0; + const size_t scale_bmm2_device_size = mSageAttn ? sizeof(float) : 0; + size_t sage_quant_space_size = mSageAttn ? sizeof(float) * batch_size * mNumHeads * mHeadSize : 0; + + if (paddedHeadSize != mHeadSize) + sage_quant_space_size + = sage_quant_space_size < (batch_size * input_seq_len * mNumHeads * paddedHeadSize * sizeof(__nv_bfloat16)) + ? (batch_size * input_seq_len * mNumHeads * paddedHeadSize * sizeof(__nv_bfloat16)) + : sage_quant_space_size; + + bool const enableRingAttn = (mCpGroup.size() > 1); + const size_t ring_q_buf_size = enableRingAttn ? sizeof(T) * batch_size * input_seq_len * local_hidden_units_ : 0; + const size_t ring_kv_buf_size + = enableRingAttn ? 2 * sizeof(T) * batch_size * input_seq_len * local_hidden_units_ : 0; + const size_t ring_softmax_stats_buf_size + = enableRingAttn ? 2 * sizeof(float) * batch_size * input_seq_len * mNumHeads : 0; + const size_t ring_block_output_size + = enableRingAttn ? sizeof(T) * batch_size * input_seq_len * local_hidden_units_ : 0; + + // Workspace pointer shift + int8_t* workspace_byte_ptr = reinterpret_cast<int8_t*>(workspace); + size_t offset = CUBLAS_WORKSPACE_SIZE; + + T* attention_mask = reinterpret_cast<T*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, attention_mask_size)); + int* cu_seqlens = reinterpret_cast<int*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, cu_seqlens_size)); + T* q_buf_2_ = reinterpret_cast<T*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, q_buf_2_size)); + T* k_buf_2_ = reinterpret_cast<T*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, k_buf_2_size)); + T* v_buf_2_ = reinterpret_cast<T*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, v_buf_2_size)); + T* qk_buf_ = reinterpret_cast<T*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, qk_buf_size)); + T* qkv_buf_2_ = reinterpret_cast<T*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, qkv_buf_2_size)); + float* qk_buf_float_ + = reinterpret_cast<float*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, qk_buf_float_size)); + int* padding_offset = reinterpret_cast<int*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, padding_offset_size)); + uint32_t* fmha_tile_counter_ptr + = reinterpret_cast<uint32_t*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, fmha_scheduler_counter)); + + __nv_fp8_e4m3* quanted_qkv_ptr + = reinterpret_cast<__nv_fp8_e4m3*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, quanted_qkv_size)); + float* q_scale_ptr = reinterpret_cast<float*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, q_scale_size)); + float* k_scale_ptr = reinterpret_cast<float*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, k_scale_size)); + float* v_scale_ptr = reinterpret_cast<float*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, v_scale_size)); + float* scale_bmm1_ptr + = reinterpret_cast<float*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, scale_bmm1_device_size)); + float* scale_bmm2_ptr + = reinterpret_cast<float*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, scale_bmm2_device_size)); + void* sage_quant_space_ptr + = reinterpret_cast<void*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, sage_quant_space_size)); + + T* ring_q_buf_ = reinterpret_cast<T*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, ring_q_buf_size)); + T* ring_kv_buf_1_ = reinterpret_cast<T*>( + tc::nextWorkspacePtr(workspace_byte_ptr, offset, ring_kv_buf_size + sizeof(int) * (batch_size + 1))); + T* ring_kv_buf_2_ = reinterpret_cast<T*>( + tc::nextWorkspacePtr(workspace_byte_ptr, offset, ring_kv_buf_size + sizeof(int) * (batch_size + 1))); + float* ring_softmax_stats_buf_ + = reinterpret_cast<float*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, ring_softmax_stats_buf_size)); + float* ring_softmax_accu_stats_buf_ + = reinterpret_cast<float*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, ring_softmax_stats_buf_size)); + T* ring_block_output_ + = reinterpret_cast<T*>(tc::nextWorkspacePtr(workspace_byte_ptr, offset, ring_block_output_size)); + + // build attention_mask, cu_seqlens, and padding_offset tensors + BuildDecoderInfoParams<T> params{}; + params.seqQOffsets = cu_seqlens; + params.paddingOffsets = padding_offset; + params.attentionMask = attention_mask; + params.seqQLengths = input_lengths; + params.batchSize = batch_size; + params.maxQSeqLength = input_seq_len; + params.numTokens = num_tokens; + params.attentionMaskType = AttentionMaskType::PADDING; + params.fmhaTileCounter = fmha_tile_counter_ptr; + if (mSageAttn) + { + params.fmhaHostBmm1Scale = 1.0f / (sqrtf(mHeadSize * 1.0f) * q_scaling); + params.fmhaBmm1Scale = scale_bmm1_ptr; + params.fmhaBmm2Scale = scale_bmm2_ptr; + } + invokeBuildDecoderInfo(params, stream); + sync_check_cuda_error(stream); + + auto const gemm_data_type = tc::CudaDataType<T>::value; + int const attention_seq_len_1 = request_seq_len; // q length + int const attention_seq_len_2 = request_seq_len; // kv length + + // If the model has relative attentiona bias, q scaling should be applied in QK gemm stage and use 1 in + // softamax stage (because to get softmax[scale(Q*K) + rel pos bias] here, q_scaling can't be applied during + // softmax phase by qk_scale); otherwise, use 1 in gemm stage and apply scaling in softmax stage + float const qk_scale + = 1.0f / (sqrtf(mHeadSize * 1.0f) * q_scaling); // q_scaling in denominator. by default q_scaling =1.0f + float const qk_scale_gemm = mRelativeAttention ? qk_scale : 1.0f; + T const qk_scale_softmax = static_cast<T>(mRelativeAttention ? 1.0f : qk_scale); + + T* linear_bias_slopes = nullptr; + + // FMHA doesn't apply to MHA with relative attention bias, i.e. softmax(QK + bias) * V + // We update mEnableContextFMHA in constructor to check this condition + if (mEnableContextFMHA) + { + if (enableRingAttn) + { + // make sure the padding part of key/value buffer is 0 + cudaMemsetAsync(ring_kv_buf_1_, 0, + reinterpret_cast<int8_t*>(ring_kv_buf_2_) - reinterpret_cast<int8_t*>(ring_kv_buf_1_), stream); + + cudaMemcpyAsync(ring_q_buf_, attention_input, ring_q_buf_size, cudaMemcpyDeviceToDevice, stream); + cudaMemcpyAsync(ring_kv_buf_1_, + const_cast<char*>(reinterpret_cast<char const*>(attention_input)) + ring_q_buf_size, ring_kv_buf_size, + cudaMemcpyDeviceToDevice, stream); + cudaMemcpyAsync(reinterpret_cast<char*>(ring_kv_buf_1_) + ring_kv_buf_size, cu_seqlens, + sizeof(int) * (batch_size + 1), cudaMemcpyDeviceToDevice, stream); + // init softmax_stats + cudaMemsetAsync(ring_softmax_accu_stats_buf_, 0, ring_softmax_stats_buf_size, stream); + +#if ENABLE_MULTI_DEVICE + // relative position of prev/next rank in cp group + int prev_rank = mCpRank > 0 ? mCpRank - 1 : mCpGroup.size() - 1; + int next_rank = (mCpRank == static_cast<int>(mCpGroup.size() - 1)) ? 0 : mCpRank + 1; +#endif // ENABLE_MULTI_DEVICE + + common::check_cuda_error(cudaStreamCreate(&mNcclStream)); + common::check_cuda_error(cudaStreamSynchronize(stream)); + + uint32_t* fmha_scheduler_counter_h = (uint32_t*) malloc(sizeof(uint32_t)); + cudaMemcpyAsync( + fmha_scheduler_counter_h, fmha_tile_counter_ptr, sizeof(uint32_t), cudaMemcpyDeviceToHost, stream); + for (size_t iter = 0; iter < mCpGroup.size(); ++iter) + { + // KV buffer used by fmha + T* ring_fmha_kv_buf_ = (iter % 2 == 0) ? ring_kv_buf_1_ : ring_kv_buf_2_; +#if ENABLE_MULTI_DEVICE + T* ring_send_kv_buf_ = (iter % 2 == 0) ? ring_kv_buf_1_ : ring_kv_buf_2_; + T* ring_recv_kv_buf_ = (iter % 2 == 0) ? ring_kv_buf_2_ : ring_kv_buf_1_; + if (iter < mCpGroup.size() - 1) + { + NCCLCHECK(ncclGroupStart()); + TLLM_CHECK_WITH_INFO(mNcclComm.get() != nullptr, "mNcclComm should be initialized before used"); + NCCLCHECK(ncclSend(ring_send_kv_buf_, + ring_kv_buf_size / sizeof(T) + sizeof(int) / sizeof(T) * (batch_size + 1), + (*getDtypeMap())[inputDesc[0].type], next_rank, *mNcclComm, mNcclStream)); + NCCLCHECK(ncclRecv(ring_recv_kv_buf_, + ring_kv_buf_size / sizeof(T) + sizeof(int) / sizeof(T) * (batch_size + 1), + (*getDtypeMap())[inputDesc[0].type], prev_rank, *mNcclComm, mNcclStream)); + NCCLCHECK(ncclGroupEnd()); + } +#else + TLLM_LOG_ERROR("Please set ENABLE_MULTI_DEVICE to enable RingAttention"); + return 1; +#endif // ENABLE_MULTI_DEVICE + // Construct the fmha params for running kernels. + MHARunnerParams fmhaParams{}; + fmhaParams.b = request_batch_size; + fmhaParams.qSeqLen = request_seq_len; + fmhaParams.kvSeqLen = request_seq_len; + fmhaParams.totalQSeqLen = request_batch_size * request_seq_len; + // Device buffer pointers. + fmhaParams.qPtr = ring_q_buf_; + fmhaParams.kvPtr = ring_fmha_kv_buf_; + if (iter == 0) + { + fmhaParams.outputPtr = context_buf_; + fmhaParams.softmaxStatsPtr = ring_softmax_accu_stats_buf_; + } + else + { + cudaMemsetAsync(ring_softmax_stats_buf_, 0, ring_softmax_stats_buf_size, stream); + fmhaParams.outputPtr = ring_block_output_; + fmhaParams.softmaxStatsPtr = ring_softmax_stats_buf_; + } + fmhaParams.cuQSeqLenPtr = cu_seqlens; + fmhaParams.cuKvSeqLenPtr + = reinterpret_cast<int*>(reinterpret_cast<char*>(ring_fmha_kv_buf_) + ring_kv_buf_size); + + fmhaParams.tileCounterPtr = fmha_tile_counter_ptr; + fmhaParams.stream = stream; + // Run the fmha kernel. + cudaMemsetAsync(fmhaParams.outputPtr, 0, ring_block_output_size, stream); + cudaMemcpyAsync(fmhaParams.tileCounterPtr, fmha_scheduler_counter_h, sizeof(uint32_t), + cudaMemcpyHostToDevice, stream); + mFmhaDispatcher->run(fmhaParams); + if (iter != 0) + { + invokeRecoverFromRA<T>((T*) context_buf_, (float*) ring_softmax_accu_stats_buf_, + (T*) ring_block_output_, (float*) ring_softmax_stats_buf_, fmhaParams.b, fmhaParams.qSeqLen, + mNumHeads, mHeadSize, cu_seqlens, stream); + } + cudaStreamSynchronize(stream); + cudaStreamSynchronize(mNcclStream); + } + common::check_cuda_error(cudaStreamDestroy(mNcclStream)); + free(fmha_scheduler_counter_h); + } + + else + { + if (mSageAttn && mHeadSize == 72 && mSageAttnQBlockSize == 64 && mSageAttnKBlockSize == 64 + && mSageAttnVBlockSize == 256) + { + sage_quant<72, 80, 64, 64, 256, __nv_bfloat16, __nv_fp8_e4m3, float>( + // host var + batch_size, mNumHeads, input_seq_len, true, true, + // device var + // q k v + attention_input, attention_input + mNumHeads * mHeadSize, + attention_input + 2 * mNumHeads * mHeadSize, + // stride + 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, cu_seqlens, + cu_seqlens, sage_quant_space_ptr, + // quant q k v + quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * paddedHeadSize, + quanted_qkv_ptr + 2 * mNumHeads * paddedHeadSize, + // quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * mHeadSize, context, + 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, + // scales + q_scale_ptr, k_scale_ptr, v_scale_ptr, stream); + + sync_check_cuda_error(stream); + } + if (mSageAttn && mHeadSize == 80 && mSageAttnQBlockSize == 64 && mSageAttnKBlockSize == 64 + && mSageAttnVBlockSize == 256) + { + sage_quant<80, 80, 64, 64, 256, __nv_bfloat16, __nv_fp8_e4m3, float>( + // host var + batch_size, mNumHeads, input_seq_len, true, true, + // device var + // q k v + attention_input, attention_input + mNumHeads * mHeadSize, + attention_input + 2 * mNumHeads * mHeadSize, + // stride + 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, cu_seqlens, + cu_seqlens, sage_quant_space_ptr, + // quant q k v + quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * paddedHeadSize, + quanted_qkv_ptr + 2 * mNumHeads * paddedHeadSize, + // quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * mHeadSize, context, + 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, + // scales + q_scale_ptr, k_scale_ptr, v_scale_ptr, stream); + + sync_check_cuda_error(stream); + } + if (mSageAttn && mHeadSize == 128 && mSageAttnQBlockSize == 64 && mSageAttnKBlockSize == 64 + && mSageAttnVBlockSize == 256) + { + sage_quant<128, 128, 64, 64, 256, __nv_bfloat16, __nv_fp8_e4m3, float>( + // host var + batch_size, mNumHeads, input_seq_len, true, true, + // device var + // q k v + attention_input, attention_input + mNumHeads * mHeadSize, + attention_input + 2 * mNumHeads * mHeadSize, + // stride + 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, cu_seqlens, + cu_seqlens, sage_quant_space_ptr, + // quant q k v + quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * paddedHeadSize, + quanted_qkv_ptr + 2 * mNumHeads * paddedHeadSize, + // quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * mHeadSize, context, + 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, + // scales + q_scale_ptr, k_scale_ptr, v_scale_ptr, stream); + + sync_check_cuda_error(stream); + } + if (mSageAttn && mHeadSize == 128 && mSageAttnQBlockSize == 64 && mSageAttnKBlockSize == 32 + && mSageAttnVBlockSize == 32) + { + sage_quant<128, 128, 64, 32, 32, __nv_bfloat16, __nv_fp8_e4m3, float>( + // host var + batch_size, mNumHeads, input_seq_len, true, true, + // device var + // q k v + attention_input, attention_input + mNumHeads * mHeadSize, + attention_input + 2 * mNumHeads * mHeadSize, + // stride + 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, cu_seqlens, + cu_seqlens, sage_quant_space_ptr, + // quant q k v + quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * paddedHeadSize, + quanted_qkv_ptr + 2 * mNumHeads * paddedHeadSize, + // quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * mHeadSize, context, + 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, + // scales + q_scale_ptr, k_scale_ptr, v_scale_ptr, stream); + + sync_check_cuda_error(stream); + } + if (mSageAttn && mHeadSize == 80 && mSageAttnQBlockSize == 64 && mSageAttnKBlockSize == 32 + && mSageAttnVBlockSize == 32) + { + sage_quant<80, 80, 64, 32, 32, __nv_bfloat16, __nv_fp8_e4m3, float>( + // host var + batch_size, mNumHeads, input_seq_len, true, true, + // device var + // q k v + attention_input, attention_input + mNumHeads * mHeadSize, + attention_input + 2 * mNumHeads * mHeadSize, + // stride + 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, cu_seqlens, + cu_seqlens, sage_quant_space_ptr, + // quant q k v + quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * paddedHeadSize, + quanted_qkv_ptr + 2 * mNumHeads * paddedHeadSize, + // quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * mHeadSize, context, + 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, + // scales + q_scale_ptr, k_scale_ptr, v_scale_ptr, stream); + + sync_check_cuda_error(stream); + } + if (mSageAttn && mHeadSize == 72 && mSageAttnQBlockSize == 64 && mSageAttnKBlockSize == 32 + && mSageAttnVBlockSize == 32) + { + sage_quant<72, 80, 64, 32, 32, __nv_bfloat16, __nv_fp8_e4m3, float>( + // host var + batch_size, mNumHeads, input_seq_len, true, true, + // device var + // q k v + attention_input, attention_input + mNumHeads * mHeadSize, + attention_input + 2 * mNumHeads * mHeadSize, + // stride + 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, 3 * mNumHeads * mHeadSize, cu_seqlens, + cu_seqlens, sage_quant_space_ptr, + // quant q k v + quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * paddedHeadSize, + quanted_qkv_ptr + 2 * mNumHeads * paddedHeadSize, + // quanted_qkv_ptr, quanted_qkv_ptr + mNumHeads * mHeadSize, context, + 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, 3 * mNumHeads * paddedHeadSize, + // scales + q_scale_ptr, k_scale_ptr, v_scale_ptr, stream); + + sync_check_cuda_error(stream); + } + + // Construct the fmha params for running kernels. + MHARunnerParams fmhaParams{}; + fmhaParams.b = request_batch_size; + fmhaParams.qSeqLen = request_seq_len; + fmhaParams.kvSeqLen = request_seq_len; + fmhaParams.totalQSeqLen = request_batch_size * request_seq_len; + // Device buffer pointers. + fmhaParams.qkvPtr = attention_input; + fmhaParams.outputPtr = context_buf_; + fmhaParams.cuQSeqLenPtr = cu_seqlens; + fmhaParams.cuKvSeqLenPtr = cu_seqlens; + fmhaParams.tileCounterPtr = fmha_tile_counter_ptr; + fmhaParams.stream = stream; + if (mSageAttn) + { + if (paddedHeadSize != mHeadSize) + fmhaParams.outputPtr = sage_quant_space_ptr; + fmhaParams.qkvPtr = quanted_qkv_ptr; + fmhaParams.scaleBmm1Ptr = scale_bmm1_ptr; + fmhaParams.scaleBmm2Ptr = scale_bmm2_ptr; + fmhaParams.qScalePtr = q_scale_ptr; + fmhaParams.kScalePtr = k_scale_ptr; + fmhaParams.vScalePtr = v_scale_ptr; + fmhaParams.qMaxNBlock = (input_seq_len + mSageAttnQBlockSize - 1) / mSageAttnQBlockSize; + fmhaParams.kMaxNBlock = (input_seq_len + mSageAttnKBlockSize - 1) / mSageAttnKBlockSize; + fmhaParams.vMaxNBlock = (input_seq_len + mSageAttnVBlockSize - 1) / mSageAttnVBlockSize; + } + + // Run the fmha kernel. + + // TODO: set it correctly for contiguous kv buffer (cross-attention). + fmhaParams.totalKvSeqLen = num_tokens; + + fmhaParams.cuKvSeqLenPtr = cu_seqlens; + fmhaParams.cuMaskRowsPtr = cu_seqlens; + fmhaParams.tileCounterPtr = fmha_tile_counter_ptr; + + fmhaParams.scaleBmm1Ptr = scale_bmm1_ptr; + fmhaParams.scaleBmm2Ptr = scale_bmm2_ptr; + fmhaParams.forceFp32Acc = mFMHAForceFP32Acc; + mFmhaDispatcher->run(fmhaParams); + sync_check_cuda_error(stream); + if (mSageAttn) + { + if (paddedHeadSize != mHeadSize && mHeadSize == 72) + { + unpadding<80, 72, __nv_bfloat16>(batch_size, mNumHeads, input_seq_len, sage_quant_space_ptr, + mNumHeads * 72, mNumHeads * 80, cu_seqlens, context_buf_, stream); + } + } + } + } + else + { + // FIXME: a temporary solution to make sure the padding part of key/value buffer is 0 + // NOTE: pointer subtraction is used below since there could be some extra gap due to alignment. + // Otherwise, we could do cudaMemsetAsync(k_buf_2_, 0, k_buf_2_size + v_buf_2_size, stream); + // cudaMemsetAsync(k_buf_2_, 0, reinterpret_cast<int8_t*>(qk_buf_) - reinterpret_cast<int8_t*>(k_buf_2_), + // stream); + // FIXME: the final solution is to change the add_fusedQKV_bias_transpose_kernel to map CTAs corresponding to + // the output shape, and set the padding part to 0. Without zero-initialize guarantee, these workspace buffers + // may contain random NaN values when IFB workload is high. + cudaMemsetAsync(k_buf_2_, 0, + reinterpret_cast<int8_t*>(v_buf_2_) - reinterpret_cast<int8_t*>(k_buf_2_) + v_buf_2_size, stream); + + // only non-FMHA path needs to split Q,K,V from QKV + invokeAddFusedQKVBiasTranspose(q_buf_2_, k_buf_2_, v_buf_2_, const_cast<T*>(attention_input), input_lengths, + mRemovePadding ? padding_offset : nullptr, batch_size, input_seq_len, num_tokens, mNumHeads, mNumHeads, + mHeadSize, 0, 0.0f, RotaryScalingType::kNONE, 0.0f, 0, PositionEmbeddingType::kLEARNED_ABSOLUTE, + (float*) nullptr, 0, stream); + + if (!mQKHalfAccum && gemm_data_type != CUDA_R_32F) + { + mCublasWrapper->stridedBatchedGemm(CUBLAS_OP_T, CUBLAS_OP_N, + attention_seq_len_2, // n + attention_seq_len_1, // m + mHeadSize, // k + qk_scale_gemm, k_buf_2_, gemm_data_type, + mHeadSize, // k + attention_seq_len_2 * mHeadSize, // n * k + q_buf_2_, gemm_data_type, + mHeadSize, // k + attention_seq_len_1 * mHeadSize, // m * k + 0.0f, qk_buf_float_, CUDA_R_32F, + attention_seq_len_2, // n + attention_seq_len_2 * attention_seq_len_1, + request_batch_size * mNumHeads, // global batch size + CUDA_R_32F); + + // add relative position bias + if (mRelativeAttention) + { + // add rel pos bias + // QK is (batch_size, local_head_num, q_length, k_length), rel pos bias is (1, local_head_num, + // max_output_len + 1, max_output_len + 1). broadcast along 1st dim. max_seq_len is already + // max_output_len + 1. In implicit mode, relative_attention_bias is rel attn table + // [num_heads, num_buckets], with necessary params (max_distance, num_buckets) passed at the end + invokeAddRelativeAttentionBiasUnaligned(qk_buf_float_, relative_attn_table, request_batch_size, + mNumHeads, attention_seq_len_1, attention_seq_len_2, stream, mMaxDistance > 0, + inputDesc[3].dims.d[1], mMaxDistance, true /* bidirectional */); + } + + MaskedSoftmaxParam<T, float> param; + param.attention_score = qk_buf_; // (batch_size, head_num, q_length, k_length) + param.qk = qk_buf_float_; // (batch_size, head_num, q_length, k_length) + param.attention_mask = attention_mask; // (batch_size, q_length, k_length) + param.batch_size = request_batch_size; + param.q_length = attention_seq_len_1; + param.k_length = attention_seq_len_2; + param.num_heads = mNumHeads; + param.qk_scale = qk_scale_softmax; + param.linear_bias_slopes = const_cast<T*>(linear_bias_slopes); // (head_num,), optional + invokeMaskedSoftmax(param, stream); + } + else + { + mCublasWrapper->stridedBatchedGemm(CUBLAS_OP_T, CUBLAS_OP_N, attention_seq_len_2, attention_seq_len_1, + mHeadSize, k_buf_2_, mHeadSize, attention_seq_len_2 * mHeadSize, q_buf_2_, mHeadSize, + attention_seq_len_1 * mHeadSize, qk_buf_, attention_seq_len_2, + attention_seq_len_2 * attention_seq_len_1, request_batch_size * mNumHeads, qk_scale_gemm, + 0.0f); // alpha, beta + + // add relative position bias + if (mRelativeAttention) + { + // add rel pos bias + // QK is (batch_size, local_head_num, q_length, k_length), rel pos bias is (1, local_head_num, + // max_output_len + 1, max_output_len + 1). broadcast along 1st dim. max_seq_len is already + // max_output_len + 1. In implicit mode, relative_attention_bias is rel attn table + // [num_heads, num_buckets], with necessary params (max_distance, num_buckets) passed at the end + invokeAddRelativeAttentionBiasUnaligned(qk_buf_, relative_attn_table, request_batch_size, mNumHeads, + attention_seq_len_1, attention_seq_len_2, stream, mMaxDistance > 0, inputDesc[3].dims.d[1], + mMaxDistance, true /* bidirectional */); + } + + MaskedSoftmaxParam<T, T> param; + param.attention_score = qk_buf_; // (batch_size, head_num, q_length, k_length) + param.qk = qk_buf_; // (batch_size, head_num, q_length, k_length) + param.attention_mask = attention_mask; // (batch_size, q_length, k_length) + param.batch_size = request_batch_size; + param.q_length = attention_seq_len_1; + param.k_length = attention_seq_len_2; + param.num_heads = mNumHeads; + param.qk_scale = qk_scale_softmax; + param.linear_bias_slopes = const_cast<T*>(linear_bias_slopes); // (head_num,), optional + invokeMaskedSoftmax(param, stream); + } + + mCublasWrapper->stridedBatchedGemm(CUBLAS_OP_N, CUBLAS_OP_N, mHeadSize, attention_seq_len_1, + attention_seq_len_2, v_buf_2_, mHeadSize, attention_seq_len_2 * mHeadSize, qk_buf_, attention_seq_len_2, + attention_seq_len_1 * attention_seq_len_2, qkv_buf_2_, mHeadSize, attention_seq_len_1 * mHeadSize, + request_batch_size * mNumHeads); + + if (!mRemovePadding) + { + invokeTransposeQKV(context_buf_, qkv_buf_2_, request_batch_size, attention_seq_len_1, mNumHeads, mHeadSize, + (float*) nullptr, 0, stream); + } + else + { + invokeTransposeAttentionOutRemovePadding(qkv_buf_2_, context_buf_, num_tokens, request_batch_size, + request_seq_len, mNumHeads, mHeadSize, padding_offset, (float*) nullptr, 0, stream); + } + } + sync_check_cuda_error(stream); + return 0; +} + +template int BertAttentionPlugin::enqueueImpl<half>(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream); + +template int BertAttentionPlugin::enqueueImpl<float>(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream); + +#ifdef ENABLE_BF16 +template int BertAttentionPlugin::enqueueImpl<__nv_bfloat16>(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream); +#endif + +int BertAttentionPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + if (mType == DataType::kHALF) + { + return enqueueImpl<half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kFLOAT) + { + return enqueueImpl<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16) + { + return enqueueImpl<__nv_bfloat16>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType BertAttentionPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return inputTypes[0]; +} + +// IPluginV2 Methods + +char const* BertAttentionPlugin::getPluginType() const noexcept +{ + return BERT_ATTENTION_PLUGIN_NAME; +} + +char const* BertAttentionPlugin::getPluginVersion() const noexcept +{ + return BERT_ATTENTION_PLUGIN_VERSION; +} + +int BertAttentionPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int BertAttentionPlugin::initialize() noexcept +{ + auto cublasHandle = getCublasHandle(); + auto cublasLtHandle = getCublasLtHandle(); + mCublasWrapper.reset(new tc::CublasMMWrapper(cublasHandle, cublasLtHandle, nullptr, nullptr)); + if (mEnableContextFMHA) + { + // Pre-checked during constructing. + Data_type data_type; + if (mType == DataType::kHALF) + { + data_type = DATA_TYPE_FP16; + } + else if (mType == DataType::kBF16) + { + data_type = DATA_TYPE_BF16; + } + else + { + TLLM_CHECK_WITH_INFO(false, "GPTAttentionPlugin received wrong data type."); + } + + // Construct the fmha runner. + MHARunnerFixedParams fmhaParams{}; + if (mSageAttn) + { + fmhaParams.dataType = DATA_TYPE_E4M3; + } + else + { + fmhaParams.dataType = data_type; + } + fmhaParams.dataTypeOut = data_type; + fmhaParams.forceFp32Acc = mFMHAForceFP32Acc; + fmhaParams.attentionMaskType = ContextAttentionMaskType::PADDING; + fmhaParams.isSPadded = !mRemovePadding; + fmhaParams.numQHeads = mNumHeads; + fmhaParams.numKvHeads = mNumHeads; + fmhaParams.headSize = mHeadSize; + fmhaParams.qScaling = mQScaling; + fmhaParams.sageBlockSizeQ = mSageAttnQBlockSize; + fmhaParams.sageBlockSizeK = mSageAttnKBlockSize; + fmhaParams.sageBlockSizeV = mSageAttnVBlockSize; + if (mSageAttn) + { + int const paddedHeadSize = ((mHeadSize + 15) / 16) * 16; + fmhaParams.headSize = paddedHeadSize; + } + + if (mCpGroup.size() > 1) + { + fmhaParams.attentionInputLayout = AttentionInputLayout::Q_CONTIGUOUS_KV; + fmhaParams.saveSoftmax = true; + } + + // Load kernels from the pre-compiled cubins. + // The KV input data type. The default is same as dataType. + fmhaParams.dataTypeKv = data_type; + fmhaParams.headSizeV = mHeadSize; + + // Load kernels from the pre-compiled cubins. + mFmhaDispatcher.reset(new FmhaDispatcher(fmhaParams)); + // Fall back to unfused MHA kernels if not supported. + mEnableContextFMHA = mFmhaDispatcher->isSupported(); + } + +#if ENABLE_MULTI_DEVICE + if (mCpGroup.size() > 1 && COMM_SESSION.getSize() > 1) + { + TLLM_LOG_TRACE("%s start for rank %d", __PRETTY_FUNCTION__, COMM_SESSION.getRank()); + mNcclComm = getComm(mCpGroup); + TLLM_LOG_TRACE("%s stop for rank %d", __PRETTY_FUNCTION__, COMM_SESSION.getRank()); + } +#endif // ENABLE_MULTI_DEVICE + + return 0; +} + +void BertAttentionPlugin::destroy() noexcept +{ + delete this; +} + +size_t BertAttentionPlugin::getSerializationSize() const noexcept +{ + return sizeof(mNumHeads) + sizeof(mHeadSize) + sizeof(mQScaling) + sizeof(mQKHalfAccum) + sizeof(mEnableContextFMHA) + + sizeof(mFMHAForceFP32Acc) + sizeof(mType) + sizeof(mRelativeAttention) + sizeof(mMaxDistance) + + sizeof(mRemovePadding) + sizeof(mSageAttn) + sizeof(mSageAttnQBlockSize) + sizeof(mSageAttnKBlockSize) + + sizeof(mSageAttnVBlockSize) + sizeof(mCpSize) + sizeof(mCpRank) + sizeof(int32_t) * mCpGroup.size(); +} + +void BertAttentionPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mNumHeads); + write(d, mHeadSize); + write(d, mQScaling); + write(d, mQKHalfAccum); + write(d, mEnableContextFMHA); + write(d, mFMHAForceFP32Acc); + write(d, mType); + write(d, mRelativeAttention); + write(d, mMaxDistance); + write(d, mRemovePadding); + write(d, mSageAttn); + write(d, mSageAttnQBlockSize); + write(d, mSageAttnKBlockSize); + write(d, mSageAttnVBlockSize); + write(d, mCpSize); + write(d, mCpRank); + for (auto it = mCpGroup.begin(); it != mCpGroup.end(); ++it) + { + write(d, *it); + } + TLLM_CHECK(d == a + getSerializationSize()); +} + +void BertAttentionPlugin::terminate() noexcept {} + +/////////////// + +BertAttentionPluginCreator::BertAttentionPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + + mPluginAttributes.emplace_back(PluginField("num_heads", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("head_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("q_scaling", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("context_fmha_type", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("do_relative_attention", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("max_distance", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("remove_padding", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("sage_attn", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("sage_attn_q_block_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("sage_attn_k_block_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("sage_attn_v_block_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("cp_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("cp_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("cp_group", nullptr, PluginFieldType::kINT32)); + + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* BertAttentionPluginCreator::getPluginName() const noexcept +{ + return BERT_ATTENTION_PLUGIN_NAME; +} + +char const* BertAttentionPluginCreator::getPluginVersion() const noexcept +{ + return BERT_ATTENTION_PLUGIN_VERSION; +} + +PluginFieldCollection const* BertAttentionPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* BertAttentionPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int num_heads{}; + int head_size{}; + ContextFMHAType context_fmha_type{}; + float q_scaling{}; + nvinfer1::DataType type{}; + bool do_relative_attention{}; + int max_distance{}; + bool remove_padding{}; + bool sage_attn{}; + int sage_attn_q_block_size{}; + int sage_attn_k_block_size{}; + int sage_attn_v_block_size{}; + int cp_size{}; + int cp_rank{}; + std::set<int> cp_group{}; + + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "num_heads")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + num_heads = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "head_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + head_size = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "q_scaling")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + q_scaling = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + else if (!strcmp(attrName, "context_fmha_type")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + context_fmha_type = static_cast<ContextFMHAType>(*(static_cast<int8_t const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "do_relative_attention")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + do_relative_attention = static_cast<bool>(*(static_cast<int8_t const*>(fields[i].data))); + } + else if (!strcmp(attrName, "max_distance")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + max_distance = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "remove_padding")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + remove_padding = static_cast<bool>(*(static_cast<int8_t const*>(fields[i].data))); + } + else if (!strcmp(attrName, "sage_attn")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + sage_attn = static_cast<bool>(*(static_cast<int8_t const*>(fields[i].data))); + if (sage_attn) + { + std::cout << "sage attn true!" << std::endl; + } + } + else if (!strcmp(attrName, "sage_attn_q_block_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + sage_attn_q_block_size = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "sage_attn_k_block_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + sage_attn_k_block_size = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "sage_attn_v_block_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + sage_attn_v_block_size = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "cp_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + cp_size = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "cp_rank")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + cp_rank = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "cp_group")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + auto const* r = static_cast<int const*>(fields[i].data); + for (int j = 0; j < fields[i].length; ++j) + { + cp_group.insert(*r); + ++r; + } + } + } + try + { + auto* obj = new BertAttentionPlugin(num_heads, head_size, q_scaling, context_fmha_type, type, + do_relative_attention, max_distance, remove_padding, sage_attn, sage_attn_q_block_size, + sage_attn_k_block_size, sage_attn_v_block_size, cp_size, cp_rank, cp_group); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* BertAttentionPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call BertAttentionPlugin::destroy() + try + { + auto* obj = new BertAttentionPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/bertAttentionPlugin/bertAttentionPlugin.h b/cpp/tensorrt_llm/plugins/bertAttentionPlugin/bertAttentionPlugin.h new file mode 100644 index 000000000000..2eb39086a005 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/bertAttentionPlugin/bertAttentionPlugin.h @@ -0,0 +1,142 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/cublasMMWrapper.h" +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/kernels/fmhaDispatcher.h" +#include "tensorrt_llm/kernels/gptKernels.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include <cassert> +#include <cuda_runtime.h> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class BertAttentionPlugin : public BasePlugin +{ +public: + BertAttentionPlugin() = delete; + + BertAttentionPlugin(int num_heads, int head_size, float q_scaling, + tensorrt_llm::kernels::ContextFMHAType context_fmha_type, nvinfer1::DataType type, + bool do_relative_attention = false, int max_distance = 0, bool remove_padding = false, bool sage_attn = false, + int sage_attn_q_block_size = 0, int sage_attn_k_block_size = 0, int sage_attn_v_block_size = 0, int cp_size = 1, + int cp_rank = 0, std::set<int> cp_group = {}); + + BertAttentionPlugin(void const* data, size_t length); + + ~BertAttentionPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + template <typename T> + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + const std::string mLayerName; + + int mNumHeads; + int mHeadSize; + float mQScaling; + nvinfer1::DataType mType; + bool mRelativeAttention = false; + int mMaxDistance = 0; + bool mRemovePadding = false; + + // unfused mha + bool mQKHalfAccum = false; + + // fmha runner (disable by default) + bool mEnableContextFMHA = false; + bool mFMHAForceFP32Acc = false; + + // sage attention + bool mSageAttn = false; + int mSageAttnQBlockSize = 0; + int mSageAttnKBlockSize = 0; + int mSageAttnVBlockSize = 0; + std::set<std::vector<int>> mSageAttnSupportedBlockSizes{{64, 64, 256}, {64, 32, 32}}; + + int mSM = tensorrt_llm::common::getSMVersion(); + + // comm group for RingAttention + int mCpSize = 1; + int mCpRank = 0; + std::set<int> mCpGroup = {}; +#if ENABLE_MULTI_DEVICE + std::shared_ptr<ncclComm_t> mNcclComm; +#endif // ENABLE_MULTI_DEVICE + cudaStream_t mNcclStream; + + // The default copy constructor will leave them as nullptr. clone() shall initialize it. + UniqPtrWNullCopy<tensorrt_llm::kernels::FmhaDispatcher> mFmhaDispatcher; + UniqPtrWNullCopy<tensorrt_llm::common::CublasMMWrapper> mCublasWrapper; +}; + +class BertAttentionPluginCreator : public BaseCreator +{ +public: + BertAttentionPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/common/CMakeLists.txt b/cpp/tensorrt_llm/plugins/common/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/common/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/common/checkMacrosPlugin.cpp b/cpp/tensorrt_llm/plugins/common/checkMacrosPlugin.cpp new file mode 100644 index 000000000000..2aab6b3675d8 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/common/checkMacrosPlugin.cpp @@ -0,0 +1,35 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "checkMacrosPlugin.h" + +#include "tensorrt_llm/common/logger.h" + +namespace tensorrt_llm::plugins +{ + +void caughtError(std::exception const& e) +{ + TLLM_LOG_EXCEPTION(e); +} + +void logError(char const* msg, char const* file, char const* fn, int line) +{ + TLLM_LOG_ERROR("Parameter check failed at: %s::%s::%d, condition: %s", file, fn, line, msg); +} + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/common/checkMacrosPlugin.h b/cpp/tensorrt_llm/plugins/common/checkMacrosPlugin.h new file mode 100644 index 000000000000..d8d8af1ef220 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/common/checkMacrosPlugin.h @@ -0,0 +1,29 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" + +namespace tensorrt_llm::plugins +{ + +void logError(char const* msg, char const* file, char const* fn, int line); + +void caughtError(std::exception const& e); + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/common/gemmPluginProfiler.cpp b/cpp/tensorrt_llm/plugins/common/gemmPluginProfiler.cpp new file mode 100644 index 000000000000..e5d6650648ab --- /dev/null +++ b/cpp/tensorrt_llm/plugins/common/gemmPluginProfiler.cpp @@ -0,0 +1,404 @@ +/* + * Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/common/cublasMMWrapper.h" +#include "tensorrt_llm/kernels/cutlass_kernels/fp8_rowwise_gemm/fp8_rowwise_gemm.h" +#include "tensorrt_llm/kernels/cutlass_kernels/fpA_intB_gemm/fpA_intB_gemm.h" +#include "tensorrt_llm/kernels/cutlass_kernels/fused_gated_gemm/fused_gated_gemm.h" +#include "tensorrt_llm/kernels/cutlass_kernels/int8_gemm/int8_gemm.h" +#include "tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePlugin.h" +#include "tensorrt_llm/plugins/lowLatencyGemmPlugin/lowLatencyGemmPlugin.h" +#include "tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/lowLatencyGemmSwigluPlugin.h" +#include "tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.h" +#if defined(USING_OSS_CUTLASS_FP4_GEMM) +#include "tensorrt_llm/kernels/cutlass_kernels/include/fp4_gemm.h" +#else +#include "fp4_gemm.h" +#endif +#if defined(USING_OSS_CUTLASS_ALLREDUCE_GEMM) +#include "tensorrt_llm/kernels/cutlass_kernels/include/allreduce_gemm_runner.h" +using GemmAllReduceImplInterface = tensorrt_llm::kernels::opened_cutlass_kernels::GemmAllReduceImplInterface; +#else +#include "allreduce_gemm_runner.h" +using GemmAllReduceImplInterface = tensorrt_llm::kernels::cutlass_kernels::GemmAllReduceImplInterface; +#endif + +#include <cstddef> + +namespace tensorrt_llm::plugins +{ + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::GemmPluginProfiler() +{ + mMNKProfileMap = std::make_shared<MNKProfileMap>(); + + // set SKIP_GEMM_PLUGIN_PROFILINGS=1 to avoid tactics profilings + auto const skipEnv = std::getenv("SKIP_GEMM_PLUGIN_PROFILINGS"); + mSkip = (skipEnv != NULL && std::stoi(skipEnv)); + if (mSkip) + { + TLLM_LOG_DEBUG( + "SKIP_GEMM_PLUGIN_PROFILINGS is set. Skipping GEMM plugin profilings. It could result in runtime error " + "if default tactic is not defined."); + } +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +void GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::serialize( + char*& buffer, GemmIdType const& gemmId) const +{ + auto mProfileMap = mMNKProfileMap->getMProfileMap(gemmId); + + // Save number of profiles for given GEMM ID + write(buffer, static_cast<int>(mProfileMap->size())); + for (auto const& pair : *mProfileMap) + { + // Save pair of M to the best GEMM config + write(buffer, pair); + } +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +void GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::deserialize( + char const*& data, GemmDims& dims, GemmIdType const& gemmId) +{ + // NOTE: this mutex is not needed since each thread owns its private map, but will put here for + // consistency + writer_lock lock(mMNKProfileMap->mutex); + + mDims = dims; + + // GemmId gemmId(dims.n, dims.k); + if (!mMNKProfileMap->existsMProfileMap(gemmId)) + { + // Create GEMM with GEMM ID if it does not exist + mMNKProfileMap->createMProfileMap(gemmId); + } + // Populate map with profiles of GEMM ID + auto profileMap = mMNKProfileMap->getMProfileMap(gemmId); + int selectedMapSize; + read(data, selectedMapSize); + for (int ii = 0; ii < selectedMapSize; ++ii) + { + std::pair<int, std::optional<Config>> config; + read(data, config); + profileMap->insert(config); + } +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +size_t GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::getSerializationSize( + GemmIdType const& gemmId) const +{ + reader_lock lock(mMNKProfileMap->mutex); + return sizeof(int) + // size of the tactics map + mMNKProfileMap->getMProfileMap(gemmId)->size() + * sizeof(std::pair<int, std::optional<Config>>); // size of the tactics map +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +int GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::getMaxProfileM() const +{ + return 8192; +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +void GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::initTmpData( + int m, int n, int k, char* workspace, size_t size, cudaStream_t stream) +{ + /* Do nothing */ +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +void GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::profileTactics(RunnerPtr const& runner, + nvinfer1::DataType const& type, GemmDims const& dims, GemmIdType const& gemmId, bool hasWeightOnlyCudaKernel) +{ + writer_lock lock(mMNKProfileMap->mutex); + + if (!dims.isInitialized()) + { + return; + } + + mRunner = runner; + mType = type; + + int const maxM = std::min(nextPowerOfTwo(dims.maxM), getMaxProfileM()); + computeTmpSize(maxM, dims.n, dims.k); + + if (!mMNKProfileMap->existsMProfileMap(gemmId)) + { + // Create map for GEMM ID + mMNKProfileMap->createMProfileMap(gemmId); + } + + if (mSkip) + { + return; + } + + auto mProfileMap = mMNKProfileMap->getMProfileMap(gemmId); + bool isAllocated{false}; + + auto profileTactics = [&mProfileMap, &isAllocated, this](int m, int n, int k) + { + if (mProfileMap->count(m) == 0) + { + if (!isAllocated) + { + // Allocate tmp data to run GEMMs + allocateTmpData(); + isAllocated = true; + } + initTmpData(m, n, k, mWorkspaceTmp, mTmpWorkspaceSizeInBytes, mStream); + auto tactics = this->getTactics(m, n, k); + + // Profile different tactics for particular m and insert best config to the map + mProfileMap->insert({m, this->profileTacticsForProblem(m, n, k, tactics)}); + } + }; + + common::check_cuda_error(cudaStreamCreate(&mStream)); + + int const startMinMRounded = nextPowerOfTwo(dims.minM); + + if (hasWeightOnlyCudaKernel) + { + // Profile tactics for finer granularity of M, + // if CUDA kernel is enabled for weight-only plugins + int minM = dims.minM; + for (int m = std::max(1, minM); m < std::min(16, maxM); m += 1) + { + profileTactics(m, dims.n, dims.k); + } + + for (int m = 16; m < maxM; m *= 2) + { + profileTactics(m, dims.n, dims.k); + } + } + else + { + // Profile tactics for CUTLASS kernel only + for (int m = std::max(1, startMinMRounded); m < maxM; m *= 2) + { + profileTactics(m, dims.n, dims.k); + } + } + + profileTactics(maxM, dims.n, dims.k); + + if (isAllocated) + { + // Free tmp data + freeTmpData(); + } + common::check_cuda_error(cudaStreamDestroy(mStream)); +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +std::optional<Config> GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::getBestConfig( + int m, GemmIdType const& gemmId) const +{ + reader_lock lock(mMNKProfileMap->mutex); + + if (mSkip) + { + TLLM_LOG_TRACE("Skip is set, no best config is set for this instance"); + return std::nullopt; + } + + int const mRounded = std::min(std::max(1, nextPowerOfTwo(m)), getMaxProfileM()); + fflush(stdout); + + if (mMNKProfileMap->getMProfileMap(gemmId)->count(m) > 0) + { + return mMNKProfileMap->getMProfileMap(gemmId)->at(m); + } + else if (mMNKProfileMap->getMProfileMap(gemmId)->count(mRounded) > 0) + { + return mMNKProfileMap->getMProfileMap(gemmId)->at(mRounded); + } + else + { + std::ostringstream msg; + msg << "Cannot find best tactic for m=" << m << " and GEMM ID " << gemmId; + TLLM_LOG_WARNING(msg.str()); + return std::nullopt; + } +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +void GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::allocateTmpData() +{ + TLLM_CHECK_WITH_INFO(mTmpWorkspaceSizeInBytes > 0, "tmpWorkspaceSizeInBytes must be larger than 0"); + auto const status = cudaMalloc(&mWorkspaceTmp, mTmpWorkspaceSizeInBytes); + TLLM_CHECK_WITH_INFO(status == cudaSuccess, "Can't allocate tmp workspace for GEMM tactics profiling."); +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +void GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::freeTmpData() +{ + auto const status = cudaFree(mWorkspaceTmp); + TLLM_CHECK_WITH_INFO(status == cudaSuccess, "Can't free tmp workspace for GEMM tactics profiling."); +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +std::optional<Config> GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::profileTacticsForProblem( + int m, int n, int k, std::vector<Config> const& tactics) +{ + TLLM_LOG_DEBUG(__PRETTY_FUNCTION__); + + float bestTime = std::numeric_limits<float>::max(); + Config bestConfig; + bool foundOne = false; + + // Iterate over all tactics for given M, N and K + for (size_t ii = 0; ii < tactics.size(); ++ii) + { + Config const& candidateConfig = tactics[ii]; + float time = std::numeric_limits<float>::max(); + try + { + if (!checkTactic(m, n, k, candidateConfig)) + { + continue; + } + // Profile particular tactic for given M, N and K + time = profileTacticForProblem(m, n, k, candidateConfig); + foundOne = true; + } + catch (std::exception const& e) + { + std::ostringstream msg; + msg << "Cannot profile configuration " << ii; + if constexpr (std::is_same_v<Config, tensorrt_llm::cutlass_extensions::CutlassGemmConfig>) + { + msg << ": " << candidateConfig.toString(); + } + msg << "\n (for" + << " m=" << m << ", n=" << n << ", k=" << k << ")" + << ", reason: \"" << e.what() << "\". Skipped"; + TLLM_LOG_TRACE(msg.str()); + cudaGetLastError(); // Reset the last cudaError to cudaSuccess. + continue; + } + + // Choose the fastest tactic + if (time < bestTime) + { + bestConfig = candidateConfig; + bestTime = time; + } + } + + if (!foundOne) + { + std::ostringstream msg; + msg << "Have not found any valid GEMM config for shape (" + << "m=" << m << ", n=" << n << ", k=" << k << "). Will try to use default or fail at runtime"; + TLLM_LOG_WARNING(msg.str()); + return std::nullopt; + } + + return {bestConfig}; +} + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +float GemmPluginProfiler<Config, RunnerPtr, GemmIdType, GemmIdHashType>::profileTacticForProblem( + int m, int n, int k, Config const& tactic) +{ + constexpr int warmup = 5; + constexpr int runs = 10; + + cudaStream_t stream = mStream; + + // Warmup the execution + for (int i = 0; i < warmup; ++i) + { + runTactic(m, n, k, tactic, mWorkspaceTmp, stream); + } + + cudaEvent_t start; + cudaEvent_t stop; + common::check_cuda_error(cudaEventCreate(&start)); + common::check_cuda_error(cudaEventCreate(&stop)); + common::check_cuda_error(cudaStreamSynchronize(stream)); + common::check_cuda_error(cudaEventRecord(start, stream)); + + // Profile GEMM + for (int i = 0; i < runs; ++i) + { + runTactic(m, n, k, tactic, mWorkspaceTmp, stream); + } + + common::check_cuda_error(cudaEventRecord(stop, stream)); + + common::check_cuda_error(cudaEventSynchronize(stop)); + + float elapsed; + common::check_cuda_error(cudaEventElapsedTime(&elapsed, start, stop)); + + common::check_cuda_error(cudaEventDestroy(start)); + common::check_cuda_error(cudaEventDestroy(stop)); + + return elapsed / runs; +} + +template class GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassInt8GemmRunnerInterface>, GemmIdCore, + GemmIdCoreHash>; + +template class GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassFpAIntBGemmRunnerInterface>, GemmIdCore, + GemmIdCoreHash>; + +template class GemmPluginProfiler<cublasLtMatmulHeuristicResult_t, + std::shared_ptr<tensorrt_llm::common::CublasMMWrapper>, GemmIdCublas, GemmIdCublasHash>; + +// TODO I dont like the dependency on the MOE plugin here, but MOE needs the full context to run profiles +template class GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, MixtureOfExpertsPlugin*, + GemmIDMoe, GemmIDMoeHash>; + +template class GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassFusedGatedGemmRunnerInterface>, GemmIdCore, + GemmIdCoreHash>; + +template class GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassFp8RowwiseGemmRunnerInterface>, GemmIdCore, + GemmIdCoreHash>; + +#if defined(USING_OSS_CUTLASS_FP4_GEMM) +template class GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassFp4GemmRunnerInterface>, GemmIdCore, GemmIdCoreHash>; +#else +template class GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + std::shared_ptr<tensorrt_llm::kernels::internal_cutlass_kernels::CutlassFp4GemmRunnerInterface>, GemmIdCore, + GemmIdCoreHash>; +#endif + +template class GemmPluginProfiler<LowLatencyGemmPluginProfiler::Config, LowLatencyGemmRunnerPtr, GemmIdCore, + GemmIdCoreHash>; + +template class GemmPluginProfiler<LowLatencyGemmSwigluPluginProfiler::Config, LowLatencyGemmSwigluRunnerPtr, GemmIdCore, + GemmIdCoreHash>; + +template class GemmPluginProfiler<GemmAllReduceImplInterface::LaunchConfig, std::shared_ptr<GemmAllReduceImplInterface>, + GemmIdCore, GemmIdCoreHash>; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/common/gemmPluginProfiler.h b/cpp/tensorrt_llm/plugins/common/gemmPluginProfiler.h new file mode 100644 index 000000000000..fe85b3b7e456 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/common/gemmPluginProfiler.h @@ -0,0 +1,332 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "pluginUtils.h" + +#include <cuda_runtime.h> + +#include <cstdlib> +#include <iostream> +#include <memory> +#include <mutex> +#include <optional> +#include <shared_mutex> +#include <sstream> +#include <unordered_map> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +struct GemmDims +{ + using DimType64 = utils::DimType64; + + DimType64 minM; + DimType64 maxM; + DimType64 n; + DimType64 k; + + GemmDims() + : minM(-1) + , maxM(-1) + , n(-1) + , k(-1) + { + } + + GemmDims(DimType64 minM_, DimType64 maxM_, DimType64 n_, DimType64 k_) + : minM(minM_) + , maxM(maxM_) + , n(n_) + , k(k_) + { + } + + [[nodiscard]] bool isInitialized() const + { + return minM >= 0 && maxM >= 0 && n >= 0 && k >= 0; + } +}; + +// Unique ID of GEMM +// In our case GEMM is uniqly identified by N and K +class GemmIdCore +{ +public: + int n; + int k; + nvinfer1::DataType dtype; + + GemmIdCore(int n_, int k_, nvinfer1::DataType const& dtype_) + : n(n_) + , k(k_) + , dtype(dtype_) + { + } + + GemmIdCore() + : n(-1) + , k(-1) + , dtype(nvinfer1::DataType::kFLOAT) // dtype does not matter here + { + } + + bool operator==(GemmIdCore const& id) const + { + return isEqual(id); + } + + friend std::ostream& operator<<(std::ostream& out, GemmIdCore const& id) + { + out << "(N;K)=(" << id.n << ";" << id.k << "),"; + out << " type=" << static_cast<int>(id.dtype); + return out; + } + +protected: + bool isEqual(GemmIdCore const& id) const + { + return n == id.n && k == id.k && dtype == id.dtype; + } +}; + +// Hash of GemmId +struct GemmIdCoreHash +{ + std::size_t operator()(GemmIdCore const& id) const + { + auto h1 = std::hash<int>{}(id.n); + auto h2 = std::hash<int>{}(id.k); + auto h3 = std::hash<int>{}(static_cast<int>(id.dtype)); + return h1 ^ h2 ^ h3; + } +}; + +class GemmIdCublas : public GemmIdCore +{ +public: + bool transA{}; + bool transB{}; + nvinfer1::DataType outputDtype; + + GemmIdCublas(int n_, int k_, nvinfer1::DataType const& dtype_, bool transA_, bool transB_, + nvinfer1::DataType const& output_dtype_) + : GemmIdCore(n_, k_, dtype_) + , transA(transA_) + , transB(transB_) + , outputDtype(output_dtype_) + { + } + + GemmIdCublas() {} + + bool operator==(GemmIdCublas const& id) const + { + return isEqual(id) && transA == id.transA && transB == id.transB && outputDtype == id.outputDtype; + } + + friend std::ostream& operator<<(std::ostream& out, GemmIdCublas const& id) + { + out << "(N;K)=(" << id.n << ";" << id.k << "),"; + out << " type=" << static_cast<int>(id.dtype); + out << " transA=" << id.transA; + out << " transB=" << id.transB; + out << " outputDtype=" << static_cast<int>(id.outputDtype); + return out; + } +}; + +// Hash of GemmIdCublas +struct GemmIdCublasHash +{ + std::size_t operator()(GemmIdCublas const& id) const + { + auto h1 = std::hash<int>{}(id.n); + auto h2 = std::hash<int>{}(id.k); + auto h3 = std::hash<int>{}(static_cast<int>(id.dtype)); + auto h4 = std::hash<bool>{}(id.transA); + auto h5 = std::hash<bool>{}(id.transB); + auto h6 = std::hash<bool>{}(static_cast<int>(id.outputDtype)); + return h1 ^ h2 ^ h3 ^ h4 ^ h5 ^ h6; + } +}; + +template <typename Config, typename RunnerPtr, typename GemmIdType, typename GemmIdHashType> +class GemmPluginProfiler +{ +public: + // Map for single GEMM for different Ms (GEMM dimension) to the best config for particular M + using MProfileMap = std::unordered_map<int, std::optional<Config>>; + using MProfileMapPtr = std::shared_ptr<MProfileMap>; + + // requires exclusive ownership to write to *this + using reader_lock = std::unique_lock<std::shared_timed_mutex>; + // requires shared ownership to read from other + using writer_lock = std::shared_lock<std::shared_timed_mutex>; + + // Struct of continuing map if GEMMs to the best profiles for different Ms + struct MNKProfileMap + { + // Mutex guarding map + std::shared_timed_mutex mutex; + // Map from GEMM Id to profile for particular GEMM + std::unordered_map<GemmIdType, MProfileMapPtr, GemmIdHashType> profileMap; + + bool existsMProfileMap(GemmIdType const& id) + { + auto const iter = profileMap.find(id); + return iter != profileMap.end(); + } + + void createMProfileMap(GemmIdType const& id) + { + profileMap[id] = std::make_shared<MProfileMap>(); + } + + MProfileMapPtr getMProfileMap(GemmIdType const& id) + { + auto const iter = profileMap.find(id); + if (iter == profileMap.end()) + { + std::ostringstream msg; + msg << "Cannot find ID (" << id << ") in the profile map. Abort."; + TLLM_THROW(msg.str()); + } + return iter->second; + } + }; + + using MNKProfileMapPtr = std::shared_ptr<MNKProfileMap>; + + GemmPluginProfiler(); + + virtual ~GemmPluginProfiler() = default; + + void serialize(char*& buffer, GemmIdType const& gemmId) const; + + void deserialize(char const*& data, GemmDims& dims, GemmIdType const& gemmId); + size_t getSerializationSize(GemmIdType const& gemmId) const; + + void profileTactics(RunnerPtr const& runner, nvinfer1::DataType const& type, GemmDims const& dims, + GemmIdType const& gemmId, bool hasWeightOnlyCudaKernel = false); + + void setSelectionTactics(MNKProfileMapPtr const& map) + { + mMNKProfileMap = map; + } + + void setTmpWorkspaceSizeInBytes(size_t bytes) + { + mTmpWorkspaceSizeInBytes = bytes; + } + + void setSkip(bool skip) + { + mSkip = mSkip || skip; + } + + std::optional<Config> getBestConfig(int m, GemmIdType const& gemmId) const; + + virtual int getMaxProfileM() const; + +protected: + virtual void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) = 0; + + virtual void computeTmpSize(size_t maxM, size_t n, size_t k) = 0; + + virtual bool checkTactic(int m, int n, int k, Config const& tactic) const + { + return true; + } + + virtual std::vector<Config> getTactics(int m, int n, int k) const = 0; + + virtual void initTmpData(int m, int n, int k, char* workspace, size_t size, cudaStream_t stream); + +private: + void allocateTmpData(); + + void freeTmpData(); + + std::optional<Config> profileTacticsForProblem(int m, int n, int k, std::vector<Config> const& tactics); + + float profileTacticForProblem(int m, int n, int k, Config const& tactic); + + int nextPowerOfTwo(int v) const + { + --v; + v |= v >> 1; + v |= v >> 2; + v |= v >> 4; + v |= v >> 8; + v |= v >> 16; + return ++v; + } + +protected: + RunnerPtr mRunner{nullptr}; + + nvinfer1::DataType mType{}; + +private: + MNKProfileMapPtr mMNKProfileMap{}; + + size_t mTmpWorkspaceSizeInBytes{0}; + + char* mWorkspaceTmp{nullptr}; + + cudaStream_t mStream; + + GemmDims mDims{}; + + bool mSkip{false}; +}; + +template <typename GemmPluginProfilerType> +class GemmPluginProfilerManager +{ +public: + using MNKProfileMap = typename GemmPluginProfilerType::MNKProfileMap; + using MNKProfileMapPtr = typename GemmPluginProfilerType::MNKProfileMapPtr; + using GemmPluginProfilerPtr = std::shared_ptr<GemmPluginProfilerType>; + + GemmPluginProfilerManager() + { + mMNKProfileMap = std::make_shared<MNKProfileMap>(); + } + + GemmPluginProfilerPtr createGemmPluginProfiler(bool inference, bool skip = false) + { + auto profiler = std::make_shared<GemmPluginProfilerType>(); + profiler->setSkip(skip); + // If the profiler is created during the engine build, + // mMNKProfileMap is shared between different profilers to minimize the time spent on the profiling + // and do not repeat profiling for the GEMMs of the same shape. + if (!inference) + { + profiler->setSelectionTactics(mMNKProfileMap); + } + return profiler; + } + +private: + MNKProfileMapPtr mMNKProfileMap{}; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/common/plugin.cpp b/cpp/tensorrt_llm/plugins/common/plugin.cpp new file mode 100644 index 000000000000..82c8bf93b13c --- /dev/null +++ b/cpp/tensorrt_llm/plugins/common/plugin.cpp @@ -0,0 +1,124 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include "checkMacrosPlugin.h" +#include <cstdint> +#include <functional> + +#ifdef _MSC_VER +#define FN_NAME __FUNCTION__ +#else +#define FN_NAME __func__ +#endif + +PluginFieldParser::PluginFieldParser(int32_t nbFields, nvinfer1::PluginField const* fields) + : mFields{fields} +{ + for (int32_t i = 0; i < nbFields; i++) + { + mMap.emplace(fields[i].name, PluginFieldParser::Record{i}); + } +} + +PluginFieldParser::~PluginFieldParser() +{ + for (auto const& [name, record] : mMap) + { + if (!record.retrieved) + { + std::stringstream ss; + ss << "unused plugin field with name: " << name; + tensorrt_llm::plugins::logError(ss.str().c_str(), __FILE__, FN_NAME, __LINE__); + } + } +} + +template <typename T> +nvinfer1::PluginFieldType toFieldType(); +#define SPECIALIZE_TO_FIELD_TYPE(T, type) \ + template <> \ + nvinfer1::PluginFieldType toFieldType<T>() \ + { \ + return nvinfer1::PluginFieldType::type; \ + } +SPECIALIZE_TO_FIELD_TYPE(half, kFLOAT16) +SPECIALIZE_TO_FIELD_TYPE(float, kFLOAT32) +SPECIALIZE_TO_FIELD_TYPE(double, kFLOAT64) +SPECIALIZE_TO_FIELD_TYPE(int8_t, kINT8) +SPECIALIZE_TO_FIELD_TYPE(int16_t, kINT16) +SPECIALIZE_TO_FIELD_TYPE(int32_t, kINT32) +SPECIALIZE_TO_FIELD_TYPE(char, kCHAR) +SPECIALIZE_TO_FIELD_TYPE(nvinfer1::Dims, kDIMS) +SPECIALIZE_TO_FIELD_TYPE(void, kUNKNOWN) +#undef SPECIALIZE_TO_FIELD_TYPE + +template <typename T> +std::optional<T> PluginFieldParser::getScalar(std::string_view const& name) +{ + auto const iter = mMap.find(name); + if (iter == mMap.end()) + { + return std::nullopt; + } + auto& record = mMap.at(name); + auto const& f = mFields[record.index]; + TLLM_CHECK(toFieldType<T>() == f.type && f.length == 1); + record.retrieved = true; + return std::optional{*static_cast<T const*>(f.data)}; +} + +#define INSTANTIATE_PluginFieldParser_getScalar(T) \ + template std::optional<T> PluginFieldParser::getScalar(std::string_view const&) +INSTANTIATE_PluginFieldParser_getScalar(half); +INSTANTIATE_PluginFieldParser_getScalar(float); +INSTANTIATE_PluginFieldParser_getScalar(double); +INSTANTIATE_PluginFieldParser_getScalar(int8_t); +INSTANTIATE_PluginFieldParser_getScalar(int16_t); +INSTANTIATE_PluginFieldParser_getScalar(int32_t); +INSTANTIATE_PluginFieldParser_getScalar(char); +INSTANTIATE_PluginFieldParser_getScalar(nvinfer1::Dims); +#undef INSTANTIATE_PluginFieldParser_getScalar + +template <typename T> +std::optional<std::set<T>> PluginFieldParser::getSet(std::string_view const& name) +{ + auto const iter = mMap.find(name); + if (iter == mMap.end()) + { + return std::nullopt; + } + auto& record = mMap.at(name); + auto const& f = mFields[record.index]; + TLLM_CHECK(toFieldType<T>() == f.type); + std::set<T> group; + auto const* r = static_cast<T const*>(f.data); + for (int j = 0; j < f.length; ++j) + { + group.insert(*r); + ++r; + } + + record.retrieved = true; + return std::optional{group}; +} + +#define INSTANTIATE_PluginFieldParser_getVector(T) \ + template std::optional<std::set<T>> PluginFieldParser::getSet(std::string_view const&) +INSTANTIATE_PluginFieldParser_getVector(int32_t); +#undef INSTANTIATE_PluginFieldParser_getVector diff --git a/cpp/tensorrt_llm/plugins/common/plugin.h b/cpp/tensorrt_llm/plugins/common/plugin.h new file mode 100644 index 000000000000..a7febe4cc13d --- /dev/null +++ b/cpp/tensorrt_llm/plugins/common/plugin.h @@ -0,0 +1,143 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/opUtils.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/plugins/common/checkMacrosPlugin.h" + +#include <NvInferRuntime.h> + +#include <cstring> +#include <map> +#include <memory> +#include <optional> +#include <set> +#include <string> +#include <unordered_map> + +namespace tensorrt_llm::plugins +{ + +using namespace tensorrt_llm::common::op; + +class BasePlugin : public nvinfer1::IPluginV2DynamicExt +{ +public: + void setPluginNamespace(char const* libNamespace) noexcept override + { + mNamespace = libNamespace; + } + + [[nodiscard]] char const* getPluginNamespace() const noexcept override + { + return mNamespace.c_str(); + } + +protected: + std::string mNamespace{api::kDefaultNamespace}; +}; + +class BasePluginV3 : public nvinfer1::IPluginV3, + public nvinfer1::IPluginV3OneCore, + public nvinfer1::IPluginV3OneBuild, + public nvinfer1::IPluginV3OneRuntime +{ +public: + void setPluginNamespace(char const* libNamespace) noexcept + { + mNamespace = libNamespace; + } + + [[nodiscard]] char const* getPluginNamespace() const noexcept override + { + return mNamespace.c_str(); + } + +protected: + std::string mNamespace{api::kDefaultNamespace}; +}; + +class BaseCreator : public nvinfer1::IPluginCreator +{ +public: + void setPluginNamespace(char const* libNamespace) noexcept override + { + mNamespace = libNamespace; + } + + [[nodiscard]] char const* getPluginNamespace() const noexcept override + { + return mNamespace.c_str(); + } + +protected: + std::string mNamespace{api::kDefaultNamespace}; +}; + +class BaseCreatorV3 : public nvinfer1::IPluginCreatorV3One +{ +public: + void setPluginNamespace(char const* libNamespace) noexcept + { + mNamespace = libNamespace; + } + + [[nodiscard]] char const* getPluginNamespace() const noexcept override + { + return mNamespace.c_str(); + } + +protected: + std::string mNamespace{api::kDefaultNamespace}; +}; + +} // namespace tensorrt_llm::plugins + +// Init with O(n) and retrieve with O(1) +class PluginFieldParser +{ +public: + // field array must remain valid when calling getScalar() later. + PluginFieldParser(int32_t nbFields, nvinfer1::PluginField const* fields); + // delete to remind accidental mis-use (copy) which may result in false-alarm warnings about unused fields. + PluginFieldParser(PluginFieldParser const&) = delete; + PluginFieldParser& operator=(PluginFieldParser const&) = delete; + // check if all fields are retrieved and emit warning if some of them are not. + ~PluginFieldParser(); + template <typename T> + std::optional<T> getScalar(std::string_view const& name); + template <typename T> + std::optional<std::set<T>> getSet(std::string_view const& name); + +private: + nvinfer1::PluginField const* mFields; + + struct Record + { + Record(int32_t idx) + : index{idx} + { + } + + int32_t const index; + bool retrieved{false}; + }; + + std::unordered_map<std::string_view, Record> mMap; +}; diff --git a/cpp/tensorrt_llm/plugins/common/pluginUtils.h b/cpp/tensorrt_llm/plugins/common/pluginUtils.h new file mode 100644 index 000000000000..ee3e59d57c6d --- /dev/null +++ b/cpp/tensorrt_llm/plugins/common/pluginUtils.h @@ -0,0 +1,78 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include <NvInferRuntime.h> + +#include "tensorrt_llm/common/logger.h" + +namespace tensorrt_llm::plugins::utils +{ +using DimType64 = int64_t; + +inline DimType64 computeMDimension(bool transA, nvinfer1::Dims const& dims) +{ + DimType64 M{1}; + if (transA) + { + for (int i = dims.nbDims - 1; i > 0; --i) + { + M *= dims.d[i]; + } + } + else + { + for (int i = 0; i < dims.nbDims - 1; ++i) + { + M *= dims.d[i]; + } + } + return M; +} + +inline DimType64 computeNDimension(bool transB, nvinfer1::Dims const& dims) +{ + DimType64 N{1}; + if (transB) + { + for (int32_t i = 0; i < dims.nbDims - 1; ++i) + { + N *= dims.d[i]; + } + } + else + { + for (int32_t i = dims.nbDims - 1; i > 0; --i) + { + N *= dims.d[i]; + } + } + return N; +} + +inline std::int32_t logErrorReturn0(char const* variable) +{ + TLLM_LOG_ERROR("Value of %s is out of range for int32_t", variable); + return 0; +} + +#define TLLM_INT32_CAST(value) \ + ((value > 0x7FFFFFFFLL || value < -0x80000000LL) ? tensorrt_llm::plugins::utils::logErrorReturn0(#value) \ + : static_cast<int32_t>(value)) + +} // namespace tensorrt_llm::plugins::utils diff --git a/cpp/tensorrt_llm/plugins/cpSplitPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/cpSplitPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cpSplitPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/cpSplitPlugin/cpSplitPlugin.cpp b/cpp/tensorrt_llm/plugins/cpSplitPlugin/cpSplitPlugin.cpp new file mode 100644 index 000000000000..221d5dac2da1 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cpSplitPlugin/cpSplitPlugin.cpp @@ -0,0 +1,356 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include <cstdio> + +#include "cpSplitPlugin.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::CpSplitPluginCreator; +using tensorrt_llm::plugins::CpSplitPlugin; + +static char const* CPSPLIT_PLUGIN_VERSION{"1"}; +static char const* CPSPLIT_PLUGIN_NAME{"CpSplit"}; +PluginFieldCollection CpSplitPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> CpSplitPluginCreator::mPluginAttributes; + +CpSplitPlugin::CpSplitPlugin() +{ + initFieldsToSerialize(); +} + +CpSplitPlugin::CpSplitPlugin(int cpSize, int cpRank) + : mCpSize(cpSize) + , mCpRank(cpRank) +{ + initFieldsToSerialize(); +} + +void CpSplitPlugin::initFieldsToSerialize() +{ + mDataToSerialize.clear(); + mDataToSerialize.emplace_back(PluginField("cp_size", &mCpSize, PluginFieldType::kINT32, 1)); + mDataToSerialize.emplace_back(PluginField("cp_rank", &mCpRank, PluginFieldType::kINT32, 1)); + mFCToSerialize.nbFields = mDataToSerialize.size(); + mFCToSerialize.fields = mDataToSerialize.data(); +} + +// IPluginV3 methods +nvinfer1::IPluginCapability* CpSplitPlugin::getCapabilityInterface(nvinfer1::PluginCapabilityType type) noexcept +{ + switch (type) + { + case PluginCapabilityType::kBUILD: return static_cast<IPluginV3OneBuild*>(this); + case PluginCapabilityType::kRUNTIME: return static_cast<IPluginV3OneRuntime*>(this); + case PluginCapabilityType::kCORE: return static_cast<IPluginV3OneCore*>(this); + } + return nullptr; +} + +nvinfer1::IPluginV3* CpSplitPlugin::clone() noexcept +{ + std::unique_ptr<CpSplitPlugin> plugin{std::make_unique<CpSplitPlugin>(*this)}; + plugin->setPluginNamespace(mNamespace.c_str()); + plugin->initFieldsToSerialize(); + return plugin.release(); +} + +// IPluginV3OneCore methods +char const* CpSplitPlugin::getPluginName() const noexcept +{ + return CPSPLIT_PLUGIN_NAME; +} + +char const* CpSplitPlugin::getPluginVersion() const noexcept +{ + return CPSPLIT_PLUGIN_VERSION; +} + +// IPluginV3OneBuild methods +int32_t CpSplitPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept +{ + return 0; +} + +int32_t CpSplitPlugin::getOutputDataTypes( + DataType* outputTypes, int32_t nbOutputs, DataType const* inputTypes, int32_t nbInputs) const noexcept +{ + outputTypes[0] = inputTypes[0]; + outputTypes[1] = DataType::kINT32; + outputTypes[2] = DataType::kINT32; + return 0; +} + +int32_t CpSplitPlugin::getOutputShapes(DimsExprs const* inputs, int32_t nbInputs, DimsExprs const* shapeInputs, + int32_t nbShapeInputs, DimsExprs* outputs, int32_t nbOutputs, IExprBuilder& exprBuilder) noexcept +{ + outputs[0].nbDims = 1; + + auto cpSize = exprBuilder.constant(mCpSize); + auto upper = inputs[0].d[0]; + auto opt = exprBuilder.operation(DimensionOperation::kCEIL_DIV, *upper, *cpSize); + outputs[0].d[0] = exprBuilder.declareSizeTensor(1, *opt, *upper); + + // We must have such an output size tensor (with dim == 0) to notify the shape of output tensor above + outputs[1].nbDims = 0; + outputs[2].nbDims = 1; + outputs[2].d[0] = upper; + return 0; +} + +bool CpSplitPlugin::supportsFormatCombination( + int32_t pos, nvinfer1::DynamicPluginTensorDesc const* inOut, int32_t nbInputs, int32_t nbOutputs) noexcept +{ + if (pos == IdxEntry::INPUT_IDS) + { + return ((inOut[pos].desc.type == DataType::kINT32) && (inOut[pos].desc.format == TensorFormat::kLINEAR)); + } + else if (pos == IdxEntry::REQUEST_TYPES || pos == IdxEntry::HOST_CONTEXT_LENGTH) + { + return inOut[pos].desc.type == DataType::kINT32; + } + else + { + return ((inOut[pos].desc.type == DataType::kINT32) && (inOut[pos].desc.format == TensorFormat::kLINEAR)); + } + return false; +} + +int32_t CpSplitPlugin::getNbOutputs() const noexcept +{ + return 3; +} + +size_t CpSplitPlugin::getWorkspaceSize(nvinfer1::DynamicPluginTensorDesc const* inputs, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* outputs, int32_t nbOutputs) const noexcept +{ + return 0; +} + +int32_t CpSplitPlugin::getValidTactics(int32_t* tactics, int32_t nbTactics) noexcept +{ + return 0; +} + +int32_t CpSplitPlugin::getNbTactics() noexcept +{ + return 0; +} + +char const* CpSplitPlugin::getTimingCacheID() noexcept +{ + return nullptr; +} + +int32_t CpSplitPlugin::getFormatCombinationLimit() noexcept +{ + return 1; +} + +char const* CpSplitPlugin::getMetadataString() noexcept +{ + return nullptr; +} + +// IPluginV3OneRuntime methods +int32_t CpSplitPlugin::setTactic(int32_t tactic) noexcept +{ + return 0; +} + +int32_t CpSplitPlugin::onShapeChange(nvinfer1::PluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::PluginTensorDesc const* out, int32_t nbOutputs) noexcept +{ + return 0; +} + +int32_t CpSplitPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // @param inputIds [tokenNum] + // @param host_request_types [batchSize]: Tensor = None (On CPU) + // The tensor on the host that indicates if a request is in context or + // generation phase. Its shape is [batch_size]. See Inflight Batching + // in docs/gpt_attention.md, + // @param host_context_lengths [batchSize]: Tensor = None (On CPU) + // A host tensor that contains the lengths of the different inputs + // outputs + // @param outputIds [tokenNum spiltted by cp] + // @param outputLength scalar + // @param joinIdx [tokenNum] + + int64_t tokenNum = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims; ++i) + { + tokenNum *= inputDesc[0].dims.d[i]; + } + + RequestType const* reqTypes = static_cast<RequestType const*>(inputs[IdxEntry::REQUEST_TYPES]); + int32_t const* hContextLengths = static_cast<int32_t const*>(inputs[IdxEntry::HOST_CONTEXT_LENGTH]); + int const* inputIds = reinterpret_cast<int const*>(inputs[IdxEntry::INPUT_IDS]); + int* outputIds = reinterpret_cast<int*>(outputs[0]); + int32_t* outputLength = reinterpret_cast<int32_t*>(outputs[1]); + int32_t* outputJoinIdx = reinterpret_cast<int32_t*>(outputs[2]); + + int32_t const nbSeq = inputDesc[IdxEntry::HOST_CONTEXT_LENGTH].dims.d[0]; + + int32_t* hInputs = new int[inputDesc[IdxEntry::INPUT_IDS].dims.d[0]]; + int32_t* hOutputs = new int[inputDesc[IdxEntry::INPUT_IDS].dims.d[0]]; + int32_t* hOutputJoinIdx = new int[inputDesc[IdxEntry::INPUT_IDS].dims.d[0]]; + cudaMemcpyAsync( + hInputs, inputIds, sizeof(int32_t) * inputDesc[IdxEntry::INPUT_IDS].dims.d[0], cudaMemcpyDeviceToHost, stream); + sync_check_cuda_error(stream); + + int32_t inputIdx = 0; + int32_t outputIdx = 0; + for (int32_t seqIdx = 0; seqIdx < nbSeq; ++seqIdx) + { + if (reqTypes[seqIdx] == RequestType::kCONTEXT) + { + auto const& ctxLength = hContextLengths[seqIdx]; + int32_t partialAverageLength = (ctxLength + mCpSize - 1) / mCpSize; + int32_t partialLength + = mCpRank == mCpSize - 1 ? ctxLength - partialAverageLength * (mCpSize - 1) : partialAverageLength; + for (int i = 0; i < partialLength; i++) + { + hOutputs[outputIdx + i] = hInputs[inputIdx + partialAverageLength * mCpRank + i]; + } + inputIdx += ctxLength; + outputIdx += partialAverageLength; + } + else if (reqTypes[seqIdx] == RequestType::kGENERATION) + { + auto const& genLength = nbSeq - seqIdx; + int32_t partialAverageLength = (genLength + mCpSize - 1) / mCpSize; + int32_t partialLength + = mCpRank == mCpSize - 1 ? genLength - partialAverageLength * (mCpSize - 1) : partialAverageLength; + for (int i = 0; i < partialLength; i++) + { + hOutputs[outputIdx + i] = hInputs[inputIdx + partialAverageLength * mCpRank + i]; + } + outputIdx += partialAverageLength; + break; + } + } + int32_t hOutputLength = outputIdx; + inputIdx = 0; + outputIdx = 0; + for (int32_t seqIdx = 0; seqIdx < nbSeq; ++seqIdx) + { + if (reqTypes[seqIdx] == RequestType::kCONTEXT) + { + auto const& ctxLength = hContextLengths[seqIdx]; + int32_t partialAverageLength = (ctxLength + mCpSize - 1) / mCpSize; + for (int32_t idx = 0; idx < ctxLength; ++idx) + { + hOutputJoinIdx[inputIdx + idx] + = idx % partialAverageLength + idx / partialAverageLength * hOutputLength + outputIdx; + } + inputIdx += ctxLength; + outputIdx += partialAverageLength; + } + else if (reqTypes[seqIdx] == RequestType::kGENERATION) + { + auto const& genLength = nbSeq - seqIdx; + int32_t partialAverageLength = (genLength + mCpSize - 1) / mCpSize; + for (int32_t idx = 0; idx < genLength; ++idx) + { + hOutputJoinIdx[inputIdx + idx] + = idx % partialAverageLength + idx / partialAverageLength * hOutputLength + outputIdx; + } + break; + } + } + cudaMemcpyAsync(outputIds, hOutputs, sizeof(int32_t) * hOutputLength, cudaMemcpyHostToDevice, stream); + cudaMemcpyAsync(outputLength, &hOutputLength, sizeof(int32_t), cudaMemcpyHostToDevice, stream); + cudaMemcpyAsync(outputJoinIdx, hOutputJoinIdx, sizeof(int32_t) * tokenNum, cudaMemcpyHostToDevice, stream); + sync_check_cuda_error(stream); + return 0; +} + +nvinfer1::IPluginV3* CpSplitPlugin::attachToContext(nvinfer1::IPluginResourceContext* context) noexcept +{ + return clone(); +} + +nvinfer1::PluginFieldCollection const* CpSplitPlugin::getFieldsToSerialize() noexcept +{ + return &mFCToSerialize; +} + +CpSplitPluginCreator::CpSplitPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("cp_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("cp_rank", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* CpSplitPluginCreator::getPluginName() const noexcept +{ + return CPSPLIT_PLUGIN_NAME; +} + +char const* CpSplitPluginCreator::getPluginVersion() const noexcept +{ + return CPSPLIT_PLUGIN_VERSION; +} + +nvinfer1::PluginFieldCollection const* CpSplitPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +nvinfer1::IPluginV3* CpSplitPluginCreator::createPlugin( + char const* name, nvinfer1::PluginFieldCollection const* fc, nvinfer1::TensorRTPhase phase) noexcept +{ + PluginField const* fields = fc->fields; + int cp_size{}; + int cp_rank{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "cp_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + cp_size = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "cp_rank")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + cp_rank = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + } + try + { + auto* obj = new CpSplitPlugin(cp_size, cp_rank); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/cpSplitPlugin/cpSplitPlugin.h b/cpp/tensorrt_llm/plugins/cpSplitPlugin/cpSplitPlugin.h new file mode 100644 index 000000000000..1dc8c15b355a --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cpSplitPlugin/cpSplitPlugin.h @@ -0,0 +1,112 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class CpSplitPlugin : public BasePluginV3 +{ +public: + CpSplitPlugin(); + CpSplitPlugin(int cpSize, int cpRank); + CpSplitPlugin(CpSplitPlugin const& p) = default; + void initFieldsToSerialize(); + + // IPluginV3 methods + nvinfer1::IPluginCapability* getCapabilityInterface(nvinfer1::PluginCapabilityType type) noexcept override; + nvinfer1::IPluginV3* clone() noexcept override; + + // IPluginV3OneCore methods + char const* getPluginName() const noexcept override; + char const* getPluginVersion() const noexcept override; + + // IPluginV3OneBuild methods + int32_t configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept override; // nochange + int32_t getOutputDataTypes(nvinfer1::DataType* outputTypes, int32_t nbOutputs, nvinfer1::DataType const* inputTypes, + int32_t nbInputs) const noexcept override; // fixed + int32_t getOutputShapes(nvinfer1::DimsExprs const* inputs, int32_t nbInputs, nvinfer1::DimsExprs const* shapeInputs, + int32_t nbShapeInputs, nvinfer1::DimsExprs* outputs, int32_t nbOutputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; // fixed + bool supportsFormatCombination(int32_t pos, nvinfer1::DynamicPluginTensorDesc const* inOut, int32_t nbInputs, + int32_t nbOutputs) noexcept override; // fixed + int32_t getNbOutputs() const noexcept override; // fixed + size_t getWorkspaceSize(nvinfer1::DynamicPluginTensorDesc const* inputs, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* outputs, int32_t nbOutputs) const noexcept override; // fixed + int32_t getValidTactics(int32_t* tactics, int32_t nbTactics) noexcept override; + int32_t getNbTactics() noexcept override; + char const* getTimingCacheID() noexcept override; + int32_t getFormatCombinationLimit() noexcept override; + char const* getMetadataString() noexcept override; + + // IPluginV3OneRuntime methods + int32_t setTactic(int32_t tactic) noexcept override; + int32_t onShapeChange(nvinfer1::PluginTensorDesc const* in, int32_t nbInputs, nvinfer1::PluginTensorDesc const* out, + int32_t nbOutputs) noexcept override; + int32_t enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept override; // fixed + nvinfer1::IPluginV3* attachToContext(nvinfer1::IPluginResourceContext* context) noexcept override; + nvinfer1::PluginFieldCollection const* getFieldsToSerialize() noexcept override; + +private: + int mCpSize; + int mCpRank; + std::vector<nvinfer1::PluginField> mDataToSerialize; + nvinfer1::PluginFieldCollection mFCToSerialize; + + enum IdxEntry + { + INPUT_IDS, + REQUEST_TYPES, + HOST_CONTEXT_LENGTH, + }; + + enum class RequestType : int32_t + { + kCONTEXT = 0, + kGENERATION = 1 + }; +}; + +class CpSplitPluginCreator : public BaseCreatorV3 +{ +public: + CpSplitPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV3* createPlugin( + char const* name, nvinfer1::PluginFieldCollection const* fc, nvinfer1::TensorRTPhase phase) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/cudaStreamPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/cudaStreamPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cudaStreamPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/cudaStreamPlugin/cudaStreamPlugin.cpp b/cpp/tensorrt_llm/plugins/cudaStreamPlugin/cudaStreamPlugin.cpp new file mode 100644 index 000000000000..802e828c9250 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cudaStreamPlugin/cudaStreamPlugin.cpp @@ -0,0 +1,293 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "cudaStreamPlugin.h" +#include "tensorrt_llm/runtime/iBuffer.h" + +#include <cassert> + +using namespace nvinfer1; +using tensorrt_llm::plugins::CudaStreamPluginCreator; +using tensorrt_llm::plugins::CudaStreamPlugin; + +static char const* CUDA_STREAM_PLUGIN_VERSION{"1"}; +static char const* CUDA_STREAM_PLUGIN_NAME{"CudaStream"}; +PluginFieldCollection CudaStreamPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> CudaStreamPluginCreator::mPluginAttributes; + +CudaStreamPlugin::CudaStreamPlugin(int sideStreamId, int nbInputs, nvinfer1::DataType type) + : mSideStreamId(sideStreamId) + , mNbInputs(nbInputs) + , mType(type) +{ + init(); +} + +CudaStreamPlugin::CudaStreamPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mSideStreamId); + read(d, mNbInputs); + read(d, mType); + + init(); + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +CudaStreamPlugin::CudaStreamPlugin(CudaStreamPlugin const& other) + : mSideStreamId(other.mSideStreamId) + , mNbInputs(other.mNbInputs) + , mType(other.mType) +{ + init(); +} + +void CudaStreamPlugin::init() +{ + mSideStreamPtr = nullptr; +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* CudaStreamPlugin::clone() const noexcept +{ + auto* plugin = new CudaStreamPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs CudaStreamPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + assert(outputIndex == 0); + return inputs[outputIndex]; +} + +bool CudaStreamPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + TLLM_CHECK_WITH_INFO(nbInputs == mNbInputs, "CudaStreamPlugin only accepts mNbInputs inputs"); + TLLM_CHECK_WITH_INFO(nbOutputs == 1, "CudaStreamPlugin only accepts 1 output"); + + auto const& desc = inOut[pos]; + if (desc.format != TensorFormat::kLINEAR) + { + return false; + } + + if (pos > 0 && pos < nbInputs) + { + return true; + } + return desc.type == mType; +} + +void CudaStreamPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t CudaStreamPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int CudaStreamPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + if (!mSideStreamPtr) + { + auto const resource_name = nvinfer1::pluginInternal::SideStream::getResourceKey(mSideStreamId); + nvinfer1::pluginInternal::SideStream side_stream{}; + mSideStreamPtr = reinterpret_cast<nvinfer1::pluginInternal::SideStream*>( + getPluginRegistry()->acquirePluginResource(resource_name.c_str(), &side_stream)); + } + mSideStreamPtr->waitSideStreamOnMainStream(stream); + size_t count = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims; ++i) + { + count *= inputDesc[0].dims.d[i]; + } + count *= tensorrt_llm::runtime::BufferDataType(inputDesc[0].type).getSize(); + TLLM_CUDA_CHECK(cudaMemcpyAsync(outputs[0], inputs[0], count, cudaMemcpyDeviceToDevice, stream)); + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType CudaStreamPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* CudaStreamPlugin::getPluginType() const noexcept +{ + return CUDA_STREAM_PLUGIN_NAME; +} + +char const* CudaStreamPlugin::getPluginVersion() const noexcept +{ + return CUDA_STREAM_PLUGIN_VERSION; +} + +int CudaStreamPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int CudaStreamPlugin::initialize() noexcept +{ + return 0; +} + +void CudaStreamPlugin::terminate() noexcept +{ + if (mSideStreamPtr) + { + auto const resource_name = nvinfer1::pluginInternal::SideStream::getResourceKey(mSideStreamId); + getPluginRegistry()->releasePluginResource(resource_name.c_str()); + mSideStreamPtr = nullptr; + } +} + +size_t CudaStreamPlugin::getSerializationSize() const noexcept +{ + return sizeof(mSideStreamId) + sizeof(mNbInputs) + sizeof(mType); +} + +void CudaStreamPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mSideStreamId); + write(d, mNbInputs); + write(d, mType); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void CudaStreamPlugin::destroy() noexcept +{ + delete this; +} + +/////////////// + +CudaStreamPluginCreator::CudaStreamPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("side_stream_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("num_inputs", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* CudaStreamPluginCreator::getPluginName() const noexcept +{ + return CUDA_STREAM_PLUGIN_NAME; +} + +char const* CudaStreamPluginCreator::getPluginVersion() const noexcept +{ + return CUDA_STREAM_PLUGIN_VERSION; +} + +PluginFieldCollection const* CudaStreamPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* CudaStreamPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int sideStreamId; + int nbInputs; + int type; + + // Read configurations from each fields + struct MapPair + { + char const* key; + int& field; + bool optional = false; + bool set = false; + }; + + std::array input_map{ + MapPair{"side_stream_id", std::ref(sideStreamId)}, + MapPair{"num_inputs", std::ref(nbInputs)}, + MapPair{"type_id", std::ref(type)}, + }; + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + for (auto& item : input_map) + { + if (!strcmp(item.key, attrName)) + { + TLLM_CHECK(fields[i].type == nvinfer1::PluginFieldType::kINT32); + TLLM_CHECK_WITH_INFO(!item.set, "Parameter %s was set twice", item.key); + item.field = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + item.set = true; + } + } + } + + for (auto& item : input_map) + { + TLLM_CHECK_WITH_INFO(item.set || item.optional, "Parameter %s is required but not set", item.key); + } + + try + { + auto* obj = new CudaStreamPlugin(sideStreamId, nbInputs, static_cast<nvinfer1::DataType>(type)); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* CudaStreamPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call CudaStreamPlugin::destroy() + try + { + auto* obj = new CudaStreamPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/cudaStreamPlugin/cudaStreamPlugin.h b/cpp/tensorrt_llm/plugins/cudaStreamPlugin/cudaStreamPlugin.h new file mode 100644 index 000000000000..5b78c3b873bb --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cudaStreamPlugin/cudaStreamPlugin.h @@ -0,0 +1,265 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "NvInferPlugin.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/runtime/cudaMemPool.h" +#include "tensorrt_llm/runtime/utils/debugUtils.h" +#include <memory> +#include <string> +#include <vector> + +namespace nvinfer1 +{ +namespace pluginInternal +{ +class SideWorkspace +{ +public: + SideWorkspace(cudaStream_t stream) + : mWorkspaceSize{0} + , mWorkspacePtr{nullptr} + , mStream{stream} + { + } + + ~SideWorkspace() + { + if (mWorkspacePtr) + { + TLLM_CUDA_CHECK(cudaFreeAsync(mWorkspacePtr, mStream)); + } + } + + void* get(size_t workspaceSize) + { + if (mWorkspacePtr && mWorkspaceSize < workspaceSize) + { + TLLM_CUDA_CHECK(cudaFreeAsync(mWorkspacePtr, mStream)); + mWorkspacePtr = nullptr; + } + if (!mWorkspacePtr) + { + mWorkspaceSize = workspaceSize; + auto pool_ptr + = tensorrt_llm::runtime::CudaMemPool::getPrimaryPoolForDevice(tensorrt_llm::common::getDevice()); + TLLM_CUDA_CHECK(cudaMallocFromPoolAsync(&mWorkspacePtr, mWorkspaceSize, pool_ptr->getPool(), mStream)); + } + return mWorkspacePtr; + } + +private: + size_t mWorkspaceSize; + void* mWorkspacePtr; + cudaStream_t mStream; +}; + +class SideStream : public IPluginResource +{ +public: + SideStream(bool init = false) + : mStream{} + , mMainEvent{} + , mSideEvent{} + , mWorkspace{} + , mInit{init} + { + // The object passed to acquirePluginResource should use the default value init=false + if (init) + { + TLLM_CUDA_CHECK(cudaStreamCreate(&mStream)); + TLLM_CUDA_CHECK(cudaEventCreateWithFlags(&mMainEvent, cudaEventDisableTiming)); + TLLM_CUDA_CHECK(cudaEventCreateWithFlags(&mSideEvent, cudaEventDisableTiming)); + mWorkspace = std::make_shared<SideWorkspace>(mStream); + } + } + + void free() + { + if (mInit) + { + mWorkspace = nullptr; + TLLM_CUDA_CHECK(cudaStreamSynchronize(mStream)); + TLLM_CUDA_CHECK(cudaStreamDestroy(mStream)); + TLLM_CUDA_CHECK(cudaEventDestroy(mMainEvent)); + TLLM_CUDA_CHECK(cudaEventDestroy(mSideEvent)); + mInit = false; + } + } + + int32_t release() noexcept override + { + try + { + free(); + } + catch (std::exception const& e) + { + return -1; + } + return 0; + } + + IPluginResource* clone() noexcept override + { + // An object is cloned only when calling acquirePluginResource for the first time for each key + std::unique_ptr<SideStream> cloned{}; + try + { + if (!mInit) + { + cloned = std::make_unique<SideStream>(/* init */ true); + } + else + { + return nullptr; + } + } + catch (std::exception const& e) + { + return nullptr; + } + return cloned.release(); + } + + ~SideStream() override + { + free(); + } + + void* getWorkspacePtr(size_t workspaceSize) + { + return mWorkspace->get(workspaceSize); + } + + cudaStream_t getStream() const + { + return mStream; + } + + void waitMainStreamOnSideStream(cudaStream_t const stream) const + { + TLLM_CUDA_CHECK(cudaEventRecord(mMainEvent, stream)); + TLLM_CUDA_CHECK(cudaStreamWaitEvent(mStream, mMainEvent)); + } + + void waitSideStreamOnMainStream(cudaStream_t const stream) const + { + TLLM_CUDA_CHECK(cudaEventRecord(mSideEvent, mStream)); + TLLM_CUDA_CHECK(cudaStreamWaitEvent(stream, mSideEvent)); + } + + void stallMainStream(char const* name, cudaStream_t const stream, std::optional<int> delay = std::nullopt) const + { + tensorrt_llm::runtime::utils::stallStream(name, stream, delay); + } + + void stallSideStream(char const* name, std::optional<int> delay = std::nullopt) const + { + tensorrt_llm::runtime::utils::stallStream(name, mStream, delay); + } + + static std::string getResourceKey(int const stream_id) + { + return "side_stream_" + std::to_string(stream_id); + } + +private: + cudaStream_t mStream; + cudaEvent_t mMainEvent; + cudaEvent_t mSideEvent; + std::shared_ptr<SideWorkspace> mWorkspace; + bool mInit; +}; + +} // namespace pluginInternal +} // namespace nvinfer1 + +namespace tensorrt_llm::plugins +{ + +class CudaStreamPlugin : public BasePlugin +{ +public: + CudaStreamPlugin(int sideStreamId, int nbInputs, nvinfer1::DataType type); + + CudaStreamPlugin(void const* data, size_t length); + + CudaStreamPlugin(CudaStreamPlugin const&); + + void init(); + + ~CudaStreamPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + const std::string mLayerName; + int mSideStreamId; + int mNbInputs; + nvinfer1::DataType mType; + nvinfer1::pluginInternal::SideStream* mSideStreamPtr; +}; + +class CudaStreamPluginCreator : public BaseCreator +{ +public: + CudaStreamPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/CMakeLists.txt new file mode 100644 index 000000000000..ea25de075f34 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/CMakeLists.txt @@ -0,0 +1,22 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# + +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/cumsumLastDimPlugin.cpp b/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/cumsumLastDimPlugin.cpp new file mode 100644 index 000000000000..927a42ebac2f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/cumsumLastDimPlugin.cpp @@ -0,0 +1,299 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "cumsumLastDimPlugin.h" +#include "tensorrt_llm/common/assert.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::CumsumLastDimPluginCreator; +using tensorrt_llm::plugins::CumsumLastDimPlugin; + +static char const* CUMSUM_LAST_DIM_PLUGIN_VERSION{"1"}; +static char const* CUMSUM_LAST_DIM_PLUGIN_NAME{"CumsumLastDim"}; +PluginFieldCollection CumsumLastDimPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> CumsumLastDimPluginCreator::mPluginAttributes; + +static constexpr SizeType32 LENGTH_LIMIT_FOR_BLOCKSCAN = 4096; + +CumsumLastDimPlugin::CumsumLastDimPlugin(SizeType32 inputLength, nvinfer1::DataType type, size_t temp_storage_bytes) + : mInputLength(inputLength) + , mTempStorageBytes(temp_storage_bytes) + , mType(type) +{ + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF) + || (mType == DataType::kINT32), + "Only support int, float, half, and bfloat16."); + if (mTempStorageBytes == 0) + { + mTempStorageBytes = getWorkspaceSizeNeeded(inputLength, type); + } +} + +// Parameterized constructor +CumsumLastDimPlugin::CumsumLastDimPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mInputLength); + read(d, mTempStorageBytes); + read(d, mType); + TLLM_CHECK(d == a + length); + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF) + || (mType == DataType::kINT32), + "Only support int, float, half, and bfloat16."); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* CumsumLastDimPlugin::clone() const noexcept +{ + auto* plugin = new CumsumLastDimPlugin(mInputLength, mType, mTempStorageBytes); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +// Outputs +// output_tensor: [batch_size, inputLength] +nvinfer1::DimsExprs CumsumLastDimPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + TLLM_CHECK_WITH_INFO(outputIndex == 0, "Only one output."); + return inputs[getInputTensorIdx()]; +} + +bool CumsumLastDimPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); +} + +void CumsumLastDimPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t CumsumLastDimPlugin::getWorkspaceSizeNeeded(SizeType32 inputLength, nvinfer1::DataType type) +{ + size_t tempStorageBytes{0}; + if (inputLength < LENGTH_LIMIT_FOR_BLOCKSCAN) // last dim unknown or small, use BlockScan + { + tempStorageBytes = 0; + } + else if (type == DataType::kINT32) + { + tempStorageBytes = invokeComputeCumsumLastDimWorkspaceSize<int>(inputLength); + } + else if (type == DataType::kHALF) + { + tempStorageBytes = invokeComputeCumsumLastDimWorkspaceSize<half>(inputLength); + } + else if (type == DataType::kFLOAT) + { + tempStorageBytes = invokeComputeCumsumLastDimWorkspaceSize<float>(inputLength); + } +#ifdef ENABLE_BF16 + else if (type == DataType::kBF16) + { + tempStorageBytes = invokeComputeCumsumLastDimWorkspaceSize<__nv_bfloat16>(inputLength); + } +#endif + return tempStorageBytes; +} + +size_t CumsumLastDimPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return mTempStorageBytes; +} + +template <typename T> +int CumsumLastDimPlugin::enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) +{ + // inputs + // 0. input_tensor [batch_size, inputLength] + // outputs + // 0. output_tensor [batch_size, inputLength] + auto const batchSize = inputDesc[getInputTensorIdx()].dims.d[0]; + auto const inputLength = inputDesc[getInputTensorIdx()].dims.d[1]; + /* + Two cases where we should use BlockScan: + 1. inputLength is small + 2. batchSize is large (since DeviceScan causes kernel launch per row) + */ + void* wp = inputLength < LENGTH_LIMIT_FOR_BLOCKSCAN || batchSize > 2 ? nullptr : workspace; + invokeCumsumLastDim<T>( + batchSize, inputLength, inputs[getInputTensorIdx()], outputs[0], wp, mTempStorageBytes, stream); + + sync_check_cuda_error(stream); + return 0; +} + +int CumsumLastDimPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + if (mType == DataType::kINT32) + { + return enqueueImpl<int>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kHALF) + { + return enqueueImpl<half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kFLOAT) + { + return enqueueImpl<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16) + { + return enqueueImpl<__nv_bfloat16>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType CumsumLastDimPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK_WITH_INFO(index == 0, "Only one output."); + return inputTypes[getInputTensorIdx()]; +} + +// IPluginV2 Methods + +char const* CumsumLastDimPlugin::getPluginType() const noexcept +{ + return CUMSUM_LAST_DIM_PLUGIN_NAME; +} + +char const* CumsumLastDimPlugin::getPluginVersion() const noexcept +{ + return CUMSUM_LAST_DIM_PLUGIN_VERSION; +} + +int CumsumLastDimPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int CumsumLastDimPlugin::initialize() noexcept +{ + return 0; +} + +void CumsumLastDimPlugin::terminate() noexcept {} + +size_t CumsumLastDimPlugin::getSerializationSize() const noexcept +{ + return sizeof(mInputLength) + sizeof(mTempStorageBytes) + sizeof(mType); +} + +void CumsumLastDimPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mInputLength); + write(d, mTempStorageBytes); + write(d, mType); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void CumsumLastDimPlugin::destroy() noexcept +{ + delete this; +} + +/////////////// + +CumsumLastDimPluginCreator::CumsumLastDimPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("input_length", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* CumsumLastDimPluginCreator::getPluginName() const noexcept +{ + return CUMSUM_LAST_DIM_PLUGIN_NAME; +} + +char const* CumsumLastDimPluginCreator::getPluginVersion() const noexcept +{ + return CUMSUM_LAST_DIM_PLUGIN_VERSION; +} + +PluginFieldCollection const* CumsumLastDimPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* CumsumLastDimPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int inputLength{}; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "input_length")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + inputLength = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + try + { + auto* obj = new CumsumLastDimPlugin(inputLength, type); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* CumsumLastDimPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call CumsumLastDimPlugin::destroy() + try + { + auto* obj = new CumsumLastDimPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/cumsumLastDimPlugin.h b/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/cumsumLastDimPlugin.h new file mode 100644 index 000000000000..3cbf4e2356dd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/cumsumLastDimPlugin/cumsumLastDimPlugin.h @@ -0,0 +1,102 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TRT_CUMSUM_LAST_DIM_PLUGIN_H +#define TRT_CUMSUM_LAST_DIM_PLUGIN_H + +#include "tensorrt_llm/kernels/cumsumLastDim.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> + +namespace tensorrt_llm::plugins +{ +class CumsumLastDimPlugin : public BasePlugin +{ +public: + using SizeType32 = tensorrt_llm::kernels::SizeType32; + + CumsumLastDimPlugin(SizeType32 inputLength, nvinfer1::DataType type, size_t tempStorageBytes = 0); + CumsumLastDimPlugin(void const* data, size_t length); + ~CumsumLastDimPlugin() override = default; + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + template <typename T> + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + size_t getWorkspaceSizeNeeded(SizeType32 inputLength, nvinfer1::DataType type); + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + using IndexType = std::int32_t; + + IndexType getInputTensorIdx() const + { + return 0; + }; + +private: + SizeType32 mInputLength; + size_t mTempStorageBytes; + nvinfer1::DataType mType; +}; + +class CumsumLastDimPluginCreator : public BaseCreator +{ +public: + CumsumLastDimPluginCreator(); + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins + +#endif diff --git a/cpp/tensorrt_llm/plugins/doraPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/doraPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/doraPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/doraPlugin/doraPlugin.cpp b/cpp/tensorrt_llm/plugins/doraPlugin/doraPlugin.cpp new file mode 100644 index 000000000000..7c980f079ceb --- /dev/null +++ b/cpp/tensorrt_llm/plugins/doraPlugin/doraPlugin.cpp @@ -0,0 +1,392 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "doraPlugin.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/runtime/iBuffer.h" + +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::DoraPlugin; +using tensorrt_llm::plugins::DoraPluginCreator; + +static char const* DORA_PLUGIN_VERSION{"1"}; +static char const* DORA_PLUGIN_NAME{"Dora"}; +PluginFieldCollection DoraPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> DoraPluginCreator::mPluginAttributes; + +DoraPlugin::DoraPlugin(std::vector<int32_t> const& outHiddenSizes, nvinfer1::DataType type, bool removeInputPadding) + : mType(type) + , mRemoveInputPadding(removeInputPadding) + , mDoraImpl(outHiddenSizes, type) +{ + mOutHiddenSizes.resize(outHiddenSizes.size()); + mOutHiddenSizes.assign(outHiddenSizes.cbegin(), outHiddenSizes.cend()); + init(); +} + +void DoraPlugin::init() +{ + // initialize data to serialize + mDataToSerialize.clear(); + mDataToSerialize.emplace_back( + "out_hidden_sizes", mOutHiddenSizes.data(), PluginFieldType::kINT32, mOutHiddenSizes.size()); + mDataToSerialize.emplace_back("type", &mType, PluginFieldType::kINT32, 1); + mDataToSerialize.emplace_back("remove_input_padding", &mRemoveInputPadding, PluginFieldType::kINT8, 1); + mFieldsToSerialize.nbFields = static_cast<int32_t>(mDataToSerialize.size()); + mFieldsToSerialize.fields = mDataToSerialize.data(); +} + +// IPluginV3 methods +nvinfer1::IPluginCapability* DoraPlugin::getCapabilityInterface(nvinfer1::PluginCapabilityType type) noexcept +{ + switch (type) + { + case PluginCapabilityType::kBUILD: return static_cast<IPluginV3OneBuild*>(this); + case PluginCapabilityType::kRUNTIME: return static_cast<IPluginV3OneRuntime*>(this); + case PluginCapabilityType::kCORE: return static_cast<IPluginV3OneCore*>(this); + } + return nullptr; +} + +nvinfer1::IPluginV3* DoraPlugin::clone() noexcept +{ + std::unique_ptr<DoraPlugin> plugin{std::make_unique<DoraPlugin>(mOutHiddenSizes, mType, mRemoveInputPadding)}; + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin.release(); +} + +// IPluginV3OneCore methods +char const* DoraPlugin::getPluginName() const noexcept +{ + return DORA_PLUGIN_NAME; +} + +char const* DoraPlugin::getPluginVersion() const noexcept +{ + return DORA_PLUGIN_VERSION; +} + +// IPluginV3OneBuild methods +int32_t DoraPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept +{ + return 0; +} + +int32_t DoraPlugin::getOutputDataTypes( + DataType* outputTypes, int32_t nbOutputs, DataType const* inputTypes, int32_t nbInputs) const noexcept +{ + try + { + TLLM_CHECK(nbOutputs == 1); + TLLM_CHECK(nbInputs == 2 + static_cast<int32_t>(mOutHiddenSizes.size()) + (mRemoveInputPadding ? 1 : 0)); + TLLM_CHECK(inputTypes[IdxEntry::kINPUT_TENSOR] == mType); + // output has the same dtype as the input, the plugin just applies scaling + outputTypes[0] = inputTypes[IdxEntry::kINPUT_TENSOR]; + } + catch (std::exception const& e) + { + caughtError(e); + } + return 0; +} + +int32_t DoraPlugin::getOutputShapes(DimsExprs const* inputs, int32_t nbInputs, DimsExprs const* shapeInputs, + int32_t nbShapeInputs, DimsExprs* outputs, int32_t nbOutputs, IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbOutputs == 1); + TLLM_CHECK(nbShapeInputs == 0); + TLLM_CHECK(nbInputs == 2 + static_cast<int32_t>(mOutHiddenSizes.size()) + (mRemoveInputPadding ? 1 : 0)); + + auto const inputTensorDims = inputs[IdxEntry::kINPUT_TENSOR]; + TLLM_CHECK(inputTensorDims.nbDims == (mRemoveInputPadding ? 2 : 3)); + + auto const lastDim = inputTensorDims.d[inputTensorDims.nbDims - 1]; + TLLM_CHECK(lastDim->isConstant()); + TLLM_CHECK(lastDim->getConstantValue() == std::accumulate(mOutHiddenSizes.cbegin(), mOutHiddenSizes.cend(), 0)); + + outputs[0].nbDims = inputTensorDims.nbDims; + for (auto dim = 0; dim < inputTensorDims.nbDims; ++dim) + { + outputs[0].d[dim] = inputTensorDims.d[dim]; + } + } + catch (std::exception const& e) + { + caughtError(e); + } + return 0; +} + +bool DoraPlugin::supportsFormatCombination( + int32_t pos, nvinfer1::DynamicPluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + auto const numModules = static_cast<int32_t>(mOutHiddenSizes.size()); + if (nbInputs != 2 + numModules + (mRemoveInputPadding ? 1 : 0)) + { + return false; + } + + bool const isInput = pos < nbInputs; + if (pos == IdxEntry::kHOST_REQUEST_TYPES) + { + return (inOut[pos].desc.type == nvinfer1::DataType::kINT32); + } + // optional host_context_lens after lora pointers + else if (pos == IdxEntry::kLORA_WEIGHTS_PTRS_START + numModules and isInput) + { + return (inOut[pos].desc.type == nvinfer1::DataType::kINT32 and mRemoveInputPadding); + } + // lora weight pointers + else if (pos >= IdxEntry::kLORA_WEIGHTS_PTRS_START and pos < IdxEntry::kLORA_WEIGHTS_PTRS_START + numModules) + { + return (inOut[pos].desc.type == nvinfer1::DataType::kINT64); + } + else if (pos != 0 and isInput) + { + TLLM_LOG_WARNING("%s: got an unexpected input at position %d", __PRETTY_FUNCTION__, pos); + return false; + } + + return (inOut[pos].desc.type == mType) and (inOut[pos].desc.format == TensorFormat::kLINEAR); +} + +int32_t DoraPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +size_t DoraPlugin::getWorkspaceSize(nvinfer1::DynamicPluginTensorDesc const* inputs, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* outputs, int32_t nbOutputs) const noexcept +{ + auto const inputTensorMax = inputs[IdxEntry::kINPUT_TENSOR].max; + auto const maxNumTokens = mRemoveInputPadding ? inputTensorMax.d[0] : inputTensorMax.d[0] * inputTensorMax.d[1]; + auto const size = mDoraImpl.getWorkspaceSize(maxNumTokens); + return size; +} + +int32_t DoraPlugin::getValidTactics(int32_t* tactics, int32_t nbTactics) noexcept +{ + return 0; +} + +int32_t DoraPlugin::getNbTactics() noexcept +{ + return 0; +} + +char const* DoraPlugin::getTimingCacheID() noexcept +{ + return nullptr; +} + +int32_t DoraPlugin::getFormatCombinationLimit() noexcept +{ + return 1; +} + +char const* DoraPlugin::getMetadataString() noexcept +{ + return nullptr; +} + +// IPluginV3OneRuntime methods +int32_t DoraPlugin::setTactic(int32_t tactic) noexcept +{ + return 0; +} + +int32_t DoraPlugin::onShapeChange(nvinfer1::PluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::PluginTensorDesc const* out, int32_t nbOutputs) noexcept +{ + return 0; +} + +int32_t DoraPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + + auto const numModules = static_cast<int32_t>(mOutHiddenSizes.size()); + auto const numReqs = inputDesc[IdxEntry::kHOST_REQUEST_TYPES].dims.d[0]; + + auto const inputTensorDesc = inputDesc[IdxEntry::kINPUT_TENSOR]; + auto const numTokens + = mRemoveInputPadding ? inputTensorDesc.dims.d[0] : inputTensorDesc.dims.d[0] * inputTensorDesc.dims.d[1]; + auto const seqLen = mRemoveInputPadding ? 0 : inputTensorDesc.dims.d[1]; + + void const* inputTensor = inputs[IdxEntry::kINPUT_TENSOR]; + auto const* hostRequestTypes = static_cast<int32_t const*>(inputs[IdxEntry::kHOST_REQUEST_TYPES]); + void const* const* loraWeightsPtrs = &inputs[IdxEntry::kLORA_WEIGHTS_PTRS_START]; + + int32_t const* hostContextLengths = mRemoveInputPadding + ? static_cast<int32_t const*>(inputs[IdxEntry::kLORA_WEIGHTS_PTRS_START + numModules]) + : nullptr; + + mExpandDoraWeightPtrs.clear(); + mExpandDoraWeightPtrs.reserve(numModules * numTokens); + + bool hasAnyDora = false; + + for (auto moduleIdx = 0; moduleIdx < numModules; moduleIdx++) + { + auto const loraWeightModulePtrs = static_cast<int64_t const*>(loraWeightsPtrs[moduleIdx]); + + int idx = 0; + for (int reqId = 0; reqId < numReqs; reqId++) + { + // loraWeightModulePtrs has 3 pointers for each module: A,B, and an optional DoRA magnitude + // the current DoRA plugin does not apply LoRA, so A and B are ignored. + RequestType const reqType = static_cast<RequestType const>(hostRequestTypes[reqId]); + auto const* modulePtr = reinterpret_cast<void const*>(loraWeightModulePtrs[reqId * 3 + 2]); + hasAnyDora = hasAnyDora or modulePtr != nullptr; + + if (reqType == RequestType::kGENERATION) + { + mExpandDoraWeightPtrs.push_back(modulePtr); + idx += 1; + } + else + { + int contextLen = (mRemoveInputPadding ? hostContextLengths[reqId] : seqLen); + + for (int contextId = 0; contextId < contextLen; contextId++) + { + mExpandDoraWeightPtrs.push_back(modulePtr); + idx += 1; + } + } + } + if (idx != numTokens) + { + TLLM_LOG_ERROR("LoraParams and input dims don't match, lora tokens %d input tokens %d", idx, numTokens); + return -1; + } + } + + if (hasAnyDora) + { + mDoraImpl.run(numTokens, inputTensor, mExpandDoraWeightPtrs.data(), outputs, workspace, stream); + } + else + { + // skip dora scaling if all requests are pure-lora + auto const inputRank = inputTensorDesc.dims.nbDims; + auto const numel + = std::accumulate(inputTensorDesc.dims.d, inputTensorDesc.dims.d + inputRank, 1, std::multiplies()); + auto const elemSize = tensorrt_llm::common::getDTypeSize(mType); + tensorrt_llm::common::cudaAutoCpy((int8_t*) outputs[0], (int8_t*) inputTensor, numel * elemSize, stream); + } + + sync_check_cuda_error(stream); + return 0; +} + +nvinfer1::IPluginV3* DoraPlugin::attachToContext(nvinfer1::IPluginResourceContext* context) noexcept +{ + return clone(); +} + +nvinfer1::PluginFieldCollection const* DoraPlugin::getFieldsToSerialize() noexcept +{ + return &mFieldsToSerialize; +} + +DoraPluginCreator::DoraPluginCreator() +{ + mPluginAttributes.clear(); + mPluginAttributes.emplace_back("num_modules", nullptr, PluginFieldType::kINT32, 1); + mPluginAttributes.emplace_back("type", nullptr, PluginFieldType::kINT32, 1); + mPluginAttributes.emplace_back("remove_input_padding", nullptr, PluginFieldType::kINT8, 1); + mFC.nbFields = static_cast<int32_t>(mPluginAttributes.size()); + mFC.fields = mPluginAttributes.data(); +} + +char const* DoraPluginCreator::getPluginName() const noexcept +{ + return DORA_PLUGIN_NAME; +} + +char const* DoraPluginCreator::getPluginVersion() const noexcept +{ + return DORA_PLUGIN_VERSION; +} + +nvinfer1::PluginFieldCollection const* DoraPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +nvinfer1::IPluginV3* DoraPluginCreator::createPlugin( + char const* name, nvinfer1::PluginFieldCollection const* fc, nvinfer1::TensorRTPhase phase) noexcept +{ + PluginField const* fields = fc->fields; + nvinfer1::DataType type{}; + bool removeInputPadding{}; + std::vector<int32_t> outHiddenSizes; + + // Read configurations from each field + for (int i = 0; i < fc->nbFields; ++i) + { + auto const field = fields[i]; + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type")) + { + TLLM_CHECK(field.type == PluginFieldType::kINT32 and field.length == 1); + type = *static_cast<nvinfer1::DataType const*>(field.data); + } + else if (!strcmp(attrName, "remove_input_padding")) + { + TLLM_CHECK(field.type == PluginFieldType::kINT8 and field.length == 1); + removeInputPadding = *static_cast<bool const*>(field.data); + } + else if (!strcmp(attrName, "out_hidden_sizes")) + { + TLLM_CHECK(field.type == PluginFieldType::kINT32); + auto const* outHiddenSizesPtr = static_cast<int32_t const*>(field.data); + outHiddenSizes.resize(field.length); + outHiddenSizes.assign(outHiddenSizesPtr, outHiddenSizesPtr + field.length); + } + else + { + TLLM_LOG_WARNING("%s: got an unexpected attribute: %s", __PRETTY_FUNCTION__, attrName); + } + } + + try + { + auto* obj = new DoraPlugin(outHiddenSizes, type, removeInputPadding); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/doraPlugin/doraPlugin.h b/cpp/tensorrt_llm/plugins/doraPlugin/doraPlugin.h new file mode 100644 index 000000000000..dfee11fdc90e --- /dev/null +++ b/cpp/tensorrt_llm/plugins/doraPlugin/doraPlugin.h @@ -0,0 +1,114 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/kernels/lora/dora.h" +#include "tensorrt_llm/plugins/common/plugin.h" + +namespace tensorrt_llm::plugins +{ + +class DoraPlugin : public BasePluginV3 +{ +public: + DoraPlugin() = delete; + DoraPlugin(std::vector<int32_t> const& outHiddenSizes, nvinfer1::DataType type, bool removeInputPadding); + DoraPlugin(DoraPlugin const& p) = default; + + // IPluginV3 methods + nvinfer1::IPluginCapability* getCapabilityInterface(nvinfer1::PluginCapabilityType type) noexcept override; + nvinfer1::IPluginV3* clone() noexcept override; + + // IPluginV3OneCore methods + char const* getPluginName() const noexcept override; + char const* getPluginVersion() const noexcept override; + + // IPluginV3OneBuild methods + int32_t configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept override; + int32_t getOutputDataTypes(nvinfer1::DataType* outputTypes, int32_t nbOutputs, nvinfer1::DataType const* inputTypes, + int32_t nbInputs) const noexcept override; + int32_t getOutputShapes(nvinfer1::DimsExprs const* inputs, int32_t nbInputs, nvinfer1::DimsExprs const* shapeInputs, + int32_t nbShapeInputs, nvinfer1::DimsExprs* outputs, int32_t nbOutputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination(int32_t pos, nvinfer1::DynamicPluginTensorDesc const* inOut, int32_t nbInputs, + int32_t nbOutputs) noexcept override; + int32_t getNbOutputs() const noexcept override; + size_t getWorkspaceSize(nvinfer1::DynamicPluginTensorDesc const* inputs, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* outputs, int32_t nbOutputs) const noexcept override; + int32_t getValidTactics(int32_t* tactics, int32_t nbTactics) noexcept override; + int32_t getNbTactics() noexcept override; + char const* getTimingCacheID() noexcept override; + int32_t getFormatCombinationLimit() noexcept override; + char const* getMetadataString() noexcept override; + + // IPluginV3OneRuntime methods + int32_t setTactic(int32_t tactic) noexcept override; + int32_t onShapeChange(nvinfer1::PluginTensorDesc const* in, int32_t nbInputs, nvinfer1::PluginTensorDesc const* out, + int32_t nbOutputs) noexcept override; + int32_t enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept override; // fixed + nvinfer1::IPluginV3* attachToContext(nvinfer1::IPluginResourceContext* context) noexcept override; + nvinfer1::PluginFieldCollection const* getFieldsToSerialize() noexcept override; + +private: + void init(); + + std::vector<nvinfer1::PluginField> mDataToSerialize; + nvinfer1::PluginFieldCollection mFieldsToSerialize; + + enum IdxEntry + { + kINPUT_TENSOR = 0, + kHOST_REQUEST_TYPES = 1, + kLORA_WEIGHTS_PTRS_START = 2 + }; + + // TODO(oargov) this is shared with the LoRA plugin, put it somewhere else + enum class RequestType : int32_t + { + kCONTEXT = 0, + kGENERATION = 1 + }; + + std::vector<int32_t> mOutHiddenSizes; + nvinfer1::DataType mType; + bool mRemoveInputPadding; + tensorrt_llm::kernels::DoraImpl mDoraImpl; + + std::vector<void const*> mExpandDoraWeightPtrs{}; +}; + +class DoraPluginCreator : public BaseCreatorV3 +{ +public: + DoraPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV3* createPlugin( + char const* name, nvinfer1::PluginFieldCollection const* fc, nvinfer1::TensorRTPhase phase) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +}; // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/eaglePlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/eaglePlugin/CMakeLists.txt new file mode 100644 index 000000000000..b6bd0439cc0c --- /dev/null +++ b/cpp/tensorrt_llm/plugins/eaglePlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/eaglePlugin/eagleDecodeDraftTokensPlugin.cpp b/cpp/tensorrt_llm/plugins/eaglePlugin/eagleDecodeDraftTokensPlugin.cpp new file mode 100644 index 000000000000..899c93855b9f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/eaglePlugin/eagleDecodeDraftTokensPlugin.cpp @@ -0,0 +1,945 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "eagleDecodeDraftTokensPlugin.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/dataType.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/kernels/samplingTopKKernels.h" +#include "tensorrt_llm/kernels/speculativeDecoding/eagleDecodingKernels.h" +#include "tensorrt_llm/kernels/speculativeDecoding/medusaDecodingKernels.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iTensor.h" + +using namespace nvinfer1; +using tensorrt_llm::plugins::EagleDecodeDraftTokensPluginCreator; +using tensorrt_llm::plugins::EagleDecodeDraftTokensPlugin; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::kernels::speculative_decoding; +using namespace tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; + +static char const* EAGLE_DECODE_DRAFT_TOKENS_PLUGIN_VERSION{"1"}; +static char const* EAGLE_DECODE_DRAFT_TOKENS_PLUGIN_NAME{"EagleDecodeDraftTokens"}; +PluginFieldCollection EagleDecodeDraftTokensPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> EagleDecodeDraftTokensPluginCreator::mPluginAttributes; + +EagleDecodeDraftTokensPlugin::EagleDecodeDraftTokensPlugin( + nvinfer1::DataType type, int32_t layerIdx, int32_t numEagleLayers, bool topKSampling) + : mDtype(type) + , mLayerIdx(layerIdx) + , mNumEagleLayers(numEagleLayers) + , mTopKSampling(topKSampling) +{ + TLLM_CHECK_WITH_INFO(mTopKSampling, "Multinomial sampling is not supported yet."); +} + +// Parameterized constructor +EagleDecodeDraftTokensPlugin::EagleDecodeDraftTokensPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mDtype); + read(d, mLayerIdx); + read(d, mNumEagleLayers); + read(d, mTopKSampling); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + static_cast<int>(length), static_cast<int>(d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* EagleDecodeDraftTokensPlugin::clone() const noexcept +{ + auto* plugin = new EagleDecodeDraftTokensPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs EagleDecodeDraftTokensPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + TLLM_CHECK(outputIndex < getNbOutputs()); + TLLM_CHECK(nbInputs == 12); + auto const batchSizeExpr = inputs[getIdx(InputIdxEntry::PATHS)].d[0]; + auto const maxDecodingTokensExpr = inputs[getIdx(InputIdxEntry::PATHS)].d[1]; + auto const maxPathLengthExpr = inputs[getIdx(InputIdxEntry::PATHS)].d[2]; + auto const maxDecodingDraftTokensExpr + = exprBuilder.operation(DimensionOperation::kSUB, *maxDecodingTokensExpr, *exprBuilder.constant(1)); + + auto const numEagleLayersExpr + = exprBuilder.operation(DimensionOperation::kSUB, *maxPathLengthExpr, *exprBuilder.constant(1)); + auto const maxDecodingDraftTokensSquareExpr + = exprBuilder.operation(DimensionOperation::kPROD, *maxDecodingDraftTokensExpr, + *maxDecodingDraftTokensExpr); // maxDecodingDraftTokensExpr * maxDecodingDraftTokensExpr + + nvinfer1::DimsExprs ret; + if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_DRAFT_TOKEN_IDS)) + { + // output_draft_token_ids: [batch_size, max_decoding_draft_tokens] + ret.nbDims = 2; + ret.d[0] = batchSizeExpr; + ret.d[1] = maxDecodingDraftTokensExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_DRAFT_LENS)) + { + // output_draft_lens: [batch_size] + ret.nbDims = 1; + ret.d[0] = batchSizeExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_PATHS)) + { + // output_path: [batch_size, max_decoding_tokens, max_path_len] + ret.nbDims = 3; + ret.d[0] = batchSizeExpr; + ret.d[1] = maxDecodingTokensExpr; + ret.d[2] = maxPathLengthExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_CURRENT_SCORES)) + { + // output_current_scores: [batch_size, max_decoding_draft_tokens] + ret.nbDims = 2; + ret.d[0] = batchSizeExpr; + ret.d[1] = maxDecodingDraftTokensExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_NEXT_EXPAND_INDICES)) + { + // output_next_expand_index + ret.nbDims = 2; + ret.d[0] = batchSizeExpr; + ret.d[1] = maxDecodingDraftTokensExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_ALL_LAYERS_SCORES)) + { + // output_all_layers_scores: + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + ret.nbDims = 3; + ret.d[0] = batchSizeExpr; + ret.d[1] = numEagleLayersExpr; + ret.d[2] = maxDecodingDraftTokensSquareExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_ALL_LAYERS_DRAFT_TOKEN_IDS)) + { + // output_all_layers_draft_token_ids: + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + ret.nbDims = 3; + ret.d[0] = batchSizeExpr; + ret.d[1] = numEagleLayersExpr; + ret.d[2] = maxDecodingDraftTokensSquareExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_ALL_LAYERS_DRAFT_TOKEN_IDS_PREDECESSOR)) + { + // output_all_layers_draft_token_ids_predecessor + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + ret.nbDims = 3; + ret.d[0] = batchSizeExpr; + ret.d[1] = numEagleLayersExpr; + ret.d[2] = maxDecodingDraftTokensSquareExpr; + } + else + { + TLLM_CHECK_WITH_INFO( + false, "Wrong outputIndex %d in EagleDecodeDraftTokensPlugin::getOutputDimensions", outputIndex); + } + return ret; +} + +bool EagleDecodeDraftTokensPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + TLLM_CHECK(nbInputs == 12 && nbOutputs == getNbOutputs()); + TLLM_CHECK(pos < nbInputs + nbOutputs); + + if (pos == getIdx(InputIdxEntry::LOGITS)) + { + // input: logits + // output: output_all_layers_scores + return (inOut[pos].type == mDtype) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == getIdx(InputIdxEntry::INPUT_ALL_LAYERS_SCORES) || pos == getIdx(InputIdxEntry::INPUT_PREV_SCORES) + || pos == nbInputs + getIdx(OutputIdxEntry::OUTPUT_ALL_LAYERS_SCORES) + || pos == nbInputs + getIdx(OutputIdxEntry::OUTPUT_CURRENT_SCORES)) + { + // input: rand_sample, input_all_layers_scores, input_prev_scores + // output: output_all_layers_scores, output_current_scores + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else + { + // input: path, num_valid_logits, use_dynamic_tree, dynamic_tree_max_topK, input_draft_token_ids, + // input_draft_lens, input_current_expand_index, input_all_layers_draft_token_ids + // output: output_draft_token_ids, output_draft_lens, output_path, output_next_expand_index + // output_all_layers_draft_token_ids, output_all_alyers_draft_token_predecessor + return (inOut[pos].type == nvinfer1::DataType::kINT32) && (inOut[pos].format == TensorFormat::kLINEAR); + } +} + +void EagleDecodeDraftTokensPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +template <typename T> +size_t EagleDecodeDraftTokensPlugin::getWorkspaceSizeType(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + size_t workspaceSize{0}; + auto const numInputLogits = inputs[getIdx(InputIdxEntry::LOGITS)].dims.d[0]; + auto const batchSize = inputs[getIdx(InputIdxEntry::PATHS)].dims.d[0]; + auto const vocabSizePadded = inputs[getIdx(InputIdxEntry::LOGITS)].dims.d[1]; + auto const maxDecodingTokens = inputs[getIdx(InputIdxEntry::PATHS)].dims.d[1]; + auto const maxDecodingDraftTokens = maxDecodingTokens - 1; + auto const maxTopK = maxDecodingDraftTokens; + auto const mNumEagleLayers = inputs[getIdx(InputIdxEntry::INPUT_ALL_LAYERS_SCORES)].dims.d[1]; + + // Greedy sampling + if (mTopKSampling) + { + // 0. The first topK sampling workspace + auto const draftTokenSamplingWorkspaceSize + = getTopKWorkspaceSize<T>(numInputLogits, /* maxTokensPerStep */ 1, /* maxTopK */ maxTopK, vocabSizePadded); + + // 1. The first TopKs [numInputLogits] + auto const topKsSize = numInputLogits * sizeof(SizeType32); + + // 2. Topks offset [batchSize] + // Each request will have different number of logits that need to be sampled + // This tensor will record the start offset of the topK for each request + auto const topKOffsetSize = batchSize * sizeof(SizeType32); + + // 3. Logits ptrs [numInputLogits] + auto const logitsPtrsSize = numInputLogits * sizeof(T*); + + // 4. The first topK sampling's output ids ptrs [numInputLogits][maxDecodingDraftTokens] + auto const firstTopKOutputIdsPtrsSize = numInputLogits * sizeof(TokenIdType*); + + // 5. The first topK sampling's output ids (temporary buffer) [numInputLogits * maxDecodingDraftTokens] + auto const firstTopKOutputIdsSize = numInputLogits * maxDecodingDraftTokens * sizeof(TokenIdType); + + // 6. Number of successors for each nodes, extract from the paths and layerId + // [batchSize * maxDecodingTokens] + auto const numSuccessorsForEachNodeSize = batchSize * maxDecodingTokens * sizeof(SizeType32); + + // 7. Flag whether to do decoding or not. SamplingTopK is done for numInputLogits tokens. + // But only sum(numValidLogitsPerRequest[:]) of them are valid. + // [batchSize * maxDecodingTokens] + auto const skipDecodeSize = numInputLogits * sizeof(bool); + + // 8. The first topK sampling's logprobs [batchSize * maxDecodingDraftTokens] + auto const firstTopKOutputLogProbsSize = numInputLogits * maxDecodingDraftTokens * sizeof(float); + + // 9. Eagle-2, the second topK sampling workspace + // Sampling from [batchSize, maxTopK * maxTopK] to [batchSize, maxTopK] + auto const secondTopKSamplingWorkspaceSize = getTopKWorkspaceSize<float>( + batchSize, /* maxTokensPerStep */ 1, /* maxTopK */ maxTopK, maxTopK * maxTopK); + + // 10. Eagle-2, the outputIds of the second topK sampling, shape [batchSize, maxDecodingTokens] + auto const secondTopKOutputIdsSize = batchSize * maxDecodingTokens * sizeof(TokenIdType); + // 11. Eagle-2, the outputIdsPtr of the second topK sampling, shape [batchSize] + auto const secondTopKOutputIdsPtrSize = batchSize * sizeof(TokenIdType*); + // 12. Eagle-2, the inputScoresPtrs of the second topK sampling, shape [batchSize] + auto const secondTopKInputScoresPtrsSize = batchSize * sizeof(float*); + // 13. Eagle-2, the outpuLogProbs of the second topK samplig, shape [batchSize, maxDecodingDraftTokens] + auto const secondTopKOutputLogProbsSize = batchSize * maxDecodingDraftTokens * sizeof(float); + + // 14. Eagle-2, the input scores pointers of the third topK sampling, shape [batchSize] + // Each points to a vocabSize = '(mNumEagleLayers - 1) * dynamicTreeMaxTopK * dynamicTreeMaxTopK + + // dynamicTreeMaxTopK' + auto const thirdTopKInputScoresPtrsSize = batchSize * sizeof(float*); + // 15. Eagle-2, the output of the third topK sampling, shape [batchSize, maxDecodingDraftTokens] + auto const thirdTopKOutputIdsSize = batchSize * maxDecodingDraftTokens * sizeof(TokenIdType); + // 16. Eagle-2, the output pointers of the third topK sampling, shape [batchSize] + auto const thirdTopKOutputIdsPtrsSize = batchSize * sizeof(TokenIdType*); + // 17. Eagle-2, the workspace of the third topK sampling + // Sampling from [batchSize, '(mNumEagleLayers - 1) * dynamicTreeMaxTopK * dynamicTreeMaxTopK + + // dynamicTreeMaxTopK'] to [batchSize, maxDecodingDraftTokens] We over-set the vocabsize here. + auto const thridTopKSamplingWorkspaceSize = getTopKWorkspaceSize<float>(batchSize, /* maxTokensPerStep */ 1, + /* maxTopK */ maxDecodingDraftTokens, mNumEagleLayers * maxDecodingDraftTokens * maxDecodingDraftTokens); + + // 18. Eagle-2, the topKs for each request in the third topK sampling + // The real topK value is min(maxDecodingDraftTokens, totalNumDraftTokensForAllLayers) + auto const thirdTopKsSize = batchSize * sizeof(SizeType32); + + SizeType32 constexpr NUM_BUFFERS{19}; + size_t workspaces[NUM_BUFFERS]; + workspaces[0] = draftTokenSamplingWorkspaceSize; + workspaces[1] = topKsSize; + workspaces[2] = topKOffsetSize; + workspaces[3] = logitsPtrsSize; + workspaces[4] = firstTopKOutputIdsPtrsSize; + workspaces[5] = firstTopKOutputIdsSize; + workspaces[6] = numSuccessorsForEachNodeSize; + workspaces[7] = skipDecodeSize; + workspaces[8] = firstTopKOutputLogProbsSize; + workspaces[9] = secondTopKSamplingWorkspaceSize; + workspaces[10] = secondTopKOutputIdsSize; + workspaces[11] = secondTopKOutputIdsPtrSize; + workspaces[12] = secondTopKInputScoresPtrsSize; + workspaces[13] = secondTopKOutputLogProbsSize; + workspaces[14] = thirdTopKInputScoresPtrsSize; + workspaces[15] = thirdTopKOutputIdsSize; + workspaces[16] = thirdTopKOutputIdsPtrsSize; + workspaces[17] = thridTopKSamplingWorkspaceSize; + workspaces[18] = thirdTopKsSize; + workspaceSize = tc::calculateTotalWorkspaceSize(workspaces, NUM_BUFFERS); + } + else + { + // TODO fill me + // Multinomial sampling + TLLM_CHECK_WITH_INFO(false, "Multinomial sampling is not supported yet."); + } + + return workspaceSize; +} + +size_t EagleDecodeDraftTokensPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + auto const logitsType = inputs[getIdx(InputIdxEntry::LOGITS)].type; + if (logitsType == nvinfer1::DataType::kFLOAT) + { + return getWorkspaceSizeType<float>(inputs, nbInputs, outputs, nbOutputs); + } + else if (logitsType == nvinfer1::DataType::kHALF) + { + return getWorkspaceSizeType<__half>(inputs, nbInputs, outputs, nbOutputs); + } + else + { + TLLM_CHECK_WITH_INFO(false, "Unsupported logits type"); + } + return 0; +} + +template <typename T> +void EagleDecodeDraftTokensPlugin::doTopKSampling(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // We allocate many buffers with 'numInputLogits' size, but the input logits will include some padding logits. + // So only 'batchSize' or 'numValidLogits' size will be actually used. + auto const numInputLogits = inputDesc[getIdx(InputIdxEntry::LOGITS)].dims.d[0]; + auto const vocabSizePadded = inputDesc[getIdx(InputIdxEntry::LOGITS)].dims.d[1]; + auto const batchSize = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[0]; + auto const maxDecodingTokens = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[1]; + auto const maxPathLen = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[2]; + auto const maxDecodingDraftTokens = maxDecodingTokens - 1; + auto const maxTopK = maxDecodingDraftTokens; + + ////////////////////////////////////////// Get plugin inputs ////////////////////////////////////////// + // Plugin inputs + // Input logits for sampling, shape: [numInputLogits, vocabSizePadded] + auto pluginInputLogits = static_cast<T const*>(inputs[getIdx(InputIdxEntry::LOGITS)]); + // Input paths, shape: [batchSize, maxDecodingTokens, maxPathLen] + auto pluginInputPaths = static_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::PATHS)]); + auto numValidLogits = static_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::NUM_VALID_LOGITS)]); + // For Eagle-2 + // Whether to use dynamic tree (i.e., Eagle-2) + auto useDynamicTree = *(static_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::USE_DYNAMIC_TREE)])); + // The max topK for dynamic tree. All the requests have the same expand topK. + // In Eagle-2, dynamicTreeMaxTopK is equal to maxNonLeavesPerLayer in the internal EagleNets. + auto dynamicTreeMaxTopK = *(static_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::DYNAMIC_TREE_MAX_TOPK)])); + // All layer's draft tokenIds, shape: [batchSize, maxDecodingDraftTokens] + auto pluginInputDraftTokenIds + = reinterpret_cast<TokenIdType const*>(inputs[getIdx(InputIdxEntry::INPUT_DRAFT_TOKEN_IDS)]); + // The number of all layer's draft tokenIds, shape: [batchSize] + auto pluginInputDraftLens = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::INPUT_DRAFT_LENS)]); + // The previous EagleNet's scores, shape: [batchSize, maxDecodingDraftTokens] + auto pluginInputPrevScores = static_cast<float const*>(inputs[getIdx(InputIdxEntry::INPUT_PREV_SCORES)]); + // The indices of the nodes that will be expand in this layer, shape: [batchSize, maxDecodingDraftTokens] + // The index is related to the final output tree, which has max_decoding_draft_tokens draft tokens. + auto pluginInputCurrentExpandIndices + = reinterpret_cast<TokenIdType const*>(inputs[getIdx(InputIdxEntry::INPUT_CURRENT_EXPAND_INDICES)]); + // The scores from all previous EagleNets, + // shape: [batchSize, mNumEagleLayers, maxDecodingDraftTokens x maxDecodingDraftTokens] + auto pluginInputAllLayersScores = static_cast<float const*>(inputs[getIdx(InputIdxEntry::INPUT_ALL_LAYERS_SCORES)]); + // The draft tokens from all previous EagleNets, + // shape: [batchSize, mNumEagleLayers, maxDecodingDraftTokens x maxDecodingDraftTokens] + auto pluginInputAllLayersDraftTokenIds + = reinterpret_cast<TokenIdType const*>(inputs[getIdx(InputIdxEntry::INPUT_ALL_LAYERS_DRAFT_TOKEN_IDS)]); + // The predecessor of all the draft tokens, + // shape: [batchSize, mNumEagleLayers, maxDecodingDraftTokens x maxDecodingDraftTokens] + auto pluginInputAllLayersDraftTokenIdsPredecessor = reinterpret_cast<SizeType32 const*>( + inputs[getIdx(InputIdxEntry::INPUT_ALL_LAYERS_DRAFT_TOKEN_IDS_PREDECESSOR)]); + + ////////////////////////////////////////// Get plugin outputs ////////////////////////////////////////// + // Plugin outputs + // All layer's draft tokenIds, shape: [batchSize, maxDecodingDraftTokens] + auto pluginOutputDraftTokenIds + = reinterpret_cast<TokenIdType*>(outputs[getIdx(OutputIdxEntry::OUTPUT_DRAFT_TOKEN_IDS)]); + // The number of all layer's draft tokenIds, shape: [batchSize] + auto pluginOutputDraftLens = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::OUTPUT_DRAFT_LENS)]); + // For Eagle-2 + // Updated paths base on this layer's sampling result, shape: [batchSize, maxDecodingTokens, maxPathLen] + auto pluginOutputPaths = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::OUTPUT_PATHS)]); + // This layer's scores, which will be used in next layers [batchSize, maxDecodingDraftTokens] + auto pluginOutputCurrentScores = static_cast<float*>(outputs[getIdx(OutputIdxEntry::OUTPUT_CURRENT_SCORES)]); + // The indices of the nodes that will be expand in next layer, shape: [batchSize, maxDecodingDraftTokens] + // The index is related to the final output tree, which has max_decoding_draft_tokens draft tokens. + auto pluginOutputNextExpandIndices + = reinterpret_cast<TokenIdType*>(outputs[getIdx(OutputIdxEntry::OUTPUT_NEXT_EXPAND_INDICES)]); + // Updated scores, shape: [batchSize, mNumEagleLayers, maxDecodingDraftTokens x maxDecodingDraftTokens] + auto pluginOutputAllLayersScores = static_cast<float*>(outputs[getIdx(OutputIdxEntry::OUTPUT_ALL_LAYERS_SCORES)]); + // Updated draft tokens, shape: [batchSize, mNumEagleLayers, maxDecodingDraftTokens x maxDecodingDraftTokens] + auto pluginOutputAllLayersDraftTokenIds + = reinterpret_cast<TokenIdType*>(outputs[getIdx(OutputIdxEntry::OUTPUT_ALL_LAYERS_DRAFT_TOKEN_IDS)]); + // Update the predecessor of the draft tokens, shape: [batchSize, mNumEagleLayers, maxDecodingDraftTokens x + // maxDecodingDraftTokens] + auto pluginOutputAllLayersDraftTokenIdsPredecessor + = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::OUTPUT_ALL_LAYERS_DRAFT_TOKEN_IDS_PREDECESSOR)]); + + ////////////////////////////////////////// Get workspaces ////////////////////////////////////////// + int8_t* workspaceBytePtr = reinterpret_cast<int8_t*>(workspace); + size_t offset{0}; + // Workspace 0: Sampling workspace. + // Treat numInputLogits as batchSize + auto const samplingWorkspaceSize + = getTopKWorkspaceSize<T>(numInputLogits, /* maxTokensPerStep */ 1, /* maxTopK */ maxTopK, vocabSizePadded); + void* workspaceSampling + = reinterpret_cast<void*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, samplingWorkspaceSize)); + + // Workspace 1: Topks tensor: shape [numInputLogits] + SizeType32* topKs = reinterpret_cast<SizeType32*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, numInputLogits * sizeof(SizeType32))); + + // Workspace 2: topKOffset tensor: shape: [batchSize], number of nodes that have successors for each requests + SizeType32* topKOffset + = reinterpret_cast<SizeType32*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(SizeType32))); + + // Workspace 3: logits pointers tensor: shape: [numInputLogits] + T const** logitsPtrs + = reinterpret_cast<T const**>(tc::nextWorkspacePtr(workspaceBytePtr, offset, numInputLogits * sizeof(T*))); + + // Workspace 4: outputIds pointers tensor: shape [numInputLogits], each points to a [maxDecodingDraftTokens] buffer + TokenIdType** firstTopKOutputIdsPtrs = reinterpret_cast<TokenIdType**>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, numInputLogits * sizeof(TokenIdType*))); + + // Workspace 5: outputIds tensor: flatten outputIds, shape [numInputLogits * maxDecodingDraftTokens] + TokenIdType* firstTopKOutputIdsFlatten = reinterpret_cast<TokenIdType*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, numInputLogits * maxDecodingDraftTokens * sizeof(TokenIdType))); + + // Workspace 6: number of successors for each nodes tensor: shape [batchSize * maxDecodingTokens] + SizeType32* numSuccessorsForEachNode = reinterpret_cast<SizeType32*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(SizeType32))); + + // Workspace 7: skip decoding mask [numInputLogits] + bool* skipDecode + = reinterpret_cast<bool*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, numInputLogits * sizeof(bool))); + + // In Eagle-1, we do not need to return logProbs + float* firstTopKOutputLogProbs = nullptr; + if (useDynamicTree) + { + // Workspace 8. The output logProbs of the first topK sampling. + // Which will be updated with the previous layer's scores (i.e., pluginInputPrevScores), and will be treat as + // the input of the second topK sampling. For mLayerIdx == 0, shape: [numInputLogits(batchSize), + // maxDecodingDraftTokens] For mLayerIdx > 0, shape: [numInputLogits(batchSize * dynamicTreeMaxTopK), + // maxDecodingDraftTokens] + firstTopKOutputLogProbs = reinterpret_cast<float*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, numInputLogits * maxDecodingDraftTokens * sizeof(float))); + } + + SizeType32 const secondTopKVocabSize = dynamicTreeMaxTopK * maxDecodingDraftTokens; + // Workspace 9: Sampling from [batchSize, dynamicTreeMaxTopK * maxDecodingDraftTokens] to [batchSize, + // dynamicTreeMaxTopK] + auto const secondTopKSamplingWorkspaceSize + = getTopKWorkspaceSize<float>(batchSize, /* maxTokensPerStep */ 1, /* maxTopK */ maxTopK, secondTopKVocabSize); + void* workspaceScoresSampling + = reinterpret_cast<void*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, secondTopKSamplingWorkspaceSize)); + + // Workspace 10: the second (scores) sampling's outputIds, shape: [batchSize, maxDecodingDraftTokens] + TokenIdType* secondTopKOutputIdsFlatten = reinterpret_cast<TokenIdType*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingDraftTokens * sizeof(TokenIdType))); + + // Workspace 11: the second (scores) sampling's outputIdsPtrs + TokenIdType** secondTopKOutputIdsPtrs = reinterpret_cast<TokenIdType**>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(TokenIdType*))); + + // Workspace 12: input scores pointers + float** secondTopKInputScoresPtrs + = reinterpret_cast<float**>(tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(float*))); + + // Workspace 13: the second sampling's outputLogProbs + float* secondTopKOutputLogProbs = reinterpret_cast<float*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingDraftTokens * sizeof(float))); + + // Workspace 14: The input scores pointers of the third topK sampling, shape [batchSize] + float** thirdTopKInputScoresPtrs + = reinterpret_cast<float**>(tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(float*))); + + // Workspace 15: The output of the third topK sampling, shape [batchSize, maxDecodingDraftTokens] + TokenIdType* thirdTopKOutputIds = reinterpret_cast<TokenIdType*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingDraftTokens * sizeof(TokenIdType))); + + // Workspace 16: The output pointers of the third topK sampling, shape [batchSize] + TokenIdType** thirdTopKOutputIdsPtrs = reinterpret_cast<TokenIdType**>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(TokenIdType*))); + + // The number of draft tokens among all layers + long const totalNumDraftTokensForAllLayers + = (mNumEagleLayers - 1) * dynamicTreeMaxTopK * dynamicTreeMaxTopK + dynamicTreeMaxTopK; + + auto const thridTopKSamplingWorkspaceSize = getTopKWorkspaceSize<float>( + batchSize, /* maxTokensPerStep */ 1, /* maxTopK */ maxDecodingDraftTokens, totalNumDraftTokensForAllLayers); + // Workspace 17: The workspace of the third topK sampling + void* workspaceThirdTopKSampling + = reinterpret_cast<void*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, thridTopKSamplingWorkspaceSize)); + + // Workspace 18. Eagle-2, the topKs for each request in the third topK sampling, shape [batchSize] + // The real topK value is min(maxDecodingDraftTokens, totalNumDraftTokensForAllLayers) + SizeType32* thirdTopKs + = reinterpret_cast<SizeType32*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(SizeType32))); + + ////////////////////////////////////////// Main logic ////////////////////////////////////////// + // Fill logitsPtrs from plugin input logits + // And fill firstTopKOutputIdsPtrs from firstTopKOutputIdsFlatten + invokeAssembleDraftLogitsOffsets(logitsPtrs, pluginInputLogits, firstTopKOutputIdsPtrs, firstTopKOutputIdsFlatten, + skipDecode, numValidLogits, numInputLogits, batchSize, maxDecodingDraftTokens, vocabSizePadded, stream); + sync_check_cuda_error(stream); + + if (useDynamicTree) + { + // For Eagle-2, the topK value between different requests are the same, all set to 'dynamicTreeMaxTopK'. + invokeSetTopKsFromDyanmicTreeMaxTopK( + mLayerIdx, batchSize, numInputLogits, topKs, topKOffset, dynamicTreeMaxTopK, numValidLogits, stream); + sync_check_cuda_error(stream); + + // Do softmax for the input logits + // We set the 'batchSize' and 'maxBatchSize' to 'numInputLogits', while 'numInputLogits' logits may contain + // some padding logits, which do not need to be calculated. + // We use 'skipDecode' list to skip these padding logits. This could avoid redundant calculations. + BiasSoftmaxParams<T> biasSoftmaxParams; + biasSoftmaxParams.logits = const_cast<T*>(pluginInputLogits); + biasSoftmaxParams.logitsPtrs = nullptr; + biasSoftmaxParams.probs = const_cast<T*>(pluginInputLogits); + biasSoftmaxParams.maxBeamWidth = 1; + biasSoftmaxParams.batchSlots = nullptr; + biasSoftmaxParams.batchSize = numInputLogits; + biasSoftmaxParams.maxBatchSize = numInputLogits; + biasSoftmaxParams.vocabSize = vocabSizePadded; + biasSoftmaxParams.vocabSizePadded = vocabSizePadded; + biasSoftmaxParams.skipSoftMax = false; + biasSoftmaxParams.batchSlotsLogits = false; + biasSoftmaxParams.skipDecode = skipDecode; + biasSoftmaxParams.checkParams(); + + invokeAddBiasSoftMax(biasSoftmaxParams, stream); + sync_check_cuda_error(stream); + } + else + { + // For Eagle-1, extract topK value from input path. + invokeExtractTopKsFromPath(pluginInputPaths, topKs, topKOffset, numSuccessorsForEachNode, mLayerIdx, batchSize, + maxDecodingTokens, maxPathLen, stream); + sync_check_cuda_error(stream); + } + + TopKSamplingKernelParams<T> params{}; + params.logProbsPtrs = logitsPtrs; // [numInputLogits][vocabSizePadded] + params.outputIdsPtrs = firstTopKOutputIdsPtrs; // [numInputLogits][maxDecodingDraftTokens] + params.workspace = workspaceSampling; + params.maxTopK = maxTopK; + params.topKs = topKs; // [numInputLogits] + params.batchSize = numInputLogits; + params.maxBatchSize = numInputLogits; + params.maxTokensPerStep = 1; + params.vocabSizePadded = vocabSizePadded; + params.returnAllSelectedTokens = true; + params.strictTopPBoundary = false; + params.skipDecode = skipDecode; + params.outputLogProbs = firstTopKOutputLogProbs; // [numInputLogits * maxDecodingDraftTokens] + params.logitsHasProbs = true; + + invokeBatchTopKSampling(params, stream); + sync_check_cuda_error(stream); + + if (useDynamicTree) + { + // When mLayerIdx == 0, we do not need to update scores. + // We take the outputLogProbs of the first topK sampling as the scores directly. + if (mLayerIdx != 0) + { + // Update firstTopKOutputLogProbs with pluginInputPrevScores, which is the scores from the previous layer + invokeUpdateScores(batchSize, dynamicTreeMaxTopK, maxDecodingDraftTokens, firstTopKOutputLogProbs, + pluginInputPrevScores, stream); + sync_check_cuda_error(stream); + + // Do the second top-dynamicTreeMaxTopK sampling among this dynamicTreeMaxTopK x dynamicTreeMaxTopK draft + // tokens. Through the second topK sampling, we obtain the dynamicTreeMaxTopK output draft tokens of this + // layer. + + // Although theoretically we only need to select 'dynamicTreeMaxTopK' draft tokens from 'dynamicTreeMaxTopK + // * dynamicTreeMaxTopK' draft tokens, we over-set vocabSize here. This is because when we write the scores + // into firstTopKOutputLogProbs, we store it in the form of [batchSize * dynamicTreeMaxTopK, + // maxDecodingDraftTokens]. For each request, these 'dynamicTreeMaxTopK * dynamicTreeMaxTopK' scores are not + // saved continuously, but in the format of [dynamicTreeMaxTopK, maxDecodingDraftTokens]. For unused + // positions, we set '-inf' to ensure that they will not be sampled. Examples: For a request, + // dynamicTreeMaxTopK == 3, the scores in its buffer ([dynamicTreeMaxTopK, maxDecodingDraftTokens]) are as + // follow: + // [[1.1, 2.2, 3.3, -inf, -inf, ...], + // [4.4, 5.5, 6.6, -inf, -inf, ...], + // [7.7, 8.8, 9.9, -inf, -inf, ...]] + + // Prepare the input of the second topK sampling. + invokeAssembleSecondTopKSamplingInputs(batchSize, dynamicTreeMaxTopK, maxDecodingDraftTokens, + firstTopKOutputLogProbs, secondTopKInputScoresPtrs, secondTopKOutputIdsFlatten, secondTopKOutputIdsPtrs, + stream); + sync_check_cuda_error(stream); + + TopKSamplingKernelParams<float> params{}; + params.logProbsPtrs = secondTopKInputScoresPtrs; + params.outputIdsPtrs = secondTopKOutputIdsPtrs; + params.workspace = workspaceScoresSampling; + params.maxTopK = maxTopK; // Same to maxDecodingTokens + params.topKs = topKs; // [batchSize], all set to dynamicTreeMaxTopK + params.batchSize = batchSize; + params.maxBatchSize = batchSize; + params.maxTokensPerStep = 1; + params.vocabSizePadded = secondTopKVocabSize; + params.returnAllSelectedTokens = true; + params.strictTopPBoundary = false; + + invokeBatchTopKSampling(params, stream); + sync_check_cuda_error(stream); + } + + // Copy this layer's scores and draft tokensId: + // 1) Copy this layer's scores to pluginOutputAllLayersScores + // 2) Copy dynamicTreeMaxTopK (or dynamicTreeMaxTopK * dynamicTreeMaxTopK) draft tokens to + // pluginOutputAllLayersDraftTokenIds 3) Set the predecessors of these draft tokens and save to + // pluginOutputAllLayersDraftTokenIdsPredecessor, + // which will be used to reconstruct the final output tree at the last layer + invokeCopyScoresAndDraftTokenIds(mLayerIdx, mNumEagleLayers, maxDecodingDraftTokens, batchSize, + dynamicTreeMaxTopK, + pluginInputCurrentExpandIndices, // The indices of the nodes that expand in this layer (i.e., the input + // logits). The index is related to the final tree. + pluginInputAllLayersScores, pluginInputAllLayersDraftTokenIds, pluginInputAllLayersDraftTokenIdsPredecessor, + pluginOutputAllLayersScores, pluginOutputAllLayersDraftTokenIds, + pluginOutputAllLayersDraftTokenIdsPredecessor, + firstTopKOutputLogProbs, // This layer's scores + firstTopKOutputIdsFlatten, // This layer's draft tokens + stream); + sync_check_cuda_error(stream); + + // Update Path + // For mLayerIdx == 0, the output of the first topK sampling are the output draft tokens of this layers. The + // update logic is simple. For mLayerIdx > 0, the output of the second topK sampling are the output draft tokens + // of this layers. 'secondTopKOutputIdsPtrs' contains the top-dynamicTreeMaxTopK selected from the second topK + // sampling. 'pluginOutputNextExpandIndices' record the selected the top-dynamicTreeMaxTopK draft token's Id of + // this layer, + // which will be used in the next layer to compute the predecessors. + // The last layer will completely reconstruct the paths, so there is no need to update the paths here. + if (mLayerIdx != mNumEagleLayers - 1) + { + invokeUpdatePath(mLayerIdx, batchSize, dynamicTreeMaxTopK, maxDecodingTokens, maxPathLen, pluginInputPaths, + pluginOutputPaths, + secondTopKOutputIdsPtrs, // if mLayerIdx == 0, secondTopKOutputIdsPtrs == nullptr, and it's useless + // during update paths + pluginOutputNextExpandIndices, stream); + sync_check_cuda_error(stream); + } + + if (mLayerIdx != 0) + { + // We will extract the real draft tokenIds and scores from 'firstTopKOutputIdsFlatten' and + // 'secondTopKInputScoresPtrs' according to the 'secondTopKOutputIdsPtrs'. And store them into + // 'secondTopKOutputIdsPtrs' and 'secondTopKOutputLogProbs' (reuse these buffers). + // secondTopKInputScoresPtrs: shape [batchSize * dynamicTreeMaxTopK, maxDecodingDraftTokens] + // The original scores, which were used to do the second TopK sampling + // secondTopKOutputIdsPtrs: shape [batchSize], each points to a [maxDecodingDraftTokens] buffer + // The output of the second TopK sampling, which are the indices of the top-dynamicTreeMaxTopK among + // 'dynamicTreeMaxTopK * dynamicTreeMaxTopK'. We need to figure out what these top-dynamicTreeMaxTopK + // draft tokens' real tokenIds. + // firstTopKOutputIdsFlatten: shape [batchSize * dynamicTreeMaxTopK, maxDecodingDraftTokens] + // The value are related to the vocabSize, which is the real tokenIds. + invokeExtractScoresAndRealDraftTokensIds(batchSize, dynamicTreeMaxTopK, maxDecodingDraftTokens, + secondTopKInputScoresPtrs, secondTopKOutputIdsPtrs, firstTopKOutputIdsFlatten, secondTopKOutputLogProbs, + stream); + sync_check_cuda_error(stream); + } + + // Copy this layer's output draft tokens and scores. + // This layer's output scores is next layer's previous scores. + // if mLayerIdx == 0, directly use the first topK's outputIds / logProbs as this layer's output draft tokens / + // scores if mLayerIdx > 0, we use the second topK's outputIds / logProbs, + // which is updated with the real draft tokenIds / logprobs in 'invokeExtractScoresAndRealDraftTokensIds' + invokeUpdateDraftTokensAndLensAndCurScores(mLayerIdx, batchSize, dynamicTreeMaxTopK, maxDecodingDraftTokens, + mLayerIdx == 0 ? firstTopKOutputIdsPtrs : secondTopKOutputIdsPtrs, pluginInputDraftTokenIds, + pluginInputDraftLens, pluginOutputDraftTokenIds, pluginOutputDraftLens, + mLayerIdx == 0 ? firstTopKOutputLogProbs : secondTopKOutputLogProbs, pluginOutputCurrentScores, stream); + sync_check_cuda_error(stream); + + if (mLayerIdx == mNumEagleLayers - 1) + { + // The maximum number of nodes on the final tree (exclude the root node) + auto const maxNodesOnFinalTree = std::min(maxDecodingDraftTokens, totalNumDraftTokensForAllLayers); + + // When reach the last EagleNet, we need to do the third sampling, which take all layers' draft tokens and + // scores as input, and then select top-maxDecodingDraftTokens draft tokens among them. We need to + // reconstruct the path/tree after the third topK sampling. + invokeAssembleThridTopKSamplingInputs(batchSize, maxDecodingDraftTokens, mNumEagleLayers, + maxNodesOnFinalTree, thirdTopKs, pluginOutputAllLayersScores, thirdTopKInputScoresPtrs, + thirdTopKOutputIds, thirdTopKOutputIdsPtrs, stream); + sync_check_cuda_error(stream); + + // 1) Do topK sampling among all previous draft tokens + TopKSamplingKernelParams<float> params{}; + params.logProbsPtrs = thirdTopKInputScoresPtrs; + params.outputIdsPtrs = thirdTopKOutputIdsPtrs; + params.workspace = workspaceThirdTopKSampling; + params.topKs = thirdTopKs; // All set to 'maxNodesOnFinalTree' + params.maxTopK = maxDecodingDraftTokens; // We set maxTopK to 'maxDecodingDraftTokens' to align the + // outputIdsPtrs offsets when written back. + params.batchSize = batchSize; + params.maxBatchSize = batchSize; + params.maxTokensPerStep = 1; + params.vocabSizePadded = totalNumDraftTokensForAllLayers; + params.returnAllSelectedTokens = true; + params.strictTopPBoundary = false; // Make sure to select topK tokens. + + invokeBatchTopKSampling(params, stream); + sync_check_cuda_error(stream); + + // 2) Reconstruct the Path + invokeReconstructFinalPath(batchSize, dynamicTreeMaxTopK, maxDecodingDraftTokens, maxDecodingTokens, + maxPathLen, mNumEagleLayers, maxNodesOnFinalTree, thirdTopKOutputIdsPtrs, + pluginOutputAllLayersDraftTokenIdsPredecessor, pluginOutputPaths, stream); + sync_check_cuda_error(stream); + + // 3) Copy this layer's outputIds to outputDraftTokenIds + invokeCopyFinalDraftTokens(batchSize, maxDecodingDraftTokens, mNumEagleLayers, maxNodesOnFinalTree, + thirdTopKOutputIdsPtrs, pluginOutputAllLayersDraftTokenIds, pluginOutputDraftTokenIds, + pluginOutputDraftLens, stream); + sync_check_cuda_error(stream); + } + } + else + { + // Eagle-1: Copy output token id from outputIdsPtrs to the plugin output buffer + invokeCopyOutputTokensIds(firstTopKOutputIdsPtrs, topKs, topKOffset, pluginInputDraftTokenIds, + pluginInputDraftLens, numValidLogits, pluginOutputDraftTokenIds, pluginOutputDraftLens, mLayerIdx, + batchSize, maxDecodingDraftTokens, pluginInputPaths, pluginOutputPaths, maxPathLen, stream); + sync_check_cuda_error(stream); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +template <typename T> +void EagleDecodeDraftTokensPlugin::enqueueType(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // TODO split batch into greedy and non-greedy and execute both paths + if (mTopKSampling) + { + doTopKSampling<T>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else + { + // TODO fill me + TLLM_CHECK_WITH_INFO(false, "Multinomial sampling is not supported yet"); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +int EagleDecodeDraftTokensPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + auto const logitsType = inputDesc[getIdx(InputIdxEntry::LOGITS)].type; + if (logitsType == nvinfer1::DataType::kFLOAT) + { + enqueueType<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (logitsType == nvinfer1::DataType::kHALF) + { + enqueueType<__half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else + { + TLLM_CHECK_WITH_INFO(false, "Unsupported logits type"); + } + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType EagleDecodeDraftTokensPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index < getNbOutputs()); + TLLM_CHECK(index < getNbOutputs()); + if (index == getIdx(OutputIdxEntry::OUTPUT_ALL_LAYERS_SCORES) + || index == getIdx(OutputIdxEntry::OUTPUT_CURRENT_SCORES)) + { + // Only output_prev_socres are float + return inputTypes[getIdx(InputIdxEntry::INPUT_ALL_LAYERS_SCORES)]; + } + else + { + // output_draft_token_ids, output_draft_lens, output_paths, output_next_expand_index, + // output_all_layers_draft_token_ids, output_all_layers_draft_token_ids_predecessor + // are all int32 type, same as path + return inputTypes[getIdx(InputIdxEntry::PATHS)]; + } +} + +// IPluginV2 Methods + +char const* EagleDecodeDraftTokensPlugin::getPluginType() const noexcept +{ + return EAGLE_DECODE_DRAFT_TOKENS_PLUGIN_NAME; +} + +char const* EagleDecodeDraftTokensPlugin::getPluginVersion() const noexcept +{ + return EAGLE_DECODE_DRAFT_TOKENS_PLUGIN_VERSION; +} + +int EagleDecodeDraftTokensPlugin::getNbOutputs() const noexcept +{ + return 8; +} + +int EagleDecodeDraftTokensPlugin::initialize() noexcept +{ + return 0; +} + +void EagleDecodeDraftTokensPlugin::terminate() noexcept {} + +size_t EagleDecodeDraftTokensPlugin::getSerializationSize() const noexcept +{ + return sizeof(mDtype) + sizeof(mLayerIdx) + sizeof(mNumEagleLayers) + sizeof(mTopKSampling); +} + +void EagleDecodeDraftTokensPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mDtype); + write(d, mLayerIdx); + write(d, mNumEagleLayers); + write(d, mTopKSampling); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void EagleDecodeDraftTokensPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +EagleDecodeDraftTokensPluginCreator::EagleDecodeDraftTokensPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("layer_idx", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("num_eagle_layers", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("top_k_sampling", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* EagleDecodeDraftTokensPluginCreator::getPluginName() const noexcept +{ + return EAGLE_DECODE_DRAFT_TOKENS_PLUGIN_NAME; +} + +char const* EagleDecodeDraftTokensPluginCreator::getPluginVersion() const noexcept +{ + return EAGLE_DECODE_DRAFT_TOKENS_PLUGIN_VERSION; +} + +PluginFieldCollection const* EagleDecodeDraftTokensPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* EagleDecodeDraftTokensPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int32_t layerIdx{}; + int32_t numEagleLayers{}; + nvinfer1::DataType type{}; + bool topKSampling{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "layer_idx")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + layerIdx = *static_cast<int32_t const*>(fields[i].data); + } + else if (!strcmp(attrName, "num_eagle_layers")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + numEagleLayers = *static_cast<int32_t const*>(fields[i].data); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "top_k_sampling")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + topKSampling = static_cast<bool>(*static_cast<int32_t const*>(fields[i].data)); + } + } + + try + { + auto* obj = new EagleDecodeDraftTokensPlugin(type, layerIdx, numEagleLayers, topKSampling); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* EagleDecodeDraftTokensPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call EagleDecodeDraftTokensPlugin::destroy() + try + { + auto* obj = new EagleDecodeDraftTokensPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/eaglePlugin/eagleDecodeDraftTokensPlugin.h b/cpp/tensorrt_llm/plugins/eaglePlugin/eagleDecodeDraftTokensPlugin.h new file mode 100644 index 000000000000..8c144a1bc073 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/eaglePlugin/eagleDecodeDraftTokensPlugin.h @@ -0,0 +1,174 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class EagleDecodeDraftTokensPlugin : public BasePlugin +{ +public: + EagleDecodeDraftTokensPlugin(nvinfer1::DataType type, int32_t layerIdx, int32_t numEagleLayers, bool topKSampling); + + EagleDecodeDraftTokensPlugin(void const* data, size_t length); + + ~EagleDecodeDraftTokensPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + enum class InputIdxEntry : int32_t + { + // 12 inputs + // [num_input_logits, vocab_size_padded] + LOGITS = 0, + // [batch_size, max_decoding_tokens, max_path_len] + PATHS, + // [1] + NUM_VALID_LOGITS, + // [1] + USE_DYNAMIC_TREE, + // [1] + DYNAMIC_TREE_MAX_TOPK, + + // [batch_size, max_decoding_draft_tokens] + INPUT_DRAFT_TOKEN_IDS, + // [batch_size] + INPUT_DRAFT_LENS, + + // [batch_size, max_decoding_draft_tokens] + INPUT_PREV_SCORES, + + // [batch_size, max_decoding_draft_tokens] + INPUT_CURRENT_EXPAND_INDICES, + + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + INPUT_ALL_LAYERS_SCORES, + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + INPUT_ALL_LAYERS_DRAFT_TOKEN_IDS, + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + INPUT_ALL_LAYERS_DRAFT_TOKEN_IDS_PREDECESSOR + }; + + enum class OutputIdxEntry : int32_t + { + // 8 outputs + // [batch_size, max_decoding_draft_tokens] + OUTPUT_DRAFT_TOKEN_IDS = 0, + // [batch_size] + OUTPUT_DRAFT_LENS, + + // [batch_size, max_decoding_tokens, max_path_len] + OUTPUT_PATHS, + + // [batch_size, max_decoding_draft_tokens] + OUTPUT_CURRENT_SCORES, + + // [batch_size, max_decoding_draft_tokens] + OUTPUT_NEXT_EXPAND_INDICES, + + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + OUTPUT_ALL_LAYERS_SCORES, + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + OUTPUT_ALL_LAYERS_DRAFT_TOKEN_IDS, + // [batch_size, num_eagle_layers, max_decoding_draft_tokens x max_decoding_draft_tokens] + OUTPUT_ALL_LAYERS_DRAFT_TOKEN_IDS_PREDECESSOR + }; + + int32_t getIdx(InputIdxEntry idx) const + { + return static_cast<int32_t>(idx); + } + + int32_t getIdx(OutputIdxEntry idx) const + { + return static_cast<int32_t>(idx); + } + +private: + template <typename T> + size_t getWorkspaceSizeType(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept; + + template <typename T> + void enqueueType(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept; + + template <typename T> + void doTopKSampling(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept; + +private: + nvinfer1::DataType mDtype; // Logit datatype + int32_t mLayerIdx{-1}; // Index of eagle layer + int32_t mNumEagleLayers{-1}; // Number of eagle layers + bool mTopKSampling; // Use TopK sampling or multinomial sampling +}; + +class EagleDecodeDraftTokensPluginCreator : public BaseCreator +{ +public: + EagleDecodeDraftTokensPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/eaglePlugin/eaglePrepareDrafterInputsPlugin.cpp b/cpp/tensorrt_llm/plugins/eaglePlugin/eaglePrepareDrafterInputsPlugin.cpp new file mode 100644 index 000000000000..2cd8c695e296 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/eaglePlugin/eaglePrepareDrafterInputsPlugin.cpp @@ -0,0 +1,548 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "eaglePrepareDrafterInputsPlugin.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/dataType.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/kernels/speculativeDecoding/eagleDecodingKernels.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iTensor.h" + +using namespace nvinfer1; +using tensorrt_llm::plugins::EaglePrepareDrafterInputsPluginCreator; +using tensorrt_llm::plugins::EaglePrepareDrafterInputsPlugin; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::kernels::speculative_decoding; +using namespace tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; + +static char const* EAGLE_PREPARE_DRAFTER_INPUTS_PLUGIN_VERSION{"1"}; +static char const* EAGLE_PREPARE_DRAFTER_INPUTS_PLUGIN_NAME{"EaglePrepareDrafterInputs"}; +PluginFieldCollection EaglePrepareDrafterInputsPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> EaglePrepareDrafterInputsPluginCreator::mPluginAttributes; + +EaglePrepareDrafterInputsPlugin::EaglePrepareDrafterInputsPlugin( + int32_t layerIdx, int32_t numLayers, int32_t maxNonLeavesPerLayer) + : mLayerIdx(layerIdx) + , mNumLayers(numLayers) + , mMaxNonLeavesPerLayer(maxNonLeavesPerLayer) +{ +} + +void EaglePrepareDrafterInputsPlugin::initFieldsToSerialize() +{ + mDataToSerialize.clear(); + mDataToSerialize.emplace_back(PluginField("layer_idx", &mLayerIdx, PluginFieldType::kINT32, 1)); + mDataToSerialize.emplace_back(PluginField("num_layers", &mNumLayers, PluginFieldType::kINT32, 1)); + mDataToSerialize.emplace_back( + PluginField("max_non_leaves_per_layer", &mMaxNonLeavesPerLayer, PluginFieldType::kINT32, 1)); + mFCToSerialize.nbFields = mDataToSerialize.size(); + mFCToSerialize.fields = mDataToSerialize.data(); +} + +nvinfer1::IPluginCapability* EaglePrepareDrafterInputsPlugin::getCapabilityInterface( + nvinfer1::PluginCapabilityType type) noexcept +{ + try + { + if (type == nvinfer1::PluginCapabilityType::kBUILD) + { + return static_cast<nvinfer1::IPluginV3OneBuild*>(this); + } + if (type == nvinfer1::PluginCapabilityType::kRUNTIME) + { + return static_cast<nvinfer1::IPluginV3OneRuntime*>(this); + } + TLLM_CHECK(type == nvinfer1::PluginCapabilityType::kCORE); + return static_cast<nvinfer1::IPluginV3OneCore*>(this); + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +// IPluginV3 methods +nvinfer1::IPluginV3* EaglePrepareDrafterInputsPlugin::clone() noexcept +{ + auto clone = std::make_unique<EaglePrepareDrafterInputsPlugin>(*this); + clone->initFieldsToSerialize(); + return clone.release(); +} + +// IPluginV3OneCore methods +char const* EaglePrepareDrafterInputsPlugin::getPluginName() const noexcept +{ + return EAGLE_PREPARE_DRAFTER_INPUTS_PLUGIN_NAME; +} + +char const* EaglePrepareDrafterInputsPlugin::getPluginVersion() const noexcept +{ + return EAGLE_PREPARE_DRAFTER_INPUTS_PLUGIN_VERSION; +} + +char const* EaglePrepareDrafterInputsPlugin::getPluginNamespace() const noexcept +{ + return tensorrt_llm::plugins::api::kDefaultNamespace; +} + +// IPluginV3OneBuild methods +int32_t EaglePrepareDrafterInputsPlugin::getNbOutputs() const noexcept +{ + return 11; +} + +int32_t EaglePrepareDrafterInputsPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept +{ + return 0; +} + +bool EaglePrepareDrafterInputsPlugin::supportsFormatCombination( + int32_t pos, nvinfer1::DynamicPluginTensorDesc const* inOut, int32_t nbInputs, int32_t nbOutputs) noexcept +{ + return (inOut[pos].desc.type == nvinfer1::DataType::kINT32) && (inOut[pos].desc.format == TensorFormat::kLINEAR); +} + +int32_t EaglePrepareDrafterInputsPlugin::getOutputDataTypes(nvinfer1::DataType* outputTypes, int32_t nbOutputs, + nvinfer1::DataType const* inputTypes, int32_t nbInputs) const noexcept +{ + outputTypes[0] = nvinfer1::DataType::kINT32; + outputTypes[1] = nvinfer1::DataType::kINT32; + outputTypes[2] = nvinfer1::DataType::kINT32; + outputTypes[3] = nvinfer1::DataType::kINT32; + outputTypes[4] = nvinfer1::DataType::kINT32; + outputTypes[5] = nvinfer1::DataType::kINT32; + outputTypes[6] = nvinfer1::DataType::kINT32; + outputTypes[7] = nvinfer1::DataType::kINT32; + outputTypes[8] = nvinfer1::DataType::kINT32; + outputTypes[9] = nvinfer1::DataType::kINT32; + outputTypes[10] = nvinfer1::DataType::kINT32; + outputTypes[11] = nvinfer1::DataType::kINT32; + return 0; +} + +int32_t EaglePrepareDrafterInputsPlugin::getOutputShapes(nvinfer1::DimsExprs const* inputs, int32_t nbInputs, + nvinfer1::DimsExprs const* shapeInputs, int32_t nbShapeInputs, nvinfer1::DimsExprs* outputs, int32_t nbOutputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + TLLM_CHECK(nbOutputs == 11); + TLLM_CHECK(nbInputs == 15); + TLLM_CHECK(nbShapeInputs == 0); + auto const numTokens = inputs[getIdx(InputIdxEntry::INPUT_IDS)].d[0]; + auto const batchSizeExpr = inputs[getIdx(InputIdxEntry::PREV_DRAFT_PATHS)].d[0]; + auto const numGenRequestsExpr = inputs[getIdx(InputIdxEntry::SPEC_DECODING_GENERATION_LENGTHS)].d[0]; + auto const numInputGenTokensExpr = inputs[getIdx(InputIdxEntry::INPUT_GEN_TOKENS)].d[0]; + auto const maxDecodingLenExpr = inputs[getIdx(InputIdxEntry::PREV_DRAFT_PATHS)].d[1]; + auto const maxPathLenExpr = inputs[getIdx(InputIdxEntry::PREV_DRAFT_PATHS)].d[2]; + + for (SizeType32 outputIndex = 0; outputIndex < nbOutputs; ++outputIndex) + { + if (outputIndex == getIdx(OutputIdxEntry::SEQUENCE_LENGTHS) + || outputIndex == getIdx(OutputIdxEntry::CONTEXT_LENGTHS) + || outputIndex == getIdx(OutputIdxEntry::SPEC_DECODING_GENERATION_LENGTHS)) + { + outputs[outputIndex] = inputs[getIdx(InputIdxEntry::SEQUENCE_LENGTHS)]; + } + else if (outputIndex == getIdx(OutputIdxEntry::SPEC_DECODING_PACKED_MASK)) + { + outputs[outputIndex].nbDims = 3; + outputs[outputIndex].d[0] = batchSizeExpr; + outputs[outputIndex].d[1] = maxDecodingLenExpr; + outputs[outputIndex].d[2] + = exprBuilder.operation(DimensionOperation::kCEIL_DIV, *maxDecodingLenExpr, *exprBuilder.constant(32)); + } + else if (outputIndex == getIdx(OutputIdxEntry::SPEC_DECODING_POSITION_OFFSETS)) + { + outputs[outputIndex].nbDims = 2; + outputs[outputIndex].d[0] = batchSizeExpr; + outputs[outputIndex].d[1] = maxDecodingLenExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::OUTPUT_IDS) + || outputIndex == getIdx(OutputIdxEntry::HIDDEN_STATES_INDICES) + || (mLayerIdx == 0 && outputIndex == getIdx(OutputIdxEntry::POSITION_IDS))) + { + if (mLayerIdx == 0) + { + // We have at most numGenRequests * (mNumLayers + 1) accepted tokens per step for gen requests and + // input_ids - numGenTokens tokens for context requests. + auto numOutputGenTokensExpr = exprBuilder.operation( + DimensionOperation::kPROD, *numGenRequestsExpr, *exprBuilder.constant(mNumLayers + 1)); + auto numInputCtxTokensExpr + = exprBuilder.operation(DimensionOperation::kSUB, *numTokens, *numInputGenTokensExpr); + outputs[outputIndex].nbDims = 1; + outputs[outputIndex].d[0] = exprBuilder.operation(DimensionOperation::kMAX, *exprBuilder.constant(1), + *exprBuilder.operation(DimensionOperation::kSUM, *numOutputGenTokensExpr, *numInputCtxTokensExpr)); + } + else + { + // At most we have mMaxNonLeavesPerLayer non-leaves at this layer. + // And in total we pass all non-leaves + all their preceding nodes. + // batchSize * mMaxNonLeavesPerLayer * layerIdx + outputs[outputIndex].nbDims = 1; + outputs[outputIndex].d[0] = exprBuilder.operation(DimensionOperation::kPROD, + *exprBuilder.operation(DimensionOperation::kPROD, *exprBuilder.constant(mLayerIdx), + *exprBuilder.constant(mMaxNonLeavesPerLayer)), + *batchSizeExpr); + } + } + else if (mLayerIdx > 0 && outputIndex == getIdx(OutputIdxEntry::POSITION_IDS)) + { + outputs[outputIndex].nbDims = 1; + outputs[outputIndex].d[0] = batchSizeExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::LAST_TOKEN_INDICES)) + { + outputs[outputIndex].nbDims = 1; + outputs[outputIndex].d[0] = exprBuilder.operation( + DimensionOperation::kPROD, *exprBuilder.constant(mMaxNonLeavesPerLayer), *batchSizeExpr); + } + else if (outputIndex == getIdx(OutputIdxEntry::NUM_LAST_TOKEN_INDICES)) + { + outputs[outputIndex].nbDims = 1; + outputs[outputIndex].d[0] = exprBuilder.constant(1); + } + else if (outputIndex == getIdx(OutputIdxEntry::HIDDEN_SIZE_BATCH_LEVEL_STARTS)) + { + // batchSize * (maxPathLen - 1) + 1 + outputs[outputIndex].nbDims = 1; + outputs[outputIndex].d[0] = exprBuilder.operation(DimensionOperation::kSUM, *exprBuilder.constant(1), + *exprBuilder.operation(DimensionOperation::kPROD, *batchSizeExpr, + *exprBuilder.operation(DimensionOperation::kSUB, *maxPathLenExpr, *exprBuilder.constant(1)))); + } + } + return 0; +} + +int32_t EaglePrepareDrafterInputsPlugin::onShapeChange(nvinfer1::PluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::PluginTensorDesc const* out, int32_t nbOutputs) noexcept +{ + return 0; +} + +nvinfer1::IPluginV3* EaglePrepareDrafterInputsPlugin::attachToContext( + nvinfer1::IPluginResourceContext* context) noexcept +{ + return clone(); +} + +PluginFieldCollection const* EaglePrepareDrafterInputsPlugin::getFieldsToSerialize() noexcept +{ + return &mFCToSerialize; +} + +size_t EaglePrepareDrafterInputsPlugin::getWorkspaceSize(nvinfer1::DynamicPluginTensorDesc const* inputs, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + size_t workspaceSize{0}; + + auto const batchSize = inputs[getIdx(InputIdxEntry::NEXT_DRAFT_PATHS)].max.d[0]; + auto const maxDecodingTokens = inputs[getIdx(InputIdxEntry::NEXT_DRAFT_PATHS)].max.d[1]; + + if (mLayerIdx > 0) + { + SizeType32 constexpr NUM_BUFFERS{9}; + size_t workspaces[NUM_BUFFERS]; + workspaces[0] = batchSize * maxDecodingTokens * sizeof(int8_t); // isLeafMask + workspaces[1] = batchSize * maxDecodingTokens * sizeof(SizeType32); // selectedDraftIndices + workspaces[2] = batchSize * maxDecodingTokens * sizeof(SizeType32); // selectedDraftPosOffsets + workspaces[3] = batchSize * sizeof(SizeType32); // numSelectedDraftIndices + workspaces[4] = batchSize * maxDecodingTokens * maxDecodingTokens * sizeof(int8_t); // selectedMasks + workspaces[5] = (batchSize + 1) * sizeof(SizeType32); // cumSumGenerationLengths + workspaces[6] = batchSize * maxDecodingTokens * sizeof(SizeType32); // nonLeavesInLevelOffsets + workspaces[7] = batchSize * maxDecodingTokens * sizeof(SizeType32); // parentNonLeafInLevelOffset + workspaces[8] = 1 * sizeof(SizeType32); // maxGenerationLength + workspaceSize = tc::calculateTotalWorkspaceSize(workspaces, NUM_BUFFERS); + } + + return workspaceSize; +} + +void EaglePrepareDrafterInputsPlugin::prepareCtxEagleNetData(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const batchSize = inputDesc[getIdx(InputIdxEntry::SEQUENCE_LENGTHS)].dims.d[0]; + + auto const numTokens = inputDesc[getIdx(InputIdxEntry::INPUT_IDS)].dims.d[0]; + auto const numGenRequests = inputDesc[getIdx(InputIdxEntry::SPEC_DECODING_GENERATION_LENGTHS)].dims.d[0]; + auto const numInputGenTokens = inputDesc[getIdx(InputIdxEntry::INPUT_GEN_TOKENS)].dims.d[0]; + + auto const maxPathLen = inputDesc[getIdx(InputIdxEntry::ACCEPTED_TOKENS)].dims.d[1]; + auto const maxDecodingTokens = inputDesc[getIdx(InputIdxEntry::NEXT_DRAFT_PATHS)].dims.d[1]; + + auto eagleNetSequenceLengths = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::SEQUENCE_LENGTHS)]); + auto eagleNetContextLengths = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::CONTEXT_LENGTHS)]); + auto outputIds = reinterpret_cast<TokenIdType*>(outputs[getIdx(OutputIdxEntry::OUTPUT_IDS)]); + auto positionIds = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::POSITION_IDS)]); + auto hiddenStatesIndices = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::HIDDEN_STATES_INDICES)]); + auto lastTokenIndices = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::LAST_TOKEN_INDICES)]); + auto numLastTokenIndices = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::NUM_LAST_TOKEN_INDICES)]); + auto hiddenSizeBatchLevelStarts + = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::HIDDEN_SIZE_BATCH_LEVEL_STARTS)]); + + auto inputIds = reinterpret_cast<TokenIdType const*>(inputs[getIdx(InputIdxEntry::INPUT_IDS)]); + auto chunkedContextNextTokens + = reinterpret_cast<TokenIdType const*>(inputs[getIdx(InputIdxEntry::CHUNKED_CONTEXT_NEXT_TOKENS)]); + auto baseNetSequenceLengths = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::SEQUENCE_LENGTHS)]); + auto baseNetContextLengths = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::CONTEXT_LENGTHS)]); + auto acceptedTokens = reinterpret_cast<TokenIdType const*>(inputs[getIdx(InputIdxEntry::ACCEPTED_TOKENS)]); + auto acceptedLens = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::ACCEPTED_LENS)]); + auto prevDraftLens = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::PREV_DRAFT_LENS)]); + auto prevPaths = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::PREV_DRAFT_PATHS)]); + auto bestPathIds = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::ACCEPTED_PATHS)]); + + auto const numOutputTokens = (numTokens - numInputGenTokens) + (numGenRequests * (mNumLayers + 1)); + cudaMemsetAsync(positionIds, 0, numOutputTokens * sizeof(SizeType32), stream); + cudaMemsetAsync(hiddenStatesIndices, 0, numOutputTokens * sizeof(SizeType32), stream); + + invokePrepareCtxEagleNetInputs(eagleNetSequenceLengths, eagleNetContextLengths, outputIds, positionIds, + hiddenStatesIndices, lastTokenIndices, numLastTokenIndices, hiddenSizeBatchLevelStarts, inputIds, + chunkedContextNextTokens, baseNetSequenceLengths, baseNetContextLengths, acceptedTokens, acceptedLens, + prevDraftLens, prevPaths, bestPathIds, batchSize, maxPathLen, maxDecodingTokens, mMaxNonLeavesPerLayer, stream); + + sync_check_cuda_error(stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void EaglePrepareDrafterInputsPlugin::prepareGenEagleNetData(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const batchSize = inputDesc[getIdx(InputIdxEntry::SEQUENCE_LENGTHS)].dims.d[0]; + auto const maxDecodingTokens = inputDesc[getIdx(InputIdxEntry::NEXT_DRAFT_PATHS)].dims.d[1]; + auto const maxPathLen = inputDesc[getIdx(InputIdxEntry::NEXT_DRAFT_PATHS)].dims.d[2]; + + auto eagleNetSequenceLengths = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::SEQUENCE_LENGTHS)]); + auto eagleNetContextLengths = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::CONTEXT_LENGTHS)]); + auto outputIds = reinterpret_cast<TokenIdType*>(outputs[getIdx(OutputIdxEntry::OUTPUT_IDS)]); + auto positionIds = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::POSITION_IDS)]); + auto specDecodingGenLengths + = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::SPEC_DECODING_GENERATION_LENGTHS)]); + auto specDecodingPositionOffsets + = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::SPEC_DECODING_POSITION_OFFSETS)]); + auto specDecodingPackedMasks + = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::SPEC_DECODING_PACKED_MASK)]); + auto hiddenStatesIndices = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::HIDDEN_STATES_INDICES)]); + auto lastTokenIndices = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::LAST_TOKEN_INDICES)]); + auto numLastTokenIndices = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::NUM_LAST_TOKEN_INDICES)]); + auto outputHiddenSizeBatchStartsPerLevel + = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::HIDDEN_SIZE_BATCH_LEVEL_STARTS)]); + + auto eagleNet0SequenceLengths + = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::SEQUENCE_LENGTHS)]); + auto eagleNet0ContextLength = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::CONTEXT_LENGTHS)]); + auto nextDraftPaths = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::NEXT_DRAFT_PATHS)]); + auto nextDraftIds = reinterpret_cast<TokenIdType const*>(inputs[getIdx(InputIdxEntry::NEXT_DRAFT_TOKENS)]); + auto inputHiddenSizeBatchStartsPerLevel + = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::HIDDEN_SIZE_BATCH_LEVEL_STARTS)]); + + int8_t* workspaceBytePtr = reinterpret_cast<int8_t*>(workspace); + size_t offset{0}; + + int8_t* isLeafMask = reinterpret_cast<int8_t*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(int8_t))); + TokenIdType* selectedDraftIndices = reinterpret_cast<TokenIdType*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(TokenIdType))); + SizeType32* selectedDraftPosOffsets = reinterpret_cast<SizeType32*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(SizeType32))); + SizeType32* numSelectedDraftIndices + = reinterpret_cast<SizeType32*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(SizeType32))); + bool* selectedMasks = reinterpret_cast<bool*>(tc::nextWorkspacePtr( + workspaceBytePtr, offset, batchSize * maxDecodingTokens * maxDecodingTokens * sizeof(int8_t))); + SizeType32* cumSumGenerationLengths = reinterpret_cast<SizeType32*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, (batchSize + 1) * sizeof(SizeType32))); + SizeType32* nonLeavesInLevelOffsets = reinterpret_cast<SizeType32*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(SizeType32))); + SizeType32* parentNonLeafInLevelOffset = reinterpret_cast<SizeType32*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(SizeType32))); + SizeType32* maxGenerationLength + = reinterpret_cast<SizeType32*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, 1 * sizeof(SizeType32))); + + cudaMemsetAsync(hiddenStatesIndices, 0, batchSize * mMaxNonLeavesPerLayer * mLayerIdx * sizeof(SizeType32), stream); + cudaMemsetAsync(selectedMasks, 0, batchSize * maxDecodingTokens * maxDecodingTokens * sizeof(int8_t), stream); + // Prefill mask setting all to leaves. + cudaMemsetAsync(isLeafMask, 1, batchSize * maxDecodingTokens * sizeof(int8_t), stream); + + PrepareGenEagleNetInputsParams params; + params.nextSequenceLengths = eagleNetSequenceLengths; + params.nextContextLengths = eagleNetContextLengths; + params.outputIds = outputIds; + params.positionIds = positionIds; + params.specDecodingGenLengths = specDecodingGenLengths; + params.specDecodingPositionOffsets = specDecodingPositionOffsets; + params.specDecodingPackedMasks = specDecodingPackedMasks; + params.hiddenStatesIndices = hiddenStatesIndices; + params.lastTokenIndices = lastTokenIndices; + params.numLastTokenIndices = numLastTokenIndices; + params.outputHiddenSizeBatchStartsPerLevel = outputHiddenSizeBatchStartsPerLevel; + + // tmp data + params.isLeafMask = isLeafMask; + params.selectedDraftIndices = selectedDraftIndices; + params.selectedDraftPosOffsets = selectedDraftPosOffsets; + params.numSelectedDraftIndices = numSelectedDraftIndices; + params.selectedMasks = selectedMasks; + params.cumSumGenerationLengths = cumSumGenerationLengths; + params.maxGenerationLength = maxGenerationLength; + params.nonLeavesInLevelOffsets = nonLeavesInLevelOffsets; + params.parentNonLeafInLevelOffset = parentNonLeafInLevelOffset; + + params.nextDraftIds = nextDraftIds; + params.eagleNet0SequenceLengths = eagleNet0SequenceLengths; + params.prevContextLengths = eagleNet0ContextLength; + params.nextPaths = nextDraftPaths; + params.inputHiddenSizeBatchStartsPerLevel = inputHiddenSizeBatchStartsPerLevel; + params.levelIdx = mLayerIdx; + params.batchSize = batchSize; + params.maxPathLen = maxPathLen; + params.maxDecodingTokens = maxDecodingTokens; + params.maxNonLeavesPerLayer = mMaxNonLeavesPerLayer; + params.stream = stream; + + params.checkParams(); + + invokePrepareGenEagleNetInputs(params); + + sync_check_cuda_error(stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +int EaglePrepareDrafterInputsPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // First EagleNet instance (EagleNet0) is always chunked context attn, + // where we process either context tokens or newly accepted tokens and append them to EagleNet KV cache. + + // For all following EagleNetX (X > 0) instances there is need for masked spec decoding attn. + // Ideally with mask for context. + // Let's say we have prompt ABCD and two variants of tokens spec decoding tokens E and F + // predicted by EagleNet0. If we draw full attn mask, it becomes: + // |A|B|C|D|E|F + // E|1|1|1|1|1|0 + // F|1|1|1|1|0|1 + // + // In the next step we predict token G from ABCDE branch and token H from ABCDF branch -- like beam search. + // And we'd need spec decoding mask that includes kv cache: + // |A|B|C|D|E|F|G|H + // G|1|1|1|1|1|0|1|0 + // H|1|1|1|1|0|1|0|1 + // + // But TRT-LLM does not support such mask for now. We can only provide + // |G|H + // G|1|0 + // H|0|1 + // , which is wrong mask. + // + // For now we WAR this by passing EFGH for the EagleNet1 with right mask + // and using only G and H logits for sampling, but that's redundant compute: + // |E|F|G|H + // E|1|0|0|0 + // F|0|1|0|0 + // G|1|0|1|0 + // H|0|1|0|1 + + if (mLayerIdx == 0) + { + prepareCtxEagleNetData(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else + { + prepareGenEagleNetData(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + + return 0; +} + +/////////////// + +EaglePrepareDrafterInputsPluginCreator::EaglePrepareDrafterInputsPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("layer_idx", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("num_layers", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("max_non_leaves_per_layer", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* EaglePrepareDrafterInputsPluginCreator::getPluginName() const noexcept +{ + return EAGLE_PREPARE_DRAFTER_INPUTS_PLUGIN_NAME; +} + +char const* EaglePrepareDrafterInputsPluginCreator::getPluginVersion() const noexcept +{ + return EAGLE_PREPARE_DRAFTER_INPUTS_PLUGIN_VERSION; +} + +PluginFieldCollection const* EaglePrepareDrafterInputsPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +nvinfer1::IPluginV3* EaglePrepareDrafterInputsPluginCreator::createPlugin( + char const* name, nvinfer1::PluginFieldCollection const* fc, nvinfer1::TensorRTPhase phase) noexcept +{ + try + { + int32_t layerIdx{0}; + int32_t numLayers{0}; + int32_t maxNonLeavesPerLayer{0}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fc->fields[i].name; + if (!strcmp(attrName, "layer_idx")) + { + TLLM_CHECK(fc->fields[i].type == PluginFieldType::kINT32); + layerIdx = *static_cast<int32_t const*>(fc->fields[i].data); + } + else if (!strcmp(attrName, "num_layers")) + { + TLLM_CHECK(fc->fields[i].type == PluginFieldType::kINT32); + numLayers = *static_cast<int32_t const*>(fc->fields[i].data); + } + else if (!strcmp(attrName, "max_non_leaves_per_layer")) + { + TLLM_CHECK(fc->fields[i].type == PluginFieldType::kINT32); + maxNonLeavesPerLayer = *static_cast<int32_t const*>(fc->fields[i].data); + } + } + return new EaglePrepareDrafterInputsPlugin(layerIdx, numLayers, maxNonLeavesPerLayer); + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +char const* EaglePrepareDrafterInputsPluginCreator::getPluginNamespace() const noexcept +{ + return tensorrt_llm::plugins::api::kDefaultNamespace; +} diff --git a/cpp/tensorrt_llm/plugins/eaglePlugin/eaglePrepareDrafterInputsPlugin.h b/cpp/tensorrt_llm/plugins/eaglePlugin/eaglePrepareDrafterInputsPlugin.h new file mode 100644 index 000000000000..0059c46f6c8d --- /dev/null +++ b/cpp/tensorrt_llm/plugins/eaglePlugin/eaglePrepareDrafterInputsPlugin.h @@ -0,0 +1,186 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class EaglePrepareDrafterInputsPlugin : public nvinfer1::IPluginV3, + public nvinfer1::IPluginV3OneCore, + public nvinfer1::IPluginV3OneBuild, + public nvinfer1::IPluginV3OneRuntime +{ +public: + EaglePrepareDrafterInputsPlugin(EaglePrepareDrafterInputsPlugin const& p) = default; + + EaglePrepareDrafterInputsPlugin(int32_t layerIdx, int32_t numLayers, int32_t maxNonLeavesPerLayer); + + nvinfer1::IPluginV3* clone() noexcept override; + + nvinfer1::IPluginCapability* getCapabilityInterface(nvinfer1::PluginCapabilityType type) noexcept override; + + void initFieldsToSerialize(); + + char const* getPluginName() const noexcept override; + char const* getPluginVersion() const noexcept override; + char const* getPluginNamespace() const noexcept override; + + int32_t getNbOutputs() const noexcept override; + + bool supportsFormatCombination( + int pos, nvinfer1::DynamicPluginTensorDesc const* inOut, int32_t nbInputs, int32_t nbOutputs) noexcept override; + int32_t configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int32_t nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept override; + + int32_t getOutputDataTypes(nvinfer1::DataType* outputTypes, int32_t nbOutputs, nvinfer1::DataType const* inputTypes, + int32_t nbInputs) const noexcept override; + + int32_t getOutputShapes(nvinfer1::DimsExprs const* inputs, int32_t nbInputs, nvinfer1::DimsExprs const* shapeInputs, + int32_t nbShapeInputs, nvinfer1::DimsExprs* outputs, int32_t nbOutputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + + int32_t onShapeChange(nvinfer1::PluginTensorDesc const* in, int32_t nbInputs, nvinfer1::PluginTensorDesc const* out, + int32_t nbOutputs) noexcept override; + + nvinfer1::IPluginV3* attachToContext(nvinfer1::IPluginResourceContext* context) noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldsToSerialize() noexcept override; + + size_t getWorkspaceSize(nvinfer1::DynamicPluginTensorDesc const* inputs, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + +private: + enum class InputIdxEntry : int32_t + { + //! [batch_size] + SEQUENCE_LENGTHS = 0, + //! [batch_size] + CONTEXT_LENGTHS, + //! [num_tokens] + INPUT_IDS, + //! [batch_size] + CHUNKED_CONTEXT_NEXT_TOKENS, + //! [batch_size, max_path_len] + ACCEPTED_TOKENS, + //! [batch_size] + ACCEPTED_LENS, + //! [batch_size] + ACCEPTED_PATHS, + //! [batch_size, max_decoding_draft_tokens] + NEXT_DRAFT_TOKENS, + //! [batch_size] + NEXT_DRAFT_LENS, + //! [batch_size, max_decoding_tokens, max_path_len] + NEXT_DRAFT_PATHS, + //! [batch_size] + PREV_DRAFT_LENS, + //! [batch_size, max_decoding_tokens, max_path_len] + PREV_DRAFT_PATHS, + //! [(max_path_len - 1) * batch_size + 1] + HIDDEN_SIZE_BATCH_LEVEL_STARTS, + //! [num_gen_tokens] + INPUT_GEN_TOKENS, + //! [num_gen_requests] + SPEC_DECODING_GENERATION_LENGTHS, + }; + + enum class OutputIdxEntry : int32_t + { + //! [batch_size] + SEQUENCE_LENGTHS = 0, + //! [batch_size] + CONTEXT_LENGTHS, + //! [batch_size] + SPEC_DECODING_GENERATION_LENGTHS, + //! [batch_size, max_decoding_tokens] + SPEC_DECODING_POSITION_OFFSETS, + //! [batchSize, maxDecodingTokens, ceil(maxDecodingTokens / 32)] + SPEC_DECODING_PACKED_MASK, + //! [batchSize * mMaxNonLeavesPerLayer * layerIdx] for layerIdx > 0 + //! [num_tokens - numGenTokens + numGenRequests * (mNumLayers + 1)] for layerIdx == 0 + OUTPUT_IDS, + //! [batchSize] for layerIdx > 0 + //! [num_tokens - numGenTokens + numGenRequests * (mNumLayers + 1)] for layerIdx == 0 + POSITION_IDS, + //! [batchSize * mMaxNonLeavesPerLayer * layerIdx] for layerIdx > 0 + //! [num_tokens - numGenTokens + numGenRequests * (mNumLayers + 1)] for layerIdx == 0 + HIDDEN_STATES_INDICES, + //! [batchSize * mMaxNonLeavesPerLayer] + LAST_TOKEN_INDICES, + //! [1] + NUM_LAST_TOKEN_INDICES, + //! [(max_path_len - 1) * batch_size + 1] + HIDDEN_SIZE_BATCH_LEVEL_STARTS, + }; + + int32_t getIdx(InputIdxEntry idx) const + { + return static_cast<int32_t>(idx); + } + + int32_t getIdx(OutputIdxEntry idx) const + { + return static_cast<int32_t>(idx); + } + +private: + void prepareCtxEagleNetData(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept; + + void prepareGenEagleNetData(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept; + +private: + int32_t mLayerIdx{0}; + int32_t mNumLayers{0}; + int32_t mMaxNonLeavesPerLayer{0}; + std::vector<nvinfer1::PluginField> mDataToSerialize; + nvinfer1::PluginFieldCollection mFCToSerialize; +}; + +class EaglePrepareDrafterInputsPluginCreator : public nvinfer1::IPluginCreatorV3One +{ +public: + EaglePrepareDrafterInputsPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + char const* getPluginNamespace() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV3* createPlugin( + char const* name, nvinfer1::PluginFieldCollection const* fc, nvinfer1::TensorRTPhase phase) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/eaglePlugin/eagleSampleAndAcceptDraftTokensPlugin.cpp b/cpp/tensorrt_llm/plugins/eaglePlugin/eagleSampleAndAcceptDraftTokensPlugin.cpp new file mode 100644 index 000000000000..5fb30f583712 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/eaglePlugin/eagleSampleAndAcceptDraftTokensPlugin.cpp @@ -0,0 +1,565 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "eagleSampleAndAcceptDraftTokensPlugin.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/dataType.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/kernels/samplingTopKKernels.h" +#include "tensorrt_llm/kernels/speculativeDecoding/common.h" +#include "tensorrt_llm/kernels/speculativeDecoding/eagleDecodingKernels.h" +#include "tensorrt_llm/kernels/speculativeDecoding/medusaDecodingKernels.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iTensor.h" + +using namespace nvinfer1; +using tensorrt_llm::plugins::EagleSampleAndAcceptDraftTokensPluginCreator; +using tensorrt_llm::plugins::EagleSampleAndAcceptDraftTokensPlugin; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::kernels::speculative_decoding; +using namespace tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; + +static char const* EAGLE_SAMPLE_AND_ACCEPT_DRAFT_TOKENS_PLUGIN_VERSION{"1"}; +static char const* EAGLE_SAMPLE_AND_ACCEPT_DRAFT_TOKENS_PLUGIN_NAME{"EagleSampleAndAcceptDraftTokens"}; +PluginFieldCollection EagleSampleAndAcceptDraftTokensPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> EagleSampleAndAcceptDraftTokensPluginCreator::mPluginAttributes; + +EagleSampleAndAcceptDraftTokensPlugin::EagleSampleAndAcceptDraftTokensPlugin(nvinfer1::DataType type) + : mDtype(type) +{ +} + +// Parameterized constructor +EagleSampleAndAcceptDraftTokensPlugin::EagleSampleAndAcceptDraftTokensPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mDtype); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* EagleSampleAndAcceptDraftTokensPlugin::clone() const noexcept +{ + auto* plugin = new EagleSampleAndAcceptDraftTokensPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs EagleSampleAndAcceptDraftTokensPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + TLLM_CHECK(nbInputs == 10); + TLLM_CHECK(outputIndex < 7); + auto const batchSizeExpr = inputs[getIdx(InputIdxEntry::PATHS)].d[0]; + auto const maxDecodingDraftTokensExpr = inputs[getIdx(InputIdxEntry::DRAFT_TOKEN_IDS)].d[1]; + auto const maxDecodingTokensExpr = inputs[getIdx(InputIdxEntry::PATHS)].d[1]; + auto const maxPathLenExpr = inputs[getIdx(InputIdxEntry::PATHS)].d[2]; + + nvinfer1::DimsExprs ret; + if (outputIndex == getIdx(OutputIdxEntry::ACCEPTED_TOKENS)) + { + ret.nbDims = 2; + ret.d[0] = batchSizeExpr; + ret.d[1] = maxPathLenExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::ACCEPTED_LENS)) + { + ret.nbDims = 1; + ret.d[0] = batchSizeExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::BEST_ACCEPTED_PATHS)) + { + ret.nbDims = 1; + ret.d[0] = batchSizeExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::NEXT_DRAFT_TOKEN_IDS)) + { + ret.nbDims = 2; + ret.d[0] = batchSizeExpr; + ret.d[1] = maxDecodingDraftTokensExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::NEXT_DRAFT_LENS)) + { + ret.nbDims = 1; + ret.d[0] = batchSizeExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::NEXT_DRAFT_PATHS)) + { + ret.nbDims = 3; + ret.d[0] = batchSizeExpr; + ret.d[1] = maxDecodingTokensExpr; + ret.d[2] = maxPathLenExpr; + } + else if (outputIndex == getIdx(OutputIdxEntry::HIDDEN_SIZE_BATCH_LEVEL_STARTS)) + { + ret.nbDims = 1; + ret.d[0] = exprBuilder.operation(DimensionOperation::kSUM, *exprBuilder.constant(1), + *exprBuilder.operation(DimensionOperation::kPROD, + *exprBuilder.operation(DimensionOperation::kSUB, *maxPathLenExpr, *exprBuilder.constant(1)), + *batchSizeExpr)); + } + return ret; +} + +bool EagleSampleAndAcceptDraftTokensPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + if (pos == getIdx(InputIdxEntry::LOGITS)) // logits + { + return (inOut[pos].type == mDtype) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == getIdx(InputIdxEntry::TEMPERATURE) || pos == getIdx(InputIdxEntry::RAND_VALIDATION) + || pos == getIdx(InputIdxEntry::POSTERIOR_ALPHA) + || pos == getIdx(InputIdxEntry::POSTERIOR_THRESHOLD)) // temperature, rand_validation + { + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else // everything else + { + return (inOut[pos].type == nvinfer1::DataType::kINT32) && (inOut[pos].format == TensorFormat::kLINEAR); + } +} + +void EagleSampleAndAcceptDraftTokensPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +template <typename T> +size_t EagleSampleAndAcceptDraftTokensPlugin::getWorkspaceSizeType(nvinfer1::PluginTensorDesc const* inputs, + int nbInputs, nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + size_t workspaceSize{0}; + + auto const vocabSizePadded = inputs[getIdx(InputIdxEntry::LOGITS)].dims.d[1]; + auto const batchSize = inputs[getIdx(InputIdxEntry::PATHS)].dims.d[0]; + auto const maxDecodingTokens = inputs[getIdx(InputIdxEntry::PATHS)].dims.d[1]; + + // Greedy sampling + // Top1 sampling workspace + auto const greedySamplingWorkspaceSize + = getTopKWorkspaceSize<T>(batchSize, maxDecodingTokens, /* maxTopK */ 1, vocabSizePadded); + + // Multinomial sampling + auto const typicalSamplingWorkspaceSize + = getTypicalAcceptanceWorkspaceSize<T>(batchSize, maxDecodingTokens, vocabSizePadded); + + auto const primarySamplingWorkspaceSize = std::max(greedySamplingWorkspaceSize, typicalSamplingWorkspaceSize); + + // Target output ids + auto const targetOutputIdsSize = batchSize * maxDecodingTokens * sizeof(TokenIdType); + // Logits ptrs + auto const logitsPtrsSize = batchSize * maxDecodingTokens * sizeof(T*); + SizeType32 constexpr NUM_BUFFERS{4}; + size_t workspaces[NUM_BUFFERS]; + workspaces[0] = targetOutputIdsSize; + workspaces[1] = primarySamplingWorkspaceSize; + workspaces[2] = logitsPtrsSize; + workspaces[3] = batchSize * sizeof(SizeType32); + workspaceSize = tc::calculateTotalWorkspaceSize(workspaces, NUM_BUFFERS); + + return workspaceSize; +} + +size_t EagleSampleAndAcceptDraftTokensPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + auto const logitsType = inputs[getIdx(InputIdxEntry::LOGITS)].type; + if (logitsType == nvinfer1::DataType::kFLOAT) + { + return getWorkspaceSizeType<float>(inputs, nbInputs, outputs, nbOutputs); + } + else if (logitsType == nvinfer1::DataType::kHALF) + { + return getWorkspaceSizeType<__half>(inputs, nbInputs, outputs, nbOutputs); + } + else + { + TLLM_CHECK_WITH_INFO(false, "Unsupported logits type"); + } + return 0; +} + +template <typename T> +void EagleSampleAndAcceptDraftTokensPlugin::samplePrimeHeadTokens(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // auto const maxNumTokens = inputDesc[getIdx(InputIdxEntry::LOGITS)].dims.d[0]; + auto const vocabSizePadded = inputDesc[getIdx(InputIdxEntry::LOGITS)].dims.d[1]; + auto const batchSize = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[0]; + auto const maxDecodingTokens = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[1]; + + auto logits = static_cast<T const*>(inputs[getIdx(InputIdxEntry::LOGITS)]); + auto prevDraftLens = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::DRAFT_LENS)]); + + int8_t* workspaceBytePtr = reinterpret_cast<int8_t*>(workspace); + size_t offset{0}; + + auto const samplingWorkspaceSize + = getTopKWorkspaceSize<T>(batchSize, maxDecodingTokens, /* maxTopK */ 1, vocabSizePadded); + + TokenIdType* outputIds = reinterpret_cast<TokenIdType*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(TokenIdType))); + void* workspaceSampling + = reinterpret_cast<void*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, samplingWorkspaceSize)); + T const** logitsPtrs = reinterpret_cast<T const**>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(T*))); + SizeType32* decodingTokens + = reinterpret_cast<SizeType32*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(SizeType32))); + + // Assemble pointers to logits + invokeAssembleTargetLogitsOffsets( + logitsPtrs, decodingTokens, logits, prevDraftLens, batchSize, maxDecodingTokens, vocabSizePadded, stream); + + sync_check_cuda_error(stream); + + TopKSamplingKernelParams<T> params; + params.logProbsPtrs = logitsPtrs; + params.outputIds = outputIds; + params.workspace = workspaceSampling; + params.maxTopK = 1; + params.batchSize = batchSize; + params.maxBatchSize = batchSize; + params.tokensPerStep = decodingTokens; + params.maxTokensPerStep = maxDecodingTokens; + params.maxSeqLen = maxDecodingTokens; + params.vocabSizePadded = vocabSizePadded; + + invokeBatchTopKSampling(params, stream); + + sync_check_cuda_error(stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +template <typename T> +void EagleSampleAndAcceptDraftTokensPlugin::doTypicalAcceptance(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // auto const maxNumTokens = inputDesc[getIdx(InputIdxEntry::LOGITS)].dims.d[0]; + auto const vocabSizePadded = inputDesc[getIdx(InputIdxEntry::LOGITS)].dims.d[1]; + + auto const batchSize = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[0]; + auto const maxDecodingTokens = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[1]; + // auto const maxPathLen = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[2]; + // auto const maxDraftPathLen = maxPathLen - 1; + + auto logits = static_cast<T const*>(inputs[getIdx(InputIdxEntry::LOGITS)]); + auto prevDraftLens = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::DRAFT_LENS)]); + + int8_t* workspaceBytePtr = reinterpret_cast<int8_t*>(workspace); + size_t offset{0}; + + // Multinomial sampling + auto const primarySamplingWorkspaceSize + = getTypicalAcceptanceWorkspaceSize<T>(batchSize, maxDecodingTokens, vocabSizePadded); + + TokenIdType* outputIds = reinterpret_cast<TokenIdType*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(TokenIdType))); + void* workspaceSampling + = reinterpret_cast<void*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, primarySamplingWorkspaceSize)); + T** logitsPtrs = reinterpret_cast<T**>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(T*))); + SizeType32* decodingTokens + = reinterpret_cast<SizeType32*>(tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * sizeof(SizeType32))); + + // Assemble pointers to logits + invokeAssembleTargetLogitsOffsets(const_cast<T const**>(logitsPtrs), decodingTokens, logits, prevDraftLens, + batchSize, maxDecodingTokens, vocabSizePadded, stream); + + sync_check_cuda_error(stream); + + TypicalAcceptanceSampling<T> params; + params.logitsPtrs = logitsPtrs; + params.generationLengths = decodingTokens; + params.temperatures = reinterpret_cast<float const*>(inputs[getIdx(InputIdxEntry::TEMPERATURE)]); + params.posteriorThresholds = reinterpret_cast<float const*>(inputs[getIdx(InputIdxEntry::POSTERIOR_THRESHOLD)]); + params.posteriorAlphas = reinterpret_cast<float const*>(inputs[getIdx(InputIdxEntry::POSTERIOR_ALPHA)]); + params.outputIds = outputIds; + params.workspace = reinterpret_cast<int8_t*>(workspaceSampling); + params.randomVals = reinterpret_cast<float const*>(inputs[getIdx(InputIdxEntry::RAND_VALIDATION)]); + + params.batchSize = batchSize; + params.maxBatchSize = batchSize; + params.maxDecodingTokens = maxDecodingTokens; + params.vocabSize = vocabSizePadded; + + if (mSmCnt <= 0) + { + auto const deviceId = tensorrt_llm::common::getDevice(); + cudaDeviceProp prop{}; + TLLM_CUDA_CHECK(cudaGetDeviceProperties(&prop, deviceId)); + mSmCnt = prop.multiProcessorCount; + } + params.smCnt = mSmCnt; + + params.checkParams(); + + typicalAcceptanceSampling(params, stream); + + sync_check_cuda_error(stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +template <typename T> +void EagleSampleAndAcceptDraftTokensPlugin::acceptDraftTokens(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + // auto const maxNumTokens = inputDesc[getIdx(InputIdxEntry::LOGITS)].dims.d[0]; + auto const vocabSizePadded = inputDesc[getIdx(InputIdxEntry::LOGITS)].dims.d[1]; + + auto const batchSize = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[0]; + auto const maxDecodingTokens = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[1]; + auto const maxPathLen = inputDesc[getIdx(InputIdxEntry::PATHS)].dims.d[2]; + auto const maxDraftPathLen = maxPathLen - 1; + + auto const useDynamicTree = *(reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::USE_DYNAMIC_TREE)])); + + int8_t* workspaceBytePtr = reinterpret_cast<int8_t*>(workspace); + size_t offset{0}; + + // auto const samplingWorkspaceSize + // = getTopKWorkspaceSize<T>(batchSize, maxDecodingTokens, /* maxTopK */ 1, vocabSizePadded); + + TokenIdType* outputIds = reinterpret_cast<TokenIdType*>( + tc::nextWorkspacePtr(workspaceBytePtr, offset, batchSize * maxDecodingTokens * sizeof(TokenIdType))); + + AcceptDraftTokensByIdsWithPathsParams<T> params; + params.outputIds = reinterpret_cast<TokenIdType*>(outputs[getIdx(OutputIdxEntry::ACCEPTED_TOKENS)]); + params.draftIds = reinterpret_cast<TokenIdType const*>(inputs[getIdx(InputIdxEntry::DRAFT_TOKEN_IDS)]); + params.targetIds = outputIds; + params.acceptedLengths = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::ACCEPTED_LENS)]); + params.paths = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::PATHS)]); + params.bestPathIds = reinterpret_cast<SizeType32*>(outputs[getIdx(OutputIdxEntry::BEST_ACCEPTED_PATHS)]); + params.batchSize = batchSize; + params.maxBatchSize = batchSize; + params.vocabSize = vocabSizePadded; + params.maxSeqLen = maxPathLen; + params.maxDraftPathLen = maxDraftPathLen; + params.maxDecodingTokens = maxDecodingTokens; + params.stream = stream; + + params.checkParams(); + + acceptDraftTokensByIdsWithPaths(params); + + if (useDynamicTree) + { + // For Eagle-2, after verification and acceptance, the original path becomes useless. + // All set to '-1' + cudaMemsetAsync(outputs[getIdx(OutputIdxEntry::NEXT_DRAFT_PATHS)], -1, + batchSize * maxDecodingTokens * maxPathLen * sizeof(SizeType32), stream); + } + else + { + // For Eagle-1 + // Copy input paths to the output + cudaMemcpyAsync(outputs[getIdx(OutputIdxEntry::NEXT_DRAFT_PATHS)], inputs[getIdx(InputIdxEntry::PATHS)], + batchSize * maxDecodingTokens * maxPathLen * sizeof(SizeType32), cudaMemcpyDeviceToDevice, stream); + } + + sync_check_cuda_error(stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +template <typename T> +void EagleSampleAndAcceptDraftTokensPlugin::enqueueType(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const greedySampling = reinterpret_cast<SizeType32 const*>(inputs[getIdx(InputIdxEntry::GREEDY_SAMPLING)])[0]; + // TODO split batch into greedy and non-greedy and execute both paths + if (greedySampling) + { + // Sample all main head tokens with Top-1. + samplePrimeHeadTokens<T>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else + { + // Typical sampling for typical acceptance. + doTypicalAcceptance<T>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + + // Accept tokens based on token ids, write the best path and best token id. + acceptDraftTokens<T>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +int EagleSampleAndAcceptDraftTokensPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + auto const logitsType = inputDesc[getIdx(InputIdxEntry::LOGITS)].type; + if (logitsType == nvinfer1::DataType::kFLOAT) + { + enqueueType<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (logitsType == nvinfer1::DataType::kHALF) + { + enqueueType<__half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else + { + TLLM_CHECK_WITH_INFO(false, "Unsupported logits type"); + } + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType EagleSampleAndAcceptDraftTokensPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index < 7); + // input 1 is draft tokens now of int32 type. All outputs are int32_t as well. + return inputTypes[getIdx(InputIdxEntry::DRAFT_TOKEN_IDS)]; +} + +// IPluginV2 Methods + +char const* EagleSampleAndAcceptDraftTokensPlugin::getPluginType() const noexcept +{ + return EAGLE_SAMPLE_AND_ACCEPT_DRAFT_TOKENS_PLUGIN_NAME; +} + +char const* EagleSampleAndAcceptDraftTokensPlugin::getPluginVersion() const noexcept +{ + return EAGLE_SAMPLE_AND_ACCEPT_DRAFT_TOKENS_PLUGIN_VERSION; +} + +int EagleSampleAndAcceptDraftTokensPlugin::getNbOutputs() const noexcept +{ + return 7; +} + +int EagleSampleAndAcceptDraftTokensPlugin::initialize() noexcept +{ + return 0; +} + +void EagleSampleAndAcceptDraftTokensPlugin::terminate() noexcept {} + +size_t EagleSampleAndAcceptDraftTokensPlugin::getSerializationSize() const noexcept +{ + return sizeof(mDtype); +} + +void EagleSampleAndAcceptDraftTokensPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mDtype); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void EagleSampleAndAcceptDraftTokensPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +EagleSampleAndAcceptDraftTokensPluginCreator::EagleSampleAndAcceptDraftTokensPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* EagleSampleAndAcceptDraftTokensPluginCreator::getPluginName() const noexcept +{ + return EAGLE_SAMPLE_AND_ACCEPT_DRAFT_TOKENS_PLUGIN_NAME; +} + +char const* EagleSampleAndAcceptDraftTokensPluginCreator::getPluginVersion() const noexcept +{ + return EAGLE_SAMPLE_AND_ACCEPT_DRAFT_TOKENS_PLUGIN_VERSION; +} + +PluginFieldCollection const* EagleSampleAndAcceptDraftTokensPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* EagleSampleAndAcceptDraftTokensPluginCreator::createPlugin( + char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + + try + { + auto* obj = new EagleSampleAndAcceptDraftTokensPlugin(type); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* EagleSampleAndAcceptDraftTokensPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call EagleSampleAndAcceptDraftTokensPlugin::destroy() + try + { + auto* obj = new EagleSampleAndAcceptDraftTokensPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/eaglePlugin/eagleSampleAndAcceptDraftTokensPlugin.h b/cpp/tensorrt_llm/plugins/eaglePlugin/eagleSampleAndAcceptDraftTokensPlugin.h new file mode 100644 index 000000000000..3b14bab83170 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/eaglePlugin/eagleSampleAndAcceptDraftTokensPlugin.h @@ -0,0 +1,167 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" + +#include <cassert> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class EagleSampleAndAcceptDraftTokensPlugin : public BasePlugin +{ +public: + EagleSampleAndAcceptDraftTokensPlugin(nvinfer1::DataType type); + + EagleSampleAndAcceptDraftTokensPlugin(void const* data, size_t length); + + ~EagleSampleAndAcceptDraftTokensPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + enum class InputIdxEntry : int32_t + { + //! [num_tokens, vocab_size_padded] + LOGITS = 0, + //! [batch_size, max_decoding_draft_tokens] + DRAFT_TOKEN_IDS, + //! [batch_size] + DRAFT_LENS, + //! [batch_size] + TEMPERATURE, + //! [batch_size, max_decoding_tokens] + RAND_VALIDATION, + //! [batch_size] + POSTERIOR_ALPHA, + //! [batch_size] + POSTERIOR_THRESHOLD, + //! [batch_size, max_decoding_tokens, max_path_len] + PATHS, + //! [1] + GREEDY_SAMPLING, + //! [1] + USE_DYNAMIC_TREE + }; + + enum class OutputIdxEntry : int32_t + { + //! [batch_size, max_path_len] + ACCEPTED_TOKENS = 0, + //! [batch_size] + ACCEPTED_LENS, + //! [batch_size] + BEST_ACCEPTED_PATHS, + //! [batch_size, max_decoding_draft_tokens] + NEXT_DRAFT_TOKEN_IDS, + //! [batch_size] + NEXT_DRAFT_LENS, + //! [batch_size, max_decoding_tokens, max_path_len] + NEXT_DRAFT_PATHS, + //! [max_draft_path_len * batch_size] + HIDDEN_SIZE_BATCH_LEVEL_STARTS, + }; + + int32_t getIdx(InputIdxEntry idx) const + { + return static_cast<int32_t>(idx); + } + + int32_t getIdx(OutputIdxEntry idx) const + { + return static_cast<int32_t>(idx); + } + +private: + template <typename T> + size_t getWorkspaceSizeType(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept; + + template <typename T> + void samplePrimeHeadTokens(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept; + + template <typename T> + void doTypicalAcceptance(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept; + + template <typename T> + void acceptDraftTokens(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept; + + template <typename T> + void enqueueType(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept; + +private: + nvinfer1::DataType mDtype; + int32_t mSmCnt{0}; +}; + +class EagleSampleAndAcceptDraftTokensPluginCreator : public BaseCreator +{ +public: + EagleSampleAndAcceptDraftTokensPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/exports.def b/cpp/tensorrt_llm/plugins/exports.def new file mode 100644 index 000000000000..5d4ac9e3e793 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/exports.def @@ -0,0 +1,19 @@ +; SPDX-FileCopyrightText: Copyright (c) 1993-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +; SPDX-License-Identifier: Apache-2.0 +; +; Licensed under the Apache License, Version 2.0 (the "License"); +; you may not use this file except in compliance with the License. +; You may obtain a copy of the License at +; +; http://www.apache.org/licenses/LICENSE-2.0 +; +; Unless required by applicable law or agreed to in writing, software +; distributed under the License is distributed on an "AS IS" BASIS, +; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +; See the License for the specific language governing permissions and +; limitations under the License. + +LIBRARY nvinfer_plugin_tensorrt_llm +EXPORTS +getPluginRegistry +initLibNvInferPlugins diff --git a/cpp/tensorrt_llm/plugins/exports.map b/cpp/tensorrt_llm/plugins/exports.map new file mode 100644 index 000000000000..c6c949775079 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/exports.map @@ -0,0 +1,34 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* Hides all symbols except those specified in the global section */ +{ + global: + initTrtLlmPlugins; + setLoggerFinder; + getPluginCreators; + getCreators; + extern "C++" { + nvinfer1::IPluginCreator::*; + nvinfer1::IPluginV2Ext::*; + nvinfer1::IPluginV2IOExt::*; + nvinfer1::PluginRegistrar*; + tensorrt_llm::plugins::api::*; + tensorrt_llm::plugins::*; + }; + local: *; +}; diff --git a/cpp/tensorrt_llm/plugins/fp4GemmPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/fp4GemmPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fp4GemmPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/fp4GemmPlugin/fp4GemmPlugin.cpp b/cpp/tensorrt_llm/plugins/fp4GemmPlugin/fp4GemmPlugin.cpp new file mode 100644 index 000000000000..05f06ae38feb --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fp4GemmPlugin/fp4GemmPlugin.cpp @@ -0,0 +1,434 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include <numeric> + +#include "fp4GemmPlugin.h" +#include "tensorrt_llm/common/assert.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::Fp4GemmPluginCreator; +using tensorrt_llm::plugins::Fp4GemmPlugin; +using tensorrt_llm::plugins::Fp4GemmPluginProfiler; +#if defined(USING_OSS_CUTLASS_FP4_GEMM) +using namespace tensorrt_llm::kernels::cutlass_kernels; +#else +using namespace tensorrt_llm::kernels::internal_cutlass_kernels; +#endif + +constexpr nvinfer1::DataType FP4_DTYPE = nvinfer1::DataType::kFP4; +constexpr nvinfer1::DataType FP8_DTYPE = nvinfer1::DataType::kFP8; + +static char const* FP4_GEMM_PLUGIN_VERSION{"1"}; +static char const* FP4_GEMM_PLUGIN_NAME{"Fp4Gemm"}; +PluginFieldCollection Fp4GemmPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> Fp4GemmPluginCreator::mPluginAttributes; + +void Fp4GemmPluginProfiler::runTactic( + int m, int n, int k, Fp4GemmPluginProfiler::Config const& tactic, char* workspace, cudaStream_t const& stream) +{ + // Workspace size required by gemm runner + // NB: this function will throw exception when selected tactic exceeds SMEM, which is then + // caught by gemmPluginProfiler and it will register this tactic as invalid + size_t wsSizeRunner = mRunner->getWorkspaceSize(m, n, k, /* batch_count */ 1); + + // Workspace size required by profiling + size_t wsByteOffset = 0; + int8_t* wsBytePointer = reinterpret_cast<int8_t*>(workspace); + void* aTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, (m * k) / 2)); + void* bTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, (n * k) / 2)); + void* dTmp = reinterpret_cast<void*>( + nextWorkspacePtr(wsBytePointer, wsByteOffset, m * n * (mType == nvinfer1::DataType::kFLOAT ? 4u : 2u))); + // SF M/N is padded along 128 and K is padded along 4. + int vector_size = 16; + int sf_round_m = ((m + 127) / 128) * 128; + int sf_round_n = ((n + 127) / 128) * 128; + int sf_round_k = ((k / vector_size + 3) / 4) * 4; + float* a_sf = reinterpret_cast<float*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, sf_round_m * sf_round_k)); + float* b_sf = reinterpret_cast<float*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, sf_round_n * sf_round_k)); + float* global_sf = reinterpret_cast<float*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, sizeof(float))); + char* workspaceTmp = reinterpret_cast<char*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, wsSizeRunner)); + + // Run profiling + mRunner->gemm(dTmp, aTmp, bTmp, a_sf, b_sf, global_sf, m, n, k, /* batch_count */ 1, tactic, workspaceTmp, + wsSizeRunner, stream); + sync_check_cuda_error(stream); +} + +void Fp4GemmPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + size_t vector_size = 16; + size_t sf_round_m = ((maxM + 127) / 128) * 128; + size_t sf_round_n = ((n + 127) / 128) * 128; + size_t sf_round_k = ((k / vector_size + 3) / 4) * 4; + std::vector<size_t> workspaces = { + (size_t) (maxM * k / 2), // A + (size_t) (n * k / 2), // B + maxM * n * (mType == nvinfer1::DataType::kFLOAT ? 4u : 2u), // D + (size_t) (sf_round_m * sf_round_k), // A_SF + (size_t) (sf_round_n * sf_round_k), // B_SF + sizeof(float), // Global_SF + mRunner->getWorkspaceSize(maxM, n, k, /* batch_count */ 1) // workspace + }; + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<Fp4GemmPluginProfiler::Config> Fp4GemmPluginProfiler::getTactics(int m, int n, int k) const +{ + return mRunner->getConfigs(); +} + +Fp4GemmPlugin::Fp4GemmPlugin( + int sfVecSize, nvinfer1::DataType OutputType, Fp4GemmPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) + , mSfVecSize(sfVecSize) + , mOutputType(OutputType) +{ + init(OutputType); +} + +Fp4GemmPlugin::Fp4GemmPlugin(void const* data, size_t length, Fp4GemmPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mSfVecSize); + read(d, mOutputType); + read(d, mDims); + + init(mOutputType); + mPluginProfiler->deserialize(d, mDims, mGemmId); + + TLLM_CHECK(d == a + length); +} + +void Fp4GemmPlugin::init(nvinfer1::DataType type) +{ + TLLM_CHECK_WITH_INFO((getSMVersion() >= 100), "FP4 Gemm not supported before Blackwell"); + TLLM_CHECK_WITH_INFO( + (mOutputType == DataType::kBF16) || (mOutputType == DataType::kFLOAT) || (mOutputType == DataType::kHALF), + "Only support float, half, bfloat16, got %d.", (int) mOutputType); + mOutputType = type; + if (mOutputType == nvinfer1::DataType::kHALF) + { + mGemmRunner = std::make_shared<CutlassFp4GemmRunner<half>>(); + } + else if (mOutputType == nvinfer1::DataType::kFLOAT) + { + mGemmRunner = std::make_shared<CutlassFp4GemmRunner<float>>(); + } +#ifdef ENABLE_BF16 + else if (mOutputType == nvinfer1::DataType::kBF16) + { + mGemmRunner = std::make_shared<CutlassFp4GemmRunner<__nv_bfloat16>>(); + } +#endif + + mGemmId = GemmIdCore(mDims.n, mDims.k, mOutputType); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* Fp4GemmPlugin::clone() const noexcept +{ + auto* plugin = new Fp4GemmPlugin(*this); + return plugin; +} + +nvinfer1::DimsExprs Fp4GemmPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + TLLM_CHECK_WITH_INFO(outputIndex == 0, "Only support one output"); + auto const& dimsInput = inputs[getInputTensorIdx()]; + auto const& dimsWeights = inputs[getWeightsTensorIdx()]; + TLLM_CHECK_WITH_INFO(dimsInput.nbDims >= 2 && dimsWeights.nbDims == 2, "Fp4GemmPlugin input dim=%d, weights dim=%d", + dimsInput.nbDims, dimsWeights.nbDims); + nvinfer1::DimsExprs ret; + if (outputIndex == 0) + { + ret.nbDims = dimsInput.nbDims; + for (int i = 0; i < dimsInput.nbDims - 1; ++i) + { + ret.d[i] = dimsInput.d[i]; + } + ret.d[dimsInput.nbDims - 1] = dimsWeights.d[0]; + } + else + { + TLLM_CHECK_WITH_INFO(outputIndex == 0, "output fp4 not supported now."); + ret.nbDims = 1; + auto vecCount = dimsInput.d[0]; + int numDim = dimsInput.nbDims; + for (int idx = 1; idx < numDim - 1; ++idx) + { + vecCount = exprBuilder.operation(nvinfer1::DimensionOperation::kPROD, *vecCount, *dimsInput.d[idx]); + } + auto constant128 = exprBuilder.constant(128); + auto alignedRowCount = exprBuilder.operation(nvinfer1::DimensionOperation::kCEIL_DIV, *vecCount, *constant128); + alignedRowCount = exprBuilder.operation(nvinfer1::DimensionOperation::kPROD, *alignedRowCount, *constant128); + auto constant4 = exprBuilder.constant(4); + auto constantSFSize = exprBuilder.constant(mSfVecSize); + auto sfColumn + = exprBuilder.operation(nvinfer1::DimensionOperation::kCEIL_DIV, *dimsInput.d[numDim - 1], *constantSFSize); + auto alignedColumnCount = exprBuilder.operation(nvinfer1::DimensionOperation::kCEIL_DIV, *sfColumn, *constant4); + alignedColumnCount + = exprBuilder.operation(nvinfer1::DimensionOperation::kPROD, *alignedColumnCount, *constant4); + auto totalSize + = exprBuilder.operation(nvinfer1::DimensionOperation::kPROD, *alignedColumnCount, *alignedRowCount); + ret.d[0] = totalSize; + } + return ret; +} + +bool Fp4GemmPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + if (inOut[pos].format != TensorFormat::kLINEAR) + { + return false; + } + if (pos == getInputTensorIdx()) + { + return (inOut[pos].type == FP4_DTYPE); + } + else if (pos == getWeightsTensorIdx()) + { + return (inOut[pos].type == FP4_DTYPE); + } + else if (pos == getInputSFTensorIdx() || pos == getWeightsSFTensorIdx()) + { + return (inOut[pos].type == FP8_DTYPE); + } + else if (pos == getGlobalSFTensorIdx()) + { + return (inOut[pos].type == DataType::kFLOAT); + } + else if (pos == nbInputs) + { + // Output + return (inOut[pos].type == DataType::kFLOAT || inOut[pos].type == DataType::kBF16 + || inOut[pos].type == DataType::kHALF); + } + return false; +} + +void Fp4GemmPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + int const maxN = in[2].max.d[0]; + int const minK = in[0].min.d[in[0].min.nbDims - 1]; + int const minN = in[2].min.d[0]; + + TLLM_CHECK_WITH_INFO(minN == maxN, "Variable out channels is not allowed"); + TLLM_CHECK_WITH_INFO(minK == maxK, "Variable in channels is not allowed"); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, maxN, maxK}; + } + mGemmId = {maxN, maxK, mOutputType}; + m_workspaceMaxSize = mGemmRunner->getWorkspaceSize(maxM, maxN, maxK, /* batch_count */ 1); +} + +size_t Fp4GemmPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return m_workspaceMaxSize; +} + +int Fp4GemmPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + // inputs + // 0. input_tensor [num_tokens, dim] + // 1. input_block_scale [num_tokens, dim / SFVecSize] (padded) + // 2. weights_tensor [out_dim, dim] + // 3. weights_block_scale [out_dim, dim / SFVecSize] (padded) + // 4. alpha (global scaling factor) [1] + // outputs + // 0. output_tensor [num_tokens, out_dim] + int64_t m = 1; + for (int i = 0; i < inputDesc[getInputTensorIdx()].dims.nbDims - 1; ++i) + { + m *= inputDesc[getInputTensorIdx()].dims.d[i]; + } + int const n = inputDesc[getWeightsTensorIdx()].dims.d[0]; + int const k = inputDesc[getWeightsTensorIdx()].dims.d[1]; + TLLM_CHECK_WITH_INFO(k % 32 == 0, "K dim should be aligned to 16 Bytes"); + int N_align = mOutputType == nvinfer1::DataType::kFLOAT ? 4u : 8u; + TLLM_CHECK_WITH_INFO(n % N_align == 0, "N dim should be aligned to 16 Bytes"); + size_t const wsSize = mGemmRunner->getWorkspaceSize(m, n, k, /* batch_count */ 1); + auto const bestTactic = mPluginProfiler->getBestConfig(m, mGemmId); + TLLM_CHECK_WITH_INFO(bestTactic, "No valid FP4 GEMM tactic"); + if (m >= 1) + { + mGemmRunner->gemm(outputs[0], inputs[0], inputs[2], inputs[1], inputs[3], + reinterpret_cast<float const*>(inputs[4]), m, n, k, /* batch_count */ 1, *bestTactic, + reinterpret_cast<char*>(workspace), wsSize, stream); + } + sync_check_cuda_error(stream); + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType Fp4GemmPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK_WITH_INFO(index == 0, "Only support one output"); + return mOutputType; +} + +// IPluginV2 Methods + +char const* Fp4GemmPlugin::getPluginType() const noexcept +{ + return FP4_GEMM_PLUGIN_NAME; +} + +char const* Fp4GemmPlugin::getPluginVersion() const noexcept +{ + return FP4_GEMM_PLUGIN_VERSION; +} + +int Fp4GemmPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int Fp4GemmPlugin::initialize() noexcept +{ + configGemm(); + return 0; +} + +void Fp4GemmPlugin::terminate() noexcept {} + +size_t Fp4GemmPlugin::getSerializationSize() const noexcept +{ + return sizeof(mSfVecSize) + // mSfVecSize + sizeof(nvinfer1::DataType) + // dtype + sizeof(mDims) + // Dimensions + mPluginProfiler->getSerializationSize(mGemmId); // selected tactics container size +} + +void Fp4GemmPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mSfVecSize); + write(d, mOutputType); + write(d, mDims); + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void Fp4GemmPlugin::destroy() noexcept +{ + delete this; +} + +void Fp4GemmPlugin::configGemm() +{ + mPluginProfiler->profileTactics(mGemmRunner, mOutputType, mDims, mGemmId); +} + +/////////////// + +Fp4GemmPluginCreator::Fp4GemmPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("sv_vec_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("output_type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* Fp4GemmPluginCreator::getPluginName() const noexcept +{ + return FP4_GEMM_PLUGIN_NAME; +} + +char const* Fp4GemmPluginCreator::getPluginVersion() const noexcept +{ + return FP4_GEMM_PLUGIN_VERSION; +} + +PluginFieldCollection const* Fp4GemmPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* Fp4GemmPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + TLLM_CHECK(fc->nbFields == 2); + int sf_vec_size{}; + nvinfer1::DataType output_type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "sf_vec_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + sf_vec_size = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "output_type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + output_type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + try + { + // Fp4GemmPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + auto pluginProfiler = mGemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false); + auto* obj = new Fp4GemmPlugin(sf_vec_size, output_type, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* Fp4GemmPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call CumsumLastDimPlugin::destroy() + try + { + // Create plugin profiler with private tactics map which is read from the serialized engine + auto pluginProfiler = mGemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true); + auto* obj = new Fp4GemmPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/fp4GemmPlugin/fp4GemmPlugin.h b/cpp/tensorrt_llm/plugins/fp4GemmPlugin/fp4GemmPlugin.h new file mode 100644 index 000000000000..9947e849d84f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fp4GemmPlugin/fp4GemmPlugin.h @@ -0,0 +1,162 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#if defined(USING_OSS_CUTLASS_FP4_GEMM) +#include "tensorrt_llm/kernels/cutlass_kernels/include/fp4_gemm.h" +#else +#include "fp4_gemm.h" +#endif + +#include <cassert> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +#if defined(USING_OSS_CUTLASS_FP4_GEMM) +using Fp4GemmRunnerPtr = std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassFp4GemmRunnerInterface>; +#else +using Fp4GemmRunnerPtr + = std::shared_ptr<tensorrt_llm::kernels::internal_cutlass_kernels::CutlassFp4GemmRunnerInterface>; +#endif + +class Fp4GemmPluginProfiler : public GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + Fp4GemmRunnerPtr, GemmIdCore, GemmIdCoreHash> +{ +public: + using Config = tensorrt_llm::cutlass_extensions::CutlassGemmConfig; + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + std::vector<Config> getTactics(int m, int n, int k) const override; + +private: + tensorrt_llm::common::QuantMode mQuantMode; +}; + +class Fp4GemmPlugin : public BasePlugin +{ +public: + using PluginProfilerPtr = std::shared_ptr<Fp4GemmPluginProfiler>; + + Fp4GemmPlugin() = delete; + + Fp4GemmPlugin(int sfVecSize, nvinfer1::DataType OutputType, PluginProfilerPtr const& pluginProfiler); + + Fp4GemmPlugin(void const* data, size_t length, PluginProfilerPtr const& pluginProfiler); + + ~Fp4GemmPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + using IndexType = std::int32_t; + + IndexType getInputTensorIdx() const + { + return 0; + }; + + IndexType getInputSFTensorIdx() const + { + return 1; + }; + + IndexType getWeightsTensorIdx() const + { + return 2; + }; + + IndexType getWeightsSFTensorIdx() const + { + return 3; + }; + + IndexType getGlobalSFTensorIdx() const + { + return 4; + } + + void init(nvinfer1::DataType type); + void configGemm(); + + Fp4GemmRunnerPtr mGemmRunner; + PluginProfilerPtr mPluginProfiler; + + int mSfVecSize; + nvinfer1::DataType mOutputType; + size_t m_workspaceMaxSize; + GemmDims mDims{}; + GemmIdCore mGemmId{}; +}; + +class Fp4GemmPluginCreator : public BaseCreator +{ +public: + Fp4GemmPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<Fp4GemmPluginProfiler> mGemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/CMakeLists.txt new file mode 100644 index 000000000000..3b714a3928fb --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp *.cu) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/fp8RowwiseGemmPlugin.cpp b/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/fp8RowwiseGemmPlugin.cpp new file mode 100644 index 000000000000..84963df50a21 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/fp8RowwiseGemmPlugin.cpp @@ -0,0 +1,422 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "fp8RowwiseGemmPlugin.h" +#include "cutlass_extensions/gemm_configs.h" + +#include <NvInferRuntimeBase.h> +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels::cutlass_kernels; +using tensorrt_llm::plugins::Fp8RowwiseGemmPluginCreator; +using tensorrt_llm::plugins::Fp8RowwiseGemmPlugin; +using tensorrt_llm::plugins::Fp8RowwiseGemmPluginProfiler; + +static char const* FP8_ROWWISE_GEMM_PLUGIN_VERSION{"1"}; +static char const* FP8_ROWWISE_GEMM_PLUGIN_NAME{"Fp8RowwiseGemm"}; +PluginFieldCollection Fp8RowwiseGemmPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> Fp8RowwiseGemmPluginCreator::mPluginAttributes; + +size_t Fp8RowwiseGemmPluginProfiler::getBytePerElement(nvinfer1::DataType type) +{ + size_t bpe; + if (type == nvinfer1::DataType::kHALF || type == nvinfer1::DataType::kBF16) + { + bpe = 2; + } + else if (type == nvinfer1::DataType::kINT8 || type == nvinfer1::DataType::kFP8) + { + bpe = 1; + } + else + { + TLLM_THROW("Not recognized/implemented"); + } + return bpe; +} + +void Fp8RowwiseGemmPluginProfiler::setQuantMode(tensorrt_llm::common::QuantMode const& quantMode) +{ + mQuantMode = quantMode; +} + +void Fp8RowwiseGemmPluginProfiler::runTactic(int m, int n, int k, Fp8RowwiseGemmPluginProfiler::Config const& tactic, + char* workspace, cudaStream_t const& stream) +{ + size_t bpeIn = getBytePerElement(nvinfer1::DataType::kFP8); + size_t bpeOut = getBytePerElement(mType); + + // Workspace size required by gemm runner + // NB: this function will throw exception when selected tactic exceeds SMEM, which is then + // caught by gemmPluginProfiler and it will register this tactic as invalid + size_t wsSizeRunner = mRunner->getWorkspaceSize(m, n, k); + + // Workspace size required by profiling + size_t wsByteOffset = 0; + int8_t* wsBytePointer = reinterpret_cast<int8_t*>(workspace); + void* aTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, m * k * bpeIn)); + void* bTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, n * k * bpeIn)); + // void* cTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, n * bpeOut)); + void* dTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, m * n * bpeOut)); + float* scaleD0Tmp = reinterpret_cast<float*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, m * sizeof(float))); + float* scaleD1Tmp = reinterpret_cast<float*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, n * sizeof(float))); + char* workspaceTmp = reinterpret_cast<char*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, wsSizeRunner)); + + // Run profiling + mRunner->gemm(dTmp, aTmp, bTmp, nullptr, mQuantMode, m, n, k, scaleD0Tmp, scaleD1Tmp, tactic, workspaceTmp, + wsSizeRunner, stream); + sync_check_cuda_error(stream); +} + +int Fp8RowwiseGemmPluginProfiler::getMaxProfileM() const +{ + // Max_num_tokens are not suggested to be set larger than 16k. + return 16384; +} + +void Fp8RowwiseGemmPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + std::vector<size_t> workspaces = { + maxM * k * getBytePerElement(nvinfer1::DataType::kFP8), // A + n * k * getBytePerElement(nvinfer1::DataType::kFP8), // B + // n * getBytePerElement(mType), // C_bias + maxM * n * getBytePerElement(mType), // D + maxM * sizeof(float), // alphaRow + n * sizeof(float), // alphaCol + maxM * sizeof(float), // alphaOutput + mRunner->getWorkspaceSize(maxM, n, k) // workspace + }; + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<Fp8RowwiseGemmPluginProfiler::Config> Fp8RowwiseGemmPluginProfiler::getTactics(int m, int n, int k) const +{ + return mRunner->getConfigs(); +} + +Fp8RowwiseGemmPlugin::Fp8RowwiseGemmPlugin( + QuantMode quantMode, nvinfer1::DataType type, Fp8RowwiseGemmPlugin::PluginProfilerPtr const& pluginProfiler) + : mQuantMode(quantMode) + , mPluginProfiler(pluginProfiler) +{ + init(type); +} + +// Parameterized constructor +Fp8RowwiseGemmPlugin::Fp8RowwiseGemmPlugin( + void const* data, size_t length, Fp8RowwiseGemmPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + nvinfer1::DataType type; + unsigned int quantMode; + read(d, quantMode); + read(d, type); + read(d, mDims); + + mQuantMode = QuantMode(quantMode); + + init(type); + + mPluginProfiler->deserialize(d, mDims, mGemmId); + + TLLM_CHECK(d == a + length); +} + +void Fp8RowwiseGemmPlugin::init(nvinfer1::DataType type) +{ + mType = type; + if (mType == nvinfer1::DataType::kHALF) + { + mGemmRunner = std::make_shared<CutlassFp8RowwiseGemmRunner<half>>(); + } +#ifdef ENABLE_BF16 + else if (mType == nvinfer1::DataType::kBF16) + { + mGemmRunner = std::make_shared<CutlassFp8RowwiseGemmRunner<__nv_bfloat16>>(); + } +#endif + else + { + TLLM_THROW("Fp8 Rowwise Gemm plugin doesn't support this type now"); + } + + mPluginProfiler->setQuantMode(mQuantMode); + + mGemmId = GemmIdCore(mDims.n, mDims.k, mType); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* Fp8RowwiseGemmPlugin::clone() const noexcept +{ + auto* plugin = new Fp8RowwiseGemmPlugin(*this); + return plugin; +} + +nvinfer1::DimsExprs Fp8RowwiseGemmPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 4); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + TLLM_CHECK(nbDimsA >= 2); + DimsExprs ret; + ret.nbDims = nbDimsA; + for (int ii = 0; ii < nbDimsA - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + ret.d[nbDimsA - 1] = inputs[1].d[0]; + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool Fp8RowwiseGemmPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + switch (pos) + { + case 0: + // activation + return inOut[pos].type == nvinfer1::DataType::kFP8 && inOut[pos].format == TensorFormat::kLINEAR; + case 1: + // weights + // Weights stored in checkpoint must have fp8 type + return inOut[pos].type == nvinfer1::DataType::kFP8 && inOut[pos].format == TensorFormat::kLINEAR; + case 2: + // scales channels + case 3: + // scales tokens + return inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + case 4: + // out + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + default: + // All other format combinations are unsupported. + return false; + } +} + +void Fp8RowwiseGemmPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + int const maxN = in[1].max.d[0]; + int const minK = in[0].min.d[in[0].min.nbDims - 1]; + int const minN = in[1].min.d[0]; + + TLLM_CHECK_WITH_INFO(minN == maxN, "Variable out channels is not allowed"); + TLLM_CHECK_WITH_INFO(minK == maxK, "Variable in channels is not allowed"); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, maxN, maxK}; + } + mGemmId = {maxN, maxK, mType}; + + mWorkspaceMaxSize = mGemmRunner->getWorkspaceSize(maxM, maxN, maxK); +} + +size_t Fp8RowwiseGemmPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return mWorkspaceMaxSize; +} + +int Fp8RowwiseGemmPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // mat1 [M(*), K] + // mat2 [N, K] + // scale_tokens [M, 1] if has_per_token_scaling else [1, 1] + // scale_channels [1, N] if has_per_channel_scaling else [1, 1] + // outputs + // mat [M(*), N] + int m = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m *= inputDesc[0].dims.d[ii]; + } + int const n = inputDesc[1].dims.d[0]; + int const k = inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]; + size_t const wsSize = mGemmRunner->getWorkspaceSize(m, n, k); + + auto const bestTactic = mPluginProfiler->getBestConfig(m, mGemmId); + TLLM_CHECK_WITH_INFO(bestTactic, "No valid GEMM tactic"); + mGemmRunner->gemm(outputs[0], inputs[0], inputs[1], nullptr, mQuantMode, m, n, k, + reinterpret_cast<float const*>(inputs[2]), reinterpret_cast<float const*>(inputs[3]), *bestTactic, + reinterpret_cast<char*>(workspace), wsSize, stream); + sync_check_cuda_error(stream); + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType Fp8RowwiseGemmPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* Fp8RowwiseGemmPlugin::getPluginType() const noexcept +{ + return FP8_ROWWISE_GEMM_PLUGIN_NAME; +} + +char const* Fp8RowwiseGemmPlugin::getPluginVersion() const noexcept +{ + return FP8_ROWWISE_GEMM_PLUGIN_VERSION; +} + +int Fp8RowwiseGemmPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int Fp8RowwiseGemmPlugin::initialize() noexcept +{ + configGemm(); // gemm profiler in action + return 0; +} + +void Fp8RowwiseGemmPlugin::terminate() noexcept {} + +size_t Fp8RowwiseGemmPlugin::getSerializationSize() const noexcept +{ + return sizeof(unsigned int) + // QuantMode + sizeof(nvinfer1::DataType) + // dtype + sizeof(mDims) + // Dimensions + mPluginProfiler->getSerializationSize(mGemmId); // selected tactics container size +} + +void Fp8RowwiseGemmPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mQuantMode.value()); + write(d, mType); + write(d, mDims); + + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void Fp8RowwiseGemmPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +void Fp8RowwiseGemmPlugin::configGemm() +{ + mPluginProfiler->profileTactics(mGemmRunner, mType, mDims, mGemmId); +} + +Fp8RowwiseGemmPluginCreator::Fp8RowwiseGemmPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("has_per_channel_scaling", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("has_per_token_scaling", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* Fp8RowwiseGemmPluginCreator::getPluginName() const noexcept +{ + return FP8_ROWWISE_GEMM_PLUGIN_NAME; +} + +char const* Fp8RowwiseGemmPluginCreator::getPluginVersion() const noexcept +{ + return FP8_ROWWISE_GEMM_PLUGIN_VERSION; +} + +PluginFieldCollection const* Fp8RowwiseGemmPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* Fp8RowwiseGemmPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + TLLM_CHECK(fc->nbFields == 3); + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + try + { + // Fp8RowwiseGemmPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + auto pluginProfiler = mGemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false); + QuantMode quantMode = QuantMode{}; + auto* obj = new Fp8RowwiseGemmPlugin(quantMode, type, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* Fp8RowwiseGemmPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call Fp8RowwiseGemmPlugin::destroy() + try + { + // Create plugin profiler with private tactics map which is read from the serialized engine + auto pluginProfiler = mGemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true); + auto* obj = new Fp8RowwiseGemmPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/fp8RowwiseGemmPlugin.h b/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/fp8RowwiseGemmPlugin.h new file mode 100644 index 000000000000..36f22ad5885d --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fp8RowwiseGemmPlugin/fp8RowwiseGemmPlugin.h @@ -0,0 +1,140 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/kernels/cutlass_kernels/fp8_rowwise_gemm/fp8_rowwise_gemm.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +using Fp8RowwiseGemmRunnerPtr + = std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassFp8RowwiseGemmRunnerInterface>; + +class Fp8RowwiseGemmPluginProfiler : public GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + Fp8RowwiseGemmRunnerPtr, GemmIdCore, GemmIdCoreHash> + +{ +public: + using Config = tensorrt_llm::cutlass_extensions::CutlassGemmConfig; + + void setQuantMode(tensorrt_llm::common::QuantMode const& quantMode); + + virtual int getMaxProfileM() const override; + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + std::vector<Config> getTactics(int m, int n, int k) const override; + +private: + size_t getBytePerElement(nvinfer1::DataType type); + + tensorrt_llm::common::QuantMode mQuantMode; +}; + +class Fp8RowwiseGemmPlugin : public BasePlugin +{ +public: + using PluginProfilerPtr = std::shared_ptr<Fp8RowwiseGemmPluginProfiler>; + + Fp8RowwiseGemmPlugin() = delete; + + Fp8RowwiseGemmPlugin( + tensorrt_llm::common::QuantMode quantMode, nvinfer1::DataType type, PluginProfilerPtr const& pluginProfiler); + + Fp8RowwiseGemmPlugin(void const* data, size_t length, PluginProfilerPtr const& profiler); + + ~Fp8RowwiseGemmPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + void init(nvinfer1::DataType type); + + void configGemm(); + +private: + const std::string mLayerName; + + Fp8RowwiseGemmRunnerPtr mGemmRunner; + tensorrt_llm::common::QuantMode mQuantMode; // not configurable yet + size_t mWorkspaceMaxSize; + + GemmDims mDims{}; + GemmIdCore mGemmId{}; + + PluginProfilerPtr mPluginProfiler; + + nvinfer1::DataType mType; +}; + +class Fp8RowwiseGemmPluginCreator : public BaseCreator +{ +public: + Fp8RowwiseGemmPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<Fp8RowwiseGemmPluginProfiler> mGemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/CMakeLists.txt new file mode 100755 index 000000000000..7cc985b60b7a --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2023 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/fusedLayernormPlugin.cpp b/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/fusedLayernormPlugin.cpp new file mode 100644 index 000000000000..541afdadc4c8 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/fusedLayernormPlugin.cpp @@ -0,0 +1,388 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "fusedLayernormPlugin.h" +#include "pluginUtils.h" +#include "tensorrt_llm/common/assert.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::FusedLayernormPluginCreator; +using tensorrt_llm::plugins::FusedLayernormPlugin; + +static char const* FUSED_LAYERNORM_PLUGIN_VERSION{"1"}; +static char const* FUSED_LAYERNORM_PLUGIN_NAME{"FusedLayernorm"}; +PluginFieldCollection FusedLayernormPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> FusedLayernormPluginCreator::mPluginAttributes; + +FusedLayernormPlugin::FusedLayernormPlugin(float eps, bool needFP32Output, bool needQuantize, nvinfer1::DataType type) + : mEps(eps) + , mNeedFP32Output(needFP32Output) + , mNeedQuantize(needQuantize) + , mType(type) +{ +} + +// Parameterized constructor +FusedLayernormPlugin::FusedLayernormPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mEps); + read(d, mNeedFP32Output); + read(d, mNeedQuantize); + read(d, mType); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* FusedLayernormPlugin::clone() const noexcept +{ + auto* plugin = new FusedLayernormPlugin(mEps, mNeedFP32Output, mNeedQuantize, mType); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs FusedLayernormPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + // Dim should be the same as input hidden states + if (!mNeedQuantize) + { + return inputs[0]; + } + + if (outputIndex == 1) // un-normed output fp16 + { + return inputs[0]; + } + if (outputIndex == 0) // quantized normed output + { + // Quantized output with int64_t data type (16 FP4 values per element). + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int di = 0; di < ret.nbDims; ++di) + { + ret.d[di] = inputs[0].d[di]; + } + return ret; + } + + // Scaling Factors. + try + { + TLLM_CHECK(outputIndex == 2); + + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int di = 0; di < ret.nbDims; ++di) + { + ret.d[di] = inputs[0].d[di]; + } + // Sequence dimension or token dimension. + // Pad to multiple of 128. + auto dimM + = exprBuilder.operation(DimensionOperation::kCEIL_DIV, *ret.d[ret.nbDims - 2], *exprBuilder.constant(128)); + ret.d[ret.nbDims - 2] = exprBuilder.operation(DimensionOperation::kPROD, *dimM, *exprBuilder.constant(128)); + // Hidden size dimension. + ret.d[ret.nbDims - 1] + = exprBuilder.operation(DimensionOperation::kCEIL_DIV, *ret.d[ret.nbDims - 1], *exprBuilder.constant(16)); + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool FusedLayernormPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + int const totalPoses = 5 + 2 * static_cast<int>(mNeedQuantize); + TLLM_CHECK(0 <= pos && pos < totalPoses); + TLLM_CHECK(nbInputs == 3 + static_cast<int>(mNeedQuantize)); + if (pos < nbInputs) + { + switch (pos) + { + case 0: + case 1: + case 2: return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + case 3: return (inOut[pos].type == nvinfer1::DataType::kFLOAT); + } + } + if (pos == nbInputs) // Normed output + { + if (mNeedQuantize) + { + // fp4 quantized output -- fp4 padded tp int64 + return (inOut[pos].type == nvinfer1::DataType::kFP4) && (inOut[pos].format == TensorFormat::kLINEAR); + } + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == nbInputs + 1) // Un-normed output + { + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + // fp4 act_per_block_scale -- fp8 padded to int32 + return (inOut[pos].type == nvinfer1::DataType::kFP8) && (inOut[pos].format == TensorFormat::kLINEAR); +} + +void FusedLayernormPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t FusedLayernormPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return sizeof(WarpSpecializedCounters); +} + +int FusedLayernormPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // input [M(*), N] + // residual [M(*), N] + // weight [N, ] + // scale [1, ] - if needQuantize + // outputs + // output [M(*), N] - fp4 padded to int64 / fp16 + // un-normed output [M(*), N] - fp16 + // act_per_block_scale - fp8 padded to int32 - if needQuantize + +#define SETUP_PARAM \ + Param param; \ + int64_t m64 = 1; \ + for (int i = 0; i < inputDesc[0].dims.nbDims - 1; ++i) \ + { \ + m64 *= inputDesc[0].dims.d[i]; \ + } \ + int const m = TLLM_INT32_CAST(m64); \ + int const n = TLLM_INT32_CAST(inputDesc[2].dims.d[0]); \ + param.m = m; \ + param.n = n; \ + param.layernorm_eps = mEps; \ + param.input = const_cast<Input*>(reinterpret_cast<Input const*>(inputs[0])); \ + param.residual = const_cast<Input*>(reinterpret_cast<Input const*>(inputs[1])); \ + param.gamma = const_cast<Input*>(reinterpret_cast<Input const*>(inputs[2])); \ + if (mNeedQuantize) \ + { \ + param.sf_scale = const_cast<float*>(reinterpret_cast<float const*>(inputs[3])); \ + } \ + param.counters = reinterpret_cast<WarpSpecializedCounters*>(workspace); \ + param.stream = stream; \ + param.normed_output = reinterpret_cast<uint32_t*>(outputs[0]); \ + param.output = reinterpret_cast<Input*>(outputs[1]); \ + param.sf_out = reinterpret_cast<uint32_t*>(outputs[2]); + +#define CLEANUP_AND_INVOKE \ + TLLM_CUDA_CHECK(cudaMemsetAsync(workspace, 0, sizeof(WarpSpecializedCounters), stream)); \ + invokeWSLayerNorm(param, true, num_sms); + + int num_sms = tensorrt_llm::common::getMultiProcessorCount(); + + if (mType == DataType::kHALF) + { + using Input = half; + using Param = WarpSpecializedParam<GeneralFP4AddBiasResidualPreLayerNormParam<Input>>; + SETUP_PARAM + CLEANUP_AND_INVOKE + } +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16) + { + using Input = __nv_bfloat16; + using Param = WarpSpecializedParam<GeneralFP4AddBiasResidualPreLayerNormParam<Input>>; + SETUP_PARAM + CLEANUP_AND_INVOKE + } +#endif + else + { + TLLM_LOG_ERROR("Unsupported data type"); + return 1; + } + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType FusedLayernormPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + // assert((mNeedFP32Output && index < 3) || (!mNeedFP32Output && index < 2)); + assert((mNeedQuantize && index < 3) || (!mNeedQuantize && index < 2)); + if (index == 0) + { + // Output 0 quantized output of layernorm - fp4 padded to int64 + if (mNeedQuantize) + { + return nvinfer1::DataType::kFP4; + } + return mType; + } + else if (index == 1) + { + // Output 1 un-normed output + return mType; + } + // Output 2 act_per_block_scale - fp8 padded to int32 + return nvinfer1::DataType::kFP8; +} + +// IPluginV2 Methods + +char const* FusedLayernormPlugin::getPluginType() const noexcept +{ + return FUSED_LAYERNORM_PLUGIN_NAME; +} + +char const* FusedLayernormPlugin::getPluginVersion() const noexcept +{ + return FUSED_LAYERNORM_PLUGIN_VERSION; +} + +int FusedLayernormPlugin::getNbOutputs() const noexcept +{ + return 2 + static_cast<int>(mNeedQuantize); +} + +int FusedLayernormPlugin::initialize() noexcept +{ + return 0; +} + +void FusedLayernormPlugin::terminate() noexcept {} + +size_t FusedLayernormPlugin::getSerializationSize() const noexcept +{ + return sizeof(mEps) + sizeof(mNeedFP32Output) + sizeof(mNeedQuantize) + sizeof(mType); +} + +void FusedLayernormPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mEps); + write(d, mNeedFP32Output); + write(d, mNeedQuantize); + write(d, mType); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void FusedLayernormPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +FusedLayernormPluginCreator::FusedLayernormPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("eps", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("need_fp32_output", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("need_quantize", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* FusedLayernormPluginCreator::getPluginName() const noexcept +{ + return FUSED_LAYERNORM_PLUGIN_NAME; +} + +char const* FusedLayernormPluginCreator::getPluginVersion() const noexcept +{ + return FUSED_LAYERNORM_PLUGIN_VERSION; +} + +PluginFieldCollection const* FusedLayernormPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* FusedLayernormPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + float eps{}; + nvinfer1::DataType type{}; + bool needFP32Output{}; + bool needQuantize{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "eps")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + eps = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "need_fp32_output")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + needFP32Output = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "need_quantize")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + needQuantize = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + } + try + { + auto* obj = new FusedLayernormPlugin(eps, needFP32Output, needQuantize, type); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* FusedLayernormPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call FusedLayernormPlugin::destroy() + try + { + auto* obj = new FusedLayernormPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/fusedLayernormPlugin.h b/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/fusedLayernormPlugin.h new file mode 100755 index 000000000000..c6c899950fdc --- /dev/null +++ b/cpp/tensorrt_llm/plugins/fusedLayernormPlugin/fusedLayernormPlugin.h @@ -0,0 +1,98 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/kernels/fusedLayernormKernels/layernorm_param.h" +#include "tensorrt_llm/kernels/fusedLayernormKernels/ws_layernorm.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class FusedLayernormPlugin : public BasePlugin +{ +public: + FusedLayernormPlugin() = delete; + + FusedLayernormPlugin(float eps, bool needFP32Output, bool needQuantize, nvinfer1::DataType type); + + FusedLayernormPlugin(void const* data, size_t length); + + ~FusedLayernormPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + float mEps; + bool mNeedFP32Output; + bool mNeedQuantize; + nvinfer1::DataType mType; + + const std::string mLayerName; +}; + +class FusedLayernormPluginCreator : public BaseCreator +{ +public: + FusedLayernormPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/CMakeLists.txt new file mode 100644 index 000000000000..1d1fa98f4132 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/CMakeLists.txt @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePlugin.cpp b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePlugin.cpp new file mode 100644 index 000000000000..08ee2af55406 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePlugin.cpp @@ -0,0 +1,721 @@ +/* + * Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "gemmAllReducePlugin.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h" +#include "tensorrt_llm/plugins/common/pluginUtils.h" + +#include <unistd.h> + +static char const* GEMM_ALLREDUCE_PLUGIN_VERSION = "1"; +static char const* GEMM_ALLREDUCE_PLUGIN_NAME = "GemmAllReduce"; +template <nvinfer1::DataType T> +using CutlassType = ::tensorrt_llm::kernels::cutlass_kernels::CutlassType<T>; + +namespace tensorrt_llm::plugins +{ +template <typename K, typename V, DataType ElementA, DataType ElementB, DataType ElementD> +static std::pair<K, V> makeEntry() +{ + return {std::make_tuple(ElementA, ElementB, ElementD), + [&]() + { + using GemmTraits + = cutlass_kernels::GemmTypes<typename CutlassType<ElementA>::type, typename CutlassType<ElementB>::type, + typename CutlassType<ElementD>::type, // C, unused + typename CutlassType<ElementD>::type, + std::conditional_t<ElementA == DataType::kFP4, cutlass::float_ue4m3_t, void>, // SFA + std::conditional_t<ElementB == DataType::kFP4, cutlass::float_ue4m3_t, void>, // SFB + cutlass::layout::RowMajor, cutlass::layout::ColumnMajor, + cutlass::layout::RowMajor, // C, unused + cutlass::layout::RowMajor>; + return new cutlass_kernels::GemmAllReduceImplRunner<GemmTraits>(); + }}; +} + +template <typename K, typename V> +static std::map<K, V> getTypedInstantiators() +{ + return std::map<K, V>({makeEntry<K, V, DataType::kHALF, DataType::kHALF, DataType::kHALF>(), + makeEntry<K, V, DataType::kBF16, DataType::kBF16, DataType::kBF16>(), + makeEntry<K, V, DataType::kFP8, DataType::kFP8, DataType::kHALF>(), + makeEntry<K, V, DataType::kFP8, DataType::kFP8, DataType::kBF16>(), + makeEntry<K, V, DataType::kFP4, DataType::kFP4, DataType::kHALF>(), + makeEntry<K, V, DataType::kFP4, DataType::kFP4, DataType::kBF16>()}); +} + +//////////////////////////////////////////////////////////// +// GemmAllReducePlugin Methods +//////////////////////////////////////////////////////////// +GemmAllReducePlugin::GemmAllReducePlugin(GemmAllReducePluginOptions const& options) + : mOptions(options) + , mGemmId(GemmIdCore(options.maxProblemShape.n, options.maxProblemShape.k, options.typeD)) + , mProfiler(mGemmPluginProfileManager.createGemmPluginProfiler(/*inference=*/options.deserialize)) +{ + // construct mapping of input/output pos to argument + int argIdx = 0; + // inputs + mArgMap[argIdx++] = TensorArg::IN_ACTIVATION; + mArgMap[argIdx++] = TensorArg::IN_WEIGHT; + if (mOptions.hasSFA) + { + mArgMap[argIdx++] = TensorArg::IN_ACTIVATION_SF; + } + if (mOptions.hasSFB) + { + mArgMap[argIdx++] = TensorArg::IN_WEIGHT_SF; + } + if (mOptions.alphaIsPtr) + { + mArgMap[argIdx++] = TensorArg::IN_ALPHA; + } + mNbInputs = argIdx; + // outputs + mArgMap[argIdx++] = TensorArg::OUT_D_UC; + mArgMap[argIdx++] = TensorArg::OUT_D_MC; + mArgMap[argIdx++] = TensorArg::OUT_D_IPC; + mNbOutputs = argIdx - mNbInputs; + + // Create mapping of argument to tensor pos + for (auto const& pair : mArgMap) + { + mArgInvMap[pair.second] = pair.first; + } + + // Use map instead of huge switch case + mTypedInstantiators = getTypedInstantiators<KeyType, ValueType>(); + + auto key = std::make_tuple(mOptions.typeA, mOptions.typeB, mOptions.typeD); + + TLLM_CHECK_WITH_INFO(mTypedInstantiators.count(key) > 0, "No cutlass gemm for impl."); + mGemm = std::shared_ptr<cutlass_kernels::GemmAllReduceImplInterface>(mTypedInstantiators[key]()); +} + +void GemmAllReducePlugin::allocatePersistentWorkspace() +{ + TLLM_CHECK(mOptions.maxProblemShape.isInitialized()); + + mWorkspaceKey = "gemm_allreduce_workspace_m" + std::to_string(mOptions.maxProblemShape.maxM); + + cutlass_kernels::GemmAllReduceImplInterface::LaunchConfig smallest_tile_config + = mGemm->getSupportedLaunchConfigs()[0]; + cutlass_kernels::GemmAllReduceImplInterface::ProblemArgs args; + args.argProblemShape(mOptions.maxProblemShape.maxM, mOptions.maxProblemShape.n, mOptions.maxProblemShape.k, 1) + .argRanks(mRank, mOptions.group) + .argLaunchConfig(smallest_tile_config); + + TLLM_CHECK(mWorkspace == nullptr); + + // Wrap persistent workspace in IPluginResource type + // so that clone() can be called to allocate memory + GemmAllReducePersistentWorkspace unallocated_resource(mGemm->getPersistentWorkspace(args)); + + // Register and allocate workspace + mWorkspace = static_cast<GemmAllReducePersistentWorkspace*>( + getPluginRegistry()->acquirePluginResource(mWorkspaceKey.c_str(), &unallocated_resource)); + TLLM_CHECK(mWorkspace != nullptr); +} + +LaunchConfig GemmAllReducePlugin::getStaticHeuristicLaunchConfig(int M) const +{ + using namespace tensorrt_llm::cutlass_extensions; + // This is only applicable when we swap and transpose A & B. + // When M is small we want to select tile that best fits it to maximize MMA efficiency. + auto filterByM = [&](std::vector<LaunchConfig> candidateConfigs) + { + std::vector<LaunchConfig> result; + if (M <= 16) + { + std::copy_if(candidateConfigs.begin(), candidateConfigs.end(), std::back_inserter(result), + [](const LaunchConfig& config) + { return config.tile_shape == TileShape::TileShape_128x16x128 and config.transposed; }); + } + else if (M <= 32) + { + std::copy_if(candidateConfigs.begin(), candidateConfigs.end(), std::back_inserter(result), + [](const LaunchConfig& config) + { return config.tile_shape == TileShape::TileShape_128x32x128 and config.transposed; }); + } + else if (M <= 64) + { + std::copy_if(candidateConfigs.begin(), candidateConfigs.end(), std::back_inserter(result), + [](const LaunchConfig& config) + { return config.tile_shape == TileShape::TileShape_128x64x128 and config.transposed; }); + } + else + { + std::copy_if(candidateConfigs.begin(), candidateConfigs.end(), std::back_inserter(result), + [](const LaunchConfig& config) + { return config.tile_shape == TileShape::TileShape_128x128x128 and config.transposed; }); + } + // If result empty then use any. + if (result.empty()) + { + result = candidateConfigs; + } + return result; + }; + + auto bestLaunchConfigs = mGemm->getSupportedLaunchConfigs(); + bestLaunchConfigs = filterByM(bestLaunchConfigs); + TLLM_CHECK(!bestLaunchConfigs.empty()); + // Return first one, because who knows which is best. + return bestLaunchConfigs.front(); +} + +static GemmAllReducePluginOptions deserializeOptions(void const*& data, size_t length) +{ + char const* begin = reinterpret_cast<char const*>(data); + char const*& end = reinterpret_cast<char const*&>(data); + GemmAllReducePluginOptions options; + options.deserialize = true; + + read(end, options.typeA); + read(end, options.typeB); + read(end, options.typeD); + read(end, options.transA); + read(end, options.transB); + read(end, options.alpha); + read(end, options.maxProblemShape); + read(end, options.groupSize); + for (int i = 0; i < options.groupSize; ++i) + { + int rank = -1; + read(end, rank); + options.group.insert(rank); + } + read(end, options.hasSFA); + read(end, options.hasSFB); + read(end, options.alphaIsPtr); + + TLLM_CHECK_WITH_INFO(end == begin + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (end - begin)); + + return options; +} + +GemmAllReducePlugin::GemmAllReducePlugin(void const* data, size_t length) + : GemmAllReducePlugin(deserializeOptions(std::ref(data), length)) +{ + if (mProfiler->useProfiler()) + { + mProfiler->deserializeFromOwnFile(mGemmId, mOptions.maxProblemShape); + } +} + +////////////////////////////////// +// IPluginV2DynamicExt Methods +////////////////////////////////// +IPluginV2DynamicExt* GemmAllReducePlugin::clone() const noexcept +{ + return new GemmAllReducePlugin(*this); +} + +DimsExprs GemmAllReducePlugin::getOutputDimensions( + int outputIndex, DimsExprs const* inputs, int nbInputs, IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == mNbInputs); // number of input tensors + TLLM_CHECK(inputs[0].nbDims == inputs[1].nbDims); + TLLM_CHECK(outputIndex < getNbOutputs()); + + // List of pointers to D on each rank + if ((nbInputs + outputIndex) == TensorArg::OUT_D_IPC) + { + DimsExprs out_dims; + out_dims.nbDims = 1; + out_dims.d[0] = exprBuilder.constant(mOptions.groupSize); + return out_dims; + } + + TLLM_CHECK(mOptions.transA == false); + TLLM_CHECK(mOptions.transB == true); + + int const nbDimsA = inputs[0].nbDims; // number of dims + int const nbDimsB = inputs[1].nbDims; + + DimsExprs out_dims; + // subtract 2 -> K from each input + out_dims.nbDims = nbDimsA + nbDimsB - 2; + + if (mOptions.transA) + { + for (int i = 1; i < nbDimsA; ++i) + { + out_dims.d[i - 1] = inputs[0].d[i]; + } + } + else + { + for (int i = 0; i < nbDimsA - 1; ++i) + { + out_dims.d[i] = inputs[0].d[i]; + } + } + if (mOptions.transB) + { + for (int i = 0; i < nbDimsB - 1; ++i) + { + out_dims.d[nbDimsA - 1 + i] = inputs[1].d[i]; + } + } + else + { + for (int i = 1; i < nbDimsB; ++i) + { + out_dims.d[nbDimsA - 2 + i] = inputs[1].d[i]; + } + } + return out_dims; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool GemmAllReducePlugin::supportsFormatCombination( + int32_t pos, PluginTensorDesc const* inOut, int32_t nbInputs, int32_t nbOutputs) noexcept +{ + // inOut[0] -> activation + // inOut[1] -> weight + // inOut[1+hasInputSF] -> activation_sf + // inOut[1+hasInputSF*2] -> weight_sf + // inOut[2+hasInputSF*2] -> output[0] = D_uc + // inOut[3+hasInputSF*2] -> output[1] = D_mc + + TLLM_CHECK_WITH_INFO(pos < mNbInputs + mNbOutputs, "Unexpected pos: %d", pos); + auto const& desc = inOut[pos]; + + TLLM_CHECK_WITH_INFO(mArgMap.count(pos) > 0, "pos %d not found in mArgMap.", pos); + TensorArg arg = mArgMap[pos]; + + auto typeExists = [&](DataType dtype, auto idx) -> bool + { + for (const auto& [key, value] : mTypedInstantiators) + { + // key format: <ActivationType, WeightType, OutputType> + if (std::get<decltype(idx)::value>(key) == dtype) + { + return true; + } + } + return false; + }; + + switch (arg) + { + case TensorArg::IN_ACTIVATION: return typeExists(desc.type, std::integral_constant<size_t, 0>{}); + case TensorArg::IN_WEIGHT: return typeExists(desc.type, std::integral_constant<size_t, 1>{}); + case TensorArg::IN_ACTIVATION_SF: + case TensorArg::IN_WEIGHT_SF: + // Assumed SF for only FP4 at the moment + return desc.type == DataType::kFP8; + case TensorArg::IN_ALPHA: return desc.type == DataType::kFLOAT; + case TensorArg::OUT_D_UC: + case TensorArg::OUT_D_MC: + case TensorArg::OUT_D_IPC: return typeExists(desc.type, std::integral_constant<size_t, 2>{}); + default: return false; + } +} + +void GemmAllReducePlugin::configurePlugin( + DynamicPluginTensorDesc const* in, int32_t nbInputs, DynamicPluginTensorDesc const* out, int32_t nbOutputs) noexcept +{ + // Get problem shape + int const nbDimsA = in[0].max.nbDims; + int const minM = utils::computeMDimension(mOptions.transA, in[0].min); + int const maxM = utils::computeMDimension(mOptions.transA, in[0].max); + int const N = utils::computeNDimension(mOptions.transB, in[1].max); + int const K = mOptions.transA ? in[0].max.d[0] : in[0].max.d[nbDimsA - 1]; + + TLLM_CHECK_WITH_INFO(out[0].desc.type == mOptions.typeD, "Output type mismatch."); + + // Ensure call from execution phase does + // not override call from build phase + if (!mOptions.maxProblemShape.isInitialized()) + { + mOptions.maxProblemShape = {minM, maxM, N, K}; + mGemmId = {N, K, mOptions.typeD}; + } + + // Build phase doesn't have COMM_SESSION (i.e built on single rank) + // so do not allocate persistent workspace + if (!isBuilding()) + { + auto getTPRank = [&]() + { + int rank = COMM_SESSION.getRank(); + auto it = std::find(mOptions.group.begin(), mOptions.group.end(), rank); + TLLM_CHECK_WITH_INFO(it != mOptions.group.end(), + "Incorrect group specified - rank " + std::to_string(rank) + " not found in group"); + return std::distance(mOptions.group.begin(), it); + }; + + mRank = getTPRank(); + + if (mWorkspace == nullptr) + { + allocatePersistentWorkspace(); + } + } +} + +size_t GemmAllReducePlugin::getWorkspaceSize( + PluginTensorDesc const* inputs, int32_t nbInputs, PluginTensorDesc const* outputs, int32_t nbOutputs) const noexcept +{ + return 0; +} + +int GemmAllReducePlugin::enqueue(PluginTensorDesc const* inputDesc, PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + // inputs[0] -> [M(*), K] + // inputs[1] -> [K, N] + // outputs[0] -> [M(*), N] unicast ptr + // outputs[1] -> [M(*), N] multicast ptr + auto const nbDimsA = inputDesc[0].dims.nbDims; + auto const M = utils::computeMDimension(mOptions.transA, inputDesc[0].dims); + auto const N = utils::computeNDimension(mOptions.transB, inputDesc[1].dims); + auto const K = mOptions.transA ? inputDesc[0].dims.d[0] : inputDesc[0].dims.d[nbDimsA - 1]; + + TLLM_CHECK_WITH_INFO(M <= mOptions.maxProblemShape.maxM, "GemmAllReducePlugin M > maxM."); + TLLM_CHECK_WITH_INFO(M > 0, "GemmAllReducePlugin M is 0."); + TLLM_CHECK_WITH_INFO(N > 0, "GemmAllReducePlugin N is 0."); + TLLM_CHECK_WITH_INFO(K > 0, "GemmAllReducePlugin K is 0."); + TLLM_CHECK_WITH_INFO(mWorkspace != nullptr, "GemmAllReducePlugin workspace is null."); + + LaunchConfig bestLaunchConfig; + if (mProfiler->useProfiler()) + { + bestLaunchConfig = mProfiler->getBestConfig(M, mGemmId).value(); + } + else + { + bestLaunchConfig = getStaticHeuristicLaunchConfig(M); + } + + void const* activation = inputs[mArgInvMap[TensorArg::IN_ACTIVATION]]; + void const* weight = inputs[mArgInvMap[TensorArg::IN_WEIGHT]]; + void* D_out_uc = outputs[mArgInvMap[TensorArg::OUT_D_UC] - mNbInputs]; + void* D_out_mc = outputs[mArgInvMap[TensorArg::OUT_D_MC] - mNbInputs]; + void* D_out_ipc = outputs[mArgInvMap[TensorArg::OUT_D_IPC] - mNbInputs]; + + TLLM_CHECK_WITH_INFO(activation != nullptr, "GemmAllReducePlugin activation is NULL"); + TLLM_CHECK_WITH_INFO(weight != nullptr, "GemmAllReducePlugin weight is NULL"); + TLLM_CHECK_WITH_INFO(D_out_uc != nullptr, "GemmAllReducePlugin out_uc is NULL"); + TLLM_CHECK_WITH_INFO(D_out_mc != nullptr, "GemmAllReducePlugin out_mc is NULL"); + TLLM_CHECK_WITH_INFO(D_out_ipc != nullptr, "GemmAllReducePlugin out_ipc is NULL"); + + cutlass_kernels::GemmAllReduceImplInterface::ProblemArgs args; + args.argProblemShape(M, N, K, 1) + .argA(activation) + .argB(weight) + .argC(nullptr) + .argD(D_out_uc, D_out_mc, (void**) D_out_ipc) + .argRanks(mRank, mOptions.group) + .argBeta(0.f) // no bias + .argLaunchConfig(bestLaunchConfig) + .argWorkspace(mWorkspace->mWorkspace.get()); + // tensor for scaling input A + if (mOptions.hasSFA) + { + void const* activation_sf = inputs[mArgInvMap[TensorArg::IN_ACTIVATION_SF]]; + TLLM_CHECK_WITH_INFO(activation_sf != nullptr, "GemmAllReducePlugin activation_sf is NULL"); + args.argAScale(activation_sf); + } + // tensor for scaling input B + if (mOptions.hasSFB) + { + void const* weight_sf = inputs[mArgInvMap[TensorArg::IN_WEIGHT_SF]]; + TLLM_CHECK_WITH_INFO(weight_sf != nullptr, "GemmAllReducePlugin weight_sf is NULL"); + args.argBScale(weight_sf); + } + // tensor for scaling output D + if (mOptions.alphaIsPtr) + { + void const* alpha_vec = inputs[mArgInvMap[TensorArg::IN_ALPHA]]; + TLLM_CHECK_WITH_INFO(alpha_vec != nullptr, "GemmAllReducePlugin alpha_vec is NULL"); + args.argAlphaPtr(reinterpret_cast<float const*>(alpha_vec)); + } + else + { + args.argAlpha(mOptions.alpha); + } + + mGemm->run(args, stream); + + return 0; +} + +////////////////////////////////// +// IPluginV2Ext Methods +////////////////////////////////// +DataType GemmAllReducePlugin::getOutputDataType(int index, DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK_WITH_INFO(index < getNbOutputs(), "Output index out of bounds: %d", index); + return mOptions.typeD; +} + +////////////////////////////////// +// IPluginV2 Methods +////////////////////////////////// +char const* GemmAllReducePlugin::getPluginType() const noexcept +{ + return GEMM_ALLREDUCE_PLUGIN_NAME; +} + +char const* GemmAllReducePlugin::getPluginVersion() const noexcept +{ + return GEMM_ALLREDUCE_PLUGIN_VERSION; +} + +int GemmAllReducePlugin::getNbOutputs() const noexcept +{ + return mNbOutputs; +} + +int GemmAllReducePlugin::initialize() noexcept +{ + if (isBuilding() && mProfiler->useProfiler()) + { + // TODO (xsimmons): interfaces between GemmPluginProfiler and Plugin + // needs to be relooked at - current interface implicitly assigns runner to profiler + // object in profileTactics() + assert(mOptions.maxProblemShape.isInitialized()); + mProfiler->profileTactics(mGemm, mOptions.typeD, mOptions.maxProblemShape, mGemmId); + } + return 0; +} + +void GemmAllReducePlugin::terminate() noexcept +{ + if (isBuilding()) // need this otherwise getComm will crash during build phase + { + return; + } + + // free mWorkspace + if (mWorkspace) + { + getPluginRegistry()->releasePluginResource(mWorkspaceKey.c_str()); + mWorkspace = nullptr; + } +} + +size_t GemmAllReducePlugin::getSerializationSize() const noexcept +{ + // cannot use sizeof(GemmAllReducePluginOptions) + // becaused need packed attribute which doesn't work on enum + // without making the enum also packed + size_t size = 0; + size += sizeof(mOptions.typeA); + size += sizeof(mOptions.typeB); + size += sizeof(mOptions.typeD); + size += sizeof(mOptions.transA); + size += sizeof(mOptions.transB); + size += sizeof(mOptions.alpha); + size += sizeof(mOptions.maxProblemShape); + size += sizeof(mOptions.groupSize); + size += mOptions.group.size() * sizeof(int); + size += sizeof(mOptions.hasSFA); + size += sizeof(mOptions.hasSFB); + size += sizeof(mOptions.alphaIsPtr); + return size; +} + +void GemmAllReducePlugin::serialize(void* buffer) const noexcept +{ + char* begin = reinterpret_cast<char*>(buffer); + char* end = reinterpret_cast<char*>(buffer); + + write(end, mOptions.typeA); + write(end, mOptions.typeB); + write(end, mOptions.typeD); + write(end, mOptions.transA); + write(end, mOptions.transB); + write(end, mOptions.alpha); + write(end, mOptions.maxProblemShape); + write(end, mOptions.groupSize); + for (auto const& rank : mOptions.group) + { + write(end, rank); + } + write(end, mOptions.hasSFA); + write(end, mOptions.hasSFB); + write(end, mOptions.alphaIsPtr); + TLLM_CHECK(end == begin + getSerializationSize()); + + // Profiler MNK->kernel mappings need to be deterministic and consistent across ranks + // to ensure correct functionality (unlike standalone GEMMs). + // Since by default each rank will generate and serialize its own profiler mapping + // this can lead to different mappings between ranks which will result in fatal + // error. Therefore only generate and use profiler mapping for single rank. + if (mProfiler->useProfiler() && COMM_SESSION.getRank() == 0) + { + mProfiler->serializeToOwnFile(mGemmId); + } +} + +void GemmAllReducePlugin::destroy() noexcept +{ + delete this; +} + +//////////////////////////////////////////////////////////// +// GemmAllReducePluginCreator Methods +//////////////////////////////////////////////////////////// +PluginFieldCollection GemmAllReducePluginCreator::mFC; +std::vector<PluginField> GemmAllReducePluginCreator::mPluginAttributes; + +GemmAllReducePluginCreator::GemmAllReducePluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back("type_a", nullptr, PluginFieldType::kINT32, 1); + mPluginAttributes.emplace_back("type_b", nullptr, PluginFieldType::kINT32, 1); + mPluginAttributes.emplace_back("type_d", nullptr, PluginFieldType::kINT32, 1); + mPluginAttributes.emplace_back("transa", nullptr, PluginFieldType::kINT32, 1); + mPluginAttributes.emplace_back("transb", nullptr, PluginFieldType::kINT32, 1); + mPluginAttributes.emplace_back("alpha", nullptr, PluginFieldType::kFLOAT32, 1); + mPluginAttributes.emplace_back("group", nullptr, PluginFieldType::kINT32, 1); + mPluginAttributes.emplace_back("has_sfa", nullptr, PluginFieldType::kINT8, 1); + mPluginAttributes.emplace_back("has_sfb", nullptr, PluginFieldType::kINT8, 1); + mPluginAttributes.emplace_back("alpha_is_ptr", nullptr, PluginFieldType::kINT8, 1); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* GemmAllReducePluginCreator::getPluginName() const noexcept +{ + return GEMM_ALLREDUCE_PLUGIN_NAME; +} + +char const* GemmAllReducePluginCreator::getPluginVersion() const noexcept +{ + return GEMM_ALLREDUCE_PLUGIN_VERSION; +} + +PluginFieldCollection const* GemmAllReducePluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* GemmAllReducePluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + GemmAllReducePluginOptions options; + options.deserialize = false; + + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type_a")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + options.typeA = *static_cast<DataType const*>(fields[i].data); + } + else if (!strcmp(attrName, "type_b")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + options.typeB = *static_cast<DataType const*>(fields[i].data); + } + else if (!strcmp(attrName, "type_d")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + options.typeD = *static_cast<DataType const*>(fields[i].data); + } + else if (!strcmp(attrName, "transa")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + options.transA = *static_cast<int const*>(fields[i].data); + } + else if (!strcmp(attrName, "transb")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + options.transB = *static_cast<int const*>(fields[i].data); + } + else if (!strcmp(attrName, "alpha")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + options.alpha = *static_cast<float const*>(fields[i].data); + } + else if (!strcmp(attrName, "group")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + auto const* ranks = static_cast<int const*>(fields[i].data); + for (int j = 0; j < fields[i].length; ++j) + { + options.group.insert(ranks[j]); + } + options.groupSize = options.group.size(); + } + else if (!strcmp(attrName, "has_sfa")) // passed in as input tensor + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + options.hasSFA = *static_cast<int8_t const*>(fields[i].data); + } + else if (!strcmp(attrName, "has_sfb")) // passed in as input tensor + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + options.hasSFB = *static_cast<int8_t const*>(fields[i].data); + } + else if (!strcmp(attrName, "alpha_is_ptr")) // passed in as input tensor + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + options.alphaIsPtr = *static_cast<int8_t const*>(fields[i].data); + } + } + + try + { + // GemmAllReducePluginCreator is unique and shared for an engine generation + auto* obj = new GemmAllReducePlugin(options); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + return nullptr; + } +} + +IPluginV2* GemmAllReducePluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call GemmAllReducePlugin::destroy() + try + { + auto* obj = new GemmAllReducePlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePlugin.h b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePlugin.h new file mode 100644 index 000000000000..457926246002 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePlugin.h @@ -0,0 +1,189 @@ +/* + * Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#if defined(USING_OSS_CUTLASS_ALLREDUCE_GEMM) +#include "tensorrt_llm/kernels/cutlass_kernels/include/allreduce_gemm_runner.h" +#else +#include "allreduce_gemm_runner.h" +#endif + +#include "gemmAllReducePluginProfiler.h" +#include "gemmAllReducePluginResource.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +using namespace nvinfer1; + +using nvinfer1::DataType; +#if defined(USING_OSS_CUTLASS_ALLREDUCE_GEMM) +namespace cutlass_kernels = ::tensorrt_llm::kernels::opened_cutlass_kernels; +#else +namespace cutlass_kernels = ::tensorrt_llm::kernels::cutlass_kernels; +#endif + +using LaunchConfig = typename cutlass_kernels::GemmAllReduceImplInterface::LaunchConfig; + +namespace tensorrt_llm::plugins +{ +struct GemmAllReducePluginOptions +{ + // Don't need to specify problem shape, this + // is specified in configurePlugin + DataType typeA; + DataType typeB; + DataType typeD; + int transA; + int transB; + float alpha; + // ranks participating in collective + std::set<int> group; + int groupSize; + // Set in configurePlugin during build phase + GemmDims maxProblemShape; + bool deserialize; // used for profiler instantiation + int8_t hasSFA = 0; + int8_t hasSFB = 0; + int8_t alphaIsPtr = 0; +}; + +class GemmAllReducePlugin : public BasePlugin +{ + friend class GemmAllReducePluginCreator; + +public: + ~GemmAllReducePlugin() override = default; + + ////////////////////////////////// + // IPluginV2DynamicExt Methods + ////////////////////////////////// + IPluginV2DynamicExt* clone() const noexcept override; + + DimsExprs getOutputDimensions( + int outputIndex, DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept override; + + // inOut[0] -> activation + // inOut[1] -> weight + // inOut[2] -> result + bool supportsFormatCombination( + int32_t pos, PluginTensorDesc const* inOut, int32_t nbInputs, int32_t nbOutputs) noexcept override; + + // in[0] -> activation + // in[1] -> weight + // no bias needed + void configurePlugin(DynamicPluginTensorDesc const* in, int32_t nbInputs, DynamicPluginTensorDesc const* out, + int32_t nbOutputs) noexcept override; + + size_t getWorkspaceSize(PluginTensorDesc const* inputs, int32_t nbInputs, PluginTensorDesc const* outputs, + int32_t nbOutputs) const noexcept override; + + // in[0] -> activation + // in[1] -> weight + // out[0] -> result_uc + // out[1] -> result_mc + int enqueue(PluginTensorDesc const* inputDesc, PluginTensorDesc const* outputDesc, void const* const* inputs, + void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + ////////////////////////////////// + // IPluginV2Ext Methods + ////////////////////////////////// + DataType getOutputDataType(int index, DataType const* inputTypes, int nbInputs) const noexcept override; + + ////////////////////////////////// + // IPluginV2 Methods + ////////////////////////////////// + char const* getPluginType() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + int getNbOutputs() const noexcept override; + + int initialize() noexcept override; + + void terminate() noexcept override; + + size_t getSerializationSize() const noexcept override; + + void serialize(void* buffer) const noexcept override; + + void destroy() noexcept override; + +private: + explicit GemmAllReducePlugin(GemmAllReducePluginOptions const& options); + // Parameterized constructor + explicit GemmAllReducePlugin(void const* data, size_t length); + + void allocatePersistentWorkspace(); + + LaunchConfig getStaticHeuristicLaunchConfig(int M) const; + + // Params that are initialized during constructor + using KeyType = std::tuple<DataType, DataType, DataType>; + using ValueType = std::function<cutlass_kernels::GemmAllReduceImplInterface*()>; + GemmAllReducePluginOptions mOptions; + int mRank = 0; + + enum TensorArg + { + IN_ACTIVATION, + IN_ACTIVATION_SF, + IN_WEIGHT, + IN_WEIGHT_SF, + IN_ALPHA, + OUT_D_UC, + OUT_D_MC, + OUT_D_IPC + }; + + std::unordered_map<int, TensorArg> mArgMap; + std::unordered_map<TensorArg, int> mArgInvMap; + int mNbInputs = 0; + int mNbOutputs = 0; + + std::map<KeyType, ValueType> mTypedInstantiators; + std::string mWorkspaceKey; + std::shared_ptr<cutlass_kernels::GemmAllReduceImplInterface> mGemm; + // Params that are initialized during configurePlugin() + GemmAllReducePersistentWorkspace* mWorkspace = nullptr; + + // Used for selecting best GEMM for given problem shapes + GemmIdCore mGemmId{}; + GemmPluginProfilerManager<GemmAllReducePluginProfiler> mGemmPluginProfileManager; + std::shared_ptr<GemmAllReducePluginProfiler> mProfiler; +}; + +class GemmAllReducePluginCreator : public BaseCreator +{ +public: + GemmAllReducePluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginProfiler.cpp b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginProfiler.cpp new file mode 100644 index 000000000000..a6f7ca2615df --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginProfiler.cpp @@ -0,0 +1,140 @@ +/* + * Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "gemmAllReducePlugin.h" +#include "tensorrt_llm/common/dataType.h" +#include "tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h" +#include "tensorrt_llm/plugins/common/pluginUtils.h" + +namespace tc = tensorrt_llm::common; + +namespace tensorrt_llm::plugins +{ +void GemmAllReducePluginProfiler::serializeToOwnFile(GemmIdCore gemmId) +{ + std::vector<char> file_buf(getSerializationSize(gemmId)); + char* begin = file_buf.data(); + char* end = file_buf.data(); + serialize(end, gemmId); + assert(end == begin + file_buf.size()); + + auto fileName = getCacheFileName(gemmId); + std::ofstream file(fileName, std::ios::binary); + TLLM_CHECK(file.is_open()); + file.write(begin, file_buf.size()); + file.flush(); + file.close(); +} + +void GemmAllReducePluginProfiler::deserializeFromOwnFile(GemmIdCore gemmId, GemmDims problemShape) +{ + auto fileName = getCacheFileName(gemmId); + std::ifstream file(fileName, std::ios::binary); + TLLM_CHECK(file.is_open()); + file.seekg(0, std::ios::end); + std::streamsize size = file.tellg(); + TLLM_CHECK(size > 0); + file.seekg(0, std::ios::beg); + + std::vector<char> file_buf(size); + file.read(file_buf.data(), size); + file.close(); + + char const* begin = const_cast<char const*>(file_buf.data()); + char const* end = begin; + deserialize(end, problemShape, gemmId); + assert(end == begin + size); +} + +bool GemmAllReducePluginProfiler::useProfiler() +{ + // char const* envDir = getenv("GEMM_AR_PLUGIN_PROFILE_DIR"); + // return envDir != nullptr; + // TODO(xsimmons): currently the profiler does not add any perf gain + // due to static heuristics being sufficient. We can re-enable this + // when we need more configurations. + return false; +} + +std::string GemmAllReducePluginProfiler::getCacheFileName(GemmIdCore gemmId) +{ + std::stringstream fileName; + char const* envDir = getenv("GEMM_AR_PLUGIN_PROFILE_DIR"); + std::string directory = envDir ? std::string(envDir) : "/tmp/"; + fileName << directory + "/gemm-AR"; + fileName << "-n" << std::to_string(gemmId.n); + fileName << "-k" << std::to_string(gemmId.k); + fileName << "-" << tc::getDtypeString(gemmId.dtype); + fileName << ".prof_cache"; + return fileName.str(); +} + +void GemmAllReducePluginProfiler::runTactic(int m, int n, int k, + cutlass_kernels::GemmAllReduceImplInterface::LaunchConfig const& tactic, char* workspace, + cudaStream_t const& stream) +{ + const size_t dtype_size = tc::getDTypeSize(mType); + char* inputA = workspace; + char* inputB = inputA + m * k * dtype_size; + char* outputD = inputB + n * k * dtype_size; + char* inputSFA = outputD + m * n * dtype_size; + char* inputSFB = inputSFA + m * k * dtype_size; + std::set<int> tpGroup = {0}; + + // Run on single-GPU + cutlass_kernels::GemmAllReduceImplInterface::ProblemArgs args; + args.argProblemShape(m, n, k, 1) + .argA((void*) inputA) + .argB((void*) inputB) + .argD((void*) outputD, /*output_mc=*/nullptr) + .argAScale((void*) inputSFA) + .argBScale((void*) inputSFB) + .argRanks(0, tpGroup) + .argAlpha(1.f) + .argBeta(0.f) // no bias + .argLaunchConfig(tactic); + + TLLM_CHECK(mRunner != nullptr); + mRunner->run(args, stream); +} + +void GemmAllReducePluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + TLLM_CHECK(maxM != 0); + TLLM_CHECK(n != 0); + TLLM_CHECK(k != 0); + // mType refers to the output data type + // WARNING: This code assumes that the output precision is >= to input precision + const size_t dtype_size = tc::getDTypeSize(mType); + size_t bytes = 0; + bytes += maxM * k * dtype_size; // A + bytes += n * k * dtype_size; // B + // No C + // Note that D is typically IPC, however, when tuning GEMM we need it to run on single GPU + bytes += maxM * n * dtype_size; // D + // scale tensors for A & B - will at most be same size as A/B + bytes += maxM * k * dtype_size; // A + bytes += n * k * dtype_size; // B + + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<cutlass_kernels::GemmAllReduceImplInterface::LaunchConfig> GemmAllReducePluginProfiler::getTactics( + int m, int n, int k) const +{ + TLLM_CHECK(mRunner != nullptr); + return mRunner->getSupportedLaunchConfigs(); +} +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginProfiler.h b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginProfiler.h new file mode 100644 index 000000000000..faacbb3b8c0f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginProfiler.h @@ -0,0 +1,68 @@ +/* + * Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#if defined(USING_OSS_CUTLASS_ALLREDUCE_GEMM) +#include "tensorrt_llm/kernels/cutlass_kernels/include/allreduce_gemm_runner.h" +#else +#include "allreduce_gemm_runner.h" +#endif +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" + +namespace tensorrt_llm::plugins +{ +/* + * Used for tuning to find best GEMM configs for different problem shapes. + * WARNING: Tuning GEMM+AR kernel may not be fully representable of real + * multi-GPU workloads as tuning only runs on single-GPU. + * IMPORTANT: TRT-LLM does not support deterministic tuning across ranks. + * Because of this, we have to serialize/deserialize our own configuration file. + */ + +#if defined(USING_OSS_CUTLASS_ALLREDUCE_GEMM) +namespace cutlass_kernels = ::tensorrt_llm::kernels::opened_cutlass_kernels; +#else +namespace cutlass_kernels = ::tensorrt_llm::kernels::cutlass_kernels; +#endif +class GemmAllReducePluginProfiler + : public GemmPluginProfiler<cutlass_kernels::GemmAllReduceImplInterface::LaunchConfig, + std::shared_ptr<cutlass_kernels::GemmAllReduceImplInterface>, GemmIdCore, GemmIdCoreHash> +{ +public: + void serializeToOwnFile(GemmIdCore gemmId); + + void deserializeFromOwnFile(GemmIdCore gemmId, GemmDims problemShape); + + bool useProfiler(); + +protected: + //////////////////////////////////// + // GemmPluginProfiler methods + //////////////////////////////////// + void runTactic(int m, int n, int k, cutlass_kernels::GemmAllReduceImplInterface::LaunchConfig const& tactic, + char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + std::vector<cutlass_kernels::GemmAllReduceImplInterface::LaunchConfig> getTactics( + int m, int n, int k) const override; + +private: + static std::string getCacheFileName(GemmIdCore gemmId); +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginResource.h b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginResource.h new file mode 100644 index 000000000000..8136bd363bd7 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmAllReducePlugin/gemmAllReducePluginResource.h @@ -0,0 +1,82 @@ +/* + * Copyright (c) 2020-2023, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "NvInferPlugin.h" + +#if defined(USING_OSS_CUTLASS_ALLREDUCE_GEMM) +#include "tensorrt_llm/kernels/cutlass_kernels/include/allreduce_gemm_runner.h" +#else +#include "allreduce_gemm_runner.h" +#endif +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/plugins/common/plugin.h" + +using namespace nvinfer1; + +namespace tensorrt_llm::plugins +{ + +#if defined(USING_OSS_CUTLASS_ALLREDUCE_GEMM) +namespace cutlass_kernels = ::tensorrt_llm::kernels::opened_cutlass_kernels; +#else +namespace cutlass_kernels = ::tensorrt_llm::kernels::cutlass_kernels; +#endif +class GemmAllReducePersistentWorkspace : public IPluginResource +{ +public: + GemmAllReducePersistentWorkspace(std::shared_ptr<cutlass_kernels::PersistentWorkspaceInterface> workspace) + : mWorkspace(workspace) + { + } + + ////////////////////////////////// + // IPluginResource Methods + ////////////////////////////////// + IPluginResource* clone() noexcept override + { + auto copy = new GemmAllReducePersistentWorkspace(mWorkspace); + // Resource initialization (if any) may be skipped for non-cloned objects + // since only clones will be registered by TensorRT. + try + { + copy->mWorkspace->allocate(); + return copy; + } + catch (std::exception const& e) + { + TLLM_LOG_ERROR(e.what()); + return nullptr; + } + } + + int32_t release() noexcept override + { + try + { + return mWorkspace->free(); + } + catch (std::exception const& e) + { + TLLM_LOG_ERROR(e.what()); + return -1; + } + } + + std::shared_ptr<cutlass_kernels::PersistentWorkspaceInterface> mWorkspace; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gemmPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/gemmPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp b/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp new file mode 100644 index 000000000000..9e06ad01d10f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.cpp @@ -0,0 +1,614 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "gemmPlugin.h" + +#include "gemmPluginProfiler.h" +#include "plugin.h" +#include "pluginUtils.h" +#include "tensorrt_llm/kernels/weightOnlyBatchedGemv/cudaCoreGemm.h" +#include "tensorrt_llm/runtime/utils/debugUtils.h" + +#include <NvInferRuntime.h> + +#include <cassert> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::GemmDims; +using tensorrt_llm::plugins::GemmPluginCreator; +using tensorrt_llm::plugins::GemmPlugin; +using tensorrt_llm::plugins::CublasLtGemmPluginProfiler; +using tensorrt_llm::plugins::CublasGemmWrapperPtr; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +static char const* GEMM_PLUGIN_VERSION{"1"}; +static char const* GEMM_PLUGIN_NAME{"Gemm"}; +PluginFieldCollection GemmPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> GemmPluginCreator::mPluginAttributes; + +void getProblemParams(cublasOperation_t& transa, cublasOperation_t& transb, int& m, int& n, int& k, int& lda, int& ldb, + int& ldc, bool transA, bool transB, int M, int N, int K, int padLda, int padLdb, int padLdc) +{ + transa = transB ? CUBLAS_OP_T : CUBLAS_OP_N; + transb = transA ? CUBLAS_OP_T : CUBLAS_OP_N; + m = N; + n = M; + k = K; + lda = transB ? K + padLdb : N + padLdb; + ldb = transA ? M + padLda : K + padLda; + ldc = N + padLdc; +} + +void runGemm(int const M, int const N, int const K, bool const transA, bool const transB, int const padLda, + int const padLdb, int const padLdc, nvinfer1::DataType const type, CublasGemmWrapperPtr const& cublasWrapperPtr, + void const* act, void const* weight, float const alpha, void* output, + std::optional<cublasLtMatmulHeuristicResult_t> const& heuristic, void* workspace, cudaStream_t stream) +{ + if (M == 0 || N == 0 || K == 0) + return; + + cublasWrapperPtr->setStream(stream); + cublasWrapperPtr->setWorkspace(workspace); + + cublasOperation_t transa, transb; + int m, n, k; + int lda, ldb, ldc; + getProblemParams(transa, transb, m, n, k, lda, ldb, ldc, transA, transB, M, N, K, padLda, padLdb, padLdc); + + cublasWrapperPtr->createDescriptors(transa, transb, m, n, k, lda, ldb, ldc); + cublasWrapperPtr->Gemm(transa, transb, m, n, k, weight, lda, act, ldb, output, ldc, alpha, 0.0f, heuristic); + cublasWrapperPtr->destroyDescriptors(); +} + +void CublasLtGemmPluginProfiler::runTactic( + int m, int n, int k, CublasLtGemmPluginProfiler::Config const& tactic, char* workspace, cudaStream_t const& stream) +{ + size_t dataSize = sizeof(half); + if (mType == nvinfer1::DataType::kFLOAT) + { + dataSize = sizeof(float); + } + + void* actPtr = reinterpret_cast<void*>(workspace); + void* weightPtr = reinterpret_cast<void*>( + nextWorkspacePtrWithAlignment(reinterpret_cast<int8_t*>(actPtr), m * k * dataSize, ALIGNMENT)); + void* outputPtr = reinterpret_cast<void*>( + nextWorkspacePtrWithAlignment(reinterpret_cast<int8_t*>(weightPtr), n * k * dataSize, ALIGNMENT)); + char* workspacePtr = reinterpret_cast<char*>( + nextWorkspacePtrWithAlignment(reinterpret_cast<int8_t*>(outputPtr), m * (n + mPadLdc) * dataSize, ALIGNMENT)); + runGemm(m, n, k, mTransA, mTransB, mPadLda, mPadLdb, mPadLdc, mType, mRunner, actPtr, weightPtr, 1.0f, outputPtr, + {tactic}, workspacePtr, stream); +} + +bool CublasLtGemmPluginProfiler::checkTactic(int m, int n, int k, Config const& tactic) const +{ + cublasOperation_t transa, transb; + int M = m, N = n, K = k; + int lda, ldb, ldc; + getProblemParams(transa, transb, m, n, k, lda, ldb, ldc, mTransA, mTransB, M, N, K, mPadLda, mPadLdb, mPadLdc); + + mRunner->createDescriptors(transa, transb, m, n, k, lda, ldb, ldc); + + auto const checkResult = mRunner->checkTactic(transa, transb, m, n, k, lda, ldb, ldc, tactic.algo); + + mRunner->destroyDescriptors(); + + return checkResult; +} + +void CublasLtGemmPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + size_t dataSize = getDTypeSize(mType); + size_t outputDataSize = getDTypeSize(mOutputType); + + std::vector<size_t> workspaces = { + maxM * k * dataSize, // A + n * k * dataSize, // B + maxM * (n + mPadLdc) * outputDataSize, // C + CUBLAS_WORKSPACE_SIZE // workspace + }; + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size(), ALIGNMENT); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<CublasLtGemmPluginProfiler::Config> CublasLtGemmPluginProfiler::getTactics(int M, int N, int K) const +{ + cublasOperation_t transa, transb; + int m, n, k; + int lda, ldb, ldc; + getProblemParams(transa, transb, m, n, k, lda, ldb, ldc, mTransA, mTransB, M, N, K, mPadLda, mPadLdb, mPadLdc); + + mRunner->createDescriptors(transa, transb, m, n, k, lda, ldb, ldc); + auto const heruistics = mRunner->getTactics(transa, transb, m, n, k, lda, ldb, ldc); + mRunner->destroyDescriptors(); + + return heruistics; +} + +GemmPlugin::GemmPlugin(int transA, int transB, int padLda, int padLdb, int padLdc, nvinfer1::DataType type, bool useFp8, + float alpha, GemmPlugin::PluginProfilerPtr const& pluginProfiler) + : mTransA(transA) + , mTransB(transB) + , mPadLda(padLda) + , mPadLdb(padLdb) + , mPadLdc(padLdc) + , mType(type) + , mOutputType(type) + , mUseFp8(useFp8) + , mAlpha(alpha) + , mPluginProfiler(pluginProfiler) +{ + init(); +} + +// Parameterized constructor +GemmPlugin::GemmPlugin(void const* data, size_t length, GemmPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mTransA); + read(d, mTransB); + read(d, mPadLda); + read(d, mPadLdb); + read(d, mPadLdc); + read(d, mType); + read(d, mUseFp8); + read(d, mAlpha); + read(d, mDims); + read(d, mOutputType); + + init(); + + mPluginProfiler->deserialize(d, mDims, mGemmId); + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +thread_local CublasGemmWrapperPtr GemmPlugin::mCublasWrapper = nullptr; + +void GemmPlugin::init() +{ + auto cublasHandle = getCublasHandle(); + auto cublasLtHandle = getCublasLtHandle(); + mCublasWrapper = std::make_shared<CublasMMWrapper>(cublasHandle, cublasLtHandle, nullptr, nullptr); + + mPluginProfiler->setTranspose(mTransA, mTransB); + mPluginProfiler->setOutputType(mOutputType); + mPluginProfiler->setPadLd(mPadLda, mPadLdb, mPadLdc); + + mGemmId = GemmIdCublas(mDims.n, mDims.k, mType, mTransA, mTransB, mOutputType); + + mArch = tensorrt_llm::common::getSMVersion(); +} + +void GemmPlugin::setGemmConfig() +{ + if (mType == nvinfer1::DataType::kHALF) + { + mCublasWrapper->setFP16GemmConfig(trtToCublasDtype(mOutputType)); + } + else if (mType == nvinfer1::DataType::kFLOAT) + { + mCublasWrapper->setFP32GemmConfig(); + } +#ifdef ENABLE_BF16 + else if (mType == nvinfer1::DataType::kBF16) + { + mCublasWrapper->setBF16GemmConfig(trtToCublasDtype(mOutputType)); + } +#endif + +#ifdef ENABLE_FP8 + if (mUseFp8) + { + mCublasWrapper->setFP8GemmConfig(trtToCublasDtype(mOutputType)); + } +#endif +} + +void GemmPlugin::configGemm() +{ + if (!mDims.isInitialized()) + { + return; + } + + setGemmConfig(); + + mPluginProfiler->profileTactics(mCublasWrapper, mType, mDims, mGemmId); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* GemmPlugin::clone() const noexcept +{ + auto* plugin = new GemmPlugin(*this); + return plugin; +} + +nvinfer1::DimsExprs GemmPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 2); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + int const nbDimsB = inputs[1].nbDims; + DimsExprs ret; + ret.nbDims = nbDimsA + nbDimsB - 2; + + if (mTransA) + { + for (int i = 1; i < nbDimsA; ++i) + { + ret.d[i - 1] = inputs[0].d[i]; + } + } + else + { + for (int i = 0; i < nbDimsA - 1; ++i) + { + ret.d[i] = inputs[0].d[i]; + } + } + if (mTransB) + { + for (int i = 0; i < nbDimsB - 1; ++i) + { + ret.d[nbDimsA - 1 + i] = exprBuilder.constant(inputs[1].d[i]->getConstantValue() + mPadLdc); + } + } + else + { + for (int i = 1; i < nbDimsB; ++i) + { + ret.d[nbDimsA - 2 + i] = exprBuilder.constant(inputs[1].d[i]->getConstantValue() + mPadLdc); + } + } + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool GemmPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + auto const& desc = inOut[pos]; + if (desc.format != TensorFormat::kLINEAR) + { + return false; + } + + if (pos < nbInputs) + { + // If use FP8, act/weight dtype should be kFP8 + if (mUseFp8) + { + return desc.type == nvinfer1::DataType::kFP8; + } + else + { + return desc.type == mType; + } + } + + return desc.type == mType || desc.type == nvinfer1::DataType::kFLOAT; +} + +void GemmPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const nbDimsA = in[0].max.nbDims; + + auto const minM = utils::computeMDimension(mTransA, in[0].min); + auto const maxM = utils::computeMDimension(mTransA, in[0].max); + auto const N = utils::computeNDimension(mTransB, in[1].max); + auto const K = static_cast<utils::DimType64>(mTransA ? in[0].max.d[0] : in[0].max.d[nbDimsA - 1]); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, N, K}; + } + mGemmId.n = N; + mGemmId.k = K; + + mOutputType = out[0].desc.type; +} + +size_t GemmPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return CUBLAS_WORKSPACE_SIZE; +} + +int GemmPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + // inputs + // mat1 [M, K] (mTransA = False) + // mat2 [K, N] (mTransB = False) + // outputs + // mat [M, N] + if (mCublasWrapper == nullptr) + { + auto cublasHandle = getCublasHandle(); + auto cublasLtHandle = getCublasLtHandle(); + mCublasWrapper = std::make_shared<CublasMMWrapper>(cublasHandle, cublasLtHandle, nullptr, nullptr); + } + setGemmConfig(); + + int const nbDimsA = inputDesc[0].dims.nbDims; + int const padM = mTransA ? mPadLda : 0; + int const padN = mTransB ? 0 : mPadLdb; + int const padK = mTransA ? 0 : mPadLda; + auto const M = utils::computeMDimension(mTransA, inputDesc[0].dims) - padM; + auto const N = utils::computeNDimension(mTransB, inputDesc[1].dims) - padN; + int const K = static_cast<utils::DimType64>( + mTransA ? inputDesc[0].dims.d[0] - padK : inputDesc[0].dims.d[nbDimsA - 1] - padK); + + bool noPadDim = padM == 0 && padN == 0 && padK == 0 && mPadLdc == 0; + bool cudaKernelSupportType = mType == nvinfer1::DataType::kHALF || mType == nvinfer1::DataType::kFLOAT + || mType == nvinfer1::DataType::kBF16; + + // skip computation for a TRT empty tensor + if (M == 0) + { + return 0; + } + + std::string mnkStr = "MNK={" + std::to_string(M) + ", " + std::to_string(N) + ", " + std::to_string(K) + "}"; + { + std::string const activationStr = "GEMM layer's activation before GEMM with " + mnkStr; + TLLM_CHECK_DEBUG_WITH_INFO( + tensorrt_llm::runtime::utils::tensorHasInvalid(M, K, mType, inputs[0], stream, activationStr) == false, + "Found invalid number (NaN or Inf) in " + activationStr); + } + + bool cudaKernelFinished = false; + bool isArch90or100 = mArch >= 90 && mArch < 120; + // TODO: sub tensor matmul is not supported in fp8 gemm cuda kernel + if (!isArch90or100 && M <= 4 && N <= 128000 && mUseFp8 && noPadDim && cudaKernelSupportType) + { + tensorrt_llm::kernels::cuda_core_gemm::Params params(reinterpret_cast<void const*>(inputs[0]), + reinterpret_cast<void const*>(inputs[1]), mAlpha, reinterpret_cast<void*>(outputs[0]), M, N, K, + CUDA_R_8F_E4M3, trtToCublasDtype(mOutputType)); + cudaKernelFinished = tensorrt_llm::kernels::cuda_core_gemm::cudaCoreGemmDispatcher(params, stream); + } + else if (!isArch90or100 && ((mArch < 90 && M <= 6) || (isArch90or100 && M <= 2)) && N <= 128000 && !mUseFp8 + && noPadDim && cudaKernelSupportType) + { + tensorrt_llm::kernels::cuda_core_gemm::Params params(reinterpret_cast<void const*>(inputs[0]), + reinterpret_cast<void const*>(inputs[1]), mAlpha, reinterpret_cast<void*>(outputs[0]), M, N, K, + trtToCublasDtype(mType), trtToCublasDtype(mOutputType)); + cudaKernelFinished = tensorrt_llm::kernels::cuda_core_gemm::cudaCoreGemmDispatcher(params, stream); + } + + if (!cudaKernelFinished) + { + auto bestTactic = mPluginProfiler->getBestConfig(M, mGemmId); + runGemm(M, N, K, mTransA, mTransB, mPadLda, mPadLdb, mPadLdc, mType, mCublasWrapper, inputs[0], inputs[1], + mAlpha, outputs[0], bestTactic, workspace, stream); + } + + { + std::string const outputStr = "GEMM layer's output after GEMM with " + mnkStr; + TLLM_CHECK_DEBUG_WITH_INFO( + tensorrt_llm::runtime::utils::tensorHasInvalid(M, N + mPadLdc, mType, outputs[0], stream, outputStr) + == false, + "Found invalid number (NaN or Inf) in " + outputStr); + } + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType GemmPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* GemmPlugin::getPluginType() const noexcept +{ + return GEMM_PLUGIN_NAME; +} + +char const* GemmPlugin::getPluginVersion() const noexcept +{ + return GEMM_PLUGIN_VERSION; +} + +int GemmPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int GemmPlugin::initialize() noexcept +{ + configGemm(); + return 0; +} + +void GemmPlugin::destroy() noexcept +{ + delete this; +} + +size_t GemmPlugin::getSerializationSize() const noexcept +{ + return sizeof(mTransA) + sizeof(mTransB) + sizeof(mPadLda) + sizeof(mPadLdb) + sizeof(mPadLdc) + sizeof(mType) + + sizeof(mDims) + sizeof(mUseFp8) + sizeof(mAlpha) + mPluginProfiler->getSerializationSize(mGemmId) + + sizeof(mOutputType); // selected tactics container size +} + +void GemmPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mTransA); + write(d, mTransB); + write(d, mPadLda); + write(d, mPadLdb); + write(d, mPadLdc); + write(d, mType); + write(d, mUseFp8); + write(d, mAlpha); + write(d, mDims); + write(d, mOutputType); + mPluginProfiler->serialize(d, mGemmId); + + TLLM_CHECK(d == a + getSerializationSize()); +} + +void GemmPlugin::terminate() noexcept {} + +/////////////// + +GemmPluginCreator::GemmPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("transA", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("transB", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("padLda", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("padLdb", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("padLdc", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("use_fp8", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* GemmPluginCreator::getPluginName() const noexcept +{ + return GEMM_PLUGIN_NAME; +} + +char const* GemmPluginCreator::getPluginVersion() const noexcept +{ + return GEMM_PLUGIN_VERSION; +} + +PluginFieldCollection const* GemmPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* GemmPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int transA{}; + int transB{}; + int padLda{}; + int padLdb{}; + int padLdc{}; + nvinfer1::DataType type{}; + int useFp8{}; + float alpha = 1.F; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "transa")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + transA = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "transb")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + transB = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "pad_lda")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + padLda = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "pad_ldb")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + padLdb = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "pad_ldc")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + padLdc = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "use_fp8")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + useFp8 = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "alpha")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + alpha = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + } + try + { + // GemmPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + // FIXME enable tactic profiler + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false, /* skip */ true); + auto* obj = new GemmPlugin(transA, transB, padLda, padLdb, padLdc, type, useFp8, alpha, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* GemmPluginCreator::deserializePlugin(char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call GemmPlugin::destroy() + try + { + // GemmPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + // FIXME enable tactic profiler + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true, /* skip */ true); + auto* obj = new GemmPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.h b/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.h new file mode 100644 index 000000000000..1ba553c23d4b --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmPlugin/gemmPlugin.h @@ -0,0 +1,169 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef TRT_GEMM_PLUGIN_H +#define TRT_GEMM_PLUGIN_H + +#include "tensorrt_llm/common/cublasMMWrapper.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" + +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +using CublasGemmWrapper = tensorrt_llm::common::CublasMMWrapper; +using CublasGemmWrapperPtr = std::shared_ptr<CublasGemmWrapper>; + +class CublasLtGemmPluginProfiler + : public GemmPluginProfiler<cublasLtMatmulHeuristicResult_t, CublasGemmWrapperPtr, GemmIdCublas, GemmIdCublasHash> +{ +public: + using Config = cublasLtMatmulHeuristicResult_t; + + void setTranspose(bool transposeA, bool transposeB) + { + mTransA = transposeA; + mTransB = transposeB; + } + + void setPadLd(int padLda, int padLdb, int padLdc) + { + mPadLda = padLda; + mPadLdb = padLdb; + mPadLdc = padLdc; + } + + void setOutputType(nvinfer1::DataType type) + { + mOutputType = type; + } + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + bool checkTactic(int m, int n, int k, Config const& tactic) const override; + + std::vector<Config> getTactics(int m, int n, int k) const override; + +private: + bool mTransA; + bool mTransB; + int mPadLda; + int mPadLdb; + int mPadLdc; + nvinfer1::DataType mOutputType; + + static constexpr size_t ALIGNMENT = 256; +}; + +class GemmPlugin : public BasePlugin +{ +public: + using PluginProfilerPtr = std::shared_ptr<CublasLtGemmPluginProfiler>; + + GemmPlugin() = delete; + + GemmPlugin(int transA, int transB, int padLda, int padLdb, int padLdc, nvinfer1::DataType type, bool useFp8, + float alpha, PluginProfilerPtr const& profiler); + + GemmPlugin(void const* data, size_t length, PluginProfilerPtr const& profiler); + + ~GemmPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + void init(); + void configGemm(); + void setGemmConfig(); + +private: + const std::string mLayerName; + + int mTransA; + int mTransB; + int mPadLda; + int mPadLdb; + int mPadLdc; + int mArch; + nvinfer1::DataType mType; + nvinfer1::DataType mOutputType; + + static thread_local CublasGemmWrapperPtr mCublasWrapper; + + GemmDims mDims{}; + GemmIdCublas mGemmId{}; + bool mUseFp8{false}; + float mAlpha{1.f}; + + PluginProfilerPtr mPluginProfiler; +}; + +class GemmPluginCreator : public BaseCreator +{ +public: + GemmPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<CublasLtGemmPluginProfiler> gemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins + +#endif // TRT_GEMM_PLUGIN_H diff --git a/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/CMakeLists.txt new file mode 100644 index 000000000000..3b714a3928fb --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp *.cu) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.cpp b/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.cpp new file mode 100644 index 000000000000..ed964ace695f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.cpp @@ -0,0 +1,446 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "gemmSwigluPlugin.h" +#include "cutlass_extensions/gemm_configs.h" + +#include <NvInferRuntimeBase.h> +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels::cutlass_kernels; +using tensorrt_llm::plugins::GemmSwigluPluginCreator; +using tensorrt_llm::plugins::GemmSwigluPlugin; +using tensorrt_llm::plugins::GemmSwigluPluginProfiler; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +static char const* GEMM_SWIGLU_PLUGIN_VERSION{"1"}; +static char const* GEMM_SWIGLU_PLUGIN_NAME{"GemmSwiglu"}; +PluginFieldCollection GemmSwigluPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> GemmSwigluPluginCreator::mPluginAttributes; + +size_t GemmSwigluPluginProfiler::getBytePerElement(nvinfer1::DataType type) +{ + size_t bpe; + if (type == nvinfer1::DataType::kHALF || type == nvinfer1::DataType::kBF16) + { + bpe = 2; + } + else if (type == nvinfer1::DataType::kINT8 || type == nvinfer1::DataType::kFP8) + { + bpe = 1; + } + else + { + TLLM_THROW("Not recognized/implemented"); + } + return bpe; +} + +void GemmSwigluPluginProfiler::setQuantMode(tensorrt_llm::common::QuantMode const& quantMode) +{ + mQuantMode = quantMode; +} + +void GemmSwigluPluginProfiler::runTactic( + int m, int n, int k, GemmSwigluPluginProfiler::Config const& tactic, char* workspace, cudaStream_t const& stream) +{ + size_t bpe = getBytePerElement(mType); + + // Workspace size required by gemm runner + // NB: this function will throw exception when selected tactic exceeds SMEM, which is then + // caught by gemmPluginProfiler and it will register this tactic as invalid + size_t wsSizeRunner = mRunner->getWorkspaceSize(m, n, k); + + // Workspace size required by profiling + size_t wsByteOffset = 0; + int8_t* wsBytePointer = reinterpret_cast<int8_t*>(workspace); + void* aTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, m * k * bpe)); + void* bTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, n * k * bpe)); + void* cTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, 1 * n * bpe)); + void* dTmp = reinterpret_cast<void*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, m * (n / 2) * bpe)); + char* workspaceTmp = reinterpret_cast<char*>(nextWorkspacePtr(wsBytePointer, wsByteOffset, wsSizeRunner)); + + // Run profiling + mRunner->gemm( + dTmp, aTmp, bTmp, cTmp, mQuantMode, m, n, k, 1.0, 1.0, 1.0, tactic, workspaceTmp, wsSizeRunner, stream); +} + +int GemmSwigluPluginProfiler::getMaxProfileM() const +{ + return 32768; +} + +void GemmSwigluPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + std::vector<size_t> workspaces = { + maxM * k * getBytePerElement(mType), // A + n * k * getBytePerElement(mType), // B + 1 * n * getBytePerElement(mType), // C_bias + maxM * (n / 2) * getBytePerElement(mType), // D + mRunner->getWorkspaceSize(maxM, n, k) // workspace + }; + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<GemmSwigluPluginProfiler::Config> GemmSwigluPluginProfiler::getTactics(int m, int n, int k) const +{ + return mRunner->getConfigs(); +} + +GemmSwigluPlugin::GemmSwigluPlugin(QuantMode quantMode, nvinfer1::DataType type, bool hasBias, float scale_d0, + float scale_d1, float scale_output, GemmSwigluPlugin::PluginProfilerPtr const& pluginProfiler) + : mQuantMode(quantMode) + , mPluginProfiler(pluginProfiler) + , mHasBias(hasBias) + , mScaleD0(scale_d0) + , mScaleD1(scale_d1) + , mScaleOutput(scale_output) +{ + init(type); +} + +// Parameterized constructor +GemmSwigluPlugin::GemmSwigluPlugin( + void const* data, size_t length, GemmSwigluPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + nvinfer1::DataType type; + unsigned int quantMode; + read(d, quantMode); + read(d, type); + read(d, mHasBias); + read(d, mScaleD0); + read(d, mScaleD1); + read(d, mScaleOutput); + read(d, mDims); + + mQuantMode = QuantMode(quantMode); + + init(type); + + mPluginProfiler->deserialize(d, mDims, mGemmId); + + TLLM_CHECK(d == a + length); +} + +void GemmSwigluPlugin::init(nvinfer1::DataType type) +{ + mType = type; + if (mType == nvinfer1::DataType::kFP8) + { + mGemmRunner = std::make_shared<CutlassFusedGatedGemmRunner<__nv_fp8_e4m3>>(); + } + else + { + TLLM_THROW("Gemm Swiglu plugin only supports fp8 now"); + } + + mPluginProfiler->setQuantMode(mQuantMode); + + mGemmId = GemmIdCore(mDims.n, mDims.k, mType); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* GemmSwigluPlugin::clone() const noexcept +{ + auto* plugin = new GemmSwigluPlugin(*this); + return plugin; +} + +nvinfer1::DimsExprs GemmSwigluPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 3); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + TLLM_CHECK(nbDimsA >= 2); + DimsExprs ret; + ret.nbDims = nbDimsA; + for (int ii = 0; ii < nbDimsA - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + ret.d[nbDimsA - 1] = exprBuilder.constant(inputs[1].d[1]->getConstantValue() / 2); + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool GemmSwigluPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + switch (pos) + { + case 0: + // activation + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + case 1: + // weights + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + case 2: + // bias + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + case 3: + // out + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + default: + // Never should be here + TLLM_CHECK(false); + return false; + } +} + +void GemmSwigluPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + int const maxN = in[1].max.d[1]; + int const minK = in[0].min.d[in[0].min.nbDims - 1]; + int const minN = in[1].min.d[1]; + + TLLM_CHECK_WITH_INFO(minN == maxN, "Variable out channels is not allowed"); + TLLM_CHECK_WITH_INFO(minK == maxK, "Variable in channels is not allowed"); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, maxN, maxK}; + } + mGemmId = {maxN, maxK, mType}; + + mWorkspaceMaxSize = mGemmRunner->getWorkspaceSize(maxM, maxN, maxK); +} + +size_t GemmSwigluPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return mWorkspaceMaxSize; +} + +int GemmSwigluPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + // inputs + // mat1 [M(*), K] + // mat2 [K, N] + // bias [1, N] + // outputs + // mat [M(*), N / 2] + int m = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m *= inputDesc[0].dims.d[ii]; + } + int const n = inputDesc[1].dims.d[1]; + int const k = inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]; + size_t const wsSize = mGemmRunner->getWorkspaceSize(m, n, k); + + auto const bestTactic = mPluginProfiler->getBestConfig(m, mGemmId); + TLLM_CHECK_WITH_INFO(bestTactic, "No valid GEMM tactic"); + mGemmRunner->gemm(outputs[0], inputs[0], inputs[1], inputs[2], mQuantMode, m, n, k, mScaleD0, mScaleD1, + mScaleOutput, *bestTactic, reinterpret_cast<char*>(workspace), wsSize, stream); + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType GemmSwigluPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* GemmSwigluPlugin::getPluginType() const noexcept +{ + return GEMM_SWIGLU_PLUGIN_NAME; +} + +char const* GemmSwigluPlugin::getPluginVersion() const noexcept +{ + return GEMM_SWIGLU_PLUGIN_VERSION; +} + +int GemmSwigluPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int GemmSwigluPlugin::initialize() noexcept +{ + configGemm(); // gemm profiler in action + return 0; +} + +void GemmSwigluPlugin::terminate() noexcept {} + +size_t GemmSwigluPlugin::getSerializationSize() const noexcept +{ + return sizeof(unsigned int) + // QuantMode + sizeof(nvinfer1::DataType) + // dtype + sizeof(bool) + // hasBias + sizeof(float) * 3 + // scales + sizeof(mDims) + // Dimensions + mPluginProfiler->getSerializationSize(mGemmId); // selected tactics container size +} + +void GemmSwigluPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mQuantMode.value()); + write(d, mType); + write(d, mHasBias); + write(d, mScaleD0); + write(d, mScaleD1); + write(d, mScaleOutput); + write(d, mDims); + + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void GemmSwigluPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +void GemmSwigluPlugin::configGemm() +{ + mPluginProfiler->profileTactics(mGemmRunner, mType, mDims, mGemmId); +} + +/////////////// + +GemmSwigluPluginCreator::GemmSwigluPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("has_bias", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("scale_d0", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("scale_d1", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("scale_output", nullptr, PluginFieldType::kFLOAT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* GemmSwigluPluginCreator::getPluginName() const noexcept +{ + return GEMM_SWIGLU_PLUGIN_NAME; +} + +char const* GemmSwigluPluginCreator::getPluginVersion() const noexcept +{ + return GEMM_SWIGLU_PLUGIN_VERSION; +} + +PluginFieldCollection const* GemmSwigluPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* GemmSwigluPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + TLLM_CHECK(fc->nbFields == 5); + nvinfer1::DataType type{}; + bool hasBias{}; + float scale_d0{}; + float scale_d1{}; + float scale_output{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "has_bias")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + hasBias = static_cast<bool>(*(static_cast<int8_t const*>(fields[i].data))); + } + else if (!strcmp(attrName, "scale_d0")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + scale_d0 = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + else if (!strcmp(attrName, "scale_d1")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + scale_d1 = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + else if (!strcmp(attrName, "scale_output")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + scale_output = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + } + try + { + // GemmSwigluPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + auto pluginProfiler = mGemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false); + QuantMode quantMode = QuantMode{}; + auto* obj = new GemmSwigluPlugin(quantMode, type, hasBias, scale_d0, scale_d1, scale_output, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* GemmSwigluPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call GemmSwigluPlugin::destroy() + try + { + // Create plugin profiler with private tactics map which is read from the serialized engine + auto pluginProfiler = mGemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true); + auto* obj = new GemmSwigluPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.cu b/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.cu new file mode 100644 index 000000000000..339c432b1113 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.cu @@ -0,0 +1,41 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2023 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "gemmSwigluPlugin.h" + +#include "cutlass/util/reference/device/tensor_fill.h" +#include "cutlass_extensions/gemm_configs.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels::cutlass_kernels; +using tensorrt_llm::plugins::GemmSwigluPluginCreator; +using tensorrt_llm::plugins::GemmSwigluPlugin; +using tensorrt_llm::plugins::GemmSwigluPluginProfiler; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +void GemmSwigluPluginProfiler::initTmpData(int m, int n, int k, char* workspace, size_t size, cudaStream_t stream) +{ + size_t bpe = getBytePerElement(mType); + + if (mType == nvinfer1::DataType::kFP8) + { + cutlass::reference::device::BlockFillRandomUniform(reinterpret_cast<cutlass::float_e4m3_t*>(workspace), + m * k + n * k + 1 * n, 42, cutlass::float_e4m3_t{128}, -cutlass::float_e4m3_t{128}, -1, 0, stream); + } +} diff --git a/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.h b/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.h new file mode 100644 index 000000000000..766e59aad258 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gemmSwigluPlugin/gemmSwigluPlugin.h @@ -0,0 +1,150 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/kernels/cutlass_kernels/fused_gated_gemm/fused_gated_gemm.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +using GemmSwigluRunnerPtr + = std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassFusedGatedGemmRunnerInterface>; + +class GemmSwigluPluginProfiler : public GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + GemmSwigluRunnerPtr, GemmIdCore, GemmIdCoreHash> + +{ +public: + using Config = tensorrt_llm::cutlass_extensions::CutlassGemmConfig; + + void setQuantMode(tensorrt_llm::common::QuantMode const& quantMode); + + virtual int getMaxProfileM() const override; + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + // TODO(anchengc) implement checkTactic + // bool checkTactic(int m, int n, int k, const Config& tactic) const override; + + std::vector<Config> getTactics(int m, int n, int k) const override; + + void initTmpData(int m, int n, int k, char* workspace, size_t size, cudaStream_t stream) override; + +private: + size_t getBytePerElement(nvinfer1::DataType type); + + tensorrt_llm::common::QuantMode mQuantMode; +}; + +class GemmSwigluPlugin : public BasePlugin +{ +public: + using PluginProfilerPtr = std::shared_ptr<GemmSwigluPluginProfiler>; + + GemmSwigluPlugin() = delete; + + GemmSwigluPlugin(tensorrt_llm::common::QuantMode quantMode, nvinfer1::DataType type, bool hasBias, float scale_d0, + float scale_d1, float scale_output, PluginProfilerPtr const& pluginProfiler); + + GemmSwigluPlugin(void const* data, size_t length, PluginProfilerPtr const& profiler); + + ~GemmSwigluPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + void init(nvinfer1::DataType type); + + void configGemm(); + // void setGemmConfig(); + +private: + const std::string mLayerName; + + GemmSwigluRunnerPtr mGemmRunner; + tensorrt_llm::common::QuantMode mQuantMode; // not configurable yet + size_t mWorkspaceMaxSize; + + GemmDims mDims{}; + GemmIdCore mGemmId{}; + + PluginProfilerPtr mPluginProfiler; + + nvinfer1::DataType mType; + bool mHasBias; + float mScaleD0; + float mScaleD1; + float mScaleOutput; +}; + +class GemmSwigluPluginCreator : public BaseCreator +{ +public: + GemmSwigluPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<GemmSwigluPluginProfiler> mGemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gptAttentionCommon/CMakeLists.txt b/cpp/tensorrt_llm/plugins/gptAttentionCommon/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gptAttentionCommon/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.cpp b/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.cpp new file mode 100644 index 000000000000..717ab3083e5f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.cpp @@ -0,0 +1,380 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "gptAttentionCommon.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/kernels/decoderMaskedMultiheadAttention/decoderXQARunner.h" +#include "tensorrt_llm/kernels/gptKernels.h" +#include <NvInferRuntimePlugin.h> +#include <cstdint> + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +namespace tc = tensorrt_llm::common; +using tensorrt_llm::plugins::GPTAttentionPluginCreatorCommon; +using tensorrt_llm::plugins::GPTAttentionPluginCommon; + +GPTAttentionPluginCommon::GPTAttentionPluginCommon(int layer_idx, int num_heads, int vision_start, int vision_length, + int num_kv_heads, int num_kv_heads_origin, int head_size, int unidirectional, float q_scaling, + float attn_logit_softcapping_scale, tensorrt_llm::kernels::PositionEmbeddingType position_embedding_type, + int rotary_embedding_dim, // for RoPE. Use 0 for non-RoPE + float rotary_embedding_base, tensorrt_llm::kernels::RotaryScalingType rotary_embedding_scale_type, + float rotary_embedding_scale, float rotary_embedding_short_m_scale, float rotary_embedding_long_m_scale, + int rotary_embedding_max_positions, int rotary_embedding_original_max_positions, int tp_size, + int tp_rank, // for ALiBi + bool unfuse_qkv_gemm, // for AutoPP + bool use_logn_scaling, // for LognScaling + tensorrt_llm::kernels::ContextFMHAType context_fmha_type, int kv_cache_quant_mode, bool remove_input_padding, + tensorrt_llm::kernels::AttentionMaskType mask_type, tensorrt_llm::kernels::BlockSparseParams block_sparse_params, + bool paged_kv_cache, int tokens_per_block, nvinfer1::DataType type, int32_t max_context_length, + bool qkv_bias_enabled, bool cross_attention, int max_distance, bool pos_shift_enabled, bool dense_context_fmha, + bool use_paged_context_fmha, bool use_fp8_context_fmha, bool has_full_attention_mask, bool use_cache, + bool is_spec_decoding_enabled, bool spec_decoding_is_generation_length_variable, + int32_t spec_decoding_max_generation_length, bool is_mla_enabled, int q_lora_rank, int kv_lora_rank, + int qk_nope_head_dim, int qk_rope_head_dim, int v_head_dim, bool fuse_fp4_quant, bool skip_attn, int cp_size, + int cp_rank, std::set<int32_t> cp_group) + : mResource{DecoderXQARunner::getResourceGlobal()} +{ + mLayerIdx = layer_idx; + mNumHeads = num_heads; + mVisionStart = vision_start; + mVisionLength = vision_length; + mNumKVHeads = num_kv_heads; + mNumKVHeadsOrigin = num_kv_heads_origin; + mHeadSize = head_size; + mUnidirectional = unidirectional; + mQScaling = q_scaling; + mAttnLogitSoftcappingScale = attn_logit_softcapping_scale; + mRotaryEmbeddingDim = rotary_embedding_dim; + mRotaryEmbeddingBase = rotary_embedding_base; + mRotaryEmbeddingScaleType = rotary_embedding_scale_type; + mRotaryEmbeddingScale = rotary_embedding_scale; + mRotaryEmbeddingShortMscale = rotary_embedding_short_m_scale; + mRotaryEmbeddingLongMscale = rotary_embedding_long_m_scale; + mRotaryEmbeddingMaxPositions = rotary_embedding_max_positions; + mRotaryEmbeddingOriginalMaxPositions = rotary_embedding_original_max_positions; + mPositionEmbeddingType = position_embedding_type; + mEnableContextFMHA = context_fmha_type != ContextFMHAType::DISABLED; + mFMHAForceFP32Acc = type == nvinfer1::DataType::kBF16; + mMaskType = mask_type; + mBlockSparseParams = block_sparse_params; + mType = type; + mMultiBlockMode = true; + mEnableXQA = true; + mKVCacheQuantMode = tc::QuantMode(kv_cache_quant_mode); + mRemovePadding = remove_input_padding; + mPagedKVCache = paged_kv_cache; + mTokensPerBlock = tokens_per_block; + mTpSize = tp_size; + mTpRank = tp_rank; + mUnfuseQkvGemm = unfuse_qkv_gemm; + mUseLognScaling = use_logn_scaling; + mMaxContextLength = max_context_length; + mQKVBiasEnabled = qkv_bias_enabled; + mCrossAttention = cross_attention; + mMaxDistance = max_distance; + mPosShiftEnabled = pos_shift_enabled; + mDenseContextFMHA = dense_context_fmha; + mPagedContextFMHA = use_paged_context_fmha; + mFP8ContextFMHA = use_fp8_context_fmha; + mFP8AttenOutput = use_fp8_context_fmha; + mHasFullAttentionMask = has_full_attention_mask; + mUseKVCache = use_cache; + mIsSpecDecodingEnabled = is_spec_decoding_enabled; + mSpecDecodingIsGenerationLengthVariable = spec_decoding_is_generation_length_variable; + mSpecDecodingMaxGenerationLength = spec_decoding_max_generation_length; + mIsMLAEnabled = is_mla_enabled; + mMLAParams = {q_lora_rank, kv_lora_rank, qk_nope_head_dim, qk_rope_head_dim, v_head_dim}; + mCpSize = cp_size; + mCpRank = cp_rank; + mCpGroup = std::move(cp_group); + mFuseFp4Quant = fuse_fp4_quant; + mSkipAttn = skip_attn; +} + +// Parameterized constructor +GPTAttentionPluginCommon::GPTAttentionPluginCommon(void const* data, size_t length) + : mResource{DecoderXQARunner::getResourceGlobal()} +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + unsigned int kvCacheQuantMode; + + read(d, mLayerIdx); + read(d, mNumHeads); + read(d, mVisionStart); + read(d, mVisionLength); + read(d, mNumKVHeads); + read(d, mNumKVHeadsOrigin); + read(d, mHeadSize); + read(d, mUnidirectional); + read(d, mQScaling); + read(d, mAttnLogitSoftcappingScale); + read(d, mPositionEmbeddingType); + read(d, mRotaryEmbeddingDim); + read(d, mRotaryEmbeddingBase); + read(d, mRotaryEmbeddingScaleType); + read(d, mRotaryEmbeddingScale); + read(d, mRotaryEmbeddingShortMscale); + read(d, mRotaryEmbeddingLongMscale); + read(d, mRotaryEmbeddingMaxPositions); + read(d, mRotaryEmbeddingOriginalMaxPositions); + read(d, mTpSize); + read(d, mTpRank); + read(d, mUnfuseQkvGemm); + read(d, mUseLognScaling); + read(d, mEnableContextFMHA); + read(d, mFMHAForceFP32Acc); + read(d, mMultiBlockMode); + read(d, mEnableXQA); + read(d, kvCacheQuantMode); + read(d, mRemovePadding); + read(d, mMaskType); + read(d, mBlockSparseParams); + read(d, mPagedKVCache); + read(d, mTokensPerBlock); + read(d, mType); + read(d, mMaxContextLength); + read(d, mQKVBiasEnabled); + read(d, mCrossAttention); + read(d, mMaxDistance); + read(d, mPosShiftEnabled); + read(d, mDenseContextFMHA); + read(d, mPagedContextFMHA); + read(d, mFP8ContextFMHA); + read(d, mFP8AttenOutput); + read(d, mHasFullAttentionMask); + read(d, mUseKVCache); + read(d, mIsSpecDecodingEnabled); + read(d, mUseSpecDecoding); + read(d, mSpecDecodingIsGenerationLengthVariable); + read(d, mSpecDecodingMaxGenerationLength); + read(d, mIsMLAEnabled); + read(d, mMLAParams); + read(d, mNbMultiBlockSemaphores); + read(d, mFuseFp4Quant); + read(d, mSkipAttn); + read(d, mCpSize); + read(d, mCpRank); + + mKVCacheQuantMode = tc::QuantMode(kvCacheQuantMode); + + uint32_t decoderXQARunnerResourceSerializedSize; + read(d, decoderXQARunnerResourceSerializedSize); + mResource->merge(DecoderXQARunnerResource(d, decoderXQARunnerResourceSerializedSize), /*initialize=*/true); + d += decoderXQARunnerResourceSerializedSize; + + mCpGroup.clear(); + int32_t groupItem = 0; + while (d != a + length) + { + read(d, groupItem); + mCpGroup.insert(groupItem); + } + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); + TLLM_CHECK_WITH_INFO((smVersion() >= 80) || (mType != nvinfer1::DataType::kBF16), + "Unsupported data type, pre SM 80 GPUs do not support bfloat16"); +} + +int GPTAttentionPluginCommon::initialize() noexcept +{ + return AttentionOp::initialize(); +} + +void GPTAttentionPluginCommon::destroy() noexcept +{ + delete this; +} + +size_t GPTAttentionPluginCommon::getCommonSerializationSize() const noexcept +{ + return sizeof(mLayerIdx) + sizeof(mNumHeads) + +sizeof(mVisionStart) + sizeof(mVisionLength) + sizeof(mNumKVHeads) + + sizeof(mNumKVHeadsOrigin) + sizeof(mHeadSize) + sizeof(mUnidirectional) + sizeof(mQScaling) + + sizeof(mAttnLogitSoftcappingScale) + sizeof(mPositionEmbeddingType) + sizeof(mRotaryEmbeddingDim) + + sizeof(mRotaryEmbeddingBase) + sizeof(mRotaryEmbeddingScaleType) + sizeof(mRotaryEmbeddingScale) + + sizeof(mRotaryEmbeddingShortMscale) + sizeof(mRotaryEmbeddingLongMscale) + + sizeof(mRotaryEmbeddingMaxPositions) + sizeof(mRotaryEmbeddingOriginalMaxPositions) + sizeof(mTpSize) + + sizeof(mTpRank) + sizeof(mEnableContextFMHA) + sizeof(mFMHAForceFP32Acc) + sizeof(mMultiBlockMode) + + sizeof(mEnableXQA) + sizeof(unsigned int) // mKVCacheQuantMode + + sizeof(mRemovePadding) + sizeof(mMaskType) + sizeof(mBlockSparseParams) + sizeof(mPagedKVCache) + + sizeof(mTokensPerBlock) + sizeof(mType) + sizeof(mMaxContextLength) + sizeof(mQKVBiasEnabled) + + sizeof(mCrossAttention) + sizeof(mMaxDistance) + sizeof(mPosShiftEnabled) + sizeof(mDenseContextFMHA) + + sizeof(mPagedContextFMHA) + sizeof(mFP8ContextFMHA) + sizeof(mFP8AttenOutput) + sizeof(mHasFullAttentionMask) + + sizeof(mUseKVCache) + sizeof(mUnfuseQkvGemm) + sizeof(mUseLognScaling) + sizeof(mIsSpecDecodingEnabled) + + sizeof(mUseSpecDecoding) + sizeof(mSpecDecodingIsGenerationLengthVariable) + + sizeof(mSpecDecodingMaxGenerationLength) + sizeof(mNbMultiBlockSemaphores) + sizeof(mIsMLAEnabled) + + sizeof(mMLAParams) + sizeof(mFuseFp4Quant) + sizeof(mSkipAttn) + + sizeof(uint32_t) // size of DecoderXQARunnerResource buffer. + + sizeof(mCpSize) + sizeof(mCpRank) + sizeof(int32_t) * mCpGroup.size() + mResource->getSerializationSize(); +} + +void GPTAttentionPluginCommon::serializeCommon(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mLayerIdx); + write(d, mNumHeads); + write(d, mVisionStart); + write(d, mVisionLength); + write(d, mNumKVHeads); + write(d, mNumKVHeadsOrigin); + write(d, mHeadSize); + write(d, mUnidirectional); + write(d, mQScaling); + write(d, mAttnLogitSoftcappingScale); + write(d, mPositionEmbeddingType); + write(d, mRotaryEmbeddingDim); + write(d, mRotaryEmbeddingBase); + write(d, mRotaryEmbeddingScaleType); + write(d, mRotaryEmbeddingScale); + write(d, mRotaryEmbeddingShortMscale); + write(d, mRotaryEmbeddingLongMscale); + write(d, mRotaryEmbeddingMaxPositions); + write(d, mRotaryEmbeddingOriginalMaxPositions); + write(d, mTpSize); + write(d, mTpRank); + write(d, mUnfuseQkvGemm); + write(d, mUseLognScaling); + write(d, mEnableContextFMHA); + write(d, mFMHAForceFP32Acc); + write(d, mMultiBlockMode); + write(d, mEnableXQA); + write(d, mKVCacheQuantMode.value()); + write(d, mRemovePadding); + write(d, mMaskType); + write(d, mBlockSparseParams); + write(d, mPagedKVCache); + write(d, mTokensPerBlock); + write(d, mType); + write(d, mMaxContextLength); + write(d, mQKVBiasEnabled); + write(d, mCrossAttention); + write(d, mMaxDistance); + write(d, mPosShiftEnabled); + write(d, mDenseContextFMHA); + write(d, mPagedContextFMHA); + write(d, mFP8ContextFMHA); + write(d, mFP8AttenOutput); + write(d, mHasFullAttentionMask); + write(d, mUseKVCache); + write(d, mIsSpecDecodingEnabled); + write(d, mUseSpecDecoding); + write(d, mSpecDecodingIsGenerationLengthVariable); + write(d, mSpecDecodingMaxGenerationLength); + write(d, mIsMLAEnabled); + write(d, mMLAParams); + write(d, mNbMultiBlockSemaphores); + write(d, mFuseFp4Quant); + write(d, mSkipAttn); + write(d, mCpSize); + write(d, mCpRank); + + // An uint32_t that specifies the size of the serialized buffer, followed by the actual content. + uint32_t decoderXQARunnerResourceSerializedSize = mResource->getSerializationSize(); + write(d, decoderXQARunnerResourceSerializedSize); + mResource->serialize(d, decoderXQARunnerResourceSerializedSize); + d += decoderXQARunnerResourceSerializedSize; + + for (auto it = mCpGroup.begin(); it != mCpGroup.end(); ++it) + { + write(d, *it); + } + TLLM_CHECK(d == a + getCommonSerializationSize()); +} + +void GPTAttentionPluginCommon::terminate() noexcept +{ + // Do nothing, destroy will always be called, so release the resources there. +} + +/////////////// + +GPTAttentionPluginCreatorCommon::GPTAttentionPluginCreatorCommon() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("layer_idx", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("num_heads", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("vision_start", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("vision_length", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("num_kv_heads", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("num_kv_heads_origin", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("layer_idx_in_cache_pool", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("head_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("unidirectional", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("q_scaling", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("attn_logit_softcapping_scale", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("position_embedding_type", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("rotary_embedding_dim", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("rotary_embedding_base", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("rotary_embedding_scale_type", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("rotary_embedding_scale", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("rotary_embedding_short_m_scale", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("rotary_embedding_long_m_scale", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("rotary_embedding_max_positions", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back( + PluginField("rotary_embedding_original_max_positions", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("tp_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("tp_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("unfuse_qkv_gemm", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("use_logn_scaling", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("context_fmha_type", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("kv_cache_quant_mode", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("remove_input_padding", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("mask_type", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("block_sparse_block_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("block_sparse_homo_head_pattern", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("block_sparse_num_local_blocks", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("block_sparse_vertical_stride", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("paged_kv_cache", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("tokens_per_block", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("max_context_length", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("qkv_bias_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("do_cross_attention", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("max_distance", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("pos_shift_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("dense_context_fmha", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("use_paged_context_fmha", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("use_fp8_context_fmha", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("has_full_attention_mask", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("use_cache", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("is_spec_decoding_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back( + PluginField("spec_decoding_is_generation_length_variable", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back( + PluginField("spec_decoding_max_generation_length", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("is_mla_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("q_lora_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("kv_lora_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("qk_nope_head_dim", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("qk_rope_head_dim", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("v_head_dim", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("fuse_fp4_quant", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("skip_attn", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("cp_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("cp_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("cp_group", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +PluginFieldCollection const* GPTAttentionPluginCreatorCommon::getFieldNames() noexcept +{ + return &mFC; +} diff --git a/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.h b/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.h new file mode 100644 index 000000000000..dd87d67aab9e --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.h @@ -0,0 +1,112 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/attentionOp.h" +#include "tensorrt_llm/common/cublasMMWrapper.h" +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/kernels/gptKernels.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::kernels +{ +class DecoderXQARunnerResource; +} + +namespace tensorrt_llm::plugins +{ + +class GPTAttentionPluginCommon : public BasePlugin, public tensorrt_llm::common::op::AttentionOp +{ +public: + GPTAttentionPluginCommon() = delete; + + GPTAttentionPluginCommon(int layer_idx, int num_heads, int vision_start, int vision_length, int num_kv_heads, + int num_kv_heads_origin, int head_size, int unidirectional, float q_scaling, float attn_logit_softcapping_scale, + tensorrt_llm::kernels::PositionEmbeddingType position_embedding_type, + int rotary_embedding_dim, // for RoPE. Use 0 for non-RoPE + float rotary_embedding_base, tensorrt_llm::kernels::RotaryScalingType rotary_embedding_scale_type, + float rotary_embedding_scale, float rotary_embedding_short_m_scale, float rotary_embedding_long_m_scale, + int rotary_embedding_max_positions, int rotary_embedding_original_max_positions, int tp_size, + int tp_rank, // for ALiBi + bool unfuse_qkv_gemm, // for AutoPP + bool use_logn_scaling, // for LognScaling + tensorrt_llm::kernels::ContextFMHAType context_fmha_type, int kv_cache_quant_mode, bool remove_input_padding, + tensorrt_llm::kernels::AttentionMaskType mask_type, + tensorrt_llm::kernels::BlockSparseParams block_sparse_params, bool paged_kv_cache, int tokens_per_block, + nvinfer1::DataType type, int32_t max_context_length, bool qkv_bias_enabled, bool cross_attention = false, + int max_distance = 0, bool pos_shift_enabled = false, bool dense_context_fmha = false, + bool use_paged_context_fmha = true, bool use_fp8_context_fmha = true, bool has_full_attention_mask = false, + bool use_cache = true, bool is_spec_decoding_enabled = false, + bool spec_decoding_is_generation_length_variable = false, int32_t spec_decoding_max_generation_length = 1, + bool is_mla_enabled = false, int q_lora_rank = 0, int kv_lora_rank = 0, int qk_nope_head_dim = 0, + int qk_rope_head_dim = 0, int v_head_dim = 0, bool fuse_fp4_quant = false, bool skip_attn = false, + int cp_size = 1, int cp_rank = 0, std::set<int32_t> cp_group = {}); + + GPTAttentionPluginCommon(void const* data, size_t length); + + ~GPTAttentionPluginCommon() override = default; + + template <typename T> + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + //! This is called on every trt Engine creation + int initialize() noexcept override; + //! This is called on every trt Engine destroy + void terminate() noexcept override; + + //! This is called on every trt ExecutionContext creation by TRT + //! Note TRT does not call the initialize on cloned plugin, so clone internally should do initialization. + template <typename T> + T* cloneImpl() const noexcept; + + //! This is called on evert trt Engine or ExecutionContext destroy. + //! None-cloned plugins will call terminate and then call destroy, while the cloned plugins will call destroy only + //! So plugin should put the resource release inside destroy. + void destroy() noexcept override; + + size_t getCommonSerializationSize() const noexcept; + void serializeCommon(void* buffer) const noexcept; + +protected: + std::string const mLayerName; + +private: + std::shared_ptr<tensorrt_llm::kernels::DecoderXQARunnerResource> mResource; +}; + +class GPTAttentionPluginCreatorCommon : public BaseCreator +{ +public: + GPTAttentionPluginCreatorCommon(); + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + template <typename T> + T* deserializePluginImpl(char const* name, void const* serialData, size_t serialLength) noexcept; + +protected: + std::vector<nvinfer1::PluginField> mPluginAttributes; + nvinfer1::PluginFieldCollection mFC{}; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommonImpl.h b/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommonImpl.h new file mode 100644 index 000000000000..51462cee6f40 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommonImpl.h @@ -0,0 +1,54 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "gptAttentionCommon.h" + +namespace tensorrt_llm::plugins +{ +template <typename T> +T* GPTAttentionPluginCommon::cloneImpl() const noexcept +{ + static_assert(std::is_base_of_v<GPTAttentionPluginCommon, T>); + auto* plugin = new T(static_cast<T const&>(*this)); + plugin->setPluginNamespace(mNamespace.c_str()); + + // Cloned plugins should be in initialized state with correct resources ready to be enqueued. + plugin->initialize(); + return plugin; +} + +template <typename T> +T* GPTAttentionPluginCreatorCommon::deserializePluginImpl( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call GPTAttentionPluginCommon::destroy() + try + { + auto* obj = new T(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/gptAttentionPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/gptAttentionPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gptAttentionPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/gptAttentionPlugin/gptAttentionPlugin.cpp b/cpp/tensorrt_llm/plugins/gptAttentionPlugin/gptAttentionPlugin.cpp new file mode 100644 index 000000000000..6f8c41c94131 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gptAttentionPlugin/gptAttentionPlugin.cpp @@ -0,0 +1,1387 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "gptAttentionPlugin.h" + +#include "tensorrt_llm/batch_manager/contextProgress.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/kernels/decoderMaskedMultiheadAttention.h" +#include "tensorrt_llm/kernels/gptKernels.h" +#include "tensorrt_llm/kernels/unfusedAttentionKernels.h" +#include "tensorrt_llm/plugins/common/checkMacrosPlugin.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.h" +#include "tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommonImpl.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/utils/debugUtils.h" + +#include <NvInferRuntimeBase.h> +#include <algorithm> +#include <cstdint> +#include <functional> +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::GPTAttentionPluginCreator; +using tensorrt_llm::plugins::GPTAttentionPlugin; + +static char const* GPT_ATTENTION_PLUGIN_VERSION{"1"}; +static char const* GPT_ATTENTION_PLUGIN_NAME{"GPTAttention"}; + +GPTAttentionPlugin::GPTAttentionPlugin(int layer_idx, int num_heads, int vision_start, int vision_length, + int num_kv_heads, int num_kv_heads_origin, int head_size, int unidirectional, float q_scaling, + float attn_logit_softcapping_scale, tensorrt_llm::kernels::PositionEmbeddingType position_embedding_type, + int rotary_embedding_dim, // for RoPE. 0 for non-RoPE + float rotary_embedding_base, tensorrt_llm::kernels::RotaryScalingType rotary_embedding_scale_type, + float rotary_embedding_scale, float rotary_embedding_short_m_scale, + float rotary_embedding_long_m_scale, // magnitude scaling factors for Phi-3 long RoPE + int rotary_embedding_max_positions, int rotary_embedding_original_max_positions, int tp_size, + int tp_rank, // for ALiBi + bool unfuse_qkv_gemm, // for AutoPP + bool use_logn_scaling, // for LognScaling + tensorrt_llm::kernels::ContextFMHAType context_fmha_type, int kv_cache_quant_mode, bool remove_input_padding, + tensorrt_llm::kernels::AttentionMaskType mask_type, tensorrt_llm::kernels::BlockSparseParams block_sparse_params, + bool paged_kv_cache, int tokens_per_block, nvinfer1::DataType type, int32_t max_context_length, + bool qkv_bias_enabled, bool cross_attention, int max_distance, bool pos_shift_enabled, bool dense_context_fmha, + bool use_paged_context_fmha, bool use_fp8_context_fmha, bool has_full_attention_mask, bool use_cache, + bool is_spec_decoding_enabled, bool spec_decoding_is_generation_length_variable, + int spec_decoding_max_generation_length, bool is_mla_enabled, int q_lora_rank, int kv_lora_rank, + int qk_nope_head_dim, int qk_rope_head_dim, int v_head_dim, bool fuse_fp4_quant, bool skip_attn, int cp_size, + int cp_rank, std::set<int32_t> cp_group) + : GPTAttentionPluginCommon(layer_idx, num_heads, vision_start, vision_length, num_kv_heads, num_kv_heads_origin, + head_size, unidirectional, q_scaling, attn_logit_softcapping_scale, position_embedding_type, + rotary_embedding_dim, rotary_embedding_base, rotary_embedding_scale_type, rotary_embedding_scale, + rotary_embedding_short_m_scale, rotary_embedding_long_m_scale, rotary_embedding_max_positions, + rotary_embedding_original_max_positions, tp_size, tp_rank, unfuse_qkv_gemm, use_logn_scaling, context_fmha_type, + kv_cache_quant_mode, remove_input_padding, mask_type, block_sparse_params, paged_kv_cache, tokens_per_block, + type, max_context_length, qkv_bias_enabled, cross_attention, max_distance, pos_shift_enabled, + dense_context_fmha, use_paged_context_fmha, use_fp8_context_fmha, has_full_attention_mask, use_cache, + is_spec_decoding_enabled, spec_decoding_is_generation_length_variable, spec_decoding_max_generation_length, + is_mla_enabled, q_lora_rank, kv_lora_rank, qk_nope_head_dim, qk_rope_head_dim, v_head_dim, fuse_fp4_quant, + skip_attn, cp_size, cp_rank, cp_group) +{ + TLLM_CHECK_WITH_INFO( + !is_mla_enabled, "GPTAttentionPlugin no longer supports MLA. Please use the PyTorch workflow instead."); + initEntryIdx(); +} + +GPTAttentionPlugin::GPTAttentionPlugin(void const* data, size_t length) + : GPTAttentionPluginCommon(data, length) +{ + initEntryIdx(); +} + +std::string GPTAttentionPlugin::toString(IdxEntry const& entry) const +{ +#define TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(name) \ + case IdxEntry::name: return #name + + switch (entry) + { + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(QKV_TENSOR); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(K_TENSOR); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(V_TENSOR); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ATTENTION_MASK); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ATTENTION_PACKED_MASK); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(SEQUENCE_LENGTH); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_PAST_KEY_VALUE_LENGTHS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_MAX_ATTENTION_WINDOW); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_SINK_TOKEN_LENGTH); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(CONTEXT_LENGTHS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(CACHE_INDIR); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(REQUEST_TYPES); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(KV_CACHE_BLOCK_OFFSETS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_KV_CACHE_BLOCK_OFFSETS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_KV_CACHE_POOL_POINTERS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_KV_CACHE_POOL_MAPPING); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(PAST_KEY_VALUE); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(KV_CACHE_QUANTIZATION_SCALE); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(KV_CACHE_DEQUANTIZATION_SCALE); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ATTENTION_OUTPUT_QUANTIZATION_SCALE); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ATTENTION_OUTPUT_SF_SCALE); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ROTARY_INV_FREQ); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ROTARY_COS_SIN); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ALIBI_SLOPES); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(RELATIVE_ATTENTION_BIAS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(CROSS_KV); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(CROSS_KV_LENGTH); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ENCODER_INPUT_LENGTH); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_CONTEXT_LENGTH); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(QKV_BIAS_TENSOR); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(SPEC_DECODING_GENERATION_LENGTHS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(SPEC_DECODING_PACKED_MASK); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(SPEC_DECODING_POSITION_OFFSETS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(SPEC_DECODING_USE); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(LONG_ROPE_ROTARY_INV_FREQ); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(LONG_ROPE_ROTARY_COS_SIN); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(MROPE_ROTARY_COS_SIN); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(MROPE_POSITION_DELTAS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_RUNTIME_PERF_KNOBS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(HOST_CONTEXT_PROGRESS); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(MLA_Q_B_PROJ_TENSOR); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(MLA_KV_B_PROJ_TENSOR); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(MLA_K_B_PROJ_TRANS_TENSOR); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(SKIP_ATTN); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(LOGN_SCALING); + TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING(ENUM_SIZE); + } +#undef TLLM_GPT_ATTN_IDX_ENTRY_TO_STRING + TLLM_LOG_TRACE(common::fmtstr("Missing string description for IdxEntry enum %lu.\n", static_cast<size_t>(entry))); + return ""; +} + +bool GPTAttentionPlugin::isEntryUsed(IdxEntry const& entry) const +{ + switch (entry) + { + case IdxEntry::QKV_TENSOR: return true; + case IdxEntry::K_TENSOR: return mUnfuseQkvGemm; + case IdxEntry::V_TENSOR: return mUnfuseQkvGemm; + case IdxEntry::ATTENTION_MASK: return useFullCustomMask(); + case IdxEntry::ATTENTION_PACKED_MASK: return useCustomMask(); + case IdxEntry::SEQUENCE_LENGTH: return useKVCache(); + case IdxEntry::HOST_PAST_KEY_VALUE_LENGTHS: return useKVCache(); + case IdxEntry::HOST_MAX_ATTENTION_WINDOW: return true; + case IdxEntry::HOST_SINK_TOKEN_LENGTH: return true; + case IdxEntry::CONTEXT_LENGTHS: return true; + case IdxEntry::CACHE_INDIR: return useKVCache(); + case IdxEntry::REQUEST_TYPES: return true; + case IdxEntry::KV_CACHE_BLOCK_OFFSETS: return useKVCache() && mPagedKVCache; + case IdxEntry::HOST_KV_CACHE_BLOCK_OFFSETS: return useKVCache() && mPagedKVCache; + case IdxEntry::HOST_KV_CACHE_POOL_POINTERS: return useKVCache() && mPagedKVCache; + case IdxEntry::HOST_KV_CACHE_POOL_MAPPING: return useKVCache() && mPagedKVCache; + case IdxEntry::PAST_KEY_VALUE: return useKVCache() && !mPagedKVCache; + case IdxEntry::KV_CACHE_QUANTIZATION_SCALE: return useKVCache() && mKVCacheQuantMode.hasKvCacheQuant(); + case IdxEntry::KV_CACHE_DEQUANTIZATION_SCALE: return useKVCache() && mKVCacheQuantMode.hasKvCacheQuant(); + case IdxEntry::ATTENTION_OUTPUT_QUANTIZATION_SCALE: return mFP8ContextFMHA; + case IdxEntry::ATTENTION_OUTPUT_SF_SCALE: return mFuseFp4Quant; + case IdxEntry::ROTARY_INV_FREQ: return isRoPE(); + case IdxEntry::ROTARY_COS_SIN: return isRoPE(); + case IdxEntry::ALIBI_SLOPES: return isALiBi(); + case IdxEntry::RELATIVE_ATTENTION_BIAS: return isRelativePosition(); + case IdxEntry::CROSS_KV: return isCrossAttention(); + case IdxEntry::CROSS_KV_LENGTH: return isCrossAttention(); + case IdxEntry::LOGN_SCALING: return isLognScaling(); + case IdxEntry::ENCODER_INPUT_LENGTH: return isCrossAttention(); + case IdxEntry::HOST_CONTEXT_LENGTH: return mRemovePadding; + case IdxEntry::QKV_BIAS_TENSOR: return mQKVBiasEnabled; + case IdxEntry::SPEC_DECODING_GENERATION_LENGTHS: return mIsSpecDecodingEnabled; + case IdxEntry::SPEC_DECODING_PACKED_MASK: return mIsSpecDecodingEnabled; + case IdxEntry::SPEC_DECODING_POSITION_OFFSETS: return mIsSpecDecodingEnabled; + case IdxEntry::SPEC_DECODING_USE: return mIsSpecDecodingEnabled; + case IdxEntry::LONG_ROPE_ROTARY_INV_FREQ: return isLongRoPE(); + case IdxEntry::LONG_ROPE_ROTARY_COS_SIN: return isLongRoPE(); + case IdxEntry::MROPE_ROTARY_COS_SIN: return isMRoPE(); + case IdxEntry::MROPE_POSITION_DELTAS: return isMRoPE(); + case IdxEntry::HOST_RUNTIME_PERF_KNOBS: return true; + case IdxEntry::HOST_CONTEXT_PROGRESS: return true; + case IdxEntry::MLA_Q_B_PROJ_TENSOR: return mIsMLAEnabled; + case IdxEntry::MLA_KV_B_PROJ_TENSOR: return mIsMLAEnabled; + case IdxEntry::MLA_K_B_PROJ_TRANS_TENSOR: return mIsMLAEnabled; + case IdxEntry::SKIP_ATTN: return mSkipAttn; + default: return false; + } +} + +void GPTAttentionPlugin::initEntryIdx() +{ + mEntryIdx.resize(static_cast<size_t>(IdxEntry::ENUM_SIZE)); + size_t entryIdx = 0; + for (size_t i = 0; i < static_cast<size_t>(IdxEntry::ENUM_SIZE); i++) + { + mEntryIdx[i] = entryIdx; + entryIdx += isEntryUsed(static_cast<IdxEntry>(i)); + } +} + +GPTAttentionPlugin::IndexType GPTAttentionPlugin::getIdx(IdxEntry const& entry) const +{ + TLLM_CHECK_WITH_INFO( + isEntryUsed(entry), common::fmtstr("getIdx() should not be used with entry %s.\n", toString(entry).data())); + return mEntryIdx[static_cast<size_t>(entry)]; +} + +// IPluginV2DynamicExt Methods +GPTAttentionPlugin* GPTAttentionPlugin::clone() const noexcept +{ + return dynamic_cast<GPTAttentionPlugin*>(this->cloneImpl<GPTAttentionPlugin>()); +} + +static int getPackedTensorHiddenDimIndex(bool removePadding) +{ + return removePadding ? 1 : 2; +} + +// NOTE: generation input length might be larger than one in the spec decoding mode. +int GPTAttentionPlugin::getGenerationInputSequenceLength( + nvinfer1::PluginTensorDesc const* inputDesc, int32_t localNbSeq, int32_t localNbTokens) const +{ + if (mRemovePadding) + { + // Speculative decoding mode might need variable generation input sequence length. + if (mIsSpecDecodingEnabled && mUseSpecDecoding) + { + TLLM_CHECK_WITH_INFO(mCpSize <= 1, "Context Parallel does not support speculative decoding mode for now"); + // SPEC_DECODING_POSITION_OFFSETS: [batch_size, max_generation_input_length]. + return inputDesc[getIdx(IdxEntry::SPEC_DECODING_POSITION_OFFSETS)].dims.d[1]; + } + else + { + if (mCpSize > 1) + { + // Given that localNbTokens == (beamSize * localNbSeq + mCpSize - 1) / mCpSize, but when mCpSize - 1 > + // localNbSeq, there are multiple choices for beamSize. Assume beamSize == 1 here. + TLLM_CHECK_WITH_INFO(localNbTokens == (localNbSeq + mCpSize - 1) / mCpSize, + "Context Parallel does not support beamSize > 1 for non-speculative decoding mode, " + "localNbTokens=%d, localNbSeq=%d", + localNbTokens, localNbSeq); + return 1; + } + // [num_tokens, local_hidden_size] where num_tokens = batch_size * generation_input_length + TLLM_CHECK_WITH_INFO(localNbTokens % localNbSeq == 0, + "seq_len should be same for all generation requests, localNbTokens=%d, localNbSeq=%d", localNbTokens, + localNbSeq); + return localNbTokens / localNbSeq; + } + } + else + { + // We don't have IFB without mRemovePadding, so just take it out from inputDesc + // [batch_size, seq_len, local_hidden_size] + return inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[1]; + } +} + +// outputs +// output_tensor [batch_size, seq_len, local_hidden_size] or [num_tokens, local_hidden_size] +// present_key_value_pool (optional if mPagedKVCache is false) [batch_size, 2, local_num_kv_heads, max_seq_len, +// head_size] +nvinfer1::DimsExprs GPTAttentionPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + if (mFuseFp4Quant) + { + TLLM_CHECK(outputIndex == 0 || outputIndex == 1 || (!mPagedKVCache && useKVCache() && outputIndex == 2)); + // Compute the output dimension for FP4 quantized tensor. Consistent with QuantizeToFP4Plugin. + if (outputIndex == 0) + { + auto ret = inputs[getIdx(IdxEntry::QKV_TENSOR)]; + return ret; + } + // Compute the output dimension for output scaling factor tensor. Consistent with QuantizeToFP4Plugin. + if (outputIndex == 1) + { + auto ret = inputs[getIdx(IdxEntry::QKV_TENSOR)]; + // Sequence dimension or token dimension. + // Pad to multiple of 128. + auto dimM = exprBuilder.operation(DimensionOperation::kCEIL_DIV, + *ret.d[getPackedTensorHiddenDimIndex(mRemovePadding) - 1], *exprBuilder.constant(128)); + ret.d[getPackedTensorHiddenDimIndex(mRemovePadding) - 1] + = exprBuilder.operation(DimensionOperation::kPROD, *dimM, *exprBuilder.constant(128)); + // Hidden size dimension. + // Div (rounding up) by 16 since 16 elements share one SF and SF padded to k%4==0. + ret.d[getPackedTensorHiddenDimIndex(mRemovePadding)] = exprBuilder.operation(DimensionOperation::kCEIL_DIV, + *ret.d[getPackedTensorHiddenDimIndex(mRemovePadding)], *exprBuilder.constant(16)); + return ret; + } + } + else + { + TLLM_CHECK(outputIndex == 0 || (!mPagedKVCache && useKVCache() && outputIndex == 1)); + if (outputIndex == 0) + { + auto ret = inputs[getIdx(IdxEntry::QKV_TENSOR)]; + // In MLA, the output dim is v_head_dim + auto const head_size = mHeadSize; + ret.d[getPackedTensorHiddenDimIndex(mRemovePadding)] = exprBuilder.operation( + DimensionOperation::kPROD, *exprBuilder.constant(head_size), *exprBuilder.constant(mNumHeads)); + return ret; + } + } + return inputs[getIdx(IdxEntry::PAST_KEY_VALUE)]; +} + +bool GPTAttentionPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + bool result = false; + int posCaseLine = -1; + if (pos == getIdx(IdxEntry::CONTEXT_LENGTHS) || pos == getIdx(IdxEntry::REQUEST_TYPES) + || pos == getIdx(IdxEntry::HOST_MAX_ATTENTION_WINDOW) || pos == getIdx(IdxEntry::HOST_SINK_TOKEN_LENGTH) + || (isEntryUsed(IdxEntry::SPEC_DECODING_PACKED_MASK) && pos == getIdx(IdxEntry::SPEC_DECODING_PACKED_MASK)) + || (isEntryUsed(IdxEntry::SPEC_DECODING_POSITION_OFFSETS) + && pos == getIdx(IdxEntry::SPEC_DECODING_POSITION_OFFSETS)) + || (isEntryUsed(IdxEntry::SPEC_DECODING_GENERATION_LENGTHS) + && pos == getIdx(IdxEntry::SPEC_DECODING_GENERATION_LENGTHS)) + || (isEntryUsed(IdxEntry::SPEC_DECODING_USE) && pos == getIdx(IdxEntry::SPEC_DECODING_USE))) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (isMRoPE() && (pos == getIdx(IdxEntry::MROPE_ROTARY_COS_SIN))) + { + return inOut[pos].type == nvinfer1::DataType::kFLOAT; + } + else if (isMRoPE() && (pos == getIdx(IdxEntry::MROPE_POSITION_DELTAS))) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (pos == getIdx(IdxEntry::HOST_RUNTIME_PERF_KNOBS) || pos == getIdx(IdxEntry::HOST_CONTEXT_PROGRESS)) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT64; + } + else if (useKVCache() + && (pos == getIdx(IdxEntry::SEQUENCE_LENGTH) || pos == getIdx(IdxEntry::HOST_PAST_KEY_VALUE_LENGTHS) + || pos == getIdx(IdxEntry::CACHE_INDIR))) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (isRoPE() && (pos == getIdx(IdxEntry::ROTARY_INV_FREQ) || pos == getIdx(IdxEntry::ROTARY_COS_SIN))) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kFLOAT; + } + else if (isLongRoPE() + && (pos == getIdx(IdxEntry::LONG_ROPE_ROTARY_INV_FREQ) || pos == getIdx(IdxEntry::LONG_ROPE_ROTARY_COS_SIN))) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kFLOAT; + } + else if (useKVCache() && mKVCacheQuantMode.hasKvCacheQuant() + && (pos == getIdx(IdxEntry::KV_CACHE_DEQUANTIZATION_SCALE) + || pos == getIdx(IdxEntry::KV_CACHE_QUANTIZATION_SCALE))) + { + // kv_scale for mType->int8/fp8 and int8/fp8->mType conversion + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (mFP8ContextFMHA && pos == getIdx(IdxEntry::ATTENTION_OUTPUT_QUANTIZATION_SCALE)) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (mFuseFp4Quant && pos == getIdx(IdxEntry::ATTENTION_OUTPUT_SF_SCALE)) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (useFullCustomMask() && pos == getIdx(IdxEntry::ATTENTION_MASK)) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kBOOL && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (useCustomMask() && pos == getIdx(IdxEntry::ATTENTION_PACKED_MASK)) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT32 && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (useKVCache() && mPagedKVCache + && (pos == getIdx(IdxEntry::KV_CACHE_BLOCK_OFFSETS) || pos == getIdx(IdxEntry::HOST_KV_CACHE_BLOCK_OFFSETS))) + { + // kv cache block offsets + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT32 && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (useKVCache() && mPagedKVCache && (pos == getIdx(IdxEntry::HOST_KV_CACHE_POOL_POINTERS))) + { + // kv cache pool pointers + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT64 && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (useKVCache() && mPagedKVCache && (pos == getIdx(IdxEntry::HOST_KV_CACHE_POOL_MAPPING))) + { + // kv cache pool mapping + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT32 && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (useKVCache() && mKVCacheQuantMode.hasInt8KvCache() + && (!mPagedKVCache && (pos == getIdx(IdxEntry::PAST_KEY_VALUE) || pos == nbInputs + 1))) + { + // If use Int8 K/V cache we require I/O KV values to int8 + posCaseLine = __LINE__; + result = (inOut[pos].type == nvinfer1::DataType::kINT8) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (useKVCache() && mKVCacheQuantMode.hasFp8KvCache() + && (!mPagedKVCache && (pos == getIdx(IdxEntry::PAST_KEY_VALUE) || pos == nbInputs + 1))) + { + // If use FP8 K/V cache we require I/O KV values to FP8 + posCaseLine = __LINE__; + result = (inOut[pos].type == nvinfer1::DataType::kFP8) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (mRemovePadding && (pos == getIdx(IdxEntry::HOST_CONTEXT_LENGTH))) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT32 && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (mCrossAttention + && (pos == getIdx(IdxEntry::CROSS_KV_LENGTH) || pos == getIdx(IdxEntry::ENCODER_INPUT_LENGTH))) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (isLognScaling() && pos == getIdx(IdxEntry::LOGN_SCALING)) + { + return inOut[pos].type == nvinfer1::DataType::kFLOAT; + } + else if (pos == nbInputs && mFuseFp4Quant) + { + // Set dtype for output FP4 quantized tensor. + posCaseLine = __LINE__; + result = (inOut[pos].type == nvinfer1::DataType::kFP4) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == nbInputs + 1 && mFuseFp4Quant) + { + // Set dtype for output scaling factor tensor. Use kINT32 as storage type (same as QuantizeToFP4Plugin). + posCaseLine = __LINE__; + result = (inOut[pos].type == nvinfer1::DataType::kFP8) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == nbInputs && mFP8ContextFMHA) + { + // Output tensor now supports fp8 data type. + posCaseLine = __LINE__; + result = (inOut[pos].type == nvinfer1::DataType::kFP8) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (mSkipAttn && pos == getIdx(IdxEntry::SKIP_ATTN)) + { + posCaseLine = __LINE__; + result = inOut[pos].type == nvinfer1::DataType::kBOOL && inOut[pos].format == TensorFormat::kLINEAR; + } + else + { + posCaseLine = __LINE__; + result = (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + TLLM_LOG_DEBUG( + "%s: pos: %d, result: %d, posCaseLine: %d", __PRETTY_FUNCTION__, pos, static_cast<int>(result), posCaseLine); + return result; +} + +template <typename T, typename KVCacheBuffer> +void GPTAttentionPlugin::configurePluginImpl(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + TLLM_CHECK(mHeadSize > 0); + + int beamWidth = -1; + if (!isCrossAttention() && useKVCache()) + { + // desc_val == -1 means beam_width is not static, we should look at min/max/opt. + // + // In prepareEnqueueGeneration, we'll prepare for all cases where beam_width doesn't exceed max. + // TODO: pass min AND max to prepareEnqueueGeneration instead of max only. + int desc_val = in[getIdx(IdxEntry::CACHE_INDIR)].desc.dims.d[1]; + int max_val = in[getIdx(IdxEntry::CACHE_INDIR)].max.d[1]; + beamWidth = desc_val == -1 ? max_val : desc_val; + } + else + { + beamWidth = 1; + } + TLLM_CHECK(beamWidth != -1); + + // Commonly, cyclic_attention_window_size, and max_attention_window_size will be the same + // unless each layer has different attention window sizes. + // the kv_cache capacity. + int max_encoder_context_len = isCrossAttention() ? in[getIdx(IdxEntry::CROSS_KV_LENGTH)].desc.dims.d[0] : 0; + int const max_attention_window_size = isCrossAttention() + ? max_encoder_context_len + : (useKVCache() ? in[getIdx(IdxEntry::CACHE_INDIR)].desc.dims.d[2] : 0); + int const cyclic_attention_window_size = max_attention_window_size; + + int const num_requests = 256; + int const sink_token_length = 0; + + EnqueueGenerationParams<T> enqueueParams; + enqueueParams.max_attention_window_size = max_attention_window_size; + enqueueParams.cyclic_attention_window_size = cyclic_attention_window_size; + enqueueParams.max_cyclic_attention_window_size = cyclic_attention_window_size; + enqueueParams.sink_token_length = sink_token_length; + enqueueParams.beam_width = beamWidth; + enqueueParams.num_requests = num_requests; + + prepareEnqueueGeneration<T, KVCacheBuffer>(enqueueParams); + + // Always reserve SemaphoreArray (for multi-block mode) as MMHA may enable multi-block mode when shared memory is + // not enough. + auto const& ctxLenTensor = in[getIdx(IdxEntry::CONTEXT_LENGTHS)]; + TLLM_CHECK_DEBUG(ctxLenTensor.max.nbDims == 1); + int32_t const max_batch_beam = in[getIdx(IdxEntry::CONTEXT_LENGTHS)].max.d[0]; + reserveSemaphoreArray(mNumHeads * max_batch_beam); +} + +template <typename T> +void GPTAttentionPlugin::configurePluginDispatchKVCacheType(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + if (mPagedKVCache) + { + configurePluginImpl<T, KVBlockArray>(in, nbInputs, out, nbOutputs); + } + else + { + configurePluginImpl<T, KVLinearBuffer>(in, nbInputs, out, nbOutputs); + } +} + +void GPTAttentionPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + if (mType == nvinfer1::DataType::kHALF) + { + configurePluginDispatchKVCacheType<half>(in, nbInputs, out, nbOutputs); + } + else if (mType == nvinfer1::DataType::kFLOAT) + { + configurePluginDispatchKVCacheType<float>(in, nbInputs, out, nbOutputs); + } +#ifdef ENABLE_BF16 + else if (mType == nvinfer1::DataType::kBF16) + { + configurePluginDispatchKVCacheType<__nv_bfloat16>(in, nbInputs, out, nbOutputs); + } +#endif +} + +size_t GPTAttentionPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + int const max_context_length = mMaxContextLength; + int const cross_kv_length = isCrossAttention() ? inputs[getIdx(IdxEntry::CROSS_KV_LENGTH)].dims.d[0] : 0; + int const max_num_seq = inputs[getIdx(IdxEntry::CONTEXT_LENGTHS)].dims.d[0]; + auto const type = inputs[getIdx(IdxEntry::QKV_TENSOR)].type; + int const max_kv_cache_length + = isCrossAttention() ? cross_kv_length : (useKVCache() ? inputs[getIdx(IdxEntry::CACHE_INDIR)].dims.d[2] : 0); + int const max_num_tokens + = mRemovePadding ? inputs[getIdx(IdxEntry::QKV_TENSOR)].dims.d[0] : max_num_seq * max_context_length; + int const max_blocks_per_sequence + = (useKVCache() && mPagedKVCache) ? inputs[getIdx(IdxEntry::KV_CACHE_BLOCK_OFFSETS)].dims.d[3] : 0; + + size_t const context_workspace_size + = getWorkspaceSizeForContext(type, max_num_seq, max_context_length, cross_kv_length, max_num_tokens); + + size_t const generation_workspace_size = getWorkspaceSizeForGeneration( + type, max_num_seq, max_kv_cache_length, max_num_tokens, max_blocks_per_sequence); + + size_t attention_input_workspace_size = 0; + + if (mUnfuseQkvGemm) + { + int const local_hidden_units_q + = inputs[getIdx(IdxEntry::QKV_TENSOR)].dims.d[getPackedTensorHiddenDimIndex(mRemovePadding)]; + int const local_hidden_units_kv + = inputs[getIdx(IdxEntry::K_TENSOR)].dims.d[getPackedTensorHiddenDimIndex(mRemovePadding)]; + size_t const size = tensorrt_llm::runtime::BufferDataType(type).getSize(); + size_t const attention_input_size = size * max_num_tokens * (local_hidden_units_q + 2 * local_hidden_units_kv); + size_t workspaces[1]; + workspaces[0] = attention_input_size; + attention_input_workspace_size = tensorrt_llm::common::calculateTotalWorkspaceSize(workspaces, 1); + } + + return std::max(context_workspace_size, generation_workspace_size) + attention_input_workspace_size; +} + +static size_t getStride(nvinfer1::Dims const& dims, int n) +{ + TLLM_CHECK(n >= 0 && n < dims.nbDims); + return std::accumulate(dims.d + n + 1, dims.d + dims.nbDims, 1, std::multiplies<size_t>{}); +} + +template <typename T, typename AttentionOutT, typename KVCacheBuffer> +int GPTAttentionPlugin::enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) +{ + TLLM_LOG_TRACE("Attention plugin start at layer %d", mLayerIdx); + + using runtime::RequestType; + + int32_t const nbSeq = inputDesc[getIdx(IdxEntry::CONTEXT_LENGTHS)].dims.d[0]; + RequestType const* reqTypes = static_cast<RequestType const*>(inputs[getIdx(IdxEntry::REQUEST_TYPES)]); + + int32_t nbContextRequests = 0; + int32_t contextTokenIdxEnd = 0; + int32_t contextTokenIdxEndForCp = 0; + // count context requests + for (int32_t seqIdx = 0; seqIdx < nbSeq; seqIdx++) + { + if (reqTypes[seqIdx] != RequestType::kCONTEXT) + { + break; + } + ++nbContextRequests; + contextTokenIdxEnd += mRemovePadding + ? static_cast<int32_t const*>(inputs[getIdx(IdxEntry::HOST_CONTEXT_LENGTH)])[seqIdx] + : inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[1]; + contextTokenIdxEndForCp += mRemovePadding + ? (static_cast<int32_t const*>(inputs[getIdx(IdxEntry::HOST_CONTEXT_LENGTH)])[seqIdx] + mCpSize - 1) + / mCpSize + : (inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[1] + mCpSize - 1) / mCpSize; + } + + for (int32_t seqIdx = nbContextRequests; seqIdx < nbSeq; seqIdx++) + { + TLLM_CHECK(reqTypes[seqIdx] == RequestType::kGENERATION); + } + + // mixed requests require mRemovePadding and mPagedKVCache + if (nbContextRequests != 0 && nbContextRequests != nbSeq) + { + TLLM_CHECK(mRemovePadding && mPagedKVCache); + } + + if (nbContextRequests > 0) + { + auto seqIdxBeg = 0; + auto tokenIdxBeg = 0; + auto localNbTokens = contextTokenIdxEnd; + enqueueSome<T, AttentionOutT, KVCacheBuffer>(seqIdxBeg, nbContextRequests, tokenIdxBeg, localNbTokens, + inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + + if (auto nbGenerationSeq = nbSeq - nbContextRequests; nbGenerationSeq > 0) + { + auto seqIdxBeg = nbContextRequests; + auto tokenIdxBeg = mCpSize > 1 ? contextTokenIdxEndForCp : contextTokenIdxEnd; + // if mRemovePadding is true, we may have IFB, and need to remove context tokens. + // if mRemovePadding is false, it is only generation requests, so just multiply batch_beam and seq_len (May not + // 1 for Parallel Decoding) + auto localNbTokens = mRemovePadding + ? inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[0] - tokenIdxBeg + : inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[0] * inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[1]; + enqueueSome<T, AttentionOutT, KVCacheBuffer>(seqIdxBeg, nbGenerationSeq, tokenIdxBeg, localNbTokens, inputDesc, + outputDesc, inputs, outputs, workspace, stream); + } + + sync_check_cuda_error(stream); + TLLM_LOG_TRACE("Attention plugin stop at layer %d", mLayerIdx); + + return 0; +} + +template <typename T, typename AttentionOutT, typename KVCacheBuffer> +int GPTAttentionPlugin::enqueueSome(int32_t seqIdxBeg, int32_t localNbSeq, int32_t tokenIdxBeg, int32_t localNbTokens, + nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) +{ + // relative_attention_bias [head_num, max_seq_len, max_seq_len] (optional in relative position) + // or [head_num, num_buckets] (optional in implicit relative attention) + // cross_kv [batch_size, seq_len, 2 * local_hidden_size] or [num_tokens, 2 * local_hidden_size] + // when enable remove_input_padding (optional in cross attention mode) + // cross_kv_length [int] max encoder input context length (optional in cross attention mode) + // encoder_input_lengths [batch_size] raw sequence lengths (optional in cross attention mode) + + using runtime::RequestType; + + auto const* const reqTypeInBatchPtr + = static_cast<RequestType const*>(inputs[getIdx(IdxEntry::REQUEST_TYPES)]) + seqIdxBeg; + bool const is_context = (reqTypeInBatchPtr[0] == RequestType::kCONTEXT); + + T const* attention_input = static_cast<T const*>(inputs[getIdx(IdxEntry::QKV_TENSOR)]) + + inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[getPackedTensorHiddenDimIndex(mRemovePadding)] + * size_t(tokenIdxBeg); + + bool changeSpecDecodingMode = false; + if (mIsSpecDecodingEnabled) + { + bool useSpecDecoding + = static_cast<bool>(reinterpret_cast<int const*>(inputs[getIdx(IdxEntry::SPEC_DECODING_USE)])[0]); + changeSpecDecodingMode = mUseSpecDecoding != useSpecDecoding; + mUseSpecDecoding = useSpecDecoding; + } + + [[maybe_unused]] MlaParams<T> mla_params; + + T const* qkv_bias = nullptr; + if (mQKVBiasEnabled) + { + qkv_bias = reinterpret_cast<T const*>(inputs[getIdx(IdxEntry::QKV_BIAS_TENSOR)]); + } + + // Note we still need context length during generation for MMHA optimization. + int32_t const max_context_q_len = [&]() + { + if (!mRemovePadding) + { + return static_cast<int>(inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[1]); + } + auto const host_context_lengths + = static_cast<int32_t const*>(inputs[getIdx(IdxEntry::HOST_CONTEXT_LENGTH)]) + seqIdxBeg; + return *std::max_element(host_context_lengths, host_context_lengths + localNbSeq); + }(); + + // Rotary inv_freq, cos_sin cache to avoid re-computing. + float const* rotary_inv_freq = nullptr; + float2 const* rotary_cos_sin = nullptr; + + bool const useLongRoPECache = isLongRoPE() && max_context_q_len > mRotaryEmbeddingOriginalMaxPositions; + if (isRoPE()) + { + auto inputName = useLongRoPECache ? IdxEntry::LONG_ROPE_ROTARY_INV_FREQ : IdxEntry::ROTARY_INV_FREQ; + rotary_inv_freq = reinterpret_cast<float const*>(inputs[getIdx(inputName)]); + } + if (isRoPE()) + { + auto inputName = useLongRoPECache ? IdxEntry::LONG_ROPE_ROTARY_COS_SIN : IdxEntry::ROTARY_COS_SIN; + rotary_cos_sin = reinterpret_cast<float2 const*>(inputs[getIdx(inputName)]); + } + + auto const mrope_rotary_cos_sin + = isMRoPE() ? reinterpret_cast<float2 const*>(inputs[getIdx(IdxEntry::MROPE_ROTARY_COS_SIN)]) : nullptr; + + auto const mrope_position_deltas + = isMRoPE() ? reinterpret_cast<int32_t const*>(inputs[getIdx(IdxEntry::MROPE_POSITION_DELTAS)]) : nullptr; + + if (mUnfuseQkvGemm) + { + int const max_seqlen = inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[mRemovePadding ? 0 : 1]; + int const batch_size = mRemovePadding ? 1 : inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[0]; + + T const* attention_input_q = static_cast<T const*>(inputs[getIdx(IdxEntry::QKV_TENSOR)]); + T const* attention_input_k = static_cast<T const*>(inputs[getIdx(IdxEntry::K_TENSOR)]); + T const* attention_input_v = static_cast<T const*>(inputs[getIdx(IdxEntry::V_TENSOR)]); + size_t const hidden_units_q + = inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims.d[getPackedTensorHiddenDimIndex(mRemovePadding)]; + size_t const hidden_units_kv + = inputDesc[getIdx(IdxEntry::K_TENSOR)].dims.d[getPackedTensorHiddenDimIndex(mRemovePadding)]; + size_t const hidden_units = hidden_units_q + 2 * hidden_units_kv; + size_t const size_qkv = sizeof(T) * hidden_units; + size_t const size_q = sizeof(T) * hidden_units_q; + size_t const size_kv = sizeof(T) * hidden_units_kv; + size_t const total_size = size_qkv * batch_size * max_seqlen; + int8_t* workspace_byte_ptr = reinterpret_cast<int8_t*>(workspace); + size_t offset = 0; + T* attention_input_qkv = reinterpret_cast<T*>(nextWorkspacePtr(workspace_byte_ptr, offset, total_size)); + workspace = reinterpret_cast<void*>(workspace_byte_ptr + offset); + + cudaMemcpy2DAsync(attention_input_qkv, size_qkv, attention_input_q, size_q, size_q, batch_size * max_seqlen, + cudaMemcpyDeviceToDevice, stream); + cudaMemcpy2DAsync(attention_input_qkv + hidden_units_q, size_qkv, attention_input_k, size_kv, size_kv, + batch_size * max_seqlen, cudaMemcpyDeviceToDevice, stream); + cudaMemcpy2DAsync(attention_input_qkv + hidden_units_q + hidden_units_kv, size_qkv, attention_input_v, size_kv, + size_kv, batch_size * max_seqlen, cudaMemcpyDeviceToDevice, stream); + + attention_input = attention_input_qkv + hidden_units * tokenIdxBeg; + } + + int const* context_q_lengths = reinterpret_cast<int const*>(inputs[getIdx(IdxEntry::CONTEXT_LENGTHS)]) + seqIdxBeg; + int const* sequence_kv_length = useKVCache() + ? static_cast<int const*>(inputs[getIdx(IdxEntry::SEQUENCE_LENGTH)]) + seqIdxBeg + : context_q_lengths; + + int max_encoder_context_len = isCrossAttention() ? inputDesc[getIdx(IdxEntry::CROSS_KV_LENGTH)].dims.d[0] : 0; + // for enc-dec model, since decoder_input_ids could be longer than 1, + // such model has an encoder context (for cross attn) and an decoder context (for self attn) + // clarify 3 lens: + // -- max_context_q_len: len of decoder input. No "max" concept, it's what it is given. + // Also called (decoder_)input_seq_length, normally 1 for encoder-decoder start token + // -- max_seq_len: max allowed len of decoder output, i.e. final results + // -- max_encoder_context_len: len of encoder input (in cross attn). Also called encoder_input_seq_length + + int const beamWidth + = isCrossAttention() ? 1 : (useKVCache() ? inputDesc[getIdx(IdxEntry::CACHE_INDIR)].dims.d[1] : 1); + + // Commonly, cyclic_attention_window_size, and max_attention_window_size will be the same + // unless each layer has different attention window sizes. + // the kv_cache capacity. + int const max_attention_window_size = isCrossAttention() + ? max_encoder_context_len + : (useKVCache() ? inputDesc[getIdx(IdxEntry::CACHE_INDIR)].dims.d[2] : 0); + // The cyclic_attention_window_size will determine the cyclic kv cache position of new tokens. + // Note that this cyclic_attention_window_size might be smaller than the actual kv cache capactity. + int const* cyclic_attention_window_sizes + = reinterpret_cast<int const*>(inputs[getIdx(IdxEntry::HOST_MAX_ATTENTION_WINDOW)]); + int const cyclic_attention_window_size + = isCrossAttention() ? max_encoder_context_len : cyclic_attention_window_sizes[mLayerIdx]; + int const sink_token_length = reinterpret_cast<int const*>(inputs[getIdx(IdxEntry::HOST_SINK_TOKEN_LENGTH)])[0]; + int const num_attn_layer = inputDesc[getIdx(IdxEntry::HOST_MAX_ATTENTION_WINDOW)].dims.d[0]; + int const max_cyclic_attention_window_size = isCrossAttention() + ? max_encoder_context_len + : *std::max_element(cyclic_attention_window_sizes, cyclic_attention_window_sizes + num_attn_layer); + bool const can_use_one_more_block = beamWidth > 1; + + float const* kv_scale_orig_quant = nullptr; + float const* kv_scale_quant_orig = nullptr; + if (useKVCache() && mKVCacheQuantMode.hasKvCacheQuant()) + { + assert(inputDesc[getIdx(IdxEntry::KV_CACHE_QUANTIZATION_SCALE)].type == nvinfer1::DataType::kFLOAT); + assert(inputDesc[getIdx(IdxEntry::KV_CACHE_DEQUANTIZATION_SCALE)].type == nvinfer1::DataType::kFLOAT); + kv_scale_orig_quant = reinterpret_cast<float const*>(inputs[getIdx(IdxEntry::KV_CACHE_QUANTIZATION_SCALE)]); + kv_scale_quant_orig = reinterpret_cast<float const*>(inputs[getIdx(IdxEntry::KV_CACHE_DEQUANTIZATION_SCALE)]); + } + + float const* attention_output_orig_quant = nullptr; + if (mFP8ContextFMHA) + { + assert(inputDesc[getIdx(IdxEntry::ATTENTION_OUTPUT_QUANTIZATION_SCALE)].type == nvinfer1::DataType::kFLOAT); + attention_output_orig_quant + = reinterpret_cast<float const*>(inputs[getIdx(IdxEntry::ATTENTION_OUTPUT_QUANTIZATION_SCALE)]); + } + float const* attention_output_sf_scale = nullptr; + if (mFuseFp4Quant) + { + assert(inputDesc[getIdx(IdxEntry::ATTENTION_OUTPUT_SF_SCALE)].type == nvinfer1::DataType::kFLOAT); + attention_output_sf_scale = reinterpret_cast<float const*>(inputs[getIdx(IdxEntry::ATTENTION_OUTPUT_SF_SCALE)]); + } + uint32_t const* attention_packed_mask = nullptr; + if (useCustomMask()) + { + assert(inputDesc[getIdx(IdxEntry::ATTENTION_PACKED_MASK)].type == nvinfer1::DataType::kINT32); + attention_packed_mask = reinterpret_cast<uint32_t const*>(inputs[getIdx(IdxEntry::ATTENTION_PACKED_MASK)]); + } + bool const* attention_mask = nullptr; + int attention_mask_stride = 0; + if (useFullCustomMask()) + { + attention_mask_stride = static_cast<int>(inputDesc[getIdx(IdxEntry::ATTENTION_MASK)].dims.d[1]); + attention_mask = reinterpret_cast<bool const*>(inputs[getIdx(IdxEntry::ATTENTION_MASK)]) + + attention_mask_stride * static_cast<size_t>(tokenIdxBeg); + } + + int max_blocks_per_sequence = 0; + kernels::KVBlockArray::DataType* block_offsets = nullptr; + void* host_primary_pool_pointer = nullptr; + void* host_secondary_pool_pointer = nullptr; + if (useKVCache() && mPagedKVCache) + { + auto const& kvCacheBlockOffsetsShape = inputDesc[getIdx(IdxEntry::KV_CACHE_BLOCK_OFFSETS)].dims; + max_blocks_per_sequence = kvCacheBlockOffsetsShape.d[kvCacheBlockOffsetsShape.nbDims - 1]; + + std::int32_t const* host_pool_mapping + = static_cast<std::int32_t const*>(inputs[getIdx(IdxEntry::HOST_KV_CACHE_POOL_MAPPING)]); + + int32_t const layerToPool = host_pool_mapping[mLayerIdx * 2]; + int32_t const layerIdxInCachePool = host_pool_mapping[mLayerIdx * 2 + 1]; + TLLM_LOG_TRACE("Layer%d: LayerCachePoolLocator{.indexOfPool=%d, .layerIdxInCachePool=%d}", mLayerIdx, + layerToPool, layerIdxInCachePool); + auto const seqStride = getStride(kvCacheBlockOffsetsShape, 1); + auto const poolStride = getStride(kvCacheBlockOffsetsShape, 0); + auto const seqOffset = seqIdxBeg * seqStride; + auto const poolOffset = layerToPool * poolStride; + + block_offsets + = reinterpret_cast<kernels::KVBlockArray::DataType*>(inputs[getIdx(IdxEntry::KV_CACHE_BLOCK_OFFSETS)]) + + poolOffset + seqOffset; + + auto const* const typed_host_pool_pointers + = static_cast<char* const*>(inputs[getIdx(IdxEntry::HOST_KV_CACHE_POOL_POINTERS)]); + + auto const cacheElemSize = (mKVCacheQuantMode.hasKvCacheQuant() ? 1 : sizeof(T)); + + auto const kv_cache_head_num = (mNumKVHeads + mCpSize - 1) / mCpSize; + auto const blockSize = mTokensPerBlock * kv_cache_head_num * mHeadSize; + auto const bytesPerBlock = blockSize * cacheElemSize; + auto const layerOffset = layerIdxInCachePool * 2 * bytesPerBlock; + + host_primary_pool_pointer = reinterpret_cast<void*>(typed_host_pool_pointers[layerToPool * 2] + layerOffset); + host_secondary_pool_pointer + = reinterpret_cast<void*>(typed_host_pool_pointers[layerToPool * 2 + 1] + layerOffset); + } + + // The index of kv cache tensor in outputs. If fuse FP4 quant, an additional scaling factor output is added before + // the kv cache tensor. + int const kvCacheIdxInOutputs = mFuseFp4Quant ? 2 : 1; + // The number of elements per storage type. For FP4 output, storage type is uint8_t. + int const numEltsPerStorageType = mFuseFp4Quant ? 2 : 1; + + AttentionOutT* context_buf_ = static_cast<AttentionOutT*>(outputs[0]) + + outputDesc[0].dims.d[getPackedTensorHiddenDimIndex(mRemovePadding)] * tokenIdxBeg / numEltsPerStorageType; + + __nv_fp8_e4m3* context_buf_sf_ = nullptr; + if (mFuseFp4Quant) + { + // The output address for FP4 scaling factor. + context_buf_sf_ = static_cast<__nv_fp8_e4m3*>(outputs[1]); + } + + void* key_value_cache = nullptr; + if (useKVCache() && !mPagedKVCache) + { + auto const cacheElemSize = (mKVCacheQuantMode.hasKvCacheQuant() ? 1 : sizeof(T)); + key_value_cache = static_cast<std::byte*>(outputs[kvCacheIdxInOutputs]) + + cacheElemSize * getStride(outputDesc[kvCacheIdxInOutputs].dims, 0) * seqIdxBeg; + void const* past_key_value_cache = inputs[getIdx(IdxEntry::PAST_KEY_VALUE)]; + if (past_key_value_cache != outputs[kvCacheIdxInOutputs]) + { + auto shape = outputDesc[kvCacheIdxInOutputs].dims; + auto const size + = cacheElemSize * std::accumulate(shape.d, shape.d + shape.nbDims, 1, std::multiplies<size_t>{}); + cudaMemcpyAsync(outputs[kvCacheIdxInOutputs], past_key_value_cache, size, cudaMemcpyDeviceToDevice, stream); + } + } + + T const* alibi_slopes = isALiBi() ? static_cast<T const*>(inputs[getIdx(IdxEntry::ALIBI_SLOPES)]) : nullptr; + + int const* spec_decoding_packed_mask = nullptr; + int const* spec_decoding_position_offsets = nullptr; + int const* spec_decoding_generation_lengths = nullptr; + int num_decoding_draft_tokens = 0; + if (mIsSpecDecodingEnabled && mUseSpecDecoding) + { + // Second dimension of spec_decoding_position_offsets is num_decoding_draft_tokens + 1. + // [batch_size, num_decoding_draft_tokens + 1] + num_decoding_draft_tokens = inputDesc[getIdx(IdxEntry::SPEC_DECODING_POSITION_OFFSETS)].dims.d[1] - 1; + if (num_decoding_draft_tokens > 0) + { + // spec_decoding_* tensors are not filled for context requests. Hence, always strting from 0th index + int32_t constexpr genSeqIdx = 0; + spec_decoding_packed_mask = static_cast<int const*>(inputs[getIdx(IdxEntry::SPEC_DECODING_PACKED_MASK)]) + + genSeqIdx * getStride(inputDesc[getIdx(IdxEntry::SPEC_DECODING_PACKED_MASK)].dims, 0); + // Packed as [num_tokens, packed_mask_size] + // Use seqIdxBeg * (num_decoding_draft_tokens + 1) here as only generation tokens have the packed_mask + // buffer. + // TODO: support variable sequence length based on generationTokenIdxBeg. + spec_decoding_packed_mask = static_cast<int const*>(inputs[getIdx(IdxEntry::SPEC_DECODING_PACKED_MASK)]) + + genSeqIdx * (num_decoding_draft_tokens + 1) + * getStride(inputDesc[getIdx(IdxEntry::SPEC_DECODING_PACKED_MASK)].dims, 0); + spec_decoding_position_offsets + = static_cast<int const*>(inputs[getIdx(IdxEntry::SPEC_DECODING_POSITION_OFFSETS)]) + + genSeqIdx * getStride(inputDesc[getIdx(IdxEntry::SPEC_DECODING_POSITION_OFFSETS)].dims, 0); + spec_decoding_generation_lengths + = static_cast<int const*>(inputs[getIdx(IdxEntry::SPEC_DECODING_GENERATION_LENGTHS)]) + genSeqIdx; + } + } + + int32_t const* host_past_kv_len_list = useKVCache() + ? static_cast<int const*>(inputs[getIdx(IdxEntry::HOST_PAST_KEY_VALUE_LENGTHS)]) + seqIdxBeg + : nullptr; + int32_t const max_context_kv_len = useKVCache() + ? *std::max_element(host_past_kv_len_list, host_past_kv_len_list + localNbSeq) + : max_context_q_len; + + int const* host_context_lengths + = mRemovePadding ? reinterpret_cast<int const*>(inputs[getIdx(IdxEntry::HOST_CONTEXT_LENGTH)]) : nullptr; + + int64_t const* runtime_perf_knobs = static_cast<int64_t const*>(inputs[getIdx(IdxEntry::HOST_RUNTIME_PERF_KNOBS)]); + + EnqueueParams<T> common_enqueue_params; + common_enqueue_params.attention_input = attention_input; + common_enqueue_params.qkv_bias = qkv_bias; + common_enqueue_params.attention_mask = attention_mask; + common_enqueue_params.rotary_inv_freq = rotary_inv_freq; + common_enqueue_params.rotary_cos_sin = rotary_cos_sin; + common_enqueue_params.max_attention_window_size = max_attention_window_size; + common_enqueue_params.cyclic_attention_window_size = cyclic_attention_window_size; + common_enqueue_params.max_cyclic_attention_window_size = max_cyclic_attention_window_size; + common_enqueue_params.can_use_one_more_block = can_use_one_more_block; + common_enqueue_params.sink_token_length = sink_token_length; + common_enqueue_params.kv_scale_orig_quant = kv_scale_orig_quant; + common_enqueue_params.kv_scale_quant_orig = kv_scale_quant_orig; + common_enqueue_params.attention_output_orig_quant = attention_output_orig_quant; + common_enqueue_params.attention_output_sf_scale = attention_output_sf_scale; + common_enqueue_params.alibi_slopes = alibi_slopes; + common_enqueue_params.context_buf = context_buf_; + common_enqueue_params.context_buf_sf = context_buf_sf_; + common_enqueue_params.key_value_cache = key_value_cache; + common_enqueue_params.block_offsets = block_offsets; + common_enqueue_params.host_primary_pool_pointer = host_primary_pool_pointer; + common_enqueue_params.host_secondary_pool_pointer = host_secondary_pool_pointer; + common_enqueue_params.num_tokens = localNbTokens; + common_enqueue_params.max_blocks_per_sequence = max_blocks_per_sequence; + common_enqueue_params.sequence_lengths = sequence_kv_length; + common_enqueue_params.context_lengths = context_q_lengths; + common_enqueue_params.host_context_lengths = host_context_lengths; + common_enqueue_params.workspace = workspace; + common_enqueue_params.runtime_perf_knobs = runtime_perf_knobs; + + if (isRelativePosition()) + { + common_enqueue_params.relative_attention_bias + = static_cast<T const*>(inputs[getIdx(IdxEntry::RELATIVE_ATTENTION_BIAS)]); + common_enqueue_params.relative_attention_bias_stride + = inputDesc[getIdx(IdxEntry::RELATIVE_ATTENTION_BIAS)].dims.d[1]; // max_seq_len or num_buckets + } + if (isLognScaling()) + { + common_enqueue_params.logn_scaling_ptr = static_cast<float const*>(inputs[getIdx(IdxEntry::LOGN_SCALING)]); + } + if (isCrossAttention()) + { + common_enqueue_params.encoder_input_lengths + = reinterpret_cast<int const*>(inputs[getIdx(IdxEntry::ENCODER_INPUT_LENGTH)]) + seqIdxBeg; + } + + if (is_context) // context stage + { + int const batch_size = localNbSeq; + int const request_batch_size = batch_size; + // num of total tokens (without paddings when remove paddings). + int num_encoder_tokens = 0; + if (isCrossAttention()) + { + if (!mRemovePadding) + { + num_encoder_tokens = request_batch_size * max_encoder_context_len; + } + else + { + num_encoder_tokens = inputDesc[getIdx(IdxEntry::CROSS_KV)].dims.d[0]; + } + } + + common_enqueue_params.input_seq_length = max_context_q_len; + common_enqueue_params.max_past_kv_length = max_context_kv_len; + EnqueueContextParams<T> enqueue_params{common_enqueue_params}; + enqueue_params.attention_packed_mask = attention_packed_mask; + enqueue_params.batch_size = batch_size; + enqueue_params.mrope_rotary_cos_sin = mrope_rotary_cos_sin; + enqueue_params.total_kv_len = enqueue_params.num_tokens; + + if (isCrossAttention()) + { + enqueue_params.cross_kv = static_cast<T const*>(inputs[getIdx(IdxEntry::CROSS_KV)]); + enqueue_params.cross_kv_length = max_encoder_context_len; + enqueue_params.num_encoder_tokens = num_encoder_tokens; + } + + enqueueContext<T, KVCacheBuffer>(enqueue_params, stream); + + { + std::string const afterContexStr = "ctx attention at layer " + std::to_string(mLayerIdx); + TLLM_LOG_TRACE("GPTAttentionPlugin - %s", afterContexStr.c_str()); + + auto progress = static_cast<batch_manager::ContextProgress* const*>( + inputs[getIdx(IdxEntry::HOST_CONTEXT_PROGRESS)])[0]; + if (progress != nullptr) + { + progress->recordEvent(mLayerIdx, stream); + } + + if (!mFuseFp4Quant) + { + TLLM_CHECK_DEBUG_WITH_INFO( + tensorrt_llm::runtime::utils::tensorHasInvalid(localNbTokens, + outputDesc[0].dims.d[getPackedTensorHiddenDimIndex(mRemovePadding)], + mFP8ContextFMHA ? nvinfer1::DataType::kFP8 : mType, context_buf_, stream, afterContexStr) + == false, + "Found invalid number (NaN or Inf) in " + afterContexStr); + } + } + } + else // generation stage; max_context_q_len == input_seq_len == 1 + { + TLLM_CHECK_WITH_INFO(useKVCache(), "KV-cache-less is only supported for context"); + int batch_beam = localNbSeq; + TLLM_CHECK(batch_beam % beamWidth == 0); + int32_t const num_requests = batch_beam / beamWidth; + + int const* cache_indir + = beamWidth == 1 ? nullptr : reinterpret_cast<int const*>(inputs[getIdx(IdxEntry::CACHE_INDIR)]); + + // Medusa: the max input sequence length if variable sequence length is needed. + int const input_seq_length = getGenerationInputSequenceLength(inputDesc, localNbSeq, localNbTokens); + int const max_past_kv_length = isCrossAttention() ? max_encoder_context_len : max_context_kv_len; + auto qkvDims = inputDesc[getIdx(IdxEntry::QKV_TENSOR)].dims; + TLLM_CHECK_WITH_INFO(input_seq_length == 1 || (mIsSpecDecodingEnabled && mUseSpecDecoding), + "Only speculative decoding mode supports input length > 1 in the generation phase, input_seq_length=%d, " + "mIsSpecDecodingEnabled=%s, nDims=%d, (" FMT_DIM ", " FMT_DIM ", " FMT_DIM ")", + input_seq_length, mIsSpecDecodingEnabled ? "true" : "false", qkvDims.nbDims, qkvDims.d[0], qkvDims.d[1], + qkvDims.d[2]); + TLLM_CHECK_WITH_INFO( + input_seq_length == num_decoding_draft_tokens + 1, "The generation input length is not expected."); + common_enqueue_params.input_seq_length = input_seq_length; + common_enqueue_params.max_past_kv_length = max_past_kv_length; + EnqueueGenerationParams<T> enqueue_params{common_enqueue_params}; + enqueue_params.beam_width = beamWidth; + enqueue_params.attention_mask_stride = attention_mask_stride; + enqueue_params.num_requests = num_requests; + enqueue_params.cache_indir = cache_indir; + enqueue_params.semaphores = multiBlockSemaphores(); + enqueue_params.host_past_key_value_lengths = host_past_kv_len_list; + enqueue_params.mrope_position_deltas = mrope_position_deltas; + if (mIsSpecDecodingEnabled && mUseSpecDecoding) + { + enqueue_params.spec_decoding_packed_mask = spec_decoding_packed_mask; + enqueue_params.spec_decoding_position_offsets = spec_decoding_position_offsets; + enqueue_params.spec_decoding_generation_lengths = spec_decoding_generation_lengths; + enqueue_params.spec_decoding_is_generation_length_variable = mSpecDecodingIsGenerationLengthVariable; + enqueue_params.spec_decoding_max_generation_length = mSpecDecodingMaxGenerationLength; + } + if (mFuseFp4Quant) + { + enqueue_params.start_token_idx_sf = tokenIdxBeg; + } + + if (changeSpecDecodingMode) + { + // mUseSpecDecoding is changed, need to re-prepare the DecoderXQARunner + prepareEnqueueGeneration<T, KVCacheBuffer>(enqueue_params); + } + + enqueueGeneration<T, KVCacheBuffer>(enqueue_params, stream); + + { + std::string const afterGenStr = "gen attention at layer " + std::to_string(mLayerIdx); + { + TLLM_CHECK_DEBUG_WITH_INFO( + tensorrt_llm::runtime::utils::tensorHasInvalid(localNbTokens, + outputDesc[0].dims.d[getPackedTensorHiddenDimIndex(mRemovePadding)], + mFP8ContextFMHA ? nvinfer1::DataType::kFP8 : mType, context_buf_, stream, afterGenStr) + == false, + "Found invalid number (NaN or Inf) in " + afterGenStr); + } + } + } + + return 0; +} + +template <typename T, typename AttentionOutT> +int GPTAttentionPlugin::enqueueDispatchKVCacheType(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) +{ + if (mPagedKVCache) + { + return enqueueImpl<T, AttentionOutT, KVBlockArray>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else + { + return enqueueImpl<T, AttentionOutT, KVLinearBuffer>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + return 0; +} + +int GPTAttentionPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + if (mSkipAttn) + { + bool const* SKIP_ATTN = reinterpret_cast<bool const*>(inputs[getIdx(IdxEntry::SKIP_ATTN)]); + if (SKIP_ATTN[0]) + { + return 0; + } + } + + if (mType == nvinfer1::DataType::kHALF) + { + if (mFuseFp4Quant) + { + return enqueueDispatchKVCacheType<half, uint8_t>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_FP8 + if (mFP8ContextFMHA) + { + return enqueueDispatchKVCacheType<half, __nv_fp8_e4m3>( + inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return enqueueDispatchKVCacheType<half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == nvinfer1::DataType::kFLOAT) + { + return enqueueDispatchKVCacheType<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_BF16 + else if (mType == nvinfer1::DataType::kBF16) + { + if (mFuseFp4Quant) + { + return enqueueDispatchKVCacheType<__nv_bfloat16, uint8_t>( + inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_FP8 + if (mFP8ContextFMHA) + { + return enqueueDispatchKVCacheType<__nv_bfloat16, __nv_fp8_e4m3>( + inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return enqueueDispatchKVCacheType<__nv_bfloat16>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType GPTAttentionPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + if (mFuseFp4Quant) + { + TLLM_CHECK(index == 0 || index == 1 || (!mPagedKVCache && useKVCache() && index == 2)); + } + else + { + TLLM_CHECK(index == 0 || (!mPagedKVCache && useKVCache() && index == 1)); + } + if (index == 0) + { + if (mFuseFp4Quant) + { + return nvinfer1::DataType::kFP4; + } + return mFP8ContextFMHA && mEnableContextFMHA ? nvinfer1::DataType::kFP8 + : inputTypes[getIdx(IdxEntry::QKV_TENSOR)]; + } + if (mFuseFp4Quant && index == 1) + { + return nvinfer1::DataType::kFP8; + } + return inputTypes[getIdx(IdxEntry::PAST_KEY_VALUE)]; +} + +// IPluginV2 Methods + +char const* GPTAttentionPlugin::getPluginType() const noexcept +{ + return GPT_ATTENTION_PLUGIN_NAME; +} + +char const* GPTAttentionPlugin::getPluginVersion() const noexcept +{ + return GPT_ATTENTION_PLUGIN_VERSION; +} + +int GPTAttentionPlugin::getNbOutputs() const noexcept +{ + int nbOutputs = mFuseFp4Quant ? 2 : 1; + if (!mPagedKVCache && useKVCache()) + { + nbOutputs += 1; + } + return nbOutputs; +} + +size_t GPTAttentionPlugin::getSerializationSize() const noexcept +{ + return GPTAttentionPluginCommon::getCommonSerializationSize(); +} + +void GPTAttentionPlugin::serialize(void* buffer) const noexcept +{ + GPTAttentionPluginCommon::serializeCommon(buffer); +} + +/////////////// + +GPTAttentionPluginCreator::GPTAttentionPluginCreator() + : GPTAttentionPluginCreatorCommon() +{ +} + +char const* GPTAttentionPluginCreator::getPluginName() const noexcept +{ + return GPT_ATTENTION_PLUGIN_NAME; +} + +char const* GPTAttentionPluginCreator::getPluginVersion() const noexcept +{ + return GPT_ATTENTION_PLUGIN_VERSION; +} + +PluginFieldCollection const* GPTAttentionPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* GPTAttentionPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginFieldParser p{fc->nbFields, fc->fields}; + + try + { + auto* obj = new GPTAttentionPlugin(p.getScalar<int32_t>("layer_idx").value(), + p.getScalar<int32_t>("num_heads").value(), p.getScalar<int32_t>("vision_start").value(), + p.getScalar<int32_t>("vision_length").value(), p.getScalar<int32_t>("num_kv_heads").value(), + p.getScalar<int32_t>("num_kv_heads_origin").value(), p.getScalar<int32_t>("head_size").value(), + p.getScalar<int32_t>("unidirectional").value(), p.getScalar<float>("q_scaling").value(), + p.getScalar<float>("attn_logit_softcapping_scale").value(), + static_cast<PositionEmbeddingType>(p.getScalar<int8_t>("position_embedding_type").value()), + p.getScalar<int32_t>("rotary_embedding_dim").value(), p.getScalar<float>("rotary_embedding_base").value(), + static_cast<RotaryScalingType>(p.getScalar<int8_t>("rotary_embedding_scale_type").value()), + p.getScalar<float>("rotary_embedding_scale").value(), + p.getScalar<float>("rotary_embedding_short_m_scale").value(), + p.getScalar<float>("rotary_embedding_long_m_scale").value(), + p.getScalar<int32_t>("rotary_embedding_max_positions").value(), + p.getScalar<int32_t>("rotary_embedding_original_max_positions").value(), + static_cast<int32_t>(p.getScalar<int32_t>("tp_size").value()), + static_cast<int32_t>(p.getScalar<int32_t>("tp_rank").value()), + static_cast<bool>(p.getScalar<int8_t>("unfuse_qkv_gemm").value()), + static_cast<bool>(p.getScalar<int8_t>("use_logn_scaling").value()), + static_cast<ContextFMHAType>(p.getScalar<int8_t>("context_fmha_type").value()), + p.getScalar<int32_t>("kv_cache_quant_mode").value(), + static_cast<bool>(p.getScalar<int8_t>("remove_input_padding").value()), + static_cast<AttentionMaskType>(p.getScalar<int32_t>("mask_type").value()), + BlockSparseParams{p.getScalar<int32_t>("block_sparse_block_size").value(), + static_cast<bool>(p.getScalar<int8_t>("block_sparse_homo_head_pattern").value()), + p.getScalar<int32_t>("block_sparse_num_local_blocks").value(), + p.getScalar<int32_t>("block_sparse_vertical_stride").value()}, + static_cast<bool>(p.getScalar<int32_t>("paged_kv_cache").value()), + p.getScalar<int32_t>("tokens_per_block").value(), + static_cast<nvinfer1::DataType>(p.getScalar<int32_t>("type_id").value()), + p.getScalar<int32_t>("max_context_length").value(), + static_cast<bool>(p.getScalar<int8_t>("qkv_bias_enabled").value()), + static_cast<bool>(p.getScalar<int8_t>("do_cross_attention").value()), + static_cast<int32_t>(p.getScalar<int32_t>("max_distance").value()), + static_cast<bool>(p.getScalar<int8_t>("pos_shift_enabled").value()), + static_cast<bool>(p.getScalar<int8_t>("dense_context_fmha").value()), + static_cast<bool>(p.getScalar<int8_t>("use_paged_context_fmha").value()), + static_cast<bool>(p.getScalar<int8_t>("use_fp8_context_fmha").value()), + static_cast<bool>(p.getScalar<int8_t>("has_full_attention_mask").value()), + static_cast<bool>(p.getScalar<int32_t>("use_cache").value()), + static_cast<bool>(p.getScalar<int8_t>("is_spec_decoding_enabled").value()), + static_cast<bool>(p.getScalar<int8_t>("spec_decoding_is_generation_length_variable").value()), + p.getScalar<int32_t>("spec_decoding_max_generation_length").value(), + static_cast<int8_t>(p.getScalar<int8_t>("is_mla_enabled").value()), + static_cast<int32_t>(p.getScalar<int32_t>("q_lora_rank").value()), + static_cast<int32_t>(p.getScalar<int32_t>("kv_lora_rank").value()), + static_cast<int32_t>(p.getScalar<int32_t>("qk_nope_head_dim").value()), + static_cast<int32_t>(p.getScalar<int32_t>("qk_rope_head_dim").value()), + static_cast<int32_t>(p.getScalar<int32_t>("v_head_dim").value()), + static_cast<bool>(p.getScalar<int8_t>("fuse_fp4_quant").value()), + static_cast<bool>(p.getScalar<int8_t>("skip_attn").value()), + static_cast<int32_t>(p.getScalar<int32_t>("cp_size").value()), + static_cast<int32_t>(p.getScalar<int32_t>("cp_rank").value()), + static_cast<std::set<int32_t>>(p.getSet<int32_t>("cp_group").value())); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* GPTAttentionPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call GPTAttentionPlugin::destroy() + try + { + auto* obj = new GPTAttentionPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/gptAttentionPlugin/gptAttentionPlugin.h b/cpp/tensorrt_llm/plugins/gptAttentionPlugin/gptAttentionPlugin.h new file mode 100644 index 000000000000..3e34703c6221 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/gptAttentionPlugin/gptAttentionPlugin.h @@ -0,0 +1,259 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "checkMacrosPlugin.h" +#include "tensorrt_llm/common/cublasMMWrapper.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/common/stringUtils.h" +#include "tensorrt_llm/kernels/contextFusedMultiHeadAttention/fmhaRunner.h" +#include "tensorrt_llm/kernels/contextFusedMultiHeadAttention/fused_multihead_attention_common.h" +#include "tensorrt_llm/kernels/gptKernels.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.h" +#include <cassert> +#include <cstddef> +#include <cstdint> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ +// batch_size = num_ctx_requests + num_gen_requests * beam_width +// num_ctx_requests = number of context requests (single sequence per request). +// num_gen_requests = number of generation requests (beam_width sequences per request). +// Context sequences have to appear first, generation sequences after + +// inputs (see GPTAttentionPlugin::isEntryUsed for when each tensor is actually used) +// 0. input_tensor [batch_size, seq_len, local_hidden_size + 2 * local_num_kv_heads * head_size] or +// [num_tokens, local_hidden_size + 2 * local_num_kv_heads * head_size] when +// enable_remove_input_padding +// 1. sequence_length [batch_size] (optional) +// 2. host_past_key_value_lengths [batch_size] (int32) (optional) +// 3. host_max_attention_window_sizes [num_layers] (int32) +// 4. host_sink_token_length [1] (int32) +// 5. context_lengths [batch_size] +// 6. cache_indir [num_gen_requests, beam_width, memory_max_len] (required in beamsearch) (optional) +// 7. host_request_types [batch_size] int32. 0: context; 1: generation: 2: none. When not in inflight-batching +// mode, +// all elements must be identical. +// 8. past_key_value_pool [batch_size, 2, local_num_kv_heads, max_seq_len, head_size] or +// block_offsets [batch_size, 2, max_blocks_per_seq] if paged kv cache (optional) +// 8.1 host_pool_pointers [2] if paged kv cache (optional) +// 9. kv_cache_quantization_scale [1] (optional) +// 10. kv_cache_dequantization_scale [1] (optional) +// 11. attention_output_quantization_scale [1] (on device, optional) +// 12. attention_mask [num_tokens, kv_seqlen] (on device, bool, optional) +// 13. attention_packed_mask [num_tokens, kv_seqlen / 32] (on device, uint32_t, optional) +// - pack masks by encoding multiple mask positions into a single 32-bit unsigned integer. +// - see kernels/contextMultiHeadAttention/fmhaPackedMask.cpp for more details. +// 14. rotary_inv_freq [head_size / 2] or [head_size] (longrope type) (float) (on device, optional) +// 15. rotary_cos_sin [max_num_embedding_positions, 2] (float) (on device, optional) +// 16. alibi_slopes [num_heads] (optional for ALiBi position embedding) +// 17. relative_attention_bias [num_heads] (optional for ALiBi position embedding) +// 18. host_context_lengths [batch_size] int32. (optional, required when remove_input_padding is true) +// 19. qkv_bias (optional) [local_hidden_size * 3] +// 20. spec_decoding_generation_lengths (optional, required when medusa is enabled) (int32_t) [batch_size] +// 21. spec_decoding_packed_mask (optional, required when medusa is enabled) (int32_t) [num_tokens, packed_mask_dim] +// packed_mask_dim = divUp(max_num_spec_decoding_tokens + 1, 32) +// 22. spec_decoding_position_offsets (optional, required when medusa is enabled) (int32_t) [batch_size, +// max_num_spec_decoding_tokens + 1] +// 23. spec_decoding_use (optional, bool) [1]: If it is set as true, enable speculative decoding +// 24. long_rope_rotary_inv_freq [head / 2] (float) (on device, optional) +// 25. long_rope_rotary_cos_sin [max_num_embedding_positions, 2] (float) (on device, optional) +// 26. host_runtime_perf_knobs (int64) +// 27. host_context_progress (void*) +// 28. position_id_tensor(MLA) [total_tokens], used for rope embedding in MLA +// 29. q_a_proj_tensor(MLA) [hidden_dim, c_q_dim + c_k_dim + ropd_dim], used to proj compacted QKV +// 30. q_a_layernorm_tensor(MLA) [c_q_dim], rmsnorm weight for compacted q +// 31. q_b_proj_tensor(MLA) [c_q_dim, head_num * head_size], weight for companted q to q in context +// 32. kv_a_proj_with_mqa_tensor(MLA) [c_q_dim, head_num * (c_k_dim + rope_dim)], weight for companted q to kdim in +// generation +// 33. kv_a_layernorm_tensor(MLA) [c_k_dim], rmsnorm weight for compacted kv +// 34. kv_b_proj_tensor(MLA) [c_k_dim, head_num * 2 * (head_size - rope_dim)], weight for compacted kv to kv in +// context +// 35. skip_attn (optional, bool) [1]: If it is set as true, skip the atteniton plugin and return +// directly. +// +// outputs +// output_tensor [batch_size, seq_len, local_hidden_size] +// present_key_value_pool (optional if not paged kv cache) [batch_size, 2, local_num_kv_heads, max_seq_len, +// head_size] + +class GPTAttentionPlugin : public GPTAttentionPluginCommon +{ +public: + GPTAttentionPlugin(int layer_idx, int num_heads, int vision_start, int vision_length, int num_kv_heads, + int num_kv_heads_origin, int head_size, int unidirectional, float q_scaling, float attn_logit_softcapping_scale, + tensorrt_llm::kernels::PositionEmbeddingType position_embedding_type, + int rotary_embedding_dim, // for RoPE. 0 for non-RoPE + float rotary_embedding_base, tensorrt_llm::kernels::RotaryScalingType rotary_embedding_scale_type, + float rotary_embedding_scale, float rotary_embedding_short_m_scale, float rotary_embedding_long_m_scale, + int rotary_embedding_max_positions, int rotary_embedding_original_max_positions, int tp_size, + int tp_rank, // for ALiBi + bool unfuse_qkv_gemm, // for AutoPP + bool use_logn_scaling, // for LognScaling + tensorrt_llm::kernels::ContextFMHAType context_fmha_type, int kv_cache_quant_mode, bool remove_input_padding, + tensorrt_llm::kernels::AttentionMaskType mask_type, + tensorrt_llm::kernels::BlockSparseParams block_sparse_params, bool paged_kv_cache, int tokens_per_block, + nvinfer1::DataType type, int32_t max_context_length, bool qkv_bias_enabled, bool cross_attention = false, + int max_distance = 0, bool pos_shift_enabled = false, bool dense_context_fmha = false, + bool use_paged_context_fmha = true, bool use_fp8_context_fmha = true, bool has_full_attention_mask = false, + bool use_cache = true, bool is_spec_decoding_enabled = false, + bool spec_decoding_is_generation_length_variable = false, int spec_decoding_max_generation_length = 1, + bool is_mla_enabled = false, int q_lora_rank = 0, int kv_lora_rank = 0, int qk_nope_head_dim = 0, + int qk_rope_head_dim = 0, int v_head_dim = 0, bool fuse_fp4_quant = false, bool skip_attn = false, + int cp_size = 1, int cp_rank = 0, std::set<int32_t> cp_group = {}); + + GPTAttentionPlugin(void const* data, size_t length); + + ~GPTAttentionPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + template <typename T, typename AttentionOutT, typename KVCacheBuffer> + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + template <typename T, typename AttentionOutT = T> + int enqueueDispatchKVCacheType(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream); + + template <typename T, typename KVCacheBuffer> + void configurePluginImpl(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept; + template <typename T> + void configurePluginDispatchKVCacheType(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + + //! This is called on every trt ExecutionContext creation by TRT + //! Note TRT does not call the initialize on cloned plugin, so clone internally should do initialization. + GPTAttentionPlugin* clone() const noexcept override; + + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + +private: + template <typename T, typename AttentionOutT, typename KVCacheBuffer> + int enqueueSome(int32_t seqIdxBeg, int32_t localNbSeq, int32_t tokenIdxBeg, int32_t localNbTokens, + nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + using IndexType = std::int32_t; + + std::vector<size_t> mEntryIdx; + enum class IdxEntry : size_t + { + QKV_TENSOR, + K_TENSOR, + V_TENSOR, + ATTENTION_MASK, + ATTENTION_PACKED_MASK, + SEQUENCE_LENGTH, + HOST_PAST_KEY_VALUE_LENGTHS, + HOST_MAX_ATTENTION_WINDOW, + HOST_SINK_TOKEN_LENGTH, + CONTEXT_LENGTHS, + CACHE_INDIR, + REQUEST_TYPES, + KV_CACHE_BLOCK_OFFSETS, + HOST_KV_CACHE_BLOCK_OFFSETS, + HOST_KV_CACHE_POOL_POINTERS, + HOST_KV_CACHE_POOL_MAPPING, + PAST_KEY_VALUE, + KV_CACHE_QUANTIZATION_SCALE, + KV_CACHE_DEQUANTIZATION_SCALE, + ATTENTION_OUTPUT_QUANTIZATION_SCALE, + ATTENTION_OUTPUT_SF_SCALE, + ROTARY_INV_FREQ, + ROTARY_COS_SIN, + ALIBI_SLOPES, + RELATIVE_ATTENTION_BIAS, + CROSS_KV, + CROSS_KV_LENGTH, + ENCODER_INPUT_LENGTH, + HOST_CONTEXT_LENGTH, + QKV_BIAS_TENSOR, + SPEC_DECODING_GENERATION_LENGTHS, + SPEC_DECODING_PACKED_MASK, + SPEC_DECODING_POSITION_OFFSETS, + SPEC_DECODING_USE, + LONG_ROPE_ROTARY_INV_FREQ, + LONG_ROPE_ROTARY_COS_SIN, + MROPE_ROTARY_COS_SIN, + MROPE_POSITION_DELTAS, + HOST_RUNTIME_PERF_KNOBS, + HOST_CONTEXT_PROGRESS, + MLA_Q_B_PROJ_TENSOR, + MLA_KV_B_PROJ_TENSOR, + MLA_K_B_PROJ_TRANS_TENSOR, + SKIP_ATTN, + LOGN_SCALING, + ENUM_SIZE, // Used to count the number of IdxEntry, must put in last + }; + + std::string toString(IdxEntry const& entry) const; + bool isEntryUsed(IdxEntry const& entry) const; + void initEntryIdx(); + IndexType getIdx(IdxEntry const& entry) const; + + // Get generation input sequence length (might be larger than 1 in the speculative decoding mode). + int getGenerationInputSequenceLength( + nvinfer1::PluginTensorDesc const* inputDesc, int32_t localNbSeq, int32_t localNbTokens) const; +}; + +class GPTAttentionPluginCreator : public GPTAttentionPluginCreatorCommon +{ +public: + GPTAttentionPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/identityPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/identityPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/identityPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/identityPlugin/identityPlugin.cpp b/cpp/tensorrt_llm/plugins/identityPlugin/identityPlugin.cpp new file mode 100644 index 000000000000..109010e7a933 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/identityPlugin/identityPlugin.cpp @@ -0,0 +1,199 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "identityPlugin.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" + +using namespace nvinfer1; +using tensorrt_llm::plugins::IdentityPluginCreator; +using tensorrt_llm::plugins::IdentityPlugin; + +static char const* IDENTITY_PLUGIN_VERSION{"1"}; +static char const* IDENTITY_PLUGIN_NAME{"Identity"}; +PluginFieldCollection IdentityPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> IdentityPluginCreator::mPluginAttributes; + +IdentityPlugin::IdentityPlugin() {} + +// Parameterized constructor +IdentityPlugin::IdentityPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* IdentityPlugin::clone() const noexcept +{ + auto* plugin = new IdentityPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs IdentityPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + return inputs[outputIndex]; +} + +bool IdentityPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + assert(0 <= pos && pos < 2); + PluginTensorDesc const& input = inOut[0]; + PluginTensorDesc const& output = inOut[1]; + switch (pos) + { + case 0: return input.format == nvinfer1::TensorFormat::kLINEAR; + case 1: return output.type == input.type && output.format == nvinfer1::TensorFormat::kLINEAR; + } + return false; +} + +void IdentityPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t IdentityPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int IdentityPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + size_t count = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims; ++i) + { + count *= inputDesc[0].dims.d[i]; + } + count *= tensorrt_llm::runtime::BufferDataType(inputDesc[0].type).getSize(); + + cudaMemcpyAsync(outputs[0], inputs[0], count, cudaMemcpyDeviceToDevice, stream); + + sync_check_cuda_error(stream); + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType IdentityPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + assert(index == 0); + return inputTypes[0]; +} + +// IPluginV2 Methods + +char const* IdentityPlugin::getPluginType() const noexcept +{ + return IDENTITY_PLUGIN_NAME; +} + +char const* IdentityPlugin::getPluginVersion() const noexcept +{ + return IDENTITY_PLUGIN_VERSION; +} + +int IdentityPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int IdentityPlugin::initialize() noexcept +{ + return 0; +} + +void IdentityPlugin::terminate() noexcept {} + +size_t IdentityPlugin::getSerializationSize() const noexcept +{ + return 0; +} + +void IdentityPlugin::serialize(void* buffer) const noexcept {} + +void IdentityPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +IdentityPluginCreator::IdentityPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* IdentityPluginCreator::getPluginName() const noexcept +{ + return IDENTITY_PLUGIN_NAME; +} + +char const* IdentityPluginCreator::getPluginVersion() const noexcept +{ + return IDENTITY_PLUGIN_VERSION; +} + +PluginFieldCollection const* IdentityPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* IdentityPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + try + { + auto* obj = new IdentityPlugin(); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* IdentityPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call IdentityPlugin::destroy() + try + { + auto* obj = new IdentityPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/identityPlugin/identityPlugin.h b/cpp/tensorrt_llm/plugins/identityPlugin/identityPlugin.h new file mode 100644 index 000000000000..9ab10601ae59 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/identityPlugin/identityPlugin.h @@ -0,0 +1,89 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class IdentityPlugin : public BasePlugin +{ +public: + IdentityPlugin(); + + IdentityPlugin(void const* data, size_t length); + + ~IdentityPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + const std::string mLayerName; +}; + +class IdentityPluginCreator : public BaseCreator +{ +public: + IdentityPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/CMakeLists.txt new file mode 100755 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/layernormQuantizationPlugin.cpp b/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/layernormQuantizationPlugin.cpp new file mode 100644 index 000000000000..02a40a00c919 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/layernormQuantizationPlugin.cpp @@ -0,0 +1,472 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "layernormQuantizationPlugin.h" +#include "pluginUtils.h" +#include "tensorrt_llm/kernels/layernormKernels.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::LayernormQuantizationPluginCreator; +using tensorrt_llm::plugins::LayernormQuantizationPlugin; + +static char const* LAYERNORM_QUANTIZATION_PLUGIN_VERSION{"1"}; +static char const* LAYERNORM_QUANTIZATION_PLUGIN_NAME{"LayernormQuantization"}; +PluginFieldCollection LayernormQuantizationPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> LayernormQuantizationPluginCreator::mPluginAttributes; + +LayernormQuantizationPlugin::LayernormQuantizationPlugin(float eps, bool useDiffOfSquares, + bool dynamicActivationScaling, bool sumPerToken, bool clampValEnabled, tensorrt_llm::common::QuantMode quantMode, + nvinfer1::DataType type, nvinfer1::DataType outputType) + : mEps(eps) + , mUseDiffOfSquares(useDiffOfSquares) + , mDynActScaling(dynamicActivationScaling) + , mType(type) + , mOutputType(outputType) + , mClampValEnabled(clampValEnabled) + , mQuantMode(quantMode) + , mSumPerToken(sumPerToken) +{ + TLLM_CHECK_WITH_INFO(mOutputType == nvinfer1::DataType::kINT8 || mOutputType == nvinfer1::DataType::kFP8, + "Only int8 or fp8 output type is allowed."); + // Check if the quant mode is valid. + TLLM_CHECK_WITH_INFO(mQuantMode.hasPerTokenScaling(), "The quant mode is not valid."); +} + +// Parameterized constructor +LayernormQuantizationPlugin::LayernormQuantizationPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mEps); + read(d, mUseDiffOfSquares); + read(d, mDynActScaling); + read(d, mSumPerToken); + read(d, mClampValEnabled); + read(d, mQuantMode); + read(d, mType); + read(d, mOutputType); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* LayernormQuantizationPlugin::clone() const noexcept +{ + auto* plugin = new LayernormQuantizationPlugin( + mEps, mUseDiffOfSquares, mDynActScaling, mSumPerToken, mClampValEnabled, mQuantMode, mType, mOutputType); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs LayernormQuantizationPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + if (outputIndex == 0) + { + // Quantized output + return inputs[outputIndex]; + } + + // Dynamic scaling or per-token sum if enabled + try + { + if (outputIndex == 1) + { + TLLM_CHECK(mDynActScaling); + } + else if (outputIndex == 2) + { + TLLM_CHECK(mSumPerToken); + } + else + { + TLLM_CHECK(false); + } + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int di = 0; di < ret.nbDims - 1; ++di) + { + ret.d[di] = inputs[0].d[di]; + } + ret.d[ret.nbDims - 1] = exprBuilder.constant(1); + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool LayernormQuantizationPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + int const totalPoses + = 6 + static_cast<int>(mClampValEnabled) + static_cast<int>(mDynActScaling) + static_cast<int>(mSumPerToken); + TLLM_CHECK(0 <= pos && pos < totalPoses); + TLLM_CHECK(nbInputs == 4 + static_cast<int>(mClampValEnabled)); + if (pos < nbInputs) + { + if (pos < 3) + { + // activatation, weight, bias + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == 3) + { + // scale + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == 4 && mClampValEnabled) + { + // clamp_max_v + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + } + else + { + auto const output_pos = pos - nbInputs; + if (output_pos == 0) + { + // Quantized output + return (inOut[pos].type == mOutputType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (output_pos == 1 && mDynActScaling) + { + // Dynamic scaling if enabled + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (output_pos == 2 && static_cast<int>(mClampValEnabled)) + { + // Clamp value + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + } + + // We should never reach this point + TLLM_CHECK_WITH_INFO(false, "The input/output is not supported."); + return false; +} + +void LayernormQuantizationPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t LayernormQuantizationPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int LayernormQuantizationPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // input [M(*), N] + // weight [N, ] + // bias [N, ] + // scale_to_int [1] + // clamp_max_v [2], contains min val, and max val (optional) + // outputs + // output [M(*), N] Normalized activations, potentially with quantization applied. + // dynamic_scaling [M(*), 1] (Optional) Per-token scales if quantization is enabled. + // token_sums [M(*), 1] (Optional) Per-token sums of all the channels (before quantization). + + int64_t m64 = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims - 1; ++i) + { + m64 *= inputDesc[0].dims.d[i]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[1].dims.d[0]); + + void const* input = inputs[0]; + void const* weight = inputs[1]; + void const* bias = inputs[2]; + void const* scale = inputs[3]; + void const* clampValPtr = mClampValEnabled ? inputs[4] : nullptr; + void* output = outputs[0]; + void* dynamic_scale = mDynActScaling ? outputs[1] : nullptr; + void* sum_per_token = mSumPerToken ? outputs[2] : nullptr; + + if (inputDesc[0].type == DataType::kFLOAT && mOutputType == DataType::kINT8) + { + dispatchDataType<float, int8_t>(nullptr, input, weight, bias, mEps, m, n, stream, mUseDiffOfSquares, + clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kFLOAT && mOutputType == DataType::kFP8) + { + dispatchDataType<float, __nv_fp8_e4m3>(nullptr, input, weight, bias, mEps, m, n, stream, mUseDiffOfSquares, + clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#endif // ENABLE_FP8 + else if (inputDesc[0].type == DataType::kHALF && mOutputType == DataType::kINT8) + { + dispatchDataType<half, int8_t>(nullptr, input, weight, bias, mEps, m, n, stream, mUseDiffOfSquares, clampValPtr, + scale, dynamic_scale, sum_per_token, output); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kHALF && mOutputType == DataType::kFP8) + { + dispatchDataType<half, __nv_fp8_e4m3>(nullptr, input, weight, bias, mEps, m, n, stream, mUseDiffOfSquares, + clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#endif // ENABLE_FP8 +#ifdef ENABLE_BF16 + else if (inputDesc[0].type == DataType::kBF16 && mOutputType == DataType::kINT8) + { + dispatchDataType<__nv_bfloat16, int8_t>(nullptr, input, weight, bias, mEps, m, n, stream, mUseDiffOfSquares, + clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kBF16 && mOutputType == DataType::kFP8) + { + dispatchDataType<__nv_bfloat16, __nv_fp8_e4m3>(nullptr, input, weight, bias, mEps, m, n, stream, + mUseDiffOfSquares, clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#endif // ENABLE_FP8 +#endif // ENABLE_BF16 + sync_check_cuda_error(stream); + return 0; +} + +template <typename T, typename QuantT> +void LayernormQuantizationPlugin::dispatchDataType(void* out, void const* input, void const* gamma, void const* beta, + float const eps, int const tokens, int const hidden_dim, cudaStream_t stream, bool use_diff_of_squares, + void const* clampValPtr, void const* scale, void* dynamic_scale, void* sum_per_token, + void* normed_output_quant) noexcept +{ + // inputs + // activation [dim0(*), dim1] + // clamp_value [2], contains min val, and max val (optional) + // outputs + // quant [dim0(*), dim1] + // scale_tokens [dim0(*), 1] + + invokeGeneralLayerNorm(reinterpret_cast<T*>(out), reinterpret_cast<T const*>(input), + reinterpret_cast<T const*>(gamma), reinterpret_cast<T const*>(beta), eps, tokens, hidden_dim, mQuantMode, + stream, use_diff_of_squares, reinterpret_cast<float const*>(clampValPtr), reinterpret_cast<float const*>(scale), + reinterpret_cast<float*>(dynamic_scale), reinterpret_cast<float*>(sum_per_token), + reinterpret_cast<QuantT*>(normed_output_quant)); +} + +// IPluginV2Ext Methods +nvinfer1::DataType LayernormQuantizationPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + assert(index <= 2); + + if (index == 0) + { + // Output 0 quantized output of layer norm + return mOutputType; + } + else if (index == 1) + { + assert(mDynActScaling); + // Output 1 dynamic act scaling + return nvinfer1::DataType::kFLOAT; + } + else if (index == 2) + { + assert(mDynActScaling && mSumPerToken); + // Output 2 per-token sums + return nvinfer1::DataType::kFLOAT; + } + + // We should never reach this point + TLLM_CHECK_WITH_INFO(false, "The output index is not supported."); + return nvinfer1::DataType::kFLOAT; +} + +// IPluginV2 Methods + +char const* LayernormQuantizationPlugin::getPluginType() const noexcept +{ + return LAYERNORM_QUANTIZATION_PLUGIN_NAME; +} + +char const* LayernormQuantizationPlugin::getPluginVersion() const noexcept +{ + return LAYERNORM_QUANTIZATION_PLUGIN_VERSION; +} + +int LayernormQuantizationPlugin::getNbOutputs() const noexcept +{ + return 1 + static_cast<int>(mDynActScaling) + static_cast<int>(mSumPerToken); +} + +int LayernormQuantizationPlugin::initialize() noexcept +{ + return 0; +} + +void LayernormQuantizationPlugin::terminate() noexcept {} + +size_t LayernormQuantizationPlugin::getSerializationSize() const noexcept +{ + return sizeof(mEps) + sizeof(mUseDiffOfSquares) + sizeof(mDynActScaling) + sizeof(mSumPerToken) + + sizeof(mClampValEnabled) + sizeof(mQuantMode) + sizeof(mType) + sizeof(mOutputType); +} + +void LayernormQuantizationPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mEps); + write(d, mUseDiffOfSquares); + write(d, mDynActScaling); + write(d, mSumPerToken); + write(d, mClampValEnabled); + write(d, mQuantMode); + write(d, mType); + write(d, mOutputType); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void LayernormQuantizationPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +LayernormQuantizationPluginCreator::LayernormQuantizationPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("eps", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("use_diff_of_squares", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("dyn_act_scaling", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("sum_per_token", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("clamp_val_enabled", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("quant_mode", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("out_type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* LayernormQuantizationPluginCreator::getPluginName() const noexcept +{ + return LAYERNORM_QUANTIZATION_PLUGIN_NAME; +} + +char const* LayernormQuantizationPluginCreator::getPluginVersion() const noexcept +{ + return LAYERNORM_QUANTIZATION_PLUGIN_VERSION; +} + +PluginFieldCollection const* LayernormQuantizationPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* LayernormQuantizationPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + tensorrt_llm::common::QuantMode quantMode{}; + float eps{}; + nvinfer1::DataType type{}; + nvinfer1::DataType outputType{}; + bool useDiffOfSquares{}; + bool dynamicActivationScaling{}; + bool sumPerToken{}; + bool clampValEnabled{}; + + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "eps")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + eps = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "dyn_act_scaling")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dynamicActivationScaling = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "use_diff_of_squares")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + useDiffOfSquares = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "sum_per_token")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + sumPerToken = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "clamp_val_enabled")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + clampValEnabled = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "quant_mode")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + quantMode = QuantMode(*(static_cast<int32_t const*>(fields[i].data))); + } + else if (!strcmp(attrName, "out_type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + outputType = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + try + { + auto* obj = new LayernormQuantizationPlugin( + eps, useDiffOfSquares, dynamicActivationScaling, sumPerToken, clampValEnabled, quantMode, type, outputType); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* LayernormQuantizationPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call LayernormQuantizationPlugin::destroy() + try + { + auto* obj = new LayernormQuantizationPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/layernormQuantizationPlugin.h b/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/layernormQuantizationPlugin.h new file mode 100644 index 000000000000..5cf3fa7e022f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/layernormQuantizationPlugin/layernormQuantizationPlugin.h @@ -0,0 +1,110 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class LayernormQuantizationPlugin : public BasePlugin +{ +public: + LayernormQuantizationPlugin(float eps, bool useDiffOfSquares, bool dynamicActivationScaling, bool sumPerToken, + bool clampValEnabled, tensorrt_llm::common::QuantMode quantMode, nvinfer1::DataType type, + nvinfer1::DataType outputType); + + LayernormQuantizationPlugin(void const* data, size_t length); + + ~LayernormQuantizationPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + template <typename T, typename QuantT> + void dispatchDataType(void* out, void const* input, void const* gamma, void const* beta, float const eps, + int const tokens, int const hidden_dim, cudaStream_t stream, bool use_diff_of_squares, void const* clampValPtr, + void const* scale, void* dynamic_scale, void* sum_per_token, void* normed_output_quant) noexcept; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + float mEps; + bool mUseDiffOfSquares; + bool mDynActScaling; + nvinfer1::DataType mType; + + const std::string mLayerName; + // The quantized output data type + nvinfer1::DataType mOutputType; + // Do we clamp the input tensor? + bool mClampValEnabled; + // The quantization mode + tensorrt_llm::common::QuantMode mQuantMode; + // Should we output the sum of channels per-token? (Used by QServe GEMM) + bool mSumPerToken; +}; + +class LayernormQuantizationPluginCreator : public BaseCreator +{ +public: + LayernormQuantizationPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/lookupPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/lookupPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lookupPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/lookupPlugin/lookupPlugin.cpp b/cpp/tensorrt_llm/plugins/lookupPlugin/lookupPlugin.cpp new file mode 100644 index 000000000000..e4d26f9e5ec6 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lookupPlugin/lookupPlugin.cpp @@ -0,0 +1,341 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include <cstdio> + +#include "lookupPlugin.h" +#include "tensorrt_llm/kernels/lookupKernels.h" +#include "tensorrt_llm/plugins/common/plugin.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::LookupPluginCreator; +using tensorrt_llm::plugins::LookupPlugin; + +static char const* LOOKUP_PLUGIN_VERSION{"1"}; +static char const* LOOKUP_PLUGIN_NAME{"Lookup"}; +PluginFieldCollection LookupPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> LookupPluginCreator::mPluginAttributes; + +LookupPlugin::LookupPlugin(nvinfer1::DataType type, int rank) + : mType(type) + , mRank(rank) +{ + mArch = tensorrt_llm::common::getSMVersion(); +} + +// Parameterized constructor +LookupPlugin::LookupPlugin(void const* data, size_t length) +{ + mArch = tensorrt_llm::common::getSMVersion(); + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mType); + read(d, mRank); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* LookupPlugin::clone() const noexcept +{ + auto* plugin = new LookupPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + plugin->initialize(); + return plugin; +} + +nvinfer1::DimsExprs LookupPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 2 || nbInputs == 3); + TLLM_CHECK(outputIndex == 0); + DimsExprs ret; + int const nbDimsInput = inputs[0].nbDims; + int const nbDimsWeight = inputs[1].nbDims; + ret.nbDims = nbDimsInput + 1; + + for (int i = 0; i < nbDimsInput; ++i) + { + ret.d[i] = inputs[0].d[i]; + } + ret.d[nbDimsInput] = inputs[1].d[nbDimsWeight - 1]; + + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool LookupPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + bool res = false; + if (nbInputs == 2) + { + switch (pos) + { + case 0: res = ((inOut[0].type == DataType::kINT32) && (inOut[0].format == TensorFormat::kLINEAR)); break; + case 1: res = ((inOut[1].type == mType) && (inOut[1].format == TensorFormat::kLINEAR)); break; + case 2: res = ((inOut[2].type == mType) && (inOut[2].format == TensorFormat::kLINEAR)); break; + default: // should NOT be here! + res = false; + } + } + else + { + TLLM_CHECK_WITH_INFO(mArch == 90, "int8 weight only lookupPlugin is only supported in SM 90 now."); + switch (pos) + { + case 0: res = ((inOut[0].type == DataType::kINT32) && (inOut[0].format == TensorFormat::kLINEAR)); break; + case 1: + res = ((inOut[1].type == DataType::kINT8 || inOut[1].type == mType) + && (inOut[1].format == TensorFormat::kLINEAR)); + break; + case 2: res = ((inOut[2].type == mType) && (inOut[2].format == TensorFormat::kLINEAR)); break; + case 3: res = ((inOut[3].type == mType) && (inOut[3].format == TensorFormat::kLINEAR)); break; + default: // should NOT be here! + res = false; + } + } + return res; +} + +void LookupPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + mNbInputs = nbInputs; +} + +size_t LookupPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int LookupPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + // inputs + // input [tokenNum] + // weight [localVocabSize, hidden] + // per_token_scales [localVocabSize], optional + // outputs + // embedding [tokenNum, hidden] + + int64_t tokenNum = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims; ++i) + { + tokenNum *= inputDesc[0].dims.d[i]; + } + + int const localVocabSize = inputDesc[1].dims.d[0]; + int const hidden = inputDesc[1].dims.d[inputDesc[1].dims.nbDims - 1]; + int const* input = reinterpret_cast<int const*>(inputs[0]); + + int offset = mRank * localVocabSize; + + if (mNbInputs == 3) + { + int8_t const* weight = reinterpret_cast<int8_t const*>(inputs[1]); + if (mType == DataType::kHALF) + { + half const* per_token_scales = reinterpret_cast<half const*>(inputs[2]); + half* output = reinterpret_cast<half*>(outputs[0]); + invokeLookUp<half, int8_t, int>( + output, input, weight, tokenNum, offset, localVocabSize, hidden, per_token_scales, stream); + } + else if (mType == DataType::kFLOAT) + { + float const* per_token_scales = reinterpret_cast<float const*>(inputs[2]); + float* output = reinterpret_cast<float*>(outputs[0]); + invokeLookUp<float, int8_t, int>( + output, input, weight, tokenNum, offset, localVocabSize, hidden, per_token_scales, stream); + } + else if (mType == DataType::kBF16) + { + __nv_bfloat16 const* per_token_scales = reinterpret_cast<__nv_bfloat16 const*>(inputs[2]); + __nv_bfloat16* output = reinterpret_cast<__nv_bfloat16*>(outputs[0]); + invokeLookUp<__nv_bfloat16, int8_t, int>( + output, input, weight, tokenNum, offset, localVocabSize, hidden, per_token_scales, stream); + } + } + else + { + if (mType == DataType::kHALF) + { + half const* weight = reinterpret_cast<half const*>(inputs[1]); + half* output = reinterpret_cast<half*>(outputs[0]); + invokeLookUp<half, half, int>( + output, input, weight, tokenNum, offset, localVocabSize, hidden, nullptr, stream); + } + else if (mType == DataType::kFLOAT) + { + float const* weight = reinterpret_cast<float const*>(inputs[1]); + float* output = reinterpret_cast<float*>(outputs[0]); + invokeLookUp<float, float, int>( + output, input, weight, tokenNum, offset, localVocabSize, hidden, nullptr, stream); + } + else if (mType == DataType::kBF16) + { + __nv_bfloat16 const* weight = reinterpret_cast<__nv_bfloat16 const*>(inputs[1]); + __nv_bfloat16* output = reinterpret_cast<__nv_bfloat16*>(outputs[0]); + invokeLookUp<__nv_bfloat16, __nv_bfloat16, int>( + output, input, weight, tokenNum, offset, localVocabSize, hidden, nullptr, stream); + } + } + sync_check_cuda_error(stream); + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType LookupPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* LookupPlugin::getPluginType() const noexcept +{ + return LOOKUP_PLUGIN_NAME; +} + +char const* LookupPlugin::getPluginVersion() const noexcept +{ + return LOOKUP_PLUGIN_VERSION; +} + +int LookupPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int LookupPlugin::initialize() noexcept +{ + return 0; +} + +void LookupPlugin::destroy() noexcept +{ + delete this; +} + +size_t LookupPlugin::getSerializationSize() const noexcept +{ + return sizeof(mType) + sizeof(mRank); +} + +void LookupPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mRank); + + TLLM_CHECK(d == a + getSerializationSize()); +} + +void LookupPlugin::terminate() noexcept {} + +/////////////// + +LookupPluginCreator::LookupPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("rank", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* LookupPluginCreator::getPluginName() const noexcept +{ + return LOOKUP_PLUGIN_NAME; +} + +char const* LookupPluginCreator::getPluginVersion() const noexcept +{ + return LOOKUP_PLUGIN_VERSION; +} + +PluginFieldCollection const* LookupPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* LookupPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + nvinfer1::DataType type{}; + int rank{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "rank")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + rank = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + } + try + { + auto* obj = new LookupPlugin(type, rank); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* LookupPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call LookupPlugin::destroy() + try + { + auto* obj = new LookupPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/lookupPlugin/lookupPlugin.h b/cpp/tensorrt_llm/plugins/lookupPlugin/lookupPlugin.h new file mode 100644 index 000000000000..4dddaa1d8bdc --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lookupPlugin/lookupPlugin.h @@ -0,0 +1,96 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class LookupPlugin : public BasePlugin +{ +public: + LookupPlugin() = delete; + + LookupPlugin(nvinfer1::DataType type, int rank); + + LookupPlugin(void const* data, size_t length); + + ~LookupPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + const std::string mLayerName; + + nvinfer1::DataType mType; + int mRank; + int mNbInputs = 0; + int mArch; +}; + +class LookupPluginCreator : public BaseCreator +{ +public: + LookupPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/loraPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/loraPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/loraPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/loraPlugin/loraPlugin.cpp b/cpp/tensorrt_llm/plugins/loraPlugin/loraPlugin.cpp new file mode 100644 index 000000000000..7a7d925a74f6 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/loraPlugin/loraPlugin.cpp @@ -0,0 +1,525 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "loraPlugin.h" + +#include "pluginUtils.h" +#include "tensorrt_llm/common/assert.h" + +#include <vector> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::LoraPluginCreator; +using tensorrt_llm::plugins::LoraPlugin; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +static char const* LORA_PLUGIN_VERSION{"1"}; +static char const* LORA_PLUGIN_NAME{"Lora"}; +PluginFieldCollection LoraPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> LoraPluginCreator::mPluginAttributes; + +LoraPlugin::LoraPlugin(int in_hidden_size, std::vector<int> out_hidden_sizes, int transA, int transB, + int num_lora_modules, nvinfer1::DataType type, LoraPlugin::PluginProfilerPtr const& pluginProfiler, + bool remove_input_padding, int max_low_rank, int weight_index) + : mTransA(transA) + , mTransB(transB) + , mType(type) + , mRemoveInputPadding(remove_input_padding) + , mNumLoraModules(num_lora_modules) + , mInHiddenSize(in_hidden_size) + , mMaxLowRank(max_low_rank) + , mWeightIndex(weight_index) + , mPluginProfiler(pluginProfiler) +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + mOutHiddenSizes.resize(mNumLoraModules); + mOutHiddenSizes.assign(out_hidden_sizes.begin(), out_hidden_sizes.end()); + init(); +} + +// Parameterized constructor +LoraPlugin::LoraPlugin(void const* data, size_t length, LoraPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mInHiddenSize); + read(d, mTransA); + read(d, mTransB); + read(d, mNumLoraModules); + read(d, mType); + read(d, mRemoveInputPadding); + read(d, mMaxLowRank); + read(d, mWeightIndex); + mOutHiddenSizes.resize(mNumLoraModules); + for (int i = 0; i < mNumLoraModules; i++) + { + read(d, mOutHiddenSizes[i]); + } + init(); + + mPluginProfiler->deserialize(d, mDims, mGemmId); + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +void LoraPlugin::init() +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + + auto cublasHandle = getCublasHandle(); + auto cublasLtHandle = getCublasLtHandle(); + auto cublasWraper = std::make_shared<CublasMMWrapper>(cublasHandle, cublasLtHandle, nullptr, nullptr); + + mLoraImpl = std::make_shared<kernels::LoraImpl>( + mInHiddenSize, mOutHiddenSizes, mTransA, mTransB, mNumLoraModules, mType, mMaxLowRank, cublasWraper); + + mPluginProfiler->setTranspose(mTransA, mTransB); + mGemmId = GemmIdCublas(mDims.n, mDims.k, mType, mTransA, mTransB, mType); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* LoraPlugin::clone() const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + auto* plugin = new LoraPlugin(*this); + return plugin; +} + +nvinfer1::DimsExprs LoraPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + try + { + TLLM_CHECK(outputIndex < mNumLoraModules); + int const nbDimsA = inputs[getInputTensorIdx()].nbDims; + DimsExprs ret; + ret.nbDims = nbDimsA; + + for (int i = 0; i < ret.nbDims; ++i) + { + ret.d[0] = 0; + } + + if (mTransA) + { + for (int i = 1; i < nbDimsA; ++i) + { + ret.d[i - 1] = inputs[getInputTensorIdx()].d[i]; + } + } + else + { + for (int i = 0; i < nbDimsA - 1; ++i) + { + ret.d[i] = inputs[getInputTensorIdx()].d[i]; + } + } + + auto const* outHiddenSize = exprBuilder.constant(mOutHiddenSizes.at(outputIndex)); + TLLM_CHECK(outHiddenSize != nullptr); + ret.d[ret.nbDims - 1] = outHiddenSize; + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool LoraPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + if (pos == getHostRequestTypesIdx()) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (pos >= getLoraRanksIdx() && pos < getLoraRanksIdx() + mNumLoraModules) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (pos >= getLoraWeightsPtrsIdx() && pos < getLoraWeightsPtrsIdx() + mNumLoraModules) + { + return inOut[pos].type == nvinfer1::DataType::kINT64; + } + else if (mRemoveInputPadding && pos == getHostContextLengthsIdx()) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else + { + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } +} + +void LoraPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + + auto const input = in[getInputTensorIdx()]; + + int const nbDimsA = input.max.nbDims; + + auto const minM = utils::computeMDimension(mTransA, input.min); + auto const maxM = utils::computeMDimension(mTransA, input.max); + auto const N = utils::computeNDimension(mTransB, in[getHostRequestTypesIdx()].max); + auto const K = static_cast<utils::DimType64>(mTransA ? input.max.d[0] : input.max.d[nbDimsA - 1]); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, N, K}; + } + mGemmId.n = N; + mGemmId.k = K; + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +size_t LoraPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + + int const nbReq = inputs[getLoraRanksIdx()].dims.d[0]; + auto const type = inputs[getInputTensorIdx()].type; + auto const numTokens = getNumTokens(inputs); + return mLoraImpl->getWorkspaceSize(numTokens, nbReq, type); +} + +int64_t LoraPlugin::getNumTokens(nvinfer1::PluginTensorDesc const* input_tensors) const +{ + int ndim = input_tensors[getInputTensorIdx()].dims.nbDims; + TLLM_CHECK_WITH_INFO( + 3 == ndim || 2 == ndim, "hidden_state dimension should be either 2 [numTokens, hidden], or 3 [b, s, hidden]"); + int64_t num_tokens = input_tensors[getInputTensorIdx()].dims.d[0]; + if (ndim == 3) + { + num_tokens *= input_tensors[getInputTensorIdx()].dims.d[1]; + } + return num_tokens; +} + +int LoraPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + if (isBuilding()) + { + return 0; + } + + auto const numReqs = inputDesc[getLoraRanksIdx()].dims.d[0]; + void const* input = inputs[getInputTensorIdx()]; + int const seqLen = mRemoveInputPadding ? 0 : inputDesc[getInputTensorIdx()].dims.d[1]; + int32_t const* reqTypes = static_cast<int32_t const*>(inputs[getHostRequestTypesIdx()]); + void const* const* loraRanks = &inputs[getLoraRanksIdx()]; + void const* const* loraWeightPtrs = &inputs[getLoraWeightsPtrsIdx()]; + int32_t const* hostContextLengths + = mRemoveInputPadding ? static_cast<int32_t const*>(inputs[getHostContextLengthsIdx()]) : nullptr; + + int numTokens = getNumTokens(inputDesc); + mExpandLoraWeightPtrs.clear(); + mExpandLoraRanks.clear(); + mExpandLoraWeightPtrs.reserve(mNumLoraModules * numTokens * 2); + mExpandLoraRanks.reserve(mNumLoraModules * numTokens); + + for (int loraModuleIdx = 0; loraModuleIdx < mNumLoraModules; loraModuleIdx++) + { + auto const loraWeightModulePtrs = static_cast<int64_t const*>(loraWeightPtrs[loraModuleIdx]); + auto const loraRankModule = static_cast<int32_t const*>(loraRanks[loraModuleIdx]); + + int idx = 0; + for (int reqId = 0; reqId < numReqs; reqId++) + { + // loraWeightModulePtrs has 3 pointers for each module: A,B, and an optional DoRA magnitude + // the current LoRA plugin does not apply DoRA scaling, so the magnitude is ignored + RequestType const reqType = static_cast<RequestType>(reqTypes[reqId]); + if (reqType == RequestType::kGENERATION) + { + mExpandLoraWeightPtrs.push_back(reinterpret_cast<void const*>(loraWeightModulePtrs[reqId * 3])); + mExpandLoraWeightPtrs.push_back(reinterpret_cast<void const*>(loraWeightModulePtrs[reqId * 3 + 1])); + mExpandLoraRanks.push_back(loraRankModule[reqId]); + idx += 1; + } + else + { + int contextLen = (mRemoveInputPadding ? hostContextLengths[reqId] : seqLen); + + for (int contextId = 0; contextId < contextLen; contextId++) + { + mExpandLoraWeightPtrs.push_back(reinterpret_cast<void const*>(loraWeightModulePtrs[reqId * 3])); + mExpandLoraWeightPtrs.push_back(reinterpret_cast<void const*>(loraWeightModulePtrs[reqId * 3 + 1])); + mExpandLoraRanks.push_back(loraRankModule[reqId]); + idx += 1; + } + } + } + + // In 1st generation phase cross attention qkv lora, cross qkv is skipped by passing an empty encoder_output + // (passing 0 to dim) getNumTokens() will get in cross qkv_lora. Skipping the check for this case. + if (numTokens > 0) + { + TLLM_CHECK_WITH_INFO(idx == numTokens, + fmtstr("LoraParams and input dims don't match, lora tokens %d input tokens %d", idx, numTokens)); + } + } + + // only used for unified gemm + auto bestTactic = mPluginProfiler->getBestConfig(numTokens, mGemmId); + mLoraImpl->setBestTactic(bestTactic); + mLoraImpl->run(numTokens, numReqs, input, mExpandLoraRanks.data(), mExpandLoraWeightPtrs.data(), mWeightIndex, + outputs, workspace, stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType LoraPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + TLLM_CHECK(index < mNumLoraModules); + return mType; +} + +// IPluginV2 Methods + +char const* LoraPlugin::getPluginType() const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + return LORA_PLUGIN_NAME; +} + +char const* LoraPlugin::getPluginVersion() const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + return LORA_PLUGIN_VERSION; +} + +int LoraPlugin::getNbOutputs() const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + return mNumLoraModules; +} + +int LoraPlugin::initialize() noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + if (!mDims.isInitialized()) + { + return 0; + } + + mLoraImpl->setGemmConfig(); + + mPluginProfiler->profileTactics(mLoraImpl->getCublasWrapper(), mType, mDims, mGemmId); + return 0; +} + +void LoraPlugin::destroy() noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + delete this; +} + +size_t LoraPlugin::getSerializationSize() const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + return sizeof(mInHiddenSize) + sizeof(mTransA) + sizeof(mTransB) + sizeof(mNumLoraModules) + sizeof(mType) + + mPluginProfiler->getSerializationSize(mGemmId) + sizeof(mRemoveInputPadding) + sizeof(mMaxLowRank) + + sizeof(mWeightIndex) + sizeof(int) * mNumLoraModules; // selected tactics container size +} + +void LoraPlugin::serialize(void* buffer) const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + char *d = static_cast<char*>(buffer), *a = d; + write(d, mInHiddenSize); + write(d, mTransA); + write(d, mTransB); + write(d, mNumLoraModules); + write(d, mType); + write(d, mRemoveInputPadding); + write(d, mMaxLowRank); + write(d, mWeightIndex); + for (int i = 0; i < mNumLoraModules; i++) + { + write(d, mOutHiddenSizes.at(i)); + } + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void LoraPlugin::terminate() noexcept {} + +/////////////// + +LoraPluginCreator::LoraPluginCreator() +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("transA", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("transB", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("num_lora_modules", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("weight_index", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* LoraPluginCreator::getPluginName() const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + return LORA_PLUGIN_NAME; +} + +char const* LoraPluginCreator::getPluginVersion() const noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + return LORA_PLUGIN_VERSION; +} + +PluginFieldCollection const* LoraPluginCreator::getFieldNames() noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + return &mFC; +} + +IPluginV2* LoraPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + + PluginField const* fields = fc->fields; + nvinfer1::DataType type{}; + int num_lora_modules{}; + int in_hidden_size{}; + int transA{}; + int transB{}; + bool remove_input_padding{}; + int max_low_rank{}; + int weight_index{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "in_hidden_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + in_hidden_size = *(static_cast<int32_t const*>(fields[i].data)); + } + else if (!strcmp(attrName, "transa")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + transA = *(static_cast<int const*>(fields[i].data)); + } + else if (!strcmp(attrName, "transb")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + transB = *(static_cast<int const*>(fields[i].data)); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "remove_input_padding")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + remove_input_padding = static_cast<bool>(*(static_cast<int8_t const*>(fields[i].data))); + } + else if (!strcmp(attrName, "max_low_rank")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + max_low_rank = *(static_cast<int const*>(fields[i].data)); + } + else if (!strcmp(attrName, "num_lora_modules")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + num_lora_modules = *(static_cast<int const*>(fields[i].data)); + } + else if (!strcmp(attrName, "weight_index")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + weight_index = *(static_cast<int const*>(fields[i].data)); + } + } + std::vector<int> out_hidden_sizes; + out_hidden_sizes.resize(num_lora_modules); + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + for (int j = 0; j < num_lora_modules; j++) + { + if (!strcmp(attrName, fmtstr("out_hidden_size_%d", j).c_str())) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + out_hidden_sizes.at(j) = *(static_cast<int const*>(fields[i].data)); + } + } + } + try + { + // LoraPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + // FIXME enable tactic profiler + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false, /* skip */ true); + auto* obj = new LoraPlugin(in_hidden_size, out_hidden_sizes, transA, transB, num_lora_modules, type, + pluginProfiler, remove_input_padding, max_low_rank, weight_index); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* LoraPluginCreator::deserializePlugin(char const* name, void const* serialData, size_t serialLength) noexcept +{ + TLLM_LOG_DEBUG("%s", __PRETTY_FUNCTION__); + // This object will be deleted when the network is destroyed, which will + // call LoraPlugin::destroy() + try + { + // LoraPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + // FIXME enable tactic profiler + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true, /* skip */ true); + auto* obj = new LoraPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/loraPlugin/loraPlugin.h b/cpp/tensorrt_llm/plugins/loraPlugin/loraPlugin.h new file mode 100644 index 000000000000..7795f7b7c76d --- /dev/null +++ b/cpp/tensorrt_llm/plugins/loraPlugin/loraPlugin.h @@ -0,0 +1,159 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef TRT_LORA_PLUGIN_H +#define TRT_LORA_PLUGIN_H +#include "tensorrt_llm/kernels/lora/lora.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/plugins/gemmPlugin/gemmPlugin.h" +#include <cassert> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class LoraPlugin : public BasePlugin +{ +public: + using PluginProfilerPtr = std::shared_ptr<CublasLtGemmPluginProfiler>; + using ImplPtr = std::shared_ptr<kernels::LoraImpl>; + using Config = cublasLtMatmulHeuristicResult_t; + + LoraPlugin() = delete; + + LoraPlugin(int in_hidden_size, std::vector<int> out_hidden_sizes, int transA, int transB, int num_lora_modules, + nvinfer1::DataType type, PluginProfilerPtr const& profiler, bool remove_input_padding, int max_low_rank, + int weight_index); + + LoraPlugin(void const* data, size_t length, PluginProfilerPtr const& profiler); + + ~LoraPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + int64_t getNumTokens(nvinfer1::PluginTensorDesc const* input_tensors) const; + void init(); + + using IndexType = std::int32_t; + + IndexType getInputTensorIdx() const + { + return 0; + } + + IndexType getHostRequestTypesIdx() const + { + return 1; + } + + IndexType getLoraRanksIdx() const + { + return 2; + } + + IndexType getLoraWeightsPtrsIdx() const + { + return 2 + mNumLoraModules; + } + + IndexType getHostContextLengthsIdx() const + { + TLLM_CHECK(mRemoveInputPadding); + return 2 + mNumLoraModules + mNumLoraModules; + } + + enum class RequestType : int32_t + { + kCONTEXT = 0, + kGENERATION = 1 + }; + +private: + const std::string mLayerName; + + std::vector<int> mOutHiddenSizes; + int mTransA; + int mTransB; + nvinfer1::DataType mType; + bool mRemoveInputPadding; + int mNumLoraModules; + int mInHiddenSize; + int mMaxLowRank; + int mWeightIndex; + + std::vector<void const*> mExpandLoraWeightPtrs{}; + std::vector<int32_t> mExpandLoraRanks{}; + + GemmDims mDims{}; + GemmIdCublas mGemmId{}; + + PluginProfilerPtr mPluginProfiler; + ImplPtr mLoraImpl; +}; + +class LoraPluginCreator : public BaseCreator +{ +public: + LoraPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<CublasLtGemmPluginProfiler> gemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins + +#endif // TRT_LORA_PLUGIN_H diff --git a/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/CMakeLists.txt new file mode 100644 index 000000000000..b6bd0439cc0c --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/lowLatencyGemmPlugin.cpp b/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/lowLatencyGemmPlugin.cpp new file mode 100644 index 000000000000..6165d6210f29 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/lowLatencyGemmPlugin.cpp @@ -0,0 +1,425 @@ + +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "lowLatencyGemmPlugin.h" +#include "low_latency_gemm.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaFp8Utils.h" +#include "tensorrt_llm/common/logger.h" +#include <NvInferRuntime.h> +#include <NvInferRuntimeBase.h> +#include <NvInferRuntimePlugin.h> +#include <cstddef> +#include <cstdint> +#include <cstdio> +#include <numeric> +#include <optional> +#include <vector> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels::internal_cutlass_kernels; +using tensorrt_llm::plugins::LowLatencyGemmPluginCreator; +using tensorrt_llm::plugins::LowLatencyGemmPlugin; +using tensorrt_llm::plugins::LowLatencyGemmPluginProfiler; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +static char const* LOW_LATENCY_GEMM_PLUGIN_VERSION{"1"}; +static char const* LOW_LATENCY_GEMM_PLUGIN_NAME{"LowLatencyGemm"}; + +PluginFieldCollection LowLatencyGemmPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> LowLatencyGemmPluginCreator::mPluginAttributes; + +using FP8Type = __nv_fp8_e4m3; + +static std::optional<float> getFloatEnv(char const* name) +{ + char const* const env = std::getenv(name); + if (env == nullptr) + { + return std::nullopt; + } + try + { + float value = std::stof(env); + return {value}; + } + catch (std::invalid_argument const& e) + { + return std::nullopt; + } + catch (std::out_of_range const& e) + { + return std::nullopt; + } +}; + +void LowLatencyGemmPluginProfiler::runTactic(int m, int n, int k, LowLatencyGemmPluginProfiler::Config const& tactic, + char* workspace, cudaStream_t const& stream) +{ + + float default_pdl_overlap_ratio = 0.5; + float default_prefetch_ratio = -1.0; + FP8Type* aTmp = reinterpret_cast<FP8Type*>(workspace); + FP8Type* bTmp + = reinterpret_cast<FP8Type*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(aTmp), m * k * sizeof(FP8Type))); + void* cTmp = reinterpret_cast<void*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(bTmp), n * k * sizeof(FP8Type))); + size_t workspaceSize = mRunner->getWorkspaceSize(m, n, k); + char* workspaceTmp = reinterpret_cast<char*>(nextWorkspacePtr( + reinterpret_cast<int8_t*>(cTmp), m * n * (mType == nvinfer1::DataType::kFLOAT ? sizeof(float) : sizeof(half)))); + mRunner->gemm(aTmp, bTmp, 1.0f, 0.0f, nullptr, cTmp, m, n, k, default_pdl_overlap_ratio, default_prefetch_ratio, + tactic, workspaceTmp, workspaceSize, stream); +} + +void LowLatencyGemmPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + + std::vector<size_t> workspaces = {maxM * k * sizeof(FP8Type), n * k * sizeof(FP8Type), + maxM * n * (mType == nvinfer1::DataType::kFLOAT ? sizeof(float) : sizeof(half)), + mRunner->getWorkspaceSize(maxM, n, k)}; + + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<LowLatencyGemmPluginProfiler::Config> LowLatencyGemmPluginProfiler::getTactics(int m, int n, int k) const +{ + return mRunner->getConfigs(); +} + +LowLatencyGemmPlugin::LowLatencyGemmPlugin( + nvinfer1::DataType type, float alpha, PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) + , mAplha(alpha) +{ + init(type); +} + +LowLatencyGemmPlugin::LowLatencyGemmPlugin(void const* data, size_t length, PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + + char const *d = reinterpret_cast<char const*>(data), *a = d; + nvinfer1::DataType type; + read(d, type); + read(d, mAplha); + read(d, mDims); + init(type); + mPluginProfiler->deserialize(d, mDims, mGemmId); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +void LowLatencyGemmPlugin::init(nvinfer1::DataType type) +{ + + mType = type; + + if (mType == nvinfer1::DataType::kFLOAT) + { + m_lowLatencyGemmRunner = std::make_shared<CutlassLowLatencyFp8GemmRunner<float>>(); + } + else if (mType == nvinfer1::DataType::kHALF) + { + m_lowLatencyGemmRunner = std::make_shared<CutlassLowLatencyFp8GemmRunner<half>>(); + } +#ifdef ENABLE_BF16 + + else if (mType == nvinfer1::DataType::kBF16) + { + m_lowLatencyGemmRunner = std::make_shared<CutlassLowLatencyFp8GemmRunner<__nv_bfloat16>>(); + } +#endif + else + { + TLLM_THROW("Unsupported data type"); + } + mGemmId = GemmIdCore(mDims.n, mDims.k, mType); +} + +nvinfer1::DimsExprs LowLatencyGemmPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 2); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + TLLM_CHECK(nbDimsA >= 2); + DimsExprs ret; + ret.nbDims = nbDimsA; + for (int ii = 0; ii < nbDimsA - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + // input[1] , weights [n,k] + ret.d[nbDimsA - 1] = inputs[1].d[0]; + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool LowLatencyGemmPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + switch (pos) + { + case 0: + // activation + return inOut[pos].type == nvinfer1::DataType::kFP8 && inOut[pos].format == TensorFormat::kLINEAR; + case 1: + // weights + // Weights stored in checkpoint must have fp8 type + return inOut[pos].type == nvinfer1::DataType::kFP8 && inOut[pos].format == TensorFormat::kLINEAR; + case 2: + // out + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + default: + // Never should be here + assert(false); + return false; + } +} + +void LowLatencyGemmPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + int const maxN = in[1].max.d[0]; + int const minK = in[0].min.d[in[0].min.nbDims - 1]; + int const minN = in[1].min.d[0]; + + TLLM_CHECK_WITH_INFO(minN == maxN, "Variable out channels is not allowed"); + TLLM_CHECK_WITH_INFO(minK == maxK, "Variable in channels is not allowed"); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, maxN, maxK}; + } + mGemmId = {maxN, maxK, mType}; + + m_workspaceMaxSize = m_lowLatencyGemmRunner->getWorkspaceSize(maxM, maxN, maxK); +} + +size_t LowLatencyGemmPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return m_workspaceMaxSize; +} + +int LowLatencyGemmPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + + // input0 activation [M,K] + // input1 weights [N,K] + // output0 [M,N] + + int64_t m64 = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m64 *= inputDesc[0].dims.d[ii]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[1].dims.d[0]); + int const k = TLLM_INT32_CAST(inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]); + int const wsSize = m_lowLatencyGemmRunner->getWorkspaceSize(m, n, k); + auto const& bestTactic = mPluginProfiler->getBestConfig(m, mGemmId); + TLLM_CHECK_WITH_INFO(bestTactic, "No valid Low Latency GEMM tactic"); + + auto env_pdl_overlap_ratio = getFloatEnv("TRTLLM_PDL_OVERLAP_RATIO"); + auto env_prefetch_ratio = getFloatEnv("TRTLLM_PREFETCH_RATIO"); + auto valid_ratio = [](std::optional<float>& env_val, float default_val) + { + if (env_val.has_value()) + { + TLLM_CHECK_WITH_INFO(env_val.value() <= 1.0f, "Valid ratio should be less than or equal to 1.0"); + return env_val.value(); + } + return default_val; + }; + float pdl_overlap_ratio = valid_ratio(env_pdl_overlap_ratio, /*default_val=*/0.5); + float prefetch_ratio = valid_ratio(env_prefetch_ratio, /*default_val=*/-1.0); + m_lowLatencyGemmRunner->gemm(const_cast<FP8Type*>(reinterpret_cast<FP8Type const*>(inputs[0])), + const_cast<FP8Type*>(reinterpret_cast<FP8Type const*>(inputs[1])), mAplha, 0.0F, nullptr, outputs[0], m, n, k, + pdl_overlap_ratio, prefetch_ratio, *bestTactic, reinterpret_cast<char*>(workspace), wsSize, stream); + + return 0; +} + +nvinfer1::DataType LowLatencyGemmPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* LowLatencyGemmPlugin::getPluginType() const noexcept +{ + return LOW_LATENCY_GEMM_PLUGIN_NAME; +} + +char const* LowLatencyGemmPlugin::getPluginVersion() const noexcept +{ + return LOW_LATENCY_GEMM_PLUGIN_VERSION; +} + +int LowLatencyGemmPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int LowLatencyGemmPlugin::initialize() noexcept +{ + configGemm(); + return 0; +} + +void LowLatencyGemmPlugin::terminate() noexcept {} + +nvinfer1::IPluginV2DynamicExt* LowLatencyGemmPlugin::clone() const noexcept +{ + auto* plugin = new LowLatencyGemmPlugin(*this); + return plugin; +} + +size_t LowLatencyGemmPlugin::getSerializationSize() const noexcept +{ + return sizeof(nvinfer1::DataType) + // dtype + sizeof(float) * 1 + // alpha + sizeof(mDims) + mPluginProfiler->getSerializationSize(mGemmId); +} + +void LowLatencyGemmPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mAplha); + write(d, mDims); + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void LowLatencyGemmPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +void LowLatencyGemmPlugin::configGemm() +{ + mPluginProfiler->profileTactics(m_lowLatencyGemmRunner, mType, mDims, mGemmId); +} + +LowLatencyGemmPluginCreator::LowLatencyGemmPluginCreator() +{ + + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("alpha", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* LowLatencyGemmPluginCreator::getPluginName() const noexcept +{ + return LOW_LATENCY_GEMM_PLUGIN_NAME; +} + +char const* LowLatencyGemmPluginCreator::getPluginVersion() const noexcept +{ + return LOW_LATENCY_GEMM_PLUGIN_VERSION; +} + +PluginFieldCollection const* LowLatencyGemmPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* LowLatencyGemmPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + float alpha{}; + nvinfer1::DataType type{}; + for (int i = 0; i < fc->nbFields; i++) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "alpha")) + { + + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + alpha = *(static_cast<float const*>(fields[i].data)); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + + try + { + + // + // GemmPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/*inference=*/false); + auto* obj = new LowLatencyGemmPlugin(type, alpha, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* LowLatencyGemmPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + try + { + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/*inference=*/true); + auto* obj = new LowLatencyGemmPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/lowLatencyGemmPlugin.h b/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/lowLatencyGemmPlugin.h new file mode 100644 index 000000000000..98b8f4807174 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lowLatencyGemmPlugin/lowLatencyGemmPlugin.h @@ -0,0 +1,135 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "low_latency_gemm.h" + +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <cstddef> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +using LowLatencyGemmRunnerPtr + = std::shared_ptr<tensorrt_llm::kernels::internal_cutlass_kernels::CutlassLowLatencyFp8GemmRunnerInterface>; + +class LowLatencyGemmPluginProfiler + : public GemmPluginProfiler< + tensorrt_llm::kernels::internal_cutlass_kernels::CutlassLowLatencyFp8GemmRunnerInterface::ConfigType, + LowLatencyGemmRunnerPtr, GemmIdCore, GemmIdCoreHash> +{ + +public: + using Config = tensorrt_llm::kernels::internal_cutlass_kernels::CutlassLowLatencyFp8GemmRunnerInterface::ConfigType; + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + std::vector<Config> getTactics(int m, int n, int k) const override; +}; + +class LowLatencyGemmPlugin : public BasePlugin +{ + +public: + using PluginProfilerPtr = std::shared_ptr<LowLatencyGemmPluginProfiler>; + + LowLatencyGemmPlugin() = delete; + + LowLatencyGemmPlugin(nvinfer1::DataType type, float alpha, PluginProfilerPtr const& pluginProfiler); + + LowLatencyGemmPlugin(void const* data, size_t length, PluginProfilerPtr const& pluginProfiler); + ~LowLatencyGemmPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + void init(nvinfer1::DataType type); + void configGemm(); + +private: + std::string const mLayerName; + + LowLatencyGemmRunnerPtr m_lowLatencyGemmRunner; + size_t m_workspaceMaxSize; + + GemmDims mDims{}; + GemmIdCore mGemmId{}; + + PluginProfilerPtr mPluginProfiler; + + nvinfer1::DataType mType; + float mAplha{1.0F}; +}; + +class LowLatencyGemmPluginCreator : public BaseCreator +{ +public: + LowLatencyGemmPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<LowLatencyGemmPluginProfiler> gemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/CMakeLists.txt new file mode 100644 index 000000000000..b6bd0439cc0c --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/lowLatencyGemmSwigluPlugin.cpp b/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/lowLatencyGemmSwigluPlugin.cpp new file mode 100644 index 000000000000..a1aa11c2f165 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/lowLatencyGemmSwigluPlugin.cpp @@ -0,0 +1,468 @@ + +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "lowLatencyGemmSwigluPlugin.h" +#include "low_latency_gemm_swiglu.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/cudaFp8Utils.h" +#include "tensorrt_llm/common/logger.h" +#include <NvInferRuntime.h> +#include <NvInferRuntimeBase.h> +#include <NvInferRuntimePlugin.h> +#include <cstddef> +#include <cstdint> +#include <cstdio> +#include <numeric> +#include <optional> +#include <vector> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels::internal_cutlass_kernels; +using tensorrt_llm::plugins::LowLatencyGemmSwigluPluginCreator; +using tensorrt_llm::plugins::LowLatencyGemmSwigluPlugin; +using tensorrt_llm::plugins::LowLatencyGemmSwigluPluginProfiler; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +static char const* LOW_LATENCY_GEMM_SWIGLU_PLUGIN_VERSION{"1"}; +static char const* LOW_LATENCY_GEMM_SWIGLU_PLUGIN_NAME{"LowLatencyGemmSwiglu"}; + +PluginFieldCollection LowLatencyGemmSwigluPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> LowLatencyGemmSwigluPluginCreator::mPluginAttributes; + +using FP8Type = __nv_fp8_e4m3; + +static std::optional<float> getFloatEnv(char const* name) +{ + char const* const env = std::getenv(name); + if (env == nullptr) + { + return std::nullopt; + } + try + { + float value = std::stof(env); + return {value}; + } + catch (std::invalid_argument const& e) + { + return std::nullopt; + } + catch (std::out_of_range const& e) + { + return std::nullopt; + } +}; + +static size_t getBytePerElement(nvinfer1::DataType type) +{ + size_t bpe; + if (type == nvinfer1::DataType::kFLOAT) + { + bpe = 4; + } + else if (type == nvinfer1::DataType::kHALF || type == nvinfer1::DataType::kBF16) + { + bpe = 2; + } + else if (type == nvinfer1::DataType::kINT8 || type == nvinfer1::DataType::kFP8) + { + bpe = 1; + } + else + { + TLLM_THROW("Not recognized/implemented"); + } + return bpe; +} + +void LowLatencyGemmSwigluPluginProfiler::runTactic(int m, int n, int k, + LowLatencyGemmSwigluPluginProfiler::Config const& tactic, char* workspace, cudaStream_t const& stream) +{ + + float default_pdl_overlap_ratio = 0.5; + float default_prefetch_ratio = -1.0; + FP8Type* aTmp = reinterpret_cast<FP8Type*>(workspace); + FP8Type* bTmp + = reinterpret_cast<FP8Type*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(aTmp), m * k * sizeof(FP8Type))); + void* dTmp = reinterpret_cast<void*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(bTmp), n * k * sizeof(FP8Type))); + size_t workspaceSize = mRunner->getWorkspaceSize(m, n, k); + char* workspaceTmp = reinterpret_cast<char*>( + nextWorkspacePtr(reinterpret_cast<int8_t*>(dTmp), (n / 2 * m * getBytePerElement(mType)))); + mRunner->gemm(aTmp, bTmp, 1.0f, 0.0f, 1.0f, 1.0f, nullptr, dTmp, m, n, k, default_pdl_overlap_ratio, + default_prefetch_ratio, tactic, workspaceTmp, workspaceSize, stream); +} + +int LowLatencyGemmSwigluPluginProfiler::getMaxProfileM() const +{ + return 32768; +} + +void LowLatencyGemmSwigluPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + + std::vector<size_t> workspaces = {maxM * k * sizeof(FP8Type), // A + n * k * sizeof(FP8Type), // B + maxM * (n / 2) * getBytePerElement(mType), // D + mRunner->getWorkspaceSize(maxM, n, k)}; // workspace + + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<LowLatencyGemmSwigluPluginProfiler::Config> LowLatencyGemmSwigluPluginProfiler::getTactics( + int m, int n, int k) const +{ + return mRunner->getConfigs(); +} + +LowLatencyGemmSwigluPlugin::LowLatencyGemmSwigluPlugin(nvinfer1::DataType type, float scale_output, float scale_d0, + float scale_d1, PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) + , mScaleOutput(scale_output) + , mScaleD0(scale_d0) + , mScaleD1(scale_d1) +{ + init(type); +} + +LowLatencyGemmSwigluPlugin::LowLatencyGemmSwigluPlugin( + void const* data, size_t length, PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + + char const *d = reinterpret_cast<char const*>(data), *a = d; + nvinfer1::DataType type; + read(d, type); + read(d, mScaleOutput); + read(d, mScaleD0); + read(d, mScaleD1); + read(d, mDims); + + init(type); + mPluginProfiler->deserialize(d, mDims, mGemmId); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +void LowLatencyGemmSwigluPlugin::init(nvinfer1::DataType type) +{ + + mType = type; + + if (mType == nvinfer1::DataType::kFP8) + { + mLowLatencyGemmSwigluRunner = std::make_shared<CutlassLowLatencyFp8GemmSwigluRunner<__nv_fp8_e4m3>>(); + } + else + { + TLLM_THROW("Unsupported data type"); + } + mGemmId = GemmIdCore(mDims.n, mDims.k, mType); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* LowLatencyGemmSwigluPlugin::clone() const noexcept +{ + auto* plugin = new LowLatencyGemmSwigluPlugin(*this); + return plugin; +} + +nvinfer1::DimsExprs LowLatencyGemmSwigluPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 2); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + TLLM_CHECK(nbDimsA >= 2); + DimsExprs ret; + ret.nbDims = nbDimsA; + for (int ii = 0; ii < nbDimsA - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + ret.d[nbDimsA - 1] = exprBuilder.constant(inputs[1].d[1]->getConstantValue() / 2); + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool LowLatencyGemmSwigluPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + switch (pos) + { + case 0: + // activation + return inOut[pos].type == nvinfer1::DataType::kFP8 && inOut[pos].format == TensorFormat::kLINEAR; + case 1: + // weights + // Weights stored in checkpoint must have fp8 type + return inOut[pos].type == nvinfer1::DataType::kFP8 && inOut[pos].format == TensorFormat::kLINEAR; + case 2: + // out + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + default: + // Never should be here + TLLM_CHECK(false); + return false; + } +} + +void LowLatencyGemmSwigluPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + int const maxN = in[1].max.d[1]; + int const minK = in[0].min.d[in[0].min.nbDims - 1]; + int const minN = in[1].min.d[1]; + + TLLM_CHECK_WITH_INFO(minN == maxN, "Variable out channels is not allowed"); + TLLM_CHECK_WITH_INFO(minK == maxK, "Variable in channels is not allowed"); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, maxN, maxK}; + } + mGemmId = {maxN, maxK, mType}; + + mWorkspaceMaxSize = mLowLatencyGemmSwigluRunner->getWorkspaceSize(maxM, maxN, maxK); +} + +size_t LowLatencyGemmSwigluPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return mWorkspaceMaxSize; +} + +int LowLatencyGemmSwigluPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + + // input0 activation [M,K] row-major + // input1 weights [K, N] col-major + // output0 [M,N / 2] row-major + + int64_t m64 = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m64 *= inputDesc[0].dims.d[ii]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[1].dims.d[1]); + int const k = TLLM_INT32_CAST(inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]); + int const wsSize = mLowLatencyGemmSwigluRunner->getWorkspaceSize(m, n, k); + auto const& bestTactic = mPluginProfiler->getBestConfig(m, mGemmId); + TLLM_CHECK_WITH_INFO(bestTactic, "No valid Low Latency GEMM SWIGLU tactic"); + + auto env_pdl_overlap_ratio = getFloatEnv("TRTLLM_PDL_OVERLAP_RATIO"); + auto env_prefetch_ratio = getFloatEnv("TRTLLM_PREFETCH_RATIO"); + auto valid_ratio = [](std::optional<float>& env_val, float default_val) + { + if (env_val.has_value()) + { + TLLM_CHECK_WITH_INFO(env_val.value() <= 1.0f, "Valid ratio should be less than or equal to 1.0"); + return env_val.value(); + } + return default_val; + }; + float pdl_overlap_ratio = valid_ratio(env_pdl_overlap_ratio, /*default_val=*/0.5); + float prefetch_ratio = valid_ratio(env_prefetch_ratio, /*default_val=*/-1.0); + mLowLatencyGemmSwigluRunner->gemm(const_cast<FP8Type*>(reinterpret_cast<FP8Type const*>(inputs[0])), + const_cast<FP8Type*>(reinterpret_cast<FP8Type const*>(inputs[1])), mScaleOutput, 0.0F, mScaleD0, mScaleD1, + nullptr, outputs[0], m, n, k, pdl_overlap_ratio, prefetch_ratio, *bestTactic, + reinterpret_cast<char*>(workspace), wsSize, stream); + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType LowLatencyGemmSwigluPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* LowLatencyGemmSwigluPlugin::getPluginType() const noexcept +{ + return LOW_LATENCY_GEMM_SWIGLU_PLUGIN_NAME; +} + +char const* LowLatencyGemmSwigluPlugin::getPluginVersion() const noexcept +{ + return LOW_LATENCY_GEMM_SWIGLU_PLUGIN_VERSION; +} + +int LowLatencyGemmSwigluPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int LowLatencyGemmSwigluPlugin::initialize() noexcept +{ + configGemm(); + return 0; +} + +void LowLatencyGemmSwigluPlugin::terminate() noexcept {} + +size_t LowLatencyGemmSwigluPlugin::getSerializationSize() const noexcept +{ + return sizeof(nvinfer1::DataType) + // dtype + sizeof(float) * 3 + // scales + sizeof(mDims) + mPluginProfiler->getSerializationSize(mGemmId); +} + +void LowLatencyGemmSwigluPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mScaleOutput); + write(d, mScaleD0); + write(d, mScaleD1); + write(d, mDims); + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void LowLatencyGemmSwigluPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +void LowLatencyGemmSwigluPlugin::configGemm() +{ + mPluginProfiler->profileTactics(mLowLatencyGemmSwigluRunner, mType, mDims, mGemmId); +} + +////////////////////////////////////////////////////////////////////////// + +LowLatencyGemmSwigluPluginCreator::LowLatencyGemmSwigluPluginCreator() +{ + + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("scale_output", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("scale_d0", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("scale_d1", nullptr, PluginFieldType::kFLOAT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* LowLatencyGemmSwigluPluginCreator::getPluginName() const noexcept +{ + return LOW_LATENCY_GEMM_SWIGLU_PLUGIN_NAME; +} + +char const* LowLatencyGemmSwigluPluginCreator::getPluginVersion() const noexcept +{ + return LOW_LATENCY_GEMM_SWIGLU_PLUGIN_VERSION; +} + +PluginFieldCollection const* LowLatencyGemmSwigluPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* LowLatencyGemmSwigluPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + TLLM_CHECK(fc->nbFields == 4); + nvinfer1::DataType type{}; + float scale_output{}; + float scale_d0{}; + float scale_d1{}; + for (int i = 0; i < fc->nbFields; i++) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "scale_output")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + scale_output = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + else if (!strcmp(attrName, "scale_d0")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + scale_d0 = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + else if (!strcmp(attrName, "scale_d1")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + scale_d1 = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + } + + try + { + + // + // LowLatencyGemmSwigluPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/*inference=*/false); + auto* obj = new LowLatencyGemmSwigluPlugin(type, scale_output, scale_d0, scale_d1, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* LowLatencyGemmSwigluPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + try + { + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/*inference=*/true); + auto* obj = new LowLatencyGemmSwigluPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/lowLatencyGemmSwigluPlugin.h b/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/lowLatencyGemmSwigluPlugin.h new file mode 100644 index 000000000000..3f73324e7740 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lowLatencyGemmSwigluPlugin/lowLatencyGemmSwigluPlugin.h @@ -0,0 +1,140 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "low_latency_gemm_swiglu.h" + +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <cstddef> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ +using LowLatencyGemmSwigluRunnerPtr + = std::shared_ptr<tensorrt_llm::kernels::internal_cutlass_kernels::CutlassLowLatencyFp8GemmSwigluRunnerInterface>; + +class LowLatencyGemmSwigluPluginProfiler + : public GemmPluginProfiler< + tensorrt_llm::kernels::internal_cutlass_kernels::CutlassLowLatencyFp8GemmSwigluRunnerInterface::ConfigType, + LowLatencyGemmSwigluRunnerPtr, GemmIdCore, GemmIdCoreHash> +{ + +public: + using Config + = tensorrt_llm::kernels::internal_cutlass_kernels::CutlassLowLatencyFp8GemmSwigluRunnerInterface::ConfigType; + + virtual int getMaxProfileM() const override; + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + std::vector<Config> getTactics(int m, int n, int k) const override; +}; + +class LowLatencyGemmSwigluPlugin : public BasePlugin +{ + +public: + using PluginProfilerPtr = std::shared_ptr<LowLatencyGemmSwigluPluginProfiler>; + + LowLatencyGemmSwigluPlugin() = delete; + + LowLatencyGemmSwigluPlugin(nvinfer1::DataType type, float scale_output, float scale_d0, float scale_d1, + PluginProfilerPtr const& pluginProfiler); + + LowLatencyGemmSwigluPlugin(void const* data, size_t length, PluginProfilerPtr const& pluginProfiler); + ~LowLatencyGemmSwigluPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + void init(nvinfer1::DataType type); + void configGemm(); + +private: + std::string const mLayerName; + + LowLatencyGemmSwigluRunnerPtr mLowLatencyGemmSwigluRunner; + size_t mWorkspaceMaxSize; + + GemmDims mDims{}; + GemmIdCore mGemmId{}; + + PluginProfilerPtr mPluginProfiler; + + nvinfer1::DataType mType; + float mScaleOutput; + float mScaleD0; + float mScaleD1; +}; + +class LowLatencyGemmSwigluPluginCreator : public BaseCreator +{ +public: + LowLatencyGemmSwigluPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<LowLatencyGemmSwigluPluginProfiler> gemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/lruPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/lruPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lruPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/lruPlugin/lruPlugin.cpp b/cpp/tensorrt_llm/plugins/lruPlugin/lruPlugin.cpp new file mode 100644 index 000000000000..9d86b8cb8acd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lruPlugin/lruPlugin.cpp @@ -0,0 +1,431 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "lruPlugin.h" +#include "tensorrt_llm/common/assert.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::lruPluginCreator; +using tensorrt_llm::plugins::lruPlugin; + +static char const* LRU_PLUGIN_VERSION{"1"}; +static char const* LRU_PLUGIN_NAME{"LRU"}; +PluginFieldCollection lruPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> lruPluginCreator::mPluginAttributes; + +lruPlugin::lruPlugin(int dim, int block_size, nvinfer1::DataType type, bool removePadding, bool pagedState, + bool yEnabled, bool yBiasEnabled, bool fuseGateEnabled, bool gateBiasEnabled) + : mDim(dim) + , mBlockSize(block_size) + , mType(type) + , mRemovePadding(removePadding) + , mPagedState(pagedState) + , mYEnabled(yEnabled) + , mYBiasEnabled(yBiasEnabled) + , mFuseGateEnabled(fuseGateEnabled) + , mGateBiasEnabled(gateBiasEnabled) +{ + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF), + "Only support float, half, and bfloat16."); +} + +// Parameterized constructor +lruPlugin::lruPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mDim); + read(d, mBlockSize); + read(d, mType); + read(d, mRemovePadding); + read(d, mPagedState); + read(d, mYEnabled); + read(d, mYBiasEnabled); + read(d, mFuseGateEnabled); + read(d, mGateBiasEnabled); + TLLM_CHECK(d == a + length); + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF), + "Only support float, half, and bfloat16."); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* lruPlugin::clone() const noexcept +{ + auto* plugin = new lruPlugin(mDim, mBlockSize, mType, mRemovePadding, mPagedState, mYEnabled, mYBiasEnabled, + mFuseGateEnabled, mGateBiasEnabled); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +// Outputs +// output_tensor: [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// state: [batch_size, dim] +nvinfer1::DimsExprs lruPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + if (outputIndex == 0) + { + return inputs[getXIdx()]; + } + return inputs[getStateIdx()]; +} + +bool lruPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + if (pos == getHostRequestTypesIdx() || pos == getLastTokenIdsIdx() || (mPagedState && pos == getSlotMappingIdx())) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (mPagedState && pos == getStateIdx()) + { + return inOut[pos].type == nvinfer1::DataType::kINT64; + } + else if (pos == getStateIdx() || pos == (nbInputs + 1)) + { + // Use float for both input and output state + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else + { + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } +} + +void lruPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t lruPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +void lruPlugin::setLruParams(lruParams& params, const size_t batch, const size_t dim, const size_t block_size, + const size_t maxSeqLen, void* statePtr, void const* x, void const* gate, void const* gate_bias, void const* gate_x, + void const* gate_x_bias, void const* gate_a, void const* gate_a_bias, void const* y, void const* y_bias, + void const* A, int const* lastTokenIds, int const* slotMapping, void* out, bool removePadding) +{ + // Reset the parameters + memset(¶ms, 0, sizeof(params)); + + params.batch = batch; + params.width = dim; + params.block_size = block_size; + params.max_seqlen = maxSeqLen; + params.remove_padding = removePadding; + + // Set the pointers and strides. + params.A_ptr = const_cast<void*>(A); + params.x_ptr = const_cast<void*>(x); + params.y_ptr = const_cast<void*>(y); + params.y_bias_ptr = const_cast<void*>(y_bias); + params.gate_ptr = const_cast<void*>(gate); + params.gate_bias_ptr = const_cast<void*>(gate_bias); + params.gate_x_ptr = const_cast<void*>(gate_x); + params.gate_x_bias_ptr = const_cast<void*>(gate_x_bias); + params.gate_a_ptr = const_cast<void*>(gate_a); + params.gate_a_bias_ptr = const_cast<void*>(gate_a_bias); + params.state_ptr = statePtr; + params.out_ptr = out; + params.last_token_ids_ptr = lastTokenIds; + params.slot_mapping_ptr = slotMapping; +} + +template <typename T> +int lruPlugin::enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) +{ + // inputs + // 0. x [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding + // 1. A [dim] + // 2. state [batch_size, dim] or host [1] containing only pointer for paged_state + // 3. host_request_types [batch_size] int32. 0: context; 1: generation; 2: none. + // 4. last_token_ids [batch_size] int32 + // 5. state_slot_mapping [batch_size] int32, optional for paged state + // 6. y [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding + // 7. y_bias [dim] + // 8. gate [batch_size, seq_len, 2 * dim] or [num_tokens, 2 * dim] for remove_input_padding + // 9. gate_bias [2 * dim] + // 10. gate_x [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding + // 11. gate_a [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding + // 12. gate_x_bias [2 * dim] + // 13. gate_a_bias [2 * dim] + // outputs + // 0. output_tensor [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding + // 1. state [batch_size, dim] + auto const batch_size = inputDesc[getHostRequestTypesIdx()].dims.d[0]; + int max_seq_len; + if (mRemovePadding) + { + max_seq_len = -1; + } + else + { + max_seq_len = inputDesc[getXIdx()].dims.d[1]; + } + + // only support context or generation, not for both of them + RequestType const* reqTypes = static_cast<RequestType const*>(inputs[getHostRequestTypesIdx()]); + + lruParams lru_params; + + int const* slotMapping = mPagedState ? static_cast<int const*>(inputs[getSlotMappingIdx()]) : nullptr; + void const* y = mYEnabled ? inputs[getYIdx()] : nullptr; + void const* y_bias = mYBiasEnabled ? inputs[getYBiasIdx()] : nullptr; + void const* gate = mFuseGateEnabled ? inputs[getGateIdx()] : nullptr; + void const* gate_bias = (mFuseGateEnabled && mGateBiasEnabled) ? inputs[getGateBiasIdx()] : nullptr; + void const* gate_x = mFuseGateEnabled ? nullptr : inputs[getGateXIdx()]; + void const* gate_a = mFuseGateEnabled ? nullptr : inputs[getGateAIdx()]; + void const* gate_x_bias = (!mFuseGateEnabled && mGateBiasEnabled) ? inputs[getGateXBiasIdx()] : nullptr; + void const* gate_a_bias = (!mFuseGateEnabled && mGateBiasEnabled) ? inputs[getGateABiasIdx()] : nullptr; + + void* statePtr = mPagedState ? *reinterpret_cast<void**>(const_cast<void*>(inputs[getStateIdx()])) : outputs[1]; + + setLruParams(lru_params, batch_size, mDim, mBlockSize, max_seq_len, statePtr, inputs[getXIdx()], gate, gate_bias, + gate_x, gate_x_bias, gate_a, gate_a_bias, y, y_bias, inputs[getAIdx()], + static_cast<int const*>(inputs[getLastTokenIdsIdx()]), slotMapping, outputs[0], mRemovePadding); + + if (reqTypes[0] == RequestType::kCONTEXT) + { + invokeRGLRU<T>(lru_params, stream); + } + else if (reqTypes[0] == RequestType::kGENERATION) + { + invokeRGLRUUpdate<T>(lru_params, stream); + } + sync_check_cuda_error(stream); + return 0; +} + +int lruPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + if (mType == DataType::kHALF) + { + return enqueueImpl<half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kFLOAT) + { + return enqueueImpl<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16) + { + return enqueueImpl<__nv_bfloat16>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType lruPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + if (index == 0) + { + return inputTypes[getXIdx()]; + } + else + { + return inputTypes[getStateIdx()]; + } +} + +// IPluginV2 Methods + +char const* lruPlugin::getPluginType() const noexcept +{ + return LRU_PLUGIN_NAME; +} + +char const* lruPlugin::getPluginVersion() const noexcept +{ + return LRU_PLUGIN_VERSION; +} + +int lruPlugin::getNbOutputs() const noexcept +{ + return mPagedState ? 1 : 2; +} + +int lruPlugin::initialize() noexcept +{ + return 0; +} + +void lruPlugin::terminate() noexcept {} + +size_t lruPlugin::getSerializationSize() const noexcept +{ + return sizeof(mDim) + sizeof(mBlockSize) + sizeof(mType) + sizeof(mRemovePadding) + sizeof(mPagedState) + + sizeof(mYEnabled) + sizeof(mYBiasEnabled) + sizeof(mFuseGateEnabled) + sizeof(mGateBiasEnabled); +} + +void lruPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mDim); + write(d, mBlockSize); + write(d, mType); + write(d, mRemovePadding); + write(d, mPagedState); + write(d, mYEnabled); + write(d, mYBiasEnabled); + write(d, mFuseGateEnabled); + write(d, mGateBiasEnabled); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void lruPlugin::destroy() noexcept +{ + delete this; +} + +/////////////// + +lruPluginCreator::lruPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("dim", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("block_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("remove_input_padding", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("paged_state", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("y_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("y_bias_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("fuse_gate_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("gate_bias_enabled", nullptr, PluginFieldType::kINT8)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* lruPluginCreator::getPluginName() const noexcept +{ + return LRU_PLUGIN_NAME; +} + +char const* lruPluginCreator::getPluginVersion() const noexcept +{ + return LRU_PLUGIN_VERSION; +} + +PluginFieldCollection const* lruPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* lruPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int dim{}; + int block_size{}; + bool removePadding{}; + bool pagedState{}; + bool yEnabled{}; + bool yBiasEnabled{}; + bool fuseGateEnabled{}; + bool gateBiasEnabled{}; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "dim")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dim = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + if (!strcmp(attrName, "block_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + block_size = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "remove_input_padding")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + removePadding = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "paged_state")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + pagedState = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "y_enabled")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + yEnabled = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "y_bias_enabled")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + yBiasEnabled = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "fuse_gate_enabled")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + fuseGateEnabled = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "gate_bias_enabled")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + gateBiasEnabled = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + } + try + { + auto* obj = new lruPlugin( + dim, block_size, type, removePadding, pagedState, yEnabled, yBiasEnabled, fuseGateEnabled, gateBiasEnabled); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* lruPluginCreator::deserializePlugin(char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call lruPlugin::destroy() + try + { + auto* obj = new lruPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/lruPlugin/lruPlugin.h b/cpp/tensorrt_llm/plugins/lruPlugin/lruPlugin.h new file mode 100644 index 000000000000..ee4e0b989b34 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/lruPlugin/lruPlugin.h @@ -0,0 +1,239 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TRT_LRU_PLUGIN_H +#define TRT_LRU_PLUGIN_H +#include "tensorrt_llm/kernels/lruKernel.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> + +namespace tensorrt_llm::plugins +{ +// batch_size = num_ctx_requests or num_gen_requests +// num_ctx_requests = number of context requests (single sequence per request). +// num_gen_requests = number of generation requests (single sequences per request). +// can not support beam search + +// inputs +// 0. x [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 1. A [dim] +// 2. state [batch_size, dim] or host [1] containing only pointer for paged_state +// 3. host_request_types [batch_size] int32. 0: context; 1: generation; 2: none. +// 4. last_token_ids [batch_size] int32 +// 5. state_slot_mapping [batch_size] int32, optional for paged state +// 6. y [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 7. y_bias [dim] +// 8. gate [batch_size, seq_len, 2 * dim] or [num_tokens, 2 * dim] for remove_input_padding +// 9. gate_bias [2 * dim] +// 10. gate_x [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 11. gate_a [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 12. gate_x_bias [2 * dim] +// 13. gate_a_bias [2 * dim] +// outputs +// 0. output_tensor [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 1. state [batch_size, dim] + +class lruPlugin : public BasePlugin +{ +public: + lruPlugin(int dim, int block_size, nvinfer1::DataType type, bool removePadding, bool pagedState, bool yEnabled, + bool yBiasEnabled, bool fuseGateEnabled, bool gateBiasEnabled); + + lruPlugin(void const* data, size_t length); + + ~lruPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + template <typename T> + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + + enum class RequestType : int32_t + { + kCONTEXT = 0, + kGENERATION = 1 + }; + +private: + using IndexType = std::int32_t; + + IndexType getXIdx() const + { + return 0; + }; + + IndexType getAIdx() const + { + return 1; + }; + + IndexType getStateIdx() const + { + return 2; + }; + + IndexType getHostRequestTypesIdx() const + { + return 3; + }; + + IndexType getLastTokenIdsIdx() const + { + return 4; + }; + + IndexType getSlotMappingIdx() const + { + if (mPagedState) + return 5; + else + return 4; + }; + + IndexType getYIdx() const + { + if (mYEnabled) + return getSlotMappingIdx() + 1; + else + return getSlotMappingIdx(); + }; + + IndexType getYBiasIdx() const + { + if (mYBiasEnabled) + return getYIdx() + 1; + else + return getYIdx(); + }; + + IndexType getGateIdx() const + { + if (mFuseGateEnabled) + return getYBiasIdx() + 1; + else + return getYBiasIdx(); + }; + + IndexType getGateBiasIdx() const + { + if (mFuseGateEnabled && mGateBiasEnabled) + return getGateIdx() + 1; + else + return getGateIdx(); + }; + + IndexType getGateXIdx() const + { + if (mFuseGateEnabled) + return getGateBiasIdx(); + else + return getGateBiasIdx() + 1; + }; + + IndexType getGateAIdx() const + { + if (mFuseGateEnabled) + return getGateXIdx(); + else + return getGateXIdx() + 1; + }; + + IndexType getGateXBiasIdx() const + { + if (!mFuseGateEnabled && mGateBiasEnabled) + return getGateAIdx() + 1; + else + return getGateAIdx(); + }; + + IndexType getGateABiasIdx() const + { + if (!mFuseGateEnabled && mGateBiasEnabled) + return getGateXBiasIdx() + 1; + else + return getGateXBiasIdx(); + }; + + static void setLruParams(tensorrt_llm::kernels::lruParams& params, + // sizes + const size_t batch, const size_t dim, const size_t block_size, const size_t maxSeqLen, + // device pointers + void* statePtr, void const* x, void const* gate, void const* gate_bias, void const* gate_x, + void const* gate_x_bias, void const* gate_a, void const* gate_a_bias, void const* y, void const* y_bias, + void const* A, int const* lastTokenIds, int const* slotMapping, void* out, bool removePadding); + +private: + int mDim; + int mBlockSize; + nvinfer1::DataType mType; + bool mRemovePadding = false; + bool mPagedState = false; + bool mYEnabled = false; + bool mYBiasEnabled = false; + bool mFuseGateEnabled = false; + bool mGateBiasEnabled = false; +}; + +class lruPluginCreator : public BaseCreator +{ +public: + lruPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins + +#endif // TRT_LRU_PLUGIN_H diff --git a/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/mambaConv1dPlugin.cpp b/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/mambaConv1dPlugin.cpp new file mode 100644 index 000000000000..16754248b84d --- /dev/null +++ b/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/mambaConv1dPlugin.cpp @@ -0,0 +1,404 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "mambaConv1dPlugin.h" +#include "tensorrt_llm/common/assert.h" +#include <algorithm> + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::MambaConv1dPluginCreator; +using tensorrt_llm::plugins::MambaConv1dPlugin; + +static char const* MAMBA_CONV1D_PLUGIN_VERSION{"1"}; +static char const* MAMBA_CONV1D_PLUGIN_NAME{"MambaConv1d"}; + +PluginFieldCollection MambaConv1dPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> MambaConv1dPluginCreator::mPluginAttributes; + +MambaConv1dPlugin::MambaConv1dPlugin(int dim, int dconv, int preStride, int postStride, nvinfer1::DataType type, + bool removePadding, bool pagedState, bool applySilu) + : mDim(dim) + , mDConv(dconv) + , mPreStride(preStride) + , mPostStride(postStride) + , mType(type) + , mRemovePadding(removePadding) + , mPagedState(pagedState) + , mApplySilu(applySilu) +{ + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF), + "Only support float, half, and bfloat16."); +} + +// Parameterized constructor +MambaConv1dPlugin::MambaConv1dPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mDim); + read(d, mDConv); + read(d, mPreStride); + read(d, mPostStride); + read(d, mType); + read(d, mRemovePadding); + read(d, mPagedState); + read(d, mApplySilu); + TLLM_CHECK(d == a + length); + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF), + "Only support float, half, and bfloat16."); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* MambaConv1dPlugin::clone() const noexcept +{ + auto* plugin + = new MambaConv1dPlugin(mDim, mDConv, mPreStride, mPostStride, mType, mRemovePadding, mPagedState, mApplySilu); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +// Outputs +// output_tensor: [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// state: [batch_size, dconv - 1, dim] +nvinfer1::DimsExprs MambaConv1dPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + if (outputIndex == 0) + { + auto ret = inputs[getInputTensorIdx()]; + ret.d[mRemovePadding ? 1 : 2] = exprBuilder.constant(mDim); + return ret; + } + return inputs[getConvStateIdx()]; +} + +bool MambaConv1dPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + if (pos == getHostRequestTypesIdx() || pos == getLastTokenIdsIdx() + || (mRemovePadding && pos == getHostContextLengthIdx()) || (mPagedState && pos == getSlotMappingIdx())) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (mPagedState && pos == getConvStateIdx()) + { + return inOut[pos].type == nvinfer1::DataType::kINT64; + } + else + { + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } +} + +void MambaConv1dPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t MambaConv1dPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +void MambaConv1dPlugin::setMambaConv1dParams(tensorrt_llm::kernels::MambaConv1dParamsBase& params, const size_t batch, + const size_t dim, const size_t maxSeqLen, const size_t dconv, const size_t preStride, const size_t postStride, + void const* inPtr, void const* stateInPtr, void* stateOutPtr, void const* convWeight, void const* convBias, + void* outPtr, int const* lastTokenIds, int const* stateSlotMapping, bool removePadding, bool applySilu) +{ + // Reset the parameters + memset(¶ms, 0, sizeof(params)); + + params.batch = batch; + params.dim = dim; + params.max_seqlen = maxSeqLen; + params.dconv = dconv; + params.pre_stride = preStride; + params.post_stride = postStride; + + params.remove_padding = removePadding; + params.apply_silu = applySilu; + + // Set the pointers and strides. + params.in_ptr = const_cast<void*>(inPtr); + params.state_in_ptr = const_cast<void*>(stateInPtr); + params.state_out_ptr = stateOutPtr; + params.weight_ptr = const_cast<void*>(convWeight); + params.bias_ptr = const_cast<void*>(convBias); + params.out_ptr = outPtr; + params.last_token_ids_ptr = lastTokenIds; + params.state_slot_mapping_ptr = stateSlotMapping; +} + +template <typename T> +int MambaConv1dPlugin::enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) +{ + // inputs + // 0. input_tensor [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding + // 1. conv_state [batch_size, dconv - 1, dim] or host [1] containing only pointer for paged_state + // 2. weight [dim, 1, dconv] + // 3. bias [dim] + // 4. host_request_types [batch_size] int32. 0: context; 1: generation; 2: none. + // 5. last_token_ids [batch_size] int32 + // 6. host_context_lengths [batch_size] int32, optional for remove_input_padding + // 7. state_slot_mapping [batch_size] int32, optional + // outputs + // 0. output_tensor [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding + // 1. conv_state [batch_size, dconv - 1, dim] + auto const batchSize = inputDesc[getHostRequestTypesIdx()].dims.d[0]; + int maxSeqLen; + if (mRemovePadding) + { + int const* host_context_length = static_cast<int const*>(inputs[getHostContextLengthIdx()]); + maxSeqLen = *std::max_element(host_context_length, host_context_length + batchSize); + } + else + { + maxSeqLen = inputDesc[getInputTensorIdx()].dims.d[1]; + } + + // only support context or generation, not for both of them + RequestType const* reqTypes = static_cast<RequestType const*>(inputs[getHostRequestTypesIdx()]); + + MambaConv1dParamsBase mambaConv1dParams; + + int const* slotMapping = mPagedState ? static_cast<int const*>(inputs[getSlotMappingIdx()]) : nullptr; + void* stateInPtr = mPagedState ? *reinterpret_cast<void**>(const_cast<void*>(inputs[getConvStateIdx()])) + : const_cast<void*>(inputs[getConvStateIdx()]); + void* stateOutPtr + = mPagedState ? *reinterpret_cast<void**>(const_cast<void*>(inputs[getConvStateIdx()])) : outputs[1]; + + setMambaConv1dParams(mambaConv1dParams, batchSize, mDim, maxSeqLen, mDConv, mPreStride, mPostStride, + inputs[getInputTensorIdx()], stateInPtr, stateOutPtr, inputs[getWeightIdx()], inputs[getBiasIdx()], outputs[0], + static_cast<int const*>(inputs[getLastTokenIdsIdx()]), slotMapping, mRemovePadding, mApplySilu); + + if (reqTypes[0] == RequestType::kCONTEXT) + { + invokeMambaConv1dContext<T>(mambaConv1dParams, stream); + } + else if (reqTypes[0] == RequestType::kGENERATION) + { + invokeMambaConv1dGeneration<T>(mambaConv1dParams, stream); + } + sync_check_cuda_error(stream); + return 0; +} + +int MambaConv1dPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + if (mType == DataType::kHALF) + { + return enqueueImpl<half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kFLOAT) + { + return enqueueImpl<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16) + { + return enqueueImpl<__nv_bfloat16>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType MambaConv1dPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + return inputTypes[getInputTensorIdx()]; +} + +// IPluginV2 Methods + +char const* MambaConv1dPlugin::getPluginType() const noexcept +{ + return MAMBA_CONV1D_PLUGIN_NAME; +} + +char const* MambaConv1dPlugin::getPluginVersion() const noexcept +{ + return MAMBA_CONV1D_PLUGIN_VERSION; +} + +int MambaConv1dPlugin::getNbOutputs() const noexcept +{ + return 2; +} + +int MambaConv1dPlugin::initialize() noexcept +{ + return 0; +} + +void MambaConv1dPlugin::terminate() noexcept {} + +size_t MambaConv1dPlugin::getSerializationSize() const noexcept +{ + return sizeof(mDim) + sizeof(mDConv) + sizeof(mPreStride) + sizeof(mPostStride) + sizeof(mType) + + sizeof(mRemovePadding) + sizeof(mPagedState) + sizeof(mApplySilu); +} + +void MambaConv1dPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mDim); + write(d, mDConv); + write(d, mPreStride); + write(d, mPostStride); + write(d, mType); + write(d, mRemovePadding); + write(d, mPagedState); + write(d, mApplySilu); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void MambaConv1dPlugin::destroy() noexcept +{ + delete this; +} + +/////////////// + +MambaConv1dPluginCreator::MambaConv1dPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("dim", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("dconv", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("pre_stride", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("post_stride", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("remove_input_padding", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("paged_state", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("apply_silu", nullptr, PluginFieldType::kINT8)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* MambaConv1dPluginCreator::getPluginName() const noexcept +{ + return MAMBA_CONV1D_PLUGIN_NAME; +} + +char const* MambaConv1dPluginCreator::getPluginVersion() const noexcept +{ + return MAMBA_CONV1D_PLUGIN_VERSION; +} + +PluginFieldCollection const* MambaConv1dPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* MambaConv1dPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int dim{}; + int dconv{}; + int pre_stride{}; + int post_stride{}; + bool removePadding{}; + bool pagedState{}; + bool applySilu{}; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "dim")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dim = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "dconv")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dconv = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "pre_stride")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + pre_stride = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "post_stride")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + post_stride = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "remove_input_padding")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + removePadding = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "paged_state")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + pagedState = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "apply_silu")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + applySilu = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + } + try + { + auto* obj + = new MambaConv1dPlugin(dim, dconv, pre_stride, post_stride, type, removePadding, pagedState, applySilu); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* MambaConv1dPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call MambaConv1dPlugin::destroy() + try + { + auto* obj = new MambaConv1dPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/mambaConv1dPlugin.h b/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/mambaConv1dPlugin.h new file mode 100644 index 000000000000..d351b1cdc237 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/mambaConv1dPlugin/mambaConv1dPlugin.h @@ -0,0 +1,176 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TRT_MAMBA_CONV1D_PLUGIN_H +#define TRT_MAMBA_CONV1D_PLUGIN_H +#include "tensorrt_llm/kernels/mambaConv1dKernels.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> + +namespace tensorrt_llm::plugins +{ +// batch_size = num_ctx_requests or num_gen_requests +// num_ctx_requests = number of context requests (single sequence per request). +// num_gen_requests = number of generation requests (single sequences per request). +// can not support beam search + +// inputs +// 0. input_tensor [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 1. conv_state [batch_size, dconv - 1, dim] or host [1] containing only pointer for paged_state +// 2. weight [1, dconv, dim] +// 3. bias [dim] +// 4. host_request_types [batch_size] int32. 0: context; 1: generation; 2: none. +// 5. last_token_ids [batch_size] int32 +// 6. host_context_lengths [batch_size] int32, optional for remove_input_padding +// 7. state_slot_mapping [batch_size] int32, optional +// outputs +// 0. output_tensor [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 1. conv_state [batch_size, dconv - 1, dim] + +class MambaConv1dPlugin : public BasePlugin +{ +public: + MambaConv1dPlugin(int dim, int dconv, int preStride, int postStride, nvinfer1::DataType type, bool removePadding, + bool pagedState, bool applySilu); + + MambaConv1dPlugin(void const* data, size_t length); + + ~MambaConv1dPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + template <typename T> + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + + enum class RequestType : int32_t + { + kCONTEXT = 0, + kGENERATION = 1 + }; + +private: + using IndexType = std::int32_t; + + IndexType getInputTensorIdx() const + { + return 0; + }; + + IndexType getConvStateIdx() const + { + return 1; + }; + + IndexType getWeightIdx() const + { + return 2; + }; + + IndexType getBiasIdx() const + { + return 3; + }; + + IndexType getHostRequestTypesIdx() const + { + return 4; + }; + + IndexType getLastTokenIdsIdx() const + { + return 5; + }; + + IndexType getHostContextLengthIdx() const + { + return 6; + }; + + IndexType getSlotMappingIdx() const + { + // if not remove input padding, host_context_length is not used, so the index is 6 + return mRemovePadding ? 7 : 6; + }; + + void setMambaConv1dParams(tensorrt_llm::kernels::MambaConv1dParamsBase& params, + // sizes + const size_t batch, const size_t dim, const size_t maxSeqLen, const size_t dconv, const size_t preStride, + const size_t postStride, + // device pointers + void const* inPtr, void const* stateInPtr, void* stateOutPtr, void const* convWeight, void const* convBias, + void* outPtr, int const* lastTokenIds, int const* stateSlotMapping, bool removePadding, bool applySilu); + +private: + int mDim; + int mDConv; + int mPreStride; + int mPostStride; + nvinfer1::DataType mType; + bool mRemovePadding = false; + bool mPagedState = false; + bool mApplySilu = true; +}; + +class MambaConv1dPluginCreator : public BaseCreator +{ +public: + MambaConv1dPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins + +#endif // TRT_MAMBA_CONV1D_PLUGIN_H diff --git a/cpp/tensorrt_llm/plugins/mixtureOfExperts/CMakeLists.txt b/cpp/tensorrt_llm/plugins/mixtureOfExperts/CMakeLists.txt new file mode 100644 index 000000000000..7cc985b60b7a --- /dev/null +++ b/cpp/tensorrt_llm/plugins/mixtureOfExperts/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2023 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.cpp b/cpp/tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.cpp new file mode 100644 index 000000000000..ccce34850730 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.cpp @@ -0,0 +1,1314 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.h" +#include "tensorrt_llm/common/cudaBf16Wrapper.h" +#include "tensorrt_llm/common/dataType.h" +#include "tensorrt_llm/common/envUtils.h" +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/utils/debugUtils.h" +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::plugins; +using tensorrt_llm::common::QuantMode; +using tensorrt_llm::common::nextWorkspacePtr; +using tensorrt_llm::common::calculateTotalWorkspaceSize; +using tensorrt_llm::plugins::MixtureOfExpertsPluginCreator; +using tensorrt_llm::plugins::MixtureOfExpertsPlugin; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +using LoraImpl = tensorrt_llm::kernels::LoraImpl; +using LoraParams = tensorrt_llm::kernels::LoraParams; + +static char const* MIXTURE_OF_EXPERTS_PLUGIN_VERSION{"1"}; +static char const* MIXTURE_OF_EXPERTS_PLUGIN_NAME{"MixtureOfExperts"}; +nvinfer1::PluginFieldCollection MixtureOfExpertsPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> MixtureOfExpertsPluginCreator::mPluginAttributes; + +MixtureOfExpertsPlugin::MixtureOfExpertsPlugin(bool remove_input_padding, int number_of_experts, int experts_per_token, + int expert_hidden_size, int expert_inter_size, int groupwise_quant_algo, int group_size, + ActivationType activation_type, nvinfer1::DataType type, nvinfer1::DataType weight_type, + nvinfer1::DataType output_type, QuantMode quant_mode, bool use_final_scales, bool use_bias, int tp_size, + int tp_rank, int ep_size, int ep_rank, bool force_determinism, int side_stream_id, + MixtureOfExpertsPluginProfilerPtr gemm_profiler_ptr, bool use_lora, nvinfer1::DataType lora_type, + LoraPluginProfilerPtr lora_profiler, int max_low_rank) + : mNumExperts(number_of_experts) + , mExpertsPerToken(experts_per_token) + , mExpertHiddenSize(expert_hidden_size) + , mExpertInterSize(expert_inter_size) + , mGroupwiseQuantAlgo(groupwise_quant_algo) + , mGroupSize(group_size) + , mActivationType(activation_type) + , mType(type) + , mWeightType(weight_type) + , mOutputType(output_type) + , mQuantMode(quant_mode) + , mUseFinalScales(use_final_scales) + , mUseBias(use_bias) + , mParallelismConfig(MOEParallelismConfig{tp_size, tp_rank, ep_size, ep_rank}) + , mUseDeterministicKernels(force_determinism) + , mSideStreamId(side_stream_id) + , mGemmProfiler(std::move(gemm_profiler_ptr)) + , mUseLora(use_lora) + , mLoraType(lora_type) + , mMaxLowRank(max_low_rank) + , mRemoveInputPadding(remove_input_padding) + , mLoraProfiler(std::move(lora_profiler)) +{ + init(); +} + +tensorrt_llm::plugins::MixtureOfExpertsPlugin::MixtureOfExpertsPlugin(MixtureOfExpertsPlugin const& other) + : mMOERunner() + , mNumExperts(other.mNumExperts) + , mExpertsPerToken(other.mExpertsPerToken) + , mExpertHiddenSize(other.mExpertHiddenSize) + , mExpertInterSize(other.mExpertInterSize) + , mGroupwiseQuantAlgo(other.mGroupwiseQuantAlgo) + , mGroupSize(other.mGroupSize) + , mActivationType(other.mActivationType) + , mType(other.mType) + , mWeightType(other.mWeightType) + , mOutputType(other.mOutputType) + , mQuantMode(other.mQuantMode) + , mUseFinalScales(other.mUseFinalScales) + , mUseBias(other.mUseBias) + , mParallelismConfig(other.mParallelismConfig) + , mDims(other.mDims) + , mUseDeterministicKernels(other.mUseDeterministicKernels) + , mSideStreamId(other.mSideStreamId) + , mGemmId1(other.mGemmId1) + , mGemmId2(other.mGemmId2) + , mGemmProfiler(other.mGemmProfiler) + , mUseLora(other.mUseLora) + , mLoraType(other.mLoraType) + , mMaxLowRank(other.mMaxLowRank) + , mRemoveInputPadding(other.mRemoveInputPadding) + , mLoraImpl1(other.mLoraImpl1) + , mLoraImpl2(other.mLoraImpl2) + , mLoraGemmId1(other.mLoraGemmId1) + , mLoraGemmId2(other.mLoraGemmId2) + , mLoraProfiler(other.mLoraProfiler) + , mLayerName(other.mLayerName) + , mNamespace(other.mNamespace) +{ + init(); +} + +size_t MixtureOfExpertsPlugin::getSerializationSize() const noexcept +{ + size_t size = sizeof(mRemoveInputPadding) + sizeof(mNumExperts) + sizeof(mExpertsPerToken) + + sizeof(mExpertHiddenSize) + sizeof(mExpertInterSize) + sizeof(mGroupwiseQuantAlgo) + sizeof(mGroupSize) + + sizeof(mActivationType) + sizeof(mType) + sizeof(mWeightType) + sizeof(mOutputType) + + sizeof(QuantMode::BaseType) + sizeof(mUseFinalScales) + sizeof(mUseBias) + sizeof(mParallelismConfig) + + sizeof(mDims) + sizeof(mUseDeterministicKernels) + sizeof(mSideStreamId) + + mGemmProfiler->getSerializationSize(mGemmId1) + mGemmProfiler->getSerializationSize(mGemmId2) + + sizeof(mUseLora) + sizeof(mLoraType) + sizeof(mMaxLowRank); + + if (hasLora()) + { + size += mLoraProfiler->getSerializationSize(mLoraGemmId1); + size += mLoraProfiler->getSerializationSize(mLoraGemmId2); + } + + return size; +} + +MixtureOfExpertsPlugin::MixtureOfExpertsPlugin(void const* data, size_t length, + MixtureOfExpertsPluginProfilerPtr gemm_profiler_ptr, LoraPluginProfilerPtr lora_profiler) + : mGemmProfiler(gemm_profiler_ptr) + , mLoraProfiler(lora_profiler) +{ + char const* d = reinterpret_cast<char const*>(data); + char const* a = d; + read(d, mRemoveInputPadding); + read(d, mNumExperts); + read(d, mExpertsPerToken); + read(d, mExpertHiddenSize); + read(d, mExpertInterSize); + read(d, mGroupwiseQuantAlgo); + read(d, mGroupSize); + read(d, mActivationType); + read(d, mType); + read(d, mWeightType); + read(d, mOutputType); + QuantMode::BaseType quant_mode; + read(d, quant_mode); + mQuantMode = QuantMode{quant_mode}; + read(d, mUseFinalScales); + read(d, mUseBias); + read(d, mParallelismConfig); + read(d, mDims); + read(d, mUseDeterministicKernels); + read(d, mSideStreamId); + read(d, mUseLora); + read(d, mLoraType); + read(d, mMaxLowRank); + + // Call init before deserialising the profiler to initialize mGemmId + init(); + mGemmProfiler->deserialize(d, mDims, mGemmId1); + mGemmProfiler->deserialize(d, mDims, mGemmId2); + + if (hasLora()) + { + mLoraProfiler->deserialize(d, mDims, mLoraGemmId1); + mLoraProfiler->deserialize(d, mDims, mLoraGemmId2); + } + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +void MixtureOfExpertsPlugin::serialize(void* buffer) const noexcept +{ + char* d = static_cast<char*>(buffer); + char* a = d; + + write(d, mRemoveInputPadding); + write(d, mNumExperts); + write(d, mExpertsPerToken); + write(d, mExpertHiddenSize); + write(d, mExpertInterSize); + write(d, mGroupwiseQuantAlgo); + write(d, mGroupSize); + write(d, mActivationType); + write(d, mType); + write(d, mWeightType); + write(d, mOutputType); + write(d, mQuantMode.value()); + write(d, mUseFinalScales); + write(d, mUseBias); + write(d, mParallelismConfig); + write(d, mDims); + write(d, mUseDeterministicKernels); + write(d, mSideStreamId); + write(d, mUseLora); + write(d, mLoraType); + write(d, mMaxLowRank); + + mGemmProfiler->serialize(d, mGemmId1); + mGemmProfiler->serialize(d, mGemmId2); + + if (hasLora()) + { + mLoraProfiler->serialize(d, mLoraGemmId1); + mLoraProfiler->serialize(d, mLoraGemmId2); + } + + TLLM_CHECK(d == a + getSerializationSize()); +} + +template <typename Type, bool NeedQuant = false> +std::unique_ptr<kernels::CutlassMoeFCRunnerInterface> switch_output_type(nvinfer1::DataType output_type) +{ + switch (output_type) + { + case nvinfer1::DataType::kFP4: + case nvinfer1::DataType::kFP8: + // TODO We need an atomic FP8 reduction for the finalize fusions + TLLM_THROW("Outputting %d directly is not currently supported", static_cast<int>(output_type)); + // return std::make_unique<kernels::CutlassMoeFCRunner<Type, Type>>(); + case nvinfer1::DataType::kHALF: + if constexpr (NeedQuant) + { + return std::make_unique<kernels::CutlassMoeFCRunner<Type, Type, half, half>>(); + } + else + { + return std::make_unique<kernels::CutlassMoeFCRunner<Type, Type, half, Type>>(); + } +#ifdef ENABLE_BF16 + case nvinfer1::DataType::kBF16: + if constexpr (NeedQuant) + { + return std::make_unique<kernels::CutlassMoeFCRunner<Type, Type, __nv_bfloat16, __nv_bfloat16>>(); + } + else + { + return std::make_unique<kernels::CutlassMoeFCRunner<Type, Type, __nv_bfloat16, Type>>(); + } +#endif + default: TLLM_THROW("Invalid output type %d", static_cast<int>(output_type)); + } +}; + +void MixtureOfExpertsPlugin::init() +{ + TLLM_CHECK_WITH_INFO(mType == DataType::kFP8 || mType == DataType::kFP4 || mOutputType == mType, + "MOE plugin only supports a different output type for FP4/FP8"); + TLLM_CHECK_WITH_INFO(mType != DataType::kFP8 || tensorrt_llm::common::getSMVersion() >= 89, + "MoE FP8 is not supported for architectures less than SM89"); + TLLM_CHECK_WITH_INFO(mType != DataType::kFP4 || (tensorrt_llm::common::getSMVersion() >= 100), + "MoE FP4 is only supported on architecture SM100 or later"); + + TLLM_CHECK_WITH_INFO(!hasLora() || mLoraType == mOutputType, "The LoraType need to keep same with moe OutputType."); + + if (mWeightType == nvinfer1::DataType::kINT8 && mQuantMode.hasInt4Weights()) + { + mWeightType = DataType::kINT4; + } + + if (mType == DataType::kHALF && mWeightType == DataType::kHALF) + { + mMOERunner = std::make_unique<kernels::CutlassMoeFCRunner<half, half>>(); + } + else if (mType == DataType::kFLOAT && mWeightType == DataType::kFLOAT) + { + mMOERunner = std::make_unique<kernels::CutlassMoeFCRunner<float, float>>(); + } + else if (mType == DataType::kHALF && mWeightType == DataType::kINT8) + { + mMOERunner = std::make_unique<kernels::CutlassMoeFCRunner<half, uint8_t>>(); + } + else if (mType == DataType::kHALF && mWeightType == DataType::kINT4) + { + mMOERunner = std::make_unique<kernels::CutlassMoeFCRunner<half, cutlass::uint4b_t>>(); + } +#ifdef ENABLE_FP8 + else if (mType == DataType::kFP8 && mWeightType == DataType::kINT4 && mOutputType == DataType::kHALF) + { + mMOERunner = std::make_unique<kernels::CutlassMoeFCRunner<__nv_fp8_e4m3, cutlass::uint4b_t, half, half>>(); + } +#endif +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16 && mWeightType == DataType::kBF16) + { + mMOERunner = std::make_unique<kernels::CutlassMoeFCRunner<__nv_bfloat16, __nv_bfloat16>>(); + } + else if (mType == DataType::kBF16 && mWeightType == DataType::kINT8) + { + mMOERunner = std::make_unique<kernels::CutlassMoeFCRunner<__nv_bfloat16, uint8_t>>(); + } + else if (mType == DataType::kBF16 && mWeightType == DataType::kINT4) + { + mMOERunner = std::make_unique<kernels::CutlassMoeFCRunner<__nv_bfloat16, cutlass::uint4b_t>>(); + } +#ifdef ENABLE_FP8 + else if (mType == DataType::kFP8 && mWeightType == DataType::kINT4 && mOutputType == DataType::kBF16) + { + mMOERunner = std::make_unique< + kernels::CutlassMoeFCRunner<__nv_fp8_e4m3, cutlass::uint4b_t, __nv_bfloat16, __nv_bfloat16>>(); + } +#endif +#endif + +#ifdef ENABLE_FP8 + if (mType == DataType::kFP8 && mWeightType == DataType::kFP8) + { + mMOERunner = switch_output_type<__nv_fp8_e4m3>(mOutputType); + } +#endif +#ifdef ENABLE_FP4 + if (mType == DataType::kFP4 && mWeightType == DataType::kFP4) + { + mMOERunner = switch_output_type<__nv_fp4_e2m1, true>(mOutputType); + } +#endif + + if (!mMOERunner) + { + TLLM_THROW( + "Could not construct the mixture of experts plugin with the requested input combination Activation: %d " + "Weight: %d Output: %d", + static_cast<int>(mType), static_cast<int>(mWeightType), static_cast<int>(mOutputType)); + } + + // Finalize fusion should be disabled if Lora is used. + mMOERunner->use_fused_finalize_ + = (mExpertsPerToken < 3 || !mUseDeterministicKernels) && !getEnvMOEDisableFinalizeFusion() && !hasLora(); + + mGemmId1 = GemmIDMoe{1, mNumExperts, mExpertsPerToken, mParallelismConfig, mExpertHiddenSize, mExpertInterSize, + mGroupSize, mActivationType, mType, mWeightType, mQuantMode, !mMOERunner->use_fused_finalize_}; + mGemmId2 = GemmIDMoe{2, mNumExperts, mExpertsPerToken, mParallelismConfig, mExpertHiddenSize, mExpertInterSize, + mGroupSize, mActivationType, mType, mWeightType, mQuantMode, !mMOERunner->use_fused_finalize_}; + mGemmProfiler->setMaxProfileM(16384 * mNumExperts / mExpertsPerToken); + + if (hasLora()) + { + auto cublasHandle = getCublasHandle(); + auto cublasLtHandle = getCublasLtHandle(); + auto cublasWrapper = std::make_shared<CublasMMWrapper>(cublasHandle, cublasLtHandle, nullptr, nullptr); + mLoraGemmId1 = GemmIdCublas(mExpertInterSize, mExpertHiddenSize, mLoraType, false, true, mLoraType); + mLoraGemmId2 = GemmIdCublas(mExpertHiddenSize, mExpertInterSize, mLoraType, false, true, mLoraType); + std::vector<int> loraOutSizes1 = {static_cast<int>(mExpertInterSize)}; + mLoraImpl1 = std::make_shared<LoraImpl>( + mExpertHiddenSize, loraOutSizes1, false, true, 1, mLoraType, mMaxLowRank, cublasWrapper); + std::vector<int> loraOutSizes2 = {static_cast<int>(mExpertHiddenSize)}; + mLoraImpl2 = std::make_shared<LoraImpl>( + mExpertInterSize, loraOutSizes2, false, true, 1, mLoraType, mMaxLowRank, cublasWrapper); + + TLLM_CUDA_CHECK(cudaEventCreate(&mMemcpyEvent)); + } + mSideStreamPtr = nullptr; + mDebugStallMain = tensorrt_llm::runtime::utils::stallStream("TLLM_DEBUG_MOE_STALL_MAIN"); + mDebugStallSide = tensorrt_llm::runtime::utils::stallStream("TLLM_DEBUG_MOE_STALL_SIDE"); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* MixtureOfExpertsPlugin::clone() const noexcept +{ + auto* plugin = new MixtureOfExpertsPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs MixtureOfExpertsPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + assert(outputIndex == getOutputTensorIndex() || outputIndex == getOutputDummyTensorIndex()); + return inputs[getInputTensorIndex()]; +} + +bool MixtureOfExpertsPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + TLLM_CHECK(0 <= pos && pos < getNbInputs() + getNbOutputs()); + TLLM_CHECK_WITH_INFO( + nbInputs == getNbInputs(), "Required input to plugin is missing. Expected %d Got %d", getNbInputs(), nbInputs); + TLLM_CHECK_WITH_INFO(nbOutputs == getNbOutputs(), "Required output to plugin is missing. Expected %d Got %d", + getNbOutputs(), nbOutputs); + + if (inOut[pos].format != TensorFormat::kLINEAR) + { + return false; + } + + if (pos == getExpertWeights1Index() || pos == getExpertWeights2Index()) + { + if (mGroupwiseQuantAlgo == 0) + { + auto normalized_weight_type + = mWeightType == nvinfer1::DataType::kINT4 ? nvinfer1::DataType::kINT8 : mWeightType; + return inOut[pos].type == normalized_weight_type; + } + else + { + return inOut[pos].type == mOutputType; + } + } + else if (pos == getTokenSelectedExpertsIndex()) + { + return inOut[pos].type == DataType::kINT32; + } + else if (pos == getTokenFinalScalesIndex()) + { + return inOut[pos].type == DataType::kFLOAT; + } + else if (pos == getExpertBias1Index() || pos == getExpertBias2Index()) + { + return inOut[pos].type == mOutputType; + } + else if (pos == nbInputs + getOutputTensorIndex()) + { + return inOut[pos].type == mOutputType; + } + else if (useSideStream() && pos == nbInputs + getOutputDummyTensorIndex()) + { + return inOut[pos].type == inOut[getInputDummyTensorIndex()].type; + } + else if (useSideStream() && pos == getInputDummyTensorIndex()) + { + return true; + } + else if (hasExpertFp8QuantScales() && getExpertFP8Dequant1Index() <= pos && pos <= getExpertFP8QuantFinalIndex()) + { + return inOut[pos].type == DataType::kFLOAT; + } + else if (hasExpertIntQuantScales() && getExpertIntQuantScale1Index() <= pos + && pos <= getExpertIntQuantScale2Index()) + { + return inOut[pos].type == mOutputType; + } + else if (hasFP4QuantScales() && getFP4GlobalActSF1Index() <= pos && pos <= getFP4GlobalSF2Index()) + { + if (pos == getFP4WeightSF1Index() || pos == getFP4WeightSF2Index()) + return inOut[pos].type == nvinfer1::DataType::kFP8; + else + return inOut[pos].type == nvinfer1::DataType::kFLOAT; + } + else if (hasLora() && hasExpertFp8QuantScales() && pos == getInputFP8DequantIndex()) + { + return inOut[pos].type == nvinfer1::DataType::kFLOAT; + } + else if (hasExpertWeightQuantZeros() && getExpertIntQuantZeros1Index() <= pos + && pos <= getExpertIntQuantZeros2Index()) + { + return inOut[pos].type == mOutputType; + } + else if (hasExpertPrequantScales() && getExpertPrequantScales1Index() <= pos + && pos <= getExpertPrequantScales2Index()) + { + return inOut[pos].type == mOutputType; + } + else if (hasGroupwiseFp8Alpha() && getExpertFp8Alpha1Index() <= pos && pos <= getExpertFp8Alpha2Index()) + { + return inOut[pos].type == DataType::kFLOAT; + } + else if (hasLora() && pos == getHostRequestTypeIndex()) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (hasLora() && (pos == getLoraFC1RanksIndex() || pos == getLoraFC2RanksIndex())) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (hasGatedLoraWeightsAndRanks() && pos == getLoraGatedRanksIndex()) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (hasLora() && (pos == getLoraFC1WeightPtrsIndex() || pos == getLoraFC2WeightPtrsIndex())) + { + return inOut[pos].type == nvinfer1::DataType::kINT64; + } + else if (hasGatedLoraWeightsAndRanks() && pos == getLoraGatedWeightPtrsIndex()) + { + return inOut[pos].type == nvinfer1::DataType::kINT64; + } + else if (hasLora() && mRemoveInputPadding && pos == getHostContextLengthIndex()) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if ((hasFP4QuantScales() || hasGroupwiseFp8Alpha()) && pos == getInputTensorIndex()) + { + return inOut[pos].type == mOutputType; + } + else + { + return inOut[pos].type == mType; + } + + return false; +} + +void MixtureOfExpertsPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + TLLM_CHECK_WITH_INFO( + nbInputs == getNbInputs(), "Required input to plugin is missing. Expected %d Got %d", getNbInputs(), nbInputs); + TLLM_CHECK_WITH_INFO(nbOutputs == getNbOutputs(), "Required output to plugin is missing. Expected %d Got %d", + getNbOutputs(), nbOutputs); + + auto in_tensor = in[getInputTensorIndex()]; + + auto const minM + = std::accumulate(in_tensor.min.d, in_tensor.min.d + in_tensor.min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM + = std::accumulate(in_tensor.max.d, in_tensor.max.d + in_tensor.max.nbDims - 1, 1, std::multiplies<int>()); + + auto weights_1 = in[getExpertWeights1Index()]; + auto weights_2 = in[getExpertWeights2Index()]; + int inner_dim_idx = getGemmShapeInnerDimIndex(); + int const maxK = weights_1.max.d[inner_dim_idx]; + int const maxN = weights_2.max.d[inner_dim_idx]; + int const minK = weights_1.min.d[inner_dim_idx]; + int const minN = weights_2.min.d[inner_dim_idx]; + + TLLM_CHECK_WITH_INFO(minN == maxN, "Variable out channels is not allowed"); + TLLM_CHECK_WITH_INFO(minK == maxK, "Variable in channels is not allowed"); + TLLM_CHECK_WITH_INFO(maxK == mExpertHiddenSize && maxN == mExpertInterSize, + "Configured tensor sizes %dx%d does not match constructor param size %ldx%ld", maxK, maxN, mExpertHiddenSize, + mExpertInterSize); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, maxN, maxK}; + } + + mGemmId1 = GemmIDMoe{1, mNumExperts, mExpertsPerToken, mParallelismConfig, mExpertHiddenSize, mExpertInterSize, + mGroupSize, mActivationType, mType, mWeightType, mQuantMode, !mMOERunner->use_fused_finalize_}; + mGemmId2 = GemmIDMoe{2, mNumExperts, mExpertsPerToken, mParallelismConfig, mExpertHiddenSize, mExpertInterSize, + mGroupSize, mActivationType, mType, mWeightType, mQuantMode, !mMOERunner->use_fused_finalize_}; + + if (hasLora()) + { + auto const N = utils::computeNDimension(true, in[getHostRequestTypeIndex()].max); + mLoraGemmId1 = GemmIdCublas(N, mExpertHiddenSize, mLoraType, false, true, mLoraType); + mLoraGemmId2 = GemmIdCublas(N, mExpertInterSize, mLoraType, false, true, mLoraType); + } +} + +auto MixtureOfExpertsPlugin::setupWorkspace(void* base_ptr, int64_t num_tokens, int num_reqs) const -> WorkspaceInfo +{ + size_t moe_workspace_size + = mMOERunner->getWorkspaceSize(num_tokens, mExpertHiddenSize, mExpertInterSize, mNumExperts, mExpertsPerToken, + mActivationType, mParallelismConfig, hasLora(), /*use_deepseek_fp8_block_scale=*/false, + /*min_latency_mode=*/false, hasExpertPrequantScales()); + + // Permutation map + size_t src_to_dest_map_size = mExpertsPerToken * num_tokens * sizeof(int); + + size_t lora_workspace_size = 0; + if (hasLora()) + { + int64_t num_reqs_lora = std::min(num_tokens * mExpertsPerToken, static_cast<int64_t>(num_reqs * mNumExperts)); + lora_workspace_size + = std::max(mLoraImpl1->getWorkspaceSize(num_tokens * mExpertsPerToken, num_reqs_lora, mLoraType), + mLoraImpl2->getWorkspaceSize(num_tokens * mExpertsPerToken, num_reqs_lora, mLoraType)); + } + + std::vector<size_t> workspaces{ + moe_workspace_size, + src_to_dest_map_size, + lora_workspace_size, + }; + + WorkspaceInfo info{}; + info.size = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + + if (base_ptr) + { + info.workspace = base_ptr; + info.src_to_dest_map = nextWorkspacePtr((int8_t*) info.workspace, moe_workspace_size); + info.lora_workspace = nextWorkspacePtr((int8_t*) info.src_to_dest_map, src_to_dest_map_size); + } + + return info; +} + +int64_t MixtureOfExpertsPlugin::getNumTokens(nvinfer1::PluginTensorDesc const* input_tensors) const +{ + int ndim = input_tensors[getInputTensorIndex()].dims.nbDims; + TLLM_CHECK_WITH_INFO( + 3 == ndim || 2 == ndim, "hidden_state dimension should be either 2 [b*s, hidden], or 3 [b, s, hidden]"); + int64_t num_tokens = input_tensors[getInputTensorIndex()].dims.d[0]; + if (ndim == 3) + { + num_tokens *= input_tensors[getInputTensorIndex()].dims.d[1]; + } + return num_tokens; +} + +size_t MixtureOfExpertsPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + TLLM_CHECK_WITH_INFO( + nbInputs == getNbInputs(), "Required input to plugin is missing. Expected %d Got %d", getNbInputs(), nbInputs); + TLLM_CHECK_WITH_INFO(nbOutputs == getNbOutputs(), "Required output to plugin is missing. Expected %d Got %d", + getNbOutputs(), nbOutputs); + + if (useSideStream()) + { + return 0; + } + int const num_tokens = getNumTokens(inputs); + int const num_lora_reqs = getNumLoraRequests(inputs); + return setupWorkspace(nullptr, num_tokens, num_lora_reqs).size; +} + +MOEParallelismConfig MixtureOfExpertsPlugin::getParallelismConfig() const +{ + return mParallelismConfig; +} + +QuantParams tensorrt_llm::plugins::MixtureOfExpertsPlugin::getQuantParams(nvinfer1::PluginTensorDesc const* inputDesc, + void const* const* inputs, int scale_1_idx, int scale_2_idx, int scale_3_idx, int scale_4_idx, int scale_5_idx, + int scale_6_idx, int scale_7_idx, int scale_8_idx) const +{ + void const* scale_1 = scale_1_idx >= 0 ? inputs[scale_1_idx] : nullptr; + void const* scale_2 = scale_2_idx >= 0 ? inputs[scale_2_idx] : nullptr; + void const* scale_3 = scale_3_idx >= 0 ? inputs[scale_3_idx] : nullptr; + void const* scale_4 = scale_4_idx >= 0 ? inputs[scale_4_idx] : nullptr; + void const* scale_5 = scale_5_idx >= 0 ? inputs[scale_5_idx] : nullptr; + void const* scale_6 = scale_6_idx >= 0 ? inputs[scale_6_idx] : nullptr; + void const* scale_7 = scale_7_idx >= 0 ? inputs[scale_7_idx] : nullptr; + void const* scale_8 = scale_8_idx >= 0 ? inputs[scale_8_idx] : nullptr; + nvinfer1::PluginTensorDesc const* desc_1 = scale_1_idx >= 0 ? &inputDesc[scale_1_idx] : nullptr; + nvinfer1::PluginTensorDesc const* desc_2 = scale_2_idx >= 0 ? &inputDesc[scale_2_idx] : nullptr; + nvinfer1::PluginTensorDesc const* desc_3 = scale_3_idx >= 0 ? &inputDesc[scale_3_idx] : nullptr; + nvinfer1::PluginTensorDesc const* desc_4 = scale_4_idx >= 0 ? &inputDesc[scale_4_idx] : nullptr; + nvinfer1::PluginTensorDesc const* desc_5 = scale_5_idx >= 0 ? &inputDesc[scale_5_idx] : nullptr; + nvinfer1::PluginTensorDesc const* desc_6 = scale_6_idx >= 0 ? &inputDesc[scale_6_idx] : nullptr; + auto const gated_inter_size = isGatedActivation(mActivationType) ? mExpertInterSize * 2 : mExpertInterSize; + auto const experts_per_node = mNumExperts / mParallelismConfig.ep_size; + if (hasExpertIntQuantScales()) + { + TLLM_CHECK(scale_1 && scale_2); + if (!hasGroupwiseIntQuantScales()) + { + TLLM_CHECK(!scale_3 && !scale_4 && !scale_5 && !scale_6); + TLLM_CHECK(desc_1->dims.nbDims == 2); + TLLM_CHECK(desc_2->dims.nbDims == 2); + TLLM_CHECK_WITH_INFO( + desc_1->dims.d[0] == experts_per_node, "Incorrect number of experts in int quant scale"); + TLLM_CHECK(desc_1->dims.d[1] == gated_inter_size); + TLLM_CHECK_WITH_INFO( + desc_2->dims.d[0] == experts_per_node, "Incorrect number of experts in int quant scale"); + TLLM_CHECK(desc_2->dims.d[1] == mExpertHiddenSize); + return QuantParams::Int(scale_1, scale_2); + } + else + { + TLLM_CHECK(desc_1->dims.nbDims == 3); + TLLM_CHECK(desc_2->dims.nbDims == 3); + TLLM_CHECK((scale_3 && scale_4) || !hasExpertPrequantScales()); + TLLM_CHECK((scale_5 && scale_6) || !hasExpertWeightQuantZeros()); + TLLM_CHECK((scale_7 && scale_8) || !hasGroupwiseFp8Alpha()); + return QuantParams::GroupWise(mGroupSize, scale_1, scale_2, scale_3, scale_4, scale_5, scale_6, + static_cast<float const*>(scale_7), static_cast<float const*>(scale_8)); + } + } + else if (hasExpertFp8QuantScales()) + { + TLLM_CHECK(scale_1 && scale_2 && scale_3); + TLLM_CHECK(scale_4 || !hasExpertFp8FinalQuantScales()); + TLLM_CHECK((scale_5 != nullptr) == hasLora()); + TLLM_CHECK(!scale_6); + TLLM_CHECK(desc_1->dims.nbDims == 2); + TLLM_CHECK(desc_2->dims.nbDims == 1); + TLLM_CHECK(desc_3->dims.nbDims == 2); + TLLM_CHECK_WITH_INFO( + desc_1->dims.d[0] == experts_per_node && desc_1->dims.d[1] == 1, "Incorrect shape for weight FP8 scale"); + TLLM_CHECK(desc_2->dims.d[0] == 1); + TLLM_CHECK_WITH_INFO( + desc_3->dims.d[0] == experts_per_node && desc_3->dims.d[1] == 1, "Incorrect shape for weight FP8 scale"); + return QuantParams::FP8(static_cast<float const*>(scale_1), static_cast<float const*>(scale_2), + static_cast<float const*>(scale_3), static_cast<float const*>(scale_4), static_cast<float const*>(scale_5)); + } + else if (hasFP4QuantScales()) + { + TLLM_CHECK(scale_1 && scale_2 && scale_3 && scale_4 && scale_5 && scale_6); + TLLM_CHECK(desc_1->dims.nbDims == 1); + TLLM_CHECK(desc_2->dims.nbDims == 3); + TLLM_CHECK(desc_3->dims.nbDims == 1); + TLLM_CHECK(desc_4->dims.nbDims == 1); + TLLM_CHECK(desc_5->dims.nbDims == 3); + TLLM_CHECK(desc_6->dims.nbDims == 1); + TLLM_CHECK(desc_1->dims.d[0] == 1); + TLLM_CHECK_WITH_INFO(desc_2->dims.d[0] == experts_per_node && desc_2->dims.d[1] == gated_inter_size + && desc_2->dims.d[2] + == mExpertHiddenSize / TmaWarpSpecializedGroupedGemmInput::NVFP4BlockScaleVectorSize, + "Incorrect shape for FP4 scale"); + TLLM_CHECK_WITH_INFO(desc_3->dims.d[0] == experts_per_node, "Incorrect shape for FP4 scale"); + TLLM_CHECK(desc_4->dims.d[0] == 1); + TLLM_CHECK_WITH_INFO(desc_5->dims.d[0] == experts_per_node && desc_5->dims.d[1] == mExpertHiddenSize + && desc_5->dims.d[2] + == mExpertInterSize / TmaWarpSpecializedGroupedGemmInput::NVFP4BlockScaleVectorSize, + "Incorrect shape for FP4 scale"); + TLLM_CHECK_WITH_INFO(desc_6->dims.d[0] == experts_per_node, "Incorrect shape for FP4 scale"); + return QuantParams::FP4(static_cast<float const*>(scale_1), + static_cast<TmaWarpSpecializedGroupedGemmInput::ElementSF const*>(scale_2), + static_cast<float const*>(scale_3), static_cast<float const*>(scale_4), + static_cast<TmaWarpSpecializedGroupedGemmInput::ElementSF const*>(scale_5), + static_cast<float const*>(scale_6)); + } + return {}; +} + +int MixtureOfExpertsPlugin::getNumLoraRequests(nvinfer1::PluginTensorDesc const* input_tensors) const +{ + if (!hasLora()) + return 0; + int num_reqs = input_tensors[getLoraFC1RanksIndex()].dims.d[0]; + return num_reqs; +} + +LoraParams MixtureOfExpertsPlugin::getLoraParams( + nvinfer1::PluginTensorDesc const* inputDesc, void const* const* inputs, void* workspace) +{ + TLLM_CHECK(hasLora()); + + int const num_reqs = getNumLoraRequests(inputDesc); + int64_t const num_tokens = getNumTokens(inputDesc); + bool is_gated_actiation = isGatedActivation(mActivationType); + + mLoraExpandFC1WeightPtrs.clear(); + mLoraExpandFC2WeightPtrs.clear(); + mLoraExpandFC1Ranks.clear(); + mLoraExpandFC2Ranks.clear(); + + mLoraExpandFC1WeightPtrs.reserve(num_tokens * 2); + mLoraExpandFC2WeightPtrs.reserve(num_tokens * 2); + mLoraExpandFC1Ranks.reserve(num_tokens); + mLoraExpandFC2Ranks.reserve(num_tokens); + + if (is_gated_actiation) + { + mLoraExpandGatedWeightPtrs.clear(); + mLoraExpandGatedRanks.clear(); + mLoraExpandGatedWeightPtrs.reserve(num_tokens * 2); + mLoraExpandGatedRanks.reserve(num_tokens); + } + + int const seq_len = mRemoveInputPadding ? 0 : inputDesc[getInputTensorIndex()].dims.d[1]; + int32_t const* req_types = static_cast<int32_t const*>(inputs[getHostRequestTypeIndex()]); + int32_t const* host_context_lens + = mRemoveInputPadding ? static_cast<int32_t const*>(inputs[getHostContextLengthIndex()]) : nullptr; + + auto const fc1_lora_weight_ptrs = static_cast<void const* const*>(inputs[getLoraFC1WeightPtrsIndex()]); + auto const fc1_lora_ranks = static_cast<int32_t const*>(inputs[getLoraFC1RanksIndex()]); + + auto const fc2_lora_weight_ptrs = static_cast<void const* const*>(inputs[getLoraFC2WeightPtrsIndex()]); + auto const fc2_lora_ranks = static_cast<int32_t const*>(inputs[getLoraFC2RanksIndex()]); + + auto const gated_lora_weight_ptrs + = is_gated_actiation ? static_cast<void const* const*>(inputs[getLoraGatedWeightPtrsIndex()]) : nullptr; + auto const gated_lora_ranks + = is_gated_actiation ? static_cast<int32_t const*>(inputs[getLoraGatedRanksIndex()]) : nullptr; + + int idx = 0; + for (int req_id = 0; req_id < num_reqs; req_id++) + { + RequestType const reqType = static_cast<RequestType const>(req_types[req_id]); + if (reqType == RequestType::kGENERATION) + { + // lora_weight_ptrs has 3 pointers for each module: A,B, and an optional DoRA magnitude + // the current LoRA implementation does not apply DoRA scaling, so the magnitude is ignored + mLoraExpandFC1WeightPtrs.push_back(fc1_lora_weight_ptrs[req_id * 3]); + mLoraExpandFC1WeightPtrs.push_back(fc1_lora_weight_ptrs[req_id * 3 + 1]); + mLoraExpandFC1Ranks.push_back(fc1_lora_ranks[req_id]); + + mLoraExpandFC2WeightPtrs.push_back(fc2_lora_weight_ptrs[req_id * 3]); + mLoraExpandFC2WeightPtrs.push_back(fc2_lora_weight_ptrs[req_id * 3 + 1]); + mLoraExpandFC2Ranks.push_back(fc2_lora_ranks[req_id]); + + if (is_gated_actiation) + { + mLoraExpandGatedWeightPtrs.push_back(gated_lora_weight_ptrs[req_id * 3]); + mLoraExpandGatedWeightPtrs.push_back(gated_lora_weight_ptrs[req_id * 3 + 1]); + mLoraExpandGatedRanks.push_back(gated_lora_ranks[req_id]); + } + + idx += 1; + } + else + { + int context_len = (mRemoveInputPadding ? host_context_lens[req_id] : seq_len); + + for (int context_id = 0; context_id < context_len; context_id++) + { + mLoraExpandFC1WeightPtrs.push_back(fc1_lora_weight_ptrs[req_id * 3]); + mLoraExpandFC1WeightPtrs.push_back(fc1_lora_weight_ptrs[req_id * 3 + 1]); + mLoraExpandFC1Ranks.push_back(fc1_lora_ranks[req_id]); + + mLoraExpandFC2WeightPtrs.push_back(fc2_lora_weight_ptrs[req_id * 3]); + mLoraExpandFC2WeightPtrs.push_back(fc2_lora_weight_ptrs[req_id * 3 + 1]); + mLoraExpandFC2Ranks.push_back(fc2_lora_ranks[req_id]); + + if (is_gated_actiation) + { + mLoraExpandGatedWeightPtrs.push_back(gated_lora_weight_ptrs[req_id * 3]); + mLoraExpandGatedWeightPtrs.push_back(gated_lora_weight_ptrs[req_id * 3 + 1]); + mLoraExpandGatedRanks.push_back(gated_lora_ranks[req_id]); + } + } + idx += context_len; + } + } + + TLLM_CHECK_WITH_INFO(idx == num_tokens, fmtstr("idx %d num_tokens %ld", idx, num_tokens)); + + return LoraParams(num_reqs, mLoraExpandFC1Ranks.data(), mLoraExpandFC1WeightPtrs.data(), mLoraExpandFC2Ranks.data(), + mLoraExpandFC2WeightPtrs.data(), mLoraImpl1, mLoraImpl2, workspace, &mMemcpyEvent, mLoraExpandGatedRanks.data(), + mLoraExpandGatedWeightPtrs.data()); +} + +int MixtureOfExpertsPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace_ptr, + cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + + int64_t const num_tokens = getNumTokens(inputDesc); + int64_t const num_reqs = getNumLoraRequests(inputDesc); + + if (useSideStream()) + { + // Prepare the side stream + if (!mSideStreamPtr) + { + auto const resource_name = nvinfer1::pluginInternal::SideStream::getResourceKey(mSideStreamId); + nvinfer1::pluginInternal::SideStream side_stream{}; + mSideStreamPtr = reinterpret_cast<nvinfer1::pluginInternal::SideStream*>( + getPluginRegistry()->acquirePluginResource(resource_name.c_str(), &side_stream)); + } + // Debug the code with the main stream stalled (only executed when the environment variable + // TLLM_DEBUG_MOE_STALL_MAIN is set and has a positive value) + mSideStreamPtr->stallMainStream("TLLM_DEBUG_MOE_STALL_MAIN", stream, mDebugStallMain); + // The side stream waits for the inputs managed by the main stream to be ready + mSideStreamPtr->waitMainStreamOnSideStream(stream); + // Provide data dependency for the shared experts running after this plugin by copying inputs on the main stream + size_t count = 1; + for (int i = 0; i < inputDesc[getInputDummyTensorIndex()].dims.nbDims; ++i) + { + count *= inputDesc[getInputDummyTensorIndex()].dims.d[i]; + } + count *= tensorrt_llm::runtime::BufferDataType(inputDesc[getInputDummyTensorIndex()].type).getSize(); + TLLM_CUDA_CHECK(cudaMemcpyAsync(outputs[getOutputDummyTensorIndex()], inputs[getInputDummyTensorIndex()], count, + cudaMemcpyDeviceToDevice, stream)); + // Switch from the main stream to the side stream + stream = mSideStreamPtr->getStream(); + // The workspace is managed by the side stream (otherwise, the lifetime of workspace may be incorrect) + auto const workspace_size = setupWorkspace(nullptr, num_tokens, num_reqs).size; + workspace_ptr = mSideStreamPtr->getWorkspacePtr(workspace_size); + } + auto workspace = setupWorkspace(workspace_ptr, num_tokens, num_reqs); + + auto w1_desc = inputDesc[getExpertWeights1Index()]; + auto w2_desc = inputDesc[getExpertWeights2Index()]; + TLLM_CHECK(w1_desc.dims.nbDims == 3); + auto const experts_per_node = mNumExperts / mParallelismConfig.ep_size; + TLLM_CHECK(w1_desc.dims.d[0] == experts_per_node); + TLLM_CHECK(w2_desc.dims.nbDims == 3); + TLLM_CHECK(w2_desc.dims.d[0] == experts_per_node); + + auto [inner_packed_elements, outer_packed_elements] = getWeightPackedElements(); + int inner_dim_idx = getGemmShapeInnerDimIndex(); + int outer_dim_idx = getGemmShapeOuterDimIndex(); + TLLM_CHECK(w1_desc.dims.d[inner_dim_idx] * inner_packed_elements == mExpertHiddenSize); + if (isGatedActivation(mActivationType)) + { + TLLM_CHECK(w1_desc.dims.d[outer_dim_idx] * outer_packed_elements == mExpertInterSize * 2); + } + else + { + TLLM_CHECK(w1_desc.dims.d[outer_dim_idx] * outer_packed_elements == mExpertInterSize); + } + + TLLM_CHECK(w2_desc.dims.d[inner_dim_idx] * inner_packed_elements == mExpertInterSize); + TLLM_CHECK(w2_desc.dims.d[outer_dim_idx] * outer_packed_elements == mExpertHiddenSize); + + QuantParams quant_params{}; + if (hasExpertIntQuantScales()) + { + if (mGroupSize > 0) + { + quant_params = getQuantParams(inputDesc, inputs, getExpertIntQuantScale1Index(), + getExpertIntQuantScale2Index(), hasExpertPrequantScales() ? getExpertPrequantScales1Index() : -1, + hasExpertPrequantScales() ? getExpertPrequantScales2Index() : -1, + hasExpertWeightQuantZeros() ? getExpertIntQuantZeros1Index() : -1, + hasExpertWeightQuantZeros() ? getExpertIntQuantZeros2Index() : -1, + hasGroupwiseFp8Alpha() ? getExpertFp8Alpha1Index() : -1, + hasGroupwiseFp8Alpha() ? getExpertFp8Alpha2Index() : -1); + } + else + { + quant_params + = getQuantParams(inputDesc, inputs, getExpertIntQuantScale1Index(), getExpertIntQuantScale2Index()); + } + } + else if (hasExpertFp8QuantScales()) + { + quant_params = getQuantParams(inputDesc, inputs, // + getExpertFP8Dequant1Index(), // + getExpertFP8Quant2Index(), // + getExpertFP8Dequant2Index(), // + hasExpertFp8FinalQuantScales() ? getExpertFP8QuantFinalIndex() : -1, + hasLora() ? getInputFP8DequantIndex() : -1); + } + else if (hasFP4QuantScales()) + { + quant_params = getQuantParams(inputDesc, inputs, // + getFP4GlobalActSF1Index(), // + getFP4WeightSF1Index(), // + getFP4GlobalSF1Index(), // + getFP4GlobalActSF2Index(), // + getFP4WeightSF2Index(), // + getFP4GlobalSF2Index() // + ); + } + + LoraParams lora_params{}; + + if (hasLora()) + { + lora_params = getLoraParams(inputDesc, inputs, workspace.lora_workspace); + auto lora_gemm1 = mLoraProfiler->getBestConfig(num_tokens, mLoraGemmId1); + auto lora_gemm2 = mLoraProfiler->getBestConfig(num_tokens, mLoraGemmId2); + + mLoraImpl1->setBestTactic(lora_gemm1); + mLoraImpl2->setBestTactic(lora_gemm2); + } + + std::optional<tensorrt_llm::cutlass_extensions::CutlassGemmConfig> gemm1; + std::optional<tensorrt_llm::cutlass_extensions::CutlassGemmConfig> gemm2; + if (common::getEnvForceDeterministicMOE()) + { + gemm1 = mMOERunner->getTactics(MoeGemmId::GEMM_1)[0]; + gemm2 = mMOERunner->getTactics(MoeGemmId::GEMM_2)[0]; + } + else + { + gemm1 = mGemmProfiler->getBestConfig(num_tokens, mGemmId1); + gemm2 = mGemmProfiler->getBestConfig(num_tokens, mGemmId2); + } + + MoeMinLatencyParams min_latency_params{}; + mMOERunner->setTactic(gemm1, gemm2); +#ifdef USING_OSS_CUTLASS_MOE_GEMM + mMOERunner->runMoe(inputs[getInputTensorIndex()], nullptr, true, + static_cast<int const*>(inputs[getTokenSelectedExpertsIndex()]), + hasFinalScales() ? static_cast<float const*>(inputs[getTokenFinalScalesIndex()]) : nullptr, + inputs[getExpertWeights1Index()], hasBias() ? inputs[getExpertBias1Index()] : nullptr, + ActivationParams(mActivationType), inputs[getExpertWeights2Index()], + hasBias() ? inputs[getExpertBias2Index()] : nullptr, quant_params, num_tokens, num_tokens, mExpertHiddenSize, + mExpertHiddenSize /*TRT does not support padding, safe to assume padded/unpadded hidden sizes are the same*/, + mExpertInterSize, mNumExperts, mExpertsPerToken, static_cast<char*>(workspace.workspace), + // Outputs + outputs[getOutputTensorIndex()], static_cast<int*>(workspace.src_to_dest_map), mParallelismConfig, + /*enable_alltoall=*/false, hasLora(), lora_params, /*use_deepseek_fp8_block_scale=*/false, + /*min_latency_mode=*/false, min_latency_params, stream); +#else + mMOERunner->runMoe(inputs[getInputTensorIndex()], nullptr, true, + static_cast<int const*>(inputs[getTokenSelectedExpertsIndex()]), + hasFinalScales() ? static_cast<float const*>(inputs[getTokenFinalScalesIndex()]) : nullptr, + inputs[getExpertWeights1Index()], hasBias() ? inputs[getExpertBias1Index()] : nullptr, + ActivationParams(mActivationType), inputs[getExpertWeights2Index()], + hasBias() ? inputs[getExpertBias2Index()] : nullptr, quant_params, num_tokens, num_tokens, mExpertHiddenSize, + mExpertInterSize, mNumExperts, mExpertsPerToken, static_cast<char*>(workspace.workspace), + // Outputs + outputs[getOutputTensorIndex()], static_cast<int*>(workspace.src_to_dest_map), mParallelismConfig, hasLora(), + lora_params, /*use_deepseek_fp8_block_scale=*/false, + /*min_latency_mode=*/false, min_latency_params, stream); +#endif + + if (useSideStream()) + { + // Debug the code with the side stream stalled (only executed when the environment variable + // TLLM_DEBUG_MOE_STALL_SIDE is set and has a positive value) + mSideStreamPtr->stallSideStream("TLLM_DEBUG_MOE_STALL_SIDE", mDebugStallSide); + } + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType MixtureOfExpertsPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == getOutputTensorIndex() || index == getOutputDummyTensorIndex()); + if (useSideStream() && index == getOutputDummyTensorIndex()) + { + return inputTypes[getInputDummyTensorIndex()]; + } + return mOutputType; +} + +// IPluginV2 Methods +char const* MixtureOfExpertsPlugin::getPluginType() const noexcept +{ + return MIXTURE_OF_EXPERTS_PLUGIN_NAME; +} + +char const* MixtureOfExpertsPlugin::getPluginVersion() const noexcept +{ + return MIXTURE_OF_EXPERTS_PLUGIN_VERSION; +} + +int MixtureOfExpertsPlugin::initialize() noexcept +{ + mGemmProfiler->setGemmToProfile(kernels::GemmProfilerBackend::GemmToProfile::GEMM_1); + mGemmProfiler->profileTactics(this, mType, mDims, mGemmId1); + mGemmProfiler->setGemmToProfile(kernels::GemmProfilerBackend::GemmToProfile::GEMM_2); + mGemmProfiler->profileTactics(this, mType, mDims, mGemmId2); + + if (hasLora()) + { + mLoraImpl1->setGemmConfig(); + mLoraImpl2->setGemmConfig(); + + mLoraProfiler->profileTactics(mLoraImpl1->getCublasWrapper(), mType, mDims, mLoraGemmId1); + mLoraProfiler->profileTactics(mLoraImpl2->getCublasWrapper(), mType, mDims, mLoraGemmId2); + } + return 0; +} + +void MixtureOfExpertsPlugin::terminate() noexcept +{ + if (mSideStreamPtr) + { + auto const resource_name = nvinfer1::pluginInternal::SideStream::getResourceKey(mSideStreamId); + getPluginRegistry()->releasePluginResource(resource_name.c_str()); + mSideStreamPtr = nullptr; + } +} + +void MixtureOfExpertsPlugin::destroy() noexcept +{ + if (hasLora()) + { + TLLM_CUDA_CHECK(cudaEventDestroy(mMemcpyEvent)); + } + // This gets called when the network containing plugin is destroyed + delete this; +} + +void MixtureOfExpertsPlugin::setPluginNamespace(char const* libNamespace) noexcept +{ + mNamespace = libNamespace; +} + +char const* MixtureOfExpertsPlugin::getPluginNamespace() const noexcept +{ + return mNamespace.c_str(); +} + +/////////////// + +char const* MixtureOfExpertsPluginCreator::getPluginName() const noexcept +{ + return MIXTURE_OF_EXPERTS_PLUGIN_NAME; +} + +char const* MixtureOfExpertsPluginCreator::getPluginVersion() const noexcept +{ + return MIXTURE_OF_EXPERTS_PLUGIN_VERSION; +} + +nvinfer1::PluginFieldCollection const* MixtureOfExpertsPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +MixtureOfExpertsPluginCreator::MixtureOfExpertsPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(nvinfer1::PluginField("remove_input_padding", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("number_of_experts", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("experts_per_token", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("expert_hidden_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("expert_inter_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("groupwise_quant_algo", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("group_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("activation_type", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("weight_type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("quant_mode", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("use_final_scales", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("use_bias", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("tp_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("tp_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("ep_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("ep_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("side_stream_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("use_lora", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("lora_type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(nvinfer1::PluginField("max_low_rank", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +IPluginV2* MixtureOfExpertsPluginCreator::createPlugin( + char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept +{ + nvinfer1::PluginField const* fields = fc->fields; + int mRemoveInputPadding{}; + int mNumExperts{}; + int mExpertsPerToken{}; + int mExpertHiddenSize{}; + int mExpertInterSize{}; + int mGroupwiseQuantAlgo{}; + int mGroupSize{}; + int mActivationType{}; + int mType{}; + int mWeightType{}; + int mOutputType{INT_MAX}; + int mQuantMode{}; + int mUseFinalScales{1}; // Default to true + int mUseBias{0}; + int mTPSize{}; + int mTPRank{}; + int mEPSize{}; + int mEPRank{}; + int mRequiresDeterminism{0}; + int mSideStreamId{0}; + int mUseLora{}; + int mLoraType{INT_MAX}; + int mMaxLowRank{0}; + + // Read configurations from each fields + struct MapPair + { + char const* key; + int& field; + bool optional = false; + bool set = false; + }; + + std::array input_map{ + MapPair{"remove_input_padding", std::ref(mRemoveInputPadding)}, + MapPair{"number_of_experts", std::ref(mNumExperts)}, + MapPair{"experts_per_token", std::ref(mExpertsPerToken)}, + MapPair{"expert_hidden_size", std::ref(mExpertHiddenSize)}, + MapPair{"expert_inter_size", std::ref(mExpertInterSize)}, + MapPair{"groupwise_quant_algo", std::ref(mGroupwiseQuantAlgo)}, + MapPair{"group_size", std::ref(mGroupSize)}, + MapPair{"activation_type", std::ref(mActivationType)}, + MapPair{"type_id", std::ref(mType)}, + MapPair{"weight_type_id", std::ref(mWeightType)}, + MapPair{"quant_mode", std::ref(mQuantMode)}, + MapPair{"tp_size", std::ref(mTPSize)}, + MapPair{"tp_rank", std::ref(mTPRank)}, + MapPair{"ep_size", std::ref(mEPSize)}, + MapPair{"ep_rank", std::ref(mEPRank)}, + MapPair{"use_lora", std::ref(mUseLora)}, + MapPair{"use_final_scales", std::ref(mUseFinalScales)}, + + // Optional + MapPair{"use_bias", std::ref(mUseBias), true}, + MapPair{"output_type_id", std::ref(mOutputType), true}, + MapPair{"force_determinism", std::ref(mRequiresDeterminism), true}, + MapPair{"side_stream_id", std::ref(mSideStreamId), true}, + MapPair{"lora_type_id", std::ref(mLoraType), true}, + MapPair{"max_low_rank", std::ref(mMaxLowRank), true}, + }; + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + for (auto& item : input_map) + { + if (!strcmp(item.key, attrName)) + { + TLLM_CHECK(fields[i].type == nvinfer1::PluginFieldType::kINT32); + TLLM_CHECK_WITH_INFO(!item.set, "Parameter %s was set twice", item.key); + item.field = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + item.set = true; + } + } + } + + for (auto& item : input_map) + { + TLLM_CHECK_WITH_INFO(item.set || item.optional, "Parameter %s is required but not set", item.key); + } + + // Output type is optional, if not set it to the same as mType + if (mOutputType == INT_MAX) + { + mOutputType = mType; + } + + if (mUseLora) + { + TLLM_CHECK_WITH_INFO(mLoraType != INT_MAX && mMaxLowRank != 0, + "MoE fuse lora, lora_type_id and max_low_rank are required but not set"); + } + + try + { + auto gemmProfiler = moePluginProfiler.createGemmPluginProfiler(/* inference */ false); + auto loraProfiler = loraPluginProfileManager.createGemmPluginProfiler(/* inference */ false, /* skip */ true); + auto* obj = new MixtureOfExpertsPlugin( + // Constructor parameters + mRemoveInputPadding, mNumExperts, mExpertsPerToken, mExpertHiddenSize, mExpertInterSize, + mGroupwiseQuantAlgo, mGroupSize, static_cast<ActivationType>(mActivationType), + static_cast<nvinfer1::DataType>(mType), static_cast<nvinfer1::DataType>(mWeightType), + static_cast<nvinfer1::DataType>(mOutputType), QuantMode(mQuantMode), mUseFinalScales != 0, mUseBias != 0, + mTPSize, mTPRank, mEPSize, mEPRank, mRequiresDeterminism != 0, mSideStreamId, gemmProfiler, mUseLora != 0, + static_cast<nvinfer1::DataType>(mLoraType), loraProfiler, mMaxLowRank); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* MixtureOfExpertsPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call MixtureOfExpertsPlugin::destroy() + try + { + auto gemmProfiler = moePluginProfiler.createGemmPluginProfiler(/* inference */ true); + auto loraProfiler = loraPluginProfileManager.createGemmPluginProfiler(/* inference */ false, /* skip */ true); + + auto* obj = new MixtureOfExpertsPlugin( + // Constructor parameters + serialData, serialLength, gemmProfiler, loraProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +void MixtureOfExpertsPluginCreator::setPluginNamespace(char const* libNamespace) noexcept +{ + mNamespace = libNamespace; +} + +char const* MixtureOfExpertsPluginCreator::getPluginNamespace() const noexcept +{ + return mNamespace.c_str(); +} + +void MixtureOfExpertsGemmProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + checkInit(); + size_t bytes = backend.getWorkspaceSize(maxM); + this->setTmpWorkspaceSizeInBytes(bytes); +} + +void MixtureOfExpertsGemmProfiler::runTactic(int m, int n, int k, MixtureOfExpertsGemmProfiler::Config const& tactic, + char* workspace_ptr_char, cudaStream_t const& stream) +{ + checkInit(); + backend.runProfiler(m, tactic, workspace_ptr_char, /*expert_weights*/ nullptr, stream); +} + +auto MixtureOfExpertsGemmProfiler::getTactics(int m, int n, int k) const -> std::vector<Config> +{ + assert(mRunner); + return mRunner->mMOERunner->getTactics(backend.mGemmToProfile); +} + +void MixtureOfExpertsGemmProfiler::initTmpData( + int m, int n, int k, char* workspace, size_t ws_size, cudaStream_t stream) +{ + checkInit(); + backend.prepare(m, workspace, /*expert_weights*/ nullptr, stream); +} + +void MixtureOfExpertsGemmProfiler::checkInit() +{ + assert(mRunner); + if (init_backend) + { + return; + } + init_backend = true; + auto& plugin = *mRunner; +#ifdef USING_OSS_CUTLASS_MOE_GEMM + backend.init(*plugin.mMOERunner, backend.mGemmToProfile, plugin.mType, plugin.mWeightType, plugin.mOutputType, + plugin.mNumExperts, plugin.mExpertsPerToken, plugin.mExpertHiddenSize, + plugin.mExpertHiddenSize /*TRT backend does not support unpadded hidden size*/, plugin.mExpertInterSize, + plugin.mGroupSize, plugin.mActivationType, plugin.hasBias(), plugin.hasLora(), /*min_latency_mode=*/false, + /*need_weights=*/true, plugin.getParallelismConfig(), /*enable_alltoall=*/false); +#else + backend.init(*plugin.mMOERunner, backend.mGemmToProfile, plugin.mType, plugin.mWeightType, plugin.mOutputType, + plugin.mNumExperts, plugin.mExpertsPerToken, plugin.mExpertHiddenSize, plugin.mExpertInterSize, + plugin.mGroupSize, plugin.mActivationType, plugin.hasBias(), plugin.hasLora(), /*min_latency_mode=*/false, + /*need_weights=*/true, plugin.getParallelismConfig()); +#endif +} diff --git a/cpp/tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.h b/cpp/tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.h new file mode 100644 index 000000000000..feb1f10cdc70 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/mixtureOfExperts/mixtureOfExpertsPlugin.h @@ -0,0 +1,637 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2023 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#ifndef TRT_MIXTURE_OF_EXPERTS_PLUGIN_H +#define TRT_MIXTURE_OF_EXPERTS_PLUGIN_H + +#include "NvInferPlugin.h" +#include "tensorrt_llm/kernels/cutlass_kernels/include/cutlass_kernel_selector.h" +#if defined(USING_OSS_CUTLASS_MOE_GEMM) +#include "tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h" +#else +#include "moe_kernels.h" +#endif +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/kernels/lora/lora.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/plugins/cudaStreamPlugin/cudaStreamPlugin.h" +#include "tensorrt_llm/plugins/gemmPlugin/gemmPlugin.h" +#include "tensorrt_llm/runtime/cudaStream.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ +namespace kernels = CUTLASS_MOE_GEMM_KERNELS_NAMESPACE; +using MoeMinLatencyParams = CUTLASS_MOE_GEMM_KERNELS_NAMESPACE::MoeMinLatencyParams; +using MOEParallelismConfig = CUTLASS_MOE_GEMM_KERNELS_NAMESPACE::MOEParallelismConfig; +using QuantParams = CUTLASS_MOE_GEMM_KERNELS_NAMESPACE::QuantParams; +using MoeGemmId = CUTLASS_MOE_GEMM_NAMESPACE::MoeGemmId; +using ActivationType = CUTLASS_MOE_GEMM_NAMESPACE::ActivationType; +using ActivationParams = CUTLASS_MOE_GEMM_KERNELS_NAMESPACE::ActivationParams; +using TmaWarpSpecializedGroupedGemmInput = CUTLASS_MOE_GEMM_NAMESPACE::TmaWarpSpecializedGroupedGemmInput; +using CUTLASS_MOE_GEMM_NAMESPACE::isGatedActivation; + +class MixtureOfExpertsGemmProfiler; +using MixtureOfExpertsPluginProfilerPtr = std::shared_ptr<MixtureOfExpertsGemmProfiler>; +using GroupwiseQuantAlgo = tensorrt_llm::common::GroupwiseQuantAlgo; + +struct GemmIDMoe +{ + int gemm_idx; + int num_experts{}; + int experts_per_token{}; + kernels::MOEParallelismConfig parallelism_config{}; + int64_t hidden{}; + int64_t inter{}; + int64_t group_size{}; + ActivationType actfn{}; + nvinfer1::DataType dtype{}; + nvinfer1::DataType wdtype{}; + tensorrt_llm::common::QuantMode quant_mode; + bool determinism_mode = false; + + bool operator==(GemmIDMoe const& id) const + { + return id.gemm_idx == gemm_idx && id.num_experts == num_experts && id.experts_per_token == experts_per_token + && id.parallelism_config == parallelism_config && id.hidden == hidden && id.inter == inter + && id.group_size == group_size && id.actfn == actfn && id.dtype == dtype && id.wdtype == wdtype + && id.quant_mode == quant_mode && id.determinism_mode == determinism_mode; + } + + friend std::ostream& operator<<(std::ostream& out, GemmIDMoe const& id) + { + out << "gemm idx, experts, experts_per_token, parallelism_config, hidden, inter, group_size, actfn, dtype, " + "weight " + "type, parallelism mode, determinism mode=" + + << id.gemm_idx << "," << id.num_experts << "," << id.experts_per_token << "," << id.parallelism_config + << "," << id.hidden << "," << id.inter << "," << id.group_size << "," << static_cast<int>(id.actfn) << "," + << static_cast<int>(id.dtype) << "," << static_cast<int>(id.wdtype) << "," << id.quant_mode.value() << "," + << id.determinism_mode; + return out; + } +}; + +// Hash of GemmIDMoe +struct GemmIDMoeHash +{ + std::size_t operator()(GemmIDMoe const& id) const + { + size_t hash = std::hash<int>{}(id.gemm_idx); + hash ^= std::hash<int>{}(id.num_experts); + hash ^= std::hash<int>{}(id.experts_per_token); + hash ^= std::hash<int>{}(id.parallelism_config.tp_size); + hash ^= std::hash<int>{}(id.parallelism_config.ep_size); + hash ^= std::hash<int>{}(id.parallelism_config.tp_rank); + hash ^= std::hash<int>{}(id.parallelism_config.ep_rank); + hash ^= std::hash<int>{}(id.hidden); + hash ^= std::hash<int>{}(id.inter); + hash ^= std::hash<int>{}(id.group_size); + hash ^= std::hash<int>{}(static_cast<int>(id.actfn)); + hash ^= std::hash<int>{}(static_cast<int>(id.dtype)); + hash ^= std::hash<int>{}(static_cast<int>(id.wdtype)); + hash ^= std::hash<int>{}(static_cast<int>(id.quant_mode.value())); + return hash; + } +}; + +class MixtureOfExpertsPlugin : public nvinfer1::IPluginV2DynamicExt +{ +public: + using LoraPluginProfilerPtr = std::shared_ptr<CublasLtGemmPluginProfiler>; + using LoraImplPtr = std::shared_ptr<tensorrt_llm::kernels::LoraImpl>; + MixtureOfExpertsPlugin() = delete; + MixtureOfExpertsPlugin(bool remove_input_padding, int number_of_experts, int experts_per_token, + int expert_hidden_size, int expert_inter_size, int groupwise_quant_algo, int group_size, + ActivationType activation_type, nvinfer1::DataType type, nvinfer1::DataType weight_type, + nvinfer1::DataType output_type, tensorrt_llm::common::QuantMode quant_mode, bool use_final_scales, + bool use_bias, int tp_size, int tp_rank, int ep_size, int ep_rank, bool force_determinism, int side_stream_id, + MixtureOfExpertsPluginProfilerPtr gemm_profiler_ptr, bool use_lora, nvinfer1::DataType lora_type, + LoraPluginProfilerPtr lora_profiler, int max_low_rank); + MixtureOfExpertsPlugin(void const* data, size_t length, MixtureOfExpertsPluginProfilerPtr gemm_profiler_ptr, + LoraPluginProfilerPtr lora_profiler); + MixtureOfExpertsPlugin(MixtureOfExpertsPlugin const&); + + void init(); + + ~MixtureOfExpertsPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + + int getNbOutputs() const noexcept override + { + return 1 + useSideStream(); + } + + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + void setPluginNamespace(char const* pluginNamespace) noexcept override; + char const* getPluginNamespace() const noexcept override; + +private: + friend class MixtureOfExpertsGemmProfiler; + std::unique_ptr<kernels::CutlassMoeFCRunnerInterface> mMOERunner{}; + int mNumExperts{}; + int mExpertsPerToken{}; + int64_t mExpertHiddenSize{}; + int64_t mExpertInterSize{}; + int64_t mGroupwiseQuantAlgo{}; + int64_t mGroupSize{}; + ActivationType mActivationType; + nvinfer1::DataType mType{}; + nvinfer1::DataType mWeightType{}; + nvinfer1::DataType mOutputType{}; + tensorrt_llm::common::QuantMode mQuantMode; + bool mUseFinalScales{}; + bool mUseBias{}; + MOEParallelismConfig mParallelismConfig{}; + + GemmDims mDims{}; + bool mUseDeterministicKernels = false; + int mSideStreamId = 0; + + int mDebugStallMain = 0; + int mDebugStallSide = 0; + + GemmIDMoe mGemmId1{}; + GemmIDMoe mGemmId2{}; + + MixtureOfExpertsPluginProfilerPtr mGemmProfiler; + + // lora related + bool mUseLora{}; + nvinfer1::DataType mLoraType{}; + int mMaxLowRank{}; + bool mRemoveInputPadding{}; + + LoraImplPtr mLoraImpl1; + LoraImplPtr mLoraImpl2; + + GemmIdCublas mLoraGemmId1{}; + GemmIdCublas mLoraGemmId2{}; + LoraPluginProfilerPtr mLoraProfiler; + + std::vector<void const*> mLoraExpandFC1WeightPtrs{}; + std::vector<void const*> mLoraExpandFC2WeightPtrs{}; + std::vector<void const*> mLoraExpandGatedWeightPtrs{}; + std::vector<int32_t> mLoraExpandFC1Ranks{}; + std::vector<int32_t> mLoraExpandFC2Ranks{}; + std::vector<int32_t> mLoraExpandGatedRanks{}; + + cudaEvent_t mMemcpyEvent; + nvinfer1::pluginInternal::SideStream* mSideStreamPtr; + + // The below are not serialised + std::string const mLayerName{}; + std::string mNamespace{}; + + struct WorkspaceInfo + { + void* workspace{}; + void* src_to_dest_map{}; + void* lora_workspace{}; + size_t size{}; + }; + + int64_t getNumTokens(nvinfer1::PluginTensorDesc const* input_tensor) const; + WorkspaceInfo setupWorkspace(void* base_ptr, int64_t num_tokens, int num_reqs = 0) const; + + MOEParallelismConfig getParallelismConfig() const; + QuantParams getQuantParams(nvinfer1::PluginTensorDesc const* inputDesc, void const* const* inputs, + int scale_1_idx = -1, int scale_2_idx = -1, int scale_3_idx = -1, int scale_4_idx = -1, int scale_5_idx = -1, + int scale_6_idx = -1, int scale_7_idx = -1, int scale_8_idx = -1) const; + + int getNumLoraRequests(nvinfer1::PluginTensorDesc const* input_tensor) const; + tensorrt_llm::kernels::LoraParams getLoraParams( + nvinfer1::PluginTensorDesc const* inputDesc, void const* const* inputs, void* workspace); + + enum class RequestType : int32_t + { + kCONTEXT = 0, + kGENERATION = 1 + }; + + using IndexType = std::int32_t; + + // Inputs + constexpr static IndexType getInputTensorIndex() + { + return 0; + } + + constexpr static IndexType getExpertWeights1Index() + { + return getInputTensorIndex() + 1; + } + + constexpr static IndexType getExpertWeights2Index() + { + return getExpertWeights1Index() + 1; + } + + constexpr static IndexType getTokenSelectedExpertsIndex() + { + return getExpertWeights2Index() + 1; + } + + // Conditional inputs, we only allocate a new index if actually used + bool hasBias() const + { + return mUseBias; + } + + bool hasFinalScales() const + { + return mUseFinalScales; + } + + bool hasExpertIntQuantScales() const + { + return mQuantMode.hasInt4Weights() || mQuantMode.hasInt8Weights(); + } + + bool hasExpertFp8QuantScales() const + { + return mQuantMode.hasFp8Qdq(); + } + + bool hasExpertFp8FinalQuantScales() const + { + return hasExpertFp8QuantScales() && mOutputType == nvinfer1::DataType::kFP8; + } + + bool hasFP4QuantScales() const + { + return mQuantMode.hasNvfp4(); + } + + bool hasGroupwiseIntQuantScales() const + { + return mGroupwiseQuantAlgo > 0; + } + + bool hasExpertWeightQuantZeros() const + { + return mGroupwiseQuantAlgo & GroupwiseQuantAlgo::ZERO; + } + + bool hasExpertPrequantScales() const + { + return mGroupwiseQuantAlgo & GroupwiseQuantAlgo::PRE_QUANT_SCALE; + } + + bool hasGroupwiseFp8Alpha() const + { + return mGroupwiseQuantAlgo & GroupwiseQuantAlgo::FP8_ALPHA; + } + + bool useSideStream() const + { + return mSideStreamId > 0; + } + + bool hasLora() const + { + return mUseLora; + } + + bool hasGatedLoraWeightsAndRanks() const + { + return mUseLora && isGatedActivation(mActivationType); + } + + IndexType getTokenFinalScalesIndex() const + { + return getTokenSelectedExpertsIndex() + hasFinalScales(); + } + + IndexType getExpertBias1Index() const + { + return getTokenFinalScalesIndex() + hasBias(); + } + + IndexType getExpertBias2Index() const + { + return getExpertBias1Index() + hasBias(); + } + + /* + * Weight-Only int quant scales + */ + IndexType getExpertIntQuantScale1Index() const + { + return getExpertBias2Index() + hasExpertIntQuantScales(); + } + + IndexType getExpertIntQuantScale2Index() const + { + return getExpertIntQuantScale1Index() + hasExpertIntQuantScales(); + } + + /* + * FP8 Quant Scales + */ + IndexType getExpertFP8Dequant1Index() const + { + return getExpertIntQuantScale2Index() + hasExpertFp8QuantScales(); + } + + IndexType getExpertFP8Quant2Index() const + { + return getExpertFP8Dequant1Index() + hasExpertFp8QuantScales(); + } + + IndexType getExpertFP8Dequant2Index() const + { + return getExpertFP8Quant2Index() + hasExpertFp8QuantScales(); + } + + IndexType getExpertFP8QuantFinalIndex() const + { + return getExpertFP8Dequant2Index() + hasExpertFp8FinalQuantScales(); + } + + IndexType getInputFP8DequantIndex() const + { + return getExpertFP8QuantFinalIndex() + (hasExpertFp8QuantScales() && hasLora()); + } + + /* + * FP4 Quant Scales + */ + IndexType getFP4GlobalActSF1Index() const + { + return getInputFP8DequantIndex() + hasFP4QuantScales(); + } + + IndexType getFP4WeightSF1Index() const + { + return getFP4GlobalActSF1Index() + hasFP4QuantScales(); + } + + IndexType getFP4GlobalSF1Index() const + { + return getFP4WeightSF1Index() + hasFP4QuantScales(); + } + + IndexType getFP4GlobalActSF2Index() const + { + return getFP4GlobalSF1Index() + hasFP4QuantScales(); + } + + IndexType getFP4WeightSF2Index() const + { + return getFP4GlobalActSF2Index() + hasFP4QuantScales(); + } + + IndexType getFP4GlobalSF2Index() const + { + return getFP4WeightSF2Index() + hasFP4QuantScales(); + } + + /* + * Groupwise Params + */ + IndexType getExpertPrequantScales1Index() const + { + return getFP4GlobalSF2Index() + hasExpertPrequantScales(); + } + + IndexType getExpertPrequantScales2Index() const + { + return getExpertPrequantScales1Index() + hasExpertPrequantScales(); + } + + IndexType getExpertIntQuantZeros1Index() const + { + return getExpertPrequantScales2Index() + hasExpertWeightQuantZeros(); + } + + IndexType getExpertIntQuantZeros2Index() const + { + return getExpertIntQuantZeros1Index() + hasExpertWeightQuantZeros(); + } + + IndexType getExpertFp8Alpha1Index() const + { + return getExpertIntQuantZeros2Index() + hasGroupwiseFp8Alpha(); + } + + IndexType getExpertFp8Alpha2Index() const + { + return getExpertFp8Alpha1Index() + hasGroupwiseFp8Alpha(); + } + + /* + * LoRA params + */ + IndexType getLoraFC1WeightPtrsIndex() const + { + return getExpertFp8Alpha2Index() + hasLora(); + } + + IndexType getLoraFC1RanksIndex() const + { + return getLoraFC1WeightPtrsIndex() + hasLora(); + } + + IndexType getLoraFC2WeightPtrsIndex() const + { + return getLoraFC1RanksIndex() + hasLora(); + } + + IndexType getLoraFC2RanksIndex() const + { + return getLoraFC2WeightPtrsIndex() + hasLora(); + } + + IndexType getLoraGatedWeightPtrsIndex() const + { + return getLoraFC2RanksIndex() + hasGatedLoraWeightsAndRanks(); + } + + IndexType getLoraGatedRanksIndex() const + { + return getLoraGatedWeightPtrsIndex() + hasGatedLoraWeightsAndRanks(); + } + + IndexType getHostRequestTypeIndex() const + { + return getLoraGatedRanksIndex() + hasLora(); + } + + IndexType getHostContextLengthIndex() const + { + return getHostRequestTypeIndex() + (mRemoveInputPadding && hasLora()); + } + + IndexType getInputDummyTensorIndex() const + { + return getHostContextLengthIndex() + useSideStream(); + } + + IndexType getNbInputs() const + { + return getInputDummyTensorIndex() + 1; + } + + // Outputs + constexpr static IndexType getOutputTensorIndex() + { + return 0; + } + + IndexType getOutputDummyTensorIndex() const + { + return getOutputTensorIndex() + useSideStream(); + } + + /** + * Get the index of the expert shape tuple that represents the inner dimension + */ + int getGemmShapeInnerDimIndex() const + { + // In weight only mode the shape is transposed + return hasExpertIntQuantScales() ? 1 : 2; + } + + /** + * Get the index of the expert shape tuple that represents the outer dimension + */ + int getGemmShapeOuterDimIndex() const + { + // In weight only mode the shape is transposed + return hasExpertIntQuantScales() ? 2 : 1; + } + + /** + * Get quantization dimension scaling factor + */ + std::pair<int, int> getWeightPackedElements() const + { + if (mGroupwiseQuantAlgo == 0) + { + return {1, mQuantMode.hasInt4Weights() ? 2 : 1}; + } + else + { + return {1, 4}; + } + } +}; + +class MixtureOfExpertsGemmProfiler + : public tensorrt_llm::plugins::GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + MixtureOfExpertsPlugin*, GemmIDMoe, GemmIDMoeHash> +{ +public: + MixtureOfExpertsGemmProfiler() + { + // NOTE: Do not access mPlugin here, since we are called from the constructor before all fields are init + } + + void setGemmToProfile(kernels::GemmProfilerBackend::GemmToProfile gemm_to_profile) + { + // Just set the backend directly. This will just be reused in checkInit(). + backend.mGemmToProfile = gemm_to_profile; + // We need to set the backend to reinitialise itself with the new GEMM + init_backend = false; + } + + void setMaxProfileM(int maxProfileM) + { + mMaxProfileM = maxProfileM; + } + + virtual int getMaxProfileM() const override + { + return mMaxProfileM; + } + +protected: + using Config = tensorrt_llm::cutlass_extensions::CutlassGemmConfig; + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + std::vector<Config> getTactics(int m, int n, int k) const override; + void initTmpData(int maxM, int n, int k, char* workspace, size_t size, cudaStream_t stream) override; + + void checkInit(); + + bool init_backend = false; + kernels::GemmProfilerBackend backend{}; + +private: + int mMaxProfileM = 0; +}; + +class MixtureOfExpertsPluginCreator : public nvinfer1::IPluginCreator +{ +public: + MixtureOfExpertsPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + + void setPluginNamespace(char const* pluginNamespace) noexcept override; + + char const* getPluginNamespace() const noexcept override; + +private: + GemmPluginProfilerManager<MixtureOfExpertsGemmProfiler> moePluginProfiler; + GemmPluginProfilerManager<CublasLtGemmPluginProfiler> loraPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; + std::string mNamespace; +}; + +} // namespace tensorrt_llm::plugins + +#endif // TRT_MIXTURE_OF_EXPERTS_PLUGIN_H diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/ncclPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/allgatherPlugin.cpp b/cpp/tensorrt_llm/plugins/ncclPlugin/allgatherPlugin.cpp new file mode 100644 index 000000000000..4825dd51bbab --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/allgatherPlugin.cpp @@ -0,0 +1,253 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "allgatherPlugin.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include <nccl.h> + +using namespace nvinfer1; +using tensorrt_llm::plugins::AllgatherPluginCreator; +using tensorrt_llm::plugins::AllgatherPlugin; + +static char const* ALLGATHER_PLUGIN_VERSION{"1"}; +static char const* ALLGATHER_PLUGIN_NAME{"AllGather"}; +PluginFieldCollection AllgatherPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> AllgatherPluginCreator::mPluginAttributes; + +AllgatherPlugin::AllgatherPlugin(std::set<int> group, nvinfer1::DataType type) + : mGroup(std::move(group)) + , mType(type) +{ +} + +// Parameterized constructor +AllgatherPlugin::AllgatherPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mType); + mGroup.clear(); + int groupItem = 0; + while (d != a + length) + { + read(d, groupItem); + mGroup.insert(groupItem); + } + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* AllgatherPlugin::clone() const noexcept +{ + auto* plugin = new AllgatherPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs AllgatherPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + auto ret = inputs[0]; + auto groupSize = exprBuilder.constant(mGroup.size()); + ret.d[0] = exprBuilder.operation(DimensionOperation::kPROD, *ret.d[0], *groupSize); + return ret; +} + +bool AllgatherPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); +} + +void AllgatherPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t AllgatherPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int AllgatherPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + size_t size = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims; ++i) + { + size *= inputDesc[0].dims.d[i]; + } + + TLLM_CHECK_WITH_INFO(mNcclComm.get() != nullptr, "mNcclComm should be initialized before used"); + NCCLCHECK(ncclAllGather(inputs[0], outputs[0], size, (*getDtypeMap())[inputDesc[0].type], *mNcclComm, stream)); + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType AllgatherPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + assert(index == 0); + return inputTypes[0]; +} + +// IPluginV2 Methods + +char const* AllgatherPlugin::getPluginType() const noexcept +{ + return ALLGATHER_PLUGIN_NAME; +} + +char const* AllgatherPlugin::getPluginVersion() const noexcept +{ + return ALLGATHER_PLUGIN_VERSION; +} + +int AllgatherPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int AllgatherPlugin::initialize() noexcept +{ + if (isBuilding()) + { + return 0; + } + TLLM_LOG_TRACE("%s start for rank %d", __PRETTY_FUNCTION__, COMM_SESSION.getRank()); + mNcclComm = getComm(mGroup); + TLLM_LOG_TRACE("%s stop for rank %d", __PRETTY_FUNCTION__, COMM_SESSION.getRank()); + return 0; +} + +void AllgatherPlugin::terminate() noexcept {} + +size_t AllgatherPlugin::getSerializationSize() const noexcept +{ + return sizeof(int) * mGroup.size() + sizeof(mType); +} + +void AllgatherPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + for (auto it = mGroup.begin(); it != mGroup.end(); ++it) + { + write(d, *it); + } + TLLM_CHECK(d == a + getSerializationSize()); +} + +void AllgatherPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +AllgatherPluginCreator::AllgatherPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("group", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* AllgatherPluginCreator::getPluginName() const noexcept +{ + return ALLGATHER_PLUGIN_NAME; +} + +char const* AllgatherPluginCreator::getPluginVersion() const noexcept +{ + return ALLGATHER_PLUGIN_VERSION; +} + +PluginFieldCollection const* AllgatherPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* AllgatherPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + std::set<int> group; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "group")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + auto const* r = static_cast<int const*>(fields[i].data); + for (int j = 0; j < fields[i].length; ++j) + { + group.insert(*r); + ++r; + } + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + + try + { + auto* obj = new AllgatherPlugin(group, type); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* AllgatherPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call AllgatherPlugin::destroy() + try + { + auto* obj = new AllgatherPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/allgatherPlugin.h b/cpp/tensorrt_llm/plugins/ncclPlugin/allgatherPlugin.h new file mode 100644 index 000000000000..3d7810e6bd49 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/allgatherPlugin.h @@ -0,0 +1,92 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class AllgatherPlugin : public BasePlugin +{ +public: + AllgatherPlugin(std::set<int> group, nvinfer1::DataType type); + + AllgatherPlugin(void const* data, size_t length); + + ~AllgatherPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + const std::string mLayerName; + std::set<int> mGroup; + nvinfer1::DataType mType; + std::shared_ptr<ncclComm_t> mNcclComm; +}; + +class AllgatherPluginCreator : public BaseCreator +{ +public: + AllgatherPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/allreducePlugin.cpp b/cpp/tensorrt_llm/plugins/ncclPlugin/allreducePlugin.cpp new file mode 100644 index 000000000000..24d9aff418f7 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/allreducePlugin.cpp @@ -0,0 +1,986 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "allreducePlugin.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/customAllReduceUtils.h" +#include "tensorrt_llm/common/dataType.h" +#include "tensorrt_llm/common/nvmlWrapper.h" +#include "tensorrt_llm/kernels/customAllReduceKernels.h" +#include "tensorrt_llm/kernels/userbuffers/ub_interface.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include <nccl.h> + +#include <unordered_set> + +using namespace nvinfer1; +using tensorrt_llm::plugins::AllreducePluginCreator; +using tensorrt_llm::plugins::AllreducePlugin; +using tensorrt_llm::kernels::AllReduceFusionOp; +using tensorrt_llm::kernels::AllReduceStrategyType; +using tensorrt_llm::kernels::AllReduceStrategyConfig; +using tensorrt_llm::mpi::MpiTag; + +static char const* ALLREDUCE_PLUGIN_VERSION{"1"}; +static char const* ALLREDUCE_PLUGIN_NAME{"AllReduce"}; +PluginFieldCollection AllreducePluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> AllreducePluginCreator::mPluginAttributes; + +AllreducePlugin::AllreducePlugin(std::set<int> group, nvinfer1::DataType type, AllReduceStrategyType strategy, + AllReduceStrategyConfig config, AllReduceFusionOp op, int32_t counter, float eps, int8_t affine, int8_t bias, + int8_t scale) + : mGroup(std::move(group)) + , mType(type) + , mStrategy(strategy) + , mConfig(config) + , mOp(op) + , mEps(eps) + , mAffine(affine) + , mBias(bias) + , mScale(scale) +{ + check(); +} + +// Parameterized constructor +AllreducePlugin::AllreducePlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mType); + read(d, mStrategy); + read(d, mConfig); + read(d, mOp); + read(d, mEps); + read(d, mAffine); + read(d, mBias); + read(d, mScale); + mGroup.clear(); + int groupItem = 0; + while (d != a + length) + { + read(d, groupItem); + mGroup.insert(groupItem); + } + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); + check(); +} + +void AllreducePlugin::check() noexcept +{ + if (mStrategy != AllReduceStrategyType::UB) + { + TLLM_CHECK(mOp != AllReduceFusionOp::LAST_PROCESS_FOR_UB); + } +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* AllreducePlugin::clone() const noexcept +{ + auto* plugin = new AllreducePlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs AllreducePlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM_QUANT_NVFP4 && mStrategy == AllReduceStrategyType::UB && mScale) + { + if (outputIndex == 0) + { + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int di = 0; di < ret.nbDims; ++di) + { + ret.d[di] = inputs[0].d[di]; + } + return ret; + } + else if (outputIndex == 2) + { + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int di = 0; di < ret.nbDims; ++di) + { + ret.d[di] = inputs[0].d[di]; + } + auto dimM = exprBuilder.operation( + DimensionOperation::kCEIL_DIV, *ret.d[ret.nbDims - 2], *exprBuilder.constant(128)); + ret.d[ret.nbDims - 2] = exprBuilder.operation(DimensionOperation::kPROD, *dimM, *exprBuilder.constant(128)); + ret.d[ret.nbDims - 1] = exprBuilder.operation( + DimensionOperation::kCEIL_DIV, *ret.d[ret.nbDims - 1], *exprBuilder.constant(16)); + return ret; + } + } + return inputs[0]; +} + +bool AllreducePlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + int base_inputs = 0; + switch (mStrategy) + { + case AllReduceStrategyType::NCCL: + case AllReduceStrategyType::UB: + case AllReduceStrategyType::NCCL_SYMMETRIC: base_inputs = 1; break; + default: base_inputs = 2; break; + } + int fusion_op_extra_inputs = 0; + int scale_idx = 0; + if (mOp != AllReduceFusionOp::NONE) + { + ++fusion_op_extra_inputs; + if (mAffine) + { + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_PREPOST_NORM) + ++fusion_op_extra_inputs; + ++fusion_op_extra_inputs; + } + if (mBias) + { + ++fusion_op_extra_inputs; + } + if (mScale) + { + scale_idx = base_inputs + fusion_op_extra_inputs; + ++fusion_op_extra_inputs; + } + } + + TLLM_CHECK(nbInputs == (base_inputs + fusion_op_extra_inputs)); + + if (pos == 1) + { + switch (mStrategy) + { + case AllReduceStrategyType::NCCL: + case AllReduceStrategyType::UB: + case AllReduceStrategyType::NCCL_SYMMETRIC: break; + default: return (inOut[pos].type == nvinfer1::DataType::kINT64) && (inOut[pos].format == TensorFormat::kLINEAR); + } + } + if (mStrategy == AllReduceStrategyType::UB) + { + if (mScale && pos == scale_idx) + { + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM_QUANT_NVFP4) + { + if (pos == nbInputs) + { + return (inOut[pos].type == nvinfer1::DataType::kFP4) && (inOut[pos].format == TensorFormat::kLINEAR); + } + if (pos == (nbInputs + 2)) + { + return (inOut[pos].type == nvinfer1::DataType::kFP8) && (inOut[pos].format == TensorFormat::kLINEAR); + } + } + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM_QUANT_FP8) + { + if (pos == nbInputs) + { + return (inOut[pos].type == nvinfer1::DataType::kFP8) && (inOut[pos].format == TensorFormat::kLINEAR); + } + } + } + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); +} + +void AllreducePlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t AllreducePlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +AllReduceStrategyType AllreducePlugin::selectImplementation( + size_t messageSize, int worldSize, nvinfer1::DataType type) noexcept +{ + bool const isAuto = (mStrategy == AllReduceStrategyType::AUTO); + + bool const forceDeterministic = common::getEnvForceDeterministicAllReduce(); + if (!mIsP2PSupported) + { + if (!isAuto) + { + TLLM_LOG_INFO("Since Peer to Peer not supported, fallback to AllReduceStrategy: NCCL_SYMMETRIC"); + } + else if (forceDeterministic) + { + TLLM_LOG_WARNING( + "Since Peer to Peer not supported, fallback to AllReduceStrategy: NCCL_SYMMETRIC. NCCL_SYMMETRIC might " + "produce " + "non-deterministic results."); + } + return AllReduceStrategyType::NCCL_SYMMETRIC; + } + + if (isAuto && !mIsNVLINKSupported && !forceDeterministic) + { + return AllReduceStrategyType::NCCL_SYMMETRIC; + } + + auto const maxWorkspaceSize = utils::customAllReduceUtils::getMaxRequiredWorkspaceSize(worldSize); + + AllReduceStrategyType strat = AllReduceStrategyType::NCCL_SYMMETRIC; + auto const messageSizeBytes = messageSize * common::getDTypeSize(type); + + if (messageSizeBytes <= maxWorkspaceSize) + { + // In some instances, the two-shot strategy has exhibited significant performance issues. + // As a temporary measure, we have disabled the two-shot strategy. + // TODO: remove this WAR after https://nvbugspro.nvidia.com/bug/4718747 is fixed. + if (!isAuto) + { + strat = mStrategy; + } + else if (forceDeterministic) + { + strat = AllReduceStrategyType::ONESHOT; + } + else if (worldSize <= 2) + { + strat = AllReduceStrategyType::ONESHOT; + } + else if (worldSize <= 4) + { + if (messageSizeBytes < 1 * 1000 * 1000) + { + strat = AllReduceStrategyType::ONESHOT; + } + else + { + strat = AllReduceStrategyType::NCCL_SYMMETRIC; + } + } + else + { + if (messageSizeBytes < 500 * 1000) + { + strat = AllReduceStrategyType::ONESHOT; + } + else + { + strat = AllReduceStrategyType::NCCL_SYMMETRIC; + } + } + + if (!kernels::configurationSupported(strat, messageSize, worldSize, type)) + { + if (!isAuto) + { + TLLM_LOG_WARNING("Since not aligned, fallback to AllReduceStrategy: NCCL_SYMMETRIC"); + } + else if (forceDeterministic) + { + TLLM_LOG_WARNING( + "Since not aligned, fallback to AllReduceStrategy: NCCL_SYMMETRIC. NCCL_SYMMETRIC might produce " + "non-deterministic results."); + } + strat = AllReduceStrategyType::NCCL_SYMMETRIC; + } + } + else + { + if (!isAuto) + { + TLLM_LOG_WARNING("Since messageSize > maxWorkspace, fallback to AllReduceStrategy: NCCL_SYMMETRIC"); + } + else if (forceDeterministic) + { + TLLM_LOG_WARNING( + "Since messageSize > maxWorkspace, fallback to AllReduceStrategy: NCCL_SYMMETRIC. NCCL_SYMMETRIC might " + "produce " + "non-deterministic results."); + } + strat = AllReduceStrategyType::NCCL_SYMMETRIC; + } + + return strat; +} + +int AllreducePlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + size_t size = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims; ++i) + { + size *= inputDesc[0].dims.d[i]; + } + + kernels::AllReduceStrategyType runtimeStrategy; + + static char* forceNcclAllReduceStrategyChar = std::getenv("FORCE_NCCL_ALL_REDUCE_STRATEGY"); + bool forceNcclAllReduceStrategy = (forceNcclAllReduceStrategyChar != nullptr); + if (forceNcclAllReduceStrategy || mStrategy == AllReduceStrategyType::NCCL) + { + runtimeStrategy = AllReduceStrategyType::NCCL; + } + else if (mStrategy == AllReduceStrategyType::NCCL_SYMMETRIC) + { + runtimeStrategy = AllReduceStrategyType::NCCL_SYMMETRIC; + } + else if (mStrategy == AllReduceStrategyType::UB) + { + runtimeStrategy = AllReduceStrategyType::UB; + } + else + { + runtimeStrategy = selectImplementation(size, mGroup.size(), mType); + } + + // Log runtime strategy + auto const rank = COMM_SESSION.getRank(); + switch (runtimeStrategy) + { + case AllReduceStrategyType::NCCL: + { + TLLM_LOG_DEBUG("AllReducePlugin strategy for rank %d: NCCL", rank); + break; + } + case AllReduceStrategyType::NCCL_SYMMETRIC: + { + TLLM_LOG_DEBUG("AllReducePlugin strategy for rank %d: NCCL_SYMMETRIC", rank); + break; + } + case AllReduceStrategyType::ONESHOT: + { + TLLM_LOG_DEBUG("AllReducePlugin strategy for rank %d: ONESHOT", rank); + break; + } + case AllReduceStrategyType::TWOSHOT: + { + TLLM_LOG_DEBUG("AllReducePlugin strategy for rank %d: TWOSHOT", rank); + break; + } + case AllReduceStrategyType::UB: + { + TLLM_LOG_DEBUG("AllReducePlugin strategy for rank %d: UB", rank); + break; + } + default: break; + } + + if (runtimeStrategy == AllReduceStrategyType::NCCL || runtimeStrategy == AllReduceStrategyType::NCCL_SYMMETRIC) + { + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM || mOp == AllReduceFusionOp::RESIDUAL_RMS_PREPOST_NORM) + { + NCCLCHECK(ncclAllReduce(inputs[0], outputs[1], size, (*getDtypeMap())[mType], ncclSum, *mNcclComm, stream)); + tensorrt_llm::kernels::AllReduceParams params; + int fusion_ptr_idx = 0; + if (mStrategy == AllReduceStrategyType::NCCL || mStrategy == AllReduceStrategyType::NCCL_SYMMETRIC) + { + fusion_ptr_idx = 1; + } + else + { + fusion_ptr_idx = 2; + } + params.fusion_params.bias_buffer = mBias ? inputs[fusion_ptr_idx++] : nullptr; + params.fusion_params.residual_buffer = inputs[fusion_ptr_idx++]; + params.fusion_params.weight_buffer = mAffine ? inputs[fusion_ptr_idx++] : nullptr; + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_PREPOST_NORM) + { + params.fusion_params.weight_buffer_pre_residual_norm = mAffine ? inputs[fusion_ptr_idx++] : nullptr; + } + params.local_output_buffer_ptr = outputs[0]; + params.elts_total = size; + params.fusion_params.hidden_size = inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]; + params.fusion_params.eps = mEps; + params.fusion_params.intermediate_buffer = outputs[1]; + TLLM_LOG_DEBUG("residualRmsNorm called"); + tensorrt_llm::kernels::residualRmsNorm(params, mType, stream, mOp); + } + else + { + NCCLCHECK(ncclAllReduce(inputs[0], outputs[0], size, (*getDtypeMap())[mType], ncclSum, *mNcclComm, stream)); + } + } + else if (runtimeStrategy == AllReduceStrategyType::UB) + { + TLLM_CHECK(!mBias); + + size_t dtype_size = tensorrt_llm::common::getDTypeSize(mType); + int hidden_size = inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]; + + TLLM_CHECK_WITH_INFO(tensorrt_llm::runtime::ub::ub_is_initialized(), "UserBuffer has not been initialized!"); + auto ub_buffer0 = tensorrt_llm::runtime::ub::ub_get(0); + auto ub_buffer1 = tensorrt_llm::runtime::ub::ub_get(1); + TLLM_CHECK(inputs[0] == ub_buffer0.addr); + auto ub_comm = tensorrt_llm::runtime::ub::ub_comm(); + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM_QUANT_FP8) + { + TLLM_CHECK(mAffine); + TLLM_CHECK(mScale); + TLLM_CHECK(outputs[0] == ub_buffer1.addr); + void* residual = const_cast<void*>(inputs[1]); + void* gamma = const_cast<void*>(inputs[2]); + float* scale = const_cast<float*>(reinterpret_cast<float const*>(inputs[3])); + tensorrt_llm::kernels::ub::allreduce2_userbuff_inplace_rmsnorm_quant_launcher(ub_buffer0.handle, 0, + ub_buffer1.handle, 0, size, hidden_size, nullptr, gamma, mEps, scale, residual, outputs[1], mType, + ub_comm, stream); + } + else if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM_QUANT_NVFP4) + { + auto ub_buffer2 = tensorrt_llm::runtime::ub::ub_get(2); + TLLM_CHECK(mAffine); + TLLM_CHECK(mScale); + TLLM_CHECK(outputs[0] == ub_buffer1.addr); + TLLM_CHECK(outputs[2] == ub_buffer2.addr); + void* residual = const_cast<void*>(inputs[1]); + void* gamma = const_cast<void*>(inputs[2]); + float* scale = const_cast<float*>(reinterpret_cast<float const*>(inputs[3])); + tensorrt_llm::kernels::ub::allreduce2_userbuff_inplace_rmsnorm_quant_fp4_launcher(ub_buffer0.handle, 0, + ub_buffer1.handle, 0, ub_buffer2.handle, 0, size, hidden_size, nullptr, gamma, mEps, scale, residual, + outputs[1], mType, ub_comm, stream); + } + else if (mOp == AllReduceFusionOp::LAST_PROCESS_FOR_UB) + { + TLLM_CHECK(outputs[1] == ub_buffer1.addr); + void* residual = const_cast<void*>(inputs[1]); + tensorrt_llm::kernels::ub::allreduce2_userbuff_inplace_launcher( + ub_buffer0.handle, 0, size, mType, ub_comm, stream); + tensorrt_llm::kernels::ub::allgather2_userbuff_residual_launcher( + ub_buffer1.handle, 0, size, hidden_size, residual, mType, ub_comm, stream); + TLLM_CUDA_CHECK( + cudaMemcpyAsync(outputs[0], ub_buffer0.addr, size * dtype_size, cudaMemcpyDeviceToDevice, stream)); + } + else if (mOp == AllReduceFusionOp::NONE) + { + tensorrt_llm::kernels::ub::allreduce2_userbuff_inplace_launcher( + ub_buffer0.handle, 0, size, mType, ub_comm, stream); + TLLM_CUDA_CHECK( + cudaMemcpyAsync(outputs[0], ub_buffer0.addr, size * dtype_size, cudaMemcpyDeviceToDevice, stream)); + } + else + { + TLLM_CHECK_WITH_INFO(false, "Unsupported UB allreduce fusion op"); + } + } + else + { + auto const tpSize = mGroup.size(); + int tpRank = 0; + for (auto const& currentRank : mGroup) + { + if (rank == currentRank) + break; + ++tpRank; + } + + int token_num = size / inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]; + int hidden_size = inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]; + auto params = tensorrt_llm::kernels::AllReduceParams::deserialize( + reinterpret_cast<int64_t*>(const_cast<void*>(inputs[1])), tpSize, tpRank, mType, token_num, hidden_size, + mOp); + + params.local_output_buffer_ptr = outputs[0]; + params.local_input_buffer_ptr = inputs[0]; + params.elts_total = size; + + int fusion_ptr_idx = 2; + params.fusion_params.bias_buffer = mBias ? inputs[fusion_ptr_idx++] : nullptr; + params.fusion_params.residual_buffer = inputs[fusion_ptr_idx++]; + params.fusion_params.weight_buffer = mAffine ? inputs[fusion_ptr_idx++] : nullptr; + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_PREPOST_NORM) + params.fusion_params.weight_buffer_pre_residual_norm = mAffine ? inputs[fusion_ptr_idx++] : nullptr; + params.fusion_params.hidden_size = hidden_size; + params.fusion_params.eps = mEps; + params.fusion_params.intermediate_buffer = outputs[1]; + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM) + { + for (size_t i = 0; i < tpSize; ++i) + { + params.fusion_params.lamport_peer_comm_buffer_ptrs[i] + = reinterpret_cast<void**>(const_cast<void*>(inputs[1]))[tpSize * 4 + i]; + params.fusion_params.lamport_peer_comm_buffer_ptrs[i + tensorrt_llm::kernels::MAX_RANKS_PER_NODE] + = reinterpret_cast<void**>(const_cast<void*>(inputs[1]))[tpSize * 5 + i]; + params.fusion_params.lamport_peer_comm_buffer_ptrs[i + tensorrt_llm::kernels::MAX_RANKS_PER_NODE * 2] + = reinterpret_cast<void**>(const_cast<void*>(inputs[1]))[tpSize * 6 + i]; + } + } + TLLM_LOG_DEBUG("customAllReduce called"); + tensorrt_llm::kernels::customAllReduce(params, mType, runtimeStrategy, mConfig, mOp, stream); + } + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType AllreducePlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index < getNbOutputs()); + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM_QUANT_NVFP4) + { + if (index == 0) + { + return nvinfer1::DataType::kFP4; + } + else if (index == 2) + { + return nvinfer1::DataType::kFP8; + } + } + if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM_QUANT_FP8) + { + if (index == 0) + { + return nvinfer1::DataType::kFP8; + } + } + return inputTypes[0]; +} + +// IPluginV2 Methods + +char const* AllreducePlugin::getPluginType() const noexcept +{ + return ALLREDUCE_PLUGIN_NAME; +} + +char const* AllreducePlugin::getPluginVersion() const noexcept +{ + return ALLREDUCE_PLUGIN_VERSION; +} + +int AllreducePlugin::getNbOutputs() const noexcept +{ + if (mOp == AllReduceFusionOp::NONE) + { + return 1; + } + else if (mOp == AllReduceFusionOp::RESIDUAL_RMS_NORM_QUANT_NVFP4) + { + return 3; + } + else + { + return 2; + } +} + +bool AllreducePlugin::isCustomAllReduceSupported(int ranks_per_node) const noexcept +{ + constexpr bool isCudaVersionSupported = +#if defined(CUDART_VERSION) && CUDART_VERSION >= 11020 + true; +#else + false; +#endif + + return isCudaVersionSupported && (ranks_per_node % 2 == 0) + && (static_cast<size_t>(ranks_per_node) <= kernels::MAX_RANKS_PER_NODE) && (ranks_per_node > 0); +} + +using tensorrt_llm::common::NvmlManager; +using tensorrt_llm::common::NVMLWrapper; + +std::set<int> getLocalGroup(std::set<int> const& group) +{ + auto const myRank = COMM_SESSION.getRank(); + auto const myLocalRank = LOCAL_COMM_SESSION.getRank(); + auto const localSize = LOCAL_COMM_SESSION.getSize(); + + std::vector<int32_t> ranks(localSize, 0); + std::vector<int32_t> localRanks(localSize, 0); + if (group.size() >= static_cast<size_t>(localSize)) + { + LOCAL_COMM_SESSION.allgather(&myRank, ranks.data(), 1, tensorrt_llm::mpi::MpiType::kINT32); + LOCAL_COMM_SESSION.allgather(&myLocalRank, localRanks.data(), 1, tensorrt_llm::mpi::MpiType::kINT32); + } + else + { + if (myRank == *group.begin()) + { + ranks.clear(); + int rank; + ranks.push_back(myRank); + for (auto it = std::next(std::begin(group), 1); it != group.end(); ++it) + { + COMM_SESSION.recvValue(rank, *it, MpiTag::kDefault); + ranks.push_back(rank); + } + for (auto it = std::next(std::begin(group), 1); it != group.end(); ++it) + { + COMM_SESSION.send(ranks.data(), localSize, tensorrt_llm::mpi::MpiType::kINT32, *it, MpiTag::kDefault); + } + + localRanks.clear(); + localRanks.push_back(myLocalRank); + for (auto it = std::next(std::begin(group), 1); it != group.end(); ++it) + { + COMM_SESSION.recvValue(rank, *it, MpiTag::kDefault); + localRanks.push_back(rank); + } + for (auto it = std::next(std::begin(group), 1); it != group.end(); ++it) + { + COMM_SESSION.send( + localRanks.data(), localSize, tensorrt_llm::mpi::MpiType::kINT32, *it, MpiTag::kDefault); + } + } + else + { + COMM_SESSION.sendValue(myRank, *group.begin(), MpiTag::kDefault); + COMM_SESSION.recv( + ranks.data(), localSize, tensorrt_llm::mpi::MpiType::kINT32, *group.begin(), MpiTag::kDefault); + + COMM_SESSION.sendValue(myLocalRank, *group.begin(), MpiTag::kDefault); + COMM_SESSION.recv( + localRanks.data(), localSize, tensorrt_llm::mpi::MpiType::kINT32, *group.begin(), MpiTag::kDefault); + } + } + + std::set<int> localGroup; + for (size_t i = 0; i < ranks.size(); ++i) + { + auto rank = ranks[i]; + if (group.find(rank) != group.end()) + { + localGroup.insert(localRanks[i]); + } + } + return localGroup; +} + +void AllreducePlugin::initGroupTopology() noexcept +{ + static std::map<std::set<int>, std::tuple<bool, bool>> cache; + if (cache.find(mGroup) != cache.end()) + { + auto [isNVLINKSupported, isP2PSupported] = cache[mGroup]; + mIsNVLINKSupported = isNVLINKSupported; + mIsP2PSupported = isP2PSupported; + return; + } + setGroupTopology(); + cache[mGroup] = {mIsNVLINKSupported, mIsP2PSupported}; +} + +void AllreducePlugin::setGroupTopology() noexcept +{ + auto const rank = COMM_SESSION.getRank(); + TLLM_LOG_INFO("Detecting local TP group for rank %d", rank); + std::set<int> localGroup = getLocalGroup(mGroup); + if (mGroup.size() != localGroup.size()) + { + mIsP2PSupported = false; + mIsNVLINKSupported = false; + TLLM_LOG_INFO("Found inter-node TP group for rank %d", rank); + return; + } + TLLM_LOG_INFO("TP group is intra-node for rank %d", rank); + + NvmlManager nvmlManager; + auto const& nvml = nvmlManager.sharedWrapper(); + std::unordered_set<int> visitedDevice; + mIsP2PSupported = true; + mIsNVLINKSupported = true; + + // Use cudaDeviceCanAccessPeer to determine whether p2p is supported, + // and use nvml to determine whether there are nvlink links between ranks. + for (int firstDeviceId : localGroup) + { + for (int secondDeviceId : localGroup) + { + if (firstDeviceId == secondDeviceId || visitedDevice.find(secondDeviceId) != visitedDevice.end()) + { + continue; + } + + int canAccessPeer = 0; + TLLM_CUDA_CHECK(cudaDeviceCanAccessPeer(&canAccessPeer, firstDeviceId, secondDeviceId)); + + if (!canAccessPeer) + { + mIsP2PSupported = false; + mIsNVLINKSupported = false; + + return; + } + + nvmlDevice_t firstDevice; + NVML_CHECK(nvml->nvmlDeviceGetHandleByIndex(firstDeviceId, &firstDevice)); + + bool isNVLINK = false; + + for (unsigned int link = 0; link < NVML_NVLINK_MAX_LINKS; link++) + { + nvmlPciInfo_t remotePciInfo; + if (nvml->nvmlDeviceGetNvLinkRemotePciInfo(firstDevice, link, &remotePciInfo) != NVML_SUCCESS) + { + continue; + } + + nvmlDevice_t remoteDevice; + auto const result = nvml->nvmlDeviceGetHandleByPciBusId(remotePciInfo.busId, &remoteDevice); + + if (result == NVML_SUCCESS) + { + // Two GPUs are connected directly through nvlink + unsigned int remoteDeviceId; + NVML_CHECK(nvml->nvmlDeviceGetIndex(remoteDevice, &remoteDeviceId)); + + if (remoteDeviceId == static_cast<unsigned int>(secondDeviceId)) + { + isNVLINK = true; + } + } + else if (result == NVML_ERROR_NOT_FOUND) + { + // Maybe Two GPUs are connected via nvswitch, + // now remotePciInfo represents the pci information of nvswitch, + // determine whether nvlink is supported by whether two GPUs are connected to the same nvswitch. + nvmlDevice_t secondDevice; + NVML_CHECK(nvml->nvmlDeviceGetHandleByIndex(secondDeviceId, &secondDevice)); + + for (unsigned int secondLink = 0; secondLink < NVML_NVLINK_MAX_LINKS; secondLink++) + { + nvmlPciInfo_t secondRemotePciInfo; + if (nvml->nvmlDeviceGetNvLinkRemotePciInfo(secondDevice, secondLink, &secondRemotePciInfo) + != NVML_SUCCESS) + { + continue; + } + + if (strcmp(remotePciInfo.busId, secondRemotePciInfo.busId) == 0) + { + isNVLINK = true; + break; + } + } + } + else + { + NVML_CHECK(result); + } + + if (isNVLINK) + { + break; + } + } + + mIsNVLINKSupported &= isNVLINK; + } + visitedDevice.insert(firstDeviceId); + } +} + +int AllreducePlugin::initialize() noexcept +{ + if (isBuilding()) + { + return 0; + } + + TLLM_LOG_TRACE("%s start for rank %d", __PRETTY_FUNCTION__, COMM_SESSION.getRank()); + mNcclComm = getComm(mGroup); + if (mStrategy != AllReduceStrategyType::NCCL) + { + initGroupTopology(); + } + + TLLM_LOG_TRACE("%s stop for rank %d", __PRETTY_FUNCTION__, COMM_SESSION.getRank()); + return 0; +} + +void AllreducePlugin::terminate() noexcept {} + +size_t AllreducePlugin::getSerializationSize() const noexcept +{ + return sizeof(int) * mGroup.size() + sizeof(mType) + sizeof(mStrategy) + sizeof(mConfig) + sizeof(mOp) + + sizeof(mEps) + sizeof(mAffine) + sizeof(mBias) + sizeof(mScale); +} + +void AllreducePlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mStrategy); + write(d, mConfig); + write(d, mOp); + write(d, mEps); + write(d, mAffine); + write(d, mBias); + write(d, mScale); + for (auto it = mGroup.begin(); it != mGroup.end(); ++it) + { + write(d, *it); + } + TLLM_CHECK(d == a + getSerializationSize()); +} + +void AllreducePlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +AllreducePluginCreator::AllreducePluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("group", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("strategy", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("config", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("fusion_op", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("counter", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("eps", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("affine", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("bias", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("scale", nullptr, PluginFieldType::kINT8)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* AllreducePluginCreator::getPluginName() const noexcept +{ + return ALLREDUCE_PLUGIN_NAME; +} + +char const* AllreducePluginCreator::getPluginVersion() const noexcept +{ + return ALLREDUCE_PLUGIN_VERSION; +} + +PluginFieldCollection const* AllreducePluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* AllreducePluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + std::set<int> group; + nvinfer1::DataType type{}; + AllReduceStrategyType strategy{}; + AllReduceStrategyConfig config{}; + AllReduceFusionOp fusion_op{}; + int32_t counter{}; + float eps{}; + int8_t affine{}; + int8_t bias{}; + int8_t scale{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "group")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + auto const* r = static_cast<int const*>(fields[i].data); + for (int j = 0; j < fields[i].length; ++j) + { + group.insert(*r); + ++r; + } + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "strategy")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + strategy = static_cast<AllReduceStrategyType>(*static_cast<int8_t const*>(fields[i].data)); + } + else if (!strcmp(attrName, "config")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + config = static_cast<AllReduceStrategyConfig>(*static_cast<int8_t const*>(fields[i].data)); + } + else if (!strcmp(attrName, "fusion_op")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + fusion_op = static_cast<AllReduceFusionOp>(*static_cast<int8_t const*>(fields[i].data)); + } + else if (!strcmp(attrName, "counter")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + counter = *static_cast<int32_t const*>(fields[i].data); + } + else if (!strcmp(attrName, "eps")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + eps = *static_cast<float const*>(fields[i].data); + } + else if (!strcmp(attrName, "affine")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + affine = *static_cast<int8_t const*>(fields[i].data); + } + else if (!strcmp(attrName, "bias")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + bias = *static_cast<int8_t const*>(fields[i].data); + } + else if (!strcmp(attrName, "scale")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + scale = *static_cast<int8_t const*>(fields[i].data); + } + } + try + { + auto* obj = new AllreducePlugin(group, type, strategy, config, fusion_op, counter, eps, affine, bias, scale); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* AllreducePluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call AllreducePlugin::destroy() + try + { + auto* obj = new AllreducePlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/allreducePlugin.h b/cpp/tensorrt_llm/plugins/ncclPlugin/allreducePlugin.h new file mode 100644 index 000000000000..881fbf3b89a5 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/allreducePlugin.h @@ -0,0 +1,114 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/kernels/customAllReduceKernels.h" +#include "tensorrt_llm/plugins/common/plugin.h" + +#include <cassert> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ +namespace tk = ::tensorrt_llm::kernels; + +class AllreducePlugin : public BasePlugin +{ +public: + AllreducePlugin(std::set<int> group, nvinfer1::DataType type, tk::AllReduceStrategyType strategy, + tk::AllReduceStrategyConfig config, tk::AllReduceFusionOp op, int32_t counter, float eps, int8_t affine, + int8_t bias, int8_t scale); + + AllreducePlugin(void const* data, size_t length); + + ~AllreducePlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + bool isCustomAllReduceSupported(int ranks_per_node) const noexcept; + void initGroupTopology() noexcept; + void setGroupTopology() noexcept; + tk::AllReduceStrategyType selectImplementation(size_t messageSize, int worldSize, nvinfer1::DataType type) noexcept; + void check() noexcept; + +private: + std::string const mLayerName; + std::set<int> mGroup; + bool mIsNVLINKSupported; + bool mIsP2PSupported; + nvinfer1::DataType mType; + tk::AllReduceStrategyType mStrategy; + tk::AllReduceStrategyConfig mConfig; + tk::AllReduceFusionOp mOp; + float mEps; + std::shared_ptr<ncclComm_t> mNcclComm; + int8_t mAffine; + int8_t mBias; + int8_t mScale; +}; + +class AllreducePluginCreator : public BaseCreator +{ +public: + AllreducePluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/recvPlugin.cpp b/cpp/tensorrt_llm/plugins/ncclPlugin/recvPlugin.cpp new file mode 100644 index 000000000000..089ed31175b2 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/recvPlugin.cpp @@ -0,0 +1,252 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "recvPlugin.h" + +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include <nccl.h> + +using namespace nvinfer1; +using tensorrt_llm::plugins::RecvPluginCreator; +using tensorrt_llm::plugins::RecvPlugin; +using tensorrt_llm::mpi::MpiTag; + +static char const* RECV_PLUGIN_VERSION{"1"}; +static char const* RECV_PLUGIN_NAME{"Recv"}; +PluginFieldCollection RecvPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> RecvPluginCreator::mPluginAttributes; + +RecvPlugin::RecvPlugin(int srcRank, nvinfer1::DataType type) + : mSrcRank(srcRank) + , mType(type) +{ +} + +// Parameterized constructor +RecvPlugin::RecvPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mType); + read(d, mSrcRank); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* RecvPlugin::clone() const noexcept +{ + auto* plugin = new RecvPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs RecvPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + return inputs[0]; +} + +bool RecvPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); +} + +void RecvPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t RecvPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int RecvPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + size_t size = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims; ++i) + { + size *= inputDesc[0].dims.d[i]; + } + TLLM_LOG_DEBUG("start ncclRecv with size %d", size); + NCCLCHECK(ncclRecv(outputs[0], size, (*getDtypeMap())[inputDesc[0].type], 0, mComm, stream)); + TLLM_LOG_DEBUG("end ncclRecv with size %d", size); + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType RecvPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + assert(index == 0); + return inputTypes[0]; +} + +// IPluginV2 Methods + +char const* RecvPlugin::getPluginType() const noexcept +{ + return RECV_PLUGIN_NAME; +} + +char const* RecvPlugin::getPluginVersion() const noexcept +{ + return RECV_PLUGIN_VERSION; +} + +int RecvPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int RecvPlugin::initialize() noexcept +{ + if (isBuilding()) + { + return 0; + } + ncclUniqueId id; + COMM_SESSION.recvValue(id, mSrcRank, MpiTag::kDefault); +// Need static connection initialization for accurate KV cache size estimation +#if defined(_WIN32) + if (getenv("NCCL_RUNTIME_CONNECT") == nullptr) + _putenv_s("NCCL_RUNTIME_CONNECT", "0"); +#else + setenv("NCCL_RUNTIME_CONNECT", "0", 0); +#endif // _WIN32 + NCCLCHECK(ncclCommInitRank(&mComm, 2, id, 1)); + return 0; +} + +void RecvPlugin::terminate() noexcept +{ + if (isBuilding()) + { + return; + } + NCCLCHECK(ncclCommDestroy(mComm)); +} + +size_t RecvPlugin::getSerializationSize() const noexcept +{ + return sizeof(mSrcRank) + sizeof(mType); +} + +void RecvPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mSrcRank); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void RecvPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +RecvPluginCreator::RecvPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("src_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* RecvPluginCreator::getPluginName() const noexcept +{ + return RECV_PLUGIN_NAME; +} + +char const* RecvPluginCreator::getPluginVersion() const noexcept +{ + return RECV_PLUGIN_VERSION; +} + +PluginFieldCollection const* RecvPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* RecvPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int srcRank{}; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "src_rank")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + srcRank = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + + try + { + auto* obj = new RecvPlugin(srcRank, type); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* RecvPluginCreator::deserializePlugin(char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call RecvPlugin::destroy() + try + { + auto* obj = new RecvPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/recvPlugin.h b/cpp/tensorrt_llm/plugins/ncclPlugin/recvPlugin.h new file mode 100644 index 000000000000..5c8eedfb5218 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/recvPlugin.h @@ -0,0 +1,90 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class RecvPlugin : public BasePlugin +{ +public: + RecvPlugin(int srcRank, nvinfer1::DataType type); + + RecvPlugin(void const* data, size_t length); + + ~RecvPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + ncclComm_t mComm; // TODO: Remove this + int mSrcRank; + nvinfer1::DataType mType; +}; + +class RecvPluginCreator : public BaseCreator +{ +public: + RecvPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/reduceScatterPlugin.cpp b/cpp/tensorrt_llm/plugins/ncclPlugin/reduceScatterPlugin.cpp new file mode 100644 index 000000000000..fe17c44fc418 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/reduceScatterPlugin.cpp @@ -0,0 +1,252 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "reduceScatterPlugin.h" + +#include <cassert> +#include <nccl.h> + +using namespace nvinfer1; +using tensorrt_llm::plugins::ReduceScatterPluginCreator; +using tensorrt_llm::plugins::ReduceScatterPlugin; + +static char const* REDUCE_SCATTER_PLUGIN_VERSION{"1"}; +static char const* REDUCE_SCATTER_PLUGIN_NAME{"ReduceScatter"}; +PluginFieldCollection ReduceScatterPluginCreator::mFC{}; +std::vector<PluginField> ReduceScatterPluginCreator::mPluginAttributes; + +ReduceScatterPlugin::ReduceScatterPlugin(std::set<int> group, nvinfer1::DataType type) + : mGroup(std::move(group)) + , mType(type) +{ +} + +// Parameterized constructor +ReduceScatterPlugin::ReduceScatterPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mType); + mGroup.clear(); + int groupItem = 0; + while (d != a + length) + { + read(d, groupItem); + mGroup.insert(groupItem); + } + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* ReduceScatterPlugin::clone() const noexcept +{ + auto* plugin = new ReduceScatterPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs ReduceScatterPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + auto output = inputs[0]; + output.d[0] + = exprBuilder.operation(DimensionOperation::kFLOOR_DIV, *output.d[0], *exprBuilder.constant(mGroup.size())); + return output; +} + +bool ReduceScatterPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); +} + +void ReduceScatterPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t ReduceScatterPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int ReduceScatterPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + size_t size = 1; + for (int i = 0; i < outputDesc[0].dims.nbDims; ++i) + { + size *= outputDesc[0].dims.d[i]; + } + + TLLM_CHECK_WITH_INFO(mNcclComm.get() != nullptr, "mNcclComm should be initialized before used"); + NCCLCHECK(ncclReduceScatter( + inputs[0], outputs[0], size, (*getDtypeMap())[inputDesc[0].type], ncclSum, *mNcclComm, stream)); + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType ReduceScatterPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + assert(index == 0); + return inputTypes[0]; +} + +// IPluginV2 Methods + +char const* ReduceScatterPlugin::getPluginType() const noexcept +{ + return REDUCE_SCATTER_PLUGIN_NAME; +} + +char const* ReduceScatterPlugin::getPluginVersion() const noexcept +{ + return REDUCE_SCATTER_PLUGIN_VERSION; +} + +int ReduceScatterPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int ReduceScatterPlugin::initialize() noexcept +{ + if (isBuilding()) + { + return 0; + } + mNcclComm = getComm(mGroup); + return 0; +} + +void ReduceScatterPlugin::terminate() noexcept {} + +size_t ReduceScatterPlugin::getSerializationSize() const noexcept +{ + return sizeof(int) * mGroup.size() + sizeof(mType); +} + +void ReduceScatterPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + for (auto it = mGroup.begin(); it != mGroup.end(); ++it) + { + write(d, *it); + } + TLLM_CHECK(d == a + getSerializationSize()); +} + +void ReduceScatterPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +ReduceScatterPluginCreator::ReduceScatterPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("group", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* ReduceScatterPluginCreator::getPluginName() const noexcept +{ + return REDUCE_SCATTER_PLUGIN_NAME; +} + +char const* ReduceScatterPluginCreator::getPluginVersion() const noexcept +{ + return REDUCE_SCATTER_PLUGIN_VERSION; +} + +PluginFieldCollection const* ReduceScatterPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* ReduceScatterPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + std::set<int> group; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "group")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + auto const* r = static_cast<int const*>(fields[i].data); + for (int j = 0; j < fields[i].length; ++j) + { + group.insert(*r); + ++r; + } + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + + try + { + auto* obj = new ReduceScatterPlugin(group, type); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* ReduceScatterPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call ReduceScatterPlugin::destroy() + try + { + auto* obj = new ReduceScatterPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/reduceScatterPlugin.h b/cpp/tensorrt_llm/plugins/ncclPlugin/reduceScatterPlugin.h new file mode 100644 index 000000000000..c630b57a2b98 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/reduceScatterPlugin.h @@ -0,0 +1,91 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class ReduceScatterPlugin : public BasePlugin +{ +public: + ReduceScatterPlugin(std::set<int> group, nvinfer1::DataType type); + + ReduceScatterPlugin(void const* data, size_t length); + + ~ReduceScatterPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + const std::string mLayerName; + std::set<int> mGroup; + nvinfer1::DataType mType; + std::shared_ptr<ncclComm_t> mNcclComm; +}; + +class ReduceScatterPluginCreator : public BaseCreator +{ +public: + ReduceScatterPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/sendPlugin.cpp b/cpp/tensorrt_llm/plugins/ncclPlugin/sendPlugin.cpp new file mode 100644 index 000000000000..81d66aa8211e --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/sendPlugin.cpp @@ -0,0 +1,255 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "sendPlugin.h" + +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include <cassert> +#include <nccl.h> + +using namespace nvinfer1; +using tensorrt_llm::plugins::SendPluginCreator; +using tensorrt_llm::plugins::SendPlugin; +using tensorrt_llm::mpi::MpiTag; + +static char const* SEND_PLUGIN_VERSION{"1"}; +static char const* SEND_PLUGIN_NAME{"Send"}; +PluginFieldCollection SendPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> SendPluginCreator::mPluginAttributes; + +SendPlugin::SendPlugin(int tgtRank, nvinfer1::DataType type) + : mTgtRank(tgtRank) + , mType(type) +{ +} + +// Parameterized constructor +SendPlugin::SendPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mType); + read(d, mTgtRank); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* SendPlugin::clone() const noexcept +{ + auto* plugin = new SendPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs SendPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + return inputs[0]; +} + +bool SendPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); +} + +void SendPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t SendPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int SendPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + size_t size = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims; ++i) + { + size *= inputDesc[0].dims.d[i]; + } + + TLLM_LOG_DEBUG("start ncclSend with size %d", size); + NCCLCHECK(ncclSend(inputs[0], size, (*getDtypeMap())[inputDesc[0].type], 1, mComm, stream)); + TLLM_LOG_DEBUG("end ncclSend with size %d", size); + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType SendPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + assert(index == 0); + return inputTypes[0]; +} + +// IPluginV2 Methods + +char const* SendPlugin::getPluginType() const noexcept +{ + return SEND_PLUGIN_NAME; +} + +char const* SendPlugin::getPluginVersion() const noexcept +{ + return SEND_PLUGIN_VERSION; +} + +int SendPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int SendPlugin::initialize() noexcept +{ + if (isBuilding()) + { + return 0; + } + + ncclUniqueId id; + ncclGetUniqueId(&id); + COMM_SESSION.sendValue(id, mTgtRank, MpiTag::kDefault); +// Need static connection initialization for accurate KV cache size estimation +#if defined(_WIN32) + if (getenv("NCCL_RUNTIME_CONNECT") == nullptr) + _putenv_s("NCCL_RUNTIME_CONNECT", "0"); +#else + setenv("NCCL_RUNTIME_CONNECT", "0", 0); +#endif // _WIN32 + NCCLCHECK(ncclCommInitRank(&mComm, 2, id, 0)); + return 0; +} + +void SendPlugin::terminate() noexcept +{ + if (isBuilding()) + { + return; + } + NCCLCHECK(ncclCommDestroy(mComm)); +} + +size_t SendPlugin::getSerializationSize() const noexcept +{ + return sizeof(mTgtRank) + sizeof(mType); +} + +void SendPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mTgtRank); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void SendPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +SendPluginCreator::SendPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("tgt_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* SendPluginCreator::getPluginName() const noexcept +{ + return SEND_PLUGIN_NAME; +} + +char const* SendPluginCreator::getPluginVersion() const noexcept +{ + return SEND_PLUGIN_VERSION; +} + +PluginFieldCollection const* SendPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* SendPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int tgtRank{}; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "tgt_rank")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + tgtRank = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + + try + { + auto* obj = new SendPlugin(tgtRank, type); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* SendPluginCreator::deserializePlugin(char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call SendPlugin::destroy() + try + { + auto* obj = new SendPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/ncclPlugin/sendPlugin.h b/cpp/tensorrt_llm/plugins/ncclPlugin/sendPlugin.h new file mode 100644 index 000000000000..0d36b0ebff28 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/ncclPlugin/sendPlugin.h @@ -0,0 +1,89 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class SendPlugin : public BasePlugin +{ +public: + SendPlugin(int tgtRank, nvinfer1::DataType type); + + SendPlugin(void const* data, size_t length); + + ~SendPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + ncclComm_t mComm; // TODO: Remove this + int mTgtRank; + nvinfer1::DataType mType; +}; + +class SendPluginCreator : public BaseCreator +{ +public: + SendPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/qserveGemmPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/qserveGemmPlugin/CMakeLists.txt new file mode 100755 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/qserveGemmPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/qserveGemmPlugin/qserveGemmPlugin.cpp b/cpp/tensorrt_llm/plugins/qserveGemmPlugin/qserveGemmPlugin.cpp new file mode 100644 index 000000000000..166f1cc32cbe --- /dev/null +++ b/cpp/tensorrt_llm/plugins/qserveGemmPlugin/qserveGemmPlugin.cpp @@ -0,0 +1,416 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "qserveGemmPlugin.h" +#include "tensorrt_llm/kernels/qserveGemm.h" +#include <cassert> +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::QServeGemmPluginCreator; +using tensorrt_llm::plugins::QServeGemmPlugin; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; +using namespace tensorrt_llm::kernels::qserve; + +static char const* QSERVE_GEMM_PLUGIN_VERSION{"1"}; +static char const* QSERVE_GEMM_PLUGIN_NAME{"QServeGemm"}; + +PluginFieldCollection QServeGemmPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> QServeGemmPluginCreator::mPluginAttributes; + +namespace tensorrt_llm::plugins +{ + +QServeGemmPlugin::QServeGemmPlugin( + // QuantMode quantMode, + nvinfer1::DataType dtype, int groupSize) +{ + init(dtype, groupSize); +} + +QServeGemmPlugin::QServeGemmPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + + nvinfer1::DataType type; + unsigned int quantMode; + int groupSize; + + read(d, quantMode); + read(d, type); + read(d, groupSize); + + read(d, mDims); + + // mQuantMode = QuantMode(quantMode); + + init(type, groupSize); + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +void QServeGemmPlugin::init(nvinfer1::DataType dtype, int groupSize) +{ + if (groupSize <= 0) + groupSize = -1; // Per-channel + mGroupSize = groupSize; + mType = dtype; + mRunner = std::make_shared<QServeGemmRunner>(); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* QServeGemmPlugin::clone() const noexcept +{ + auto* plugin = new QServeGemmPlugin(*this); + return plugin; +} + +nvinfer1::DimsExprs QServeGemmPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 6); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + TLLM_CHECK(nbDimsA >= 2); + DimsExprs ret; + ret.nbDims = nbDimsA; + for (int ii = 0; ii < nbDimsA - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + ret.d[nbDimsA - 1] = inputs[1].d[0]; + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool QServeGemmPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + if (mGroupSize != -1) + { // Per-group + switch (pos) + { + case 0: + // activation + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + case 1: + // uint4 weights packed in int8 + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + case 2: + // int8 weight s2_zeros + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + case 3: + // int8 weight s2_scales + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + case 4: + // fp16 weight s1_scales + return inOut[pos].type == nvinfer1::DataType::kHALF && inOut[pos].format == TensorFormat::kLINEAR; + case 5: + // fp16 activation scales + return inOut[pos].type == nvinfer1::DataType::kHALF && inOut[pos].format == TensorFormat::kLINEAR; + case 6: + // fp16 output activation + return inOut[pos].type == nvinfer1::DataType::kHALF && inOut[pos].format == TensorFormat::kLINEAR; + default: return false; + } + } + + else + { // Per-channel + switch (pos) + { + case 0: + // activation + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + case 1: + // uint4 weights packed in int8 + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + case 2: + // fp16 s1_scales + return inOut[pos].type == nvinfer1::DataType::kHALF && inOut[pos].format == TensorFormat::kLINEAR; + case 3: + // fp16 s1_szeros + return inOut[pos].type == nvinfer1::DataType::kHALF && inOut[pos].format == TensorFormat::kLINEAR; + case 4: + // fp16 act_sums + return inOut[pos].type == nvinfer1::DataType::kHALF && inOut[pos].format == TensorFormat::kLINEAR; + case 5: + // fp16 act_scales + return inOut[pos].type == nvinfer1::DataType::kHALF && inOut[pos].format == TensorFormat::kLINEAR; + case 6: + // fp16 output activation + return inOut[pos].type == nvinfer1::DataType::kHALF && inOut[pos].format == TensorFormat::kLINEAR; + default: return false; + } + } +} + +void QServeGemmPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + int const maxN = in[1].max.d[0]; + int const minK = in[0].min.d[in[0].min.nbDims - 1]; + int const minN = in[1].min.d[0]; + + TLLM_CHECK_WITH_INFO(minN == maxN, "Variable out channels is not allowed"); + TLLM_CHECK_WITH_INFO(minK == maxK, "Variable in channels is not allowed"); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, maxN, maxK}; + } + m_workspaceMaxSize = mRunner->getWorkspaceSize(maxM, maxN, maxK); +} + +size_t QServeGemmPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return m_workspaceMaxSize; +} + +int QServeGemmPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept +{ + // inputs + + // Per group: + // activation [M, K] int8_t Quantized sint8 activations + // weights [N, K/2] int8_t Quantized uint4 weights (packed as int8_t) + // s2_zeros [K/group_size, N] int8_t Level-2 sint8 scaled zeros of weights + // s2_scales [K/group_size, N] int8_t Level-2 sint8 scales of weights + // s1_scales [N] half Level-1 fp16 scales of weights + // act_scales [M] half Scales of activations + + // Per channel: + // activation [M, K] int8_t Quantized sint8 activations + // weights [N, K/2] int8_t Quantized uint4 weights (packed as int8_t) + // s1_scales [N] half Level-1 scales of weights + // s1_szeros [N] half Level-1 scaled zeros of weights + // act_sums [M] half Per-token sums of activations + // act_scales [M] half Scales of activations + + // outputs + // mat [M(*), N] half + + int64_t m64 = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m64 *= inputDesc[0].dims.d[ii]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[1].dims.d[0]); + int const k = TLLM_INT32_CAST(inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]); + + // TODO: Implement optimized kernels if (m <= 4) + + if (mGroupSize != -1) + { + ParamsPerGroup params = {reinterpret_cast<int8_t const*>(inputs[0]), // A + reinterpret_cast<int8_t const*>(inputs[1]), // B + reinterpret_cast<int8_t const*>(inputs[2]), // s2_zeros + reinterpret_cast<int8_t const*>(inputs[3]), // s2_scales + reinterpret_cast<half const*>(inputs[4]), // s1_scales + reinterpret_cast<half const*>(inputs[5]), // act_scales + reinterpret_cast<half*>(outputs[0]), // C + m, n, k}; + mRunner->gemmPerGroup(params, stream); + } + else + { + ParamsPerChannel params = {reinterpret_cast<int8_t const*>(inputs[0]), // A + reinterpret_cast<int8_t const*>(inputs[1]), // B + reinterpret_cast<half const*>(inputs[2]), // s1_scales + reinterpret_cast<half const*>(inputs[3]), // s1_szeros + reinterpret_cast<half const*>(inputs[4]), // act_sums + reinterpret_cast<half const*>(inputs[5]), // act_scales + reinterpret_cast<half*>(outputs[0]), // C + m, n, k}; + mRunner->gemmPerChannel(params, stream); + } + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType QServeGemmPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* QServeGemmPlugin::getPluginType() const noexcept +{ + return QSERVE_GEMM_PLUGIN_NAME; +} + +char const* QServeGemmPlugin::getPluginVersion() const noexcept +{ + return QSERVE_GEMM_PLUGIN_VERSION; +} + +int QServeGemmPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int QServeGemmPlugin::initialize() noexcept +{ + configGemm(); + return 0; +} + +void QServeGemmPlugin::terminate() noexcept {} + +size_t QServeGemmPlugin::getSerializationSize() const noexcept +{ + return sizeof(mQuantMode) + // QuantMode + sizeof(mType) + // dtype + sizeof(mGroupSize) + // GroupSize + sizeof(mDims); // Dimensions +} + +void QServeGemmPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mQuantMode.value()); + write(d, mType); + write(d, mGroupSize); + write(d, mDims); + + TLLM_CHECK(d == a + getSerializationSize()); +} + +void QServeGemmPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +void QServeGemmPlugin::configGemm() {} + +/////////////// + +QServeGemmPluginCreator::QServeGemmPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.push_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.push_back(PluginField("group_size", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* QServeGemmPluginCreator::getPluginName() const noexcept +{ + return QSERVE_GEMM_PLUGIN_NAME; +} + +char const* QServeGemmPluginCreator::getPluginVersion() const noexcept +{ + return QSERVE_GEMM_PLUGIN_VERSION; +} + +PluginFieldCollection const* QServeGemmPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* QServeGemmPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + // We do not use any fields for now. + + PluginField const* fields = fc->fields; + + // bool perTokenScaling, perChannelScaling; + DataType dtype{}; + int group_size = -1; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dtype = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + // Only supports fp16 for now. + assert(dtype == nvinfer1::DataType::kHALF); + } + else if (!strcmp(attrName, "group_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + group_size = *static_cast<int const*>(fields[i].data); + // Currently only support per-channel or g128. + assert(group_size == -1 || group_size == 128); + } + } + try + { + // QServeGemmPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + // auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false); + // QuantMode quantMode = QuantMode::fromQuantAlgo("W4A8_QSERVE"); + auto* obj = new QServeGemmPlugin(dtype, group_size); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* QServeGemmPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call QServeGemmPlugin::destroy() + try + { + // Create plugin profiler with private tactics map which is read from the serialized engine + // auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true); + auto* obj = new QServeGemmPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/qserveGemmPlugin/qserveGemmPlugin.h b/cpp/tensorrt_llm/plugins/qserveGemmPlugin/qserveGemmPlugin.h new file mode 100644 index 000000000000..086460863c4f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/qserveGemmPlugin/qserveGemmPlugin.h @@ -0,0 +1,112 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <memory> +#include <string> +#include <tensorrt_llm/kernels/qserveGemm.h> + +namespace tensorrt_llm::plugins +{ + +using QServeGemmRunnerPtr = std::shared_ptr<tensorrt_llm::kernels::qserve::QServeGemmRunner>; + +class QServeGemmPlugin : public BasePlugin +{ +public: + // using PluginProfilerPtr = std::shared_ptr<QServeGemmPluginProfiler>; + + QServeGemmPlugin(void const* data, size_t length); + + QServeGemmPlugin(nvinfer1::DataType dtype, int groupSize); + + ~QServeGemmPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + void init(nvinfer1::DataType dtype, int groupSize); + + void configGemm(); + + std::string const mLayerName; + + QServeGemmRunnerPtr mRunner; + + tensorrt_llm::common::QuantMode mQuantMode; // Not used for now + GemmDims mDims{}; + + size_t m_workspaceMaxSize; + + // Only supports fp16 output for now. + nvinfer1::DataType mType; + + int mGroupSize; +}; + +class QServeGemmPluginCreator : public BaseCreator +{ +public: + QServeGemmPluginCreator(); + + QServeGemmPluginCreator(void const* data, size_t length); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/CMakeLists.txt new file mode 100755 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/quantizePerTokenPlugin.cpp b/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/quantizePerTokenPlugin.cpp new file mode 100644 index 000000000000..23d0b80390e3 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/quantizePerTokenPlugin.cpp @@ -0,0 +1,353 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "quantizePerTokenPlugin.h" +#include "tensorrt_llm/kernels/quantization.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels; +using tensorrt_llm::plugins::QuantizePerTokenPluginCreator; +using tensorrt_llm::plugins::QuantizePerTokenPlugin; + +static char const* QUANTIZE_PER_TOKEN_PLUGIN_VERSION{"1"}; +static char const* QUANTIZE_PER_TOKEN_PLUGIN_NAME{"QuantizePerToken"}; +PluginFieldCollection QuantizePerTokenPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> QuantizePerTokenPluginCreator::mPluginAttributes; + +QuantizePerTokenPlugin::QuantizePerTokenPlugin( + nvinfer1::DataType outputType, QuantMode quantMode, bool clampValEnabled, bool sumPerToken) + : mOutputType{outputType} + , mQuantMode{quantMode} + , mClampValEnabled{clampValEnabled} + , mSumPerToken{sumPerToken} +{ + TLLM_CHECK_WITH_INFO(mOutputType == nvinfer1::DataType::kINT8 || mOutputType == nvinfer1::DataType::kFP8, + "Only int8 or fp8 output type is allowed."); + // Check if the quant mode is valid. + TLLM_CHECK_WITH_INFO(mQuantMode.hasPerTokenScaling(), "The quant mode is not valid."); +} + +// Parameterized constructor +QuantizePerTokenPlugin::QuantizePerTokenPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mOutputType); + read(d, mQuantMode); + read(d, mClampValEnabled); + read(d, mSumPerToken); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* QuantizePerTokenPlugin::clone() const noexcept +{ + auto* plugin = new QuantizePerTokenPlugin(mOutputType, mQuantMode, mClampValEnabled, mSumPerToken); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs QuantizePerTokenPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs <= 2); + TLLM_CHECK(outputIndex <= 2); + if (outputIndex == 2) + { + // Per token sums. + TLLM_CHECK(mSumPerToken); + } + + if (outputIndex == 0) + { + // Quantized input + return inputs[0]; + } + + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int ii = 0; ii < ret.nbDims - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + ret.d[ret.nbDims - 1] = exprBuilder.constant(1); + // [M(*), 1] dynamic per token scales or sums + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool QuantizePerTokenPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + if (pos == 0) + { + // activation + return (inOut[pos].type == nvinfer1::DataType::kFLOAT || inOut[pos].type == nvinfer1::DataType::kHALF +#ifdef ENABLE_BF16 + || inOut[pos].type == nvinfer1::DataType::kBF16 +#endif + ) + && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (pos == 1 && mClampValEnabled) + { + // clamp_max_v + return inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (pos == 1 + int(mClampValEnabled)) + { + // quantized activation + return inOut[pos].type == mOutputType && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (pos == 2 + int(mClampValEnabled)) + { + // scales + return inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (pos == 3 + int(mClampValEnabled)) + { + TLLM_CHECK(mSumPerToken); + // per-token sums + return inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + } + + // Never should be here + assert(false); + return false; +} + +void QuantizePerTokenPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t QuantizePerTokenPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +template <typename T, typename QuantT> +void QuantizePerTokenPlugin::dispatchDataType(void* output, void const* input, void const* clampValPtr, void* scalePtr, + void* sumPtr, int dim0, int dim1, cudaStream_t stream) noexcept +{ + // inputs + // activation [dim0(*), dim1] + // clamp_value [2], contains min val, and max val (optional) + // outputs + // quant [dim0(*), dim1] + // scale_tokens [dim0(*), 1] + + invokePerTokenQuantization(reinterpret_cast<QuantT*>(output), reinterpret_cast<T const*>(input), dim0, dim1, + reinterpret_cast<float const*>(clampValPtr), reinterpret_cast<float*>(scalePtr), + reinterpret_cast<float*>(sumPtr), mQuantMode, stream); +} + +int QuantizePerTokenPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // activation [M(*), K] + // clamp_value [2], contains min val, and max val (optional) + // outputs + // quant [M(*), K] Quantized activations. + // scale_tokens [M(*), 1] Per-token scales. + // token_sums [M(*), 1] (Optional) Per-token sums of all the channels (before quantization). + + int64_t m = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m *= inputDesc[0].dims.d[ii]; + } + int64_t const k = inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]; + + void const* clampValPtr = mClampValEnabled ? inputs[1] : nullptr; + void* sumPtr = mSumPerToken ? outputs[2] : nullptr; + + if (inputDesc[0].type == DataType::kFLOAT && mOutputType == DataType::kINT8) + { + dispatchDataType<float, int8_t>(outputs[0], inputs[0], clampValPtr, outputs[1], sumPtr, m, k, stream); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kFLOAT && mOutputType == DataType::kFP8) + { + dispatchDataType<float, __nv_fp8_e4m3>(outputs[0], inputs[0], clampValPtr, outputs[1], sumPtr, m, k, stream); + } +#endif // ENABLE_FP8 + else if (inputDesc[0].type == DataType::kHALF && mOutputType == DataType::kINT8) + { + dispatchDataType<half, int8_t>(outputs[0], inputs[0], clampValPtr, outputs[1], sumPtr, m, k, stream); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kHALF && mOutputType == DataType::kFP8) + { + dispatchDataType<half, __nv_fp8_e4m3>(outputs[0], inputs[0], clampValPtr, outputs[1], sumPtr, m, k, stream); + } +#endif // ENABLE_FP8 +#ifdef ENABLE_BF16 + else if (inputDesc[0].type == DataType::kBF16 && mOutputType == DataType::kINT8) + { + dispatchDataType<__nv_bfloat16, int8_t>(outputs[0], inputs[0], clampValPtr, outputs[1], sumPtr, m, k, stream); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kBF16 && mOutputType == DataType::kFP8) + { + dispatchDataType<__nv_bfloat16, __nv_fp8_e4m3>( + outputs[0], inputs[0], clampValPtr, outputs[1], sumPtr, m, k, stream); + } +#endif // ENABLE_FP8 +#endif // ENABLE_BF16 + sync_check_cuda_error(stream); + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType QuantizePerTokenPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(nbInputs >= 1); + TLLM_CHECK(index <= 2); + if (index == 2) + { + // Per token sums. + TLLM_CHECK(mSumPerToken); + } + return index == 0 ? mOutputType : nvinfer1::DataType::kFLOAT; +} + +// IPluginV2 Methods + +char const* QuantizePerTokenPlugin::getPluginType() const noexcept +{ + return QUANTIZE_PER_TOKEN_PLUGIN_NAME; +} + +char const* QuantizePerTokenPlugin::getPluginVersion() const noexcept +{ + return QUANTIZE_PER_TOKEN_PLUGIN_VERSION; +} + +int QuantizePerTokenPlugin::getNbOutputs() const noexcept +{ + return 2 + static_cast<int>(mSumPerToken); +} + +int QuantizePerTokenPlugin::initialize() noexcept +{ + return 0; +} + +void QuantizePerTokenPlugin::terminate() noexcept {} + +size_t QuantizePerTokenPlugin::getSerializationSize() const noexcept +{ + return sizeof(mOutputType) + sizeof(mQuantMode) + sizeof(mClampValEnabled) + sizeof(mSumPerToken); +} + +void QuantizePerTokenPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mOutputType); + write(d, mQuantMode); + write(d, mClampValEnabled); + write(d, mSumPerToken); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void QuantizePerTokenPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +QuantizePerTokenPluginCreator::QuantizePerTokenPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("quant_mode", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("clamp_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("sum_per_token", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* QuantizePerTokenPluginCreator::getPluginName() const noexcept +{ + return QUANTIZE_PER_TOKEN_PLUGIN_NAME; +} + +char const* QuantizePerTokenPluginCreator::getPluginVersion() const noexcept +{ + return QUANTIZE_PER_TOKEN_PLUGIN_VERSION; +} + +PluginFieldCollection const* QuantizePerTokenPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* QuantizePerTokenPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginFieldParser p{fc->nbFields, fc->fields}; + try + { + auto* obj = new QuantizePerTokenPlugin(static_cast<nvinfer1::DataType>(p.getScalar<int32_t>("type_id").value()), + QuantMode(p.getScalar<int32_t>("quant_mode").value()), + static_cast<bool>(p.getScalar<int8_t>("clamp_enabled").value()), + static_cast<bool>(p.getScalar<int32_t>("sum_per_token").value())); + + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* QuantizePerTokenPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call QuantizePerTokenPlugin::destroy() + try + { + auto* obj = new QuantizePerTokenPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/quantizePerTokenPlugin.h b/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/quantizePerTokenPlugin.h new file mode 100644 index 000000000000..47b218acfd28 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizePerTokenPlugin/quantizePerTokenPlugin.h @@ -0,0 +1,104 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class QuantizePerTokenPlugin : public BasePlugin +{ +public: + QuantizePerTokenPlugin(nvinfer1::DataType outputType, tensorrt_llm::common::QuantMode quantMode, + bool clampValEnabled, bool sumPerToken); + + QuantizePerTokenPlugin(void const* data, size_t length); + + ~QuantizePerTokenPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + template <typename T, typename QuantT> + void dispatchDataType(void* output, void const* input, void const* clampValPtr, void* scalePtr, void* sumPtr, + int dim0, int dim1, cudaStream_t stream) noexcept; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + std::string const mLayerName; + // The quantized output data type. + nvinfer1::DataType mOutputType; + // The quantization mode. + tensorrt_llm::common::QuantMode mQuantMode; + // Do we clamp the input tensor ? + bool mClampValEnabled; + // Do we output the per-token sum? + bool mSumPerToken; +}; + +class QuantizePerTokenPluginCreator : public BaseCreator +{ +public: + QuantizePerTokenPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/CMakeLists.txt new file mode 100755 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/quantizeTensorPlugin.cpp b/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/quantizeTensorPlugin.cpp new file mode 100644 index 000000000000..cacb32b809bf --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/quantizeTensorPlugin.cpp @@ -0,0 +1,250 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "quantizeTensorPlugin.h" +#include "tensorrt_llm/kernels/quantization.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using tensorrt_llm::plugins::QuantizeTensorPluginCreator; +using tensorrt_llm::plugins::QuantizeTensorPlugin; + +static char const* QUANTIZE_TENSOR_PLUGIN_VERSION{"1"}; +static char const* QUANTIZE_TENSOR_PLUGIN_NAME{"QuantizeTensor"}; +PluginFieldCollection QuantizeTensorPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> QuantizeTensorPluginCreator::mPluginAttributes; + +QuantizeTensorPlugin::QuantizeTensorPlugin() {} + +// Parameterized constructor +QuantizeTensorPlugin::QuantizeTensorPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* QuantizeTensorPlugin::clone() const noexcept +{ + return new QuantizeTensorPlugin(*this); +} + +nvinfer1::DimsExprs QuantizeTensorPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 2); + TLLM_CHECK(outputIndex < 1); + // Quantized input + return inputs[0]; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool QuantizeTensorPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + switch (pos) + { + case 0: + // activation + return (inOut[pos].type == nvinfer1::DataType::kFLOAT || inOut[pos].type == nvinfer1::DataType::kHALF +#ifdef ENABLE_BF16 + || inOut[pos].type == nvinfer1::DataType::kBF16 +#endif + ) + && inOut[pos].format == TensorFormat::kLINEAR; + case 1: + // scales + return inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + case 2: + // quantized activation + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + default: + // Never should be here + TLLM_CHECK(false); + return false; + } +} + +void QuantizeTensorPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t QuantizeTensorPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int QuantizeTensorPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // activation [M(*), K] + // scale [1, 1] + // outputs + // quant [M(*), K] + + int64_t numElts = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims; ++ii) + { + numElts *= inputDesc[0].dims.d[ii]; + } + + if (inputDesc[0].type == DataType::kFLOAT) + { + invokeQuantization<float>(reinterpret_cast<int8_t*>(outputs[0]), reinterpret_cast<float const*>(inputs[0]), + numElts, reinterpret_cast<float const*>(inputs[1]), stream, mProp.maxGridSize[0]); + } + else if (inputDesc[0].type == DataType::kHALF) + { + invokeQuantization<half>(reinterpret_cast<int8_t*>(outputs[0]), reinterpret_cast<half const*>(inputs[0]), + numElts, reinterpret_cast<float const*>(inputs[1]), stream, mProp.maxGridSize[0]); + } +#ifdef ENABLE_BF16 + else if (inputDesc[0].type == DataType::kBF16) + { + invokeQuantization<__nv_bfloat16>(reinterpret_cast<int8_t*>(outputs[0]), + reinterpret_cast<__nv_bfloat16 const*>(inputs[0]), numElts, reinterpret_cast<float const*>(inputs[1]), + stream, mProp.maxGridSize[0]); + } +#endif + sync_check_cuda_error(stream); + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType QuantizeTensorPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(nbInputs == 2); + TLLM_CHECK(index == 0); + return nvinfer1::DataType::kINT8; +} + +// IPluginV2 Methods + +char const* QuantizeTensorPlugin::getPluginType() const noexcept +{ + return QUANTIZE_TENSOR_PLUGIN_NAME; +} + +char const* QuantizeTensorPlugin::getPluginVersion() const noexcept +{ + return QUANTIZE_TENSOR_PLUGIN_VERSION; +} + +int QuantizeTensorPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int QuantizeTensorPlugin::initialize() noexcept +{ + int deviceId = 0; + tensorrt_llm::common::check_cuda_error(cudaGetDevice(&deviceId)); + tensorrt_llm::common::check_cuda_error(cudaGetDeviceProperties(&mProp, deviceId)); + return 0; +} + +void QuantizeTensorPlugin::terminate() noexcept {} + +size_t QuantizeTensorPlugin::getSerializationSize() const noexcept +{ + return 0; +} + +void QuantizeTensorPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + TLLM_CHECK(d == a + getSerializationSize()); +} + +void QuantizeTensorPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +QuantizeTensorPluginCreator::QuantizeTensorPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* QuantizeTensorPluginCreator::getPluginName() const noexcept +{ + return QUANTIZE_TENSOR_PLUGIN_NAME; +} + +char const* QuantizeTensorPluginCreator::getPluginVersion() const noexcept +{ + return QUANTIZE_TENSOR_PLUGIN_VERSION; +} + +PluginFieldCollection const* QuantizeTensorPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* QuantizeTensorPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + try + { + auto* obj = new QuantizeTensorPlugin(); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* QuantizeTensorPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call QuantizeTensorPlugin::destroy() + try + { + auto* obj = new QuantizeTensorPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/quantizeTensorPlugin.h b/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/quantizeTensorPlugin.h new file mode 100644 index 000000000000..6f1ce864ec35 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizeTensorPlugin/quantizeTensorPlugin.h @@ -0,0 +1,92 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class QuantizeTensorPlugin : public BasePlugin +{ +public: + QuantizeTensorPlugin(); + + QuantizeTensorPlugin(void const* data, size_t length); + + ~QuantizeTensorPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + const std::string mLayerName; + cudaDeviceProp mProp; +}; + +class QuantizeTensorPluginCreator : public BaseCreator +{ +public: + QuantizeTensorPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/quantizeToFP4Plugin.cpp b/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/quantizeToFP4Plugin.cpp new file mode 100644 index 000000000000..b5eaffeeda2a --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/quantizeToFP4Plugin.cpp @@ -0,0 +1,301 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "quantizeToFP4Plugin.h" +#include "pluginUtils.h" +#include "tensorrt_llm/kernels/quantization.h" +#include <NvInferRuntimeBase.h> + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::QuantizeToFP4PluginCreator; +using tensorrt_llm::plugins::QuantizeToFP4Plugin; + +constexpr nvinfer1::DataType FP4_DTYPE = nvinfer1::DataType::kFP4; +constexpr nvinfer1::DataType FP8_DTYPE = nvinfer1::DataType::kFP8; + +static char const* QUANT_FP4_PLUGIN_VERSION{"1"}; +static char const* QUANT_FP4_PLUGIN_NAME{"QuantizeToFP4"}; +PluginFieldCollection QuantizeToFP4PluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> QuantizeToFP4PluginCreator::mPluginAttributes; + +QuantizeToFP4Plugin::QuantizeToFP4Plugin(){}; + +// Parameterized constructor +QuantizeToFP4Plugin::QuantizeToFP4Plugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* QuantizeToFP4Plugin::clone() const noexcept +{ + auto* plugin = new QuantizeToFP4Plugin(); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs QuantizeToFP4Plugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + // Quantized output in FP4 datatype. + if (outputIndex == 0) + { + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int di = 0; di < ret.nbDims; ++di) + { + ret.d[di] = inputs[0].d[di]; + } + // // Div up by 16 as the storage type has 16 FP4 values per element. + // ret.d[ret.nbDims - 1] + // = exprBuilder.operation(DimensionOperation::kCEIL_DIV, *ret.d[ret.nbDims - 1], + // *exprBuilder.constant(16)); + return ret; + } + // Scaling Factors in FP8. + else if (outputIndex == 1) + { + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int di = 0; di < ret.nbDims; ++di) + { + ret.d[di] = inputs[0].d[di]; + } + // Sequence dimension or token dimension. + // Pad to multiple of 128. + auto dimM + = exprBuilder.operation(DimensionOperation::kCEIL_DIV, *ret.d[ret.nbDims - 2], *exprBuilder.constant(128)); + ret.d[ret.nbDims - 2] = exprBuilder.operation(DimensionOperation::kPROD, *dimM, *exprBuilder.constant(128)); + // Hidden size dimension. + // Div (rounding up) by 16 since 16 elements share one SF and SF padded to k%4==0. + ret.d[ret.nbDims - 1] + = exprBuilder.operation(DimensionOperation::kCEIL_DIV, *ret.d[ret.nbDims - 1], *exprBuilder.constant(16)); + return ret; + } + return DimsExprs{}; +} + +bool QuantizeToFP4Plugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + // half input + float global_sf + fp4 output (e2m1) + fp8 SF output. + int const totalPoses = 2 + 2; + TLLM_CHECK(0 <= pos && pos < totalPoses); + TLLM_CHECK(nbInputs == 2); + switch (pos) + { + case 0: + return (inOut[pos].type == nvinfer1::DataType::kHALF || inOut[pos].type == nvinfer1::DataType::kBF16 + || inOut[pos].type == nvinfer1::DataType::kFP8) + && (inOut[pos].format == TensorFormat::kLINEAR); + case 1: return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + case 2: return (inOut[pos].type == FP4_DTYPE) && (inOut[pos].format == TensorFormat::kLINEAR); + case 3: return (inOut[pos].type == FP8_DTYPE) && (inOut[pos].format == TensorFormat::kLINEAR); + default: break; + } + return false; +} + +void QuantizeToFP4Plugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t QuantizeToFP4Plugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +int QuantizeToFP4Plugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // input [M(*), N] half data type + // SF scale [1] float data type + // used to scale SF from input range to fp8 range (448.f / (MaxVal of input / 6.f)) + // outputs + // output [M(*), N] fp4 storage (E2M1) + // SF output [M, N / 16] fp8 storage (UE4M3) + + int64_t m64 = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims - 1; ++i) + { + m64 *= inputDesc[0].dims.d[i]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]); + + TLLM_CHECK_WITH_INFO(n % 16 == 0, "the N dimension must be multiple of 16."); + + float const* SFScale = static_cast<float const*>(inputs[1]); + int64_t* output = reinterpret_cast<int64_t*>(outputs[0]); + int32_t* SFoutput = reinterpret_cast<int32_t*>(outputs[1]); + + DataType inputDtype = inputDesc[0].type; + + switch (inputDtype) + { + case DataType::kHALF: + { + auto input = reinterpret_cast<half const*>(inputs[0]); + invokeFP4Quantization(1, m, n, input, SFScale, output, SFoutput, false, QuantizationSFLayout::SWIZZLED, + mMultiProcessorCount, stream); + break; + } + + case DataType::kBF16: + { + auto input = reinterpret_cast<__nv_bfloat16 const*>(inputs[0]); + invokeFP4Quantization(1, m, n, input, SFScale, output, SFoutput, false, QuantizationSFLayout::SWIZZLED, + mMultiProcessorCount, stream); + break; + } + + case DataType::kFP8: + { + auto input = reinterpret_cast<__nv_fp8_e4m3 const*>(inputs[0]); + invokeFP4Quantization(1, m, n, input, SFScale, output, SFoutput, false, QuantizationSFLayout::SWIZZLED, + mMultiProcessorCount, stream); + break; + } + + default: TLLM_LOG_ERROR("only half, bfloat16 and fp8 data type are supported."); break; + } + + // Use UE4M3 scales by default. + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType QuantizeToFP4Plugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + if (index == 0) + { + // Output 0 quantized output. + return FP4_DTYPE; + } + // Output 1 SF (scaling factors). + return FP8_DTYPE; +} + +// IPluginV2 Methods + +char const* QuantizeToFP4Plugin::getPluginType() const noexcept +{ + return QUANT_FP4_PLUGIN_NAME; +} + +char const* QuantizeToFP4Plugin::getPluginVersion() const noexcept +{ + return QUANT_FP4_PLUGIN_VERSION; +} + +int QuantizeToFP4Plugin::getNbOutputs() const noexcept +{ + return 2; +} + +int QuantizeToFP4Plugin::initialize() noexcept +{ + return 0; +} + +void QuantizeToFP4Plugin::terminate() noexcept {} + +size_t QuantizeToFP4Plugin::getSerializationSize() const noexcept +{ + return 0; +} + +void QuantizeToFP4Plugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + TLLM_CHECK(d == a + getSerializationSize()); +} + +void QuantizeToFP4Plugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +QuantizeToFP4PluginCreator::QuantizeToFP4PluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* QuantizeToFP4PluginCreator::getPluginName() const noexcept +{ + return QUANT_FP4_PLUGIN_NAME; +} + +char const* QuantizeToFP4PluginCreator::getPluginVersion() const noexcept +{ + return QUANT_FP4_PLUGIN_VERSION; +} + +PluginFieldCollection const* QuantizeToFP4PluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* QuantizeToFP4PluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + try + { + auto* obj = new QuantizeToFP4Plugin(); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* QuantizeToFP4PluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call QuantizeToFP4Plugin::destroy() + try + { + auto* obj = new QuantizeToFP4Plugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/quantizeToFP4Plugin.h b/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/quantizeToFP4Plugin.h new file mode 100644 index 000000000000..b584837a447a --- /dev/null +++ b/cpp/tensorrt_llm/plugins/quantizeToFP4Plugin/quantizeToFP4Plugin.h @@ -0,0 +1,90 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class QuantizeToFP4Plugin : public BasePlugin +{ +public: + QuantizeToFP4Plugin(); + + QuantizeToFP4Plugin(void const* data, size_t length); + + ~QuantizeToFP4Plugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + const std::string mLayerName; + int const mMultiProcessorCount = tensorrt_llm::common::getMultiProcessorCount(); +}; + +class QuantizeToFP4PluginCreator : public BaseCreator +{ +public: + QuantizeToFP4PluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/CMakeLists.txt new file mode 100755 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/rmsnormQuantizationPlugin.cpp b/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/rmsnormQuantizationPlugin.cpp new file mode 100644 index 000000000000..16d0bf2dc356 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/rmsnormQuantizationPlugin.cpp @@ -0,0 +1,452 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "rmsnormQuantizationPlugin.h" +#include "pluginUtils.h" +#include "tensorrt_llm/kernels/rmsnormKernels.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::RmsnormQuantizationPluginCreator; +using tensorrt_llm::plugins::RmsnormQuantizationPlugin; + +static char const* RMSNORM_QUANTIZATION_PLUGIN_VERSION{"1"}; +static char const* RMSNORM_QUANTIZATION_PLUGIN_NAME{"RmsnormQuantization"}; +PluginFieldCollection RmsnormQuantizationPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> RmsnormQuantizationPluginCreator::mPluginAttributes; + +RmsnormQuantizationPlugin::RmsnormQuantizationPlugin(float eps, bool dynamicActivationScaling, bool sumPerToken, + bool clampValEnabled, QuantMode quantMode, nvinfer1::DataType type, nvinfer1::DataType outputType) + : mEps(eps) + , mDynActScaling(dynamicActivationScaling) + , mType(type) + , mOutputType{outputType} + , mClampValEnabled{clampValEnabled} + , mQuantMode{quantMode} + , mSumPerToken(sumPerToken) +{ + TLLM_CHECK_WITH_INFO(mOutputType == nvinfer1::DataType::kINT8 || mOutputType == nvinfer1::DataType::kFP8, + "Only int8 or fp8 output type is allowed."); + // Check if the quant mode is valid. + TLLM_CHECK_WITH_INFO(mQuantMode.hasPerTokenScaling(), "The quant mode is not valid."); +} + +// Parameterized constructor +RmsnormQuantizationPlugin::RmsnormQuantizationPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mEps); + read(d, mDynActScaling); + read(d, mSumPerToken); + read(d, mClampValEnabled); + read(d, mQuantMode); + read(d, mType); + read(d, mOutputType); + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* RmsnormQuantizationPlugin::clone() const noexcept +{ + auto* plugin = new RmsnormQuantizationPlugin( + mEps, mDynActScaling, mSumPerToken, mClampValEnabled, mQuantMode, mType, mOutputType); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs RmsnormQuantizationPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + if (outputIndex == 0) + { + // Quantized output + return inputs[outputIndex]; + } + + // Dynamic scaling or per-token sum if enabled. + try + { + if (outputIndex == 1) + { + TLLM_CHECK(mDynActScaling); + } + else if (outputIndex == 2) + { + TLLM_CHECK(mSumPerToken); + } + else + { + TLLM_CHECK(false); + } + + DimsExprs ret; + ret.nbDims = inputs[0].nbDims; + for (int di = 0; di < ret.nbDims - 1; ++di) + { + ret.d[di] = inputs[0].d[di]; + } + ret.d[ret.nbDims - 1] = exprBuilder.constant(1); + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool RmsnormQuantizationPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + int const totalPoses + = 6 + static_cast<int>(mClampValEnabled) + static_cast<int>(mDynActScaling) + static_cast<int>(mSumPerToken); + TLLM_CHECK(0 <= pos && pos < totalPoses); + TLLM_CHECK(nbInputs == 4 + static_cast<int>(mClampValEnabled)); + if (pos < nbInputs) + { + if (pos < 3) + { + // activation, weight, bias + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == 3) + { + // scale + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == 4 && mClampValEnabled) + { + // clamp_max_v + return inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + } + } + else if (pos == 4 + int(mClampValEnabled)) + { + // Quantized output + return (inOut[pos].type == mOutputType) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == 5 + int(mClampValEnabled)) + { + // Dynamic scaling if enabled + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (pos == 6 + int(mClampValEnabled)) + { + // Per-token activation sum if enabled + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + + // Never should be here + TLLM_CHECK_WITH_INFO(false, "The input/output is not supported."); + return false; +} + +void RmsnormQuantizationPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t RmsnormQuantizationPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return 0; +} + +template <typename T, typename QuantT> +void RmsnormQuantizationPlugin::dispatchDataType(void* out, void const* input, void const* gamma, void const* beta, + float const eps, int const tokens, int const hidden_dim, cudaStream_t stream, void const* clampValPtr, + void const* scale, void* dynamic_scale, void* sum_per_token, void* normed_output_quant) noexcept +{ + // inputs + // activation [dim0(*), dim1] + // clamp_value [2], contains min val, and max val (optional) + // outputs + // quant [dim0(*), dim1] + // scale_tokens [dim0(*), 1] + + invokeGeneralRmsNorm(reinterpret_cast<T*>(out), reinterpret_cast<T const*>(input), + reinterpret_cast<T const*>(gamma), reinterpret_cast<T const*>(beta), eps, tokens, hidden_dim, mQuantMode, + stream, reinterpret_cast<float const*>(clampValPtr), reinterpret_cast<float const*>(scale), + reinterpret_cast<float*>(dynamic_scale), reinterpret_cast<float*>(sum_per_token), + reinterpret_cast<QuantT*>(normed_output_quant)); +} + +int RmsnormQuantizationPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // input [M(*), N] + // weight [N, ] + // bias [N, ] + // scale_to_int [1] + // clamp_value [2], contains min val, and max val (optional) + // outputs + // output [M(*), N] Normalized activations, potentially with quantization applied. + // dynamic_scaling [M(*), 1] (Optional) Per-token scales if quantization is enabled. + // token_sums [M(*), 1] (Optional) Per-token sums of all the channels (before quantization). + + int64_t m64 = 1; + for (int i = 0; i < inputDesc[0].dims.nbDims - 1; ++i) + { + m64 *= inputDesc[0].dims.d[i]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[1].dims.d[0]); + + void const* input = inputs[0]; + void const* weight = inputs[1]; + void const* bias = inputs[2]; + void const* scale = inputs[3]; + void const* clampValPtr = mClampValEnabled ? inputs[4] : nullptr; + void* output = outputs[0]; + void* dynamic_scale = mDynActScaling ? outputs[1] : nullptr; + void* sum_per_token = mSumPerToken ? outputs[2] : nullptr; + + if (inputDesc[0].type == DataType::kFLOAT && mOutputType == DataType::kINT8) + { + dispatchDataType<float, int8_t>( + nullptr, input, weight, bias, mEps, m, n, stream, clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kFLOAT && mOutputType == DataType::kFP8) + { + dispatchDataType<float, __nv_fp8_e4m3>( + nullptr, input, weight, bias, mEps, m, n, stream, clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#endif // ENABLE_FP8 + else if (inputDesc[0].type == DataType::kHALF && mOutputType == DataType::kINT8) + { + dispatchDataType<half, int8_t>( + nullptr, input, weight, bias, mEps, m, n, stream, clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kHALF && mOutputType == DataType::kFP8) + { + dispatchDataType<half, __nv_fp8_e4m3>( + nullptr, input, weight, bias, mEps, m, n, stream, clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#endif // ENABLE_FP8 +#ifdef ENABLE_BF16 + else if (inputDesc[0].type == DataType::kBF16 && mOutputType == DataType::kINT8) + { + dispatchDataType<__nv_bfloat16, int8_t>( + nullptr, input, weight, bias, mEps, m, n, stream, clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#ifdef ENABLE_FP8 + else if (inputDesc[0].type == DataType::kBF16 && mOutputType == DataType::kFP8) + { + dispatchDataType<__nv_bfloat16, __nv_fp8_e4m3>( + nullptr, input, weight, bias, mEps, m, n, stream, clampValPtr, scale, dynamic_scale, sum_per_token, output); + } +#endif // ENABLE_FP8 +#endif // ENABLE_BF16 + sync_check_cuda_error(stream); + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType RmsnormQuantizationPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + assert(index <= 2); + + if (index == 0) + { + // Output 0 quantized output of layer norm + return mOutputType; + } + if (index == 1) + { + assert(mDynActScaling); + // Output 1 dynamic act scaling + return nvinfer1::DataType::kFLOAT; + } + // index == 2 + { + assert(mDynActScaling && mSumPerToken); + // Output 2 per token sum + return nvinfer1::DataType::kFLOAT; + } +} + +// IPluginV2 Methods + +char const* RmsnormQuantizationPlugin::getPluginType() const noexcept +{ + return RMSNORM_QUANTIZATION_PLUGIN_NAME; +} + +char const* RmsnormQuantizationPlugin::getPluginVersion() const noexcept +{ + return RMSNORM_QUANTIZATION_PLUGIN_VERSION; +} + +int RmsnormQuantizationPlugin::getNbOutputs() const noexcept +{ + return 1 + static_cast<int>(mDynActScaling) + static_cast<int>(mSumPerToken); +} + +int RmsnormQuantizationPlugin::initialize() noexcept +{ + return 0; +} + +void RmsnormQuantizationPlugin::terminate() noexcept {} + +size_t RmsnormQuantizationPlugin::getSerializationSize() const noexcept +{ + return sizeof(mOutputType) + sizeof(mClampValEnabled) + sizeof(mEps) + sizeof(mDynActScaling) + sizeof(mSumPerToken) + + sizeof(mType) + sizeof(mQuantMode); +} + +void RmsnormQuantizationPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mEps); + write(d, mDynActScaling); + write(d, mSumPerToken); + write(d, mClampValEnabled); + write(d, mQuantMode); + write(d, mType); + write(d, mOutputType); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void RmsnormQuantizationPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +RmsnormQuantizationPluginCreator::RmsnormQuantizationPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("eps", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("dyn_act_scaling", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("sum_per_token", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("clamp_enabled", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("quant_mode", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("out_type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* RmsnormQuantizationPluginCreator::getPluginName() const noexcept +{ + return RMSNORM_QUANTIZATION_PLUGIN_NAME; +} + +char const* RmsnormQuantizationPluginCreator::getPluginVersion() const noexcept +{ + return RMSNORM_QUANTIZATION_PLUGIN_VERSION; +} + +PluginFieldCollection const* RmsnormQuantizationPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* RmsnormQuantizationPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + nvinfer1::DataType outputType{}; + QuantMode quantMode; + bool clampValEnabled = false; + float eps{}; + nvinfer1::DataType type{}; + bool dynamicActivationScaling{}; + bool sumPerToken{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "quant_mode")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + quantMode = QuantMode(*(static_cast<int32_t const*>(fields[i].data))); + } + else if (!strcmp(attrName, "out_type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + outputType = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "clamp_enabled")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + clampValEnabled = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "eps")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + eps = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "dyn_act_scaling")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dynamicActivationScaling = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "sum_per_token")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + sumPerToken = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + } + try + { + auto* obj = new RmsnormQuantizationPlugin( + eps, dynamicActivationScaling, sumPerToken, clampValEnabled, quantMode, type, outputType); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* RmsnormQuantizationPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call RmsnormQuantizationPlugin::destroy() + try + { + auto* obj = new RmsnormQuantizationPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/rmsnormQuantizationPlugin.h b/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/rmsnormQuantizationPlugin.h new file mode 100644 index 000000000000..762a9bb8de1b --- /dev/null +++ b/cpp/tensorrt_llm/plugins/rmsnormQuantizationPlugin/rmsnormQuantizationPlugin.h @@ -0,0 +1,108 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +class RmsnormQuantizationPlugin : public BasePlugin +{ +public: + RmsnormQuantizationPlugin(float eps, bool dynamicActivationScaling, bool sumPerToken, bool clampValEnabled, + tensorrt_llm::common::QuantMode quantMode, nvinfer1::DataType type, nvinfer1::DataType outputType); + + RmsnormQuantizationPlugin(void const* data, size_t length); + + ~RmsnormQuantizationPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + template <typename T, typename QuantT> + void dispatchDataType(void* out, void const* input, void const* gamma, void const* beta, float const eps, + int const tokens, int const hidden_dim, cudaStream_t stream, void const* clampValPtr, void const* scale, + void* dynamic_scale, void* normed_output_quant, void* act_sum) noexcept; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + float mEps; + bool mDynActScaling; + nvinfer1::DataType mType; + + std::string const mLayerName; + // The quantized output data type. + nvinfer1::DataType mOutputType; + // Do we clamp the input tensor ? + bool mClampValEnabled; + // The quantization mode. + tensorrt_llm::common::QuantMode mQuantMode; + // Should we output the sum of channels per-token? (Used by QServe GEMM) + bool mSumPerToken; +}; + +class RmsnormQuantizationPluginCreator : public BaseCreator +{ +public: + RmsnormQuantizationPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/selectiveScanPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/selectiveScanPlugin/CMakeLists.txt new file mode 100644 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/selectiveScanPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/selectiveScanPlugin/selectiveScanPlugin.cpp b/cpp/tensorrt_llm/plugins/selectiveScanPlugin/selectiveScanPlugin.cpp new file mode 100644 index 000000000000..3e60182f28c2 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/selectiveScanPlugin/selectiveScanPlugin.cpp @@ -0,0 +1,594 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "selectiveScanPlugin.h" +#include "tensorrt_llm/common/assert.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::SelectiveScanPluginCreator; +using tensorrt_llm::plugins::SelectiveScanPlugin; + +static char const* SELECTIVE_SCAN_PLUGIN_VERSION{"1"}; +static char const* SELECTIVE_SCAN_PLUGIN_NAME{"SelectiveScan"}; +PluginFieldCollection SelectiveScanPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> SelectiveScanPluginCreator::mPluginAttributes; + +SelectiveScanPlugin::SelectiveScanPlugin(int dim, int dstate, int dtRank, int nHeads, int nGroups, int chunkSize, + bool deltaSoftplus, nvinfer1::DataType type, bool removePadding, bool pagedState, bool zEnabled, bool isMamba2) + : mDim(dim) + , mDState(dstate) + , mDtRank(dtRank) + , mNHeads(nHeads) + , mNGroups(nGroups) + , mChunkSize(chunkSize) + , mDeltaSoftplus(deltaSoftplus) + , mType(type) + , mRemovePadding(removePadding) + , mPagedState(pagedState) + , mZEnabled(zEnabled) + , mIsMamba2(isMamba2) + , mDriver(tensorrt_llm::common::CUDADriverWrapper::getInstance()) +{ + TLLM_CHECK_WITH_INFO( + (mChunkSize == 256 || mChunkSize == 128) || (!mIsMamba2), "Only support CHUNK_SIZE 256 or 128"); + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF), + "Only support float, half, and bfloat16."); +} + +// Parameterized constructor +SelectiveScanPlugin::SelectiveScanPlugin(void const* data, size_t length) + : mDriver(tensorrt_llm::common::CUDADriverWrapper::getInstance()) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mDim); + read(d, mDState); + read(d, mDtRank); + read(d, mNHeads); + read(d, mNGroups); + read(d, mChunkSize); + read(d, mDeltaSoftplus); + read(d, mType); + read(d, mRemovePadding); + read(d, mPagedState); + read(d, mZEnabled); + read(d, mIsMamba2); + TLLM_CHECK(d == a + length); + TLLM_CHECK_WITH_INFO( + (mChunkSize == 256 || mChunkSize == 128) || (!mIsMamba2), "Only support CHUNK_SIZE 256 or 128"); + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF), + "Only support float, half, and bfloat16."); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* SelectiveScanPlugin::clone() const noexcept +{ + auto* plugin = new SelectiveScanPlugin(mDim, mDState, mDtRank, mNHeads, mNGroups, mChunkSize, mDeltaSoftplus, mType, + mRemovePadding, mPagedState, mZEnabled, mIsMamba2); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +// Outputs +// output_tensor: [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// state: [batch_size, dstate, dim] +nvinfer1::DimsExprs SelectiveScanPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + if (outputIndex == 0) + { + if (mIsMamba2) + { + auto ret = inputs[getInputTensorIdx()]; + ret.d[mRemovePadding ? 1 : 2] = exprBuilder.constant(mDim); + return ret; + } + else + { + return inputs[getInputTensorIdx()]; + } + } + return inputs[getStateIdx()]; +} + +bool SelectiveScanPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + if (pos == getHostRequestTypesIdx() || pos == getLastTokenIdsIdx() + || (mRemovePadding && pos == getHostContextLengthIdx()) || (mPagedState && pos == getSlotMappingIdx())) + { + return inOut[pos].type == nvinfer1::DataType::kINT32; + } + else if (pos == getAIdx() || pos == getDeltaBiasIdx() || pos == getDIdx()) + { + return (inOut[pos].type == nvinfer1::DataType::kFLOAT) && (inOut[pos].format == TensorFormat::kLINEAR); + } + else if (mPagedState && pos == getStateIdx()) + { + return inOut[pos].type == nvinfer1::DataType::kINT64; + } + else + { + return (inOut[pos].type == mType) && (inOut[pos].format == TensorFormat::kLINEAR); + } +} + +void SelectiveScanPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t SelectiveScanPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + if (!mIsMamba2) + return 0; + + int const NUM_BUFFERS = 6; + size_t workspaces[NUM_BUFFERS]; + + if (mRemovePadding) + { + int B = inputs[getLastTokenIdsIdx()].dims.d[0]; + int BxL = inputs[getInputTensorIdx()].dims.d[0]; // num_tokens + int H = mNHeads; + int P = mDim / H; + int G = mNGroups; + int N = mDState; + int Q = mChunkSize; + int BxC = (BxL + Q - 1) / Q + B; + + workspaces[0] = long(BxC) * H * N * P * 2; // g_mxOs_ + workspaces[1] = long(BxC) * H * N * P * 4; // g_mxSt_ in float + workspaces[2] = long(BxC) * H * Q * 4; // g_mxdc_ in float + workspaces[3] = long(BxC) * H * Q * 4; // g_mxdA_ in float + workspaces[4] = long(BxC) * G * Q * Q * 2; // g_mxCB_ + workspaces[5] = 1024; // TMA descs + } + else + { + int B = inputs[getInputTensorIdx()].dims.d[0]; + int L = inputs[getInputTensorIdx()].dims.d[1]; + int H = mNHeads; + int P = mDim / H; + int G = mNGroups; + int N = mDState; + int Q = mChunkSize; + int C = (L + Q - 1) / Q; + + workspaces[0] = long(B * C) * H * N * P * 2; // g_mxOs_ + workspaces[1] = long(B * C) * H * N * P * 4; // g_mxSt_ in float + workspaces[2] = long(B * C) * H * Q * 4; // g_mxdc_ in float + workspaces[3] = long(B * C) * H * Q * 4; // g_mxdA_ in float + workspaces[4] = long(B * C) * G * Q * Q * 2; // g_mxCB_ + workspaces[5] = 1024; // TMA descs + } + + return calculateTotalWorkspaceSize(workspaces, NUM_BUFFERS); +} + +void SelectiveScanPlugin::setSSMParams(SSMParamsBase& params, const size_t batch, const size_t dim, + const size_t maxSeqLen, const size_t numTokens, const size_t dstate, const size_t dtRank, const size_t nHeads, + const size_t nGroups, const size_t chunkSize, void* statePtr, void const* x, void const* delta, + void const* deltaBias, void const* A, void const* BC, void const* D, void const* z, void* osPtr, void* stPtr, + void* dcPtr, void* dAPtr, void* cbPtr, void* descPtr, int const* lastTokenIds, int const* slotMapping, void* out, + bool deltaSoftplus, bool removePadding) +{ + // Reset the parameters + memset(¶ms, 0, sizeof(params)); + + params.batch = batch; + params.dim = dim; + params.max_seqlen = maxSeqLen; + params.num_tokens = numTokens; + params.dstate = dstate; + params.dt_rank = dtRank; + params.nheads = nHeads; + params.ngroups = nGroups; + params.chunk_size = chunkSize; + + params.delta_softplus = deltaSoftplus; + params.remove_padding = removePadding; + params.is_mamba2 = mIsMamba2; + + // Set the pointers and strides. + params.u_ptr = const_cast<void*>(x); + params.delta_ptr = const_cast<void*>(delta); + params.A_ptr = const_cast<void*>(A); + params.BC_ptr = const_cast<void*>(BC); + params.D_ptr = const_cast<void*>(D); + params.delta_bias_ptr = const_cast<void*>(deltaBias); + params.out_ptr = out; + params.x_ptr = statePtr; + params.z_ptr = const_cast<void*>(z); + params.Os_ptr = osPtr; + params.St_ptr = stPtr; + params.dc_ptr = dcPtr; + params.dA_ptr = dAPtr; + params.CB_ptr = cbPtr; + params.desc_ptr = descPtr; + params.last_token_ids_ptr = lastTokenIds; + params.slot_mapping_ptr = slotMapping; +} + +template <typename T> +int SelectiveScanPlugin::enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) +{ + // inputs + // 0. input_tensor [batch_size, max_seq_len, dim] or [num_tokens, dim] + // 1. state mamba: [batch_size, dstate, dim] or host [1] containing only pointer for paged_state + // mamba2: [batch_size, nheads, dstate, dim] or host [1] containing only pointer for paged_state + // 2. delta, mamba: [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding + // mamba2: [batch_size, seq_len, nheads] or [num_tokens, nheads] for remove_input_padding + // 3. delta_bias, [dim] for mamba, [nheads] for mamba2 + // 4. A, [dstate, dim] for mamba, [nheads] for mamba2 + // 5. BC, mamba: [batch_size, seq_len, dstate * 2] or [num_tokens, dstate * 2] for remove_input_padding + // mamba2: [batch_size, seq_len, ngroups * dstate * 2] or [num_tokens, ngroups * dstate * 2] for + // remove_input_padding + // 6. D, [dim] for mamba, [nheads] for mamba2 + // 7. host_request_types [batch_size] int32. 0: context; 1: generation. + // 8. last_token_ids [batch_size] int32 + // 9. host_context_lengths [batch_size] int32, optional for remove_input_padding + // 10. state_slot_mapping [batch_size] int32, optional for paged state + // 11. z [batch_size, max_seq_len, dim] or [num_tokens, dim] + // outputs + // 0. output_tensor [batch_size, max_seq_len, dim] or [num_tokens, dim] + // 1. state, [batch_size, dstate, dim] for mamba, [batch_size, nheads, dstate, dim] for mamba2 + auto const batch_size = inputDesc[getHostRequestTypesIdx()].dims.d[0]; + int max_seq_len; + if (mRemovePadding) + { + int const* host_context_length = static_cast<int const*>(inputs[getHostContextLengthIdx()]); + max_seq_len = *std::max_element(host_context_length, host_context_length + batch_size); + } + else + { + max_seq_len = inputDesc[getInputTensorIdx()].dims.d[1]; + } + + // only support context or generation, not for both of them + RequestType const* reqTypes = static_cast<RequestType const*>(inputs[getHostRequestTypesIdx()]); + + SSMParamsBase ssm_params; + + int const* slotMapping = mPagedState ? static_cast<int const*>(inputs[getSlotMappingIdx()]) : nullptr; + void const* z = mZEnabled ? inputs[getZIdx()] : nullptr; + + void* statePtr = mPagedState ? *reinterpret_cast<void**>(const_cast<void*>(inputs[getStateIdx()])) : outputs[1]; + + // Workspace pointer shift + int8_t* workspace_byte_ptr = reinterpret_cast<int8_t*>(workspace); + size_t offset = 0; + + T* mxOs = nullptr; + float* mxSt = nullptr; + float* mxdc = nullptr; + float* mxdA = nullptr; + T* mxCB = nullptr; + void* descs = nullptr; + + if (!mIsMamba2 || reqTypes[0] == RequestType::kGENERATION) /* no workspace needed */ + ; + else if (mRemovePadding) + { + int B = inputDesc[getLastTokenIdsIdx()].dims.d[0]; + int BxL = inputDesc[getInputTensorIdx()].dims.d[0]; // num_tokens + int H = mNHeads; + int P = mDim / H; + int G = mNGroups; + int N = mDState; + int Q = mChunkSize; + int BxC = (BxL + Q - 1) / Q + B; + + mxOs = reinterpret_cast<T*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(BxC) * H * N * P * 2)); + mxSt = reinterpret_cast<float*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(BxC) * H * N * P * 4)); + mxdc = reinterpret_cast<float*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(BxC) * H * Q * 4)); + mxdA = reinterpret_cast<float*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(BxC) * H * Q * 4)); + mxCB = reinterpret_cast<T*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(BxC) * G * Q * Q * 2)); + descs = nextWorkspacePtr(workspace_byte_ptr, offset, 1024); + } + else + { + int B = inputDesc[getInputTensorIdx()].dims.d[0]; + int L = inputDesc[getInputTensorIdx()].dims.d[1]; + int H = mNHeads; + int P = mDim / H; + int G = mNGroups; + int N = mDState; + int Q = mChunkSize; + int C = (L + Q - 1) / Q; + + mxOs = reinterpret_cast<T*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(B * C) * H * N * P * 2)); + mxSt = reinterpret_cast<float*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(B * C) * H * N * P * 4)); + mxdc = reinterpret_cast<float*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(B * C) * H * Q * 4)); + mxdA = reinterpret_cast<float*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(B * C) * H * Q * 4)); + mxCB = reinterpret_cast<T*>(nextWorkspacePtr(workspace_byte_ptr, offset, long(B * C) * G * Q * Q * 2)); + descs = nextWorkspacePtr(workspace_byte_ptr, offset, 1024); + } + + int numTokens = inputDesc[getInputTensorIdx()].dims.d[0]; + if (!mRemovePadding) + numTokens *= inputDesc[getInputTensorIdx()].dims.d[1]; + + setSSMParams(ssm_params, batch_size, mDim, max_seq_len, numTokens, mDState, mDtRank, mNHeads, mNGroups, mChunkSize, + statePtr, inputs[getInputTensorIdx()], inputs[getDeltaIdx()], inputs[getDeltaBiasIdx()], inputs[getAIdx()], + inputs[getBCIdx()], inputs[getDIdx()], z, mxOs, mxSt, mxdc, mxdA, mxCB, descs, + static_cast<int const*>(inputs[getLastTokenIdsIdx()]), slotMapping, outputs[0], mDeltaSoftplus, mRemovePadding); + + if (reqTypes[0] == RequestType::kCONTEXT) + { + if (mIsMamba2) + { + invokeChunkScan<T, float>(ssm_params, stream, mDriver.get()); + } + else + { + invokeSelectiveScan<T, float>(ssm_params, stream); + } + } + else if (reqTypes[0] == RequestType::kGENERATION) + { + invokeSelectiveScanUpdate<T, float>(ssm_params, stream); + } + sync_check_cuda_error(stream); + return 0; +} + +int SelectiveScanPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + if (isBuilding()) + { + return 0; + } + if (mType == DataType::kHALF) + { + return enqueueImpl<half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kFLOAT) + { + return enqueueImpl<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16) + { + return enqueueImpl<__nv_bfloat16>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType SelectiveScanPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + if (index == 0) + { + return inputTypes[getInputTensorIdx()]; + } + else + { + return inputTypes[getStateIdx()]; + } +} + +// IPluginV2 Methods + +char const* SelectiveScanPlugin::getPluginType() const noexcept +{ + return SELECTIVE_SCAN_PLUGIN_NAME; +} + +char const* SelectiveScanPlugin::getPluginVersion() const noexcept +{ + return SELECTIVE_SCAN_PLUGIN_VERSION; +} + +int SelectiveScanPlugin::getNbOutputs() const noexcept +{ + return mPagedState ? 1 : 2; +} + +int SelectiveScanPlugin::initialize() noexcept +{ + return 0; +} + +void SelectiveScanPlugin::terminate() noexcept {} + +size_t SelectiveScanPlugin::getSerializationSize() const noexcept +{ + return sizeof(mDim) + sizeof(mDState) + sizeof(mDtRank) + sizeof(mNHeads) + sizeof(mNGroups) + sizeof(mChunkSize) + + sizeof(mDeltaSoftplus) + sizeof(mType) + sizeof(mRemovePadding) + sizeof(mPagedState) + sizeof(mZEnabled) + + sizeof(mIsMamba2); +} + +void SelectiveScanPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mDim); + write(d, mDState); + write(d, mDtRank); + write(d, mNHeads); + write(d, mNGroups); + write(d, mChunkSize); + write(d, mDeltaSoftplus); + write(d, mType); + write(d, mRemovePadding); + write(d, mPagedState); + write(d, mZEnabled); + write(d, mIsMamba2); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void SelectiveScanPlugin::destroy() noexcept +{ + delete this; +} + +/////////////// + +SelectiveScanPluginCreator::SelectiveScanPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("dim", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("dstate", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("dt_rank", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("nheads", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("ngroups", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("chunk_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("delta_softplus", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("remove_input_padding", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("paged_state", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("z_enabled", nullptr, PluginFieldType::kINT8)); + mPluginAttributes.emplace_back(PluginField("is_mamba2", nullptr, PluginFieldType::kINT8)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* SelectiveScanPluginCreator::getPluginName() const noexcept +{ + return SELECTIVE_SCAN_PLUGIN_NAME; +} + +char const* SelectiveScanPluginCreator::getPluginVersion() const noexcept +{ + return SELECTIVE_SCAN_PLUGIN_VERSION; +} + +PluginFieldCollection const* SelectiveScanPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* SelectiveScanPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int dim{}; + int dstate{}; + int dtRank{}; + int nHeads{}; + int nGroups{}; + int chunkSize{}; + bool deltaSoftplus{}; + bool removePadding{}; + bool pagedState{}; + bool zEnabled{}; + bool isMamab2{}; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "dim")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dim = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "dstate")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dstate = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "dt_rank")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + dtRank = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "nheads")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + nHeads = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "ngroups")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + nGroups = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "chunk_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + chunkSize = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "delta_softplus")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + deltaSoftplus = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "remove_input_padding")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + removePadding = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "paged_state")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + pagedState = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "z_enabled")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + zEnabled = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + else if (!strcmp(attrName, "is_mamba2")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT8); + isMamab2 = static_cast<bool>(*(static_cast<bool const*>(fields[i].data))); + } + } + try + { + auto* obj = new SelectiveScanPlugin(dim, dstate, dtRank, nHeads, nGroups, chunkSize, deltaSoftplus, type, + removePadding, pagedState, zEnabled, isMamab2); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* SelectiveScanPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call SelectiveScanPlugin::destroy() + try + { + auto* obj = new SelectiveScanPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/selectiveScanPlugin/selectiveScanPlugin.h b/cpp/tensorrt_llm/plugins/selectiveScanPlugin/selectiveScanPlugin.h new file mode 100644 index 000000000000..96cb86fc4cbb --- /dev/null +++ b/cpp/tensorrt_llm/plugins/selectiveScanPlugin/selectiveScanPlugin.h @@ -0,0 +1,218 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TRT_SELECTIVE_SCAN_PLUGIN_H +#define TRT_SELECTIVE_SCAN_PLUGIN_H +#include "tensorrt_llm/kernels/selectiveScan/selectiveScan.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> + +namespace tensorrt_llm::plugins +{ +// batch_size = num_ctx_requests or num_gen_requests +// num_ctx_requests = number of context requests (single sequence per request). +// num_gen_requests = number of generation requests (single sequences per request). +// can not support beam search + +// inputs +// 0. input_tensor [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 1. state, mamba: [batch_size, dstate, dim] or host [1] containing only pointer for paged_state +// mamba2: [batch_size, nheads, dstate, dim] or host [1] containing only pointer for paged_state +// 2. delta, mamba: [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// mamba2: [batch_size, seq_len, nheads] or [num_tokens, nheads] for remove_input_padding +// 3. delta_bias, [dim] for mamba, [nheads] for mamba2 +// 4. A, [dstate, dim] for mamba, [nheads] for mamba2 +// 5. BC, mamba: [batch_size, seq_len, dstate * 2] or [num_tokens, dstate * 2] for remove_input_padding +// mamba2: [batch_size, seq_len, ngroups * dstate * 2] or [num_tokens, ngroups * dstate * 2] for +// remove_input_padding +// 6. D, [dim] for mamba, [nheads] for mamba2 +// 7. host_request_types [batch_size] int32. 0: context; 1: generation; 2: none. +// 8. last_token_ids [batch_size] int32 +// 9. host_context_lengths [batch_size] int32, optional for remove_input_padding +// 10. state_slot_mapping [batch_size] int32, optional for paged state +// 11. z [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// outputs +// 0. output_tensor [batch_size, seq_len, dim] or [num_tokens, dim] for remove_input_padding +// 1. state, [batch_size, dstate, dim] for mamba, [batch_size, nheads, dstate, dim] for mamba2 + +class SelectiveScanPlugin : public BasePlugin +{ +public: + SelectiveScanPlugin(int dim, int dstate, int dtRank, int nHeads, int nGroups, int chunkSize, bool deltaSoftplus, + nvinfer1::DataType type, bool removePadding, bool pagedState, bool zEnabled, bool isMamba2); + + SelectiveScanPlugin(void const* data, size_t length); + + ~SelectiveScanPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + template <typename T> + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + + enum class RequestType : int32_t + { + kCONTEXT = 0, + kGENERATION = 1 + }; + +private: + using IndexType = std::int32_t; + + IndexType getInputTensorIdx() const + { + return 0; + }; + + IndexType getStateIdx() const + { + return 1; + }; + + IndexType getDeltaIdx() const + { + return 2; + }; + + IndexType getDeltaBiasIdx() const + { + return 3; + }; + + IndexType getAIdx() const + { + return 4; + }; + + IndexType getBCIdx() const + { + return 5; + }; + + IndexType getDIdx() const + { + return 6; + }; + + IndexType getHostRequestTypesIdx() const + { + return 7; + }; + + IndexType getLastTokenIdsIdx() const + { + return 8; + }; + + IndexType getHostContextLengthIdx() const + { + if (mRemovePadding) + return 9; + else + return 8; + }; + + IndexType getSlotMappingIdx() const + { + if (mPagedState) + return getHostContextLengthIdx() + 1; + else + return getHostContextLengthIdx(); + }; + + IndexType getZIdx() const + { + if (mZEnabled) + return getSlotMappingIdx() + 1; + else + return getSlotMappingIdx(); + }; + + void setSSMParams(tensorrt_llm::kernels::SSMParamsBase& params, + // sizes + const size_t batch, const size_t dim, const size_t maxSeqLen, const size_t numTokens, const size_t dstate, + const size_t dtRank, const size_t nHeads, const size_t nGroups, const size_t chunkSize, + // device pointers + void* statePtr, void const* x, void const* delta, void const* deltaBias, void const* A, void const* BC, + void const* D, void const* z, void* osPtr, void* stPtr, void* dcPtr, void* dAPtr, void* cbPtr, void* descs, + int const* lastTokenIds, int const* slotMapping, void* out, bool deltaSoftplus, bool removePadding); + +private: + int mDim; + int mDState; + int mDtRank; + int mNHeads; + int mNGroups; + int mChunkSize; + bool mDeltaSoftplus; + nvinfer1::DataType mType; + bool mRemovePadding = false; + bool mPagedState = false; + bool mZEnabled = true; + bool mIsMamba2 = false; + std::shared_ptr<tensorrt_llm::common::CUDADriverWrapper> mDriver; +}; + +class SelectiveScanPluginCreator : public BaseCreator +{ +public: + SelectiveScanPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins + +#endif // TRT_SELECTIVE_SCAN_PLUGIN_H diff --git a/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/CMakeLists.txt new file mode 100755 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/smoothQuantGemmPlugin.cpp b/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/smoothQuantGemmPlugin.cpp new file mode 100644 index 000000000000..718d8b7e830d --- /dev/null +++ b/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/smoothQuantGemmPlugin.cpp @@ -0,0 +1,431 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "smoothQuantGemmPlugin.h" +#include "tensorrt_llm/kernels/weightOnlyBatchedGemv/int8SQ.h" +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels::cutlass_kernels; +using tensorrt_llm::plugins::SmoothQuantGemmPluginCreator; +using tensorrt_llm::plugins::SmoothQuantGemmPlugin; +using tensorrt_llm::plugins::SmoothQuantGemmPluginProfiler; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +static char const* SQ_GEMM_PLUGIN_VERSION{"1"}; +static char const* SQ_GEMM_PLUGIN_NAME{"SmoothQuantGemm"}; +PluginFieldCollection SmoothQuantGemmPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> SmoothQuantGemmPluginCreator::mPluginAttributes; + +void SmoothQuantGemmPluginProfiler::runTactic(int m, int n, int k, SmoothQuantGemmPluginProfiler::Config const& tactic, + char* workspace, cudaStream_t const& stream) +{ + int8_t* aTmp = reinterpret_cast<int8_t*>(workspace); + int8_t* bTmp = nextWorkspacePtr(aTmp, m * k * sizeof(int8_t)); + void* cTmp = reinterpret_cast<void*>(nextWorkspacePtr(bTmp, n * k * sizeof(int8_t))); + float* alphaRowTmp = reinterpret_cast<float*>( + nextWorkspacePtr(reinterpret_cast<int8_t*>(cTmp), m * n * (mType == nvinfer1::DataType::kFLOAT ? 4 : 2))); + float* alphaColTmp + = reinterpret_cast<float*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(alphaRowTmp), m * sizeof(float))); + char* workspaceTmp + = reinterpret_cast<char*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(alphaColTmp), n * sizeof(float))); + + int const wsSize = mRunner->getWorkspaceSize(m, n, k); + + mRunner->gemm( + aTmp, bTmp, mQuantMode, alphaColTmp, alphaRowTmp, cTmp, m, n, k, tactic, workspaceTmp, wsSize, stream); +} + +void SmoothQuantGemmPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + std::vector<size_t> workspaces = { + maxM * k * sizeof(int8_t), // A + n * k * sizeof(int8_t), // B + maxM * n * (mType == nvinfer1::DataType::kFLOAT ? 4u : 2u), // C + maxM * sizeof(float), // alphaRow + n * sizeof(float), // alphaCol + mRunner->getWorkspaceSize(maxM, n, k) // workspace + }; + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<SmoothQuantGemmPluginProfiler::Config> SmoothQuantGemmPluginProfiler::getTactics(int m, int n, int k) const +{ + return mRunner->getConfigs(); +} + +SmoothQuantGemmPlugin::SmoothQuantGemmPlugin( + QuantMode quantMode, nvinfer1::DataType type, SmoothQuantGemmPlugin::PluginProfilerPtr const& pluginProfiler) + : mQuantMode(quantMode) + , mPluginProfiler(pluginProfiler) +{ + init(type); +} + +// Parameterized constructor +SmoothQuantGemmPlugin::SmoothQuantGemmPlugin( + void const* data, size_t length, SmoothQuantGemmPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + nvinfer1::DataType type; + unsigned int quantMode; + read(d, quantMode); + read(d, type); + read(d, mDims); + + mQuantMode = QuantMode(quantMode); + + init(type); + + mPluginProfiler->deserialize(d, mDims, mGemmId); + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +void SmoothQuantGemmPlugin::init(nvinfer1::DataType type) +{ + mType = type; + if (mType == nvinfer1::DataType::kHALF) + { + m_sqGemmRunner = std::make_shared<CutlassInt8GemmRunner<half>>(); + } + else if (mType == nvinfer1::DataType::kFLOAT) + { + m_sqGemmRunner = std::make_shared<CutlassInt8GemmRunner<float>>(); + } + else if (mType == nvinfer1::DataType::kINT32) + { + m_sqGemmRunner = std::make_shared<CutlassInt8GemmRunner<int32_t>>(); + } +#ifdef ENABLE_BF16 + else if (mType == nvinfer1::DataType::kBF16) + { + m_sqGemmRunner = std::make_shared<CutlassInt8GemmRunner<__nv_bfloat16>>(); + } +#endif + + mPluginProfiler->setQuantMode(mQuantMode); + + mGemmId = GemmIdCore(mDims.n, mDims.k, mType); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* SmoothQuantGemmPlugin::clone() const noexcept +{ + auto* plugin = new SmoothQuantGemmPlugin(*this); + return plugin; +} + +nvinfer1::DimsExprs SmoothQuantGemmPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + try + { + TLLM_CHECK(nbInputs == 4); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + TLLM_CHECK(nbDimsA >= 2); + DimsExprs ret; + ret.nbDims = nbDimsA; + for (int ii = 0; ii < nbDimsA - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + ret.d[nbDimsA - 1] = inputs[1].d[0]; + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool SmoothQuantGemmPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + switch (pos) + { + case 0: + // activation + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + case 1: + // weights + // Weights stored in checkpoint must have int8 type + return inOut[pos].type == nvinfer1::DataType::kINT8 && inOut[pos].format == TensorFormat::kLINEAR; + case 2: + // scales channels + case 3: + // scales tokens + return inOut[pos].type == nvinfer1::DataType::kFLOAT && inOut[pos].format == TensorFormat::kLINEAR; + case 4: + // out + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + default: + // Never should be here + assert(false); + return false; + } +} + +void SmoothQuantGemmPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + int const maxN = in[1].max.d[0]; + int const minK = in[0].min.d[in[0].min.nbDims - 1]; + int const minN = in[1].min.d[0]; + + TLLM_CHECK_WITH_INFO(minN == maxN, "Variable out channels is not allowed"); + TLLM_CHECK_WITH_INFO(minK == maxK, "Variable in channels is not allowed"); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, maxN, maxK}; + } + mGemmId = {maxN, maxK, mType}; + + m_workspaceMaxSize = m_sqGemmRunner->getWorkspaceSize(maxM, maxN, maxK); +} + +size_t SmoothQuantGemmPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return m_workspaceMaxSize; +} + +int SmoothQuantGemmPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // mat1 [M(*), K] + // mat2 [N, K] + // scale_tokens [M, 1] if has_per_token_scaling else [1, 1] + // scale_channels [1, N] if has_per_channel_scaling else [1, 1] + // outputs + // mat [M(*), N] + int64_t m64 = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m64 *= inputDesc[0].dims.d[ii]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[1].dims.d[0]); + int const k = TLLM_INT32_CAST(inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]); + int const wsSize = m_sqGemmRunner->getWorkspaceSize(m, n, k); + if (m <= 4) + { + tensorrt_llm::kernels::smooth_quant::Params params(reinterpret_cast<int8_t const*>(inputs[0]), + reinterpret_cast<int8_t const*>(inputs[1]), reinterpret_cast<float const*>(inputs[2]), + reinterpret_cast<float const*>(inputs[3]), reinterpret_cast<void*>(outputs[0]), m, n, k, mQuantMode); + if (mType == nvinfer1::DataType::kHALF) + { + tensorrt_llm::kernels::smooth_quant::int8_sq_launcher<half>(params, stream); + } + else if (mType == nvinfer1::DataType::kFLOAT) + { + tensorrt_llm::kernels::smooth_quant::int8_sq_launcher<float>(params, stream); + } +#ifdef ENABLE_BF16 + else if (mType == nvinfer1::DataType::kBF16) + { + tensorrt_llm::kernels::smooth_quant::int8_sq_launcher<__nv_bfloat16>(params, stream); + } +#endif + else if (mType == nvinfer1::DataType::kINT32) + { + tensorrt_llm::kernels::smooth_quant::int8_sq_launcher<int>(params, stream); + } + } + else + { + auto const& bestTactic = mPluginProfiler->getBestConfig(m, mGemmId); + TLLM_CHECK_WITH_INFO(bestTactic, "No valid SQ GEMM tactic"); + m_sqGemmRunner->gemm(reinterpret_cast<int8_t const*>(inputs[0]), reinterpret_cast<int8_t const*>(inputs[1]), + mQuantMode, reinterpret_cast<float const*>(inputs[3]), reinterpret_cast<float const*>(inputs[2]), + reinterpret_cast<void*>(outputs[0]), m, n, k, *bestTactic, reinterpret_cast<char*>(workspace), wsSize, + stream); + } + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType SmoothQuantGemmPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* SmoothQuantGemmPlugin::getPluginType() const noexcept +{ + return SQ_GEMM_PLUGIN_NAME; +} + +char const* SmoothQuantGemmPlugin::getPluginVersion() const noexcept +{ + return SQ_GEMM_PLUGIN_VERSION; +} + +int SmoothQuantGemmPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int SmoothQuantGemmPlugin::initialize() noexcept +{ + configGemm(); + return 0; +} + +void SmoothQuantGemmPlugin::terminate() noexcept {} + +size_t SmoothQuantGemmPlugin::getSerializationSize() const noexcept +{ + return sizeof(unsigned int) + // QuantMode + sizeof(nvinfer1::DataType) + // dtype + sizeof(mDims) + // Dimensions + mPluginProfiler->getSerializationSize(mGemmId); // selected tactics container size +} + +void SmoothQuantGemmPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mQuantMode.value()); + write(d, mType); + write(d, mDims); + + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void SmoothQuantGemmPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +void SmoothQuantGemmPlugin::configGemm() +{ + mPluginProfiler->profileTactics(m_sqGemmRunner, mType, mDims, mGemmId); +} + +/////////////// + +SmoothQuantGemmPluginCreator::SmoothQuantGemmPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("has_per_channel_scaling", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("has_per_token_scaling", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* SmoothQuantGemmPluginCreator::getPluginName() const noexcept +{ + return SQ_GEMM_PLUGIN_NAME; +} + +char const* SmoothQuantGemmPluginCreator::getPluginVersion() const noexcept +{ + return SQ_GEMM_PLUGIN_VERSION; +} + +PluginFieldCollection const* SmoothQuantGemmPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* SmoothQuantGemmPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + bool perTokenScaling{}; + bool perChannelScaling{}; + nvinfer1::DataType type{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "has_per_channel_scaling")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + perChannelScaling = static_cast<bool>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "has_per_token_scaling")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + perTokenScaling = static_cast<bool>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + try + { + // SmoothQuantGemmPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false); + QuantMode quantMode = QuantMode::fromDescription(true, true, perTokenScaling, perChannelScaling, false, false, + false, false, false, false, false, false, false, false, false, false); + auto* obj = new SmoothQuantGemmPlugin(quantMode, type, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* SmoothQuantGemmPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call SmoothQuantGemmPlugin::destroy() + try + { + // Create plugin profiler with private tactics map which is read from the serialized engine + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true); + auto* obj = new SmoothQuantGemmPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/smoothQuantGemmPlugin.h b/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/smoothQuantGemmPlugin.h new file mode 100644 index 000000000000..3cabf558076b --- /dev/null +++ b/cpp/tensorrt_llm/plugins/smoothQuantGemmPlugin/smoothQuantGemmPlugin.h @@ -0,0 +1,140 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/kernels/cutlass_kernels/int8_gemm/int8_gemm.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::plugins +{ + +using perfMapType = std::unordered_map<int, tensorrt_llm::cutlass_extensions::CutlassGemmConfig>; +using SqGemmRunnerPtr = std::shared_ptr<tensorrt_llm::kernels::cutlass_kernels::CutlassInt8GemmRunnerInterface>; + +class SmoothQuantGemmPluginProfiler : public GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + SqGemmRunnerPtr, GemmIdCore, GemmIdCoreHash> +{ +public: + using Config = tensorrt_llm::cutlass_extensions::CutlassGemmConfig; + + void setQuantMode(tensorrt_llm::common::QuantMode const& quantMode) + { + mQuantMode = quantMode; + } + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + std::vector<Config> getTactics(int m, int n, int k) const override; + +private: + tensorrt_llm::common::QuantMode mQuantMode; +}; + +class SmoothQuantGemmPlugin : public BasePlugin +{ +public: + using PluginProfilerPtr = std::shared_ptr<SmoothQuantGemmPluginProfiler>; + + SmoothQuantGemmPlugin() = delete; + + SmoothQuantGemmPlugin( + tensorrt_llm::common::QuantMode quantMode, nvinfer1::DataType type, PluginProfilerPtr const& pluginProfiler); + + SmoothQuantGemmPlugin(void const* data, size_t length, PluginProfilerPtr const& pluginProfiler); + + ~SmoothQuantGemmPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + void init(nvinfer1::DataType type); + + void configGemm(); + +private: + const std::string mLayerName; + + SqGemmRunnerPtr m_sqGemmRunner; + tensorrt_llm::common::QuantMode mQuantMode; + size_t m_workspaceMaxSize; + + GemmDims mDims{}; + GemmIdCore mGemmId{}; + + PluginProfilerPtr mPluginProfiler; + + nvinfer1::DataType mType; +}; + +class SmoothQuantGemmPluginCreator : public BaseCreator +{ +public: + SmoothQuantGemmPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<SmoothQuantGemmPluginProfiler> gemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/topkLastDimPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/topkLastDimPlugin/CMakeLists.txt new file mode 100644 index 000000000000..6b4e3d8d9e0f --- /dev/null +++ b/cpp/tensorrt_llm/plugins/topkLastDimPlugin/CMakeLists.txt @@ -0,0 +1,22 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# + +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/topkLastDimPlugin/topkLastDimPlugin.cpp b/cpp/tensorrt_llm/plugins/topkLastDimPlugin/topkLastDimPlugin.cpp new file mode 100644 index 000000000000..072bfc9c8fc4 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/topkLastDimPlugin/topkLastDimPlugin.cpp @@ -0,0 +1,316 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "topkLastDimPlugin.h" +#include "tensorrt_llm/common/assert.h" + +using namespace nvinfer1; +using namespace tensorrt_llm::kernels; +using namespace tensorrt_llm::common; +using tensorrt_llm::plugins::TopkLastDimPluginCreator; +using tensorrt_llm::plugins::TopkLastDimPlugin; + +static char const* TOPK_LAST_DIM_PLUGIN_VERSION{"1"}; +static char const* TOPK_LAST_DIM_PLUGIN_NAME{"TopkLastDim"}; +PluginFieldCollection TopkLastDimPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> TopkLastDimPluginCreator::mPluginAttributes; + +TopkLastDimPlugin::TopkLastDimPlugin(nvinfer1::DataType type, int32_t k, bool is_largest) + : mType(type) + , mK(k) // To avoid data-dependent shape, enforce K to be non-dynamic + , mIsLargest(is_largest) +{ + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF) + || (mType == DataType::kINT32), + "Only support int, float, half, and bfloat16."); +} + +// Parameterized constructor +TopkLastDimPlugin::TopkLastDimPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + read(d, mType); + read(d, mK); + read(d, mIsLargest); + TLLM_CHECK(d == a + length); + TLLM_CHECK_WITH_INFO((mType == DataType::kBF16) || (mType == DataType::kFLOAT) || (mType == DataType::kHALF) + || (mType == DataType::kINT32), + "Only support int, float, half, and bfloat16."); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* TopkLastDimPlugin::clone() const noexcept +{ + auto* plugin = new TopkLastDimPlugin(mType, mK, mIsLargest); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +// Outputs +// out_val or out_idx: [batch_size, K] +nvinfer1::DimsExprs TopkLastDimPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + TLLM_CHECK_WITH_INFO(outputIndex < 2, "Only 2 outputs."); + nvinfer1::DimsExprs output(inputs[0]); + int numDim = output.nbDims; + output.d[numDim - 1] = exprBuilder.constant(mK); + return output; +} + +bool TopkLastDimPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + bool res = inOut[pos].format == TensorFormat::kLINEAR; + if (pos < 2) // input and out_val tensor must be the same type as the plugin + { + res = res && inOut[pos].type == mType; + } + else if (pos == 2) // out_idx must be int32 + { + res = res && inOut[pos].type == DataType::kINT32; + } + return res; +} + +void TopkLastDimPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t TopkLastDimPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + // extract shape info and then call helper + auto const batchSize = inputs[getInputTensorIdx()].dims.d[0]; + auto const inputLength = inputs[getInputTensorIdx()].dims.d[1]; + size_t tempStorageBytes{}; + if (mType == DataType::kINT32) + { + tempStorageBytes = invokeComputeTopkLastDimWorkspaceSize<int>(batchSize, inputLength, mK, mIsLargest); + } + else if (mType == DataType::kHALF) + { + tempStorageBytes = invokeComputeTopkLastDimWorkspaceSize<half>(batchSize, inputLength, mK, mIsLargest); + } + else if (mType == DataType::kFLOAT) + { + tempStorageBytes = invokeComputeTopkLastDimWorkspaceSize<float>(batchSize, inputLength, mK, mIsLargest); + } +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16) + { + tempStorageBytes = invokeComputeTopkLastDimWorkspaceSize<__nv_bfloat16>(batchSize, inputLength, mK, mIsLargest); + } +#endif + return tempStorageBytes; +} + +template <typename T> +int TopkLastDimPlugin::enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) +{ + // inputs + // 0. input_tensor [batch_size, inputLength] + // outputs + // 0. output_values [batch_size, k] + // 1. output_indices [batch_size, k] + auto const batchSize = inputDesc[getInputTensorIdx()].dims.d[0]; + auto const inputLength = inputDesc[getInputTensorIdx()].dims.d[1]; + if (batchSize == 0) + { + // nothing to do for empty tensor + return 0; + } + + invokeTopkLastDim<T>( + batchSize, inputLength, mK, mIsLargest, inputs[getInputTensorIdx()], outputs[0], outputs[1], workspace, stream); + + sync_check_cuda_error(stream); + return 0; +} + +int TopkLastDimPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + if (mType == DataType::kINT32) + { + return enqueueImpl<int>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kHALF) + { + return enqueueImpl<half>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kFLOAT) + { + return enqueueImpl<float>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#ifdef ENABLE_BF16 + else if (mType == DataType::kBF16) + { + return enqueueImpl<__nv_bfloat16>(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } +#endif + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType TopkLastDimPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK_WITH_INFO(index < 2, "Only 2 outputs."); + nvinfer1::DataType data_type; + if (index == 1) + { + data_type = DataType::kINT32; + } + else + { + data_type = inputTypes[getInputTensorIdx()]; + } + return data_type; +} + +// IPluginV2 Methods + +char const* TopkLastDimPlugin::getPluginType() const noexcept +{ + return TOPK_LAST_DIM_PLUGIN_NAME; +} + +char const* TopkLastDimPlugin::getPluginVersion() const noexcept +{ + return TOPK_LAST_DIM_PLUGIN_VERSION; +} + +int TopkLastDimPlugin::getNbOutputs() const noexcept +{ + return 2; +} + +int TopkLastDimPlugin::initialize() noexcept +{ + return 0; +} + +void TopkLastDimPlugin::terminate() noexcept {} + +size_t TopkLastDimPlugin::getSerializationSize() const noexcept +{ + return sizeof(mType) + sizeof(mK) + sizeof(mIsLargest); +} + +void TopkLastDimPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mK); + write(d, mIsLargest); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void TopkLastDimPlugin::destroy() noexcept +{ + delete this; +} + +/////////////// + +TopkLastDimPluginCreator::TopkLastDimPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("k", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("is_largest", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* TopkLastDimPluginCreator::getPluginName() const noexcept +{ + return TOPK_LAST_DIM_PLUGIN_NAME; +} + +char const* TopkLastDimPluginCreator::getPluginVersion() const noexcept +{ + return TOPK_LAST_DIM_PLUGIN_VERSION; +} + +PluginFieldCollection const* TopkLastDimPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* TopkLastDimPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + nvinfer1::DataType type{}; + int32_t k{}; + bool is_largest{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "k")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + k = static_cast<int32_t>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "is_largest")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + is_largest = static_cast<int32_t>(*(static_cast<int const*>(fields[i].data))) != 0; + } + } + try + { + auto* obj = new TopkLastDimPlugin(type, k, is_largest); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* TopkLastDimPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call TopkLastDimPlugin::destroy() + try + { + auto* obj = new TopkLastDimPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/topkLastDimPlugin/topkLastDimPlugin.h b/cpp/tensorrt_llm/plugins/topkLastDimPlugin/topkLastDimPlugin.h new file mode 100644 index 000000000000..0ca38ccfe105 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/topkLastDimPlugin/topkLastDimPlugin.h @@ -0,0 +1,99 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TRT_TOPK_LAST_DIM_PLUGIN_H +#define TRT_TOPK_LAST_DIM_PLUGIN_H + +#include "tensorrt_llm/kernels/topkLastDim.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include <cassert> + +namespace tensorrt_llm::plugins +{ +class TopkLastDimPlugin : public BasePlugin +{ +public: + TopkLastDimPlugin(nvinfer1::DataType type, int32_t k, bool largest); + TopkLastDimPlugin(void const* data, size_t length); + ~TopkLastDimPlugin() override = default; + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + template <typename T> + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + using IndexType = std::int32_t; + + IndexType getInputTensorIdx() const + { + return 0; + }; + +private: + nvinfer1::DataType mType; + int32_t mK; + bool mIsLargest; +}; + +class TopkLastDimPluginCreator : public BaseCreator +{ +public: + TopkLastDimPluginCreator(); + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins + +#endif diff --git a/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/CMakeLists.txt new file mode 100755 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/weightOnlyGroupwiseQuantMatmulPlugin.cpp b/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/weightOnlyGroupwiseQuantMatmulPlugin.cpp new file mode 100644 index 000000000000..85f0cf011293 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/weightOnlyGroupwiseQuantMatmulPlugin.cpp @@ -0,0 +1,657 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "weightOnlyGroupwiseQuantMatmulPlugin.h" + +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels::cutlass_kernels; +using tensorrt_llm::plugins::WeightOnlyGroupwiseQuantMatmulPluginCreator; +using tensorrt_llm::plugins::WeightOnlyGroupwiseQuantMatmulPlugin; +using tensorrt_llm::plugins::WeightOnlyGroupwiseQuantGemmPluginProfiler; +using tensorrt_llm::plugins::WeightOnlyGemmRunnerPtr; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +static char const* WOQ_GROUPWISE_MATMUL_PLUGIN_VERSION{"1"}; +static char const* WOQ_GROUPWISE_MATMUL_PLUGIN_NAME{"WeightOnlyGroupwiseQuantMatmul"}; +PluginFieldCollection WeightOnlyGroupwiseQuantMatmulPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> WeightOnlyGroupwiseQuantMatmulPluginCreator::mPluginAttributes; + +void WeightOnlyGroupwiseQuantGemmPluginProfiler::runTactic(int m, int n, int k, + WeightOnlyGroupwiseQuantGemmPluginProfiler::Config const& tactic, char* workspace, cudaStream_t const& stream) +{ + // Quantized weights are packed in FP16 format (INT4*4 -> FP16, INT8*2 -> FP16) + int const originalN = mQuantAlgo & GroupwiseQuantAlgo::INT8_WEIGHT ? n * FP16_INT8_RATIO : n * FP16_INT4_RATIO; + half* actPtr = reinterpret_cast<half*>(workspace); + void* weightPtr = nextWorkspacePtr(reinterpret_cast<int8_t*>(actPtr), m * k * sizeof(half)); + half* inputScalesPtr + = reinterpret_cast<half*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(weightPtr), n * k * sizeof(float))); + half* zerosPtr = reinterpret_cast<half*>( + nextWorkspacePtr(reinterpret_cast<int8_t*>(inputScalesPtr), k * originalN * sizeof(half) / mGroupSize)); + half* biasesPtr = reinterpret_cast<half*>( + nextWorkspacePtr(reinterpret_cast<int8_t*>(zerosPtr), k * originalN * sizeof(half) / mGroupSize)); + half* outputPtr = reinterpret_cast<half*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(biasesPtr), n * sizeof(half))); + char* workspacePtr + = reinterpret_cast<char*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(outputPtr), m * originalN * sizeof(half))); + if ((mQuantAlgo & GroupwiseQuantAlgo::ZERO) == 0) + { + zerosPtr = nullptr; + } + if ((mQuantAlgo & GroupwiseQuantAlgo::BIAS) == 0) + { + biasesPtr = nullptr; + } + + if (tactic.enableCudaKernel) + { + // run CUDA kernel + void const* pre_quant_scale_ptr = nullptr; + bool apply_alpha_in_advance = false; + float alpha = 1.0; + tensorrt_llm::kernels::weight_only::Params params{actPtr, pre_quant_scale_ptr, weightPtr, inputScalesPtr, + zerosPtr, biasesPtr, outputPtr, alpha, m, originalN, k, mGroupSize, mCudaKernelType, + apply_alpha_in_advance}; + tensorrt_llm::kernels::weight_only::kernel_launcher(mArch, params, stream); + } + else + { + // run CUTLASS kernel + int const wsSize = mRunner->getWorkspaceSize(m, originalN, k); + if (mQuantAlgo & GroupwiseQuantAlgo::INT8_WEIGHT) + { + mRunner->gemm(actPtr, reinterpret_cast<int8_t*>(weightPtr), inputScalesPtr, zerosPtr, biasesPtr, outputPtr, + m, originalN, k, mGroupSize, tactic, workspacePtr, wsSize, stream); + } + else + { + mRunner->gemm(actPtr, reinterpret_cast<cutlass::uint4b_t*>(weightPtr), inputScalesPtr, zerosPtr, biasesPtr, + outputPtr, m, originalN, k, mGroupSize, tactic, workspacePtr, wsSize, stream); + } + } +} + +void WeightOnlyGroupwiseQuantGemmPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + // Quantized weights are packed in FP16 format (INT4*4 -> FP16, INT8*2 -> FP16) + int const originalN = mQuantAlgo & GroupwiseQuantAlgo::INT8_WEIGHT ? n * FP16_INT8_RATIO : n * FP16_INT4_RATIO; + std::vector<size_t> workspaces = { + maxM * k * sizeof(half), // A + k * n * sizeof(float), // B + k * originalN * sizeof(half) / mGroupSize, // scales + k * originalN * sizeof(half) / mGroupSize, // zeros + originalN * sizeof(half), // biases + maxM * originalN * sizeof(half), // C + mRunner->getWorkspaceSize(maxM, originalN, k) // workspace + }; + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<WeightOnlyGroupwiseQuantGemmPluginProfiler::Config> WeightOnlyGroupwiseQuantGemmPluginProfiler::getTactics( + int m, int n, int k) const +{ + return mRunner->getConfigs(); +} + +bool WeightOnlyGroupwiseQuantGemmPluginProfiler::checkTactic(int m, int n, int k, Config const& tactic) const +{ + // stop to profile Cuda kernel for m >= 16 + if (tactic.enableCudaKernel) + { + return m < 16; + } + return true; +} + +WeightOnlyGroupwiseQuantMatmulPlugin::WeightOnlyGroupwiseQuantMatmulPlugin(nvinfer1::DataType type, int quant_algo, + int group_size, float alpha, WeightOnlyGroupwiseQuantMatmulPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + init(type, quant_algo, group_size, alpha); +} + +// Parameterized constructor +WeightOnlyGroupwiseQuantMatmulPlugin::WeightOnlyGroupwiseQuantMatmulPlugin( + void const* data, size_t length, WeightOnlyGroupwiseQuantMatmulPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + nvinfer1::DataType type; + int quant_algo = 0; + int group_size = 0; + float alpha = 1.0f; + read(d, type); + read(d, quant_algo); + read(d, group_size); + read(d, alpha); + read(d, mDims); + + init(type, quant_algo, group_size, alpha); + + mPluginProfiler->deserialize(d, mDims, mGemmId); + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +template <typename ActivationType, typename WeightType, typename OutputType, typename ScaleZeroType, + cutlass::WeightOnlyQuantOp QuantOp> +using GemmRunner = tensorrt_llm::kernels::cutlass_kernels::CutlassFpAIntBGemmRunner<ActivationType, WeightType, QuantOp, + ScaleZeroType, OutputType, OutputType>; + +template <typename ActivationType, typename WeightType, typename OutputType, typename ScaleZeroType = OutputType> +WeightOnlyGemmRunnerPtr selectGemmRunnerForZERO(int quant_algo) +{ + if (quant_algo & GroupwiseQuantAlgo::ZERO) + { + return std::make_shared<GemmRunner<ActivationType, WeightType, OutputType, ScaleZeroType, + cutlass::WeightOnlyQuantOp::FINEGRAINED_SCALE_AND_ZEROS>>(); + } + else + { + return std::make_shared<GemmRunner<ActivationType, WeightType, OutputType, ScaleZeroType, + cutlass::WeightOnlyQuantOp::FINEGRAINED_SCALE_ONLY>>(); + } +} + +template <typename ActivationType> +WeightOnlyGemmRunnerPtr selectGemmRunnerForWeightType(int quant_algo) +{ + if (quant_algo & GroupwiseQuantAlgo::INT8_WEIGHT) + { + return selectGemmRunnerForZERO<ActivationType, uint8_t, ActivationType>(quant_algo); + } + else + { + return selectGemmRunnerForZERO<ActivationType, cutlass::uint4b_t, ActivationType>(quant_algo); + } +} + +void WeightOnlyGroupwiseQuantMatmulPlugin::init(nvinfer1::DataType type, int quant_algo, int group_size, float alpha) +{ + mArch = tensorrt_llm::common::getSMVersion(); + mType = type; + mQuantAlgo = quant_algo; + mGroupSize = group_size; + + // quant_algo = int8_weight * 16 + fp8_alpha * 8 + pre_quant_scale * 4 + zero * 2 + bias + mPreQuantScaleInputIdx = (quant_algo & GroupwiseQuantAlgo::PRE_QUANT_SCALE) ? 1 : 0; + mWeightInputIdx = mPreQuantScaleInputIdx + 1; + mScalesInputIdx = mWeightInputIdx + 1; + mZerosInputIdx = (quant_algo & GroupwiseQuantAlgo::ZERO) ? mScalesInputIdx + 1 : mScalesInputIdx; + mBiasesInputIdx = (quant_algo & GroupwiseQuantAlgo::BIAS) ? mZerosInputIdx + 1 : mZerosInputIdx; + + if (mType == nvinfer1::DataType::kHALF) + { + // CUTLASS kernel selection + if (quant_algo & GroupwiseQuantAlgo::FP8_ALPHA) + { + mAlpha = alpha; + + // Ada & Hopper style kernels + if (mArch < 89) + { + TLLM_THROW("W4A(fp)8 kernel is unsupported on pre-Ada (sm<89) architectures!"); + } + assert(!(quant_algo & GroupwiseQuantAlgo::INT8_WEIGHT) && "W4A(fp)8 kernel requires INT4 weight!"); + m_weightOnlyGroupwiseGemmRunner + = selectGemmRunnerForZERO<__nv_fp8_e4m3, cutlass::uint4b_t, half>(quant_algo); + } + else + { + m_weightOnlyGroupwiseGemmRunner = selectGemmRunnerForWeightType<half>(quant_algo); + } + // CUDA kernel selection + if (quant_algo & GroupwiseQuantAlgo::INT8_WEIGHT) + { + // INT8 weight + mCudaKernelEnabled = tensorrt_llm::kernels::weight_only::is_supported( + mArch, tensorrt_llm::kernels::weight_only::KernelType::FP16Int8Groupwise); + mCudaKernelType = tensorrt_llm::kernels::weight_only::KernelType::FP16Int8Groupwise; + } + else + { + // INT4 weight + mCudaKernelEnabled = tensorrt_llm::kernels::weight_only::is_supported( + mArch, tensorrt_llm::kernels::weight_only::KernelType::FP16Int4Groupwise); + mCudaKernelType = tensorrt_llm::kernels::weight_only::KernelType::FP16Int4Groupwise; + } + } +#if defined(ENABLE_BF16) + else if (mType == nvinfer1::DataType::kBF16) + { + // CUTLASS kernel selection + if (quant_algo & GroupwiseQuantAlgo::FP8_ALPHA) + { + mAlpha = alpha; + + // FP8 requires at least sm89 devices + if (mArch < 89) + { + TLLM_THROW("W4A(fp)8 kernel is unsupported on pre-Ada (sm<89) architectures!"); + } + assert(!(quant_algo & GroupwiseQuantAlgo::INT8_WEIGHT) && "W4A(fp)8 kernel requires INT4 weight!"); + m_weightOnlyGroupwiseGemmRunner + = selectGemmRunnerForZERO<__nv_fp8_e4m3, cutlass::uint4b_t, __nv_bfloat16, half>(quant_algo); + } + else + { + m_weightOnlyGroupwiseGemmRunner = selectGemmRunnerForWeightType<__nv_bfloat16>(quant_algo); + } + // CUDA kernel selection + if (quant_algo & GroupwiseQuantAlgo::INT8_WEIGHT) + { + // INT8 weight + mCudaKernelEnabled = tensorrt_llm::kernels::weight_only::is_supported( + mArch, tensorrt_llm::kernels::weight_only::KernelType::BF16Int8Groupwise); + mCudaKernelType = tensorrt_llm::kernels::weight_only::KernelType::BF16Int8Groupwise; + } + else + { + // INT4 weight + mCudaKernelEnabled = tensorrt_llm::kernels::weight_only::is_supported( + mArch, tensorrt_llm::kernels::weight_only::KernelType::BF16Int4Groupwise); + mCudaKernelType = tensorrt_llm::kernels::weight_only::KernelType::BF16Int4Groupwise; + } + } +#endif + else + { + TLLM_THROW("Unsupported data type"); + } + mPluginProfiler->setQuantAlgo(mQuantAlgo); + mPluginProfiler->setGroupSize(mGroupSize); + if (mCudaKernelEnabled) + { + mPluginProfiler->setCudaKernelType(mCudaKernelType, mArch); + } + mGemmId = GemmIdCore(mDims.n, mDims.k, mType); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* WeightOnlyGroupwiseQuantMatmulPlugin::clone() const noexcept +{ + auto* plugin = new WeightOnlyGroupwiseQuantMatmulPlugin(*this); + return plugin; +} + +void WeightOnlyGroupwiseQuantMatmulPlugin::configGemm() +{ + mPluginProfiler->profileTactics(m_weightOnlyGroupwiseGemmRunner, mType, mDims, mGemmId, mCudaKernelEnabled); +} + +nvinfer1::DimsExprs WeightOnlyGroupwiseQuantMatmulPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + + // inputs + // 0 activations [M, K] + // 1 pre-quant scales [K] (optional) + // 2 weights [K, N/2] + // 3 scales [K // group_size, N] + // 4 zeros [K // group_size, N] (optional) + // 5 biases [N] (optional) + + try + { + TLLM_CHECK(nbInputs == mBiasesInputIdx + 1); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + int const nbDimsB = inputs[mWeightInputIdx].nbDims; + TLLM_CHECK(nbDimsA >= 2); + TLLM_CHECK(nbDimsB == 2); + DimsExprs ret; + ret.nbDims = nbDimsA; + for (int ii = 0; ii < nbDimsA - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + + // int4/int8 weight only quant (INT4*4 -> FP16, INT8*2 -> FP16) + int const weight_multiplier = mQuantAlgo & GroupwiseQuantAlgo::INT8_WEIGHT ? FP16_INT8_RATIO : FP16_INT4_RATIO; + ret.d[nbDimsA - 1] = exprBuilder.constant(inputs[mWeightInputIdx].d[1]->getConstantValue() * weight_multiplier); + + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool WeightOnlyGroupwiseQuantMatmulPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + if (pos < nbInputs + 1) + { + return inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + } + else + { + // Never should be here + assert(false); + return false; + } +} + +void WeightOnlyGroupwiseQuantMatmulPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + + // Quantized weights are packed in FP16 format (INT4*4 -> FP16, INT8*2 -> FP16) + int const weight_multiplier = mQuantAlgo & GroupwiseQuantAlgo::INT8_WEIGHT ? FP16_INT8_RATIO : FP16_INT4_RATIO; + int const maxN = in[mWeightInputIdx].max.d[1] * weight_multiplier; + + auto const K = maxK; + auto const N = maxN / weight_multiplier; + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, N, K}; + } + mGemmId = {N, K, mType}; + + size_t smoothedActSize = static_cast<size_t>(maxM) * static_cast<size_t>(maxK) + * (in[0].desc.type == nvinfer1::DataType::kFLOAT ? sizeof(float) : sizeof(half)); + m_workspaceMaxSize = smoothedActSize + m_weightOnlyGroupwiseGemmRunner->getWorkspaceSize(maxM, maxN, maxK); +} + +size_t WeightOnlyGroupwiseQuantMatmulPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return m_workspaceMaxSize; +} + +template <typename ActType> +void pre_quant_scale_for_act(int const m, int const k, int const mQuantAlgo, int const mPreQuantScaleInputIdx, + void const* const* inputs, void* workspace, cudaStream_t stream) +{ + // Apply pre-quant per channel scale on activations + if (mQuantAlgo & GroupwiseQuantAlgo::FP8_ALPHA) + { + tensorrt_llm::kernels::apply_per_channel_scale_kernel_launcher<ActType, __nv_fp8_e4m3>( + reinterpret_cast<__nv_fp8_e4m3*>(workspace), reinterpret_cast<ActType const*>(inputs[0]), + reinterpret_cast<ActType const*>(inputs[mPreQuantScaleInputIdx]), m, k, nullptr, stream); + } + else + { + tensorrt_llm::kernels::apply_per_channel_scale_kernel_launcher<ActType, ActType>( + reinterpret_cast<ActType*>(workspace), reinterpret_cast<ActType const*>(inputs[0]), + reinterpret_cast<ActType const*>(inputs[mPreQuantScaleInputIdx]), m, k, nullptr, stream); + } +} + +int WeightOnlyGroupwiseQuantMatmulPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // 0 activations [M, K] + // 1 pre-quant scales [K] + // 2 weights [K, N/2] + // 3 scales [K // group_size, N] + // 4 zeros [K // group_size, N] + // 5 biases [N] + // outputs + // mat [M, N] + + int64_t m64 = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m64 *= inputDesc[0].dims.d[ii]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[mWeightInputIdx].dims.d[1]); + int const k = TLLM_INT32_CAST(inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]); + + // get best tactic and check if CUDA kernel should be used + bool use_cuda_kernel = false; + auto const& bestTactic = mPluginProfiler->getBestConfig(m, mGemmId); + TLLM_CHECK_WITH_INFO(bestTactic, + "No valid weight only groupwise GEMM tactic(It is usually caused by the failure to execute all " + "candidate configurations of the CUTLASS kernel, please pay attention to the warning information " + "when building the engine.)"); + use_cuda_kernel = bestTactic->enableCudaKernel; + + bool use_pre_quant_scale = mQuantAlgo & GroupwiseQuantAlgo::PRE_QUANT_SCALE; + half const* zeros_ptr + = (mQuantAlgo & GroupwiseQuantAlgo::ZERO) ? reinterpret_cast<half const*>(inputs[mZerosInputIdx]) : nullptr; + half const* biases_ptr + = (mQuantAlgo & GroupwiseQuantAlgo::BIAS) ? reinterpret_cast<half const*>(inputs[mBiasesInputIdx]) : nullptr; + half const* act_ptr = reinterpret_cast<half const*>(inputs[0]); + + if (use_pre_quant_scale && !use_cuda_kernel) + { + // Apply pre-quant per channel scale on activations + act_ptr = reinterpret_cast<half const*>(workspace); + if (mType == nvinfer1::DataType::kHALF) + { + pre_quant_scale_for_act<half>(m, k, mQuantAlgo, mPreQuantScaleInputIdx, inputs, workspace, stream); + } +#if defined(ENABLE_BF16) + else if (mType == nvinfer1::DataType::kBF16) + { + pre_quant_scale_for_act<__nv_bfloat16>(m, k, mQuantAlgo, mPreQuantScaleInputIdx, inputs, workspace, stream); + } +#endif + } + +#if defined(ENABLE_BF16) + TLLM_CHECK_WITH_INFO(mType == nvinfer1::DataType::kHALF || mType == nvinfer1::DataType::kBF16, + "No valid weightOnlyGropwiseQuantMatmul configuration"); +#else + TLLM_CHECK_WITH_INFO(mType == nvinfer1::DataType::kHALF, "No valid weightOnlyGropwiseQuantMatmul configuration"); +#endif + + // Quantized weights are packed in FP16 format (INT4*4 -> FP16, INT8*2 -> FP16) + int real_n = mQuantAlgo & GroupwiseQuantAlgo::INT8_WEIGHT ? n * FP16_INT8_RATIO : n * FP16_INT4_RATIO; + + if (use_cuda_kernel) + { + // Apply CUDA kernel + void const* pre_quant_scale_ptr = nullptr; + if (use_pre_quant_scale) + pre_quant_scale_ptr = inputs[mPreQuantScaleInputIdx]; + void const* cuda_kernel_act_ptr = inputs[0]; + void const* cuda_kernel_weight_ptr = inputs[mWeightInputIdx]; + void const* cuda_kernel_scales_ptr = inputs[mScalesInputIdx]; + void* cuda_kernel_out_ptr = outputs[0]; + tensorrt_llm::kernels::weight_only::Params params{cuda_kernel_act_ptr, pre_quant_scale_ptr, + cuda_kernel_weight_ptr, cuda_kernel_scales_ptr, zeros_ptr, biases_ptr, cuda_kernel_out_ptr, mAlpha, m, + real_n, k, mGroupSize, mCudaKernelType, static_cast<bool>(mQuantAlgo & GroupwiseQuantAlgo::FP8_ALPHA)}; + tensorrt_llm::kernels::weight_only::kernel_launcher(mArch, params, stream); + } + else + { + // Apply CUTLASS kernel + int const ws_bytes = m_weightOnlyGroupwiseGemmRunner->getWorkspaceSize(m, real_n, k); + int32_t* weight_ptr = const_cast<int32_t*>(reinterpret_cast<int32_t const*>(inputs[mWeightInputIdx])); + m_weightOnlyGroupwiseGemmRunner->gemm(act_ptr, weight_ptr, inputs[mScalesInputIdx], zeros_ptr, biases_ptr, + mAlpha, outputs[0], m, real_n, k, mGroupSize, *bestTactic, + reinterpret_cast<char*>(workspace) + m * k * sizeof(half), ws_bytes, stream); + } + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType WeightOnlyGroupwiseQuantMatmulPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* WeightOnlyGroupwiseQuantMatmulPlugin::getPluginType() const noexcept +{ + return WOQ_GROUPWISE_MATMUL_PLUGIN_NAME; +} + +char const* WeightOnlyGroupwiseQuantMatmulPlugin::getPluginVersion() const noexcept +{ + return WOQ_GROUPWISE_MATMUL_PLUGIN_VERSION; +} + +int WeightOnlyGroupwiseQuantMatmulPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int WeightOnlyGroupwiseQuantMatmulPlugin::initialize() noexcept +{ + configGemm(); + return 0; +} + +void WeightOnlyGroupwiseQuantMatmulPlugin::terminate() noexcept {} + +size_t WeightOnlyGroupwiseQuantMatmulPlugin::getSerializationSize() const noexcept +{ + return sizeof(nvinfer1::DataType) + // mType + sizeof(int) + // mQuantAlgo + sizeof(int) + // mGroupSize + sizeof(float) + // mAlpha + sizeof(mDims) + // Dimensions + mPluginProfiler->getSerializationSize(mGemmId); // selected tactics container size +} + +void WeightOnlyGroupwiseQuantMatmulPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mQuantAlgo); + write(d, mGroupSize); + write(d, mAlpha); + write(d, mDims); + + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void WeightOnlyGroupwiseQuantMatmulPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +WeightOnlyGroupwiseQuantMatmulPluginCreator::WeightOnlyGroupwiseQuantMatmulPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("quant_algo", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("group_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("alpha", nullptr, PluginFieldType::kFLOAT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* WeightOnlyGroupwiseQuantMatmulPluginCreator::getPluginName() const noexcept +{ + return WOQ_GROUPWISE_MATMUL_PLUGIN_NAME; +} + +char const* WeightOnlyGroupwiseQuantMatmulPluginCreator::getPluginVersion() const noexcept +{ + return WOQ_GROUPWISE_MATMUL_PLUGIN_VERSION; +} + +PluginFieldCollection const* WeightOnlyGroupwiseQuantMatmulPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* WeightOnlyGroupwiseQuantMatmulPluginCreator::createPlugin( + char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + nvinfer1::DataType type{}; + int QuantAlgo{}; + int GroupSize{}; + float Alpha{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "quant_algo")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + QuantAlgo = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "group_size")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + GroupSize = static_cast<int>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + else if (!strcmp(attrName, "alpha")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kFLOAT32); + Alpha = static_cast<float>(*(static_cast<float const*>(fields[i].data))); + } + } + try + { + // WeightOnlyGroupwiseQuantMatmulPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false); + auto* obj = new WeightOnlyGroupwiseQuantMatmulPlugin(type, QuantAlgo, GroupSize, Alpha, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* WeightOnlyGroupwiseQuantMatmulPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call weightOnlyGroupwiseQuantMatmulPlugin::destroy() + try + { + // Create plugin profiler with private tactics map which is read from the serialized engine + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true); + auto* obj = new WeightOnlyGroupwiseQuantMatmulPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/weightOnlyGroupwiseQuantMatmulPlugin.h b/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/weightOnlyGroupwiseQuantMatmulPlugin.h new file mode 100644 index 000000000000..94e98ce0f5c0 --- /dev/null +++ b/cpp/tensorrt_llm/plugins/weightOnlyGroupwiseQuantMatmulPlugin/weightOnlyGroupwiseQuantMatmulPlugin.h @@ -0,0 +1,186 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/kernels/cutlass_kernels/fpA_intB_gemm/fpA_intB_gemm.h" +#include "tensorrt_llm/kernels/preQuantScaleKernel.h" +#include "tensorrt_llm/kernels/weightOnlyBatchedGemv//kernelLauncher.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/weightOnlyQuantMatmulPlugin.h" + +#include <cutlass/numeric_types.h> + +#include <cassert> +#include <cuda_runtime.h> +#include <memory> +#include <set> +#include <string> +#include <vector> + +// The blank line here is to avoid clang-format -sort-includes option reordering these two cutlass header files and +// breaking dependencies +#include "cutlass/integer_subbyte.h" + +namespace tensorrt_llm::plugins +{ + +using WeightOnlyGemmRunner = tensorrt_llm::kernels::cutlass_kernels::CutlassFpAIntBGemmRunnerInterface; +using WeightOnlyGemmRunnerPtr = std::shared_ptr<WeightOnlyGemmRunner>; +using KernelType = tensorrt_llm::kernels::weight_only::KernelType; + +class WeightOnlyGroupwiseQuantGemmPluginProfiler + : public GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, WeightOnlyGemmRunnerPtr, + GemmIdCore, GemmIdCoreHash> +{ +public: + using Config = tensorrt_llm::cutlass_extensions::CutlassGemmConfig; + + void setQuantAlgo(int quantAlgo) + { + mQuantAlgo = quantAlgo; + } + + void setGroupSize(int groupSize) + { + mGroupSize = groupSize; + } + + void setCudaKernelType(KernelType cudaKernelType, int arch) + { + mCudaKernelType = cudaKernelType; + mArch = arch; + } + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + std::vector<Config> getTactics(int m, int n, int k) const override; + + bool checkTactic(int m, int n, int k, Config const& tactic) const override; + +private: + int mQuantAlgo; + int mGroupSize; + KernelType mCudaKernelType; + int mArch; +}; + +class WeightOnlyGroupwiseQuantMatmulPlugin : public BasePlugin +{ +public: + using PluginProfilerPtr = std::shared_ptr<WeightOnlyGroupwiseQuantGemmPluginProfiler>; + + WeightOnlyGroupwiseQuantMatmulPlugin() = delete; + + WeightOnlyGroupwiseQuantMatmulPlugin( + nvinfer1::DataType type, int quant_algo, int group_size, float alpha, PluginProfilerPtr const& profiler); + + WeightOnlyGroupwiseQuantMatmulPlugin(void const* data, size_t length, PluginProfilerPtr const& profiler); + + ~WeightOnlyGroupwiseQuantMatmulPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + // group_size: 64, 128 + void init(nvinfer1::DataType type, int quant_algo, int group_size, float alpha); + + void configGemm(); + +private: + const std::string mLayerName; + + WeightOnlyGemmRunnerPtr m_weightOnlyGroupwiseGemmRunner; + size_t m_workspaceMaxSize; + nvinfer1::DataType mType; + bool mCudaKernelEnabled; + tensorrt_llm::kernels::weight_only::KernelType mCudaKernelType; + int mArch; + + // When M is smaller than this value, we trigger a fast path + // I.e. a tailored kernel instead of cutlass. + + int mQuantAlgo; + + int mGroupSize; + + float mAlpha = 1.0f; + + int mPreQuantScaleInputIdx; + int mWeightInputIdx; + int mScalesInputIdx; + int mZerosInputIdx; + int mBiasesInputIdx; + + GemmDims mDims{}; + GemmIdCore mGemmId{}; + + PluginProfilerPtr mPluginProfiler; +}; + +class WeightOnlyGroupwiseQuantMatmulPluginCreator : public BaseCreator +{ +public: + WeightOnlyGroupwiseQuantMatmulPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<WeightOnlyGroupwiseQuantGemmPluginProfiler> gemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/CMakeLists.txt b/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/CMakeLists.txt new file mode 100755 index 000000000000..86876224fccd --- /dev/null +++ b/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/CMakeLists.txt @@ -0,0 +1,21 @@ +# +# SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +# +file(GLOB SRCS *.cpp) +set(PLUGIN_SOURCES ${PLUGIN_SOURCES} ${SRCS}) +set(PLUGIN_SOURCES + ${PLUGIN_SOURCES} + PARENT_SCOPE) diff --git a/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/weightOnlyQuantMatmulPlugin.cpp b/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/weightOnlyQuantMatmulPlugin.cpp new file mode 100644 index 000000000000..f3ed07fafaff --- /dev/null +++ b/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/weightOnlyQuantMatmulPlugin.cpp @@ -0,0 +1,507 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "weightOnlyQuantMatmulPlugin.h" + +#include <numeric> + +using namespace nvinfer1; +using namespace tensorrt_llm::common; +using namespace tensorrt_llm::kernels::cutlass_kernels; +using tensorrt_llm::plugins::WeightOnlyQuantMatmulPluginCreator; +using tensorrt_llm::plugins::WeightOnlyQuantMatmulPlugin; +using tensorrt_llm::plugins::WeightOnlyQuantGemmPluginProfiler; +using tensorrt_llm::plugins::read; +using tensorrt_llm::plugins::write; + +static char const* WOQ_MATMUL_PLUGIN_VERSION{"1"}; +static char const* WOQ_MATMUL_PLUGIN_NAME{"WeightOnlyQuantMatmul"}; +PluginFieldCollection WeightOnlyQuantMatmulPluginCreator::mFC{}; +std::vector<nvinfer1::PluginField> WeightOnlyQuantMatmulPluginCreator::mPluginAttributes; + +void WeightOnlyQuantGemmPluginProfiler::runTactic(int m, int n, int k, + WeightOnlyQuantGemmPluginProfiler::Config const& tactic, char* workspace, cudaStream_t const& stream) +{ + int const originalN = n * getWeightTypeMultiplier(mWeightTypeId); + half* actPtr = reinterpret_cast<half*>(workspace); + int8_t* weightPtr + = reinterpret_cast<int8_t*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(actPtr), m * k * sizeof(half))); + half* scalesPtr + = reinterpret_cast<half*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(weightPtr), n * k * sizeof(int8_t))); + half* outputPtr + = reinterpret_cast<half*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(scalesPtr), originalN * sizeof(half))); + char* workspacePtr + = reinterpret_cast<char*>(nextWorkspacePtr(reinterpret_cast<int8_t*>(outputPtr), m * originalN * sizeof(half))); + + int const wsSize = mRunner->getWorkspaceSize(m, originalN, k); + + if (tactic.enableCudaKernel) + { + // run CUDA kernel + tensorrt_llm::kernels::weight_only::Params params{actPtr, nullptr, weightPtr, scalesPtr, nullptr, nullptr, + outputPtr, 1.f, m, originalN, k, 0, mCudaKernelType}; + tensorrt_llm::kernels::weight_only::kernel_launcher(mArch, params, stream); + } + else + { + // run CUTLASS kernel + if (mWeightTypeId == WeightTypeId::INT8) + { + mRunner->gemm( + actPtr, weightPtr, scalesPtr, outputPtr, m, originalN, k, tactic, workspacePtr, wsSize, stream); + } + else + { + mRunner->gemm(actPtr, reinterpret_cast<cutlass::uint4b_t*>(weightPtr), scalesPtr, outputPtr, m, originalN, + k, tactic, workspacePtr, wsSize, stream); + } + } +} + +void WeightOnlyQuantGemmPluginProfiler::computeTmpSize(size_t maxM, size_t n, size_t k) +{ + int const originalN = n * getWeightTypeMultiplier(mWeightTypeId); + std::vector<size_t> workspaces = { + maxM * k * sizeof(half), // A + n * k * sizeof(int8_t), // B + originalN * sizeof(half), // scales + maxM * originalN * sizeof(half), // C + mRunner->getWorkspaceSize(maxM, originalN, k) // workspace + }; + size_t bytes = calculateTotalWorkspaceSize(workspaces.data(), workspaces.size()); + setTmpWorkspaceSizeInBytes(bytes); +} + +std::vector<WeightOnlyQuantGemmPluginProfiler::Config> WeightOnlyQuantGemmPluginProfiler::getTactics( + int m, int n, int k) const +{ + return mRunner->getConfigs(); +} + +bool WeightOnlyQuantGemmPluginProfiler::checkTactic(int m, int n, int k, Config const& tactic) const +{ + // stop to profile Cuda kernel for m >= 16 + if (tactic.enableCudaKernel) + { + return m < 16; + } + return true; +} + +WeightOnlyQuantMatmulPlugin::WeightOnlyQuantMatmulPlugin(nvinfer1::DataType type, WeightTypeId weightTypeId, + WeightOnlyQuantMatmulPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + init(type, weightTypeId); +} + +// Parameterized constructor +WeightOnlyQuantMatmulPlugin::WeightOnlyQuantMatmulPlugin( + void const* data, size_t length, WeightOnlyQuantMatmulPlugin::PluginProfilerPtr const& pluginProfiler) + : mPluginProfiler(pluginProfiler) +{ + char const *d = reinterpret_cast<char const*>(data), *a = d; + nvinfer1::DataType type; + WeightTypeId weightTypeId; + read(d, type); + read(d, weightTypeId); + read(d, mDims); + + init(type, weightTypeId); + + mPluginProfiler->deserialize(d, mDims, mGemmId); + + TLLM_CHECK_WITH_INFO(d == a + length, + "Expected length (%d) != real length (%d). This is often " + "caused by using different TensorRT LLM version to build " + "engine and run engine.", + (int) length, (int) (d - a)); +} + +void WeightOnlyQuantMatmulPlugin::init(nvinfer1::DataType type, WeightTypeId weightTypeId) +{ + mArch = tensorrt_llm::common::getSMVersion(); + mType = type; + mWeightTypeId = weightTypeId; + + if (mWeightTypeId == WeightTypeId::INT8) + { + if (mType == nvinfer1::DataType::kHALF) + { + m_weightOnlyGemmRunner = std::make_shared< + CutlassFpAIntBGemmRunner<half, uint8_t, cutlass::WeightOnlyQuantOp::PER_COLUMN_SCALE_ONLY>>(); + mCudaKernelEnabled = tensorrt_llm::kernels::weight_only::is_supported( + mArch, tensorrt_llm::kernels::weight_only::KernelType::FP16Int8PerChannel); + mCudaKernelType = tensorrt_llm::kernels::weight_only::KernelType::FP16Int8PerChannel; + } +#if defined(ENABLE_BF16) + else if (mType == nvinfer1::DataType::kBF16) + { + m_weightOnlyGemmRunner = std::make_shared< + CutlassFpAIntBGemmRunner<__nv_bfloat16, uint8_t, cutlass::WeightOnlyQuantOp::PER_COLUMN_SCALE_ONLY>>(); + mCudaKernelEnabled = tensorrt_llm::kernels::weight_only::is_supported( + mArch, tensorrt_llm::kernels::weight_only::KernelType::BF16Int8PerChannel); + mCudaKernelType = tensorrt_llm::kernels::weight_only::KernelType::BF16Int8PerChannel; + } +#endif + else + { + TLLM_CHECK(false); + } + } + else if (mWeightTypeId == WeightTypeId::INT4) + { + if (mType == nvinfer1::DataType::kHALF) + { + m_weightOnlyGemmRunner = std::make_shared< + CutlassFpAIntBGemmRunner<half, cutlass::uint4b_t, cutlass::WeightOnlyQuantOp::PER_COLUMN_SCALE_ONLY>>(); + mCudaKernelEnabled = tensorrt_llm::kernels::weight_only::is_supported( + mArch, tensorrt_llm::kernels::weight_only::KernelType::FP16Int4PerChannel); + mCudaKernelType = tensorrt_llm::kernels::weight_only::KernelType::FP16Int4PerChannel; + } +#if defined(ENABLE_BF16) + else if (mType == nvinfer1::DataType::kBF16) + { + m_weightOnlyGemmRunner = std::make_shared<CutlassFpAIntBGemmRunner<__nv_bfloat16, cutlass::uint4b_t, + cutlass::WeightOnlyQuantOp::PER_COLUMN_SCALE_ONLY>>(); + mCudaKernelEnabled = tensorrt_llm::kernels::weight_only::is_supported( + mArch, tensorrt_llm::kernels::weight_only::KernelType::BF16Int4PerChannel); + mCudaKernelType = tensorrt_llm::kernels::weight_only::KernelType::BF16Int4PerChannel; + } +#endif + else + { + TLLM_CHECK(false); + } + } + else + { + TLLM_CHECK(false); + } + + mPluginProfiler->setWeightTypeId(mWeightTypeId); + if (mCudaKernelEnabled) + { + mPluginProfiler->setCudaKernelType(mCudaKernelType, mArch); + } + mGemmId = GemmIdCore(mDims.n, mDims.k, mType); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* WeightOnlyQuantMatmulPlugin::clone() const noexcept +{ + auto* plugin = new WeightOnlyQuantMatmulPlugin(*this); + return plugin; +} + +void WeightOnlyQuantMatmulPlugin::configGemm() +{ + mPluginProfiler->profileTactics(m_weightOnlyGemmRunner, mType, mDims, mGemmId, mCudaKernelEnabled); +} + +nvinfer1::DimsExprs WeightOnlyQuantMatmulPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + // input [m1, m2, m3, ... , k] + // weight [k, n] for int8, [k, n/2] for int4 + + try + { + TLLM_CHECK(nbInputs == 3); + TLLM_CHECK(outputIndex == 0); + int const nbDimsA = inputs[0].nbDims; + int const nbDimsB = inputs[1].nbDims; + TLLM_CHECK(nbDimsA >= 2); + TLLM_CHECK(nbDimsB == 2); + DimsExprs ret; + ret.nbDims = nbDimsA; + for (int ii = 0; ii < nbDimsA - 1; ++ii) + { + ret.d[ii] = inputs[0].d[ii]; + } + if (mWeightTypeId == WeightTypeId::INT8) + { + // int8 weight only quant + ret.d[nbDimsA - 1] = exprBuilder.constant(inputs[1].d[1]->getConstantValue()); + } + else + { + // int4 weight only quant + ret.d[nbDimsA - 1] = exprBuilder.constant(inputs[1].d[1]->getConstantValue() * INT8_INT4_RATIO); + } + return ret; + } + catch (std::exception const& e) + { + caughtError(e); + } + return DimsExprs{}; +} + +bool WeightOnlyQuantMatmulPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + switch (pos) + { + case 0: + // activation + return inOut[0].type == mType && inOut[0].format == TensorFormat::kLINEAR; + case 1: + // weights + // Weights are required to be int8, but will be reinterpreted as int4 in enqueue if required + // Weights stored in checkpoint should have int8/int4 type + return inOut[1].type == nvinfer1::DataType::kINT8 && inOut[1].format == TensorFormat::kLINEAR; + case 2: + // scales channels + return inOut[2].type == mType && inOut[2].format == TensorFormat::kLINEAR; + case 3: + // out + return inOut[3].type == mType && inOut[3].format == TensorFormat::kLINEAR; + default: + // Never should be here + assert(false); + return false; + } +} + +void WeightOnlyQuantMatmulPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ + auto const minM = std::accumulate(in[0].min.d, in[0].min.d + in[0].min.nbDims - 1, 1, std::multiplies<int>()); + auto const maxM = std::accumulate(in[0].max.d, in[0].max.d + in[0].max.nbDims - 1, 1, std::multiplies<int>()); + + int const maxK = in[0].max.d[in[0].max.nbDims - 1]; + int const maxN = in[1].max.d[1] * getWeightTypeMultiplier(mWeightTypeId); + + auto const K = maxK; + auto const N = maxN / getWeightTypeMultiplier(mWeightTypeId); + + if (!mDims.isInitialized()) + { + mDims = {minM, maxM, N, K}; + } + + mGemmId = {N, K, mType}; + + m_workspaceMaxSize = m_weightOnlyGemmRunner->getWorkspaceSize(maxM, maxN, maxK); +} + +size_t WeightOnlyQuantMatmulPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + return m_workspaceMaxSize; +} + +int WeightOnlyQuantMatmulPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + // inputs + // mat1 [M1, M2,..., K] + // mat2 [K, N] for int8, [K, N/2] for int4 + // scale_channels [N] + // outputs + // mat [M, N] + + int64_t m64 = 1; + for (int ii = 0; ii < inputDesc[0].dims.nbDims - 1; ++ii) + { + m64 *= inputDesc[0].dims.d[ii]; + } + int const m = TLLM_INT32_CAST(m64); + int const n = TLLM_INT32_CAST(inputDesc[1].dims.d[1]); + int const k = TLLM_INT32_CAST(inputDesc[0].dims.d[inputDesc[0].dims.nbDims - 1]); + + if (m == 0) + return 0; + +#if defined(ENABLE_BF16) + TLLM_CHECK_WITH_INFO(mType == nvinfer1::DataType::kHALF || mType == nvinfer1::DataType::kBF16, + "No valid weightOnlyQuantMatmul configuration"); +#else + TLLM_CHECK_WITH_INFO(mType == nvinfer1::DataType::kHALF, "No valid weightOnlyQuantMatmul configuration"); +#endif + int real_n = mWeightTypeId == WeightTypeId::INT4 ? n * INT8_INT4_RATIO : n; + + // get best tactic and check if CUDA kernel should be used + bool use_cuda_kernel = false; + auto const& bestTactic = mPluginProfiler->getBestConfig(m, mGemmId); + TLLM_CHECK_WITH_INFO(bestTactic, + "No valid weight only per-channel GEMM tactic(It is usually caused by the failure to execute all candidate " + "configurations of the CUTLASS kernel, please pay attention to the warning information when building the " + "engine.)"); + use_cuda_kernel = bestTactic->enableCudaKernel; + if (use_cuda_kernel) + { + void const* cuda_kernel_act_ptr = inputs[0]; + void const* cuda_kernel_weight_ptr = inputs[1]; + void const* cuda_kernel_scales_ptr = inputs[2]; + void* cuda_kernel_out_ptr = outputs[0]; + tensorrt_llm::kernels::weight_only::Params params(cuda_kernel_act_ptr, nullptr, cuda_kernel_weight_ptr, + cuda_kernel_scales_ptr, nullptr, nullptr, cuda_kernel_out_ptr, 1.f, m, real_n, k, 0, mCudaKernelType); + tensorrt_llm::kernels::weight_only::kernel_launcher(mArch, params, stream); + } + else + { + int const ws_size = m_weightOnlyGemmRunner->getWorkspaceSize(m, real_n, k); + + m_weightOnlyGemmRunner->gemm(inputs[0], inputs[1], inputs[2], outputs[0], m, real_n, k, *bestTactic, + reinterpret_cast<char*>(workspace), ws_size, stream); + } + + return 0; +} + +// IPluginV2Ext Methods +nvinfer1::DataType WeightOnlyQuantMatmulPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + TLLM_CHECK(index == 0); + return mType; +} + +// IPluginV2 Methods + +char const* WeightOnlyQuantMatmulPlugin::getPluginType() const noexcept +{ + return WOQ_MATMUL_PLUGIN_NAME; +} + +char const* WeightOnlyQuantMatmulPlugin::getPluginVersion() const noexcept +{ + return WOQ_MATMUL_PLUGIN_VERSION; +} + +int WeightOnlyQuantMatmulPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int WeightOnlyQuantMatmulPlugin::initialize() noexcept +{ + configGemm(); + return 0; +} + +void WeightOnlyQuantMatmulPlugin::terminate() noexcept {} + +size_t WeightOnlyQuantMatmulPlugin::getSerializationSize() const noexcept +{ + return sizeof(mWeightTypeId) + // mWeightTypeId + sizeof(nvinfer1::DataType) + // mType + sizeof(mDims) + // Dimensions + mPluginProfiler->getSerializationSize(mGemmId); // selected tactics container size +} + +void WeightOnlyQuantMatmulPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast<char*>(buffer), *a = d; + write(d, mType); + write(d, mWeightTypeId); + write(d, mDims); + + mPluginProfiler->serialize(d, mGemmId); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void WeightOnlyQuantMatmulPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +/////////////// + +WeightOnlyQuantMatmulPluginCreator::WeightOnlyQuantMatmulPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("weight_type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* WeightOnlyQuantMatmulPluginCreator::getPluginName() const noexcept +{ + return WOQ_MATMUL_PLUGIN_NAME; +} + +char const* WeightOnlyQuantMatmulPluginCreator::getPluginVersion() const noexcept +{ + return WOQ_MATMUL_PLUGIN_VERSION; +} + +PluginFieldCollection const* WeightOnlyQuantMatmulPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* WeightOnlyQuantMatmulPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + nvinfer1::DataType type{}; + WeightTypeId weightTypeId{}; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "weight_type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + weightTypeId = static_cast<WeightTypeId>(*(static_cast<int const*>(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + TLLM_CHECK(fields[i].type == PluginFieldType::kINT32); + type = static_cast<nvinfer1::DataType>(*(static_cast<nvinfer1::DataType const*>(fields[i].data))); + } + } + try + { + // WeightOnlyGroupwiseQuantMatmulPluginCreator is unique and shared for an engine generation + // Create plugin profiler with shared tactics map + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ false); + auto* obj = new WeightOnlyQuantMatmulPlugin(type, weightTypeId, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} + +IPluginV2* WeightOnlyQuantMatmulPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call WeightOnlyQuantMatmulPlugin::destroy() + try + { + // Create plugin profiler with private tactics map which is read from the serialized engine + auto pluginProfiler = gemmPluginProfileManager.createGemmPluginProfiler(/* inference */ true); + auto* obj = new WeightOnlyQuantMatmulPlugin(serialData, serialLength, pluginProfiler); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + caughtError(e); + } + return nullptr; +} diff --git a/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/weightOnlyQuantMatmulPlugin.h b/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/weightOnlyQuantMatmulPlugin.h new file mode 100644 index 000000000000..3177d8297d2d --- /dev/null +++ b/cpp/tensorrt_llm/plugins/weightOnlyQuantMatmulPlugin/weightOnlyQuantMatmulPlugin.h @@ -0,0 +1,175 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/common/quantization.h" +#include "tensorrt_llm/kernels/cutlass_kernels/fpA_intB_gemm/fpA_intB_gemm.h" +#include "tensorrt_llm/kernels/weightOnlyBatchedGemv/kernelLauncher.h" +#include "tensorrt_llm/plugins/common/gemmPluginProfiler.h" +#include "tensorrt_llm/plugins/common/plugin.h" + +#include <cassert> +#include <cutlass/numeric_types.h> +#include <memory> +#include <set> +#include <string> +#include <vector> + +// The blank line here is to avoid clang-format -sort-includes option reordering these two cutlass header files and +// breaking dependencies +#include "cutlass/integer_subbyte.h" + +namespace tensorrt_llm::plugins +{ +enum class WeightTypeId +{ + INT8 = 1, + INT4 = 2, +}; + +constexpr int32_t FP16_BITS = 16; +constexpr int32_t INT8_BITS = 8; +constexpr int32_t INT4_BITS = 4; +constexpr int32_t INT8_INT4_RATIO = INT8_BITS / INT4_BITS; +constexpr int32_t FP16_INT4_RATIO = FP16_BITS / INT4_BITS; +constexpr int32_t FP16_INT8_RATIO = FP16_BITS / INT8_BITS; + +inline int32_t getWeightTypeMultiplier(WeightTypeId weightTypeId) +{ + return weightTypeId == WeightTypeId::INT8 ? 1 : INT8_INT4_RATIO; +} + +using WeightOnlyGemmRunner = tensorrt_llm::kernels::cutlass_kernels::CutlassFpAIntBGemmRunnerInterface; +using WeightOnlyGemmRunnerPtr = std::shared_ptr<WeightOnlyGemmRunner>; +using KernelType = tensorrt_llm::kernels::weight_only::KernelType; + +class WeightOnlyQuantGemmPluginProfiler : public GemmPluginProfiler<tensorrt_llm::cutlass_extensions::CutlassGemmConfig, + WeightOnlyGemmRunnerPtr, GemmIdCore, GemmIdCoreHash> +{ +public: + using Config = tensorrt_llm::cutlass_extensions::CutlassGemmConfig; + + void setWeightTypeId(WeightTypeId weightId) + { + mWeightTypeId = weightId; + } + + void setCudaKernelType(KernelType cudaKernelType, int arch) + { + mCudaKernelType = cudaKernelType; + mArch = arch; + } + +protected: + void runTactic(int m, int n, int k, Config const& tactic, char* workspace, cudaStream_t const& stream) override; + + void computeTmpSize(size_t maxM, size_t n, size_t k) override; + + std::vector<Config> getTactics(int m, int n, int k) const override; + + bool checkTactic(int m, int n, int k, Config const& tactic) const override; + +private: + WeightTypeId mWeightTypeId; + KernelType mCudaKernelType; + int mArch; +}; + +class WeightOnlyQuantMatmulPlugin : public BasePlugin +{ +public: + using PluginProfilerPtr = std::shared_ptr<WeightOnlyQuantGemmPluginProfiler>; + WeightOnlyQuantMatmulPlugin() = delete; + + WeightOnlyQuantMatmulPlugin(nvinfer1::DataType type, WeightTypeId weightTypeId, PluginProfilerPtr const& profiler); + + WeightOnlyQuantMatmulPlugin(void const* data, size_t length, PluginProfilerPtr const& profiler); + + ~WeightOnlyQuantMatmulPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + +private: + void init(nvinfer1::DataType type, WeightTypeId weightTypeId); + + void configGemm(); + +private: + const std::string mLayerName; + + WeightOnlyGemmRunnerPtr m_weightOnlyGemmRunner; + size_t m_workspaceMaxSize; + nvinfer1::DataType mType; + WeightTypeId mWeightTypeId; + bool mCudaKernelEnabled; + tensorrt_llm::kernels::weight_only::KernelType mCudaKernelType; + int mArch; + + GemmDims mDims{}; + GemmIdCore mGemmId{}; + + PluginProfilerPtr mPluginProfiler; +}; + +class WeightOnlyQuantMatmulPluginCreator : public BaseCreator +{ +public: + WeightOnlyQuantMatmulPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + +private: + GemmPluginProfilerManager<WeightOnlyQuantGemmPluginProfiler> gemmPluginProfileManager; + static nvinfer1::PluginFieldCollection mFC; + static std::vector<nvinfer1::PluginField> mPluginAttributes; +}; + +} // namespace tensorrt_llm::plugins diff --git a/cpp/tensorrt_llm/runtime/CMakeLists.txt b/cpp/tensorrt_llm/runtime/CMakeLists.txt index 11a9391c0e69..ca81fbb0f6cd 100644 --- a/cpp/tensorrt_llm/runtime/CMakeLists.txt +++ b/cpp/tensorrt_llm/runtime/CMakeLists.txt @@ -26,6 +26,7 @@ set(SRCS eagleBuffers.cpp explicitDraftTokensBuffers.cpp lookaheadBuffers.cpp + layerProfiler.cpp loraManager.cpp loraUtils.cpp loraModule.cpp @@ -50,6 +51,9 @@ set(SRCS promptTuningParams.cpp runtimeKernels.cu tllmBuffers.cpp + tllmRuntime.cpp + tllmStreamReaders.cpp + tllmLogger.cpp workerPool.cpp worldConfig.cpp virtualMemory.cpp) diff --git a/cpp/tensorrt_llm/runtime/bufferManager.cpp b/cpp/tensorrt_llm/runtime/bufferManager.cpp index 58257516a387..3de42a253158 100644 --- a/cpp/tensorrt_llm/runtime/bufferManager.cpp +++ b/cpp/tensorrt_llm/runtime/bufferManager.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tllmBuffers.h" #include <cstring> @@ -38,7 +37,7 @@ BufferManager::BufferManager(CudaStreamPtr stream, bool trimPool) mPool = CudaMemPool::getPrimaryPoolForDevice(mStream->getDevice()); } -BufferManager::IBufferPtr BufferManager::gpu(std::size_t size, tensorrt_llm::DataType type) const +BufferManager::IBufferPtr BufferManager::gpu(std::size_t size, nvinfer1::DataType type) const { if (auto vmAllocator = getVirtualMemoryAllocator()) { @@ -52,7 +51,7 @@ BufferManager::IBufferPtr BufferManager::gpu(std::size_t size, tensorrt_llm::Dat return gpuSync(size, type); } -BufferManager::ITensorPtr BufferManager::gpu(tensorrt_llm::Dims dims, tensorrt_llm::DataType type) const +BufferManager::ITensorPtr BufferManager::gpu(nvinfer1::Dims dims, nvinfer1::DataType type) const { if (auto vmAllocator = getVirtualMemoryAllocator()) { @@ -66,7 +65,7 @@ BufferManager::ITensorPtr BufferManager::gpu(tensorrt_llm::Dims dims, tensorrt_l return gpuSync(dims, type); } -BufferManager::IBufferPtr BufferManager::gpuSync(std::size_t size, tensorrt_llm::DataType type) +BufferManager::IBufferPtr BufferManager::gpuSync(std::size_t size, nvinfer1::DataType type) { if (auto vmAllocator = getVirtualMemoryAllocator()) { @@ -75,7 +74,7 @@ BufferManager::IBufferPtr BufferManager::gpuSync(std::size_t size, tensorrt_llm: return std::make_unique<StaticDeviceBuffer>(size, type, CudaAllocator{}); } -BufferManager::ITensorPtr BufferManager::gpuSync(tensorrt_llm::Dims dims, tensorrt_llm::DataType type) +BufferManager::ITensorPtr BufferManager::gpuSync(nvinfer1::Dims dims, nvinfer1::DataType type) { if (auto vmAllocator = getVirtualMemoryAllocator()) { @@ -84,48 +83,47 @@ BufferManager::ITensorPtr BufferManager::gpuSync(tensorrt_llm::Dims dims, tensor return std::make_unique<StaticDeviceTensor>(dims, type, CudaAllocator{}); } -BufferManager::IBufferPtr BufferManager::cpu(std::size_t size, tensorrt_llm::DataType type) +BufferManager::IBufferPtr BufferManager::cpu(std::size_t size, nvinfer1::DataType type) { return std::make_unique<HostBuffer>(size, type); } -BufferManager::ITensorPtr BufferManager::cpu(tensorrt_llm::Dims dims, tensorrt_llm::DataType type) +BufferManager::ITensorPtr BufferManager::cpu(nvinfer1::Dims dims, nvinfer1::DataType type) { return std::make_unique<HostTensor>(dims, type); } -BufferManager::IBufferPtr BufferManager::pinned(std::size_t size, tensorrt_llm::DataType type) +BufferManager::IBufferPtr BufferManager::pinned(std::size_t size, nvinfer1::DataType type) { return std::make_unique<PinnedBuffer>(size, type); } -BufferManager::ITensorPtr BufferManager::pinned(tensorrt_llm::Dims dims, tensorrt_llm::DataType type) +BufferManager::ITensorPtr BufferManager::pinned(nvinfer1::Dims dims, nvinfer1::DataType type) { return std::make_unique<PinnedTensor>(dims, type); } -BufferManager::IBufferPtr BufferManager::pinnedPool(std::size_t size, tensorrt_llm::DataType type) +BufferManager::IBufferPtr BufferManager::pinnedPool(std::size_t size, nvinfer1::DataType type) { return std::make_unique<PinnedPoolBuffer>(size, type); } -BufferManager::ITensorPtr BufferManager::pinnedPool(tensorrt_llm::Dims dims, tensorrt_llm::DataType type) +BufferManager::ITensorPtr BufferManager::pinnedPool(nvinfer1::Dims dims, nvinfer1::DataType type) { return std::make_unique<PinnedPoolTensor>(dims, type); } -BufferManager::IBufferPtr BufferManager::managed(std::size_t size, tensorrt_llm::DataType type) +BufferManager::IBufferPtr BufferManager::managed(std::size_t size, nvinfer1::DataType type) { return std::make_unique<UVMBuffer>(size, type); } -BufferManager::ITensorPtr BufferManager::managed(tensorrt_llm::Dims dims, tensorrt_llm::DataType type) +BufferManager::ITensorPtr BufferManager::managed(nvinfer1::Dims dims, nvinfer1::DataType type) { return std::make_unique<UVMTensor>(dims, type); } -BufferManager::ITensorPtr BufferManager::ipcNvls( - std::set<int> ranks, tensorrt_llm::Dims dims, tensorrt_llm::DataType type) +BufferManager::ITensorPtr BufferManager::ipcNvls(std::set<int> ranks, nvinfer1::Dims dims, nvinfer1::DataType type) { return std::make_unique<MulticastTensor>(dims, type, ranks); } @@ -189,7 +187,7 @@ void BufferManager::copy(IBuffer const& src, IBuffer& dst) const } BufferManager::IBufferPtr BufferManager::allocate( - MemoryType memoryType, std::size_t size, tensorrt_llm::DataType type) const + MemoryType memoryType, std::size_t size, nvinfer1::DataType type) const { switch (memoryType) { @@ -204,7 +202,7 @@ BufferManager::IBufferPtr BufferManager::allocate( } BufferManager::ITensorPtr BufferManager::allocate( - MemoryType memoryType, tensorrt_llm::Dims dims, tensorrt_llm::DataType type) const + MemoryType memoryType, nvinfer1::Dims dims, nvinfer1::DataType type) const { switch (memoryType) { diff --git a/cpp/tensorrt_llm/runtime/bufferView.h b/cpp/tensorrt_llm/runtime/bufferView.h index a001d05f1f8b..236b89d7d455 100644 --- a/cpp/tensorrt_llm/runtime/bufferView.h +++ b/cpp/tensorrt_llm/runtime/bufferView.h @@ -17,7 +17,6 @@ #pragma once #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/iBuffer.h" #include <string> @@ -71,7 +70,7 @@ class BufferView : virtual public IBuffer return mBuffer->getCapacity() - mOffset; } - [[nodiscard]] tensorrt_llm::DataType getDataType() const override + [[nodiscard]] nvinfer1::DataType getDataType() const override { return mBuffer->getDataType(); } diff --git a/cpp/tensorrt_llm/runtime/decoderState.cpp b/cpp/tensorrt_llm/runtime/decoderState.cpp index 83037b2431cb..b5851dc1c2d2 100644 --- a/cpp/tensorrt_llm/runtime/decoderState.cpp +++ b/cpp/tensorrt_llm/runtime/decoderState.cpp @@ -16,7 +16,6 @@ #include "tensorrt_llm/runtime/decoderState.h" #include "tensorrt_llm/batch_manager/llmRequest.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/decodingCommon.h" #include "tensorrt_llm/runtime/runtimeKernels.h" @@ -28,10 +27,10 @@ using TensorPtr = DecoderState::TensorPtr; BeamSearchBuffers::BeamSearchBuffers(BufferManager const& bufferManager) : mOutputBeamHypotheses{} - , mCumLogProbsTmp(bufferManager.emptyTensor(MemoryType::kGPU, tensorrt_llm::DataType::kFLOAT)) + , mCumLogProbsTmp(bufferManager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kFLOAT)) { mOutputBeamHypotheses.empty(bufferManager); - mCumLogProbsTmp = bufferManager.emptyTensor(MemoryType::kGPU, tensorrt_llm::DataType::kFLOAT); + mCumLogProbsTmp = bufferManager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kFLOAT); int device; cudaGetDevice(&device); @@ -55,8 +54,8 @@ DecoderState::DecoderState() } void DecoderState::setup(SizeType32 maxNumSequences, SizeType32 maxBeamWidth, SizeType32 maxAttentionWindow, - SizeType32 sinkTokenLength, SizeType32 maxSequenceLength, tensorrt_llm::DataType dtype, - ModelConfig const& modelConfig, WorldConfig const& worldConfig, BufferManager const& bufferManager) + SizeType32 sinkTokenLength, SizeType32 maxSequenceLength, nvinfer1::DataType dtype, ModelConfig const& modelConfig, + WorldConfig const& worldConfig, BufferManager const& bufferManager) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); setupBuffers(dtype, bufferManager); @@ -65,7 +64,7 @@ void DecoderState::setup(SizeType32 maxNumSequences, SizeType32 maxBeamWidth, Si TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); } -void DecoderState::setupBuffers(tensorrt_llm::DataType dtype, BufferManager const& bufferManager) +void DecoderState::setupBuffers(nvinfer1::DataType dtype, BufferManager const& bufferManager) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); auto constexpr nvTokenIdType = TRTDataType<TokenIdType>::value; @@ -115,7 +114,7 @@ void DecoderState::setupBuffers(tensorrt_llm::DataType dtype, BufferManager cons } void DecoderState::setupSpeculativeDecoding(SpeculativeDecodingMode const& speculativeDecodingMode, - SizeType32 maxTokensPerEngineStep, tensorrt_llm::DataType dtype, ModelConfig const& modelConfig, + SizeType32 maxTokensPerEngineStep, nvinfer1::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig, BufferManager const& bufferManager) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); @@ -125,8 +124,8 @@ void DecoderState::setupSpeculativeDecoding(SpeculativeDecodingMode const& specu TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); } -void DecoderState::setupSpeculativeDecodingBuffers(SpeculativeDecodingMode const speculativeDecodingMode, - tensorrt_llm::DataType dtype, BufferManager const& bufferManager) +void DecoderState::setupSpeculativeDecodingBuffers( + SpeculativeDecodingMode const speculativeDecodingMode, nvinfer1::DataType dtype, BufferManager const& bufferManager) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); @@ -152,13 +151,13 @@ void DecoderState::setupSpeculativeDecodingBuffers(SpeculativeDecodingMode const if (speculativeDecodingMode.predictsDraftTokens()) { speculativeDecodingOutputs.nextDraftTokens - = bufferManager.emptyTensor(MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + = bufferManager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); if (speculativeDecodingMode.variableDraftLength()) { speculativeDecodingOutputs.nextDraftTokensLen - = bufferManager.emptyTensor(MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + = bufferManager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); speculativeDecodingOutputs.prevDraftTokensLen - = bufferManager.emptyTensor(MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + = bufferManager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); } } if (speculativeDecodingMode.isLookaheadDecoding()) @@ -168,11 +167,11 @@ void DecoderState::setupSpeculativeDecodingBuffers(SpeculativeDecodingMode const if (speculativeDecodingMode.needsKVCacheRewind()) { speculativeDecodingOutputs.acceptedTokensLen - = bufferManager.emptyTensor(MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + = bufferManager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); speculativeDecodingOutputs.acceptedLengthsCumSum - = bufferManager.emptyTensor(MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + = bufferManager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); speculativeDecodingOutputs.pathsOffsets - = bufferManager.emptyTensor(MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + = bufferManager.emptyTensor(MemoryType::kGPU, nvinfer1::DataType::kINT32); } dOutput->speculativeDecodingOutputs = speculativeDecodingOutputs; diff --git a/cpp/tensorrt_llm/runtime/decodingLayerWorkspace.cpp b/cpp/tensorrt_llm/runtime/decodingLayerWorkspace.cpp index c5098bf777e0..f8a4fa4e7467 100644 --- a/cpp/tensorrt_llm/runtime/decodingLayerWorkspace.cpp +++ b/cpp/tensorrt_llm/runtime/decodingLayerWorkspace.cpp @@ -15,12 +15,11 @@ */ #include "tensorrt_llm/runtime/decodingLayerWorkspace.h" -#include "tensorrt_llm/common/tllmDataType.h" #include <utility> tensorrt_llm::runtime::DecodingLayerWorkspace::DecodingLayerWorkspace(std::shared_ptr<BufferManager> bufferManager, - tensorrt_llm::layers::DecoderDomain const& decoderDomain, tensorrt_llm::DataType logitsType, + tensorrt_llm::layers::DecoderDomain const& decoderDomain, nvinfer1::DataType logitsType, size_t workspaceBufferSizeInBytes) : mBufferManager(std::move(bufferManager)) , mBatchSlotsDevice( @@ -83,8 +82,7 @@ void tensorrt_llm::runtime::DecodingLayerWorkspace::resize(size_t minSize) } tensorrt_llm::runtime::DecodingLayerWorkspace::TensorPtr -tensorrt_llm::runtime::DecodingLayerWorkspace::getWorkspaceAsDeviceTensor( - ITensor::Shape shape, tensorrt_llm::DataType type) +tensorrt_llm::runtime::DecodingLayerWorkspace::getWorkspaceAsDeviceTensor(ITensor::Shape shape, nvinfer1::DataType type) { auto const sizeInBytes = ITensor::volume(shape) * BufferDataType(type).getSize(); return std::make_shared<GenericTensor<BorrowingAllocator<MemoryType::kGPU>>>( diff --git a/cpp/tensorrt_llm/runtime/decodingLayerWorkspace.h b/cpp/tensorrt_llm/runtime/decodingLayerWorkspace.h index 68d3d54124f5..c2688b51139f 100644 --- a/cpp/tensorrt_llm/runtime/decodingLayerWorkspace.h +++ b/cpp/tensorrt_llm/runtime/decodingLayerWorkspace.h @@ -19,7 +19,6 @@ #include <memory> #include "tensorrt_llm/common/dataType.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/common/workspace.h" #include "tensorrt_llm/layers/decodingParams.h" #include "tensorrt_llm/runtime/bufferManager.h" @@ -40,7 +39,7 @@ class DecodingLayerWorkspace using BufferPtr = IBuffer::SharedPtr; DecodingLayerWorkspace(std::shared_ptr<BufferManager> bufferManager, layers::DecoderDomain const& decoderDomain, - tensorrt_llm::DataType logitsType, size_t workspaceBufferSizeInBytes); + nvinfer1::DataType logitsType, size_t workspaceBufferSizeInBytes); DecodingLayerWorkspace() = delete; @@ -72,7 +71,7 @@ class DecodingLayerWorkspace [[nodiscard]] TensorPtr getDeviceRuntimeLogits() const; ///@brief Gets a tensor with the given shape and type at the start of the device workspace. - TensorPtr getWorkspaceAsDeviceTensor(ITensor::Shape shape, tensorrt_llm::DataType type); + TensorPtr getWorkspaceAsDeviceTensor(ITensor::Shape shape, nvinfer1::DataType type); /// @brief A convenience function to copy the content of a standard vector to a device workspace. template <typename T, typename Alloc> @@ -113,7 +112,7 @@ class DecodingLayerWorkspace { size_t lastTensorOffset = 0; auto alignedSizeCalculator - = [&lastTensorOffset](std::pair<ITensor::Shape, tensorrt_llm::DataType> const& tensorDescriptor) + = [&lastTensorOffset](std::pair<ITensor::Shape, nvinfer1::DataType> const& tensorDescriptor) { auto const& [shape, type] = tensorDescriptor; auto const sizeInBytes = ITensor::volume(shape) * tensorrt_llm::common::getDTypeSize(type); diff --git a/cpp/tensorrt_llm/runtime/eagleBuffers.cpp b/cpp/tensorrt_llm/runtime/eagleBuffers.cpp index e0f2198c3e58..097fd95f49aa 100644 --- a/cpp/tensorrt_llm/runtime/eagleBuffers.cpp +++ b/cpp/tensorrt_llm/runtime/eagleBuffers.cpp @@ -19,7 +19,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/speculativeDecoding/eagleDecodingKernels.h" #include "tensorrt_llm/kernels/speculativeDecoding/explicitDraftTokensKernels.h" #include "tensorrt_llm/runtime/common.h" @@ -42,51 +41,50 @@ void EagleBuffers::Inputs::create(SizeType32 maxNumSequences, BufferManager cons auto const numEagleLayers = speculativeDecodingModule.getMaxDraftPathLen(); auto constexpr TRTTokenIdType = runtime::TRTDataType<runtime::TokenIdType>::value; - temperatures = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kFLOAT); - randomDataSample = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kFLOAT); + temperatures = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kFLOAT); + randomDataSample = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kFLOAT); randomDataValidation - = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingTokens}), tensorrt_llm::DataType::kFLOAT); + = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingTokens}), nvinfer1::DataType::kFLOAT); draftTokens = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingDraftTokens}), TRTTokenIdType); - draftLens = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + draftLens = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); draftPaths - = manager.gpu(ITensor::makeShape({maxNumSequences, maxNumPaths, maxPathLen}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences, maxNumPaths, maxPathLen}), nvinfer1::DataType::kINT32); draftPathsHost = BufferManager::pinnedPool( - ITensor::makeShape({maxNumSequences, maxNumPaths, maxPathLen}), tensorrt_llm::DataType::kINT32); - specDecodingGenerationLengths = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxNumSequences, maxNumPaths, maxPathLen}), nvinfer1::DataType::kINT32); + specDecodingGenerationLengths = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); specDecodingGenerationLengthsHost - = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); specDecodingPackedMasks = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingTokens, common::ceilDiv(maxDecodingTokens, 32)}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); specDecodingPositionOffsets - = manager.gpu(ITensor::makeShape({maxNumSequences * maxDecodingTokens}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences * maxDecodingTokens}), nvinfer1::DataType::kINT32); eagleNetCtxRequestTypesHost - = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); eagleNetCtxContextLengthsHost - = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); eagleNetCtxPastKeyValueLengthsHost - = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); eagleNetGenRequestTypesHost - = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); eagleNetGenContextLengthsHost - = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); eagleNetGenPastKeyValueLengthsHost - = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); inputGenTokensHost = BufferManager::pinnedPool( - ITensor::makeShape({maxNumSequences * maxDecodingTokens}), tensorrt_llm::DataType::kINT32); - chunkedContextNextTokens = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); - useSpecDecoding = manager.cpu(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxNumSequences * maxDecodingTokens}), nvinfer1::DataType::kINT32); + chunkedContextNextTokens = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); + useSpecDecoding = manager.cpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); // Eagle-2 - useDynamicTreeHost = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); - dynamicTreeMaxTopKHost = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); - prevScores - = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingDraftTokens}), tensorrt_llm::DataType::kFLOAT); + useDynamicTreeHost = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + dynamicTreeMaxTopKHost = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + prevScores = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingDraftTokens}), nvinfer1::DataType::kFLOAT); currentExpandIndices = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingDraftTokens}), TRTTokenIdType); allLayersScores = manager.gpu( ITensor::makeShape({maxNumSequences, numEagleLayers, maxDecodingDraftTokens * maxDecodingDraftTokens}), - tensorrt_llm::DataType::kFLOAT); + nvinfer1::DataType::kFLOAT); allLayersDraftTokenIds = manager.gpu( ITensor::makeShape({maxNumSequences, numEagleLayers, maxDecodingDraftTokens * maxDecodingDraftTokens}), TRTTokenIdType); @@ -116,63 +114,58 @@ EagleBuffers::EagleBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, run auto constexpr TRTTokenIdType = runtime::TRTDataType<runtime::TokenIdType>::value; // input tensors - engineInputs.temperatures = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kFLOAT); - engineInputs.posteriorAlpha = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kFLOAT); - engineInputs.posteriorThreshold = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kFLOAT); - posteriorAlphaHost = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kFLOAT); - posteriorThresholdHost = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kFLOAT); - greedySamplingHost = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + engineInputs.temperatures = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kFLOAT); + engineInputs.posteriorAlpha = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kFLOAT); + engineInputs.posteriorThreshold = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kFLOAT); + posteriorAlphaHost = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kFLOAT); + posteriorThresholdHost = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kFLOAT); + greedySamplingHost = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); engineInputs.draftTokens = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingDraftTokens}), TRTTokenIdType); - engineInputs.draftLens = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + engineInputs.draftLens = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); engineInputs.draftPaths - = manager.gpu(ITensor::makeShape({maxNumSequences, numPaths, pathLen}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences, numPaths, pathLen}), nvinfer1::DataType::kINT32); engineInputs.specDecodingGenerationLengths - = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); engineInputs.specDecodingPositionOffsets - = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineInputs.specDecodingPackedMasks - = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineInputs.specDecodingPackedMasks = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); - engineInputs.randomDataSample = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kFLOAT); - engineInputs.randomDataValidation = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kFLOAT); + engineInputs.randomDataSample = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kFLOAT); + engineInputs.randomDataValidation = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kFLOAT); engineInputs.eagleNetCtxRequestTypesHost - = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kINT32); + = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kINT32); engineInputs.eagleNetCtxContextLengthsHost - = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kINT32); + = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kINT32); engineInputs.eagleNetCtxPastKeyValueLengthsHost - = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kINT32); + = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kINT32); engineInputs.eagleNetGenRequestTypesHost - = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kINT32); + = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kINT32); engineInputs.eagleNetGenContextLengthsHost - = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kINT32); + = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kINT32); engineInputs.eagleNetGenPastKeyValueLengthsHost - = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kINT32); - engineInputs.inputGenTokensHost - = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kINT32); - engineInputs.chunkedContextNextTokens - = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineInputs.useSpecDecoding = BufferManager::cpu(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kINT32); + engineInputs.inputGenTokensHost = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kINT32); + engineInputs.chunkedContextNextTokens = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineInputs.useSpecDecoding = BufferManager::cpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); bufferCast<SizeType32>(*engineInputs.useSpecDecoding)[0] = 1; - chunkedContextNextTokensHost - = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, tensorrt_llm::DataType::kINT32); + chunkedContextNextTokensHost = manager.emptyTensor(runtime::MemoryType::kPINNEDPOOL, nvinfer1::DataType::kINT32); // Eagle-2 - engineInputs.useDynamicTreeHost - = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + engineInputs.useDynamicTreeHost = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); engineInputs.dynamicTreeMaxTopKHost - = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); engineInputs.prevScores - = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingDraftTokens}), tensorrt_llm::DataType::kFLOAT); + = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingDraftTokens}), nvinfer1::DataType::kFLOAT); engineInputs.currentExpandIndices = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingDraftTokens}), TRTTokenIdType); engineInputs.allLayersScores = manager.gpu( ITensor::makeShape({maxNumSequences, numEagleLayers, maxDecodingDraftTokens * maxDecodingDraftTokens}), - tensorrt_llm::DataType::kFLOAT); + nvinfer1::DataType::kFLOAT); engineInputs.allLayersDraftTokenIds = manager.gpu( ITensor::makeShape({maxNumSequences, numEagleLayers, maxDecodingDraftTokens * maxDecodingDraftTokens}), TRTTokenIdType); @@ -183,24 +176,24 @@ EagleBuffers::EagleBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, run // output tensors engineOutputs.nextDraftTokens = manager.gpu(ITensor::makeShape({maxNumSequences, numPaths, pathLen}), TRTTokenIdType); - engineOutputs.nextDraftLens = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + engineOutputs.nextDraftLens = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); engineOutputs.nextDraftPaths - = manager.gpu(ITensor::makeShape({maxNumSequences, numPaths, pathLen}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences, numPaths, pathLen}), nvinfer1::DataType::kINT32); engineOutputs.acceptedTokens - = manager.gpu(ITensor::makeShape({maxNumSequences, pathLen}), tensorrt_llm::DataType::kINT32); - engineOutputs.acceptedLens = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); - engineOutputs.acceptedPaths = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences, pathLen}), nvinfer1::DataType::kINT32); + engineOutputs.acceptedLens = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); + engineOutputs.acceptedPaths = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); engineOutputs.chunkedContextNextTokens - = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); // helper tensors scanReduceTempStorageBytes = tksd::invokeScanReduceGenerationLengths( maxNumSequences, nullptr, nullptr, 0, nullptr, nullptr, manager.getStream().get()); scanReduceTempStorage = manager.gpu(scanReduceTempStorageBytes); - cumSumGenerationLengths = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - maxGenerationLength = manager.gpu(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + cumSumGenerationLengths = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + maxGenerationLength = manager.gpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); // pre-allocate empty tensors reshape(0, maxNumSequences, modelConfig); @@ -527,15 +520,15 @@ void EagleBuffers::setFromInputs(RequestVector const& contextRequests, RequestVe switch (dtype) { - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: setFromInputs<float>( contextRequests, genRequests, vocabSizePadded, seqSlots, draftBuffers, *eagleModule, manager); break; - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: setFromInputs<half>( contextRequests, genRequests, vocabSizePadded, seqSlots, draftBuffers, *eagleModule, manager); break; - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: setFromInputs<__nv_bfloat16>( contextRequests, genRequests, vocabSizePadded, seqSlots, draftBuffers, *eagleModule, manager); break; diff --git a/cpp/tensorrt_llm/runtime/explicitDraftTokensBuffers.cpp b/cpp/tensorrt_llm/runtime/explicitDraftTokensBuffers.cpp index 89c74e6f9349..ed205ca0e117 100644 --- a/cpp/tensorrt_llm/runtime/explicitDraftTokensBuffers.cpp +++ b/cpp/tensorrt_llm/runtime/explicitDraftTokensBuffers.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/speculativeDecoding/explicitDraftTokensKernels.h" #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/iBuffer.h" @@ -41,24 +40,23 @@ void ExplicitDraftTokensBuffers::Inputs::create(SizeType32 maxNumSequences, Buff auto constexpr TRTTokenIdType = runtime::TRTDataType<runtime::TokenIdType>::value; auto const dtype = modelConfig.getDataType(); - maxGenLengthHost = manager.pinned(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + maxGenLengthHost = manager.pinned(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); temperatures = manager.gpu(ITensor::makeShape({maxNumSequences}), dtype); - positionIdsBase = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); - generationLengths = manager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); - generationLengthsHost = manager.pinned(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32); + positionIdsBase = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); + generationLengths = manager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); + generationLengthsHost = manager.pinned(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32); randomDataSample = manager.gpu(ITensor::makeShape({maxNumSequences}), dtype); randomDataValidation = manager.gpu(ITensor::makeShape({maxNumSequences, maxNumPaths, maxDraftPathLen}), dtype); draftTokens = manager.gpu(ITensor::makeShape({maxNumSequences, maxNumPaths, maxPathLen}), TRTTokenIdType); draftIndices - = manager.gpu(ITensor::makeShape({maxNumSequences, maxNumPaths, maxPathLen}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences, maxNumPaths, maxPathLen}), nvinfer1::DataType::kINT32); draftProbs = manager.gpu(ITensor::makeShape({maxNumSequences, maxNumPaths, maxDraftPathLen, vocabSizePadded}), dtype); packedMasks = manager.gpu(ITensor::makeShape({maxNumSequences, maxDecodingTokens, common::ceilDiv(maxDecodingTokens, 32)}), - tensorrt_llm::DataType::kINT32); - positionIds - = manager.gpu(ITensor::makeShape({maxNumSequences * maxDecodingTokens}), tensorrt_llm::DataType::kINT32); - useSpecDecoding = manager.cpu(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); + positionIds = manager.gpu(ITensor::makeShape({maxNumSequences * maxDecodingTokens}), nvinfer1::DataType::kINT32); + useSpecDecoding = manager.cpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); } ExplicitDraftTokensBuffers::ExplicitDraftTokensBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, @@ -83,53 +81,52 @@ ExplicitDraftTokensBuffers::ExplicitDraftTokensBuffers(SizeType32 maxBatchSize, auto const dtype = modelConfig.getDataType(); // input tensors - engineInputs.requestTypesDevice = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + engineInputs.requestTypesDevice = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); engineInputs.temperatures = manager.emptyTensor(runtime::MemoryType::kGPU, dtype); engineInputs.draftTokens = manager.gpu(ITensor::makeShape({maxNumSequences, numBeams, beamLength}), TRTTokenIdType); engineInputs.draftIndices - = manager.gpu(ITensor::makeShape({maxNumSequences, numBeams, beamLength}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences, numBeams, beamLength}), nvinfer1::DataType::kINT32); engineInputs.draftProbs = manager.gpu(ITensor::makeShape({maxNumSequences, numBeams, beamDraftLength, vocabSizePadded}), dtype); - engineInputs.generationLengths = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineInputs.positionIds = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineInputs.positionOffsets = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineInputs.packedMasks = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + engineInputs.generationLengths = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineInputs.positionIds = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineInputs.positionOffsets = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineInputs.packedMasks = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); engineInputs.randomDataSample = manager.emptyTensor(runtime::MemoryType::kGPU, dtype); engineInputs.randomDataValidation = manager.emptyTensor(runtime::MemoryType::kGPU, dtype); - engineInputs.positionIdsBase = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineInputs.useSpecDecoding = manager.cpu(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + engineInputs.positionIdsBase = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineInputs.useSpecDecoding = manager.cpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); bufferCast<SizeType32>(*engineInputs.useSpecDecoding)[0] = 1; // output tensors engineOutputs.nextDraftTokens = manager.gpu(ITensor::makeShape({maxNumSequences, numBeams, beamLength}), TRTTokenIdType); engineOutputs.nextDraftIndices - = manager.gpu(ITensor::makeShape({maxNumSequences, numBeams, beamLength}), tensorrt_llm::DataType::kINT32); + = manager.gpu(ITensor::makeShape({maxNumSequences, numBeams, beamLength}), nvinfer1::DataType::kINT32); engineOutputs.nextDraftProbs = manager.gpu(ITensor::makeShape({maxNumSequences, numBeams, beamDraftLength, vocabSizePadded}), dtype); - engineOutputs.maxGenToken = manager.gpu(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); - engineOutputs.totalGenToken = manager.gpu(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + engineOutputs.maxGenToken = manager.gpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + engineOutputs.totalGenToken = manager.gpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); - engineOutputs.nextGenerationLengths - = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineOutputs.nextPositionOffsets = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineOutputs.masks = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kBOOL); + engineOutputs.nextGenerationLengths = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineOutputs.nextPositionOffsets = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineOutputs.masks = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kBOOL); engineOutputs.nextFlatTokens = manager.emptyTensor(runtime::MemoryType::kGPU, TRTTokenIdType); - engineOutputs.bestPathLengths = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineOutputs.bestPathIndices = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); - engineOutputs.packedPositionIds = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + engineOutputs.bestPathLengths = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineOutputs.bestPathIndices = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); + engineOutputs.packedPositionIds = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); // helper tensors auto const& stream = manager.getStream(); scanTempStorageBytes = tksd::invokeScanGenerationLengths(nullptr, 0, nullptr, nullptr, maxNumSequences, stream.get()); scanTempStorage = manager.gpu(scanTempStorageBytes); - cumSumGenerationLengths = manager.emptyTensor(runtime::MemoryType::kGPU, tensorrt_llm::DataType::kINT32); + cumSumGenerationLengths = manager.emptyTensor(runtime::MemoryType::kGPU, nvinfer1::DataType::kINT32); // pre-allocate empty tensors reshape(0, maxNumSequences, modelConfig); @@ -298,15 +295,15 @@ void ExplicitDraftTokensBuffers::setFromInputs(SizeType32 numCtxSequences, SizeT switch (dtype) { - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: setFromInputs<float>(numCtxSequences, numGenSequences, vocabSizePadded, seqSlots, draftBuffers, contextPositionIds, *explicitDraftTokensModule, stream); break; - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: setFromInputs<half>(numCtxSequences, numGenSequences, vocabSizePadded, seqSlots, draftBuffers, contextPositionIds, *explicitDraftTokensModule, stream); break; - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: setFromInputs<__nv_bfloat16>(numCtxSequences, numGenSequences, vocabSizePadded, seqSlots, draftBuffers, contextPositionIds, *explicitDraftTokensModule, stream); break; diff --git a/cpp/tensorrt_llm/runtime/gptDecoder.cpp b/cpp/tensorrt_llm/runtime/gptDecoder.cpp index e1ac1717af45..930877206462 100644 --- a/cpp/tensorrt_llm/runtime/gptDecoder.cpp +++ b/cpp/tensorrt_llm/runtime/gptDecoder.cpp @@ -21,7 +21,7 @@ #include "tensorrt_llm/layers/dynamicDecodeLayer.h" #include "tensorrt_llm/runtime/decodingLayerWorkspace.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <memory> @@ -121,7 +121,7 @@ void GptDecoder<T>::disableLookahead( template <typename T> void GptDecoder<T>::setup(SamplingConfig const& samplingConfig, size_t batchSize, TensorConstPtr const& batchSlots, - std::optional<DecodingOutput> const& output, std::optional<tensorrt_llm::DataType> explicitDraftTokensDType, + std::optional<DecodingOutput> const& output, std::optional<nvinfer1::DataType> explicitDraftTokensDType, std::optional<std::vector<TensorConstPtr>> const& lookaheadPrompt, std::optional<std::vector<tle::LookaheadDecodingConfig>> const& lookaheadAlgoConfigs) { diff --git a/cpp/tensorrt_llm/runtime/gptDecoderBatched.cpp b/cpp/tensorrt_llm/runtime/gptDecoderBatched.cpp index 7b3a12ed7a2c..c55d02093afc 100644 --- a/cpp/tensorrt_llm/runtime/gptDecoderBatched.cpp +++ b/cpp/tensorrt_llm/runtime/gptDecoderBatched.cpp @@ -22,7 +22,6 @@ #include "tensorrt_llm/batch_manager/decoderBuffers.h" #include "tensorrt_llm/batch_manager/llmRequest.h" #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/kernels/decodingKernels.h" #include "tensorrt_llm/runtime/bufferManager.h" @@ -75,7 +74,7 @@ void GptDecoderBatched::disableLookahead(RequestVector const& genRequests, Tenso } void GptDecoderBatched::setup(executor::DecodingMode const& mode, SizeType32 maxNumSequences, SizeType32 maxBeamWidth, - tensorrt_llm::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig) + nvinfer1::DataType dtype, ModelConfig const& modelConfig, WorldConfig const& worldConfig) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); TLLM_CHECK(maxNumSequences > 0); diff --git a/cpp/tensorrt_llm/runtime/gptJsonConfig.cpp b/cpp/tensorrt_llm/runtime/gptJsonConfig.cpp index 47310a9a1282..311f63eaf1e7 100644 --- a/cpp/tensorrt_llm/runtime/gptJsonConfig.cpp +++ b/cpp/tensorrt_llm/runtime/gptJsonConfig.cpp @@ -20,7 +20,6 @@ #include "modelConfig.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/eagleModule.h" #include "tensorrt_llm/runtime/explicitDraftTokensModule.h" #include "tensorrt_llm/runtime/jsonSerialization.h" @@ -81,14 +80,14 @@ std::optional<FieldType> parseJsonFieldOptional(Json const& json, std::string_vi return value; } -tensorrt_llm::DataType strToDType(std::string type) +nvinfer1::DataType strToDType(std::string type) { - static std::map<std::string, tensorrt_llm::DataType> const typeMap = {{"int64", tensorrt_llm::DataType::kINT64}, - {"int32", tensorrt_llm::DataType::kINT32}, {"int", tensorrt_llm::DataType::kINT32}, - {"float32", tensorrt_llm::DataType::kFLOAT}, {"bfloat16", tensorrt_llm::DataType::kBF16}, - {"float16", tensorrt_llm::DataType::kHALF}, {"bool", tensorrt_llm::DataType::kBOOL}, - {"uint8", tensorrt_llm::DataType::kUINT8}, {"int8", tensorrt_llm::DataType::kINT8}, - {"fp8", tensorrt_llm::DataType::kFP8}, {"int4", tensorrt_llm::DataType::kINT4}}; + static std::map<std::string, nvinfer1::DataType> const typeMap = {{"int64", nvinfer1::DataType::kINT64}, + {"int32", nvinfer1::DataType::kINT32}, {"int", nvinfer1::DataType::kINT32}, + {"float32", nvinfer1::DataType::kFLOAT}, {"bfloat16", nvinfer1::DataType::kBF16}, + {"float16", nvinfer1::DataType::kHALF}, {"bool", nvinfer1::DataType::kBOOL}, + {"uint8", nvinfer1::DataType::kUINT8}, {"int8", nvinfer1::DataType::kINT8}, {"fp8", nvinfer1::DataType::kFP8}, + {"int4", nvinfer1::DataType::kINT4}}; TLLM_CHECK_WITH_INFO(typeMap.count(type) > 0, type + " not found in strToDtype."); return typeMap.at(type); @@ -141,14 +140,14 @@ std::vector<ModelConfig::LayerType> buildLayerTypes( return result; } -ModelConfig parseMultimodalConfig(Json const& json, tensorrt_llm::DataType dataType) +ModelConfig parseMultimodalConfig(Json const& json, nvinfer1::DataType dataType) { return ModelConfig{128, 10, 10, 0, 1, 128, dataType}; // use dummy values because vision engines of multimodal models does not record this info in config } ModelConfig createModelConfig(Json const& json, bool engineVersionNone, SizeType32 tensorParallelism, - SizeType32 contextParallelism, tensorrt_llm::DataType dataType) + SizeType32 contextParallelism, nvinfer1::DataType dataType) { auto const& config = engineVersionNone ? json.at("builder_config") : json.at("pretrained_config"); auto const multiModalName = parseJsonFieldOptional<std::string>(config, "model_name"); @@ -249,14 +248,14 @@ ModelConfig createModelConfig(Json const& json, bool engineVersionNone, SizeType modelConfig.setLayerTypes(layerTypes); // Set logits datatype - auto logitsDtype = tensorrt_llm::DataType::kFLOAT; + auto logitsDtype = nvinfer1::DataType::kFLOAT; if (logitsDtypeStr == "float32") { - logitsDtype = tensorrt_llm::DataType::kFLOAT; + logitsDtype = nvinfer1::DataType::kFLOAT; } else if (logitsDtypeStr == "float16") { - logitsDtype = tensorrt_llm::DataType::kHALF; + logitsDtype = nvinfer1::DataType::kHALF; } else { @@ -491,15 +490,15 @@ GptJsonConfig parseJson(InputType&& input) { if (precision == "float32") { - return tensorrt_llm::DataType::kFLOAT; + return nvinfer1::DataType::kFLOAT; } if (precision == "float16") { - return tensorrt_llm::DataType::kHALF; + return nvinfer1::DataType::kHALF; } if (precision == "bfloat16") { - return tensorrt_llm::DataType::kBF16; + return nvinfer1::DataType::kBF16; } TLLM_THROW("Model data type '%s' not supported", precision.c_str()); }(); diff --git a/cpp/tensorrt_llm/runtime/iBuffer.cpp b/cpp/tensorrt_llm/runtime/iBuffer.cpp index 82574b658b39..77707a0e4cf8 100644 --- a/cpp/tensorrt_llm/runtime/iBuffer.cpp +++ b/cpp/tensorrt_llm/runtime/iBuffer.cpp @@ -20,7 +20,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferView.h" #include <cuda_runtime_api.h> @@ -49,7 +48,7 @@ IBuffer::UniquePtr IBuffer::slice(IBuffer::SharedPtr buffer, std::size_t offset, return std::make_unique<BufferView>(std::move(buffer), offset, size); } -IBuffer::UniquePtr IBuffer::wrap(void* data, tensorrt_llm::DataType type, std::size_t size, std::size_t capacity) +IBuffer::UniquePtr IBuffer::wrap(void* data, nvinfer1::DataType type, std::size_t size, std::size_t capacity) { TLLM_CHECK_WITH_INFO(size <= capacity, "Requested size is larger than capacity"); auto memoryType = IBuffer::memoryType(data); @@ -92,17 +91,17 @@ char const* IBuffer::getDataTypeName(DataType dataType) { switch (dataType) { - case tensorrt_llm::DataType::kINT64: return DataTypeTraits<tensorrt_llm::DataType::kINT64>::name; - case tensorrt_llm::DataType::kINT32: return DataTypeTraits<tensorrt_llm::DataType::kINT32>::name; - case tensorrt_llm::DataType::kFLOAT: return DataTypeTraits<tensorrt_llm::DataType::kFLOAT>::name; - case tensorrt_llm::DataType::kBF16: return DataTypeTraits<tensorrt_llm::DataType::kBF16>::name; - case tensorrt_llm::DataType::kHALF: return DataTypeTraits<tensorrt_llm::DataType::kHALF>::name; - case tensorrt_llm::DataType::kBOOL: return DataTypeTraits<tensorrt_llm::DataType::kBOOL>::name; - case tensorrt_llm::DataType::kUINT8: return DataTypeTraits<tensorrt_llm::DataType::kUINT8>::name; - case tensorrt_llm::DataType::kINT8: return DataTypeTraits<tensorrt_llm::DataType::kINT8>::name; - case tensorrt_llm::DataType::kFP8: return DataTypeTraits<tensorrt_llm::DataType::kFP8>::name; - case tensorrt_llm::DataType::kINT4: [[fallthrough]] /* do nothing */; - case tensorrt_llm::DataType::kFP4: [[fallthrough]] /* do nothing */; + case nvinfer1::DataType::kINT64: return DataTypeTraits<nvinfer1::DataType::kINT64>::name; + case nvinfer1::DataType::kINT32: return DataTypeTraits<nvinfer1::DataType::kINT32>::name; + case nvinfer1::DataType::kFLOAT: return DataTypeTraits<nvinfer1::DataType::kFLOAT>::name; + case nvinfer1::DataType::kBF16: return DataTypeTraits<nvinfer1::DataType::kBF16>::name; + case nvinfer1::DataType::kHALF: return DataTypeTraits<nvinfer1::DataType::kHALF>::name; + case nvinfer1::DataType::kBOOL: return DataTypeTraits<nvinfer1::DataType::kBOOL>::name; + case nvinfer1::DataType::kUINT8: return DataTypeTraits<nvinfer1::DataType::kUINT8>::name; + case nvinfer1::DataType::kINT8: return DataTypeTraits<nvinfer1::DataType::kINT8>::name; + case nvinfer1::DataType::kFP8: return DataTypeTraits<nvinfer1::DataType::kFP8>::name; + case nvinfer1::DataType::kINT4: [[fallthrough]] /* do nothing */; + case nvinfer1::DataType::kFP4: [[fallthrough]] /* do nothing */; default: TLLM_THROW("Unknown data type"); } } diff --git a/cpp/tensorrt_llm/runtime/iTensor.cpp b/cpp/tensorrt_llm/runtime/iTensor.cpp index 70b31707130a..f78b25fdb19a 100644 --- a/cpp/tensorrt_llm/runtime/iTensor.cpp +++ b/cpp/tensorrt_llm/runtime/iTensor.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/memoryUtils.h" #include "tensorrt_llm/common/stringUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/tensorView.h" #include "tensorrt_llm/runtime/tllmBuffers.h" @@ -73,22 +72,22 @@ ITensor::UniquePtr ITensor::slice(SharedPtr tensor, Shape const& offsetDims, ITe return std::make_unique<TensorView>(std::move(tensor), offset, volume(dims), dims); } -ITensor::UniquePtr ITensor::view(IBuffer::SharedPtr buffer, tensorrt_llm::Dims const& dims) +ITensor::UniquePtr ITensor::view(IBuffer::SharedPtr buffer, nvinfer1::Dims const& dims) { auto const size = buffer->getSize(); return std::make_unique<TensorView>(std::move(buffer), 0, size, dims); } -tensorrt_llm::Dims ITensor::makeShape(std::initializer_list<ITensor::DimType64> const& dims) +nvinfer1::Dims ITensor::makeShape(std::initializer_list<ITensor::DimType64> const& dims) { - TLLM_CHECK_WITH_INFO(dims.size() <= tensorrt_llm::Dims::MAX_DIMS, "Number of dimensions is too large"); - tensorrt_llm::Dims shape{}; + TLLM_CHECK_WITH_INFO(dims.size() <= nvinfer1::Dims::MAX_DIMS, "Number of dimensions is too large"); + nvinfer1::Dims shape{}; shape.nbDims = static_cast<decltype(Shape::nbDims)>(dims.size()); std::copy(dims.begin(), dims.end(), shape.d); return shape; } -std::string ITensor::toString(tensorrt_llm::Dims const& dims) +std::string ITensor::toString(nvinfer1::Dims const& dims) { if (dims.nbDims < 0) { @@ -104,8 +103,7 @@ std::string ITensor::toString(tensorrt_llm::Dims const& dims) } } -ITensor::UniquePtr ITensor::wrap( - void* data, tensorrt_llm::DataType type, tensorrt_llm::Dims const& shape, std::size_t capacity) +ITensor::UniquePtr ITensor::wrap(void* data, nvinfer1::DataType type, nvinfer1::Dims const& shape, std::size_t capacity) { auto const size = volumeNonNegative(shape); TLLM_CHECK_WITH_INFO(size <= capacity, "Requested size is larger than capacity"); @@ -232,18 +230,18 @@ std::ostream& tensorrt_llm::runtime::operator<<(std::ostream& out, ITensor const { switch (tensor.getDataType()) { - case tensorrt_llm::DataType::kFLOAT: printTensor<float>(tensor, out); break; - case tensorrt_llm::DataType::kHALF: printTensor<half, float>(tensor, out); break; - case tensorrt_llm::DataType::kBOOL: printTensor<bool>(tensor, out); break; - case tensorrt_llm::DataType::kINT8: printTensor<std::int8_t, std::int32_t>(tensor, out); break; - case tensorrt_llm::DataType::kINT32: printTensor<std::int32_t>(tensor, out); break; - case tensorrt_llm::DataType::kINT64: printTensor<std::int64_t>(tensor, out); break; - case tensorrt_llm::DataType::kUINT8: printTensor<std::uint8_t, std::int32_t>(tensor, out); break; + case nvinfer1::DataType::kFLOAT: printTensor<float>(tensor, out); break; + case nvinfer1::DataType::kHALF: printTensor<half, float>(tensor, out); break; + case nvinfer1::DataType::kBOOL: printTensor<bool>(tensor, out); break; + case nvinfer1::DataType::kINT8: printTensor<std::int8_t, std::int32_t>(tensor, out); break; + case nvinfer1::DataType::kINT32: printTensor<std::int32_t>(tensor, out); break; + case nvinfer1::DataType::kINT64: printTensor<std::int64_t>(tensor, out); break; + case nvinfer1::DataType::kUINT8: printTensor<std::uint8_t, std::int32_t>(tensor, out); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: printTensor<__nv_bfloat16, float>(tensor, out); break; + case nvinfer1::DataType::kBF16: printTensor<__nv_bfloat16, float>(tensor, out); break; #endif #ifdef ENABLE_FP8 - case tensorrt_llm::DataType::kFP8: printTensor<__nv_fp8_e4m3, float>(tensor, out); break; + case nvinfer1::DataType::kFP8: printTensor<__nv_fp8_e4m3, float>(tensor, out); break; #endif default: TLLM_THROW("Unsupported data type"); } diff --git a/cpp/tensorrt_llm/runtime/ipcUtils.cpp b/cpp/tensorrt_llm/runtime/ipcUtils.cpp index 48368844f850..23a7e28a4f27 100644 --- a/cpp/tensorrt_llm/runtime/ipcUtils.cpp +++ b/cpp/tensorrt_llm/runtime/ipcUtils.cpp @@ -20,7 +20,7 @@ #include "tensorrt_llm/common/workspace.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntimeBase.h> #include <cstddef> namespace tensorrt_llm::runtime @@ -83,7 +83,7 @@ void IpcMemory::allocateIpcMemory(std::size_t bufferSize, BufferManager const& m // IPC handles. If we want to support stream-ordered allocations here, we need to create another pool with the // correct handle type. auto const ipcAlignedBufferSize = common::alignSize(bufferSize, 1LU << 21); - mBuffer = BufferManager::gpuSync(ipcAlignedBufferSize, tensorrt_llm::DataType::kUINT8); + mBuffer = BufferManager::gpuSync(ipcAlignedBufferSize, nvinfer1::DataType::kUINT8); manager.setZero(*mBuffer); auto* bufferPtr = mBuffer->data(); @@ -149,7 +149,7 @@ AllReduceBuffers::AllReduceBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWi { auto const tpSize = worldConfig.getTensorParallelism(); mAllReduceCommPtrs = BufferManager::cpu( - ITensor::makeShape({static_cast<SizeType32>(7) * tpSize + 3}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({static_cast<SizeType32>(7) * tpSize + 3}), nvinfer1::DataType::kINT64); } else { @@ -178,7 +178,7 @@ AllReduceBuffers::AllReduceBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWi mAllReduceCommPtrs = BufferManager::cpu(ITensor::makeShape({static_cast<SizeType32>(mIpcMemoryHandles.size()) * tpSize + 3}), - tensorrt_llm::DataType::kINT64); + nvinfer1::DataType::kINT64); auto commPtrs = BufferRange<void*>(*mAllReduceCommPtrs); // Start from 1 since 0 represents released state for barrier at the beginning of the all_reduce. // The last element is the barrier flag counter. @@ -211,9 +211,9 @@ AllReduceBuffers::AllReduceBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWi void lamportInitializeAll(void* buffer_0, void* buffer_1, void* buffer_2, size_t size) { #if ENABLE_MULTI_DEVICE - tensorrt_llm::kernels::lamportInitialize(buffer_0, size / sizeof(half), tensorrt_llm::DataType::kHALF, 0); - tensorrt_llm::kernels::lamportInitialize(buffer_1, size / sizeof(half), tensorrt_llm::DataType::kHALF, 0); - tensorrt_llm::kernels::lamportInitialize(buffer_2, size / sizeof(half), tensorrt_llm::DataType::kHALF, 0); + tensorrt_llm::kernels::lamportInitialize(buffer_0, size / sizeof(half), nvinfer1::DataType::kHALF, 0); + tensorrt_llm::kernels::lamportInitialize(buffer_1, size / sizeof(half), nvinfer1::DataType::kHALF, 0); + tensorrt_llm::kernels::lamportInitialize(buffer_2, size / sizeof(half), nvinfer1::DataType::kHALF, 0); cudaDeviceSynchronize(); #endif } diff --git a/cpp/tensorrt_llm/runtime/layerProfiler.cpp b/cpp/tensorrt_llm/runtime/layerProfiler.cpp new file mode 100644 index 000000000000..4c3c9779cedb --- /dev/null +++ b/cpp/tensorrt_llm/runtime/layerProfiler.cpp @@ -0,0 +1,97 @@ +/* + * Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/runtime/layerProfiler.h" +#include <iomanip> +#include <iostream> +#include <numeric> +#include <sstream> + +using namespace tensorrt_llm::runtime; + +void LayerProfiler::reportLayerTime(char const* layerName, float timeMs) noexcept +{ + if (mIterator == mLayers.end()) + { + bool const first = !mLayers.empty() && mLayers.begin()->name == layerName; + mUpdatesCount += mLayers.empty() || first; + if (first) + { + mIterator = mLayers.begin(); + } + else + { + mLayers.emplace_back(); + mLayers.back().name = layerName; + mIterator = mLayers.end() - 1; + } + } + + mIterator->timeMs.push_back(timeMs); + ++mIterator; +} + +float LayerProfiler::getTotalTime() const noexcept +{ + auto const plusLayerTime = [](float accumulator, LayerProfile const& lp) + { return accumulator + std::accumulate(lp.timeMs.begin(), lp.timeMs.end(), 0.F, std::plus<float>()); }; + return std::accumulate(mLayers.begin(), mLayers.end(), 0.0F, plusLayerTime); +} + +std::string LayerProfiler::getLayerProfile() noexcept +{ + std::string const nameHdr(" Layer"); + std::string const timeHdr(" Time(ms)"); + + float const totalTimeMs = getTotalTime(); + + auto const timeLength = timeHdr.size(); + + std::unordered_map<std::string, float> layer2times; + std::vector<std::string> layer_order; + for (auto const& p : mLayers) + { + if (!layer2times.count(p.name)) + { + layer2times[p.name] = 0; + layer_order.push_back(p.name); + } + for (auto const& t : p.timeMs) + { + layer2times[p.name] += t; + } + } + + std::stringstream ss; + ss << "\n=== Per-layer Profile ===\n" << timeHdr << nameHdr << "\n"; + + for (auto const& name : layer_order) + { + if (layer2times[name] == 0.0f) + { + continue; + } + ss << std::setw(timeLength) << std::fixed << std::setprecision(2) << layer2times[name] << " " << name << "\n"; + } + + ss << std::setw(timeLength) << std::fixed << std::setprecision(2) << totalTimeMs << " Total\n"; + ss << "\n"; + + // clear data + mLayers.clear(); + + return ss.str(); +} diff --git a/cpp/tensorrt_llm/runtime/layerProfiler.h b/cpp/tensorrt_llm/runtime/layerProfiler.h new file mode 100644 index 000000000000..bcae1546de3d --- /dev/null +++ b/cpp/tensorrt_llm/runtime/layerProfiler.h @@ -0,0 +1,47 @@ +/* + * Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/runtime/common.h" +#include <vector> + +#include <NvInfer.h> + +namespace tensorrt_llm::runtime +{ +struct LayerProfile +{ + std::string name; + std::vector<float> timeMs; +}; + +class LayerProfiler : public nvinfer1::IProfiler +{ + +public: + void reportLayerTime(char const* layerName, float timeMs) noexcept override; + + std::string getLayerProfile() noexcept; + +private: + [[nodiscard]] float getTotalTime() const noexcept; + + std::vector<LayerProfile> mLayers; + std::vector<LayerProfile>::iterator mIterator{mLayers.begin()}; + int32_t mUpdatesCount{0}; +}; +} // namespace tensorrt_llm::runtime diff --git a/cpp/tensorrt_llm/runtime/lookaheadBuffers.cpp b/cpp/tensorrt_llm/runtime/lookaheadBuffers.cpp index 5e77046c47e0..ef800ef218e4 100644 --- a/cpp/tensorrt_llm/runtime/lookaheadBuffers.cpp +++ b/cpp/tensorrt_llm/runtime/lookaheadBuffers.cpp @@ -16,23 +16,208 @@ */ #include "tensorrt_llm/runtime/lookaheadBuffers.h" -#include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include "tensorrt_llm/layers/lookaheadDecodingUtils.h" namespace tensorrt_llm::runtime { LookaheadDecodingBuffers::LookaheadDecodingBuffers( SizeType32 maxNumSequences, SizeType32 maxTokensPerStep, BufferManager const& bufferManager) - : generationLengths(bufferManager.gpu(ITensor::makeShape({maxNumSequences}), tensorrt_llm::DataType::kINT32)) + : generationLengths(bufferManager.gpu(ITensor::makeShape({maxNumSequences}), nvinfer1::DataType::kINT32)) , positionOffsets( - bufferManager.gpu(ITensor::makeShape({maxNumSequences, maxTokensPerStep}), tensorrt_llm::DataType::kINT32)) + bufferManager.gpu(ITensor::makeShape({maxNumSequences, maxTokensPerStep}), nvinfer1::DataType::kINT32)) , packedMasks(bufferManager.gpu(ITensor::makeShape({maxNumSequences, maxTokensPerStep, static_cast<ITensor::DimType64>(common::divUp(maxTokensPerStep, 32))}), - tensorrt_llm::DataType::kINT32)) + nvinfer1::DataType::kINT32)) , positionIds( - bufferManager.gpu(ITensor::makeShape({maxNumSequences, maxTokensPerStep}), tensorrt_llm::DataType::kINT32)) + bufferManager.gpu(ITensor::makeShape({maxNumSequences, maxTokensPerStep}), nvinfer1::DataType::kINT32)) { } +LookaheadRuntimeBuffers::LookaheadRuntimeBuffers(SizeType32 maxBatchSize, SizeType32 maxBeamWidth, + BufferManager const& manager, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + executor::DecodingConfig const& /* decodingConfig */, TllmRuntime const& runtime) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + TLLM_CHECK_WITH_INFO(maxBeamWidth == 1, "Lookahead decoding does not support beam search"); + + auto const tokensPerStep = modelConfig.getMaxDecodingTokens(); + auto const numPackedMasks = static_cast<ITensor::DimType64>(tensorrt_llm::common::divUp(tokensPerStep, 32)); + + cumSumLength = manager.pinned(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + + packedMasksDevice + = manager.gpu(ITensor::makeShape({maxBatchSize * tokensPerStep, numPackedMasks}), nvinfer1::DataType::kINT32); + positionOffsetsDevice = manager.gpu(ITensor::makeShape({maxBatchSize, tokensPerStep}), nvinfer1::DataType::kINT32); + generationLengthsDevice = manager.gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + positionIdsDevice = manager.gpu(ITensor::makeShape({maxBatchSize, tokensPerStep}), nvinfer1::DataType::kINT32); + + packedMaskHost = manager.cpu(packedMasksDevice->getShape(), nvinfer1::DataType::kINT32); + positionOffsetsHost = manager.cpu(positionOffsetsDevice->getShape(), nvinfer1::DataType::kINT32); + generationLengthsHost = manager.cpu(generationLengthsDevice->getShape(), nvinfer1::DataType::kINT32); + positionIdsHost = manager.cpu(positionIdsDevice->getShape(), nvinfer1::DataType::kINT32); + + packedMaskHostCopy = manager.cpu(packedMasksDevice->getShape(), nvinfer1::DataType::kINT32); + positionOffsetsHostCopy = manager.cpu(positionOffsetsDevice->getShape(), nvinfer1::DataType::kINT32); + generationLengthsHostCopy = manager.cpu(generationLengthsDevice->getShape(), nvinfer1::DataType::kINT32); + positionIdsHostCopy = manager.cpu(positionIdsDevice->getShape(), nvinfer1::DataType::kINT32); + + batchSlotsHostCopy = manager.cpu(generationLengthsDevice->getShape(), nvinfer1::DataType::kINT32); + + useSpecDecoding = manager.cpu(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + bufferCast<SizeType32>(*useSpecDecoding)[0] = 1; + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void LookaheadRuntimeBuffers::setFromInputs(SizeType32 numCtxSequences, SizeType32 numGenSequences, + ITensor const& requestTypes, ITensor const& seqSlots, LookaheadDecodingBuffers const& decoderLookaheadBuffers, + TllmRuntime const& runtime, ModelConfig const& modelConfig, WorldConfig const& worldConfig) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const& manager = runtime.getBufferManager(); + + auto const tokensPerStep = modelConfig.getMaxDecodingTokens(); + + manager.copy(seqSlots, *batchSlotsHostCopy); + manager.copy(*decoderLookaheadBuffers.generationLengths, *generationLengthsHostCopy); + manager.copy(*decoderLookaheadBuffers.positionOffsets, *positionOffsetsHostCopy); + manager.copy(*decoderLookaheadBuffers.packedMasks, *packedMaskHostCopy); + manager.copy(*decoderLookaheadBuffers.positionIds, *positionIdsHostCopy); + + manager.getStream().synchronize(); + + BufferRange<SizeType32 const> batchSlotsRange(*batchSlotsHostCopy); + BufferRange<SizeType32> cumSumLengthRange(*cumSumLength); + + SizeType32 maxGenerationLength = 0; + for (SizeType32 bi = 0; bi < numGenSequences; bi++) + { + SizeType32 gbi = batchSlotsRange[bi + numCtxSequences]; + SizeType32 theLength = BufferRange<SizeType32>(*generationLengthsHostCopy)[gbi]; + maxGenerationLength = std::max(maxGenerationLength, theLength); + } + + auto positionOffsetShape = positionOffsetsHost->getShape(); + positionOffsetShape.d[1] = maxGenerationLength; + positionOffsetsHost->reshape(positionOffsetShape); + positionOffsetsDevice->reshape(positionOffsetShape); + + auto positionIdsShape = positionIdsHostCopy->getShape(); + auto positionIdsShape1D = ITensor::makeShape({ITensor::volume(positionIdsShape)}); + positionIdsHostCopy->reshape(positionIdsShape1D); + positionIdsHost->reshape(positionIdsShape1D); + + cumSumLengthRange[0] = 0; + for (SizeType32 bi = 0; bi < numGenSequences; bi++) + { + SizeType32 gbi = batchSlotsRange[bi + numCtxSequences]; + SizeType32 theLength = BufferRange<SizeType32>(*generationLengthsHostCopy)[gbi]; + + manager.copy(*ITensor::at(generationLengthsHostCopy, {gbi}), *ITensor::at(generationLengthsHost, {bi})); + + manager.copy(*ITensor::slice(positionOffsetsHostCopy, {gbi, 0}, theLength), + *ITensor::slice(positionOffsetsHost, {bi, 0}, theLength)); + + manager.copy(*ITensor::slice(packedMaskHostCopy, gbi * tokensPerStep, theLength), + *ITensor::slice(packedMaskHost, cumSumLengthRange[0], theLength)); + + manager.copy(*ITensor::slice(positionIdsHostCopy, gbi * tokensPerStep, theLength), + *ITensor::slice(positionIdsHost, cumSumLengthRange[0], theLength)); + + cumSumLengthRange[0] += theLength; + } + + positionIdsHostCopy->reshape(positionIdsShape); + positionIdsHost->reshape(positionIdsShape); + positionIdsDevice->reshape(positionIdsShape); + + manager.copy(*ITensor::slice(generationLengthsHost, 0, numGenSequences), + *ITensor::slice(generationLengthsDevice, 0, numGenSequences)); + manager.copy(*ITensor::slice(positionOffsetsHost, 0, numGenSequences), + *ITensor::slice(positionOffsetsDevice, 0, numGenSequences)); + manager.copy(*ITensor::slice(packedMaskHost, 0, numGenSequences * tokensPerStep), + *ITensor::slice(packedMasksDevice, 0, numGenSequences * tokensPerStep)); + manager.copy( + *ITensor::slice(positionIdsHost, 0, numGenSequences), *ITensor::slice(positionIdsDevice, 0, numGenSequences)); + positionIdsDevice->reshape(ITensor::makeShape({cumSumLengthRange[0]})); + + manager.getStream().synchronize(); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void LookaheadRuntimeBuffers::reshape(SizeType32 numCtxSequences, SizeType32 numGenSequences, SizeType32 tokensPerStep) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const numSequences = numGenSequences; + + auto packedMaskShape = packedMasksDevice->getShape(); + packedMaskShape.d[0] = numSequences * tokensPerStep; + packedMasksDevice->reshape(packedMaskShape); + packedMaskHost->reshape(packedMaskShape); + + auto generationLengthsShape = generationLengthsDevice->getShape(); + generationLengthsShape.d[0] = numSequences; + generationLengthsDevice->reshape(generationLengthsShape); + generationLengthsHost->reshape(generationLengthsShape); + + auto positionOffsetsShape = positionOffsetsDevice->getShape(); + positionOffsetsShape.d[0] = numSequences; + positionOffsetsDevice->reshape(positionOffsetsShape); + positionOffsetsHost->reshape(positionOffsetsShape); + + auto positionIdsShape = positionIdsDevice->getShape(); + positionIdsShape.d[0] = numSequences; + positionIdsDevice->reshape(positionIdsShape); + positionIdsHost->reshape(positionIdsShape); + + auto batchSlotsShape = batchSlotsHostCopy->getShape(); + batchSlotsShape.d[0] = numCtxSequences + numGenSequences; + batchSlotsHostCopy->reshape(batchSlotsShape); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void LookaheadRuntimeBuffers::enableLookaheadDecoding(SizeType32 maxBatchSize, SizeType32 tokensPerStep) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + auto const numPackedMasks = static_cast<ITensor::DimType64>(tensorrt_llm::common::divUp(tokensPerStep, 32)); + packedMasksDevice->reshape(ITensor::makeShape({maxBatchSize * tokensPerStep, numPackedMasks})); + generationLengthsDevice->reshape(ITensor::makeShape({maxBatchSize})); + positionOffsetsDevice->reshape(ITensor::makeShape({maxBatchSize, tokensPerStep})); + bufferCast<SizeType32>(*useSpecDecoding)[0] = 1; + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void LookaheadRuntimeBuffers::disableLookaheadDecoding() +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + packedMasksDevice->reshape(ITensor::makeShape({1, 1})); + generationLengthsDevice->reshape(ITensor::makeShape({1})); + positionOffsetsDevice->reshape(ITensor::makeShape({1, 1})); + bufferCast<SizeType32>(*useSpecDecoding)[0] = 0; + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void LookaheadRuntimeBuffers::insertInputTensors( + TensorMap& inputBuffers, TensorMap& /* outputBuffers */, WorldConfig const& /* worldConfig */) const +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + inputBuffers.insert_or_assign("spec_decoding_packed_mask", packedMasksDevice); + inputBuffers.insert_or_assign("spec_decoding_generation_lengths", generationLengthsDevice); + inputBuffers.insert_or_assign("spec_decoding_position_offsets", positionOffsetsDevice); + inputBuffers.insert_or_assign("spec_decoding_use", useSpecDecoding); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + } // namespace tensorrt_llm::runtime diff --git a/cpp/tensorrt_llm/runtime/loraCache.cpp b/cpp/tensorrt_llm/runtime/loraCache.cpp index 36fb0363816f..3dbb814f058b 100644 --- a/cpp/tensorrt_llm/runtime/loraCache.cpp +++ b/cpp/tensorrt_llm/runtime/loraCache.cpp @@ -23,7 +23,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/loraUtils.h" #include <memory> #include <mutex> @@ -538,15 +537,15 @@ void LoraCache::splitTransposeCpu(ITensor& output, ITensor const& input, SizeTyp switch (input.getDataType()) { - case tensorrt_llm::DataType::kINT32: splitTransposeCpuInner<SizeType32>(output, input, tpSize, tpRank); break; - case tensorrt_llm::DataType::kFLOAT: splitTransposeCpuInner<float>(output, input, tpSize, tpRank); break; - case tensorrt_llm::DataType::kHALF: splitTransposeCpuInner<half>(output, input, tpSize, tpRank); break; - case tensorrt_llm::DataType::kINT8: splitTransposeCpuInner<int8_t>(output, input, tpSize, tpRank); break; + case nvinfer1::DataType::kINT32: splitTransposeCpuInner<SizeType32>(output, input, tpSize, tpRank); break; + case nvinfer1::DataType::kFLOAT: splitTransposeCpuInner<float>(output, input, tpSize, tpRank); break; + case nvinfer1::DataType::kHALF: splitTransposeCpuInner<half>(output, input, tpSize, tpRank); break; + case nvinfer1::DataType::kINT8: splitTransposeCpuInner<int8_t>(output, input, tpSize, tpRank); break; #ifdef ENABLE_FP8 - case tensorrt_llm::DataType::kFP8: splitTransposeCpuInner<__nv_fp8_e4m3>(output, input, tpSize, tpRank); break; + case nvinfer1::DataType::kFP8: splitTransposeCpuInner<__nv_fp8_e4m3>(output, input, tpSize, tpRank); break; #endif // ENABLE_FP8 #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: splitTransposeCpuInner<__nv_bfloat16>(output, input, tpSize, tpRank); break; + case nvinfer1::DataType::kBF16: splitTransposeCpuInner<__nv_bfloat16>(output, input, tpSize, tpRank); break; #endif // ENABLE_BF16 default: TLLM_CHECK_WITH_INFO(false, "data type not supported"); } diff --git a/cpp/tensorrt_llm/runtime/loraManager.cpp b/cpp/tensorrt_llm/runtime/loraManager.cpp index 1d25ea20c8e4..8d7ebe389853 100644 --- a/cpp/tensorrt_llm/runtime/loraManager.cpp +++ b/cpp/tensorrt_llm/runtime/loraManager.cpp @@ -26,6 +26,8 @@ #include "tensorrt_llm/runtime/utils/runtimeUtils.h" #include "tensorrt_llm/runtime/worldConfig.h" +#include <NvInferRuntime.h> + namespace tensorrt_llm::runtime { diff --git a/cpp/tensorrt_llm/runtime/loraUtils.cpp b/cpp/tensorrt_llm/runtime/loraUtils.cpp index da7f1475ddec..3c5e95162474 100644 --- a/cpp/tensorrt_llm/runtime/loraUtils.cpp +++ b/cpp/tensorrt_llm/runtime/loraUtils.cpp @@ -17,7 +17,6 @@ #include "tensorrt_llm/runtime/loraUtils.h" #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/modelConfig.h" @@ -58,7 +57,7 @@ void loraValidateRequestTensorDims(std::optional<ITensor::SharedPtr> const& optR keys->getShape().d[0] == expectedBatchSize, "Expected batch dimension to be 1 for each lora request"); TLLM_CHECK_WITH_INFO(weights->getMemoryType() != MemoryType::kGPU, "Expected lora weights to be in CPU memory"); TLLM_CHECK_WITH_INFO(keys->getMemoryType() != MemoryType::kGPU, "Expected lora weights to be in CPU memory"); - TLLM_CHECK_WITH_INFO(keys->getDataType() == tensorrt_llm::DataType::kINT32, + TLLM_CHECK_WITH_INFO(keys->getDataType() == nvinfer1::DataType::kINT32, "Expected lora keys to have TYPE_INT32 but was " + std::string(keys->getDataTypeName())); TLLM_CHECK_WITH_INFO(keys->getShape().d[1] == weights->getShape().d[1], diff --git a/cpp/tensorrt_llm/runtime/ncclCommunicator.cpp b/cpp/tensorrt_llm/runtime/ncclCommunicator.cpp index 7edfba42f935..b76efb75952a 100644 --- a/cpp/tensorrt_llm/runtime/ncclCommunicator.cpp +++ b/cpp/tensorrt_llm/runtime/ncclCommunicator.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/envUtils.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/ipcNvlsMemory.h" #include "tensorrt_llm/runtime/utils/multiDeviceUtils.h" @@ -103,19 +102,19 @@ void initNcclCommProbeWithTimeout(ncclUniqueId const& id, int worldSize, int ran } } -ncclDataType_t toNcclType(tensorrt_llm::DataType dataType) +ncclDataType_t toNcclType(nvinfer1::DataType dataType) { switch (dataType) { - case tensorrt_llm::DataType::kFLOAT: return ncclFloat32; - case tensorrt_llm::DataType::kHALF: return ncclHalf; - case tensorrt_llm::DataType::kINT8: return ncclInt8; - case tensorrt_llm::DataType::kINT32: return ncclInt32; - case tensorrt_llm::DataType::kUINT8: return ncclUint8; - case tensorrt_llm::DataType::kINT64: return ncclInt64; - case tensorrt_llm::DataType::kFP8: return ncclUint8; + case nvinfer1::DataType::kFLOAT: return ncclFloat32; + case nvinfer1::DataType::kHALF: return ncclHalf; + case nvinfer1::DataType::kINT8: return ncclInt8; + case nvinfer1::DataType::kINT32: return ncclInt32; + case nvinfer1::DataType::kUINT8: return ncclUint8; + case nvinfer1::DataType::kINT64: return ncclInt64; + case nvinfer1::DataType::kFP8: return ncclUint8; #if ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: return ncclBfloat16; + case nvinfer1::DataType::kBF16: return ncclBfloat16; #endif // ENABLE_BF16 default: TLLM_THROW("Unsupported data type: %d", static_cast<int>(dataType)); } @@ -124,7 +123,7 @@ ncclDataType_t toNcclType(tensorrt_llm::DataType dataType) } // namespace void NcclCommunicator::send( - void const* sendbuff, size_t count, tensorrt_llm::DataType dataType, int peer, CudaStream const& stream) const + void const* sendbuff, size_t count, nvinfer1::DataType dataType, int peer, CudaStream const& stream) const { #if ENABLE_MULTI_DEVICE TLLM_NCCL_CHECK(ncclSend(sendbuff, count, toNcclType(dataType), peer, mComm, stream.get())); @@ -134,7 +133,7 @@ void NcclCommunicator::send( } void NcclCommunicator::receive( - void* sendbuff, size_t count, tensorrt_llm::DataType dataType, int peer, CudaStream const& stream) const + void* sendbuff, size_t count, nvinfer1::DataType dataType, int peer, CudaStream const& stream) const { #if ENABLE_MULTI_DEVICE TLLM_NCCL_CHECK(ncclRecv(sendbuff, count, toNcclType(dataType), peer, mComm, stream.get())); diff --git a/cpp/tensorrt_llm/runtime/ncclCommunicator.h b/cpp/tensorrt_llm/runtime/ncclCommunicator.h index 21d7f116e95b..76cce4beab8a 100644 --- a/cpp/tensorrt_llm/runtime/ncclCommunicator.h +++ b/cpp/tensorrt_llm/runtime/ncclCommunicator.h @@ -16,7 +16,6 @@ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/cudaStream.h" #include "tensorrt_llm/runtime/iBuffer.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" @@ -58,10 +57,9 @@ class NcclCommunicator private: void send( - void const* sendbuff, size_t count, tensorrt_llm::DataType dataType, int peer, CudaStream const& stream) const; + void const* sendbuff, size_t count, nvinfer1::DataType dataType, int peer, CudaStream const& stream) const; - void receive( - void* sendbuff, size_t count, tensorrt_llm::DataType dataType, int peer, CudaStream const& stream) const; + void receive(void* sendbuff, size_t count, nvinfer1::DataType dataType, int peer, CudaStream const& stream) const; static ncclComm_t createComm(int worldSize, int rank, mpi::MpiComm const& mpiComm); diff --git a/cpp/tensorrt_llm/runtime/runtimeKernels.cu b/cpp/tensorrt_llm/runtime/runtimeKernels.cu index b22d36052370..3b3dbcac894a 100644 --- a/cpp/tensorrt_llm/runtime/runtimeKernels.cu +++ b/cpp/tensorrt_llm/runtime/runtimeKernels.cu @@ -21,7 +21,7 @@ #include "tensorrt_llm/kernels/speculativeDecoding/kvCacheUpdateKernels.h" #include "tensorrt_llm/runtime/runtimeKernels.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntimeBase.h> #include <cuda_runtime.h> using namespace tensorrt_llm::runtime; @@ -333,13 +333,13 @@ void invokeFillBatch(IBuffer& buffer, IBuffer const& slotIndices, std::size_t sl { switch (buffer.getDataType()) { - case tensorrt_llm::DataType::kINT32: + case nvinfer1::DataType::kINT32: invokeFillBatch<std::int32_t>(buffer, slotIndices, slotStride, values, stream); break; - case tensorrt_llm::DataType::kINT8: + case nvinfer1::DataType::kINT8: invokeFillBatch<std::int8_t>(buffer, slotIndices, slotStride, values, stream); break; - case tensorrt_llm::DataType::kFLOAT: invokeFillBatch<float>(buffer, slotIndices, slotStride, values, stream); break; + case nvinfer1::DataType::kFLOAT: invokeFillBatch<float>(buffer, slotIndices, slotStride, values, stream); break; default: TLLM_THROW("data type not supported"); } } @@ -349,15 +349,13 @@ void invokeGatherBatch(IBuffer& buffer, IBuffer const& values, IBuffer const& sl { switch (buffer.getDataType()) { - case tensorrt_llm::DataType::kINT32: + case nvinfer1::DataType::kINT32: invokeGatherBatch<std::int32_t>(buffer, values, slotIndices, slotStride, stream); break; - case tensorrt_llm::DataType::kINT8: + case nvinfer1::DataType::kINT8: invokeGatherBatch<std::int8_t>(buffer, values, slotIndices, slotStride, stream); break; - case tensorrt_llm::DataType::kFLOAT: - invokeGatherBatch<float>(buffer, values, slotIndices, slotStride, stream); - break; + case nvinfer1::DataType::kFLOAT: invokeGatherBatch<float>(buffer, values, slotIndices, slotStride, stream); break; default: TLLM_THROW("data type not supported"); } } @@ -410,12 +408,12 @@ void scatterTensor(ITensor& output, ITensor const& input, SizeType32 beamWidth, { switch (input.getDataType()) { - case tensorrt_llm::DataType::kINT32: invokeScatterTensor<SizeType32>(output, input, beamWidth, stream); break; - case tensorrt_llm::DataType::kFLOAT: invokeScatterTensor<float>(output, input, beamWidth, stream); break; - case tensorrt_llm::DataType::kHALF: invokeScatterTensor<half>(output, input, beamWidth, stream); break; - case tensorrt_llm::DataType::kINT8: invokeScatterTensor<int8_t>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kINT32: invokeScatterTensor<SizeType32>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kFLOAT: invokeScatterTensor<float>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kHALF: invokeScatterTensor<half>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kINT8: invokeScatterTensor<int8_t>(output, input, beamWidth, stream); break; #ifdef ENABLE_FP8 - case tensorrt_llm::DataType::kFP8: invokeScatterTensor<__nv_fp8_e4m3>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kFP8: invokeScatterTensor<__nv_fp8_e4m3>(output, input, beamWidth, stream); break; #endif // ENABLE_FP8 default: TLLM_THROW("data type not supported"); } @@ -425,15 +423,15 @@ void tileTensor(ITensor& output, ITensor const& input, SizeType32 beamWidth, Cud { switch (input.getDataType()) { - case tensorrt_llm::DataType::kINT32: invokeTileTensor<SizeType32>(output, input, beamWidth, stream); break; - case tensorrt_llm::DataType::kFLOAT: invokeTileTensor<float>(output, input, beamWidth, stream); break; - case tensorrt_llm::DataType::kHALF: invokeTileTensor<half>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kINT32: invokeTileTensor<SizeType32>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kFLOAT: invokeTileTensor<float>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kHALF: invokeTileTensor<half>(output, input, beamWidth, stream); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: invokeTileTensor<__nv_bfloat16>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kBF16: invokeTileTensor<__nv_bfloat16>(output, input, beamWidth, stream); break; #endif // ENABLE_BF16 - case tensorrt_llm::DataType::kINT8: invokeTileTensor<int8_t>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kINT8: invokeTileTensor<int8_t>(output, input, beamWidth, stream); break; #ifdef ENABLE_FP8 - case tensorrt_llm::DataType::kFP8: invokeTileTensor<__nv_fp8_e4m3>(output, input, beamWidth, stream); break; + case nvinfer1::DataType::kFP8: invokeTileTensor<__nv_fp8_e4m3>(output, input, beamWidth, stream); break; #endif // ENABLE_FP8 default: TLLM_THROW("data type not supported"); } @@ -446,22 +444,22 @@ void mergeLogitsFragments(BufferManager const& bufferManager, ITensor& output, { switch (output.getDataType()) { - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: invokeMergeLogitsFragments<float>(bufferManager, output, fragmentsVector, cachePointerDevice, cachePointerHost, firstBatchSlotIdx, microBatchSize, beamWidth, stream, stepOffset); break; - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: invokeMergeLogitsFragments<half>(bufferManager, output, fragmentsVector, cachePointerDevice, cachePointerHost, firstBatchSlotIdx, microBatchSize, beamWidth, stream, stepOffset); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: invokeMergeLogitsFragments<__nv_bfloat16>(bufferManager, output, fragmentsVector, cachePointerDevice, cachePointerHost, firstBatchSlotIdx, microBatchSize, beamWidth, stream, stepOffset); break; #endif // ENABLE_BF16 #ifdef ENABLE_FP8 - case tensorrt_llm::DataType::kFP8: + case nvinfer1::DataType::kFP8: invokeMergeLogitsFragments<__nv_fp8_e4m3>(bufferManager, output, fragmentsVector, cachePointerDevice, cachePointerHost, firstBatchSlotIdx, microBatchSize, beamWidth, stream, stepOffset); break; diff --git a/cpp/tensorrt_llm/runtime/tensorView.h b/cpp/tensorrt_llm/runtime/tensorView.h index d9e65b0efe23..17e7fb719415 100644 --- a/cpp/tensorrt_llm/runtime/tensorView.h +++ b/cpp/tensorrt_llm/runtime/tensorView.h @@ -16,7 +16,6 @@ #pragma once -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferView.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -46,19 +45,19 @@ class TensorView : virtual public ITensor, public BufferView mDims.d[0] = size; } - TensorView(IBuffer::SharedPtr const& buffer, size_t offset, size_t size, tensorrt_llm::Dims const& dims) + TensorView(IBuffer::SharedPtr const& buffer, size_t offset, size_t size, nvinfer1::Dims const& dims) : BufferView{buffer, offset, size} , mDims{dims} { Base::resize(ITensor::volumeNonNegative(dims)); } - [[nodiscard]] tensorrt_llm::Dims const& getShape() const override + [[nodiscard]] nvinfer1::Dims const& getShape() const override { return mDims; } - void reshape(tensorrt_llm::Dims const& dims) override + void reshape(nvinfer1::Dims const& dims) override { Base::resize(ITensor::volumeNonNegative(dims)); mDims = dims; @@ -82,6 +81,6 @@ class TensorView : virtual public ITensor, public BufferView return shape.nbDims > 0 && shape.d[0] > 0 ? ITensor::volume(shape) / shape.d[0] : 0; } - tensorrt_llm::Dims mDims{}; + nvinfer1::Dims mDims{}; }; } // namespace tensorrt_llm::runtime diff --git a/cpp/tensorrt_llm/runtime/tllmBuffers.cpp b/cpp/tensorrt_llm/runtime/tllmBuffers.cpp index 4876d5b87bf6..ff7ed04001d3 100644 --- a/cpp/tensorrt_llm/runtime/tllmBuffers.cpp +++ b/cpp/tensorrt_llm/runtime/tllmBuffers.cpp @@ -15,7 +15,6 @@ */ #include "tensorrt_llm/runtime/tllmBuffers.h" -#include "tensorrt_llm/common/tllmDataType.h" namespace tensorrt_llm::runtime { @@ -63,12 +62,12 @@ std::shared_ptr<MulticastBuffer> MulticastTensorView::lock() const /////////////////////////////////////// // MulticastTensorView ITensor methods /////////////////////////////////////// -tensorrt_llm::Dims const& MulticastTensorView::getShape() const +nvinfer1::Dims const& MulticastTensorView::getShape() const { return mDims; } -void MulticastTensorView::reshape(tensorrt_llm::Dims const& dims) +void MulticastTensorView::reshape(nvinfer1::Dims const& dims) { auto new_size = nonNegative(volume(dims)); if (new_size > getCapacity()) @@ -103,7 +102,7 @@ std::size_t MulticastTensorView::getCapacity() const return lock()->getCapacity(); } -tensorrt_llm::DataType MulticastTensorView::getDataType() const +nvinfer1::DataType MulticastTensorView::getDataType() const { return lock()->getDataType(); } diff --git a/cpp/tensorrt_llm/runtime/tllmBuffers.h b/cpp/tensorrt_llm/runtime/tllmBuffers.h index d023823de5b2..faed36537e5c 100644 --- a/cpp/tensorrt_llm/runtime/tllmBuffers.h +++ b/cpp/tensorrt_llm/runtime/tllmBuffers.h @@ -27,7 +27,7 @@ #include "tensorrt_llm/runtime/memoryCounters.h" #include "tensorrt_llm/runtime/virtualMemory.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_runtime_api.h> #include <algorithm> @@ -550,7 +550,7 @@ class GenericBuffer : virtual public IBuffer, TAllocator // Inherit from TAlloca //! //! \brief Construct an empty buffer. //! - explicit GenericBuffer(tensorrt_llm::DataType type, TAllocator allocator = {}) // NOLINT(*-pro-type-member-init) + explicit GenericBuffer(nvinfer1::DataType type, TAllocator allocator = {}) // NOLINT(*-pro-type-member-init) : GenericBuffer{0, type, std::move(allocator)} { } @@ -559,7 +559,7 @@ class GenericBuffer : virtual public IBuffer, TAllocator // Inherit from TAlloca //! \brief Construct a buffer with the specified allocation size in number of elements. //! explicit GenericBuffer( // NOLINT(*-pro-type-member-init) - std::size_t size, tensorrt_llm::DataType type, TAllocator allocator = {}) + std::size_t size, nvinfer1::DataType type, TAllocator allocator = {}) : GenericBuffer{size, size, type, std::move(allocator)} { } @@ -636,7 +636,7 @@ class GenericBuffer : virtual public IBuffer, TAllocator // Inherit from TAlloca //! //! \brief Returns the type of the buffer. //! - [[nodiscard]] tensorrt_llm::DataType getDataType() const override + [[nodiscard]] nvinfer1::DataType getDataType() const override { return mType; } @@ -687,8 +687,7 @@ class GenericBuffer : virtual public IBuffer, TAllocator // Inherit from TAlloca } protected: - explicit GenericBuffer( - std::size_t size, std::size_t capacity, tensorrt_llm::DataType type, TAllocator allocator = {}) + explicit GenericBuffer(std::size_t size, std::size_t capacity, nvinfer1::DataType type, TAllocator allocator = {}) : TAllocator{std::move(allocator)} , mSize{size} , mCapacity{capacity} @@ -701,14 +700,14 @@ class GenericBuffer : virtual public IBuffer, TAllocator // Inherit from TAlloca private: std::size_t mSize{0}, mCapacity{0}; - tensorrt_llm::DataType mType; + nvinfer1::DataType mType; void* mBuffer; }; class MulticastBuffer : virtual public IBuffer { public: - explicit MulticastBuffer(tensorrt_llm::DataType type, std::set<int> const& ranks) + explicit MulticastBuffer(nvinfer1::DataType type, std::set<int> const& ranks) : mSize(0) , mCapacity(0) , mType(type) @@ -717,7 +716,7 @@ class MulticastBuffer : virtual public IBuffer TLLM_CHECK(ranks.size() > 1); } - explicit MulticastBuffer(size_t size, tensorrt_llm::DataType type, std::set<int> const& ranks) + explicit MulticastBuffer(size_t size, nvinfer1::DataType type, std::set<int> const& ranks) : mSize(0) , mCapacity(0) , mType(type) @@ -818,7 +817,7 @@ class MulticastBuffer : virtual public IBuffer return mCapacity; } - [[nodiscard]] tensorrt_llm::DataType getDataType() const override + [[nodiscard]] nvinfer1::DataType getDataType() const override { return mType; } @@ -854,7 +853,7 @@ class MulticastBuffer : virtual public IBuffer private: std::size_t mSize = 0; std::size_t mCapacity = 0; - tensorrt_llm::DataType mType; + nvinfer1::DataType mType; std::set<int> mRanks; IpcNvlsHandle* mHandle; }; @@ -883,7 +882,7 @@ class GenericTensor : virtual public ITensor, public GenericBuffer<TAllocator> //! //! \brief Construct an empty tensor. //! - explicit GenericTensor(tensorrt_llm::DataType type, TAllocator allocator = {}) + explicit GenericTensor(nvinfer1::DataType type, TAllocator allocator = {}) : Base{type, std::move(allocator)} { mDims.nbDims = 0; @@ -892,14 +891,14 @@ class GenericTensor : virtual public ITensor, public GenericBuffer<TAllocator> //! //! \brief Construct a tensor with the specified allocation dimensions. //! - explicit GenericTensor(tensorrt_llm::Dims const& dims, tensorrt_llm::DataType type, TAllocator allocator = {}) + explicit GenericTensor(nvinfer1::Dims const& dims, nvinfer1::DataType type, TAllocator allocator = {}) : Base{nonNegative(volume(dims)), type, std::move(allocator)} , mDims{dims} { } explicit GenericTensor( - tensorrt_llm::Dims const& dims, std::size_t capacity, tensorrt_llm::DataType type, TAllocator allocator = {}) + nvinfer1::Dims const& dims, std::size_t capacity, nvinfer1::DataType type, TAllocator allocator = {}) : Base{nonNegative(volume(dims)), capacity, type, std::move(allocator)} , mDims{dims} { @@ -924,12 +923,12 @@ class GenericTensor : virtual public ITensor, public GenericBuffer<TAllocator> return *this; } - [[nodiscard]] tensorrt_llm::Dims const& getShape() const override + [[nodiscard]] nvinfer1::Dims const& getShape() const override { return mDims; } - void reshape(tensorrt_llm::Dims const& dims) override + void reshape(nvinfer1::Dims const& dims) override { Base::resize(nonNegative(volume(dims))); mDims = dims; @@ -947,7 +946,7 @@ class GenericTensor : virtual public ITensor, public GenericBuffer<TAllocator> } private: - tensorrt_llm::Dims mDims{}; + nvinfer1::Dims mDims{}; }; // Forward declaration @@ -972,9 +971,9 @@ class MulticastTensorView : virtual public ITensor ///////////////////// // ITensor methods ///////////////////// - [[nodiscard]] tensorrt_llm::Dims const& getShape() const override; + [[nodiscard]] nvinfer1::Dims const& getShape() const override; - void reshape(tensorrt_llm::Dims const& dims) override; + void reshape(nvinfer1::Dims const& dims) override; ///////////////////// // IBuffer methods @@ -984,7 +983,7 @@ class MulticastTensorView : virtual public ITensor [[nodiscard]] std::size_t getCapacity() const override; - [[nodiscard]] tensorrt_llm::DataType getDataType() const override; + [[nodiscard]] nvinfer1::DataType getDataType() const override; [[nodiscard]] MemoryType getMemoryType() const override; @@ -1017,7 +1016,7 @@ class MulticastTensorView : virtual public ITensor std::weak_ptr<MulticastTensor> mTensor; ViewType mViewType; - tensorrt_llm::Dims mDims{}; + nvinfer1::Dims mDims{}; }; class MulticastTensor : virtual public ITensor, @@ -1027,13 +1026,13 @@ class MulticastTensor : virtual public ITensor, public: using Base = MulticastBuffer; - explicit MulticastTensor(tensorrt_llm::DataType type, std::set<int> const& ranks) + explicit MulticastTensor(nvinfer1::DataType type, std::set<int> const& ranks) : Base(type, ranks) { mDims.nbDims = 0; } - explicit MulticastTensor(tensorrt_llm::Dims const& dims, tensorrt_llm::DataType type, std::set<int> const& ranks) + explicit MulticastTensor(nvinfer1::Dims const& dims, nvinfer1::DataType type, std::set<int> const& ranks) : Base(nonNegative(volume(dims)), type, ranks) , mDims(dims) { @@ -1069,12 +1068,12 @@ class MulticastTensor : virtual public ITensor, ///////////////////// // ITensor methods ///////////////////// - [[nodiscard]] tensorrt_llm::Dims const& getShape() const override + [[nodiscard]] nvinfer1::Dims const& getShape() const override { return mDims; } - void reshape(tensorrt_llm::Dims const& dims) override + void reshape(nvinfer1::Dims const& dims) override { Base::resize(nonNegative(volume(dims))); mDims = dims; @@ -1092,7 +1091,7 @@ class MulticastTensor : virtual public ITensor, } private: - tensorrt_llm::Dims mDims{}; + nvinfer1::Dims mDims{}; }; using DeviceTensor = GenericTensor<CudaAllocatorAsync>; diff --git a/cpp/tensorrt_llm/runtime/tllmLogger.cpp b/cpp/tensorrt_llm/runtime/tllmLogger.cpp new file mode 100644 index 000000000000..586ab2f4ae95 --- /dev/null +++ b/cpp/tensorrt_llm/runtime/tllmLogger.cpp @@ -0,0 +1,62 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tllmLogger.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" + +using namespace tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; + +void TllmLogger::log(nvinfer1::ILogger::Severity severity, nvinfer1::AsciiChar const* msg) noexcept +{ + switch (severity) + { + case nvinfer1::ILogger::Severity::kINTERNAL_ERROR: + case nvinfer1::ILogger::Severity::kERROR: TLLM_LOG_ERROR(msg); break; + case nvinfer1::ILogger::Severity::kWARNING: TLLM_LOG_WARNING(msg); break; + case nvinfer1::ILogger::Severity::kINFO: TLLM_LOG_INFO(msg); break; + case nvinfer1::ILogger::Severity::kVERBOSE: TLLM_LOG_DEBUG(msg); break; + default: TLLM_LOG_TRACE(msg); break; + } +} + +nvinfer1::ILogger::Severity TllmLogger::getLevel() +{ + auto* const logger = tc::Logger::getLogger(); + switch (logger->getLevel()) + { + case tc::Logger::Level::ERROR: return nvinfer1::ILogger::Severity::kERROR; + case tc::Logger::Level::WARNING: return nvinfer1::ILogger::Severity::kWARNING; + case tc::Logger::Level::INFO: return nvinfer1::ILogger::Severity::kINFO; + case tc::Logger::Level::DEBUG: + case tc::Logger::Level::TRACE: return nvinfer1::ILogger::Severity::kVERBOSE; + default: return nvinfer1::ILogger::Severity::kINTERNAL_ERROR; + } +} + +void TllmLogger::setLevel(nvinfer1::ILogger::Severity level) +{ + auto* const logger = tc::Logger::getLogger(); + switch (level) + { + case nvinfer1::ILogger::Severity::kINTERNAL_ERROR: + case nvinfer1::ILogger::Severity::kERROR: logger->setLevel(tc::Logger::Level::ERROR); break; + case nvinfer1::ILogger::Severity::kWARNING: logger->setLevel(tc::Logger::Level::WARNING); break; + case nvinfer1::ILogger::Severity::kINFO: logger->setLevel(tc::Logger::Level::INFO); break; + case nvinfer1::ILogger::Severity::kVERBOSE: logger->setLevel(tc::Logger::Level::TRACE); break; + default: TLLM_THROW("Unsupported severity"); + } +} diff --git a/cpp/tensorrt_llm/runtime/tllmRuntime.cpp b/cpp/tensorrt_llm/runtime/tllmRuntime.cpp new file mode 100644 index 000000000000..7c2ca4747213 --- /dev/null +++ b/cpp/tensorrt_llm/runtime/tllmRuntime.cpp @@ -0,0 +1,831 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "tllmRuntime.h" +#include "common.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/nvtxUtils.h" +#include "tensorrt_llm/common/safetensors.h" +#include "tensorrt_llm/executor/tensor.h" +#include "tensorrt_llm/kernels/userbuffers/ub_interface.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tllmLogger.h" +#include "tllmStreamReaders.h" + +#include "nlohmann/json.hpp" +#include <NvInferRuntime.h> + +#include <algorithm> +#include <cstddef> +#include <limits> +#include <memory> +#include <optional> +#include <string> +#include <type_traits> +#include <utility> +#include <vector> + +using namespace tensorrt_llm::runtime; +using TensorMap = StringPtrMap<ITensor>; + +namespace +{ +static_assert(std::is_signed<SizeType32>::value, "SizeType32 must be signed"); + +nvinfer1::Dims shapeToDims(std::vector<std::size_t> const& shape) +{ + TLLM_CHECK(shape.size() <= nvinfer1::Dims::MAX_DIMS); + nvinfer1::Dims dims; + auto constexpr dim_max = std::numeric_limits<ITensor::DimType64>::max(); + dims.nbDims = static_cast<std::int32_t>(shape.size()); + for (std::size_t i = 0; i < shape.size(); ++i) + { + // shape[i] >= 0 because it has unsigned type. Check upper bound: + TLLM_CHECK(shape[i] <= static_cast<std::size_t>(dim_max)); + dims.d[i] = static_cast<ITensor::DimType64>(shape[i]); + } + return dims; +} + +std::vector<std::size_t> dimsToShape(nvinfer1::Dims const& dims) +{ + TLLM_CHECK(dims.nbDims >= 0); + std::vector<std::size_t> shape(dims.nbDims); + for (std::int32_t i = 0; i < dims.nbDims; ++i) + { + TLLM_CHECK(dims.d[i] >= 0); + shape[i] = static_cast<std::size_t>(dims.d[i]); + } + return shape; +} + +tensorrt_llm::runtime::TllmLogger defaultLogger{}; + +void setWeightStreaming(nvinfer1::ICudaEngine& engine, float const gpuWeightsPercent) +{ + if (gpuWeightsPercent < 1) + { + int64_t streamableSize = engine.getStreamableWeightsSize(); + int64_t budget = gpuWeightsPercent * streamableSize; + TLLM_LOG_INFO("Set gpu weights percent to %f, which is %lld bytes. Valid range: %lld bytes - %lld bytes.", + gpuWeightsPercent, budget, 0, streamableSize); + engine.setWeightStreamingBudgetV2(budget); + } +} + +class LayerInfo +{ +public: + LayerInfo(std::optional<std::string> name, std::string type) + : name(std::move(name)) + , type(std::move(type)){}; + std::optional<std::string> name; + std::string type; +}; + +void assessLikelihoodOfRuntimeAllocation( + nvinfer1::ICudaEngine const& engine, nvinfer1::IEngineInspector const& engineInspector) + +{ + TLLM_LOG_INFO("Inspecting the engine to identify potential runtime issues..."); + auto const profilingVerbosity = engine.getProfilingVerbosity(); + if (profilingVerbosity != nvinfer1::ProfilingVerbosity::kDETAILED) + { + TLLM_LOG_INFO( + "The profiling verbosity of the engine does not allow this analysis to proceed. Re-build the engine with " + "'detailed' profiling verbosity to get more diagnostics."); + return; + } + auto const* const layerTypeKey = "LayerType"; + auto const* const nameKey = "Name"; + auto const numLayers = engine.getNbLayers(); + TLLM_LOG_INFO("Model has %i layers.", numLayers); + std::vector<SizeType32> indexes(numLayers); + std::iota(indexes.begin(), indexes.end(), 0); + std::vector<std::optional<LayerInfo>> layerInfos(numLayers); + std::transform(indexes.cbegin(), indexes.cend(), layerInfos.begin(), + [&](SizeType32 const idx) + { + auto const* const layerInfo + = engineInspector.getLayerInformation(idx, nvinfer1::LayerInformationFormat::kJSON); + + // Needs to be copied explicitly, see documentation of `getLayerInformation`. + auto const layerInfoCopy = std::string(layerInfo); + auto const jsonLayerInfo = nlohmann::json::parse(layerInfoCopy); + auto const layerJsonType = jsonLayerInfo.type(); + if (layerJsonType != nlohmann::detail::value_t::object) + { + return std::optional<LayerInfo>{}; + } + if (!jsonLayerInfo.contains(layerTypeKey)) + { + return std::optional<LayerInfo>{}; + } + auto const& typeJson = jsonLayerInfo.at(layerTypeKey); + if (typeJson.type() != nlohmann::detail::value_t::string) + { + return std::optional<LayerInfo>{}; + } + std::optional<std::string> name{}; + if (jsonLayerInfo.contains(nameKey)) + { + auto const& nameJson = jsonLayerInfo.at(nameKey); + auto const nameJsonType = nameJson.type(); + if (nameJsonType == nlohmann::detail::value_t::string) + { + name = nameJson.get<std::string>(); + } + } + return std::make_optional(LayerInfo{name, typeJson.get<std::string>()}); + }); + auto const layersWithInfoEnd = std::partition( + layerInfos.begin(), layerInfos.end(), [](std::optional<LayerInfo> const& info) { return info.has_value(); }); + if (layersWithInfoEnd == layerInfos.begin()) + { + TLLM_LOG_INFO("Engine layer infos could not be parsed into useful information."); + return; + } + auto const allocateLayersEnd = std::partition(layerInfos.begin(), layersWithInfoEnd, + [](std::optional<LayerInfo> const& info) { return info.value().type == "allocate"; }); + auto numWarnings = 0; + for (auto layerInfo = layerInfos.begin(); layerInfo != allocateLayersEnd; layerInfo++) + { + auto constexpr maxNumWarnings = 25; + if (numWarnings < maxNumWarnings) + { + auto const layerName = layerInfo->value().name.value_or(""); + TLLM_LOG_WARNING( + "Layer '%s' has type '%s', which could lead to large runtime memory allocations. Performance " + "might be degraded and / or you might run out of memory.", + layerName.c_str(), layerInfo->value().type.c_str()); + } + numWarnings++; + } + if (numWarnings > 0) + { + TLLM_LOG_WARNING( + "There were a total of %i layers with type 'allocate'. Some warnings might have been silenced to keep the " + "output concise.", + numWarnings); + } +} + +} // namespace + +TllmRuntime::TllmRuntime(RawEngine const& rawEngine, nvinfer1::ILogger* logger, bool useGpuDirectStorage, + float gpuWeightsPercent, bool useShapeInference) + : mStream(std::make_shared<CudaStream>()) + , mBufferManager{mStream, true} // Ensure to trim the memory pool on destruction. + , mRuntime{nvinfer1::createInferRuntime(static_cast<bool>(logger) ? *logger : defaultLogger)} + , mUseShapeInference{useShapeInference} + , mUserBufferEnabled{false} +{ + auto const startTime = std::chrono::high_resolution_clock::now(); + + switch (rawEngine.getType()) + { + case RawEngine::Type::FilePath: + { + if (useGpuDirectStorage) + { + TLLM_LOG_INFO("GDS is used to load the engine!"); + auto reader = GDSStreamReader(rawEngine.getPath()); + mEngine.reset(mRuntime->deserializeCudaEngine(reader)); + } + else + { + auto reader = StreamReader(rawEngine.getPath()); + mEngine.reset(mRuntime->deserializeCudaEngine(reader)); + } + break; + } + case RawEngine::Type::AddressWithSize: + mEngine.reset(mRuntime->deserializeCudaEngine(rawEngine.getAddress(), rawEngine.getSize())); + break; + case RawEngine::Type::HostMemory: + mEngine.reset( + mRuntime->deserializeCudaEngine(rawEngine.getHostMemory()->data(), rawEngine.getHostMemory()->size())); + break; + default: TLLM_THROW("Unsupported raw engine type."); + } + + auto const elapsedMs + = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::high_resolution_clock::now() - startTime); + + TLLM_LOG_INFO("Engine load time %lld ms", elapsedMs); + + TLLM_CHECK_WITH_INFO(mEngine != nullptr, "Failed to deserialize cuda engine."); + mEngineInspector.reset(mEngine->createEngineInspector()); + assessLikelihoodOfRuntimeAllocation(*mEngine, *mEngineInspector); + setWeightStreaming(getEngine(), gpuWeightsPercent); + auto const devMemorySize = mEngine->getDeviceMemorySizeV2(); + mEngineBuffer = mBufferManager.gpu(devMemorySize); + // Print context memory size for CI/CD to track. + TLLM_LOG_INFO("[MemUsageChange] Allocated %.2f MiB for execution context memory.", + static_cast<double>(devMemorySize) / 1048576.0); + + cacheTensorNames(); +} + +void TllmRuntime::cacheTensorNames() +{ + for (std::int32_t i = 0; i < mEngine->getNbIOTensors(); ++i) + { + auto const* const name = mEngine->getIOTensorName(i); + if (mEngine->getTensorIOMode(name) == nvinfer1::TensorIOMode::kINPUT) + { + mInputTensorNames.emplace_back(name); + } + else if (mEngine->getTensorIOMode(name) == nvinfer1::TensorIOMode::kOUTPUT) + { + mOutputTensorNames.emplace_back(name); + } + } +} + +nvinfer1::IExecutionContext& TllmRuntime::addContext(std::int32_t profileIndex) +{ + TLLM_CHECK(0 <= profileIndex && profileIndex < mEngine->getNbOptimizationProfiles()); + mContexts.emplace_back(mEngine->createExecutionContextWithoutDeviceMemory()); + if (!mContexts.back()) + { + if (mEngine->getStreamableWeightsSize() > 0) + { + TLLM_THROW("Failed to allocate memory for weights. Please try reducing --gpu_weights_percent."); + } + else + { + TLLM_THROW("Internal Error: Failed to create an execution context."); + } + } + auto& context = *mContexts.back(); + context.setDeviceMemoryV2(mEngineBuffer->data(), static_cast<int64_t>(mEngineBuffer->getCapacity())); + + if (tensorrt_llm::common::Logger::getLogger()->isEnabled(tensorrt_llm::common::Logger::TRACE) + && mContexts.size() == 1) + { + // Print engine information only once + printEngineInfo(); + } + + context.setOptimizationProfileAsync(profileIndex, mStream->get()); + // If nvtx verbosity is DETAILED, print an info about potential perf overhead. + if (context.getNvtxVerbosity() == nvinfer1::ProfilingVerbosity::kDETAILED) + { + TLLM_LOG_INFO( + "The engine was built with kDETAILED profiling verbosity, which may result in small overheads at runtime."); + } + return context; +} + +void TllmRuntime::printEngineInfo() +{ + auto& context = *(mContexts[0]); + int const nIO = mEngine->getNbIOTensors(); // Count of input / output tensor + int const nOP = mEngine->getNbOptimizationProfiles(); // Count of Optimization Profile + std::size_t maxNameWidth = 0; + std::size_t maxShapeWidth = 0; + + // Get information of engine input / output + std::vector<std::string> tensorNameList{}; + tensorNameList.reserve(nIO); + for (int i = 0; i < nIO; ++i) + { + tensorNameList.emplace_back(mEngine->getIOTensorName(i)); + } + std::vector<std::map<std::string, std::string>> tensorInfo(nIO); // Tensor Information Vector + std::vector<std::vector<std::vector<nvinfer1::Dims64>>> profileInfo(nIO); // Tensor Optimization Profile Vector + for (int i = 0; i < nIO; ++i) + { + auto const& name = tensorNameList[i]; + char const* nameC{name.c_str()}; // name of C-style + maxNameWidth = std::max(maxNameWidth, name.size()); + tensorInfo[i]["mode"] = mEngine->getTensorIOMode(nameC) == nvinfer1::TensorIOMode::kINPUT ? "I" : "O"; + tensorInfo[i]["location"] + = mEngine->getTensorLocation(nameC) == nvinfer1::TensorLocation::kDEVICE ? "GPU" : "CPU"; + tensorInfo[i]["data_type"] = dataTypeToString(mEngine->getTensorDataType(nameC)); + tensorInfo[i]["build_shape"] = shapeToString(mEngine->getTensorShape(nameC)); + maxShapeWidth = std::max(maxShapeWidth, tensorInfo[i]["build_shape"].size()); + if (tensorInfo[i]["mode"] == "I") + { + std::vector<std::vector<nvinfer1::Dims64>> topPerTensor(nOP); + for (int k = 0; k < nOP; ++k) + { + if (tensorInfo[i]["location"] == std::string("GPU")) + { + std::vector<nvinfer1::Dims64> top(3); + top[0] = mEngine->getProfileShape(nameC, k, nvinfer1::OptProfileSelector::kMIN); + top[1] = mEngine->getProfileShape(nameC, k, nvinfer1::OptProfileSelector::kOPT); + top[2] = mEngine->getProfileShape(nameC, k, nvinfer1::OptProfileSelector::kMAX); + topPerTensor[k] = top; + maxShapeWidth = std::max(maxShapeWidth, shapeToString(top[2]).size()); + } + else + { + // Shape input tensor, not used in TRT-LLM support yet + std::vector<nvinfer1::Dims64> top(3); + int const nDim = mEngine->getTensorShape(nameC).nbDims; + nvinfer1::Dims64 tensorShape{nDim, {-1}}; + int const* pos = nullptr; + pos = mEngine->getProfileTensorValues(nameC, k, nvinfer1::OptProfileSelector::kMIN); + std::copy(pos, pos + nDim, tensorShape.d); + top[0] = tensorShape; + pos = mEngine->getProfileTensorValues(nameC, k, nvinfer1::OptProfileSelector::kOPT); + std::copy(pos, pos + nDim, tensorShape.d); + top[1] = tensorShape; + pos = mEngine->getProfileTensorValues(nameC, k, nvinfer1::OptProfileSelector::kMAX); + std::copy(pos, pos + nDim, tensorShape.d); + top[2] = tensorShape; + topPerTensor[k] = top; + } + } + profileInfo[i] = topPerTensor; + } + else + { + profileInfo[i] = std::vector<std::vector<nvinfer1::Dims64>>(nOP); + } + } + // Set input shape to get output shape + for (int k = 0; k < nOP; ++k) + { + for (int j = 0; j < 3; ++j) // Min, Opt, Max + { + for (int i = 0; i < nIO; ++i) + { + auto const& name = tensorNameList[i]; + char const* nameC = name.c_str(); + if (tensorInfo[i]["mode"] == "I") + { + if (tensorInfo[i]["location"] == std::string("GPU")) + { + context.setInputShape(nameC, profileInfo[i][k][j]); + } + else + { + // Shape input tensor, not used in TRT-LLM support yet + context.setInputTensorAddress(nameC, profileInfo[i][k][j].d); + } + } + else + { + TLLM_CHECK_WITH_INFO(context.allInputDimensionsSpecified(), "Input dimensions not specified"); + TLLM_CHECK_WITH_INFO(context.allInputShapesSpecified(), "Input shapes not specified"); + if (tensorInfo[i]["location"] == std::string("GPU")) + { + profileInfo[i][k].push_back(context.getTensorShape(nameC)); + } + else + { + // Shape input tensor, not used in TRT-LLM support yet + int const nDim = mEngine->getTensorShape(nameC).nbDims; + nvinfer1::Dims64 tensorShape{nDim, {}}; + int const* pos = reinterpret_cast<int const*>(context.getTensorAddress(nameC)); + std::copy(pos, pos + nDim, tensorShape.d); + profileInfo[i][k].push_back(tensorShape); + } + } + } + } + } + + // Print information of engine input / output + std::string info; + TLLM_LOG_TRACE("Information of engine input / output."); + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth + 24, '=')); + info = alignText("Name", maxNameWidth) + "|I/O|Location|DataType|" + alignText("Shape", maxShapeWidth) + "|"; + TLLM_LOG_TRACE(info.c_str()); + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth + 24, '-')); + for (int i = 0; i < nIO; ++i) + { + info = alignText(tensorNameList[i], maxNameWidth, false) + "|"; + info += alignText(tensorInfo[i]["mode"], 3) + "|"; + info += alignText(tensorInfo[i]["location"], 8) + "|"; + info += alignText(tensorInfo[i]["data_type"], 8) + "|"; + info += alignText(tensorInfo[i]["build_shape"], maxShapeWidth) + "|"; + TLLM_LOG_TRACE(info.c_str()); + } + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth + 24, '=')); + // Print information of optimization profile + TLLM_LOG_TRACE("Information of optimization profile."); + for (int k = 0; k < nOP; ++k) + { + TLLM_LOG_TRACE("Optimization Profile %d:", k); + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth * 3 + 4, '=')); + info = alignText("Name", maxNameWidth) + "|"; + info += alignText("Min", maxShapeWidth) + "|"; + info += alignText("Opt", maxShapeWidth) + "|"; + info += alignText("Max", maxShapeWidth) + "|"; + TLLM_LOG_TRACE(info.c_str()); + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth * 3 + 4, '-')); + for (int i = 0; i < nIO; ++i) + { + auto const& top = profileInfo[i][k]; + info = alignText(tensorNameList[i], maxNameWidth, false) + "|"; + info += alignText(shapeToString(top[0]), maxShapeWidth) + "|"; + info += alignText(shapeToString(top[1]), maxShapeWidth) + "|"; + info += alignText(shapeToString(top[2]), maxShapeWidth) + "|"; + TLLM_LOG_TRACE(info.c_str()); + } + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth * 3 + 4, '=')); + } +} + +void TllmRuntime::printContextInfo(SizeType32 contextIndex) +{ + auto const& context = *(mContexts[contextIndex]); + int const nIO = mEngine->getNbIOTensors(); // Count of input / output tensor + std::size_t maxNameWidth = 0; + std::size_t maxShapeWidth = 0; + std::vector<std::tuple<std::string, bool, std::string>> tensorInfo(nIO); + for (int i = 0; i < nIO; ++i) + { + auto const name = std::string(mEngine->getIOTensorName(i)); + bool const isInput = mEngine->getTensorIOMode(name.c_str()) == nvinfer1::TensorIOMode::kINPUT; + auto const shape = shapeToString(context.getTensorShape(name.c_str())); + tensorInfo[i] = std::make_tuple(name, isInput, shape); + maxNameWidth = std::max(maxNameWidth, name.size()); + maxShapeWidth = std::max(maxShapeWidth, shape.size()); + // Shape input tensor is not considered in TRT-LLM yet + } + + TLLM_LOG_TRACE("Information of context input / output."); + TLLM_LOG_TRACE("Using Optimization Profile: %d", contextIndex); + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth + 6, '=')); + std::string info = alignText("Name", maxNameWidth) + "|I/O|" + alignText("Shape", maxShapeWidth) + "|"; + TLLM_LOG_TRACE(info.c_str()); + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth + 6, '-')); + for (int i = 0; i < nIO; ++i) + { + auto const& [name, isInput, shape] = tensorInfo[i]; + info = alignText(name, maxNameWidth, false) + "|"; + info += alignText(isInput ? "I" : "O", 3) + "|"; + info += alignText(shape, maxShapeWidth) + "|"; + TLLM_LOG_TRACE(info.c_str()); + } + TLLM_LOG_TRACE(std::string(maxNameWidth + maxShapeWidth + 6, '=')); +} + +void TllmRuntime::clearContexts() +{ + for (auto& context : mContexts) + { + context.reset(); + } + mContexts.clear(); +} + +bool TllmRuntime::executeContext(SizeType32 contextIndex) const +{ + NVTX3_FUNC_RANGE(); + auto& context = getContext(contextIndex); + auto res = context.enqueueV3(mStream->get()); + sync_check_cuda_error(mStream->get()); + return res; +} + +void TllmRuntime::setInputTensorsImpl(SizeType32 contextIndex, TensorMap const& tensorMap, bool throwOnMiss) +{ + NVTX3_FUNC_RANGE(); + auto& context = getContext(contextIndex); + for (auto const& name : mInputTensorNames) + { + auto const pos = tensorMap.find(name); + if (pos == tensorMap.end()) + { + if (throwOnMiss) + { + auto expectedShape = mEngine->getTensorShape(name.c_str()); + TLLM_THROW("Input tensor '%s' not found; expected shape: %s", name.c_str(), + ITensor::toString(expectedShape).c_str()); + } + else + { + continue; + } + } + + auto const& tensor = pos->second; + auto const tensorDtype = tensor->getDataType(); + auto const engineDtype = mEngine->getTensorDataType(name.c_str()); + // WAR: TRT does not support mixed FP8 and FP16 input, so engine expects FP16 tensors. + TLLM_CHECK_WITH_INFO(tensorDtype == engineDtype + || (tensorDtype == nvinfer1::DataType::kFP8 && engineDtype == nvinfer1::DataType::kHALF), + "%s: expected type %d, provided type %d", name.c_str(), static_cast<std::int32_t>(engineDtype), + static_cast<std::int32_t>(tensorDtype)); + + auto tensorShape = tensor->getShape(); + + // Change shape of `cache_indirection` for Variable-Beam-Width-Search + // TODO: remove this hack if beamWidth of each request are passed into GptAttentionPlugin by input tensor + if (name == "cache_indirection" && mCurrentBeamWidths.size() > 0) + { + SizeType32 const beamWidth = getCurrentBeamWidth(); + if (tensorShape.d[1] != beamWidth) + { + tensorShape.d[1] = beamWidth; + TLLM_LOG_TRACE("Change shape of cache_indirection to %s", ITensor::toString(tensorShape).c_str()); + } + } + + auto const setInputShapeSuccess = context.setInputShape(name.c_str(), tensorShape); + if (!setInputShapeSuccess) + { + auto const minShape + = mEngine->getProfileShape(name.c_str(), contextIndex, nvinfer1::OptProfileSelector::kMIN); + auto const maxShape + = mEngine->getProfileShape(name.c_str(), contextIndex, nvinfer1::OptProfileSelector::kMAX); + + TLLM_THROW("Tensor '%s' has invalid shape %s, expected in range min %s, max %s", name.c_str(), + ITensor::toString(tensorShape).c_str(), ITensor::toString(minShape).c_str(), + ITensor::toString(maxShape).c_str()); + } + auto* const data = tensor->data(); + if (static_cast<bool>(data)) + { + context.setInputTensorAddress(name.c_str(), data); + } + else + { + TLLM_CHECK_WITH_INFO(tensor->getSize() == 0, std::string("Invalid data for tensor: ") + name); + // TensorRT runtime does not support nullptr. + if (!mDummyTensor) + { + mDummyTensor = mBufferManager.gpu(ITensor::makeShape({1})); + } + context.setInputTensorAddress(name.c_str(), mDummyTensor->data()); + } + } +} + +void TllmRuntime::setStaticInputTensors(TensorMap const& tensorMap) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_FUNC_RANGE(); + + TLLM_CHECK_WITH_INFO(getNbContexts() > 0, "Contexts should be created before calling setStaticInputTensors"); + for (auto contextIndex = 0; contextIndex < getNbContexts(); ++contextIndex) + { + setInputTensorsImpl(contextIndex, tensorMap, false); + } + + // move static input tensor names to separate vector + auto const begin = mInputTensorNames.begin(); + auto end = mInputTensorNames.end(); + for (auto const& [name, tensor] : tensorMap) + { + end = std::remove(begin, end, name); + } + mInputTensorNames.erase(end, mInputTensorNames.end()); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TllmRuntime::setInputTensors(SizeType32 contextIndex, TensorMap const& tensorMap) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_FUNC_RANGE(); + setInputTensorsImpl(contextIndex, tensorMap, true); + + auto& context = getContext(contextIndex); + if (mUseShapeInference) + { + NVTX3_SCOPED_RANGE(infer_shapes); + char const* missing = nullptr; + auto const nbMissing = context.inferShapes(1, &missing); + if (nbMissing > 0) + { + TLLM_THROW("Input shape not specified: %s", missing); + } + else if (nbMissing < 0) + { + TLLM_THROW("Invalid input shape"); + } + } + + { + NVTX3_SCOPED_RANGE(final_checks); + TLLM_CHECK_WITH_INFO(context.allInputDimensionsSpecified(), "Input dimensions not specified"); + TLLM_CHECK_WITH_INFO(context.allInputShapesSpecified(), "Input shapes not specified"); + } + + // Print shape of input / output tensors for the TRT engine + if (tensorrt_llm::common::Logger::getLogger()->isEnabled(tensorrt_llm::common::Logger::TRACE)) + { + printContextInfo(contextIndex); + } + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TllmRuntime::setOutputTensors(SizeType32 contextIndex, TensorMap& tensorMap) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_FUNC_RANGE(); + if (isUserBufferEnabled()) + { + // This function will identify the output tensors in the network that need to be bound as UB buffers + // and bind the corresponding buffers to them based on their names. + setUserBufferTensors(contextIndex, tensorMap); + } + + auto& context = getContext(contextIndex); + for (auto const& name : mOutputTensorNames) + { + auto const engineDtype = mEngine->getTensorDataType(name.c_str()); + auto const pos = tensorMap.find(name); + if (pos != tensorMap.end()) + { + auto const& tensor = pos->second; + auto const tensorDtype = tensor->getDataType(); + // WAR: TRT does not support mixed FP8 and FP16 input, so engine expects FP16 tensors. + TLLM_CHECK_WITH_INFO(tensorDtype == engineDtype + || (tensorDtype == nvinfer1::DataType::kFP8 && engineDtype == nvinfer1::DataType::kHALF), + "%s: expected type %d, provided type %d", name.c_str(), static_cast<std::int32_t>(engineDtype), + static_cast<std::int32_t>(tensorDtype)); + + if (mUseShapeInference) + { + auto const dims = context.getTensorShape(name.c_str()); + tensor->reshape(dims); + } + context.setTensorAddress(name.c_str(), tensor->data()); + } + else if (mUseShapeInference) + { + auto const dims = context.getTensorShape(name.c_str()); + auto tensor = ITensor::SharedPtr(mBufferManager.gpu(dims, engineDtype)); + tensorMap.insert(pos, std::make_pair(name, tensor)); + context.setTensorAddress(name.c_str(), tensor->data()); + } + else + { + TLLM_THROW("Tensor %s is not found in tensorMap and shape inference is not allowed", name.c_str()); + } + } + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +void TllmRuntime::setUserBufferTensors(SizeType32 contextIndex, TensorMap& tensorMap) +{ + auto startsWith = [](std::string const& str, std::string const& prefix) -> bool + { return str.size() > prefix.size() && str.compare(0, prefix.size(), prefix) == 0; }; + std::string const prefix(tensorrt_llm::runtime::ub::tensor_prefix); + auto& context = getContext(contextIndex); + for (auto const& name : mOutputTensorNames) + { + auto const pos = tensorMap.find(name); + if (pos != tensorMap.end() || !startsWith(name, prefix)) + { + continue; + } + auto const engineDtype = mEngine->getTensorDataType(name.c_str()); + auto const dims = context.getTensorShape(name.c_str()); + void* ubBuffer = nullptr; + if (name[prefix.size()] == '0') + { + ubBuffer = tensorrt_llm::runtime::ub::ub_get(0).addr; + } + else if (name[prefix.size()] == '1') + { + ubBuffer = tensorrt_llm::runtime::ub::ub_get(1).addr; + } + else if (name[prefix.size()] == '2') + { + ubBuffer = tensorrt_llm::runtime::ub::ub_get(2).addr; + } + else + { + TLLM_CHECK(false); + } + auto tensor = ITensor::SharedPtr(ITensor::wrap(ubBuffer, engineDtype, dims)); + tensorMap.insert(pos, std::make_pair(name, tensor)); + context.setTensorAddress(name.c_str(), ubBuffer); + } +} + +void TllmRuntime::initializeUserBuffer(tensorrt_llm::runtime::WorldConfig const& world_config, SizeType32 maxBatchSize, + SizeType32 maxBeamWidth, SizeType32 maxSequenceLength, SizeType32 hiddenSize, + std::optional<SizeType32> maxNumTokens) +{ + auto startsWith = [](std::string const& str, std::string const& prefix) -> bool + { return str.size() > prefix.size() && str.compare(0, prefix.size(), prefix) == 0; }; + std::string const prefix(tensorrt_llm::runtime::ub::tensor_prefix); + bool useNVFP4Model = false; + for (auto const& name : mOutputTensorNames) + { + if (startsWith(name, prefix)) + { + mUserBufferEnabled = true; + if (name[prefix.size()] == '2') + { + useNVFP4Model = true; + break; + } + } + } + if (!mUserBufferEnabled) + { + return; + } + // The hidden size returned by ModelConfig is the real hidden size divided by the TP size. + auto const tpSize = world_config.getTensorParallelism(); + size_t const realHiddenSize = hiddenSize * tpSize; + size_t const tokensNum = maxNumTokens.value_or(maxBatchSize * maxBeamWidth * maxSequenceLength); + TLLM_CHECK(tokensNum > 0); + size_t const elemNum = tokensNum * realHiddenSize; + TLLM_LOG_INFO("[UserBuffer] MaxBatchSize %d, maxBeamWidth %d, maxSequenceLength %d, maxNumTokens %d, select %lu", + maxBatchSize, maxBeamWidth, maxSequenceLength, maxNumTokens.has_value() ? maxNumTokens.value() : 0, tokensNum); + tensorrt_llm::runtime::ub::ub_initialize(world_config); + tensorrt_llm::runtime::ub::ub_allocate(elemNum * sizeof(half)); + tensorrt_llm::runtime::ub::ub_allocate(elemNum * sizeof(half)); + if (useNVFP4Model) + { + tensorrt_llm::runtime::ub::ub_allocate(elemNum * sizeof(uint8_t) / 16); + } +} + +CudaStream const& TllmRuntime::getStream() const +{ + return *mStream; +} + +bool TllmRuntime::hasLayerProfiler(SizeType32 contextId) const +{ + return mContexts[contextId]->getProfiler() != nullptr; +} + +void TllmRuntime::setLayerProfiler() +{ + mLayerProfiler = std::make_unique<LayerProfiler>(); + for (auto& context : mContexts) + { + context->setProfiler(mLayerProfiler.get()); + context->setEnqueueEmitsProfile(false); + } +} + +std::string TllmRuntime::getLayerProfileInfo() const +{ + TLLM_CHECK(mLayerProfiler); + return mLayerProfiler->getLayerProfile(); +} + +void TllmRuntime::reportToProfiler(SizeType32 contextId) +{ + mContexts[contextId]->reportToProfiler(); +} + +void TllmRuntime::loadManagedWeights(RawEngine const& rawEngine, int localRank) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + NVTX3_FUNC_RANGE(); + auto& engine = getEngine(); + auto& manager = getBufferManager(); + if (rawEngine.getManagedWeightsMapOpt().has_value()) + { + TLLM_LOG_DEBUG("Loading managed weights from raw engine"); + auto executorMap = rawEngine.getManagedWeightsMapOpt().value(); + for (auto const& [name, weight] : executorMap) + { + TLLM_LOG_DEBUG("Loading managed weight: %s", name.c_str()); + auto iTensor = tensorrt_llm::executor::detail::toITensor(weight); + auto weightsDevice = std::shared_ptr<ITensor>{manager.copyFrom(*iTensor, MemoryType::kGPU)}; + mManagedWeightsMap.insert(std::make_pair(name, weightsDevice)); + } + } + else + { + TLLM_LOG_DEBUG("Loading managed weights from file"); + auto const enginePath = rawEngine.getPathOpt(); + TLLM_CHECK_WITH_INFO(enginePath.has_value(), "Engine path is not set."); + auto weightPath + = enginePath->parent_path() / ("rank" + std::to_string(localRank) + "_managed_weights.safetensors"); + auto managed_weights = common::safetensors::ISafeTensor::open(weightPath.string().c_str()); + for (auto const& name : managed_weights->keys()) + { + TLLM_LOG_DEBUG("Loading managed weight: %s", name.c_str()); + auto const weight = managed_weights->getTensor(name.c_str()); + TLLM_CHECK(weight->dtype() == engine.getTensorDataType(name.c_str())); + auto weightsDevice + = std::shared_ptr<ITensor>{manager.allocate(MemoryType::kGPU, weight->trtDims(), weight->dtype())}; + manager.copy(weight->data(), *weightsDevice, MemoryType::kCPU); + mManagedWeightsMap.insert(std::make_pair(name, weightsDevice)); + } + } + setStaticInputTensors(mManagedWeightsMap); + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} diff --git a/cpp/tensorrt_llm/runtime/tllmRuntime.h b/cpp/tensorrt_llm/runtime/tllmRuntime.h new file mode 100644 index 000000000000..dfef06d8b45f --- /dev/null +++ b/cpp/tensorrt_llm/runtime/tllmRuntime.h @@ -0,0 +1,243 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/layerProfiler.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/worldConfig.h" +#include <NvInferRuntime.h> + +#include <cstdint> +#include <memory> +#include <set> +#include <string> +#include <vector> + +namespace tensorrt_llm::runtime +{ +class TllmRuntime +{ +public: + using TensorMap = StringPtrMap<ITensor>; + + explicit TllmRuntime(RawEngine const& rawEngine, nvinfer1::ILogger* logger, bool useGpuDirectStorage = false, + float gpuWeightsPercent = 1.0f, bool useShapeInference = true); + + SizeType32 getNbContexts() const + { + return static_cast<SizeType32>(mContexts.size()); + } + + nvinfer1::IExecutionContext& getContext(SizeType32 contextIndex) const + { + return *mContexts.at(contextIndex); + } + + SizeType32 getNbProfiles() const + { + return static_cast<SizeType32>(mEngine->getNbOptimizationProfiles()); + } + + /// @brief If multiple TensorRT optimization profiles are built in the engine, this function selects the + /// corresponding profile that is going to be used based on the runtime shape, for now, TensorRT LLM only split + /// multiple profiles on the num_tokens dimension, hence the profile index is selected based on which profile + /// handles the actual num_tokens + /// @return The index of the selected TensorRT optimization profile + [[nodiscard]] SizeType32 getOptProfileId(int numTokens, std::vector<SizeType32> const& splitPoints) const + { + if (getNbProfiles() == 1) + { + return 0; + } + auto const it = std::lower_bound(splitPoints.begin(), splitPoints.end(), numTokens); + auto const optProfileId = std::distance(splitPoints.begin(), it); + return optProfileId; + } + + nvinfer1::IExecutionContext& addContext(std::int32_t profileIndex); + + void clearContexts(); + + /// @brief Set input tensors from tensorMap for all contexts. + /// @details The function can be used to set static input tensors for all iterations. If a tensor was set this way, + /// it doesn't need to included in calls to setInputTensors anymore. + void setStaticInputTensors(TensorMap const& tensorMap); + + /// @brief Set input tensors from tensorMap for context at contextIndex. + /// @details The function expects that all input tensors (excluding the ones set by setStaticInputTensors) are + /// contained in the tensorMap. If a tensor is missing, has a bad shape or type, it will throw. + void setInputTensors(SizeType32 contextIndex, TensorMap const& tensorMap); + + /// @brief Set output tensors from tensorMap for context at contextIndex. + /// @details The function expects that all output tensors are contained in the tensorMap. If a tensor is missing and + /// shape inference is enabled, it will allocate the tensor on GPU and insert it into the tensorMap. Otherwise it + /// will throw. + void setOutputTensors(SizeType32 contextIndex, TensorMap& tensorMap); + + bool executeContext(SizeType32 contextIndex) const; + + CudaStream const& getStream() const; + + BufferManager::CudaStreamPtr getStreamPtr() + { + return mStream; + } + + nvinfer1::ICudaEngine& getEngine() + { + return *mEngine; + } + + nvinfer1::ICudaEngine const& getEngine() const + { + return *mEngine; + } + + nvinfer1::IEngineInspector& getEngineInspector() + { + return *mEngineInspector; + } + + nvinfer1::IEngineInspector const& getEngineInspector() const + { + return *mEngineInspector; + } + + BufferManager& getBufferManager() + { + return mBufferManager; + } + + BufferManager const& getBufferManager() const + { + return mBufferManager; + } + + void setLayerProfiler(); + bool hasLayerProfiler(SizeType32 contextId) const; + std::string getLayerProfileInfo() const; + void reportToProfiler(SizeType32 contextId); + void loadManagedWeights(RawEngine const& rawEngine, int localRank); + void initializeUserBuffer(tensorrt_llm::runtime::WorldConfig const& world_config, SizeType32 maxBatchSize, + SizeType32 maxBeamWidth, SizeType32 maxSequenceLength, SizeType32 hiddenSize, + std::optional<SizeType32> maxNumTokens); + + bool isUserBufferEnabled() const + { + return mUserBufferEnabled; + } + + void setCurrentBeamWidths(std::vector<SizeType32> const& beamWidth) noexcept + { + mCurrentBeamWidths = beamWidth; + } + + [[nodiscard]] SizeType32 const& getCurrentBeamWidth() const noexcept + { + // At present, all requests of a batch must have the same beam width in one generation step (or they will not + // be batched together). So, the beam widths in `mCurrentBeamWidths` are the same. + // Corresponding changes must be done if Diverse-Beam-Width-Search (DBWS, requests with diverse beam width in + // a batch in one generation step) is supported in the future. + TLLM_CHECK_WITH_INFO(mCurrentBeamWidths.size() > 0, "`mCurrentBeamWidths` is empty."); + bool const isEqual = std::all_of(mCurrentBeamWidths.begin(), mCurrentBeamWidths.end(), + [&](int elem) { return elem == mCurrentBeamWidths.front(); }); + TLLM_CHECK_WITH_INFO(isEqual, "beam widths in `mCurrentBeamWidths` are not all equal."); + return mCurrentBeamWidths.front(); + } + +private: + void cacheTensorNames(); + + void setInputTensorsImpl(SizeType32 contextIndex, TensorMap const& tensorMap, bool throwOnMiss); + + void setUserBufferTensors(SizeType32 contextIndex, TensorMap& tensorMap); + + void printEngineInfo(); + + void printContextInfo(SizeType32 contextIndex); + + // Tool functions for `printEngineInfo()`. + static std::string shapeToString(nvinfer1::Dims64 const& dim) + { + std::string output("("); + if (dim.nbDims == 0) + { + return output + ")"; + } + for (int i = 0; i < dim.nbDims - 1; ++i) + { + output += std::to_string(dim.d[i]) + ", "; + } + output += std::to_string(dim.d[dim.nbDims - 1]) + ")"; + return output; + } + + static std::string dataTypeToString(nvinfer1::DataType type) + { + switch (type) + { + case nvinfer1::DataType::kINT64: return "INT64"; + case nvinfer1::DataType::kINT32: return "INT32"; + case nvinfer1::DataType::kFLOAT: return "FP32"; + case nvinfer1::DataType::kBF16: return "BF16"; + case nvinfer1::DataType::kHALF: return "FP16"; + case nvinfer1::DataType::kBOOL: return "BOOL"; + case nvinfer1::DataType::kUINT8: return "UINT8"; + case nvinfer1::DataType::kINT8: return "INT8"; + case nvinfer1::DataType::kFP8: return "FP8"; + case nvinfer1::DataType::kINT4: return "INT4"; + case nvinfer1::DataType::kFP4: return "FP4"; + default: return "UNKNOWN"; + } + return ""; + } + + static std::string alignText( + std::string const& text, int const width, bool const bCenter = true, char const blank = ' ') + { + int textLen = text.size(); + int padLeft = 0; + int padRight = 0; + padLeft = bCenter ? (width - textLen) / 2 : 0; + padRight = width - padLeft - textLen; + return std::string(padLeft, blank) + text + std::string(padRight, blank); + } + + BufferManager::CudaStreamPtr mStream; + BufferManager mBufferManager; + std::unique_ptr<nvinfer1::IRuntime> mRuntime; + std::unique_ptr<nvinfer1::ICudaEngine> mEngine; + BufferManager::IBufferPtr mEngineBuffer; + std::vector<std::unique_ptr<nvinfer1::IExecutionContext>> mContexts; + std::unique_ptr<ITensor> mDummyTensor; + std::unique_ptr<nvinfer1::IEngineInspector> mEngineInspector; + std::unique_ptr<LayerProfiler> mLayerProfiler; + bool mUseShapeInference; + TensorMap mManagedWeightsMap; + // List of input tensor names. + // Names of static tensors are removed from this list when setStaticInputTensors is called. + std::vector<std::string> mInputTensorNames; + std::vector<std::string> mOutputTensorNames; + + bool mUserBufferEnabled; + // For Variable-Beam-Width-Search + std::vector<SizeType32> mCurrentBeamWidths; +}; +} // namespace tensorrt_llm::runtime diff --git a/cpp/tensorrt_llm/runtime/tllmStreamReaders.cpp b/cpp/tensorrt_llm/runtime/tllmStreamReaders.cpp new file mode 100644 index 000000000000..55440bbe714f --- /dev/null +++ b/cpp/tensorrt_llm/runtime/tllmStreamReaders.cpp @@ -0,0 +1,217 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tllmStreamReaders.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" + +#include <cufile.h> +#include <dlfcn.h> +#include <fcntl.h> +#include <filesystem> +#include <fstream> +#include <string> +#include <unistd.h> + +// Non-GDS StreamReader + +StreamReader::StreamReader(std::filesystem::path fp) +{ + mFile.open(fp.string(), std::ios::binary | std::ios::in); + TLLM_CHECK_WITH_INFO(mFile.good(), std::string("Error opening engine file: " + fp.string())); +} + +StreamReader::~StreamReader() +{ + if (mFile.is_open()) + { + mFile.close(); + } +} + +int64_t StreamReader::read(void* destination, int64_t nbBytes) +{ + if (!mFile.good()) + { + return -1; + } + + mFile.read(static_cast<char*>(destination), nbBytes); + + return mFile.gcount(); +} + +// StreamReader using GDS + +GDSStreamReader::GDSStreamReader(std::filesystem::path const& filePath) +{ + auto const start_time = std::chrono::high_resolution_clock::now(); + initializeDriver(); + auto const elapsed_ms + = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::high_resolution_clock::now() - start_time); + + TLLM_LOG_INFO("GDS driver initialization time %lld ms", elapsed_ms); + + open(filePath); +} + +bool GDSStreamReader::open(std::string const& filepath) +{ + if (!initializeDriver()) + { + TLLM_LOG_INFO("Failed to initialize cuFile driver"); + return false; + } + + int32_t const ret = ::open(filepath.c_str(), O_CREAT | O_RDWR | O_DIRECT, 0664); + + if (ret < 0) + { + TLLM_LOG_INFO("Failed to open engine file"); + return false; + } + + mFd = ret; + mFileSize = lseek(mFd, 0, SEEK_END); + lseek(mFd, 0, SEEK_SET); + + CUfileDescr_t fileDescr; + memset((void*) &fileDescr, 0, sizeof(fileDescr)); + fileDescr.handle.fd = mFd; + fileDescr.type = CU_FILE_HANDLE_TYPE_OPAQUE_FD; + + CUfileError_t gdsStatus = cuFileHandleRegister(&mFileHandle, &fileDescr); + + if (gdsStatus.err != CU_FILE_SUCCESS) + { + TLLM_LOG_INFO("Failed to cuFileHandleRegister"); + ::close(mFd); + return false; + } + return true; +} + +void GDSStreamReader::close() +{ + if (mFd >= 0) + { + ::close(mFd); + mFd = -1; + } +} + +GDSStreamReader::~GDSStreamReader() +{ + if (mFileHandle) + { + cuFileHandleDeregister(mFileHandle); + mFileHandle = nullptr; + } + + if (mDriverInitialized) + { + cuFileDriverClose(); + } +} + +bool GDSStreamReader::seek(int64_t offset, nvinfer1::SeekPosition where) noexcept +{ + switch (where) + { + case nvinfer1::SeekPosition::kSET: mCursor = offset; return true; + case nvinfer1::SeekPosition::kCUR: mCursor += offset; return true; + case nvinfer1::SeekPosition::kEND: mCursor = -offset; return true; + default: return false; + } + return true; +} + +int64_t GDSStreamReader::read(void* dest, int64_t bytes, cudaStream_t stream) noexcept +{ + cudaPointerAttributes attributes{}; + if (cudaPointerGetAttributes(&attributes, dest) != cudaSuccess) + { + TLLM_LOG_INFO("cudaPointerGetAttributes failed"); + } + + off_t destOffset = 0; + void* destBase = dest; + + if (attributes.type == cudaMemoryTypeDevice) + { + CUdeviceptr cuDest = reinterpret_cast<CUdeviceptr>(dest); + CUdeviceptr cuBufBase = 0; + size_t cuBufSize = 0; + + cuMemGetAddressRange(&cuBufBase, &cuBufSize, cuDest); + destOffset += cuDest - cuBufBase; + destBase = reinterpret_cast<void*>(cuBufBase); + } + cuFileRead(this->mFileHandle, destBase, bytes, mCursor, destOffset); + + mCursor += bytes; + return bytes; +} + +void GDSStreamReader::reset() +{ + lseek(mFd, 0, SEEK_SET); + mCursor = 0; +} + +[[nodiscard]] bool GDSStreamReader::isOpen() const +{ + bool open = mFd >= 0; + return open; +} + +bool GDSStreamReader::initializeDriver() +{ + if (mDriverInitialized) + { + return true; + } + + mCuFileLibHandle = dlopen("libcufile.so", RTLD_LAZY | RTLD_GLOBAL); + if (!mCuFileLibHandle) + { + TLLM_LOG_INFO("Failed to dlopen libcufile.so"); + return false; + } + + // Load the required functions + *reinterpret_cast<void**>(&cuFileDriverOpen) = dlsym(mCuFileLibHandle, "cuFileDriverOpen"); + *reinterpret_cast<void**>(&cuFileHandleRegister) = dlsym(mCuFileLibHandle, "cuFileHandleRegister"); + *reinterpret_cast<void**>(&cuFileHandleDeregister) = dlsym(mCuFileLibHandle, "cuFileHandleDeregister"); + *reinterpret_cast<void**>(&cuFileDriverClose) = dlsym(mCuFileLibHandle, "cuFileDriverClose"); + *reinterpret_cast<void**>(&cuFileRead) = dlsym(mCuFileLibHandle, "cuFileRead"); + + if (!cuFileDriverOpen || !cuFileHandleRegister || !cuFileHandleDeregister || !cuFileDriverClose || !cuFileRead) + { + TLLM_LOG_INFO("Failed to dlsym libcufile.so"); + return false; + } + + CUfileError_t gdsStatus = cuFileDriverOpen(); + if (gdsStatus.err != CU_FILE_SUCCESS) + { + TLLM_LOG_INFO("cuFileDriverOpen failed"); + return false; + } + + mDriverInitialized = true; + return true; +} diff --git a/cpp/tensorrt_llm/runtime/tllmStreamReaders.h b/cpp/tensorrt_llm/runtime/tllmStreamReaders.h new file mode 100644 index 000000000000..943f0bb3e32e --- /dev/null +++ b/cpp/tensorrt_llm/runtime/tllmStreamReaders.h @@ -0,0 +1,71 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include <NvInferRuntime.h> + +#include <cufile.h> +#include <filesystem> +#include <fstream> + +class StreamReader final : public nvinfer1::IStreamReader +{ +public: + StreamReader(std::filesystem::path fp); + + virtual ~StreamReader(); + + int64_t read(void* destination, int64_t nbBytes) final; + +private: + std::ifstream mFile; +}; + +class GDSStreamReader final : public nvinfer1::IStreamReaderV2 +{ +public: + explicit GDSStreamReader(std::filesystem::path const& filePath); + + virtual ~GDSStreamReader(); + + void close(); + + [[nodiscard]] bool isOpen() const; + + bool open(std::string const& filepath); + + int64_t read(void* dest, int64_t bytes, cudaStream_t stream) noexcept final; + + void reset(); + + bool seek(int64_t offset, nvinfer1::SeekPosition where) noexcept final; + +private: + bool initializeDriver(); + + void* mCuFileLibHandle{}; + CUfileHandle_t mFileHandle{nullptr}; + bool mDriverInitialized{false}; + int32_t mFd{-1}; + int64_t mCursor{0}; + int64_t mFileSize{0}; + + CUfileError_t (*cuFileDriverOpen)(){}; + CUfileError_t (*cuFileHandleRegister)(CUfileHandle_t*, CUfileDescr_t*){}; + CUfileError_t (*cuFileHandleDeregister)(CUfileHandle_t){}; + CUfileError_t (*cuFileDriverClose)(){}; + ssize_t (*cuFileRead)(CUfileHandle_t, void*, size_t, int64_t, int64_t){}; +}; diff --git a/cpp/tensorrt_llm/runtime/utils/debugUtils.cu b/cpp/tensorrt_llm/runtime/utils/debugUtils.cu index d4aaa8244c11..661dacd9a7ac 100644 --- a/cpp/tensorrt_llm/runtime/utils/debugUtils.cu +++ b/cpp/tensorrt_llm/runtime/utils/debugUtils.cu @@ -26,7 +26,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include <cfloat> #include <string> @@ -168,7 +167,7 @@ template <typename T> bool tensorHasInvalid(ITensor const& tensor, BufferManager const& manager, std::string const& infoStr) { printLogitsKeyInfo<T>(tensor, infoStr); - auto foundInvalid = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + auto foundInvalid = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); auto foundInvalidPtr = bufferCast<int32_t>(*foundInvalid); foundInvalidPtr[0] = 0; auto const size = tensor.getSize(); @@ -185,24 +184,24 @@ template bool tensorHasInvalid<__nv_fp8_e4m3>( ITensor const& tensor, BufferManager const& manager, std::string const& infoStr); bool tensorHasInvalid( - size_t M, size_t K, tensorrt_llm::DataType type, void const* data, cudaStream_t stream, std::string const& infoStr) + size_t M, size_t K, nvinfer1::DataType type, void const* data, cudaStream_t stream, std::string const& infoStr) { auto tensorView = ITensor::wrap( const_cast<void*>(data), type, ITensor::makeShape({static_cast<int32_t>(M), static_cast<int32_t>(K)})); auto manager = BufferManager(std::make_shared<CudaStream>(stream)); - if (type == tensorrt_llm::DataType::kFLOAT) + if (type == nvinfer1::DataType::kFLOAT) { return tensorHasInvalid<float>(*tensorView, manager, infoStr); } - else if (type == tensorrt_llm::DataType::kHALF) + else if (type == nvinfer1::DataType::kHALF) { return tensorHasInvalid<half>(*tensorView, manager, infoStr); } - else if (type == tensorrt_llm::DataType::kBF16) + else if (type == nvinfer1::DataType::kBF16) { return tensorHasInvalid<__nv_bfloat16>(*tensorView, manager, infoStr); } - else if (type == tensorrt_llm::DataType::kFP8) + else if (type == nvinfer1::DataType::kFP8) { return tensorHasInvalid<__nv_fp8_e4m3>(*tensorView, manager, infoStr); } diff --git a/cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp b/cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp index ae508bbdbc3b..0f8f31082e96 100644 --- a/cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp +++ b/cpp/tensorrt_llm/runtime/utils/mpiUtils.cpp @@ -597,16 +597,9 @@ MpiComm::~MpiComm() noexcept #if ENABLE_MULTI_DEVICE if (mFreeComm && mComm) { - // Calling MPI_Comm_free after MPI has been finalized is undefined behavior. - // We need this check to prevent heap corruption during program exit when - // static MpiComm objects are created. - int finalized = 0; - if (MPI_Finalized(&finalized) == MPI_SUCCESS && !finalized) + if (MPI_Comm_free(&mComm) != MPI_SUCCESS) { - if (MPI_Comm_free(&mComm) != MPI_SUCCESS) - { - TLLM_LOG_ERROR("MPI_Comm_free failed"); - } + TLLM_LOG_ERROR("MPI_Comm_free failed"); } } #endif // ENABLE_MULTI_DEVICE diff --git a/cpp/tensorrt_llm/runtime/utils/numpyUtils.cpp b/cpp/tensorrt_llm/runtime/utils/numpyUtils.cpp index 931fbf0203da..6f95704455d3 100644 --- a/cpp/tensorrt_llm/runtime/utils/numpyUtils.cpp +++ b/cpp/tensorrt_llm/runtime/utils/numpyUtils.cpp @@ -20,9 +20,9 @@ #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/memoryUtils.h" #include "tensorrt_llm/common/stringUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" +#include <NvInferRuntime.h> #include <sstream> #include <stdexcept> @@ -34,9 +34,9 @@ namespace tc = tensorrt_llm::common; namespace tensorrt_llm::runtime::utils { -std::string getNumpyTypeDesc(tensorrt_llm::DataType type) +std::string getNumpyTypeDesc(nvinfer1::DataType type) { - using dt = tensorrt_llm::DataType; + using dt = nvinfer1::DataType; static std::unordered_map<dt, std::string> const type_map{{dt::kBOOL, "?"}, {dt::kUINT8, "u1"}, {dt::kINT8, "i1"}, {dt::kINT32, "i4"}, {dt::kINT64, "i8"}, {dt::kHALF, "f2"}, {dt::kFLOAT, "f4"}}; @@ -51,11 +51,11 @@ std::string getNumpyTypeDesc(tensorrt_llm::DataType type) return type_map.count(type) > 0 ? type_map.at(type) : "x"; } -tensorrt_llm::DataType typeFromNumpyDesc(std::string const& type) +nvinfer1::DataType typeFromNumpyDesc(std::string const& type) { TLLM_LOG_DEBUG("numpy type: %s", type.c_str()); - using dt = tensorrt_llm::DataType; + using dt = nvinfer1::DataType; static std::unordered_map<std::string, dt> const type_map{{"?", dt::kBOOL}, {"u1", dt::kUINT8}, {"i1", dt::kINT8}, {"i4", dt::kINT32}, {"i8", dt::kINT64}, {"f2", dt::kHALF}, {"f4", dt::kFLOAT}}; TLLM_CHECK_WITH_INFO(type_map.count(type) > 0, "numpy data type '" + type + "' not supported"); @@ -102,7 +102,7 @@ void parseNpyIntro(FILE*& f_ptr, uint32_t& header_len, uint32_t& start_data) start_data = 8 + 2 * npy_major + header_len; } -int parseNpyHeader(FILE*& f_ptr, uint32_t header_len, tensorrt_llm::DataType& type, std::vector<size_t>& shapeVec) +int parseNpyHeader(FILE*& f_ptr, uint32_t header_len, nvinfer1::DataType& type, std::vector<size_t>& shapeVec) { char* header_c = (char*) malloc(header_len * sizeof(char)); TLLM_CHECK_WITH_INFO(header_c != nullptr, "Failed to allocate memory for npy header"); @@ -168,11 +168,11 @@ int parseNpyHeader(FILE*& f_ptr, uint32_t header_len, tensorrt_llm::DataType& ty uint32_t header_len, start_data; utils::parseNpyIntro(f_ptr, header_len, start_data); - tensorrt_llm::DataType type; + nvinfer1::DataType type; std::vector<size_t> shape; utils::parseNpyHeader(f_ptr, header_len, type, shape); - tensorrt_llm::Dims dims; + nvinfer1::Dims dims; dims.nbDims = shape.size(); std::copy(shape.begin(), shape.end(), dims.d); @@ -203,10 +203,10 @@ void saveNpy(BufferManager const& manager, ITensor const& tensor, std::string co auto const dtype = tensor.getDataType(); #ifdef ENABLE_BF16 - if (dtype == tensorrt_llm::DataType::kBF16) + if (dtype == nvinfer1::DataType::kBF16) { TLLM_CHECK(where == MemoryType::kGPU); - auto tensorFp32 = manager.gpu(shape, tensorrt_llm::DataType::kFLOAT); + auto tensorFp32 = manager.gpu(shape, nvinfer1::DataType::kFLOAT); auto dataFp32 = bufferCast<float>(*tensorFp32); auto dataBf16 = bufferCast<__nv_bfloat16 const>(tensor); tc::invokeCudaD2DcpyConvert(dataFp32, dataBf16, tensorSize); diff --git a/cpp/tensorrt_llm/runtime/utils/pgUtils.cpp b/cpp/tensorrt_llm/runtime/utils/pgUtils.cpp index 4f15dec76781..77e21d6d5bfd 100644 --- a/cpp/tensorrt_llm/runtime/utils/pgUtils.cpp +++ b/cpp/tensorrt_llm/runtime/utils/pgUtils.cpp @@ -41,10 +41,4 @@ void init_pg(c10::intrusive_ptr<c10d::ProcessGroup> const& process_group_world, pg_local = process_group_local; } -void shutdown_pg() -{ - pg_world.reset(); - pg_local.reset(); -} - } // namespace tensorrt_llm::pg_utils diff --git a/cpp/tensorrt_llm/runtime/utils/runtimeUtils.h b/cpp/tensorrt_llm/runtime/utils/runtimeUtils.h index da94419eff64..f131ab3419bf 100644 --- a/cpp/tensorrt_llm/runtime/utils/runtimeUtils.h +++ b/cpp/tensorrt_llm/runtime/utils/runtimeUtils.h @@ -25,6 +25,7 @@ namespace tensorrt_llm::runtime { +class TllmRuntime; namespace utils { diff --git a/cpp/tensorrt_llm/runtime/virtualMemory.cpp b/cpp/tensorrt_llm/runtime/virtualMemory.cpp index c2ca710db9ac..0d08012a29d8 100644 --- a/cpp/tensorrt_llm/runtime/virtualMemory.cpp +++ b/cpp/tensorrt_llm/runtime/virtualMemory.cpp @@ -16,7 +16,6 @@ #include "tensorrt_llm/runtime/virtualMemory.h" #include "bufferManager.h" -#include "tensorrt_llm/common/tllmDataType.h" #include <forward_list> #include <shared_mutex> @@ -142,8 +141,8 @@ void OffloadConfigurator::teardown(CUmemGenericAllocationHandle, bool destructin { switch (mBackType) { - case MemoryType::kCPU: mBackedStorage = BufferManager::cpu(mSize, tensorrt_llm::DataType::kINT8); break; - case MemoryType::kPINNED: mBackedStorage = BufferManager::pinned(mSize, tensorrt_llm::DataType::kINT8); break; + case MemoryType::kCPU: mBackedStorage = BufferManager::cpu(mSize, nvinfer1::DataType::kINT8); break; + case MemoryType::kPINNED: mBackedStorage = BufferManager::pinned(mSize, nvinfer1::DataType::kINT8); break; default: TLLM_THROW("Unknown memory type: %d", static_cast<int32_t>(mBackType)); } } diff --git a/cpp/tensorrt_llm/testing/CMakeLists.txt b/cpp/tensorrt_llm/testing/CMakeLists.txt new file mode 100644 index 000000000000..646302929817 --- /dev/null +++ b/cpp/tensorrt_llm/testing/CMakeLists.txt @@ -0,0 +1,38 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +include(FetchContent) + +set(SRCS modelSpec.cpp) + +include_directories(${API_INCLUDE_DIR}/tensorrt_llm/runtime) + +if(NOT WIN32) + # additional warnings + # + # Ignore overloaded-virtual warning. We intentionally change parameters of + # some methods in derived class. + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall") + if(WARNING_IS_ERROR) + message(STATUS "Treating warnings as errors in GCC compilation") + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Werror") + endif() +else() # Windows + # warning level 4 + set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /W4") +endif() + +add_library(testing_src OBJECT ${SRCS}) +set_property(TARGET testing_src PROPERTY POSITION_INDEPENDENT_CODE ON) +set_property(TARGET testing_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON) diff --git a/cpp/tensorrt_llm/testing/modelSpec.cpp b/cpp/tensorrt_llm/testing/modelSpec.cpp new file mode 100644 index 000000000000..bc868a157a1c --- /dev/null +++ b/cpp/tensorrt_llm/testing/modelSpec.cpp @@ -0,0 +1,303 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "modelSpec.h" +#include "tensorrt_llm/common/dataType.h" + +#include <numeric> + +namespace tensorrt_llm::testing +{ + +std::string ModelSpec::getQuantMethodString() const +{ + switch (mQuantMethod) + { + case QuantMethod::kNONE: + // Bypass here. + break; + case QuantMethod::kSMOOTH_QUANT: return "sq"; break; + default: throw std::runtime_error("Unsupported quant method"); break; + } + + return ""; +} + +std::string ModelSpec::getKVCacheTypeString() const +{ + switch (mKVCacheType) + { + case KVCacheType::kDISABLED: return "no-cache"; break; + case KVCacheType::kPAGED: return "paged"; break; + case KVCacheType::kCONTINUOUS: return "continuous"; break; + default: throw std::runtime_error("Unsupported KV cache type"); break; + } + + return ""; +} + +std::string ModelSpec::getSpeculativeDecodingModeString() const +{ + if (mSpecDecodingMode.isLookaheadDecoding()) + { + return "la-decoding"; + } + else if (mSpecDecodingMode.isDraftTokensExternal()) + { + return "draft-tokens"; + } + else if (mSpecDecodingMode.isNone()) + { + // Bypass here. + } + else if (mSpecDecodingMode.isExplicitDraftTokens()) + { + return "explicit-draft-tokens"; + } + else if (mSpecDecodingMode.isMedusa()) + { + return "medusa"; + } + else if (mSpecDecodingMode.isEagle()) + { + return "eagle"; + } + else + { + throw std::runtime_error("Unsupported decoding mode"); + } + + return ""; +} + +std::string ModelSpec::getCapacitySchedulerString() const +{ + if (mCapacitySchedulerPolicy) + { + if (mCapacitySchedulerPolicy.value() == tensorrt_llm::executor::CapacitySchedulerPolicy::kMAX_UTILIZATION) + { + return "MaxUtilization"; + } + else if (mCapacitySchedulerPolicy.value() + == tensorrt_llm::executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT) + { + return "GuaranteedNoEvict"; + } + else if (mCapacitySchedulerPolicy.value() == tensorrt_llm::executor::CapacitySchedulerPolicy::kSTATIC_BATCH) + { + return "StaticBatch"; + } + else + { + throw std::runtime_error("Unsupported capacity scheduler"); + } + } + return ""; +} + +std::string ModelSpec::getInputFile() const +{ + return mInputFile; +} + +std::string ModelSpec::getModelPath() const +{ + std::vector<std::string> ret; + + ret.emplace_back(getDtypeString()); + + if (mUseGptAttentionPlugin || mUseMambaPlugin) + { + if (mUseGptAttentionPlugin && mUseMambaPlugin) + { + throw std::runtime_error("Cannot use both GPT attention plugin and MAMBA plugin"); + } + + ret.emplace_back("plugin"); + } + else + { + ret.emplace_back("default"); + } + + if (mUsePackedInput) + { + ret.emplace_back("packed"); + } + + ret.emplace_back(getKVCacheTypeString()); + + if (mMaxInputLength) + { + ret.emplace_back("in" + std::to_string(mMaxInputLength)); + } + + ret.emplace_back(getSpeculativeDecodingModeString()); + + if (mUseLoraPlugin) + { + ret.emplace_back("lora"); + } + + ret.emplace_back(getQuantMethodString()); + + if (mUseMultipleProfiles) + { + ret.emplace_back("nprofiles"); + } + + if (mGatherLogits) + { + ret.emplace_back("gather"); + } + + auto finalRet = std::accumulate(ret.begin(), ret.end(), std::string(), + [](std::string& a, std::string& b) + { + if (a.empty()) + { + return b; + } + else + { + return b.empty() ? a : a + "_" + b; + } + }); + + return finalRet; +} + +std::string ModelSpec::getResultsFileInternal(OutputContentType outputContentType) const +{ + std::vector<std::string> ret; + + if (mInputFile == "input_tokens_long.npy") + { + ret.emplace_back("output_tokens_long"); + } + else + { + ret.emplace_back("output_tokens"); + } + + if (mMaxOutputLength) + { + ret.emplace_back("out" + std::to_string(mMaxOutputLength)); + } + + ret.emplace_back(getDtypeString()); + + if (mUseGptAttentionPlugin || mUseMambaPlugin) + { + if (mUseGptAttentionPlugin && mUseMambaPlugin) + { + throw std::runtime_error("Cannot use both GPT attention plugin and MAMBA plugin"); + } + ret.emplace_back("plugin"); + } + + if (mUsePackedInput) + { + ret.emplace_back("packed"); + } + + ret.emplace_back(getKVCacheTypeString()); + + ret.emplace_back(getQuantMethodString()); + + if (mGatherLogits) + { + ret.emplace_back("gather"); + } + + ret.emplace_back("tp" + std::to_string(mTPSize)); + + ret.emplace_back("pp" + std::to_string(mPPSize)); + + ret.emplace_back("cp" + std::to_string(mCPSize)); + + if (mEnableContextFMHAFp32Acc) + { + ret.emplace_back("fmhafp32acc"); + } + + switch (outputContentType) + { + case OutputContentType::kNONE: + // Bypass here. + break; + case OutputContentType::kCONTEXT_LOGITS: ret.emplace_back("logits_context"); break; + case OutputContentType::kGENERATION_LOGITS: ret.emplace_back("logits_generation"); break; + case OutputContentType::kLOG_PROBS: ret.emplace_back("log_probs"); break; + case OutputContentType::kCUM_LOG_PROBS: ret.emplace_back("cum_log_probs"); break; + default: throw std::runtime_error("Unsupported output content type"); break; + } + + auto finalRet = std::accumulate(ret.begin(), ret.end(), std::string(), + [](std::string& a, std::string& b) + { + if (a.empty()) + { + return b; + } + else + { + return b.empty() ? a : a + "_" + b; + } + }); + return finalRet + ".npy"; +} + +std::string ModelSpec::getResultsFile() const +{ + return mOtherModelSpecToCompare ? mOtherModelSpecToCompare->getResultsFileInternal(OutputContentType::kNONE) + : getResultsFileInternal(OutputContentType::kNONE); +} + +std::string ModelSpec::getGenerationLogitsFile() const +{ + return mOtherModelSpecToCompare + ? mOtherModelSpecToCompare->getResultsFileInternal(OutputContentType::kGENERATION_LOGITS) + : getResultsFileInternal(OutputContentType::kGENERATION_LOGITS); +} + +std::string ModelSpec::getContextLogitsFile() const +{ + return mOtherModelSpecToCompare + ? mOtherModelSpecToCompare->getResultsFileInternal(OutputContentType::kCONTEXT_LOGITS) + : getResultsFileInternal(OutputContentType::kCONTEXT_LOGITS); +} + +std::string ModelSpec::getCumLogProbsFile() const +{ + return mOtherModelSpecToCompare + ? mOtherModelSpecToCompare->getResultsFileInternal(OutputContentType::kCUM_LOG_PROBS) + : getResultsFileInternal(OutputContentType::kCUM_LOG_PROBS); +} + +std::string ModelSpec::getLogProbsFile() const +{ + return mOtherModelSpecToCompare ? mOtherModelSpecToCompare->getResultsFileInternal(OutputContentType::kLOG_PROBS) + : getResultsFileInternal(OutputContentType::kLOG_PROBS); +} + +std::string ModelSpec::getDtypeString() const +{ + return tensorrt_llm::common::getDtypeString(mDataType); +} + +} // namespace tensorrt_llm::testing diff --git a/cpp/tensorrt_llm/testing/modelSpec.h b/cpp/tensorrt_llm/testing/modelSpec.h new file mode 100644 index 000000000000..5b6f88dcd135 --- /dev/null +++ b/cpp/tensorrt_llm/testing/modelSpec.h @@ -0,0 +1,342 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "NvInfer.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/speculativeDecodingMode.h" + +#include <filesystem> +#include <vector> + +namespace tensorrt_llm::testing +{ + +using tensorrt_llm::runtime::SizeType32; +using tensorrt_llm::runtime::SpeculativeDecodingMode; +using KVCacheType = tensorrt_llm::runtime::ModelConfig::KVCacheType; + +enum class QuantMethod +{ + kNONE, + kSMOOTH_QUANT, +}; + +enum class OutputContentType +{ + kNONE, + kCONTEXT_LOGITS, + kGENERATION_LOGITS, + kLOG_PROBS, + kCUM_LOG_PROBS +}; + +class ModelSpec +{ +public: + ModelSpec(std::string const& inputFile, nvinfer1::DataType dtype, + std::shared_ptr<ModelSpec> otherModelSpecToCompare = nullptr) + : mInputFile{std::move(inputFile)} + , mDataType{dtype} + , mOtherModelSpecToCompare(otherModelSpecToCompare) + { + } + + ModelSpec& setInputFile(std::string const& inputFile) + { + mInputFile = inputFile; + return *this; + } + + ModelSpec& useGptAttentionPlugin() + { + mUseGptAttentionPlugin = true; + return *this; + } + + ModelSpec& usePackedInput() + { + mUsePackedInput = true; + return *this; + } + + ModelSpec& setKVCacheType(KVCacheType kvCacheType) + { + mKVCacheType = kvCacheType; + return *this; + } + + ModelSpec& setKVCacheReuse(bool kvCacheReuse) + { + mKVCacheReuse = kvCacheReuse; + return *this; + } + + ModelSpec& useDecoderPerRequest() + { + mDecoderPerRequest = true; + return *this; + } + + ModelSpec& useTensorParallelism(int tensorParallelism) + { + mTPSize = tensorParallelism; + return *this; + } + + ModelSpec& usePipelineParallelism(int pipelineParallelism) + { + mPPSize = pipelineParallelism; + return *this; + } + + ModelSpec& useContextParallelism(int contextParallelism) + { + mCPSize = contextParallelism; + return *this; + } + + ModelSpec& setDraftTokens(SizeType32 maxDraftTokens) + { + mMaxDraftTokens = maxDraftTokens; + return *this; + } + + ModelSpec& useAcceptByLogits() + { + mAcceptDraftByLogits = true; + return *this; + } + + ModelSpec& useMambaPlugin() + { + mUseMambaPlugin = true; + return *this; + } + + ModelSpec& gatherLogits() + { + mGatherLogits = true; + return *this; + } + + ModelSpec& replaceLogits() + { + mReplaceLogits = true; + return *this; + } + + ModelSpec& returnLogProbs() + { + mReturnLogProbs = true; + return *this; + } + + ModelSpec& smokeTest() + { + mSmokeTest = true; + return *this; + } + + ModelSpec& useMedusa() + { + mSpecDecodingMode = SpeculativeDecodingMode::Medusa(); + return *this; + } + + ModelSpec& useEagle() + { + mSpecDecodingMode = SpeculativeDecodingMode::Eagle(); + return *this; + } + + ModelSpec& useLookaheadDecoding() + { + mSpecDecodingMode = SpeculativeDecodingMode::LookaheadDecoding(); + return *this; + } + + ModelSpec& useExplicitDraftTokensDecoding() + { + mSpecDecodingMode = SpeculativeDecodingMode::ExplicitDraftTokens(); + return *this; + } + + ModelSpec& useDraftTokensExternalDecoding() + { + mSpecDecodingMode = SpeculativeDecodingMode::DraftTokensExternal(); + return *this; + } + + [[nodiscard]] bool useLogits() const + { + return mGatherLogits || mReplaceLogits; + } + + ModelSpec& useMultipleProfiles() + { + mUseMultipleProfiles = true; + return *this; + } + + ModelSpec& enableContextFMHAFp32Acc() + { + mEnableContextFMHAFp32Acc = true; + return *this; + } + + [[nodiscard]] bool getEnableContextFMHAFp32Acc() const + { + return mEnableContextFMHAFp32Acc; + } + + ModelSpec& setMaxInputLength(SizeType32 maxInputLength) + { + mMaxInputLength = maxInputLength; + return *this; + } + + ModelSpec& setMaxOutputLength(SizeType32 maxOutputLength) + { + mMaxOutputLength = maxOutputLength; + return *this; + } + + ModelSpec& setQuantMethod(QuantMethod quantMethod) + { + mQuantMethod = quantMethod; + return *this; + } + + ModelSpec& useLoraPlugin() + { + mUseLoraPlugin = true; + return *this; + } + + ModelSpec& collectGenerationLogitsFile() + { + mCollectGenerationLogits = true; + return *this; + } + + ModelSpec& collectContextLogitsFile() + { + mCollectContextLogits = true; + return *this; + } + + ModelSpec& collectCumLogProbsFile() + { + mCollectCumLogProbs = true; + return *this; + } + + ModelSpec& collectLogProbsFile() + { + mCollectLogProbs = true; + return *this; + } + + ModelSpec& capacitySchedulerPolicy(tensorrt_llm::executor::CapacitySchedulerPolicy policy) + { + mCapacitySchedulerPolicy = policy; + return *this; + } + + friend std::ostream& operator<<(std::ostream& os, ModelSpec const& modelSpec) + { + return os << modelSpec.getModelPath(); + } + + // Computed properties + [[nodiscard]] std::string getInputFile() const; + + [[nodiscard]] std::string getModelPath() const; + + [[nodiscard]] std::string getResultsFileInternal( + OutputContentType outputContentType = OutputContentType::kNONE) const; + + [[nodiscard]] std::string getResultsFile() const; + [[nodiscard]] std::string getGenerationLogitsFile() const; + + [[nodiscard]] std::string getContextLogitsFile() const; + + [[nodiscard]] std::string getCumLogProbsFile() const; + + [[nodiscard]] std::string getLogProbsFile() const; + + [[nodiscard]] std::string getDtypeString() const; + + [[nodiscard]] std::string getQuantMethodString() const; + + [[nodiscard]] std::string getKVCacheTypeString() const; + + [[nodiscard]] std::string getSpeculativeDecodingModeString() const; + + [[nodiscard]] std::string getCapacitySchedulerString() const; + + static ModelSpec getDefaultModelSpec() + { + static ModelSpec modelSpec{"input_tokens.npy", nvinfer1::DataType::kHALF}; + modelSpec.useGptAttentionPlugin().setKVCacheType(KVCacheType::kPAGED).usePackedInput(); + + return modelSpec; + } + + std::string mInputFile; + nvinfer1::DataType mDataType; + + bool mUseGptAttentionPlugin{false}; + bool mUsePackedInput{false}; + KVCacheType mKVCacheType{KVCacheType::kCONTINUOUS}; + bool mKVCacheReuse{false}; + bool mDecoderPerRequest{false}; + int mPPSize{1}; + int mTPSize{1}; + int mCPSize{1}; + int mMaxDraftTokens{0}; + bool mAcceptDraftByLogits{false}; + bool mUseMambaPlugin{false}; + bool mGatherLogits{false}; + bool mReplaceLogits{false}; + bool mReturnLogProbs{false}; + bool mSmokeTest{false}; + bool mUseMultipleProfiles{false}; + int mMaxInputLength{0}; + int mMaxOutputLength{0}; + bool mUseLoraPlugin{false}; + bool mEnableContextFMHAFp32Acc{false}; + + // Flags to store whether model spec wants collect these outputs, you could call getXXXFile() if you need the name. + bool mCollectGenerationLogits{false}; + bool mCollectContextLogits{false}; + bool mCollectCumLogProbs{false}; + bool mCollectLogProbs{false}; + QuantMethod mQuantMethod{QuantMethod::kNONE}; + + SpeculativeDecodingMode mSpecDecodingMode{SpeculativeDecodingMode::None()}; + + std::optional<tensorrt_llm::executor::CapacitySchedulerPolicy> mCapacitySchedulerPolicy{std::nullopt}; + + // Sometimes, we need to compare with another model spec for golden results. + std::shared_ptr<ModelSpec> mOtherModelSpecToCompare{nullptr}; +}; + +}; // namespace tensorrt_llm::testing diff --git a/cpp/tensorrt_llm/thop/CMakeLists.txt b/cpp/tensorrt_llm/thop/CMakeLists.txt index b95426f580fe..0517ab597dfe 100644 --- a/cpp/tensorrt_llm/thop/CMakeLists.txt +++ b/cpp/tensorrt_llm/thop/CMakeLists.txt @@ -31,11 +31,8 @@ endif() add_library(th_utils STATIC thUtils.cpp) set_property(TARGET th_utils PROPERTY POSITION_INDEPENDENT_CODE ON) set_property(TARGET th_utils PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON) -# Declare the dependency on the main shared library explicitly so consumers -# (e.g. thUtilsTest) place it after th_utils on the link line; this was -# previously satisfied transitively via the removed TensorRT plugin target. -target_link_libraries(th_utils PUBLIC ${SHARED_TARGET} ${TORCH_LIBRARIES} - ${CUBLAS_LIB} ${CURAND_LIB}) +target_link_libraries(th_utils PUBLIC ${TORCH_LIBRARIES} ${CUBLAS_LIB} + ${CURAND_LIB}) # TODO This does not compile with internal cutlass MOE gemm add_library( @@ -73,7 +70,6 @@ add_library( fp8Quantize.cpp dsv3FusedAGemmOp.cpp deepseekV4QNormOp.cpp - deepseekV4BlockTableOp.cpp inverseRopeFp8QuantOp.cpp fusedQKNormRopeOp.cpp fusedDiTQKNormRopeOp.cpp @@ -85,7 +81,6 @@ add_library( fusedAddRMSNormQuant.cpp fusedActivationQuant.cpp fusedGatedRMSNormQuant.cpp - rmsNormFp4Quant.cpp fusedTopkSoftmax.cpp gatherTreeOp.cpp groupRmsNormOp.cpp diff --git a/cpp/tensorrt_llm/thop/IndexerTopKOp.cpp b/cpp/tensorrt_llm/thop/IndexerTopKOp.cpp index fc29041bbf93..1981c417dbae 100644 --- a/cpp/tensorrt_llm/thop/IndexerTopKOp.cpp +++ b/cpp/tensorrt_llm/thop/IndexerTopKOp.cpp @@ -19,6 +19,7 @@ #include "tensorrt_llm/kernels/IndexerTopK.h" +// #include <NvInferRuntime.h> // #include <c10/cuda/CUDAStream.h> // #include <cassert> // #include <set> diff --git a/cpp/tensorrt_llm/thop/allgatherOp.cpp b/cpp/tensorrt_llm/thop/allgatherOp.cpp index 5f7d4571d258..0d92aa966901 100644 --- a/cpp/tensorrt_llm/thop/allgatherOp.cpp +++ b/cpp/tensorrt_llm/thop/allgatherOp.cpp @@ -20,6 +20,7 @@ #include "tensorrt_llm/runtime/utils/mpiUtils.h" #include "tensorrt_llm/runtime/utils/pgUtils.h" +#include <NvInferRuntime.h> #include <c10/cuda/CUDAStream.h> #include <cassert> #include <set> diff --git a/cpp/tensorrt_llm/thop/allreduceOp.cpp b/cpp/tensorrt_llm/thop/allreduceOp.cpp index 38cc79379f6d..4096116fdb42 100644 --- a/cpp/tensorrt_llm/thop/allreduceOp.cpp +++ b/cpp/tensorrt_llm/thop/allreduceOp.cpp @@ -23,7 +23,6 @@ #include "tensorrt_llm/common/ncclUtils.h" #include "tensorrt_llm/common/nvmlWrapper.h" #include "tensorrt_llm/common/opUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/MiniMaxReduceRMSKernel.h" #include "tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.h" #include "tensorrt_llm/kernels/communicationKernels/customLowPrecisionAllReduceKernels.h" @@ -62,6 +61,7 @@ #include <limits> #include <unordered_set> +// using namespace nvinfer1; using tensorrt_llm::kernels::AllReduceFusionOp; using tensorrt_llm::kernels::AllReduceStrategyType; using tensorrt_llm::mpi::MpiTag; @@ -234,8 +234,8 @@ std::set<int> getLocalGroupTorch(std::set<int> const& group) class AllreduceOp { public: - AllreduceOp(std::set<int> group, tensorrt_llm::DataType type, AllReduceStrategyType strategy, AllReduceFusionOp op, - float eps) + AllreduceOp( + std::set<int> group, nvinfer1::DataType type, AllReduceStrategyType strategy, AllReduceFusionOp op, float eps) : mGroup(std::move(group)) , mIsNVLINKSupported(false) , mIsP2PSupported(false) @@ -248,7 +248,7 @@ class AllreduceOp } AllreduceOp(std::set<int> group, c10::intrusive_ptr<c10d::ProcessGroup> const& process_group_, - tensorrt_llm::DataType type, AllReduceStrategyType strategy, AllReduceFusionOp op, float eps) + nvinfer1::DataType type, AllReduceStrategyType strategy, AllReduceFusionOp op, float eps) : mGroup(std::move(group)) , mIsNVLINKSupported(false) , mIsP2PSupported(false) @@ -348,7 +348,7 @@ class AllreduceOp { TORCH_CHECK(norm_weight, "norm_weight is required for residual rms norm allreduce"); TORCH_CHECK(!bias, "bias is not supported for residual rms norm allreduce"); - TORCH_CHECK(mType == tensorrt_llm::DataType::kHALF || mType == tensorrt_llm::DataType::kBF16); + TORCH_CHECK(mType == nvinfer1::DataType::kHALF || mType == nvinfer1::DataType::kBF16); auto [norm_out, ub_buffer1] = torch_ext::create_userbuffers_tensor(input.sizes(), input.scalar_type()); tensorrt_llm::kernels::ub::allreduce2_userbuff_rmsnorm_launcher(ub_buffer0.handle, 0, ub_buffer1.handle, 0, size, hidden_size, nullptr, norm_weight.value().data_ptr(), mEps, residual.value().data_ptr(), @@ -1461,7 +1461,7 @@ class AllreduceOp bool mIsNVLINKSupported; bool mIsP2PSupported; bool mIsMNNVLSupported; - tensorrt_llm::DataType mType; + nvinfer1::DataType mType; AllReduceStrategyType mStrategy; AllReduceFusionOp mOp; float mEps; diff --git a/cpp/tensorrt_llm/thop/attentionOp.cpp b/cpp/tensorrt_llm/thop/attentionOp.cpp index 811810a8d5a2..6f0d24eb6bba 100644 --- a/cpp/tensorrt_llm/thop/attentionOp.cpp +++ b/cpp/tensorrt_llm/thop/attentionOp.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/attentionOp.h" #include "tensorrt_llm/common/attentionWorkspace.h" #include "tensorrt_llm/common/dataType.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/flashMLA/flash_mla.h" #include "tensorrt_llm/kernels/gptKernels.h" #include "tensorrt_llm/kernels/mlaKernels.h" @@ -379,9 +378,7 @@ class RunnerBase std::optional<torch::Tensor> flash_mla_num_splits, bool trtllm_gen_jit_warmup, std::optional<int64_t> compressed_kv_cache_pool_ptr, bool const is_cross, std::optional<torch::Tensor> cross_kv, std::optional<torch::Tensor> relative_attention_bias, - std::optional<torch::Tensor> quant_scale_qkv = std::nullopt, - std::optional<torch::Tensor> dsv4_inv_rope_cos_sin_cache = std::nullopt, - bool enable_dsv4_epilogue_fusion = false) const + std::optional<torch::Tensor> quant_scale_qkv = std::nullopt) const = 0; }; @@ -450,8 +447,8 @@ class Runner : public RunnerBase std::optional<torch::Tensor> flash_mla_tile_scheduler_metadata, std::optional<torch::Tensor> flash_mla_num_splits, bool trtllm_gen_jit_warmup, std::optional<int64_t> compressed_kv_cache_pool_ptr, bool const is_cross, std::optional<torch::Tensor> cross_kv, - std::optional<torch::Tensor> relative_attention_bias, std::optional<torch::Tensor> quant_scale_qkv, - std::optional<torch::Tensor> dsv4_inv_rope_cos_sin_cache, bool enable_dsv4_epilogue_fusion) const override + std::optional<torch::Tensor> relative_attention_bias, + std::optional<torch::Tensor> quant_scale_qkv) const override { auto stream = at::cuda::getCurrentCUDAStream(qkv_or_q.get_device()); T* attention_input = static_cast<T*>(qkv_or_q.slice(0, token_offset).data_ptr()); @@ -459,8 +456,7 @@ class Runner : public RunnerBase T* v_ptr = nullptr; AttentionOutT* context_buf = static_cast<AttentionOutT*>(output.slice(0, token_offset).data_ptr()); TORCH_CHECK(!op.mFuseFp4Quant || output_sf.has_value()); - TORCH_CHECK(!enable_dsv4_epilogue_fusion || output_sf.has_value()); - void* context_buf_sf = (op.mFuseFp4Quant || enable_dsv4_epilogue_fusion) ? output_sf->data_ptr() : nullptr; + void* context_buf_sf = op.mFuseFp4Quant ? output_sf->data_ptr() : nullptr; // Rotary inv_freq, cos_sin cache to avoid re-computing. float const* rotary_inv_freq_ptr = nullptr; @@ -590,32 +586,6 @@ class Runner : public RunnerBase mla_params.context_buf = reinterpret_cast<T*>(context_buf); mla_params.cos_sin_cache = rotary_cos_sin_ptr; - if (enable_dsv4_epilogue_fusion) - { - TORCH_CHECK(dsv4_inv_rope_cos_sin_cache.has_value(), - "DSv4 fused epilogue requires inverse-RoPE cos/sin cache."); - auto const& cos_sin_cache = dsv4_inv_rope_cos_sin_cache.value(); - auto const& output_sf_tensor = output_sf.value(); - TORCH_CHECK(cos_sin_cache.scalar_type() == torch::kFloat32, - "DSv4 fused epilogue cos/sin cache must be float32."); - TORCH_CHECK( - output.scalar_type() == torch::kFloat8_e4m3fn, "DSv4 fused epilogue output must be float8_e4m3fn."); - TORCH_CHECK(output.dim() == 3 && output.is_contiguous(), - "DSv4 fused epilogue output must be contiguous [groups, tokens, K]."); - TORCH_CHECK(output_sf_tensor.scalar_type() == torch::kFloat32, - "DSv4 fused epilogue output_sf must be float32."); - TORCH_CHECK(output_sf_tensor.dim() == 3 && output_sf_tensor.is_contiguous(), - "DSv4 fused epilogue output_sf must be contiguous [groups, K/128, padded_tokens]."); - TORCH_CHECK(output.size(1) >= num_tokens, "DSv4 fused epilogue output token dimension is too small."); - TORCH_CHECK(op.mMLAParams.v_head_dim > 0 && op.mMLAParams.v_head_dim % 128 == 0, - "DSv4 fused epilogue requires v_head_dim to be a positive multiple of 128."); - TORCH_CHECK(output_sf_tensor.size(2) >= num_tokens, - "DSv4 fused epilogue output_sf token dimension is too small."); - - mla_params.dsv4_epilogue_fusion.enabled = true; - mla_params.dsv4_epilogue_fusion.cos_sin_cache = static_cast<float const*>(cos_sin_cache.data_ptr()); - mla_params.dsv4_epilogue_fusion.scale_buf_m = static_cast<int32_t>(output_sf_tensor.size(2)); - } mla_params.batch_size = num_seqs; mla_params.acc_q_len = num_tokens; mla_params.head_num = op.mNumHeads; @@ -1084,9 +1054,7 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to bool sage_attn_qk_int8, int64_t num_contexts, int64_t num_ctx_tokens, bool trtllm_gen_jit_warmup, std::optional<int64_t> compressed_kv_cache_pool_ptr, bool const is_cross, std::optional<torch::Tensor> cross_kv, std::optional<torch::Tensor> relative_attention_bias, int64_t relative_attention_max_distance, - std::optional<int64_t> spec_decoding_target_max_draft_tokens, std::optional<torch::Tensor> quant_scale_qkv, - std::optional<torch::Tensor> dsv4_inv_rope_cos_sin_cache, bool enable_dsv4_epilogue_fusion, - bool const force_prepare_spec_dec_tree_mask) + std::optional<int64_t> spec_decoding_target_max_draft_tokens, std::optional<torch::Tensor> quant_scale_qkv) { TLLM_LOG_TRACE("Attention op starts at layer %d", local_layer_idx); // Use these tensors to infer if the attention is using KV cache @@ -1125,7 +1093,7 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to bool const is_fp4_out = out_dtype == torch::kUInt8; RunnerPtr runner; - if (dtype == tensorrt_llm::DataType::kHALF) + if (dtype == nvinfer1::DataType::kHALF) { if (is_fp8_out) { @@ -1141,13 +1109,13 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to runner = std::make_shared<Runner<half>>(); } } - else if (dtype == tensorrt_llm::DataType::kFLOAT) + else if (dtype == nvinfer1::DataType::kFLOAT) { TLLM_CHECK(out_dtype == torch::kFloat32); runner = std::make_shared<Runner<float>>(); } #ifdef ENABLE_BF16 - else if (dtype == tensorrt_llm::DataType::kBF16) + else if (dtype == nvinfer1::DataType::kBF16) { if (is_fp8_out) { @@ -1170,7 +1138,7 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to auto op = std::make_shared<AttentionOp>(); op->mType = dtype; - op->mFMHAForceFP32Acc = dtype == tensorrt_llm::DataType::kBF16; + op->mFMHAForceFP32Acc = dtype == nvinfer1::DataType::kBF16; op->mLayerIdx = local_layer_idx; op->mNumHeads = num_heads; op->mNumKVHeads = num_kv_heads; @@ -1182,7 +1150,6 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to op->mTokensPerBlock = tokens_per_block.value_or(0); op->mFP8GenerationMLA = false; op->mFuseFp4Quant = is_fp4_out; - op->mFusesDsv4InvRopeFp8Quant = enable_dsv4_epilogue_fusion; op->mMaxContextLength = max_context_length; op->mMaxSeqLen = max_seq_len; op->mMaxNumRequests = max_num_requests; @@ -1239,7 +1206,6 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to { op->mSpecDecodingTargetMaxGenLen = static_cast<int32_t>(spec_decoding_target_max_draft_tokens.value()) + 1; } - op->mForcePrepareSpecDecTreeMask = force_prepare_spec_dec_tree_mask; op->mUseSparseAttention = false; op->mUseTllmGenSparseAttentionPaged = false; @@ -1387,7 +1353,7 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to num_sparse_topk_value, sparse_mla_topk_lens, cu_q_seqlens, cu_kv_seqlens, fmha_scheduler_counter, mla_bmm1_scale, mla_bmm2_scale, quant_q_buffer, flash_mla_tile_scheduler_metadata, flash_mla_num_splits, trtllm_gen_jit_warmup, compressed_kv_cache_pool_ptr, is_cross, cross_kv, relative_attention_bias, - quant_scale_qkv, dsv4_inv_rope_cos_sin_cache, enable_dsv4_epilogue_fusion); + quant_scale_qkv); } if ((num_generations > 0) && (attn_input_type != AttentionInputType::ContextOnly)) @@ -1410,7 +1376,7 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to num_sparse_topk_value, sparse_mla_topk_lens, cu_q_seqlens, cu_kv_seqlens, fmha_scheduler_counter, mla_bmm1_scale, mla_bmm2_scale, quant_q_buffer, flash_mla_tile_scheduler_metadata, flash_mla_num_splits, trtllm_gen_jit_warmup, compressed_kv_cache_pool_ptr, is_cross, cross_kv, relative_attention_bias, - quant_scale_qkv, dsv4_inv_rope_cos_sin_cache, enable_dsv4_epilogue_fusion); + quant_scale_qkv); } TLLM_LOG_TRACE("Attention op stops at layer %d", local_layer_idx); @@ -1434,7 +1400,7 @@ bool attention_supports_nvfp4_output(int64_t const num_heads, int64_t const num_ } auto op = std::make_shared<AttentionOp>(); - op->mType = tensorrt_llm::DataType::kHALF; + op->mType = nvinfer1::DataType::kHALF; op->mNumHeads = num_heads; op->mNumKVHeads = num_kv_heads; op->mHeadSize = head_size; diff --git a/cpp/tensorrt_llm/thop/attentionOp.h b/cpp/tensorrt_llm/thop/attentionOp.h index f28209166e0d..94b067ae51be 100644 --- a/cpp/tensorrt_llm/thop/attentionOp.h +++ b/cpp/tensorrt_llm/thop/attentionOp.h @@ -94,9 +94,7 @@ void attention(torch::Tensor q, std::optional<torch::Tensor> k, std::optional<to std::optional<torch::Tensor> cross_kv = std::nullopt, std::optional<torch::Tensor> relative_attention_bias = std::nullopt, int64_t relative_attention_max_distance = 0, std::optional<int64_t> spec_decoding_target_max_draft_tokens = std::nullopt, - std::optional<torch::Tensor> quant_scale_qkv = std::nullopt, - std::optional<torch::Tensor> dsv4_inv_rope_cos_sin_cache = std::nullopt, bool enable_dsv4_epilogue_fusion = false, - bool const force_prepare_spec_dec_tree_mask = false); + std::optional<torch::Tensor> quant_scale_qkv = std::nullopt); struct KvCachePoolPointers { diff --git a/cpp/tensorrt_llm/thop/compressorOp.cpp b/cpp/tensorrt_llm/thop/compressorOp.cpp index 5fdfb5a85806..42083eb7aab3 100644 --- a/cpp/tensorrt_llm/thop/compressorOp.cpp +++ b/cpp/tensorrt_llm/thop/compressorOp.cpp @@ -38,10 +38,6 @@ void compressorPagedKvCompressOp(torch::Tensor kv_score, // [m, 2*state_dim] bf1 torch::Tensor cu_kv_comp, // [bsz+1] int32 int64_t batch_size, int64_t page_size, int64_t head_dim, int64_t compress_ratio, int64_t next_n) { - constexpr int64_t kMinNextN = 1; - constexpr int64_t kMaxNextN = 8; - TORCH_CHECK(next_n >= kMinNextN && next_n <= kMaxNextN, "next_n must be in [1, 8], got ", next_n); - auto stream = at::cuda::getCurrentCUDAStream(); int kv_score_eb = static_cast<int>(kv_score.element_size()); int state_eb = static_cast<int>(paged_kv.element_size()); diff --git a/cpp/tensorrt_llm/thop/cublasFp4ScaledMM.cpp b/cpp/tensorrt_llm/thop/cublasFp4ScaledMM.cpp index 8e4da99dbadc..a9ad46ad8f04 100644 --- a/cpp/tensorrt_llm/thop/cublasFp4ScaledMM.cpp +++ b/cpp/tensorrt_llm/thop/cublasFp4ScaledMM.cpp @@ -16,6 +16,7 @@ #include "tensorrt_llm/common/cublasMMWrapper.h" #include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/plugins/common/plugin.h" #include "tensorrt_llm/thop/outputTensor.h" #include "tensorrt_llm/thop/thUtils.h" #include "userbuffersTensor.h" diff --git a/cpp/tensorrt_llm/thop/cublasScaledMM.cpp b/cpp/tensorrt_llm/thop/cublasScaledMM.cpp index 62f51f7b06f9..dea6f51363e5 100644 --- a/cpp/tensorrt_llm/thop/cublasScaledMM.cpp +++ b/cpp/tensorrt_llm/thop/cublasScaledMM.cpp @@ -18,6 +18,8 @@ #include "tensorrt_llm/common/cublasMMWrapper.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/kernels/userbuffers/ub_interface.h" +#include "tensorrt_llm/plugins/common/plugin.h" +#include "tensorrt_llm/plugins/gemmPlugin/gemmPlugin.h" #include "tensorrt_llm/runtime/torchUtils.h" #include "tensorrt_llm/thop/outputTensor.h" #include "tensorrt_llm/thop/thUtils.h" diff --git a/cpp/tensorrt_llm/thop/deepseekV4BlockTableOp.cpp b/cpp/tensorrt_llm/thop/deepseekV4BlockTableOp.cpp deleted file mode 100644 index 8eb20a88bc1c..000000000000 --- a/cpp/tensorrt_llm/thop/deepseekV4BlockTableOp.cpp +++ /dev/null @@ -1,201 +0,0 @@ -/* - * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include "tensorrt_llm/kernels/deepseekV4BlockTable.h" - -#include <ATen/cuda/CUDAContext.h> -#include <c10/cuda/CUDAException.h> -#include <c10/cuda/CUDAGuard.h> -#include <limits> -#include <torch/extension.h> - -namespace th = torch; -namespace tk = tensorrt_llm::kernels; - -TRTLLM_NAMESPACE_BEGIN - -namespace torch_ext -{ -namespace -{ - -void checkInt32Tensor(th::Tensor const& tensor, char const* name) -{ - TORCH_CHECK(tensor.scalar_type() == th::kInt32, name, " must be int32"); -} - -void checkCudaContiguousTensor(th::Tensor const& tensor, char const* name, int device) -{ - TORCH_CHECK(tensor.is_cuda(), name, " must be a CUDA tensor"); - TORCH_CHECK(tensor.get_device() == device, name, " must be on the same CUDA device as output"); - TORCH_CHECK(tensor.is_contiguous(), name, " must be contiguous"); -} - -void checkLayerAttnShape(th::Tensor const& tensor, char const* name, int64_t numLayers, int64_t numAttnTypes) -{ - TORCH_CHECK(tensor.dim() == 2, name, " must be 2D [num_layers, num_attn_types]"); - TORCH_CHECK(tensor.size(0) == numLayers && tensor.size(1) == numAttnTypes, name, - " must match pool_ids shape [num_layers, num_attn_types]"); -} - -int32_t checkedInt32Size(int64_t value, char const* name) -{ - TORCH_CHECK(value <= std::numeric_limits<int32_t>::max(), name, " exceeds int32 range"); - return static_cast<int32_t>(value); -} - -void checkCommonInputs(th::Tensor const& blockOffsets, th::Tensor const& copyIdx, th::Tensor const& poolIds, - th::Tensor const& validPool, th::Tensor const& scales, th::Tensor const& layerOffsets, th::Tensor const& output) -{ - TORCH_CHECK(output.is_cuda(), "output must be a CUDA tensor"); - int const device = output.get_device(); - TORCH_CHECK(output.is_contiguous(), "output must be contiguous"); - checkCudaContiguousTensor(blockOffsets, "block_offsets", device); - checkCudaContiguousTensor(copyIdx, "copy_idx", device); - checkCudaContiguousTensor(poolIds, "pool_ids", device); - checkCudaContiguousTensor(validPool, "valid_pool", device); - checkCudaContiguousTensor(scales, "scales", device); - checkCudaContiguousTensor(layerOffsets, "layer_offsets", device); - - checkInt32Tensor(blockOffsets, "block_offsets"); - checkInt32Tensor(copyIdx, "copy_idx"); - TORCH_CHECK(poolIds.scalar_type() == th::kInt64, "pool_ids must be int64"); - TORCH_CHECK(validPool.scalar_type() == th::kBool, "valid_pool must be bool"); - checkInt32Tensor(scales, "scales"); - checkInt32Tensor(layerOffsets, "layer_offsets"); - checkInt32Tensor(output, "output"); - - TORCH_CHECK(blockOffsets.dim() == 4, "block_offsets must be 4D [num_pools, table_capacity, 2, max_blocks]"); - TORCH_CHECK(blockOffsets.size(2) == 2, "block_offsets dim 2 must be 2"); - TORCH_CHECK(copyIdx.dim() == 1, "copy_idx must be 1D"); - TORCH_CHECK(poolIds.dim() == 2, "pool_ids must be 2D [num_layers, num_attn_types]"); - TORCH_CHECK(output.dim() == 4, "output must be 4D [num_layers, num_attn_types, num_tables, max_blocks]"); - - int64_t const numLayers = poolIds.size(0); - int64_t const numAttnTypes = poolIds.size(1); - int64_t const numTables = copyIdx.size(0); - int64_t const maxBlocksPerSeq = blockOffsets.size(3); - - checkLayerAttnShape(validPool, "valid_pool", numLayers, numAttnTypes); - checkLayerAttnShape(scales, "scales", numLayers, numAttnTypes); - checkLayerAttnShape(layerOffsets, "layer_offsets", numLayers, numAttnTypes); - - TORCH_CHECK(output.size(0) == numLayers && output.size(1) == numAttnTypes && output.size(2) == numTables - && output.size(3) == maxBlocksPerSeq, - "output shape must be [pool_ids.size(0), pool_ids.size(1), copy_idx.size(0), block_offsets.size(3)]"); -} - -} // namespace - -void deepseekV4ComputeSlidingBlockTables(th::Tensor const& blockOffsets, th::Tensor const& copyIdx, - th::Tensor const& poolIds, th::Tensor const& validPool, th::Tensor const& scales, th::Tensor const& layerOffsets, - th::Tensor const& output) -{ - checkCommonInputs(blockOffsets, copyIdx, poolIds, validPool, scales, layerOffsets, output); - c10::cuda::CUDAGuard const deviceGuard(output.device()); - - int32_t const numPools = checkedInt32Size(blockOffsets.size(0), "num_pools"); - int32_t const copyIdxCapacity = checkedInt32Size(blockOffsets.size(1), "copy_idx_capacity"); - int32_t const numLayers = checkedInt32Size(poolIds.size(0), "num_layers"); - int32_t const numAttnTypes = checkedInt32Size(poolIds.size(1), "num_attn_types"); - int32_t const numTables = checkedInt32Size(copyIdx.size(0), "num_tables"); - int32_t const maxBlocksPerSeq = checkedInt32Size(blockOffsets.size(3), "max_blocks_per_seq"); - - auto stream = at::cuda::getCurrentCUDAStream(output.get_device()); - tk::invokeDeepseekV4ComputeSlidingBlockTables(blockOffsets.data_ptr<int32_t>(), copyIdx.data_ptr<int32_t>(), - poolIds.data_ptr<int64_t>(), validPool.data_ptr<bool>(), scales.data_ptr<int32_t>(), - layerOffsets.data_ptr<int32_t>(), output.data_ptr<int32_t>(), numPools, copyIdxCapacity, numLayers, - numAttnTypes, numTables, maxBlocksPerSeq, stream); - C10_CUDA_KERNEL_LAUNCH_CHECK(); -} - -void deepseekV4ComputeSlidingBlockTablesWithScratch(th::Tensor const& blockOffsets, th::Tensor const& copyIdx, - th::Tensor const& poolIds, th::Tensor const& validPool, th::Tensor const& scales, th::Tensor const& layerOffsets, - th::Tensor const& scratchPages, th::Tensor const& scratchBegs, th::Tensor const& scratchEnds, - th::Tensor const& scratchSlots, th::Tensor const& numContexts, th::Tensor const& output) -{ - checkCommonInputs(blockOffsets, copyIdx, poolIds, validPool, scales, layerOffsets, output); - int const device = output.get_device(); - checkCudaContiguousTensor(scratchPages, "scratch_pages", device); - checkCudaContiguousTensor(scratchBegs, "scratch_begs", device); - checkCudaContiguousTensor(scratchEnds, "scratch_ends", device); - checkCudaContiguousTensor(scratchSlots, "scratch_slots", device); - checkCudaContiguousTensor(numContexts, "num_contexts", device); - - checkInt32Tensor(scratchPages, "scratch_pages"); - checkInt32Tensor(scratchBegs, "scratch_begs"); - checkInt32Tensor(scratchEnds, "scratch_ends"); - checkInt32Tensor(scratchSlots, "scratch_slots"); - checkInt32Tensor(numContexts, "num_contexts"); - - int64_t const numLayers = poolIds.size(0); - int64_t const numAttnTypes = poolIds.size(1); - checkLayerAttnShape(scratchPages, "scratch_pages", numLayers, numAttnTypes); - TORCH_CHECK(scratchBegs.dim() == 2, "scratch_begs must be 2D [num_pools, scratch_capacity]"); - TORCH_CHECK(scratchEnds.dim() == 2, "scratch_ends must be 2D [num_pools, scratch_capacity]"); - TORCH_CHECK(scratchSlots.dim() == 3, "scratch_slots must be 3D [num_pools, scratch_capacity, max_scratch_slots]"); - TORCH_CHECK(scratchBegs.size(0) == blockOffsets.size(0), "scratch_begs.size(0) must match num_pools"); - TORCH_CHECK(scratchEnds.size(0) == scratchBegs.size(0) && scratchEnds.size(1) == scratchBegs.size(1), - "scratch_ends shape must match scratch_begs"); - TORCH_CHECK(scratchSlots.size(0) == scratchBegs.size(0) && scratchSlots.size(1) == scratchBegs.size(1), - "scratch_slots first two dimensions must match scratch_begs"); - TORCH_CHECK(scratchBegs.size(1) <= output.size(2), "scratch_capacity must not exceed num_tables"); - TORCH_CHECK(numContexts.dim() == 0 && numContexts.numel() == 1, "num_contexts must be a scalar tensor"); - TORCH_CHECK(scratchSlots.size(2) > 0 || scratchBegs.size(1) == 0, - "max_scratch_slots must be positive when scratch_capacity is nonzero"); - - c10::cuda::CUDAGuard const deviceGuard(output.device()); - int32_t const numPools = checkedInt32Size(blockOffsets.size(0), "num_pools"); - int32_t const copyIdxCapacity = checkedInt32Size(blockOffsets.size(1), "copy_idx_capacity"); - int32_t const numLayers32 = checkedInt32Size(numLayers, "num_layers"); - int32_t const numAttnTypes32 = checkedInt32Size(numAttnTypes, "num_attn_types"); - int32_t const numTables = checkedInt32Size(copyIdx.size(0), "num_tables"); - int32_t const maxBlocksPerSeq = checkedInt32Size(blockOffsets.size(3), "max_blocks_per_seq"); - int32_t const scratchCapacity = checkedInt32Size(scratchBegs.size(1), "scratch_capacity"); - int32_t const maxScratchSlots = checkedInt32Size(scratchSlots.size(2), "max_scratch_slots"); - - auto stream = at::cuda::getCurrentCUDAStream(output.get_device()); - tk::invokeDeepseekV4ComputeSlidingBlockTablesWithScratch(blockOffsets.data_ptr<int32_t>(), - copyIdx.data_ptr<int32_t>(), poolIds.data_ptr<int64_t>(), validPool.data_ptr<bool>(), - scales.data_ptr<int32_t>(), layerOffsets.data_ptr<int32_t>(), scratchPages.data_ptr<int32_t>(), - scratchBegs.data_ptr<int32_t>(), scratchEnds.data_ptr<int32_t>(), scratchSlots.data_ptr<int32_t>(), - numContexts.data_ptr<int32_t>(), output.data_ptr<int32_t>(), numPools, copyIdxCapacity, numLayers32, - numAttnTypes32, numTables, maxBlocksPerSeq, scratchCapacity, maxScratchSlots, stream); - C10_CUDA_KERNEL_LAUNCH_CHECK(); -} - -} // namespace torch_ext - -TRTLLM_NAMESPACE_END - -TORCH_LIBRARY_FRAGMENT(trtllm, m) -{ - m.def( - "deepseek_v4_compute_sliding_block_tables(Tensor block_offsets, Tensor copy_idx, Tensor pool_ids, " - "Tensor valid_pool, Tensor scales, Tensor layer_offsets, Tensor(a!) output) -> ()"); - m.def( - "deepseek_v4_compute_sliding_block_tables_with_scratch(Tensor block_offsets, Tensor copy_idx, " - "Tensor pool_ids, Tensor valid_pool, Tensor scales, Tensor layer_offsets, Tensor scratch_pages, " - "Tensor scratch_begs, Tensor scratch_ends, Tensor scratch_slots, Tensor num_contexts, " - "Tensor(a!) output) -> ()"); -} - -TORCH_LIBRARY_IMPL(trtllm, CUDA, m) -{ - m.impl("deepseek_v4_compute_sliding_block_tables", &tensorrt_llm::torch_ext::deepseekV4ComputeSlidingBlockTables); - m.impl("deepseek_v4_compute_sliding_block_tables_with_scratch", - &tensorrt_llm::torch_ext::deepseekV4ComputeSlidingBlockTablesWithScratch); -} diff --git a/cpp/tensorrt_llm/thop/dynamicDecodeOp.cpp b/cpp/tensorrt_llm/thop/dynamicDecodeOp.cpp index 228b2c614ab7..8e9e817bbb51 100644 --- a/cpp/tensorrt_llm/thop/dynamicDecodeOp.cpp +++ b/cpp/tensorrt_llm/thop/dynamicDecodeOp.cpp @@ -16,7 +16,6 @@ #include "tensorrt_llm/thop/dynamicDecodeOp.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/kernels/decodingCommon.h" #include "tensorrt_llm/runtime/bufferManager.h" @@ -55,7 +54,7 @@ FtDynamicDecode<T>::FtDynamicDecode(size_t const maxBatchSize, size_t const maxB auto bufferManager = std::make_shared<tensorrt_llm::runtime::BufferManager>(cudaStreamPtr); mFinishedSum = bufferManager->pinnedPool( - tr::ITensor::makeShape({static_cast<int32_t>(maxBatchSize)}), tensorrt_llm::DataType::kINT32); + tr::ITensor::makeShape({static_cast<int32_t>(maxBatchSize)}), nvinfer1::DataType::kINT32); mDynamicDecodeLayer = std::make_shared<tl::DynamicDecodeLayer<T>>(tle::DecodingMode::Auto(), decodingDomain, bufferManager); mBatchSlots = tr::getDefaultBatchSlots(maxBatchSize); diff --git a/cpp/tensorrt_llm/thop/dynamicTreeOp.cpp b/cpp/tensorrt_llm/thop/dynamicTreeOp.cpp index 0a236ced8c35..0b862108a4e6 100644 --- a/cpp/tensorrt_llm/thop/dynamicTreeOp.cpp +++ b/cpp/tensorrt_llm/thop/dynamicTreeOp.cpp @@ -25,6 +25,13 @@ namespace tk = tensorrt_llm::kernels::speculative_decoding; TRTLLM_NAMESPACE_BEGIN +namespace kernels::speculative_decoding +{ +th::Tensor computeProbsFromLogits(th::Tensor const& logits, th::Tensor const& temperatures, + th::optional<th::Tensor> const& topK, th::optional<th::Tensor> const& topP, bool skipTemperature, + runtime::SizeType32 kMax); +} // namespace kernels::speculative_decoding + namespace torch_ext { @@ -119,6 +126,26 @@ void verify_dynamic_tree_greedy_out_packed_op(th::Tensor& candidates, th::Tensor targetPredict.data_ptr<int32_t>(), treeValid.data_ptr<bool>(), batchSize, numDraftTokens, numSpecStep, stream); } +th::Tensor compute_probs_from_logits_op(th::Tensor logits, th::Tensor temperatures, th::optional<th::Tensor> topK, + th::optional<th::Tensor> topP, bool skipTemperature) +{ + TORCH_CHECK(logits.is_cuda(), "logits must be a CUDA tensor"); + TORCH_CHECK(temperatures.is_cuda(), "temperatures must be a CUDA tensor"); + TORCH_CHECK(logits.dim() == 2, "logits must be a 2D tensor"); + TORCH_CHECK(temperatures.dim() == 1, "temperatures must be a 1D tensor"); + TORCH_CHECK(logits.size(0) == temperatures.size(0), "logits and temperatures size mismatch"); + if (topK.has_value() && topK->defined()) + { + TORCH_CHECK(topK->is_cuda(), "top_k must be a CUDA tensor"); + } + if (topP.has_value() && topP->defined()) + { + TORCH_CHECK(topP->is_cuda(), "top_p must be a CUDA tensor"); + } + + return tk::computeProbsFromLogits(logits, temperatures, topK, topP, skipTemperature, /*kMax=*/0); +} + //! \brief Target-only rejection sampling verify op (no draft probabilities needed). void verify_dynamic_tree_rejection_out_op(th::Tensor& draftTokens, th::Tensor& targetProbs, th::Tensor& retrieveNextToken, th::Tensor& retrieveNextSibling, th::Tensor& treeValid, th::Tensor& acceptIndex, @@ -259,3 +286,16 @@ TORCH_LIBRARY_IMPL(trtllm, CUDA, m) { m.impl("verify_dynamic_tree_rejection_out_op", &tensorrt_llm::torch_ext::verify_dynamic_tree_rejection_out_op); } + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "compute_probs_from_logits_op(" + "Tensor logits, Tensor temperatures, Tensor? top_k=None, Tensor? top_p=None, " + "bool skip_temperature=False) -> Tensor"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("compute_probs_from_logits_op", &tensorrt_llm::torch_ext::compute_probs_from_logits_op); +} diff --git a/cpp/tensorrt_llm/thop/fp8Quantize.cpp b/cpp/tensorrt_llm/thop/fp8Quantize.cpp index 43eea8cff838..35ba5c440e18 100644 --- a/cpp/tensorrt_llm/thop/fp8Quantize.cpp +++ b/cpp/tensorrt_llm/thop/fp8Quantize.cpp @@ -209,38 +209,6 @@ std::tuple<at::Tensor, at::Tensor> fp8_quantize_1x128_packed_ue8m0(at::Tensor co return {valueE4M3.slice(0, 0, m), packedScale}; } - -std::tuple<at::Tensor, at::Tensor> fp8_quantize_1x128_cutedsl_ue8m0(at::Tensor const& self) -{ - CHECK_TH_CUDA(self); - CHECK_CONTIGUOUS(self); - - TORCH_CHECK(self.scalar_type() == at::ScalarType::BFloat16, "Input matrix dtype must be BF16."); - TORCH_CHECK(self.dim() == 2, "input must be a matrix"); - TORCH_CHECK(tensorrt_llm::common::isSM100Family(), - "fp8_quantize_1x128_cutedsl_ue8m0 currently only supports SM100 (Blackwell)."); - - auto const m = self.sizes()[0]; - auto const k = self.sizes()[1]; - TORCH_CHECK(m <= std::numeric_limits<int32_t>::max(), "M must be within int32"); - TORCH_CHECK(k <= std::numeric_limits<int32_t>::max(), "K must be within int32"); - TORCH_CHECK(k % 128 == 0, "K must be divisible by the production FP8 block size 128, but got ", k); - - at::Tensor valueE4M3 - = at::detail::empty_cuda({m, k}, at::ScalarType::Float8_e4m3fn, self.device(), /* stride */ std::nullopt); - auto const paddedM = (m + 127) / 128 * 128; - auto const sfCols = (k / 32 + 3) / 4 * 4; - at::Tensor scaleE8M0 - = at::detail::empty_cuda({paddedM * sfCols}, at::ScalarType::Byte, self.device(), /* stride */ std::nullopt); - - auto stream = at::cuda::getCurrentCUDAStream(self.get_device()); - tensorrt_llm::kernels::fp8_blockscale_gemm::launch_fp8_quantize_1x128_cutedsl_bf16_e4m3( - reinterpret_cast<__nv_fp8_e4m3*>(valueE4M3.data_ptr()), scaleE8M0.data_ptr<uint8_t>(), - reinterpret_cast<__nv_bfloat16 const*>(self.data_ptr()), static_cast<int>(m), static_cast<int>(k), - static_cast<int>(paddedM), stream); - - return {valueE4M3, scaleE8M0}; -} } // namespace torch_ext TRTLLM_NAMESPACE_END @@ -250,7 +218,6 @@ TORCH_LIBRARY_FRAGMENT(trtllm, m) m.def("fp8_quantize_1x128(Tensor input, bool use_ue8m0=False) -> (Tensor, Tensor)"); m.def("fp8_batched_quantize_1x128_permute102(Tensor input) -> (Tensor, Tensor)"); m.def("fp8_quantize_1x128_packed_ue8m0(Tensor input) -> (Tensor, Tensor)"); - m.def("fp8_quantize_1x128_cutedsl_ue8m0(Tensor input) -> (Tensor, Tensor)"); } TORCH_LIBRARY_IMPL(trtllm, CUDA, m) @@ -258,5 +225,4 @@ TORCH_LIBRARY_IMPL(trtllm, CUDA, m) m.impl("fp8_quantize_1x128", &tensorrt_llm::torch_ext::fp8_quantize_1x128); m.impl("fp8_batched_quantize_1x128_permute102", &tensorrt_llm::torch_ext::fp8_batched_quantize_1x128_permute102); m.impl("fp8_quantize_1x128_packed_ue8m0", &tensorrt_llm::torch_ext::fp8_quantize_1x128_packed_ue8m0); - m.impl("fp8_quantize_1x128_cutedsl_ue8m0", &tensorrt_llm::torch_ext::fp8_quantize_1x128_cutedsl_ue8m0); } diff --git a/cpp/tensorrt_llm/thop/groupRmsNormOp.cpp b/cpp/tensorrt_llm/thop/groupRmsNormOp.cpp index c1f3f41c5a1c..c408a8c286fb 100644 --- a/cpp/tensorrt_llm/thop/groupRmsNormOp.cpp +++ b/cpp/tensorrt_llm/thop/groupRmsNormOp.cpp @@ -17,7 +17,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/dataType.h" #include "tensorrt_llm/common/opUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/groupRmsNormKernels/groupRmsNormKernels.h" #include "tensorrt_llm/runtime/torchUtils.h" #include "tensorrt_llm/thop/thUtils.h" @@ -101,9 +100,9 @@ void groupRMSNormBase(torch::TensorList const& inputs, torch::TensorList const& /* Handle dtype conversion */ \ switch (dtype) \ { \ - case torch::ScalarType::Half: params.dtype = tensorrt_llm::DataType::kHALF; break; \ - case torch::ScalarType::BFloat16: params.dtype = tensorrt_llm::DataType::kBF16; break; \ - case torch::ScalarType::Float: params.dtype = tensorrt_llm::DataType::kFLOAT; break; \ + case torch::ScalarType::Half: params.dtype = nvinfer1::DataType::kHALF; break; \ + case torch::ScalarType::BFloat16: params.dtype = nvinfer1::DataType::kBF16; break; \ + case torch::ScalarType::Float: params.dtype = nvinfer1::DataType::kFLOAT; break; \ default: TORCH_CHECK(false, "Unsupported data type"); \ } \ tensorrt_llm::kernels::group_rms_norm::GroupRMSNormBaseKernelLauncher<n>(params); \ @@ -182,9 +181,9 @@ void groupRMSNormLargeBatch(torch::TensorList const& inputs, torch::TensorList c // Handle dtype conversion switch (dtype) { - case torch::ScalarType::Half: params.dtype = tensorrt_llm::DataType::kHALF; break; - case torch::ScalarType::BFloat16: params.dtype = tensorrt_llm::DataType::kBF16; break; - case torch::ScalarType::Float: params.dtype = tensorrt_llm::DataType::kFLOAT; break; + case torch::ScalarType::Half: params.dtype = nvinfer1::DataType::kHALF; break; + case torch::ScalarType::BFloat16: params.dtype = nvinfer1::DataType::kBF16; break; + case torch::ScalarType::Float: params.dtype = nvinfer1::DataType::kFLOAT; break; default: TORCH_CHECK(false, "Unsupported data type"); } @@ -261,9 +260,9 @@ void groupRMSNormHeuristic(torch::TensorList const& inputs, torch::TensorList co /* Handle dtype conversion */ \ switch (dtype) \ { \ - case torch::ScalarType::Half: params.dtype = tensorrt_llm::DataType::kHALF; break; \ - case torch::ScalarType::BFloat16: params.dtype = tensorrt_llm::DataType::kBF16; break; \ - case torch::ScalarType::Float: params.dtype = tensorrt_llm::DataType::kFLOAT; break; \ + case torch::ScalarType::Half: params.dtype = nvinfer1::DataType::kHALF; break; \ + case torch::ScalarType::BFloat16: params.dtype = nvinfer1::DataType::kBF16; break; \ + case torch::ScalarType::Float: params.dtype = nvinfer1::DataType::kFLOAT; break; \ default: TORCH_CHECK(false, "Unsupported data type"); \ } \ \ diff --git a/cpp/tensorrt_llm/thop/loraOp.cpp b/cpp/tensorrt_llm/thop/loraOp.cpp index 6957987be6b8..b35ca2608625 100644 --- a/cpp/tensorrt_llm/thop/loraOp.cpp +++ b/cpp/tensorrt_llm/thop/loraOp.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/cublasMMWrapper.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/opUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/cuda_graph_grouped_gemm.h" #include "tensorrt_llm/kernels/lora/lora.h" #include "tensorrt_llm/kernels/lora/loraGroupGEMMParamFillRowReorderFusion.h" @@ -152,11 +151,11 @@ std::vector<th::Tensor> lora_grouped_gemm(th::Tensor const& input, th::Tensor co { outHiddenSizes[i] = output_hidden_sizes[i]; } - tensorrt_llm::DataType loraRuntimeDataType; + nvinfer1::DataType loraRuntimeDataType; switch (input.scalar_type()) { - case torch::kFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kHALF; break; - case torch::kBFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kBF16; break; + case torch::kFloat16: loraRuntimeDataType = nvinfer1::DataType::kHALF; break; + case torch::kBFloat16: loraRuntimeDataType = nvinfer1::DataType::kBF16; break; default: throw std::invalid_argument("Invalid dtype, only supports float16, bfloat16"); } @@ -222,11 +221,11 @@ void lora_grouped_gemm_cuda_graph(th::Tensor const& lora_in_sizes, // [layer_mod auto* splitk_offsets_gpu = reinterpret_cast<int64_t*>(const_cast<void*>(splitk_offsets.data_ptr())); // Get data type - tensorrt_llm::DataType loraRuntimeDataType; + nvinfer1::DataType loraRuntimeDataType; switch (dtype) { - case torch::kFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kHALF; break; - case torch::kBFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kBF16; break; + case torch::kFloat16: loraRuntimeDataType = nvinfer1::DataType::kHALF; break; + case torch::kBFloat16: loraRuntimeDataType = nvinfer1::DataType::kBF16; break; default: TORCH_CHECK(false, "Invalid dtype, only supports float16, bfloat16, got %s", c10::toString(dtype)); } @@ -302,11 +301,11 @@ void lora_group_gemm_param_fill_row_reorder_fusion(th::Tensor const& in_sizes, / int32_t const module_count = static_cast<int32_t>(in_sizes.size(0)); // Get data type info - tensorrt_llm::DataType loraRuntimeDataType; + nvinfer1::DataType loraRuntimeDataType; switch (dtype) { - case torch::kFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kHALF; break; - case torch::kBFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kBF16; break; + case torch::kFloat16: loraRuntimeDataType = nvinfer1::DataType::kHALF; break; + case torch::kBFloat16: loraRuntimeDataType = nvinfer1::DataType::kBF16; break; default: TORCH_CHECK(false, "Invalid dtype, only supports float16, bfloat16, got %s", c10::toString(dtype)); } diff --git a/cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp b/cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp index e985eb943ee1..fc45afd792bb 100644 --- a/cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp +++ b/cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp @@ -15,7 +15,6 @@ */ #include "tensorrt_llm/common/envUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" #include "tensorrt_llm/thop/moeAlltoAllMeta.h" @@ -43,13 +42,12 @@ inline size_t alignOffset(size_t offset, size_t alignment) return (offset + alignment - 1) & ~(alignment - 1); } -inline bool hasActiveRankMask(torch::optional<torch::Tensor> const& maskTensor) -{ - return maskTensor.has_value() && maskTensor.value().defined(); -} - -// Resolve a provided rank-mask tensor into a fixed-width uint64 array. On failure -// (wrong dtype / device / shape), throw at the Python op boundary rather than launch. +// Resolve an optional rank-mask tensor into a fixed-width uint64 array. +// If the caller did not provide a mask, default to "all ranks active" (all bits set), which +// reproduces the pre-fault-tolerance behavior bit-for-bit. +// +// On failure (wrong dtype / device / shape), throws via TORCH_CHECK so the error surfaces +// at the Python op boundary rather than the kernel launch. inline void resolveActiveRankMask(torch::optional<torch::Tensor> const& maskTensor, int64_t epRank, uint64_t (&out)[tensorrt_llm::kernels::moe_comm::kRankMaskWords]) { @@ -57,7 +55,14 @@ inline void resolveActiveRankMask(torch::optional<torch::Tensor> const& maskTens using tensorrt_llm::kernels::moe_comm::kMaxRanks; TORCH_CHECK( epRank >= 0 && epRank < kMaxRanks, "epRank must be in the range [0, ", kMaxRanks, ") for active_rank_mask"); - TORCH_CHECK(hasActiveRankMask(maskTensor), "active_rank_mask must be defined"); + if (!maskTensor.has_value() || !maskTensor.value().defined()) + { + for (int w = 0; w < kRankMaskWords; ++w) + { + out[w] = ~uint64_t{0}; + } + return; + } torch::Tensor const& t = maskTensor.value(); TORCH_CHECK(t.is_cpu(), "active_rank_mask must be a CPU tensor"); TORCH_CHECK(t.scalar_type() == torch::kUInt64, "active_rank_mask must have dtype uint64"); @@ -217,7 +222,7 @@ std::tuple<std::vector<torch::Tensor>, int64_t, torch::Tensor> moeA2ADispatchOp( torch::Tensor const& tokenSelectedExperts, std::vector<torch::Tensor> const& inputPayloads, torch::Tensor const& workspace, torch::Tensor const& metainfo, int64_t runtimeMaxTokensPerRank, int64_t epRank, int64_t epSize, int64_t topK, int64_t numExperts, torch::optional<torch::Tensor> eplbLocalStats, - bool enableRankMask, torch::optional<torch::Tensor> activeRankMask) + torch::optional<torch::Tensor> activeRankMask) { using tensorrt_llm::kernels::moe_comm::PayloadDescriptor; using tensorrt_llm::kernels::moe_comm::MoeA2ADispatchParams; @@ -398,15 +403,9 @@ std::tuple<std::vector<torch::Tensor>, int64_t, torch::Tensor> moeA2ADispatchOp( params.eplb_local_stats = nullptr; } - params.enable_rank_mask = enableRankMask; - if (params.enable_rank_mask) - { - resolveActiveRankMask(activeRankMask, epRank, params.active_rank_mask); - } - else - { - TORCH_CHECK(!hasActiveRankMask(activeRankMask), "active_rank_mask requires enable_rank_mask=True"); - } + // Resolve the optional active-rank mask. Default (no mask) = all bits set, which + // exactly reproduces the pre-fault-tolerance kernel behavior. + resolveActiveRankMask(activeRankMask, epRank, params.active_rank_mask); params.stream = at::cuda::getCurrentCUDAStream(); @@ -461,7 +460,7 @@ std::tuple<std::vector<torch::Tensor>, int64_t, torch::Tensor> moeA2ADispatchOp( // In both cases, the combine kernel reads from the workspace at 'combinePayloadOffset'. torch::Tensor moeA2ACombineOp(torch::Tensor const& payload, int64_t localNumTokens, torch::Tensor const& workspace, torch::Tensor const& metainfo, int64_t runtimeMaxTokensPerRank, int64_t epRank, int64_t epSize, int64_t topK, - int64_t combinePayloadOffset, bool payloadInWorkspace, bool useLowPrecision, bool enableRankMask, + int64_t combinePayloadOffset, bool payloadInWorkspace, bool useLowPrecision = false, torch::optional<torch::Tensor> activeRankMask = torch::nullopt) { using tensorrt_llm::kernels::moe_comm::MoeA2ACombineParams; @@ -485,20 +484,20 @@ torch::Tensor moeA2ACombineOp(torch::Tensor const& payload, int64_t localNumToke TORCH_CHECK(epRank >= 0 && epRank < epSize, "epRank must be in the range [0, epSize)"); TORCH_CHECK(topK > 0 && topK <= kMaxTopK, "topK must be in the range (0, kMaxTopK]"); - // Map torch dtype to tensorrt_llm::DataType - tensorrt_llm::DataType nvDtype = tensorrt_llm::DataType::kFLOAT; + // Map torch dtype to nvinfer1::DataType + nvinfer1::DataType nvDtype = nvinfer1::DataType::kFLOAT; auto scalarType = payload.scalar_type(); if (scalarType == at::kHalf) { - nvDtype = tensorrt_llm::DataType::kHALF; + nvDtype = nvinfer1::DataType::kHALF; } else if (scalarType == at::kBFloat16) { - nvDtype = tensorrt_llm::DataType::kBF16; + nvDtype = nvinfer1::DataType::kBF16; } else if (scalarType == at::kFloat) { - nvDtype = tensorrt_llm::DataType::kFLOAT; + nvDtype = nvinfer1::DataType::kFLOAT; } else { @@ -571,15 +570,8 @@ torch::Tensor moeA2ACombineOp(torch::Tensor const& payload, int64_t localNumToke params.recv_buffers[target_rank] = target_workspace_ptr + combinePayloadOffset; } - params.enable_rank_mask = enableRankMask; - if (params.enable_rank_mask) - { - resolveActiveRankMask(activeRankMask, epRank, params.active_rank_mask); - } - else - { - TORCH_CHECK(!hasActiveRankMask(activeRankMask), "active_rank_mask requires enable_rank_mask=True"); - } + // Resolve the optional active-rank mask. Default (no mask) = all bits set. + resolveActiveRankMask(activeRankMask, epRank, params.active_rank_mask); params.stream = at::cuda::getCurrentCUDAStream(); @@ -675,13 +667,13 @@ TORCH_LIBRARY_FRAGMENT(trtllm, module) "Tensor(a!->*) workspace, Tensor metainfo, int runtime_max_tokens_per_rank, " "int ep_rank, int ep_size, int top_k, int num_experts, " "Tensor? eplb_local_stats=None, " - "bool enable_rank_mask=False, Tensor? active_rank_mask=None) -> (Tensor(a!)[], int, Tensor(a!))"); + "Tensor? active_rank_mask=None) -> (Tensor(a!)[], int, Tensor(a!))"); module.def( "moe_a2a_combine(Tensor(a) payload, int local_num_tokens," "Tensor(a!) workspace, Tensor metainfo, int runtime_max_tokens_per_rank, " "int ep_rank, int ep_size, int top_k, int combine_payload_offset, " "bool payload_in_workspace, bool use_low_precision=False, " - "bool enable_rank_mask=False, Tensor? active_rank_mask=None) -> Tensor"); + "Tensor? active_rank_mask=None) -> Tensor"); module.def( "moe_a2a_initialize(Tensor(a!) workspace, int ep_rank, int ep_size, int max_num_tokens_per_rank, " "int? eplb_stats_num_experts=None) -> Tensor"); diff --git a/cpp/tensorrt_llm/thop/moeOp.cpp b/cpp/tensorrt_llm/thop/moeOp.cpp index 81c975241a2e..4a938455488b 100644 --- a/cpp/tensorrt_llm/thop/moeOp.cpp +++ b/cpp/tensorrt_llm/thop/moeOp.cpp @@ -30,7 +30,6 @@ #include "tensorrt_llm/common/config.h" #include "tensorrt_llm/common/dataType.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/common/workspace.h" #include "tensorrt_llm/kernels/cuda_graph_grouped_gemm.h" #include "tensorrt_llm/kernels/cutlass_kernels/fp8_blockscale_gemm/fp8_blockscale_gemm.h" @@ -88,8 +87,7 @@ enum class MoeLoraRequestType : int32_t // --------------------------------------------------------------------------- inline void moeLoraGroupedGemmRunImpl(::tensorrt_llm::kernels::cutlass_kernels::MoeLoraGroupedGemmModule const& mod, int64_t num_permuted_tokens, int64_t in_hidden_size, int64_t max_lora_rank, int64_t dtype_bytes, - int64_t splitk_slices, void const* input_base, void* output_base, tensorrt_llm::DataType data_type, - cudaStream_t stream) + int64_t splitk_slices, void const* input_base, void* output_base, nvinfer1::DataType data_type, cudaStream_t stream) { TLLM_CHECK_WITH_INFO(mod.permuted_ranks_dev != nullptr, "Grouped-GEMM LoRA module is missing permuted ranks buffer (forgot to populate grouped_gemm?)."); @@ -1208,17 +1206,17 @@ class FusedMoeRunner : public torch::CustomClassHolder // ===== LoRA helpers ===== - // Map a torch dtype to the TRT-LLM tensorrt_llm::DataType used to size the + // Map a torch dtype to the TRT-LLM nvinfer1::DataType used to size the // grouped-GEMM low-rank scratch. Kept as a const member (not static) so the // FP8 case can read mOutputDtype to pick the fp16/bf16 LoRA compute dtype. - tensorrt_llm::DataType loraTypeFromActDtype(c10::ScalarType dtype) const + nvinfer1::DataType loraTypeFromActDtype(c10::ScalarType dtype) const { switch (dtype) { - case c10::ScalarType::Half: return tensorrt_llm::DataType::kHALF; - case c10::ScalarType::Float: return tensorrt_llm::DataType::kFLOAT; + case c10::ScalarType::Half: return nvinfer1::DataType::kHALF; + case c10::ScalarType::Float: return nvinfer1::DataType::kFLOAT; #ifdef ENABLE_BF16 - case c10::ScalarType::BFloat16: return tensorrt_llm::DataType::kBF16; + case c10::ScalarType::BFloat16: return nvinfer1::DataType::kBF16; #endif #ifdef ENABLE_FP8 case c10::ScalarType::Float8_e4m3fn: diff --git a/cpp/tensorrt_llm/thop/noAuxTcOp.cpp b/cpp/tensorrt_llm/thop/noAuxTcOp.cpp index 4dfb20072734..e445206e1d78 100644 --- a/cpp/tensorrt_llm/thop/noAuxTcOp.cpp +++ b/cpp/tensorrt_llm/thop/noAuxTcOp.cpp @@ -20,6 +20,7 @@ #include "tensorrt_llm/kernels/noAuxTcKernels.h" +// #include <NvInferRuntime.h> // #include <c10/cuda/CUDAStream.h> // #include <cassert> // #include <set> diff --git a/cpp/tensorrt_llm/thop/reducescatterOp.cpp b/cpp/tensorrt_llm/thop/reducescatterOp.cpp index a50ca1862f76..40f89e40ff75 100644 --- a/cpp/tensorrt_llm/thop/reducescatterOp.cpp +++ b/cpp/tensorrt_llm/thop/reducescatterOp.cpp @@ -20,6 +20,7 @@ #include "tensorrt_llm/runtime/utils/mpiUtils.h" #include "tensorrt_llm/runtime/utils/pgUtils.h" +#include <NvInferRuntime.h> #include <c10/cuda/CUDAStream.h> #include <torch/extension.h> #if ENABLE_MULTI_DEVICE diff --git a/cpp/tensorrt_llm/thop/rmsNormFp4Quant.cpp b/cpp/tensorrt_llm/thop/rmsNormFp4Quant.cpp deleted file mode 100644 index 420cb80df461..000000000000 --- a/cpp/tensorrt_llm/thop/rmsNormFp4Quant.cpp +++ /dev/null @@ -1,264 +0,0 @@ -/* - * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -#include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/kernels/quantization.h" -#include "tensorrt_llm/kernels/rmsNormFp4QuantKernels.h" -#include "tensorrt_llm/runtime/torchUtils.h" -#include "tensorrt_llm/thop/thUtils.h" - -#include <ATen/cuda/CUDAContext.h> -#include <ATen/cuda/EmptyTensor.h> - -#include <vector> - -TRTLLM_NAMESPACE_BEGIN - -namespace torch_ext -{ - -// Fused residual-add + RMSNorm + NVFP4 input-quantize in one kernel. -// Replaces the (flashinfer fused_add_rmsnorm + standalone fp4_quantize) pair on -// the no-allreduce / attention-DP path. Each rank operates on its local tokens. -// -// Inputs: -// hidden_states : [..., hidden_size] BF16/FP16 — read-only. The residual sum -// (hidden_states + residual) is returned as a fresh -// residual_out tensor; hidden_states itself is not mutated, so -// the op is functionalizable under torch.compile. -// residual : [..., hidden_size] same dtype, read-only. -// norm_weight : [hidden_size] same dtype, RMSNorm gamma. -// scale_factor : [] float32, = (448 * 6) / amax for static-quant Linear. -// eps : RMSNorm epsilon. -// return_norm_out : when true, also return the BF16 normed value (needed by -// DSA indexer's pre_indexer_proj). -// -// Returns: [quant_out, scale_out, residual_out] or -// [norm_out, quant_out, scale_out, residual_out] when return_norm_out. -std::vector<at::Tensor> fused_add_rmsnorm_fp4_quantize(at::Tensor const& hidden_states, at::Tensor residual, - at::Tensor const& norm_weight, at::Tensor const& scale_factor, double eps, bool return_norm_out) -{ - CHECK_TH_CUDA(hidden_states); - CHECK_CONTIGUOUS(hidden_states); - CHECK_TH_CUDA(residual); - CHECK_CONTIGUOUS(residual); - CHECK_TH_CUDA(norm_weight); - CHECK_CONTIGUOUS(norm_weight); - CHECK_INPUT(scale_factor, torch::kFloat32); - TORCH_CHECK( - hidden_states.scalar_type() == residual.scalar_type(), "hidden_states and residual must have matching dtype"); - TORCH_CHECK(hidden_states.scalar_type() == norm_weight.scalar_type(), - "hidden_states and norm_weight must have matching dtype"); - - auto const& input_shape = hidden_states.sizes(); - auto const rank = input_shape.size(); - TORCH_CHECK(rank >= 2, "hidden_states should be >=2D"); - int64_t m = 1; - for (size_t i = 0; i < rank - 1; i++) - { - m *= input_shape[i]; - } - auto const k = input_shape[rank - 1]; - int64_t const sf_vec_size = 16; - TORCH_CHECK(k % sf_vec_size == 0, "hidden_size must be divisible by 16"); - - std::vector<int64_t> quant_shape(input_shape.begin(), input_shape.end()); - quant_shape[rank - 1] = k / 2; - at::Tensor quant_out = at::detail::empty_cuda(quant_shape, FLOAT4_E2M1X2, hidden_states.device(), std::nullopt); - at::Tensor scale_out = at::detail::empty_cuda({tensorrt_llm::computeSwizzledLayoutSFSize(m, k / sf_vec_size)}, - SF_DTYPE, hidden_states.device(), std::nullopt); - - at::Tensor norm_out; - void* norm_out_ptr = nullptr; - if (return_norm_out) - { - norm_out = at::detail::empty_cuda( - input_shape.vec(), hidden_states.scalar_type(), hidden_states.device(), std::nullopt); - norm_out_ptr = norm_out.mutable_data_ptr(); - } - - // Freshly allocate residual_out (input + residual is written here by the - // kernel) the same way as quant_out/scale_out and the ws op - // (fusedAddRMSNormQuant.cpp): empty_cuda, not empty_like. empty_cuda is a - // plain allocation, whereas empty_like carries the input's layout and can be - // lowered to a separate node in the torch.compile trace. The kernel reads - // hidden_states (intermediate_buffer) read-only and writes the sum into this - // distinct buffer, so hidden_states is never mutated and no output aliases an - // input (the op stays functionalizable) -- with no pre-kernel copy. - at::Tensor residual_out - = at::detail::empty_cuda(input_shape.vec(), hidden_states.scalar_type(), hidden_states.device(), std::nullopt); - - tensorrt_llm::kernels::RmsNormFp4QuantParams params{}; - params.bias_buffer = nullptr; - params.residual_buffer = residual.data_ptr(); - params.weight_buffer = norm_weight.data_ptr(); - params.intermediate_buffer = hidden_states.data_ptr(); - params.scale_factor_ptr = static_cast<float const*>(scale_factor.data_ptr()); - params.quant_out = quant_out.mutable_data_ptr(); - params.scale_out = scale_out.mutable_data_ptr(); - params.norm_out = norm_out_ptr; - params.residual_out_buffer = residual_out.mutable_data_ptr(); - params.hidden_size = static_cast<int>(k); - params.eps = static_cast<float>(eps); - params.elts_total = hidden_states.numel(); - params.sf_layout = tensorrt_llm::QuantizationSFLayout::SWIZZLED; - - auto const stream = at::cuda::getCurrentCUDAStream(hidden_states.get_device()); - auto const dtype = tensorrt_llm::runtime::TorchUtils::dataType(hidden_states.scalar_type()); - - tensorrt_llm::kernels::residualRmsNormFp4Quant(params, dtype, stream); - - // residual_out holds the residual sum (= original hidden + original residual). - if (return_norm_out) - { - return {norm_out, quant_out, scale_out, residual_out}; - } - return {quant_out, scale_out, residual_out}; -} - -// Residual-less variant of fused_add_rmsnorm_fp4_quantize. Replaces the -// (flashinfer rmsnorm + standalone fp4_quantize) pair on intra-layer paths that -// have NO residual add — e.g. DSv3.2/Kimi-K2.5 MLA's q_a_layernorm feeding the -// static-NVFP4 q_b_proj. The kernel reads intermediate_buffer (=hidden_states), -// skips the residual add (residual_buffer == nullptr selects Residual=false in -// the launcher), RMSNorms it, and FP4-quantizes the result. hidden_states is -// NOT modified (no residual write-back happens when Residual=false). -// -// Inputs: -// hidden_states : [..., hidden_size] BF16/FP16 — read-only. -// norm_weight : [hidden_size] same dtype, RMSNorm gamma. -// scale_factor : [] float32, = (448 * 6) / amax for static-quant Linear. -// eps : RMSNorm epsilon. -// return_norm_out : when true, also return the BF16 normed value. -// -// Returns: [quant_out, scale_out] or [norm_out, quant_out, scale_out] when -// return_norm_out. -std::vector<at::Tensor> fused_rmsnorm_fp4_quantize(at::Tensor const& hidden_states, at::Tensor const& norm_weight, - at::Tensor const& scale_factor, double eps, bool return_norm_out) -{ - CHECK_TH_CUDA(hidden_states); - CHECK_TH_CUDA(norm_weight); - CHECK_CONTIGUOUS(norm_weight); - CHECK_INPUT(scale_factor, torch::kFloat32); - TORCH_CHECK(hidden_states.scalar_type() == norm_weight.scalar_type(), - "hidden_states and norm_weight must have matching dtype"); - - auto const& input_shape = hidden_states.sizes(); - auto const rank = input_shape.size(); - TORCH_CHECK(rank >= 2, "hidden_states should be >=2D"); - // hidden_states may be a column-slice of a wider projection (e.g. the leading - // q_lora_rank columns of kv_a_proj_with_mqa): its last dim is unit-stride but - // its row stride may exceed hidden_size. We read it in place via an input row - // stride and skip the otherwise-required contiguous copy. The kernel only - // reads with this stride; all outputs are written packed. - TORCH_CHECK(hidden_states.stride(rank - 1) == 1, "hidden_states last dim must be unit-stride"); - // All leading dims must be densely packed on top of the row pitch so that a - // single per-row element stride describes the flattened [m, k] layout. The - // only permitted non-packing is a row pitch larger than k (a column slice). - for (size_t i = 0; i + 2 < rank; i++) - { - TORCH_CHECK(hidden_states.stride(i) == hidden_states.stride(i + 1) * input_shape[i + 1], - "hidden_states leading dims must be densely packed"); - } - int64_t m = 1; - for (size_t i = 0; i < rank - 1; i++) - { - m *= input_shape[i]; - } - auto const k = input_shape[rank - 1]; - int64_t const sf_vec_size = 16; - TORCH_CHECK(k % sf_vec_size == 0, "hidden_size must be divisible by 16"); - // Element stride between consecutive logical rows. For a contiguous tensor - // this equals k, so input_row_stride==0 (packed) and behavior is identical. - int64_t const row_stride = hidden_states.stride(rank - 2); - int const input_row_stride = (row_stride == k) ? 0 : static_cast<int>(row_stride); - - std::vector<int64_t> quant_shape(input_shape.begin(), input_shape.end()); - quant_shape[rank - 1] = k / 2; - at::Tensor quant_out = at::detail::empty_cuda(quant_shape, FLOAT4_E2M1X2, hidden_states.device(), std::nullopt); - at::Tensor scale_out = at::detail::empty_cuda({tensorrt_llm::computeSwizzledLayoutSFSize(m, k / sf_vec_size)}, - SF_DTYPE, hidden_states.device(), std::nullopt); - - at::Tensor norm_out; - void* norm_out_ptr = nullptr; - if (return_norm_out) - { - // The kernel writes norm_out packed (stride hidden_size), so allocate a - // packed (contiguous) tensor rather than mirroring a possibly-strided - // input layout. - norm_out = at::detail::empty_cuda( - input_shape.vec(), hidden_states.scalar_type(), hidden_states.device(), std::nullopt); - norm_out_ptr = norm_out.mutable_data_ptr(); - } - - // residual_buffer == nullptr selects the Residual=false kernel path: the - // kernel RMSNorms intermediate_buffer (=hidden_states) directly without any - // add or write-back, so hidden_states is left unmodified. - tensorrt_llm::kernels::RmsNormFp4QuantParams params{}; - params.bias_buffer = nullptr; - params.residual_buffer = nullptr; - params.weight_buffer = norm_weight.data_ptr(); - params.intermediate_buffer = hidden_states.data_ptr(); - params.scale_factor_ptr = static_cast<float const*>(scale_factor.data_ptr()); - params.quant_out = quant_out.mutable_data_ptr(); - params.scale_out = scale_out.mutable_data_ptr(); - params.norm_out = norm_out_ptr; - params.hidden_size = static_cast<int>(k); - params.eps = static_cast<float>(eps); - params.elts_total = hidden_states.numel(); - params.sf_layout = tensorrt_llm::QuantizationSFLayout::SWIZZLED; - params.input_row_stride = input_row_stride; - - auto const stream = at::cuda::getCurrentCUDAStream(hidden_states.get_device()); - auto const dtype = tensorrt_llm::runtime::TorchUtils::dataType(hidden_states.scalar_type()); - - tensorrt_llm::kernels::residualRmsNormFp4Quant(params, dtype, stream); - - if (return_norm_out) - { - return {norm_out, quant_out, scale_out}; - } - return {quant_out, scale_out}; -} - -} // namespace torch_ext - -TRTLLM_NAMESPACE_END - -TORCH_LIBRARY_FRAGMENT(trtllm, m) -{ - m.def( - "fused_add_rmsnorm_fp4_quantize(" - "Tensor hidden_states," - "Tensor residual," - "Tensor norm_weight," - "Tensor scale_factor," - "float eps," - "bool return_norm_out) -> Tensor[]"); - m.def( - "fused_rmsnorm_fp4_quantize(" - "Tensor hidden_states," - "Tensor norm_weight," - "Tensor scale_factor," - "float eps," - "bool return_norm_out) -> Tensor[]"); -} - -TORCH_LIBRARY_IMPL(trtllm, CUDA, m) -{ - m.impl("fused_add_rmsnorm_fp4_quantize", &tensorrt_llm::torch_ext::fused_add_rmsnorm_fp4_quantize); - m.impl("fused_rmsnorm_fp4_quantize", &tensorrt_llm::torch_ext::fused_rmsnorm_fp4_quantize); -} diff --git a/cpp/tensorrt_llm/thop/thUtils.cpp b/cpp/tensorrt_llm/thop/thUtils.cpp index c151414127fa..97fe6acaab7b 100644 --- a/cpp/tensorrt_llm/thop/thUtils.cpp +++ b/cpp/tensorrt_llm/thop/thUtils.cpp @@ -15,7 +15,7 @@ */ #include "tensorrt_llm/thop/thUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <array> TRTLLM_NAMESPACE_BEGIN @@ -25,12 +25,12 @@ namespace torch_ext tensorrt_llm::runtime::ITensor::Shape convert_shape(torch::Tensor tensor) { - constexpr auto trtMaxDims = tensorrt_llm::Dims::MAX_DIMS; + constexpr auto trtMaxDims = nvinfer1::Dims::MAX_DIMS; auto const torchTensorNumDims = tensor.dim(); TLLM_CHECK_WITH_INFO(torchTensorNumDims <= trtMaxDims, "TensorRT supports at most %i tensor dimensions. Found a Torch tensor with %li dimensions.", trtMaxDims, torchTensorNumDims); - auto result = tensorrt_llm::Dims{}; + auto result = nvinfer1::Dims{}; result.nbDims = static_cast<int32_t>(torchTensorNumDims); for (int i = 0; i < torchTensorNumDims; i++) { diff --git a/cpp/tensorrt_llm/thop/trtllmGenFusedOps.h b/cpp/tensorrt_llm/thop/trtllmGenFusedOps.h index 9e55e71be045..2b3dead3fb57 100644 --- a/cpp/tensorrt_llm/thop/trtllmGenFusedOps.h +++ b/cpp/tensorrt_llm/thop/trtllmGenFusedOps.h @@ -66,15 +66,14 @@ trtllmGenGenerationPreprocess(torch::Tensor qkv_input, torch::Tensor workspace, std::optional<torch::Tensor> host_kv_cache_pool_pointers, std::optional<torch::Tensor> host_kv_cache_pool_mapping, std::optional<torch::Tensor> kv_scale_orig_quant, std::optional<torch::Tensor> kv_scale_quant_orig, std::optional<torch::Tensor> attention_output_orig_quant, std::optional<torch::Tensor> rotary_inv_freq, - std::optional<torch::Tensor> rotary_cos_sin, std::optional<torch::Tensor> mrope_position_deltas, int64_t layer_idx, - int64_t seq_offset, int64_t num_heads, int64_t num_kv_heads, int64_t head_size, int64_t tokens_per_block, - int64_t kv_cache_quant_mode, int64_t max_attention_window_size, int64_t cyclic_attention_window_size, - int64_t num_tokens, int64_t batch_beam, int64_t input_seq_length, int64_t max_past_kv_length, - int64_t rotary_embedding_dim, double rotary_embedding_base, int64_t rotary_embedding_scale_type, - double rotary_embedding_scale, int64_t rotary_embedding_max_positions, int64_t position_embedding_type, - double bmm1_scale, double bmm2_scale, bool fp8_context_fmha, int64_t predicted_tokens_per_seq, - int64_t attention_chunk_size, int64_t multi_processor_count, int64_t total_num_blocks, int64_t kv_factor, - bool need_build_kv_cache_metadata, bool cross_attention = false); + std::optional<torch::Tensor> rotary_cos_sin, int64_t layer_idx, int64_t seq_offset, int64_t num_heads, + int64_t num_kv_heads, int64_t head_size, int64_t tokens_per_block, int64_t kv_cache_quant_mode, + int64_t max_attention_window_size, int64_t cyclic_attention_window_size, int64_t num_tokens, int64_t batch_beam, + int64_t input_seq_length, int64_t max_past_kv_length, int64_t rotary_embedding_dim, double rotary_embedding_base, + int64_t rotary_embedding_scale_type, double rotary_embedding_scale, int64_t rotary_embedding_max_positions, + int64_t position_embedding_type, double bmm1_scale, double bmm2_scale, bool fp8_context_fmha, + int64_t predicted_tokens_per_seq, int64_t attention_chunk_size, int64_t multi_processor_count, + int64_t total_num_blocks, int64_t kv_factor, bool need_build_kv_cache_metadata, bool cross_attention = false); } // namespace torch_ext diff --git a/cpp/tensorrt_llm/thop/trtllmGenQKVProcessOp.cpp b/cpp/tensorrt_llm/thop/trtllmGenQKVProcessOp.cpp index bc80d8446445..64931f687680 100644 --- a/cpp/tensorrt_llm/thop/trtllmGenQKVProcessOp.cpp +++ b/cpp/tensorrt_llm/thop/trtllmGenQKVProcessOp.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/common/attentionOp.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/quantization.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/gptKernels.h" #include "tensorrt_llm/kernels/kvCacheUtils.h" #include "tensorrt_llm/kernels/unfusedAttentionKernels.h" @@ -402,16 +401,16 @@ trtllmGenContextPreprocess(torch::Tensor qkv_input, torch::Tensor workspace, tor switch (qkvDtype) { - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: tensorrt_llm::kernels::invokeQKVPreprocessing( reinterpret_cast<QKVPreprocessingParams<float, KVBlockArray>&>(qkvParams), stream); break; - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: tensorrt_llm::kernels::invokeQKVPreprocessing( reinterpret_cast<QKVPreprocessingParams<half, KVBlockArray>&>(qkvParams), stream); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: tensorrt_llm::kernels::invokeQKVPreprocessing( reinterpret_cast<QKVPreprocessingParams<__nv_bfloat16, KVBlockArray>&>(qkvParams), stream); break; @@ -550,16 +549,16 @@ void trtllmGenContextPostprocess(torch::Tensor qkv_input, torch::Tensor workspac switch (qkvDtype) { - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: tensorrt_llm::kernels::invokeKvCachePostprocessing( reinterpret_cast<QKVPreprocessingParams<float, KVBlockArray>&>(qkvParams), stream); break; - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: tensorrt_llm::kernels::invokeKvCachePostprocessing( reinterpret_cast<QKVPreprocessingParams<half, KVBlockArray>&>(qkvParams), stream); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: tensorrt_llm::kernels::invokeKvCachePostprocessing( reinterpret_cast<QKVPreprocessingParams<__nv_bfloat16, KVBlockArray>&>(qkvParams), stream); break; @@ -578,17 +577,16 @@ trtllmGenGenerationPreprocess(torch::Tensor qkv_input, torch::Tensor workspace, std::optional<torch::Tensor> host_kv_cache_pool_pointers, std::optional<torch::Tensor> host_kv_cache_pool_mapping, std::optional<torch::Tensor> kv_scale_orig_quant, std::optional<torch::Tensor> kv_scale_quant_orig, std::optional<torch::Tensor> attention_output_orig_quant, std::optional<torch::Tensor> rotary_inv_freq, - std::optional<torch::Tensor> rotary_cos_sin, std::optional<torch::Tensor> mrope_position_deltas, - int64_t const layer_idx, int64_t const seq_offset, int64_t const num_heads, int64_t const num_kv_heads, - int64_t const head_size, int64_t const tokens_per_block, int64_t const kv_cache_quant_mode, - int64_t const max_attention_window_size, int64_t const cyclic_attention_window_size, int64_t const num_tokens, - int64_t const batch_beam, int64_t const input_seq_length, int64_t const max_past_kv_length, - int64_t const rotary_embedding_dim, double const rotary_embedding_base, int64_t const rotary_embedding_scale_type, - double const rotary_embedding_scale, int64_t const rotary_embedding_max_positions, - int64_t const position_embedding_type, double const bmm1_scale, double const bmm2_scale, - bool const fp8_context_fmha, int64_t const predicted_tokens_per_seq, int64_t const attention_chunk_size, - int64_t const multi_processor_count, int64_t const total_num_blocks, int64_t const kv_factor, - bool const need_build_kv_cache_metadata, bool const cross_attention) + std::optional<torch::Tensor> rotary_cos_sin, int64_t const layer_idx, int64_t const seq_offset, + int64_t const num_heads, int64_t const num_kv_heads, int64_t const head_size, int64_t const tokens_per_block, + int64_t const kv_cache_quant_mode, int64_t const max_attention_window_size, + int64_t const cyclic_attention_window_size, int64_t const num_tokens, int64_t const batch_beam, + int64_t const input_seq_length, int64_t const max_past_kv_length, int64_t const rotary_embedding_dim, + double const rotary_embedding_base, int64_t const rotary_embedding_scale_type, double const rotary_embedding_scale, + int64_t const rotary_embedding_max_positions, int64_t const position_embedding_type, double const bmm1_scale, + double const bmm2_scale, bool const fp8_context_fmha, int64_t const predicted_tokens_per_seq, + int64_t const attention_chunk_size, int64_t const multi_processor_count, int64_t const total_num_blocks, + int64_t const kv_factor, bool const need_build_kv_cache_metadata, bool const cross_attention) { TORCH_CHECK(host_kv_cache_pool_pointers.has_value(), "host_kv_cache_pool_pointers is required."); TORCH_CHECK(host_kv_cache_pool_mapping.has_value(), "host_kv_cache_pool_mapping is required."); @@ -703,7 +701,7 @@ trtllmGenGenerationPreprocess(torch::Tensor qkv_input, torch::Tensor workspace, qkvParams.spec_decoding_position_offsets = isMultiTokenGen ? optPtr<int>(spec_decoding_position_offsets) : nullptr; qkvParams.mrope_rotary_cos_sin = nullptr; - qkvParams.mrope_position_deltas = optPtr<int>(mrope_position_deltas); + qkvParams.mrope_position_deltas = nullptr; qkvParams.batch_size = static_cast<int>(batch_beam); qkvParams.max_input_seq_len = static_cast<int>(input_seq_length); qkvParams.max_kv_seq_len = static_cast<int>(max_past_kv_length); @@ -734,16 +732,16 @@ trtllmGenGenerationPreprocess(torch::Tensor qkv_input, torch::Tensor workspace, switch (qkvDtype) { - case tensorrt_llm::DataType::kFLOAT: + case nvinfer1::DataType::kFLOAT: tensorrt_llm::kernels::invokeQKVPreprocessing( reinterpret_cast<QKVPreprocessingParams<float, KVBlockArray>&>(qkvParams), stream); break; - case tensorrt_llm::DataType::kHALF: + case nvinfer1::DataType::kHALF: tensorrt_llm::kernels::invokeQKVPreprocessing( reinterpret_cast<QKVPreprocessingParams<half, KVBlockArray>&>(qkvParams), stream); break; #ifdef ENABLE_BF16 - case tensorrt_llm::DataType::kBF16: + case nvinfer1::DataType::kBF16: tensorrt_llm::kernels::invokeQKVPreprocessing( reinterpret_cast<QKVPreprocessingParams<__nv_bfloat16, KVBlockArray>&>(qkvParams), stream); break; diff --git a/cpp/tests/CMakeLists.txt b/cpp/tests/CMakeLists.txt index 99e16ea8b147..0a06d40ee85e 100644 --- a/cpp/tests/CMakeLists.txt +++ b/cpp/tests/CMakeLists.txt @@ -25,13 +25,8 @@ include_directories( ${PROJECT_SOURCE_DIR}/include ${cutlass_source_dir}/include ${cutlass_source_dir}/tools/util/include - ${PROJECT_SOURCE_DIR}/tests/batch_manager) - -# Tests previously inherited the MPI include dirs transitively through the -# removed TensorRT plugin target's PUBLIC includes. -if(ENABLE_MULTI_DEVICE) - include_directories(${MPI_C_INCLUDE_DIRS}) -endif() + ${PROJECT_SOURCE_DIR}/tests/batch_manager + ${PROJECT_SOURCE_DIR}/tests/utils) set(TOP_LEVEL_DIR "${PROJECT_SOURCE_DIR}/..") @@ -43,12 +38,13 @@ function(add_gtest test_name test_src) ${ARGN}) add_executable(${test_name} ${test_src}) - target_link_libraries(${test_name} PUBLIC gmock_main) + target_link_libraries(${test_name} PUBLIC gmock_main TensorRT::OnnxParser) if(NOT ARGS_NO_GTEST_MAIN) target_link_libraries(${test_name} PUBLIC gtest_main) endif() if(NOT ARGS_NO_TLLM_LINKAGE) - target_link_libraries(${test_name} PUBLIC ${SHARED_TARGET}) + target_link_libraries(${test_name} PUBLIC ${SHARED_TARGET} + nvinfer_plugin_tensorrt_llm) if(WIN32) target_link_libraries(${test_name} PRIVATE context_attention_src) endif() @@ -70,4 +66,6 @@ function(add_gtest test_name test_src) add_dependencies(google-tests ${test_name}) endfunction() +add_subdirectory(utils) add_subdirectory(unit_tests) +add_subdirectory(e2e_tests) diff --git a/cpp/tests/e2e_tests/CMakeLists.txt b/cpp/tests/e2e_tests/CMakeLists.txt new file mode 100644 index 000000000000..f5deb048a180 --- /dev/null +++ b/cpp/tests/e2e_tests/CMakeLists.txt @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +add_subdirectory(batch_manager) +add_subdirectory(executor) diff --git a/cpp/tests/e2e_tests/batch_manager/CMakeLists.txt b/cpp/tests/e2e_tests/batch_manager/CMakeLists.txt new file mode 100644 index 000000000000..875e12eb975f --- /dev/null +++ b/cpp/tests/e2e_tests/batch_manager/CMakeLists.txt @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +# guidedDecoderTest requires model tokenizer info, so it's easier to run it with +# e2e tests instead of unit tests. +add_gtest(guidedDecoderTest guidedDecoderTest.cpp) +add_gtest(trtEncoderModelTest trtEncoderModelTest.cpp) +add_gtest(trtGptModelTest trtGptModelTest.cpp) +add_gtest(trtGptModelRealDecoderTest trtGptModelRealDecoderTest.cpp) +target_link_libraries(trtGptModelRealDecoderTest PRIVATE testingUtils) diff --git a/cpp/tests/e2e_tests/batch_manager/guidedDecoderTest.cpp b/cpp/tests/e2e_tests/batch_manager/guidedDecoderTest.cpp new file mode 100644 index 000000000000..7b262cacb27d --- /dev/null +++ b/cpp/tests/e2e_tests/batch_manager/guidedDecoderTest.cpp @@ -0,0 +1,227 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TOP_LEVEL_DIR +#error "Define TOP_LEVEL_DIR" +#endif + +#include <fstream> +#include <gtest/gtest.h> +#include <nlohmann/json.hpp> + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/batch_manager/decoderBuffers.h" +#include "tensorrt_llm/batch_manager/guidedDecoder.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/executor/executor.h" + +using namespace tensorrt_llm::runtime; +using namespace tensorrt_llm::batch_manager; +namespace texec = tensorrt_llm::executor; + +namespace +{ +auto const TEST_RESOURCE_PATH = std::filesystem::path{TOP_LEVEL_DIR} / "cpp/tests/resources"; +auto const DATA_PATH = TEST_RESOURCE_PATH / "data"; +auto const GPT_XGRAMMAR_TOKENIZER_INFO_PATH = DATA_PATH / "gpt2" / "xgrammar_tokenizer_info.json"; +auto const LLAMA_XGRAMMAR_TOKENIZER_INFO_PATH = DATA_PATH / "Llama-3.2-1B" / "xgrammar_tokenizer_info.json"; +} // namespace + +class GuidedDecoderTest : public ::testing::Test +{ +public: + using TensorPtr = ITensor::SharedPtr; + using VecTokens = std::vector<TokenIdType>; + using RequestIdType = std::uint64_t; + using RequestVector = std::vector<std::shared_ptr<LlmRequest>>; + + void SetUp() override + { + mStream = std::make_shared<CudaStream>(); + mRuntimeBufferManager = std::make_shared<BufferManager>(mStream); + } + + void TearDown() override {} + + void initData(std::filesystem::path tokenizerInfoPath, SizeType32 vocabSizePadded, VecTokens outputIds, + std::vector<int32_t> expectedNumRejected) + { + mLogitsDtype = nvinfer1::DataType::kFLOAT; + mMaxNumRequests = 16; + + mVocabSizePadded = vocabSizePadded; + auto const tokenizerInfo = nlohmann::json::parse(std::ifstream{tokenizerInfoPath}); + auto const encodedVocab = tokenizerInfo["encoded_vocab"].template get<std::vector<std::string>>(); + auto const tokenizerStr = tokenizerInfo["tokenizer_str"].template get<std::string>(); + auto const stopTokenIds = tokenizerInfo["stop_token_ids"].template get<std::vector<TokenIdType>>(); + texec::GuidedDecodingConfig guidedDecodingConfig( + texec::GuidedDecodingConfig::GuidedDecodingBackend::kXGRAMMAR, encodedVocab, tokenizerStr, stopTokenIds); + mGuidedDecoder = std::make_shared<GuidedDecoder>( + guidedDecodingConfig, mMaxNumRequests, mVocabSizePadded, mLogitsDtype, *mRuntimeBufferManager); + + mLogits.resize(mMaxNumRequests); + mLogitsHost.resize(mMaxNumRequests); + for (int i = 0; i < mMaxNumRequests; i++) + { + mLogits[i] = mRuntimeBufferManager->gpu(ITensor::makeShape({mVocabSizePadded}), mLogitsDtype); + mLogitsHost[i] = BufferManager::pinned(ITensor::makeShape({mVocabSizePadded}), mLogitsDtype); + } + + mOutputIds = outputIds; + mExpectedNumRejected = expectedNumRejected; + } + + void resetLogits() + { + for (int i = 0; i < mMaxNumRequests; i++) + { + auto logitsHostData = bufferCast<float>(*mLogitsHost[i]); + for (int j = 0; j < mVocabSizePadded; j++) + { + logitsHostData[j] = 0.0f; + } + mRuntimeBufferManager->copy(*(mLogitsHost[i]), *(mLogits[i])); + } + } + + void syncLogitsToHost() + { + for (int i = 0; i < mMaxNumRequests; i++) + { + mRuntimeBufferManager->copy(*(mLogits[i]), *(mLogitsHost[i])); + } + } + + int32_t countRejected(int i) + { + int32_t numRejected = 0; + for (int j = 0; j < mVocabSizePadded; j++) + { + auto logitsHostData = bufferCast<float>(*mLogitsHost[i]); + if (logitsHostData[j] < -1e6) + { + numRejected++; + } + } + return numRejected; + } + + void runTest() + { + auto llmReq1 = std::make_shared<LlmRequest>(1, 100, std::make_shared<VecTokens>(10), SamplingConfig(), false); + texec::GuidedDecodingParams guidedDecodingParams(texec::GuidedDecodingParams::GuideType::kJSON); + llmReq1->setGuidedDecodingParams(guidedDecodingParams); + llmReq1->mSeqSlot = 1; + + auto llmReq2 = std::make_shared<LlmRequest>(1, 100, std::make_shared<VecTokens>(10), SamplingConfig(), false); + llmReq2->mSeqSlot = 2; + + RequestVector contextRequests{llmReq1, llmReq2}; + RequestVector generationRequests{}; + ScheduledRequests scheduledRequests{contextRequests, generationRequests}; + DecoderInputBuffers decoderInputBuffers(mMaxNumRequests, 1, *mRuntimeBufferManager); + + for (auto const& requests : {scheduledRequests.contextRequests, scheduledRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + decoderInputBuffers.decoderRequests.push_back(llmReq); + } + } + decoderInputBuffers.decoderLogits = mLogits; + + // Context phase + resetLogits(); + mGuidedDecoder->build(scheduledRequests); + mGuidedDecoder->execute(decoderInputBuffers, *mRuntimeBufferManager); + syncLogitsToHost(); + mRuntimeBufferManager->getStream().synchronize(); + + // Move request to generation phase + contextRequests.pop_back(); + contextRequests.pop_back(); + llmReq1->setState(LlmRequestState::kGENERATION_IN_PROGRESS); + generationRequests.push_back(llmReq1); + llmReq2->setState(LlmRequestState::kGENERATION_IN_PROGRESS); + generationRequests.push_back(llmReq2); + + decoderInputBuffers.decoderRequests.clear(); + for (auto const& requests : {scheduledRequests.contextRequests, scheduledRequests.generationRequests}) + { + for (auto const& llmReq : requests) + { + decoderInputBuffers.decoderRequests.push_back(llmReq); + } + } + + EXPECT_EQ(countRejected(0), mExpectedNumRejected[0]); + EXPECT_EQ(countRejected(1), 0); + + // Generation phase + for (int i = 0; i < mOutputIds.size(); i++) + { + llmReq1->addNewToken(mOutputIds[i], 0); + llmReq2->addNewToken(mOutputIds[i], 0); + + resetLogits(); + mGuidedDecoder->build(scheduledRequests); + mGuidedDecoder->execute(decoderInputBuffers, *mRuntimeBufferManager); + syncLogitsToHost(); + mRuntimeBufferManager->getStream().synchronize(); + + EXPECT_EQ(countRejected(0), mExpectedNumRejected[i + 1]); + EXPECT_EQ(countRejected(1), 0); + } + } + +private: + SizeType32 mMaxNumRequests; + SizeType32 mVocabSizePadded; + nvinfer1::DataType mLogitsDtype; + + std::vector<TensorPtr> mLogits; // [mBatchSize, mVocabSizePadded] + std::vector<TensorPtr> mLogitsHost; // [mBatchSize, mVocabSizePadded] + + std::shared_ptr<BufferManager> mRuntimeBufferManager; + std::shared_ptr<CudaStream> mStream; + std::shared_ptr<GuidedDecoder> mGuidedDecoder; + + VecTokens mOutputIds; + std::vector<int32_t> mExpectedNumRejected; +}; + +TEST_F(GuidedDecoderTest, GptTokenizer) +{ + VecTokens outputIds{4895, 824, 312, 1298, 366, 27743, 7934, 49793, 1600, 366, 12961, 19703, 4668, 1298, 366, 54, + 4537, 17, 12, 17469, 7919, 1600, 366, 3903, 10394, 1298, 366, 1485, 405, 41022, 20662}; + std::vector<int32_t> expectedNumRejected{50251, 219, 219, 219, 48558, 219, 219, 219, 219, 50191, 219, 219, 219, 219, + 48558, 219, 219, 219, 219, 219, 219, 219, 50191, 219, 219, 219, 48558, 219, 219, 219, 219, 50256}; + initData(GPT_XGRAMMAR_TOKENIZER_INFO_PATH, 50257, outputIds, expectedNumRejected); + runTest(); +} + +TEST_F(GuidedDecoderTest, LlamaTokenizer) +{ + VecTokens outputIds{6377, 893, 333, 1115, 376, 27247, 6779, 7898, 545, 613, 376, 8926, 17830, 1115, 376, 29956, + 7228, 29906, 29899, 10399, 7734, 613, 376, 4980, 2103, 1115, 376, 29896, 29941, 29900, 29900, 341, 29890, 567, + 9092}; + std::vector<int32_t> expectedNumRejected{128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, + 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, + 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235, 128235}; + initData(LLAMA_XGRAMMAR_TOKENIZER_INFO_PATH, 128256, outputIds, expectedNumRejected); + runTest(); +} diff --git a/cpp/tests/e2e_tests/batch_manager/trtEncoderModelTest.cpp b/cpp/tests/e2e_tests/batch_manager/trtEncoderModelTest.cpp new file mode 100644 index 000000000000..dce09539bd45 --- /dev/null +++ b/cpp/tests/e2e_tests/batch_manager/trtEncoderModelTest.cpp @@ -0,0 +1,219 @@ + +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TOP_LEVEL_DIR +#error "Define TOP_LEVEL_DIR" +#endif + +#include "tensorrt_llm/batch_manager/trtEncoderModel.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/gptJsonConfig.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/utils/numpyUtils.h" +#include "tensorrt_llm/runtime/utils/runtimeUtils.h" + +#include <gmock/gmock.h> +#include <gtest/gtest.h> + +#include <filesystem> +#include <vector> + +using namespace tensorrt_llm::runtime; +namespace fs = std::filesystem; + +using TensorPtr = ITensor::SharedPtr; + +namespace +{ +auto const TEST_RESOURCE_PATH = fs::path{TOP_LEVEL_DIR} / "cpp/tests/resources"; +auto const ENC_DEC_BASE = TEST_RESOURCE_PATH / "models/enc_dec/trt_engines"; +auto const ENC_DEC_ENGINE_BASE = TEST_RESOURCE_PATH / "models/enc_dec/trt_engines"; +auto const BART_TP1_PP1_ENCODER_RMPAD_DIR = "bart-large-cnn/1-gpu/float16/tp1/encoder"; +auto const BART_TP2_PP1_ENCODER_RMPAD_DIR = "bart-large-cnn/2-gpu/float16/tp2/encoder"; +auto const BART_TP2_PP2_ENCODER_RMPAD_DIR = "bart-large-cnn/4-gpu/float16/tp2/encoder"; +auto const T5_TP1_PP1_ENCODER_RMPAD_DIR = "t5-small/1-gpu/float16/tp1/encoder"; +auto const ENC_DEC_DATA_BASE = TEST_RESOURCE_PATH / "data/enc_dec"; +} // namespace + +namespace tensorrt_llm::batch_manager +{ + +class EncoderModelTestSingleGPU : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type-member-init) +{ +protected: + EncoderModelTestSingleGPU(std::filesystem::path const& modelPath) + : mModelConfig(1, 2, 1, 1, 1, 1, nvinfer1::DataType::kFLOAT) + , mModelPath(modelPath) + { + } + + EncoderModelTestSingleGPU() + : EncoderModelTestSingleGPU(ENC_DEC_ENGINE_BASE / T5_TP1_PP1_ENCODER_RMPAD_DIR) + { + } + + void SetUp() override + { + std::filesystem::path trtEnginePath = mModelPath; + + mBeamWidth = 1; + + mLogger = std::make_shared<TllmLogger>(); + + initTrtLlmPlugins(mLogger.get()); + + auto const json = GptJsonConfig::parse(trtEnginePath / "config.json"); + mModelConfig = json.getModelConfig(); + mWorldConfig = WorldConfig::mpi(json.getGpusPerNode(), json.getTensorParallelism(), + json.getPipelineParallelism(), json.getContextParallelism()); + mVocabSizePadded = mModelConfig.getVocabSizePadded(mWorldConfig.getSize()); + + auto const enginePath = trtEnginePath / json.engineFilename(mWorldConfig); + auto const dtype = mModelConfig.getDataType(); + + ASSERT_TRUE(fs::exists(enginePath)); + mEngineBuffer = utils::loadEngine(enginePath.string()); + + mStream = std::make_unique<CudaStream>(); + mManager = std::make_unique<BufferManager>(mStream); + } + + void TearDown() override {} + + int32_t mMaxNumRequests; + int32_t mMaxSeqLen; + int32_t mBeamWidth; + int32_t mVocabSizePadded; + // SamplingConfig mSamplingConfig; + std::string mDataPath; + std::shared_ptr<nvinfer1::ILogger> mLogger; + ModelConfig mModelConfig; + WorldConfig mWorldConfig; + std::vector<std::uint8_t> mEngineBuffer; + std::unique_ptr<BufferManager> mManager; + BufferManager::CudaStreamPtr mStream; + std::filesystem::path mModelPath; +}; + +// test for TP2PP2 +class TrtEncoderModelTestMultiGPU : public EncoderModelTestSingleGPU +{ +protected: + TrtEncoderModelTestMultiGPU() + : EncoderModelTestSingleGPU(ENC_DEC_ENGINE_BASE / BART_TP2_PP2_ENCODER_RMPAD_DIR) + { + } +}; + +namespace +{ + +void runEncoderTest(std::unique_ptr<BufferManager>& bufferManager, ModelConfig const& modelConfig, + WorldConfig const& worldConfig, std::vector<std::uint8_t> const& engineBuffer, + std::shared_ptr<nvinfer1::ILogger>& logger) +{ + using VecTokens = LlmRequest::VecTokens; + using TokenIdType = LlmRequest::TokenIdType; + + auto inputsIdsHost + = utils::loadNpy(*bufferManager, (ENC_DEC_DATA_BASE / "input_ids.npy").string(), MemoryType::kCPU); + auto inputsIdsPtr = bufferCast<SizeType32>(*inputsIdsHost); + auto inputLengthsHost + = utils::loadNpy(*bufferManager, (ENC_DEC_DATA_BASE / "input_lengths.npy").string(), MemoryType::kCPU); + auto inputLengthsPtr = bufferCast<SizeType32>(*inputLengthsHost); + auto encoderOutput + = utils::loadNpy(*bufferManager, (ENC_DEC_DATA_BASE / "encoder_output.npy").string(), MemoryType::kCPU); + auto encoderOutputPrt = bufferCast<half>(*encoderOutput); + + SizeType32 const nbRequests = inputLengthsHost->getShape().d[0]; + SizeType32 const stride = inputsIdsHost->getShape().d[1]; + SizeType32 const hiddenSize = encoderOutput->getShape().d[1]; + ASSERT_EQ(nbRequests, inputsIdsHost->getShape().d[0]); + + // std::vector<std::shared_ptr<VecTokens>> inputIds(nbRequests); + RequestVector requestList; + for (SizeType32 i = 0; i < nbRequests; i++) + { + SizeType32 length = inputLengthsPtr[i]; + auto currentInputId = std::make_shared<VecTokens>(0); + currentInputId->insert(currentInputId->end(), inputsIdsPtr, inputsIdsPtr + length); + executor::Request req(*currentInputId, 1); + req.setEncoderInputTokenIds(*currentInputId); + auto request = std::make_shared<LlmRequest>(i, req); + inputsIdsPtr += stride; + requestList.push_back(request); + } + + tensorrt_llm::executor::ExecutorConfig executorConfig{}; + auto trtEncoderModel = std::make_shared<TrtEncoderModel>( + modelConfig, worldConfig, runtime::RawEngine(engineBuffer.data(), engineBuffer.size()), logger, executorConfig); + + trtEncoderModel->forward(requestList); + + if (worldConfig.isLastPipelineParallelRank() && worldConfig.getTensorParallelRank() == 0) + { + auto arrayEqual = [](auto it0, auto it1, SizeType32 length) + { + SizeType32 nbNotEqual = 0; + for (SizeType32 i = 0; i < length; i++) + { + auto v0 = static_cast<float>(*it0); + auto v1 = static_cast<float>(*it1); + if (std::abs(v0 - v1) > 1e-3) + { + nbNotEqual++; + } + it0++; + it1++; + } + return static_cast<double>(nbNotEqual) / length; + }; + ASSERT_EQ(requestList.size(), inputLengthsHost->getShape().d[0]); + { + auto curLengthPtr = inputLengthsPtr; + auto curOutPtr = encoderOutputPrt; + for (auto const& req : requestList) + { + ASSERT_TRUE(req->getEncoderOutputHost()) << "Encoder output is empty!"; + EXPECT_EQ(req->getState(), LlmRequestState::kCONTEXT_INIT); + auto actualOut = bufferCast<half>(*(req->getEncoderOutputHost())); + auto unequalFraction = arrayEqual(curOutPtr, actualOut, *curLengthPtr); + EXPECT_TRUE(unequalFraction == 0) + << "Req " << req->mRequestId << ": " << unequalFraction << " of outputs are different"; + curOutPtr += *curLengthPtr * hiddenSize; + curLengthPtr++; + } + } + } +} + +} // Anonymous namespace + +TEST_F(EncoderModelTestSingleGPU, Forward) +{ + runEncoderTest(mManager, mModelConfig, mWorldConfig, mEngineBuffer, mLogger); +} + +TEST_F(TrtEncoderModelTestMultiGPU, Forward) +{ + + runEncoderTest(mManager, mModelConfig, mWorldConfig, mEngineBuffer, mLogger); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tests/e2e_tests/batch_manager/trtGptModelRealDecoderTest.cpp b/cpp/tests/e2e_tests/batch_manager/trtGptModelRealDecoderTest.cpp new file mode 100644 index 000000000000..f401a31305d3 --- /dev/null +++ b/cpp/tests/e2e_tests/batch_manager/trtGptModelRealDecoderTest.cpp @@ -0,0 +1,1741 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/trtGptModel.h" +#include "tensorrt_llm/batch_manager/trtGptModelFactory.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tensorrt_llm/runtime/utils/numpyUtils.h" +#include "tensorrt_llm/testing/modelSpec.h" +#include "tests/utils/common.h" + +#include <gmock/gmock.h> +#include <gtest/gtest.h> + +#include <cstdint> +#include <cstdlib> +#include <memory> +#include <optional> +#include <vector> + +using namespace tensorrt_llm::testing; +using namespace tensorrt_llm::runtime; +using namespace tensorrt_llm::runtime::utils; +using namespace tensorrt_llm::batch_manager; +namespace fs = std::filesystem; +namespace tc = tensorrt_llm::common; +namespace texec = tensorrt_llm::executor; +using tensorrt_llm::testing::ModelSpec; +using tensorrt_llm::testing::KVCacheType; +using tensorrt_llm::testing::QuantMethod; + +namespace +{ +using TensorPtr = tensorrt_llm::runtime::ITensor::SharedPtr; + +auto constexpr GPT_MODEL_DIR = "gpt2"; +auto constexpr GPTJ_MODEL_DIR = "gpt-j-6b"; +auto constexpr LLAMA_MODEL_DIR = "Llama-3.2-1B"; +auto constexpr MEDUSA_MODEL_DIR = "vicuna-7b-medusa"; +auto constexpr EAGLE_MODEL_DIR = "vicuna-7b-eagle"; +auto constexpr MAMBA_MODEL_DIR = "mamba-2.8b-hf"; +auto constexpr RECURRENTGEMMA_MODEL_DIR = "recurrentgemma-2b"; +auto constexpr EXPLICIT_DRAFT_MODEL_DIR = "vicuna-7b-redrafter"; +auto constexpr CHATGLM_MODEL_DIR = "chatglm-6b"; +auto constexpr GLM_MODEL_DIR = "glm-10b"; + +auto constexpr FP8_GPT_ATTENTION_PLUGIN_IFB_PACKED_PATH = "fp8-plugin"; + +auto constexpr INPUT_FILE = "input_tokens.npy"; +auto constexpr INPUT_LLAMA_FILE = "input_tokens_llama.npy"; +auto constexpr INPUT_VICUNA_FILE = "input_vicuna.npy"; +auto constexpr LONG_INPUT_FILE = "input_tokens_long.npy"; +auto constexpr CHATGLM_INPUT_FILE = "input_tokens_chatglm-6b.npy"; +auto constexpr GLM_INPUT_FILE = "input_tokens_glm-10b.npy"; + +auto constexpr LLAMA_END_ID = 128001; +auto constexpr LLAMA_PAD_ID = 128001; + +struct ModelParams +{ + char const* baseDir; + ModelIds ids; + + friend std::ostream& operator<<(std::ostream& os, ModelParams const& modelParams) + { + return os << "baseDir: " << modelParams.baseDir << ", ids: (" << modelParams.ids.padId << "," + << modelParams.ids.endId << ")"; + } +}; + +} // namespace + +class TrtModelRealDecoderTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type-member-init) +{ +protected: + TrtModelRealDecoderTest() {} + + void SetUp() override + { + mDeviceCount = tc::getDeviceCount(); + if (mDeviceCount == 0) + { + GTEST_SKIP() << "No GPUs found"; + } + + mLogger = std::make_shared<TllmLogger>(); + + initTrtLlmPlugins(mLogger.get()); + } + + void TearDown() override {} + + int mDeviceCount{}; + std::shared_ptr<nvinfer1::ILogger> mLogger{}; +}; + +enum class TrtGptModelIfbTestType +{ + BULK, + WAVEFRONT, + RANDOM +}; + +namespace +{ + +void verifyOutput(RequestList const& finishedRequestList, + std::unordered_map<SizeType32, TestData> const& beamWidthTestData, std::vector<SizeType32> const& givenInputLengths, + SizeType32 nbGivenInputs, ModelSpec const& modelSpec) +{ + auto const checkRawLogits = modelSpec.mOtherModelSpecToCompare ? false : modelSpec.mGatherLogits; + auto const smokeTest = modelSpec.mSmokeTest; + auto const returnLogProbs = modelSpec.mReturnLogProbs; + auto const checkAcceptedTokenLogits = modelSpec.mAcceptDraftByLogits; + + if (smokeTest) + { + return; + } + + for (auto const& llmReqPtr : finishedRequestList) + { + auto const& llmReq = *llmReqPtr; + auto const requestId = llmReq.mRequestId; + auto const [givenInputIdx, givenInputLength] + = getRequestGivenInputIdxLength(requestId, nbGivenInputs, givenInputLengths); + auto const reqBeamWidth = llmReq.mSamplingConfig.beamWidth; + auto const& testData = beamWidthTestData.at(reqBeamWidth); + auto const* const expectedOutputData = bufferCast<TokenIdType const>(*testData.expectedOutputIds); + auto const expectedOutputLengths = testData.expectedOutputLengths; + auto const acceptedDraftTokensLengths = testData.acceptedDraftTokensLengths; + auto const endId = testData.endIds[givenInputIdx]; + auto const maxSeqLen = testData.maxSeqLen; + auto const draftLogits = testData.draftLogits; + auto const expectedGenerationLogits = testData.expectedGenerationLogits; + auto const expectedContextLogits = testData.expectedContextLogits; + auto const expectedCumLogProbs = testData.expectedCumLogProbs; + auto const expectedLogProbs = testData.expectedLogProbs; + auto const draftTokens = llmReq.getDraftTokens(); + auto const isDraftTokensExternal = modelSpec.mSpecDecodingMode.isDraftTokensExternal(); + auto const inputLength = givenInputLength + static_cast<SizeType32>(isDraftTokensExternal); + + for (auto beam = 0; beam < reqBeamWidth; ++beam) + { + auto const expectedOutputLength = expectedOutputLengths[givenInputIdx * reqBeamWidth + beam]; + auto const predictedTokens = llmReq.getTokens(beam); + + auto numPredTokens = static_cast<SizeType32>(predictedTokens.size() - inputLength); + if (isDraftTokensExternal && !draftTokens->empty()) + { + numPredTokens + = std::min(numPredTokens, acceptedDraftTokensLengths[givenInputIdx * reqBeamWidth + beam] + 1); + } + if (modelSpec.mSpecDecodingMode.isMedusa() || modelSpec.mSpecDecodingMode.isLookaheadDecoding() + || modelSpec.mSpecDecodingMode.isExplicitDraftTokens() || modelSpec.mSpecDecodingMode.isEagle()) + { + // WAR to ensure bulk execution of spec decoding. + // We hope that no request in batch can finish 2x faster than any other request. + // For the cases when BS < 8, some predicted tokens are mismatched to reference data. + numPredTokens /= 2; + } + + if (modelSpec.mKVCacheType == KVCacheType::kDISABLED) + { + EXPECT_EQ(numPredTokens, 1) << "b: " << requestId << " beam: " << beam; + } + else + { + EXPECT_EQ(predictedTokens.size(), expectedOutputLength) << "b: " << requestId << " beam: " << beam; + } + + bool anyMismatch = false; + for (auto i = 0; i < numPredTokens; ++i) + { + // Use the expected data for that beamWidth + auto const expectIndex = tc::flat_index3(givenInputIdx, beam, inputLength + i, reqBeamWidth, maxSeqLen); + + auto const expectedToken = expectedOutputData[expectIndex]; + if (expectedToken == endId) + { + break; + } + auto const predictIndex = inputLength + i; + auto const predictedToken = predictedTokens.at(predictIndex); + EXPECT_EQ(predictedToken, expectedToken) << "b: " << requestId << " beam: " << beam << " i: " << i; + anyMismatch |= (predictedToken != expectedToken); + } + EXPECT_FALSE(anyMismatch) << "b: " << requestId << " beam: " << beam; + + if (returnLogProbs) + { + auto cumLogProbs = llmReq.getCumLogProbs(); + auto* const reqExpectedCumLogProbs = bufferCast<float>(*expectedCumLogProbs[requestId]); + EXPECT_TRUE(almostEqual(reqExpectedCumLogProbs[beam], cumLogProbs[beam])); + + auto logProbs = llmReq.getLogProbs(beam); + auto expectedLogProbsBeam = std::shared_ptr(ITensor::slice(expectedLogProbs[requestId], beam, 1)); + expectedLogProbsBeam->squeeze(0); + auto* const reqExpectedLogProbs = bufferCast<float>(*expectedLogProbsBeam); + + for (auto i = 0; i < numPredTokens; ++i) + { + EXPECT_TRUE(almostEqual(reqExpectedLogProbs[inputLength + i], logProbs[i], 5e-2, 5e-2)) + << "expectedLogProbs : " << reqExpectedLogProbs[inputLength + i] + << " logProbs : " << logProbs[i]; + } + } + + if (checkAcceptedTokenLogits && llmReq.hasDraftTokens()) + { + TLLM_CHECK_WITH_INFO(reqBeamWidth == 1, "speculative decoding only works for beam width == 1"); + + TensorPtr const& acceptedTokensLogits = llmReq.getGenerationLogitsHost(); + auto const acceptedTokensLogitsShape = acceptedTokensLogits->getShape(); + + EXPECT_EQ(acceptedTokensLogitsShape.nbDims, 3); + EXPECT_EQ(1, acceptedTokensLogitsShape.d[0]); + EXPECT_EQ(numPredTokens, acceptedTokensLogitsShape.d[1]); + + TensorPtr const& expectedLogits = ITensor::slice(expectedGenerationLogits[requestId], 1, numPredTokens); + + // For hyperparameters + // Greater tolerance for the accepted logits of the target model. + float atol = 0.f; + float rtol = 0.01f; + EXPECT_TRUE(compareLogits(*expectedLogits, *acceptedTokensLogits, atol, rtol)); + } + + if (checkRawLogits) + { + // Check generation logits + TensorPtr const& expectedGenerationLogitsSliced + = ITensor::slice(expectedGenerationLogits[requestId], 0, numPredTokens); + + TensorPtr const& llmReqGeneration = llmReq.getGenerationLogitsHost(); + auto llmReqGenerationShape = llmReqGeneration->getShape(); + + TensorPtr generationLogitsBeam = nullptr; + if (llmReq.isStreaming()) + { + // Expect generation logits shape: [outputLength, beamWidth, vocabSizePad] + EXPECT_EQ(reqBeamWidth, llmReqGenerationShape.d[1]); + EXPECT_EQ(reqBeamWidth, 1); // Streaming mode does not support beam > 1 + llmReqGeneration->squeeze(1); // [outputLength, vocabSizePad] + generationLogitsBeam = llmReqGeneration; + } + else + { + // Expect generation logits shape: [beamWidth, outputLength, vocabSizePad] + EXPECT_EQ(reqBeamWidth, llmReqGenerationShape.d[0]); + generationLogitsBeam + = std::shared_ptr(ITensor::slice(llmReqGeneration, beam, 1)); // [1, outputLength, vocabSizePad] + generationLogitsBeam->squeeze(0); // [outputLength, vocabSizePad] + } + TensorPtr const& generationLogitsSliced = ITensor::slice(generationLogitsBeam, 0, numPredTokens); + EXPECT_TRUE(compareLogits(*expectedGenerationLogitsSliced, *generationLogitsSliced)); + } + } + + if (checkRawLogits) + { + // Check context logits + TensorPtr const& llmReqContext = llmReq.getContextLogitsHost(); + auto llmReqContextShape = llmReqContext->getShape(); + EXPECT_EQ(llmReqContextShape.nbDims, 2); + EXPECT_EQ(llmReq.mPromptLen, llmReqContextShape.d[0]); + EXPECT_TRUE(compareLogits(*expectedContextLogits[requestId], *llmReqContext)); + } + } +} + +// Pick a different endId at random from one of the expected tokens +std::vector<TokenIdType> pickRandomEndIds(TestData const& testData, std::vector<SizeType32> const& givenInputLengths, + SizeType32 const maxNewTokens, bool replaceLogits) +{ + auto const nbGivenInputs = testData.nbGivenInputs; + auto const beamWidth = testData.beamWidth; + auto* const expectedOutputData = bufferCast<TokenIdType>(*testData.expectedOutputIds); + + std::vector<TokenIdType> endIds; + + // For IFB, pick one of the output tokens as endId + for (SizeType32 bi = 0; bi < nbGivenInputs; ++bi) + { + TokenIdType skippedEndId0 = 0; + TokenIdType skippedEndId1 = 0; + SizeType32 endIdIndex = 0; + TokenIdType endId = 0; + auto const endIdRow = bi; + auto const inputLength = givenInputLengths.at(endIdRow); + do + { + auto const endIdBeam = std::rand() % beamWidth; + auto const firstOutputIndex + = tc::flat_index3(endIdRow, endIdBeam, inputLength, beamWidth, testData.maxSeqLen); + // We do not use the 1st token for EndId because of Speculative Decoding test design + // We skip 1st token because minLength is 1 + auto const endIdCol = 2 + (std::rand() % std::max(maxNewTokens - 2, 1)); + endIdIndex = firstOutputIndex + endIdCol; + skippedEndId0 = expectedOutputData[firstOutputIndex]; + skippedEndId1 = expectedOutputData[firstOutputIndex + 1]; + endId = expectedOutputData[endIdIndex]; + } while (endId == skippedEndId0 || endId == skippedEndId1); + // Workaround: The first example has endIdIndex 14, where the generation logits are almost same at + // token ids 257 and 373, which causes unstable generation results. Hence, we use the one previous + // token as endId. + if (bi == 0 && !replaceLogits) + { + endId = expectedOutputData[endIdIndex - 1]; + } + endIds.push_back(endId); + } + + return endIds; +} + +TestData loadTestData(ModelSpec const& modelSpec, ModelIds const modelIds, BeamResult const& beamResult, + ITensor const& givenInput, SizeType32 const maxBeamWidth, bool const useRandomEndId, bool const replaceLogits, + BufferManager& manager) +{ + auto const [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(givenInput, modelIds.padId); + auto const& [beamWidth, resultsFile, contextLogitsFile, genLogitsFile, cumLogProbsFile, logProbsFile] = beamResult; + + TestData testData{nbGivenInputs, beamWidth}; + testData.expectedOutputIds = loadNpy(manager, resultsFile.string(), MemoryType::kCPU); + + auto* const expectedOutputData = bufferCast<TokenIdType>(*testData.expectedOutputIds); + + auto const& outputShape = testData.expectedOutputIds->getShape(); + EXPECT_EQ(outputShape.nbDims, 2); + EXPECT_EQ(nbGivenInputs * beamWidth, outputShape.d[0]); + testData.maxSeqLen = static_cast<SizeType32>(outputShape.d[1]); + EXPECT_LE(maxInputLength, testData.maxSeqLen); + EXPECT_LE(beamWidth, maxBeamWidth); + + auto const maxNewTokens = testData.maxSeqLen - maxInputLength; + + std::srand(42); + + if (useRandomEndId) + { + testData.endIds = pickRandomEndIds(testData, givenInputLengths, maxNewTokens, replaceLogits); + } + else + { + testData.endIds.insert(testData.endIds.end(), nbGivenInputs, modelIds.endId); + } + + if (modelSpec.useLogits()) + { + testData.loadContextLogits(contextLogitsFile, givenInputLengths, manager); + } + if (modelSpec.useLogits() || modelSpec.mAcceptDraftByLogits) + { + testData.loadGenerationLogits(genLogitsFile, manager); + } + if (modelSpec.mReturnLogProbs) + { + testData.loadLogProbs(cumLogProbsFile, logProbsFile, manager); + } + + for (SizeType32 bi = 0; bi < nbGivenInputs; ++bi) + { + auto const endId = testData.endIds[bi]; + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + SizeType32 expectedLen = givenInputLengths[bi] + maxNewTokens; + for (SizeType32 si = givenInputLengths[bi]; si < testData.maxSeqLen; ++si) + { + auto const expectIndex = tc::flat_index2((bi * beamWidth + beam), si, testData.maxSeqLen); + if (expectedOutputData[expectIndex] == endId) + { + expectedLen = si; + break; + } + } + // Fill new EOS token to the expected data + for (SizeType32 si = expectedLen; si < testData.maxSeqLen; ++si) + { + auto const expectIndex = tc::flat_index2((bi * beamWidth + beam), si, testData.maxSeqLen); + expectedOutputData[expectIndex] = endId; + } + + testData.expectedOutputLengths[bi * beamWidth + beam] = expectedLen; + } + } + + if (modelSpec.mMaxDraftTokens > 0) + { + testData.makeDraft( + modelSpec.mMaxDraftTokens, modelSpec.mAcceptDraftByLogits, genLogitsFile, givenInputLengths, manager); + } + + return testData; +} + +std::tuple<std::vector<SizeType32>, std::unordered_map<SizeType32, TestData>> loadTestData(ModelSpec const& modelSpec, + ModelIds const modelIds, BeamResults const& resultsFilesBeamWidths, ITensor const& givenInput, + SizeType32 const maxBeamWidth, bool const useRandomEndId, bool const replaceLogits, BufferManager& manager) +{ + // Map between beam width, and expected results for that beam width + std::unordered_map<SizeType32, TestData> beamWidthTestData; + std::vector<SizeType32> beamWidths; + + for (auto const& beamResult : resultsFilesBeamWidths) + { + auto const beamWidth = beamResult.beamWidth; + + EXPECT_EQ(std::find(beamWidths.begin(), beamWidths.end(), beamWidth), beamWidths.end()); + beamWidths.push_back(beamWidth); + + auto testData = loadTestData( + modelSpec, modelIds, beamResult, givenInput, maxBeamWidth, useRandomEndId, replaceLogits, manager); + beamWidthTestData.emplace(beamWidth, std::move(testData)); + } + + return {std::move(beamWidths), std::move(beamWidthTestData)}; +} + +RequestList runGptModelInference(std::shared_ptr<TrtGptModel>& trtGptModel, std::vector<SizeType32> const& beamWidths, + std::unordered_map<SizeType32, TestData> const& beamWidthTestData, SizeType32 batchSize, SizeType32 nbGivenInputs, + SizeType32 maxInputLength, SizeType32 padId, std::vector<SizeType32> const& givenInputLengths, + TokenIdType const* givenInputData, ModelSpec const& modelSpec, TrtGptModelIfbTestType testType, int maxReqPerStep, + bool prepopulateKVCache, bool enableStreamingMode, bool enableBlockReuse) +{ + // Fill the requests using givenInput + // requestList will have batchSize requests + RequestList requestList; + + SizeType32 requestId = 0; + RequestList finishedRequestList; + std::vector<SizeType32> reqVec; + // Advance the requests until they are all finished + if (COMM_SESSION.getRank() == 0) + { + SizeType32 numReq = 0; + while (numReq < batchSize) + { + // Add appropriate number of requests in each iteration. For WAVEFRONT, this is always 1. + // For RANDOM, it could be any integer <= maxReqPerStep including 0. + SizeType32 reqThisStep{0}; + switch (testType) + { + case TrtGptModelIfbTestType::WAVEFRONT: reqThisStep = 1; break; + case TrtGptModelIfbTestType::RANDOM: reqThisStep = rand() % (maxReqPerStep + 1); break; + case TrtGptModelIfbTestType::BULK: [[fallthrough]]; + default: reqThisStep = batchSize; break; + } + reqThisStep = std::min(reqThisStep, (batchSize - numReq)); + reqVec.push_back(reqThisStep); + numReq += reqThisStep; + } + } + COMM_SESSION.bcast(reqVec, 0); + + SizeType32 reqVecIdx = 0; + while (requestId < batchSize || !requestList.empty()) + { + SizeType32 reqThisStep = reqVecIdx < reqVec.size() ? reqVec[reqVecIdx++] : 0; + for (SizeType32 req = 0; req < reqThisStep; req++) + { + // Alternate between beamWidths + SizeType32 beamWidth = beamWidths.at(requestId % beamWidths.size()); + auto const& testData = beamWidthTestData.at(beamWidth); + auto const* const expectedOutputData = bufferCast<TokenIdType const>(*testData.expectedOutputIds); + auto const maxSeqLen = testData.maxSeqLen; + + SamplingConfig samplingConfig{beamWidth}; + samplingConfig.temperature = std::vector{1.0f}; + samplingConfig.minLength = std::vector{1}; + samplingConfig.randomSeed = std::vector{static_cast<uint64_t>(42ull)}; + samplingConfig.topK = std::vector{1}; + samplingConfig.topP = std::vector{0.0f}; + samplingConfig.draftAcceptanceThreshold = std::vector{0.3f}; + samplingConfig.noRepeatNgramSize = std::vector{1 << 30}; + + auto const [givenInputIdx, inputLength] + = getRequestGivenInputIdxLength(requestId, nbGivenInputs, givenInputLengths); + SizeType32 endId = testData.endIds[givenInputIdx]; + + auto maxNewTokens = maxSeqLen - maxInputLength; + // Run model only to produce a single token and prepopulate KV cache + if (prepopulateKVCache || modelSpec.mKVCacheType == KVCacheType::kDISABLED) + { + maxNewTokens = 1; + } + auto const* const seqBegin = givenInputData + givenInputIdx * maxInputLength; + auto tokens = std::make_shared<std::vector<int32_t>>(seqBegin, seqBegin + inputLength); + if (!prepopulateKVCache && modelSpec.mMaxDraftTokens > 0) + { + // Append the 1st predicted token to the prompt to get the match with prepopulated KV cache + auto const expectIndex = tc::flat_index3(givenInputIdx, 0, inputLength, 1, maxSeqLen); + auto expectedToken = expectedOutputData[expectIndex]; + tokens->push_back(expectedToken); + // subtract this token from maxNewTokens + maxNewTokens -= 1; + } + auto r = std::make_shared<LlmRequest>(requestId, maxNewTokens, tokens, samplingConfig, false, endId, padId); + + auto const& draftTokens = testData.draftTokens[givenInputIdx]; + auto draftLogits = modelSpec.mAcceptDraftByLogits + ? std::make_optional<ITensor::SharedPtr>(testData.draftLogits[givenInputIdx]) + : std::nullopt; + if (!prepopulateKVCache && !draftTokens.empty()) + { + r->setDraftTokens(std::make_shared<std::vector<TokenIdType>>(draftTokens)); + r->setDraftLogits(draftLogits); + } + + SizeType32 maxDraftTokens{0}; + if (trtGptModel->getModelConfig().hasSpeculativeDecodingModule()) + { + maxDraftTokens + = trtGptModel->getModelConfig().getSpeculativeDecodingModulePtr()->getMaxDecodingDraftTokens(); + } + r->validate(trtGptModel->getMaxInputLen(), trtGptModel->getMaxSequenceLen(), maxDraftTokens, + trtGptModel->getVocabSizePadded(), std::nullopt, enableBlockReuse); + + if (enableStreamingMode) + { + r->setReturnAllGeneratedTokens(true); // Test allGeneratedTokens in this test + r->setStreaming(true); + } + + auto const vocabSizePadded + = trtGptModel->getModelConfig().getVocabSizePadded(trtGptModel->getWorldConfig().getSize()); + auto const logitDatatype = trtGptModel->getLogitDataType(); + if (modelSpec.mGatherLogits) + { + r->setReturnContextLogits(true); + r->setReturnGenerationLogits(true); + r->allocContextLogitsHost(vocabSizePadded, logitDatatype); + r->allocGenerationLogitsHost(vocabSizePadded, logitDatatype); + } + + if (!prepopulateKVCache && modelSpec.mAcceptDraftByLogits && !draftTokens.empty()) + { + r->allocTargetModelAcceptedTokenLogitsHost(vocabSizePadded, logitDatatype); + r->setReturnGenerationLogits(true); + } + + if (modelSpec.mReplaceLogits) + { + LlmRequest::LogitsPostProcessor logitsCb + = [&testData](uint64_t rId, tensorrt_llm::runtime::ITensor::SharedPtr& logits, + LlmRequest::BeamTokens const& tokens, + tensorrt_llm::runtime::BufferManager::CudaStreamPtr streamPtr, std::optional<uint64_t> cId) + { + auto const expectedGenerationLogits = testData.expectedGenerationLogits[rId]; + auto const expectedContextLogits = testData.expectedContextLogits[rId]; + auto const acceptedDraftTokensLengths = testData.acceptedDraftTokensLengths[rId]; + + auto const beamWidth = tokens.size(); + TLLM_CHECK_WITH_INFO(beamWidth == 1, "Logits substitution is not supported for beam search"); + + auto const genLogitsOffset = tokens[0].size() - expectedContextLogits->getShape().d[0]; + // TODO: Avoid static cast in TRT 10.0 + auto const numLogits = static_cast<SizeType32>(logits->getShape().d[0]); + auto const numVerifyLogits = std::min(numLogits, acceptedDraftTokensLengths + 1); + + TensorPtr logitsSlice = ITensor::slice(logits, 0, numVerifyLogits); + + auto manager = BufferManager(streamPtr); + TensorPtr logitsHost = manager.copyFrom(*logitsSlice, MemoryType::kCPU); + manager.getStream().synchronize(); + + TensorPtr refLogitsHost + = ITensor::slice(expectedGenerationLogits, genLogitsOffset, numVerifyLogits); + + EXPECT_TRUE(compareLogits(*refLogitsHost, *logitsHost, 0.f, 1e-2)) << "reqId: " << rId; + + manager.copy(*refLogitsHost, *logitsSlice); + }; + + r->mLogitsPostProcessor = logitsCb; + } + + if (modelSpec.mReturnLogProbs) + { + r->setReturnLogProbs(true); + } + requestList.push_back(r); + ++requestId; + } + + // Advance all active requests by one step + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + + // Check which requests are done, move them out + for (auto it = requestList.cbegin(); it != requestList.cend();) + { + if ((*it)->isGenerationCompleteState()) + { + finishedRequestList.push_back(*it); + requestList.erase(it++); + } + else + { + ++it; + } + } + } + return finishedRequestList; +} + +void runIfbTest(fs::path const& modelPath, ModelSpec const& modelSpec, ModelIds const modelIds, + TrtGptModelType modelType, std::vector<int32_t> const& batchSizes, BeamResults const& resultsFilesBeamWidths, + TrtGptModelIfbTestType testType, int maxReqPerStep, texec::ExecutorConfig const& executorConfig, + bool enableStreamingMode, bool useRandomEndId) +{ + auto manager = BufferManager(std::make_shared<CudaStream>()); + auto const padId = modelIds.padId; + + // Load input data + ASSERT_TRUE(fs::exists(DATA_PATH)); + auto const inputPath = DATA_PATH / modelSpec.mInputFile; + auto const& givenInput = loadNpy(manager, inputPath.string(), MemoryType::kCPU); + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(*givenInput, padId); + auto const* const givenInputData = bufferCast<TokenIdType const>(*givenInput); + + auto const& inputShape = givenInput->getShape(); + ASSERT_EQ(inputShape.nbDims, 2); + ASSERT_GT(inputShape.d[0], 0); + + auto const maxBeamWidth = executorConfig.getMaxBeamWidth(); + // Load expected outputs for each beam width value + auto [beamWidths, beamWidthTestData] = loadTestData(modelSpec, modelIds, resultsFilesBeamWidths, *givenInput, + maxBeamWidth, useRandomEndId, modelSpec.mReplaceLogits, manager); + + int const worldSize = modelSpec.mTPSize * modelSpec.mPPSize * modelSpec.mCPSize; + auto const worldConfig = WorldConfig::mpi(worldSize, modelSpec.mTPSize, modelSpec.mPPSize, modelSpec.mCPSize); + + ASSERT_TRUE(fs::exists(modelPath)); + + for (auto batchSize : batchSizes) + { + std::cout << "=== batchSize:" << batchSize << " ===\n"; + + auto trtGptModel = TrtGptModelFactory::create(modelPath, modelType, executorConfig, false); + + if (modelSpec.mKVCacheType == KVCacheType::kDISABLED) + { + ASSERT_FALSE(trtGptModel->hasKVCacheManager()); + } + + // Prepopulate KV cache for speculative decoding test + bool const prepopulateKVCache = modelSpec.mMaxDraftTokens > 0; + auto finishedRequestList = runGptModelInference(trtGptModel, beamWidths, beamWidthTestData, batchSize, + nbGivenInputs, maxInputLength, padId, givenInputLengths, givenInputData, modelSpec, testType, maxReqPerStep, + prepopulateKVCache, enableStreamingMode, modelSpec.mKVCacheReuse); + + if (prepopulateKVCache) + { + // Call the 2nd time with prefilled KV cache + finishedRequestList = runGptModelInference(trtGptModel, beamWidths, beamWidthTestData, batchSize, + nbGivenInputs, maxInputLength, padId, givenInputLengths, givenInputData, modelSpec, testType, + maxReqPerStep, false, enableStreamingMode, modelSpec.mKVCacheReuse); + } + + // WAR: disabled verification because of switched beams for different batch composition + if (worldConfig.isFirstPipelineParallelRank() + && (testType == TrtGptModelIfbTestType::BULK || maxBeamWidth == 1)) + { + bool shouldVerify = true; + + if (testType == TrtGptModelIfbTestType::BULK) + { + if (modelSpec.mKVCacheType == KVCacheType::kDISABLED && maxBeamWidth != 1) + { + // For disabled KV cache, only verify when maxBeamWidth is 1, the reason is we only compare with + // results with KV cache enabled case and usually, beams search results locate in last token while + // disabled KV cache only get exactly one new token. + shouldVerify = false; + } + } + + if (shouldVerify) + { + verifyOutput(finishedRequestList, beamWidthTestData, givenInputLengths, nbGivenInputs, modelSpec); + } + } + } +} + +struct BeamConfig +{ + SizeType32 maxBeamWidth; + std::vector<SizeType32> beamWidths; +}; + +} // namespace + +using ParamType = std::tuple<ModelParams, ModelSpec, TrtGptModelType, TrtGptModelIfbTestType, BeamConfig, // id: 0-4 + std::optional<int32_t>, // 5. maxTokensInPagedKvCache + std::optional<float>, // 6. freeGpuMemoryFraction + bool, // 7. enableTrtOverlap + bool, // 8. enableChunkedContext + bool, // 9. enableStreamingMode + bool, // 10. enableCudaGraphMode + std::optional<size_t>, // 11. hostCacheSize + bool, // 12. useRandomEndId + std::vector<SizeType32>, // 13. batchSizes + std::optional<SizeType32> // 14. maxNumTokens + >; + +std::string generateTestName(testing::TestParamInfo<ParamType> const& info) +{ + auto const modelSpec = std::get<1>(info.param); + std::string name; + switch (modelSpec.mDataType) + { + case nvinfer1::DataType::kFLOAT: name.append("Float"); break; + case nvinfer1::DataType::kHALF: name.append("Half"); break; + case nvinfer1::DataType::kINT8: name.append("Int8"); break; + case nvinfer1::DataType::kINT32: name.append("Int32"); + case nvinfer1::DataType::kBOOL: name.append("Bool"); break; + case nvinfer1::DataType::kUINT8: name.append("UInt8"); break; + case nvinfer1::DataType::kFP8: name.append("Float8"); break; + case nvinfer1::DataType::kBF16: name.append("BFloat16"); break; + case nvinfer1::DataType::kINT4: name.append("Int4"); break; + case nvinfer1::DataType::kFP4: name.append("Fp4"); break; + default: throw std::runtime_error("Unsupported DataType"); break; + } + + auto const modelType = std::get<2>(info.param); + switch (modelType) + { + case TrtGptModelType::InflightBatching: name.append("IbModel"); break; + case TrtGptModelType::InflightFusedBatching: name.append("FusedIbModel"); break; + default: name.append("DefaultModel"); break; + } + + switch (modelSpec.mKVCacheType) + { + case KVCacheType::kCONTINUOUS: name.append("ContinuousKVCache"); break; + case KVCacheType::kPAGED: name.append("PagedKVCache"); break; + case KVCacheType::kDISABLED: name.append("NoKVCache"); break; + default: throw std::runtime_error("Unknown KVCacheType"); break; + } + + auto const testType = std::get<3>(info.param); + switch (testType) + { + case TrtGptModelIfbTestType::BULK: name.append("Bulk"); break; + case TrtGptModelIfbTestType::WAVEFRONT: name.append("Wavefront"); break; + case TrtGptModelIfbTestType::RANDOM: name.append("Random"); break; + default: name.append("DefaultTest"); break; + } + BeamConfig const beamConfig = std::get<4>(info.param); + name.append("MaxBeamWidth" + std::to_string(beamConfig.maxBeamWidth)); + for (auto const beamWdith : beamConfig.beamWidths) + { + name.append("Bw" + std::to_string(beamWdith)); + } + + auto const maxTokensInPagedKvCache = std::get<5>(info.param); + if (maxTokensInPagedKvCache.has_value()) + { + name.append("KvCacheSize" + std::to_string(maxTokensInPagedKvCache.value())); + } + + auto const freeGpuMemoryFraction = std::get<6>(info.param); + if (freeGpuMemoryFraction.has_value()) + { + name.append("GpuFrac"); + } + + auto const enableTrtOverlap = std::get<7>(info.param); + if (enableTrtOverlap) + { + name.append("TrtOverlap"); + } + + auto const enableChunkedContext = std::get<8>(info.param); + if (enableChunkedContext) + { + name.append("Chunked"); + } + + if (modelSpec.mTPSize > 1) + { + name.append("TP" + std::to_string(modelSpec.mTPSize)); + } + + if (modelSpec.mPPSize > 1) + { + name.append("PP" + std::to_string(modelSpec.mPPSize)); + } + + if (modelSpec.mCPSize > 1) + { + name.append("CP" + std::to_string(modelSpec.mCPSize)); + } + + auto const useRandomEndId = std::get<12>(info.param); + if (useRandomEndId) + { + name.append("EndId"); + } + + if (modelSpec.mMaxDraftTokens > 0) + { + name.append("DraftTokens" + std::to_string(modelSpec.mMaxDraftTokens)); + } + + if (modelSpec.mAcceptDraftByLogits) + { + name.append("AcceptByLogits"); + } + + if (modelSpec.mCapacitySchedulerPolicy) + { + name.append(modelSpec.getCapacitySchedulerString()); + } + + auto const enableStreamingMode = std::get<9>(info.param); + if (enableStreamingMode) + { + name.append("Streaming"); + } + + auto const enableCudaGraphMode = std::get<10>(info.param); + if (enableCudaGraphMode) + { + name.append("CudaGraph"); + } + + auto const enableHostCache = std::get<11>(info.param); + if (enableHostCache) + { + name.append("SecondaryOffloading"); + } + + return name; +} + +class ParamTest : public TrtModelRealDecoderTest, public ::testing::WithParamInterface<ParamType> +{ +}; + +TEST_P(ParamTest, Test) +{ + + auto const& beamConfig = std::get<4>(GetParam()); + auto const& beamWidths = beamConfig.beamWidths; + + auto const modelParams = std::get<0>(GetParam()); + auto const modelIds = modelParams.ids; + auto const* const modelDir = modelParams.baseDir; + auto const modelSpec = std::get<1>(GetParam()); + + auto const useRandomEndId = std::get<12>(GetParam()); + + auto const batchSizes = std::get<13>(GetParam()); + + std::ostringstream gpuSizePath; + gpuSizePath << "tp" << modelSpec.mTPSize << "-pp" << modelSpec.mPPSize << "-cp" << modelSpec.mCPSize; + gpuSizePath << "-gpu"; + + auto const modelPath{ENGINE_PATH / modelDir / modelSpec.getModelPath() / gpuSizePath.str()}; + + auto const inputPath = DATA_PATH / modelSpec.mInputFile; + + BeamResults beamResults; + beamResults.reserve(beamWidths.size()); + for (auto beamWidth : beamWidths) + { + fs::path resultsPath + = DATA_PATH / modelDir / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + fs::path generationLogitsPath + = modelSpec.mCollectGenerationLogits ? (resultsPath / modelSpec.getGenerationLogitsFile()).string() : ""; + fs::path contextLogitsPath + = modelSpec.mCollectContextLogits ? (resultsPath / modelSpec.getContextLogitsFile()).string() : ""; + fs::path cumLogProbsPath + = modelSpec.mCollectCumLogProbs ? (resultsPath / modelSpec.getCumLogProbsFile()).string() : ""; + fs::path logProbsPath = modelSpec.mCollectLogProbs ? (resultsPath / modelSpec.getLogProbsFile()).string() : ""; + + beamResults.emplace_back(beamWidth, (resultsPath / modelSpec.getResultsFile()).string(), contextLogitsPath, + generationLogitsPath, cumLogProbsPath, logProbsPath); + } + + auto const modelType = std::get<2>(GetParam()); + auto const testType = std::get<3>(GetParam()); + auto const enableStreamingMode = std::get<9>(GetParam()); + auto const cudaGraphMode = std::get<10>(GetParam()); + + if (!(modelSpec.mUsePackedInput + && (modelSpec.mKVCacheType == KVCacheType::kPAGED || modelSpec.mKVCacheType == KVCacheType::kDISABLED))) + { + GTEST_SKIP() << "Inflight batching requires packed input and (paged KV cache or disabled KV cache)."; + } + + if (!modelSpec.mUsePackedInput && useRandomEndId) + { + GTEST_SKIP() << "Test does not support endId test with padded inputs"; + } + + for (auto beamWidth : beamWidths) + { + if (useRandomEndId && beamWidth > 1) + { + GTEST_SKIP() << "Test does not support endId test with beam search"; + } + + if (modelSpec.mMaxDraftTokens > 0 && beamWidth > 1) + { + GTEST_SKIP() << "Target model in speculative decoding does not support beam search"; + } + } + + auto executorConfig = texec::ExecutorConfig{}; + + auto const maxTokens = std::get<5>(GetParam()); + auto const enableBlockReuse = modelSpec.mMaxDraftTokens > 0 || modelSpec.mKVCacheReuse; + auto const freeGpuMemoryFraction = std::get<6>(GetParam()); + auto const hostCacheSize = std::get<11>(GetParam()); + auto const kvCacheConfig = texec::KvCacheConfig{ + enableBlockReuse, maxTokens, std::nullopt, std::nullopt, freeGpuMemoryFraction, hostCacheSize}; + executorConfig.setKvCacheConfig(kvCacheConfig); + + executorConfig.setEnableTrtOverlap(std::get<7>(GetParam())); + executorConfig.setEnableChunkedContext(std::get<8>(GetParam())); + auto const maxNumTokens = std::get<14>(GetParam()); + if (maxNumTokens.has_value()) + { + executorConfig.setMaxNumTokens(maxNumTokens.value()); + } + executorConfig.setNormalizeLogProbs(false); + executorConfig.setMaxBeamWidth(beamConfig.maxBeamWidth); + executorConfig.setGatherGenerationLogits(modelSpec.mCollectGenerationLogits); + auto extendedRuntimePerfKnobConfig = texec::ExtendedRuntimePerfKnobConfig{}; + extendedRuntimePerfKnobConfig.setCudaGraphMode(cudaGraphMode); + executorConfig.setExtendedRuntimePerfKnobConfig(extendedRuntimePerfKnobConfig); + + auto const capacitySchedulerPolicy + = modelSpec.mCapacitySchedulerPolicy.value_or(texec::CapacitySchedulerPolicy::kMAX_UTILIZATION); + executorConfig.setSchedulerConfig(texec::SchedulerConfig{capacitySchedulerPolicy}); + + if (modelSpec.mSpecDecodingMode == SpeculativeDecodingMode::LookaheadDecoding()) + { + auto decodingConfig = texec::DecodingConfig{}; + decodingConfig.setLookaheadDecodingConfig(texec::LookaheadDecodingConfig(5, 5, 5)); + executorConfig.setDecodingConfig(decodingConfig); + } + + for (auto beamWidth : beamWidths) + { + if (executorConfig.getEnableTrtOverlap() && beamWidth > 1) + { + GTEST_SKIP() << "TrtOverlap is not supported with beam search"; + } + } + + if (executorConfig.getEnableTrtOverlap() && modelSpec.mMaxDraftTokens > 0) + { + GTEST_SKIP() << "TrtOverlap is not supported with speculative decoding"; + } + + // Warning: This should be the last check before running the test. + // It will initialize MPI which can take significant time. + if (modelSpec.mTPSize * modelSpec.mPPSize * modelSpec.mCPSize != COMM_SESSION.getSize()) + { + GTEST_SKIP() << "Model's world size " << modelSpec.mPPSize * modelSpec.mTPSize * modelSpec.mCPSize + << " is not equal to the system world size"; + } + + runIfbTest(modelPath, modelSpec, modelIds, modelType, batchSizes, beamResults, testType, 2, executorConfig, + enableStreamingMode, useRandomEndId); +} + +auto constexpr gptModelParams = ModelParams{GPT_MODEL_DIR, ModelIds{50256, 50256}}; + +std::shared_ptr<ModelSpec> getGptDraftTestsCompareModelSpec() +{ + auto pModelSpec = std::make_shared<ModelSpec>(INPUT_FILE, nvinfer1::DataType::kHALF); + pModelSpec->useGptAttentionPlugin(); + pModelSpec->gatherLogits(); + pModelSpec->usePackedInput(); + pModelSpec->setKVCacheType(KVCacheType::kPAGED); + + return pModelSpec; +} + +std::shared_ptr<ModelSpec> getMedusaTestsCompareModelSpec() +{ + auto pModelSpec = std::make_shared<ModelSpec>(LONG_INPUT_FILE, nvinfer1::DataType::kHALF); + pModelSpec->useGptAttentionPlugin(); + pModelSpec->usePackedInput(); + pModelSpec->setKVCacheType(KVCacheType::kPAGED); + pModelSpec->setMaxOutputLength(128); + + return pModelSpec; +} + +std::shared_ptr<ModelSpec> getEagleTestsCompareModelSpec() +{ + auto pModelSpec = std::make_shared<ModelSpec>(LONG_INPUT_FILE, nvinfer1::DataType::kHALF); + pModelSpec->useGptAttentionPlugin(); + pModelSpec->usePackedInput(); + pModelSpec->setKVCacheType(KVCacheType::kPAGED); + pModelSpec->setMaxOutputLength(128); + + return pModelSpec; +} + +std::shared_ptr<ModelSpec> getGptChunkedContextTestsCompareModelSpec() +{ + auto pModelSpec = std::make_shared<ModelSpec>(LONG_INPUT_FILE, nvinfer1::DataType::kHALF); + pModelSpec->useGptAttentionPlugin(); + pModelSpec->usePackedInput(); + pModelSpec->setKVCacheType(KVCacheType::kPAGED); + pModelSpec->setMaxInputLength(128); + + return pModelSpec; +} + +INSTANTIATE_TEST_SUITE_P(GptTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput(), + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF, + []() -> std::shared_ptr<ModelSpec> + { + auto pModelSpec = std::make_shared<ModelSpec>(INPUT_FILE, nvinfer1::DataType::kHALF); + pModelSpec->useGptAttentionPlugin().setKVCacheType(KVCacheType::kPAGED).usePackedInput(); + return pModelSpec; + }()} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kDISABLED) + .usePackedInput()), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values( + // TODO: enable more tests when mixed beam width is supported + BeamConfig{1, {1}}, BeamConfig{2, {2}} // , BeamConfig{2, {1, 2}} + ), + testing::Values(std::nullopt, 1280), // maxTokensInPagedKvCache + testing::Values(std::nullopt, 0.4), // freeGpuMemoryFraction + testing::Values(true), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptRandomEndIdTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput()), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values( + // TODO: enable more tests when mixed beam width is supported + BeamConfig{1, {1}}, BeamConfig{2, {2}} // , BeamConfig{2, {1, 2}} + ), + testing::Values(std::nullopt, 1280), // maxTokensInPagedKvCache + testing::Values(std::nullopt, 0.4), // freeGpuMemoryFraction + testing::Values(true), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(true), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptKVOffloadingTest, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // + ModelSpec{LONG_INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput() + .setKVCacheReuse(true)), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), + testing::Values(256), // maxTokensInPagedKvCache + testing::Values(std::nullopt, 0.4), // freeGpuMemoryFraction + testing::Values(true), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(100000000), // hostCacheSize + testing::Values(false, true), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptCudaGraphTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput() + .capacitySchedulerPolicy(texec::CapacitySchedulerPolicy::kSTATIC_BATCH), + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput() + .capacitySchedulerPolicy(texec::CapacitySchedulerPolicy::kMAX_UTILIZATION)), + testing::Values(TrtGptModelType::InflightBatching, TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values( + // TODO: enable more tests when mixed beam width is supported + BeamConfig{1, {1}}, BeamConfig{2, {2}} // , BeamConfig{2, {1, 2}} + ), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(true), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptSwitchBwTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput()), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values( + // TODO: enable more tests when mixed beam width is supported + BeamConfig{2, {1}} // , BeamConfig{2, {1, 2}} + ), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{4}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptNProfilesTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values(ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useMultipleProfiles()), + testing::Values(TrtGptModelType::InflightFusedBatching), testing::Values(TrtGptModelIfbTestType::BULK), + testing::Values( + // TODO: enable more tests when mixed beam width is supported + BeamConfig{1, {1}}, BeamConfig{2, {2}} // , BeamConfig{2, {1, 2}} + ), + testing::Values(std::nullopt, 1280), // maxTokensInPagedKvCache + testing::Values(std::nullopt, 0.4), // freeGpuMemoryFraction + testing::Values(true), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(true), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptSqTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values(ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .setQuantMethod(QuantMethod::kSMOOTH_QUANT), + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF, + []() -> std::shared_ptr<ModelSpec> + { + auto pModelSpec = std::make_shared<ModelSpec>(INPUT_FILE, nvinfer1::DataType::kHALF); + pModelSpec->useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .setQuantMethod(QuantMethod::kSMOOTH_QUANT); + return pModelSpec; + }()} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kDISABLED) + .setQuantMethod(QuantMethod::kSMOOTH_QUANT)), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values( + // TODO: enable more tests when mixed beam width is supported + // FIXME: disabled flaky beam search tests (https://nvbugspro.nvidia.com/bug/4646234) + BeamConfig{1, {1}} //, BeamConfig{2, {2}} + ), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +// disabled because paused requests generate different tokens after resuming +INSTANTIATE_TEST_SUITE_P(DISABLED_GptChunkedContextTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF, getGptChunkedContextTestsCompareModelSpec()} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .setMaxInputLength(128)), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values(TrtGptModelIfbTestType::BULK), // TrtGptModelIfbTestType + testing::Values(BeamConfig{1, {1}}), // beam config + testing::Values(257), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptChunkedLongContextTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // + ModelSpec{LONG_INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .setMaxInputLength(128), + ModelSpec{LONG_INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useDraftTokensExternalDecoding() + .setDraftTokens(5)), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values(TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, + TrtGptModelIfbTestType::RANDOM), // TrtGptModelIfbTestType + testing::Values(BeamConfig{1, {1}}), // beam config + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(true), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(64) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptDraftTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF, getGptDraftTestsCompareModelSpec()} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useDraftTokensExternalDecoding() + .setDraftTokens(5) + .replaceLogits() + .collectGenerationLogitsFile() + .collectContextLogitsFile(), + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF, getGptDraftTestsCompareModelSpec()} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useDraftTokensExternalDecoding() + .setDraftTokens(5) + .useAcceptByLogits() + .replaceLogits() + .collectGenerationLogitsFile() + .collectContextLogitsFile()), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), // beamConfig + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false, true), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptLogitsTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // modelSpec + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .gatherLogits() + .collectGenerationLogitsFile() + .collectContextLogitsFile()), + testing::Values(TrtGptModelType::InflightBatching, TrtGptModelType::InflightFusedBatching), // modelType + testing::Values(TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, + TrtGptModelIfbTestType::RANDOM), // testType + testing::Values(BeamConfig{1, {1}}), // beamConfig + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false, true), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(true), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptLogProbsTests, ParamTest, + testing::Combine(testing::Values(gptModelParams), + testing::Values( + // modelSpec + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .returnLogProbs() + .collectCumLogProbsFile() + .collectLogProbsFile()), + testing::Values(TrtGptModelType::InflightFusedBatching), // modelType + testing::Values(TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, + TrtGptModelIfbTestType::RANDOM), // testType + testing::Values(BeamConfig{1, {1}}), // beamConfig + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptjTests, ParamTest, + testing::Combine(testing::Values(ModelParams{GPTJ_MODEL_DIR, {50256, 50256}}), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kCONTINUOUS) + .usePackedInput(), + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput() + + ), + testing::Values(TrtGptModelType::InflightFusedBatching), + // WAR: disable wavefront and random tests on because of switched beams + testing::Values(TrtGptModelIfbTestType::BULK + /* , TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM */), + testing::Values( + // TODO: enable more tests when mixed beam width is supported + BeamConfig{1, {1}}, BeamConfig{2, {2}} // , BeamConfig{2, {1, 2}} + ), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(MambaTests, ParamTest, + testing::Combine(testing::Values(ModelParams{MAMBA_MODEL_DIR, {0, 1}}), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kCONTINUOUS) + .usePackedInput(), + ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput() + + ), + testing::Values(TrtGptModelType::InflightBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(RecurrentGemmaTests, ParamTest, + testing::Combine(testing::Values(ModelParams{RECURRENTGEMMA_MODEL_DIR, {0, 1}}), + testing::Values(ModelSpec{INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput() + + ), + testing::Values(TrtGptModelType::InflightBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(LlamaTests, ParamTest, + testing::Combine(testing::Values(ModelParams{LLAMA_MODEL_DIR, {LLAMA_END_ID, LLAMA_PAD_ID}}), + testing::Values( + // + ModelSpec{INPUT_LLAMA_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput(), + ModelSpec{INPUT_LLAMA_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .usePipelineParallelism(4), + ModelSpec{INPUT_LLAMA_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useTensorParallelism(4), + ModelSpec{INPUT_LLAMA_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .usePipelineParallelism(2) + .useTensorParallelism(2) + + ), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values( + // TODO: enable more tests when mixed beam width is supported + BeamConfig{1, {1}}, BeamConfig{2, {2}} // , BeamConfig{2, {1, 2}} + ), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(ChatGlmTests, ParamTest, + testing::Combine(testing::Values(ModelParams{CHATGLM_MODEL_DIR, {130005, 3}}), + testing::Values( + // + ModelSpec{CHATGLM_INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED)), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(false, true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +// ChatGlm0Tests is for glm-10b. +INSTANTIATE_TEST_SUITE_P(ChatGlm0Tests, ParamTest, + testing::Combine(testing::Values(ModelParams{GLM_MODEL_DIR, {50258, 50256}}), + testing::Values( + // + ModelSpec{GLM_INPUT_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED)), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +// https://nvbugspro.nvidia.com/bug/4640177 +// WAVEFRONT and RANDOM are disabled because of the accuracy mismatch +INSTANTIATE_TEST_SUITE_P(MedusaTests, ParamTest, + testing::Combine(testing::Values(ModelParams{MEDUSA_MODEL_DIR, {2, 2}}), + testing::Values( + // + ModelSpec{INPUT_VICUNA_FILE, nvinfer1::DataType::kHALF, getMedusaTestsCompareModelSpec()} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useMedusa()), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(true, false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(EagleTests, ParamTest, + testing::Combine(testing::Values(ModelParams{EAGLE_MODEL_DIR, {2, 2}}), + testing::Values( + // + ModelSpec{INPUT_VICUNA_FILE, nvinfer1::DataType::kHALF, getEagleTestsCompareModelSpec()} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useEagle()), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(true, false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(LlamaLookaheadDecodingTests, ParamTest, + testing::Combine(testing::Values(ModelParams{LLAMA_MODEL_DIR, {LLAMA_END_ID, LLAMA_PAD_ID}}), + testing::Values( + // + ModelSpec{INPUT_LLAMA_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useLookaheadDecoding()), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), // beamConfig + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(false), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(true), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 16}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + + generateTestName); + +INSTANTIATE_TEST_SUITE_P(ExplicitDraftTokensDecodingTests, ParamTest, + testing::Combine(testing::Values(ModelParams{EXPLICIT_DRAFT_MODEL_DIR, {2, 2}}), + testing::Values( + // + ModelSpec{INPUT_VICUNA_FILE, nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useExplicitDraftTokensDecoding() + .setMaxOutputLength(128)), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values(BeamConfig{1, {1}}), // beamConfig + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + + generateTestName); + +#ifdef ENABLE_FP8 +// Using IFB-enabled engine +INSTANTIATE_TEST_SUITE_P(GptjFP8Tests, ParamTest, + testing::Combine(testing::Values(ModelParams{GPTJ_MODEL_DIR, {50256, 50256}}), + testing::Values( + // + ModelSpec{INPUT_FILE, nvinfer1::DataType::kFP8} + .useGptAttentionPlugin() + .setKVCacheType(KVCacheType::kPAGED) + .usePackedInput() + + ), + testing::Values(TrtGptModelType::InflightFusedBatching), + testing::Values( + TrtGptModelIfbTestType::BULK, TrtGptModelIfbTestType::WAVEFRONT, TrtGptModelIfbTestType::RANDOM), + testing::Values( + // TODO: enable more tests when supported + BeamConfig{1, {1}} // , BeamConfig{2, {2}}, BeamConfig{2, {1, 2}} + ), + testing::Values(std::nullopt), // maxTokensInPagedKvCache + testing::Values(0.4), // freeGpuMemoryFraction + testing::Values(false), // enableTrtOverlap + testing::Values(true), // enableChunkedContext + testing::Values(false), // enableStreamingMode + testing::Values(false), // enableCudaGraphMode + testing::Values(std::nullopt), // hostCacheSize + testing::Values(false), // useRandomEndId + testing::Values(std::vector<SizeType32>{1, 2, 8}), // batchSizes + testing::Values(std::nullopt) // maxNumTokens + ), + generateTestName); + +#endif diff --git a/cpp/tests/e2e_tests/batch_manager/trtGptModelTest.cpp b/cpp/tests/e2e_tests/batch_manager/trtGptModelTest.cpp new file mode 100644 index 000000000000..268e9bf9a238 --- /dev/null +++ b/cpp/tests/e2e_tests/batch_manager/trtGptModelTest.cpp @@ -0,0 +1,1328 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TOP_LEVEL_DIR +#error "Define TOP_LEVEL_DIR" +#endif + +#include "tensorrt_llm/batch_manager/trtGptModel.h" +#include "tensorrt_llm/batch_manager/kvCacheManager.h" +#include "tensorrt_llm/batch_manager/trtGptModelInflightBatching.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/gptJsonConfig.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/testing/modelSpec.h" + +#include <gmock/gmock.h> +#include <gtest/gtest.h> + +#include <filesystem> +#include <memory> +#include <vector> + +using ::testing::ElementsAre; +using namespace tensorrt_llm::runtime; +namespace fs = std::filesystem; +using tensorrt_llm::testing::ModelSpec; +using tensorrt_llm::testing::KVCacheType; + +using TensorPtr = ITensor::SharedPtr; + +namespace +{ +auto const TEST_RESOURCE_PATH = fs::path{TOP_LEVEL_DIR} / "cpp/tests/resources"; +auto const ENGINE_PATH = TEST_RESOURCE_PATH / "models/rt_engine"; +auto const GPT_MODEL_PATH = ENGINE_PATH / "gpt2"; +auto const LLAMA_MODEL_PATH = ENGINE_PATH / "Llama-3.2-1B"; +} // namespace + +namespace tensorrt_llm::batch_manager +{ + +class TrtGptModelTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type-member-init) +{ +protected: + TrtGptModelTest(std::filesystem::path const& modelPath) + : mModelConfig(1, 1, 1, 0, 1, 1, nvinfer1::DataType::kFLOAT) + , mModelPath(modelPath) + { + } + + TrtGptModelTest() + : TrtGptModelTest(GPT_MODEL_PATH / GetModelSpec().getModelPath() / "tp1-pp1-cp1-gpu") + { + } + + static ModelSpec& GetModelSpec() + { + static ModelSpec modelSpec{"input_tokens.npy", nvinfer1::DataType::kHALF}; + modelSpec.useGptAttentionPlugin().usePackedInput().setKVCacheType(KVCacheType::kPAGED); + return modelSpec; + } + + void SetUp() override + { + std::filesystem::path trtEnginePath = mModelPath; + + mBeamWidth = 1; + + mLogger = std::make_shared<TllmLogger>(); + + initTrtLlmPlugins(mLogger.get()); + + auto const json = GptJsonConfig::parse(trtEnginePath / "config.json"); + mModelConfig = json.getModelConfig(); + mMaxNumRequests = mModelConfig.getMaxBatchSize(); + mMaxSeqLen = mModelConfig.getMaxSequenceLen(); + mWorldConfig = WorldConfig::mpi(); + mVocabSizePadded = mModelConfig.getVocabSizePadded(mWorldConfig.getSize()); + + auto const enginePath = trtEnginePath / json.engineFilename(mWorldConfig); + auto const dtype = mModelConfig.getDataType(); + + mRawEngine.reset(new RawEngine(enginePath)); + + mSamplingConfig.temperature = std::vector{1.0f}; + mSamplingConfig.minLength = std::vector{1}; + mSamplingConfig.randomSeed = std::vector{static_cast<uint64_t>(42ul)}; + mSamplingConfig.topK = std::vector{0}; + mSamplingConfig.topP = std::vector{0.0f}; + mSamplingConfig.noRepeatNgramSize = std::vector{1 << 30}; + + mStream = std::make_unique<CudaStream>(); + mManager = std::make_unique<BufferManager>(mStream); + } + + void TearDown() override {} + + // Thin wrapper around the private TrtGptModelInflightBatching::changeBeamWidth(). + static void changeBeamWidth(std::shared_ptr<TrtGptModelInflightBatching> const& model, SizeType32 beamWidth) + { + model->changeBeamWidth(beamWidth); + } + + void forwardRequestsToCompletion( + std::shared_ptr<TrtGptModel> const& trtGptModel, RequestList& requestList, SizeType32 maxNumIterations) + { + SizeType32 numFinished = 0; + SizeType32 numIterations = 0; + while (numFinished < requestList.size() && numIterations < maxNumIterations) + { + if (numIterations > maxNumIterations) + { + FAIL() << "Iterations never finished"; + } + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + numFinished = 0; + for (auto& request : requestList) + { + if (request->isGenerationCompleteState()) + { + ++numFinished; + } + } + ++numIterations; + } + } + + int32_t mMaxNumRequests; + int32_t mMaxSeqLen; + int32_t mBeamWidth; + int32_t mVocabSizePadded; + SamplingConfig mSamplingConfig; + std::string mDataPath; + std::shared_ptr<nvinfer1::ILogger> mLogger; + ModelConfig mModelConfig; + WorldConfig mWorldConfig; + std::unique_ptr<RawEngine> mRawEngine; + std::unique_ptr<BufferManager> mManager; + BufferManager::CudaStreamPtr mStream; + std::filesystem::path mModelPath; +}; + +class TrtGptModelLoraTest : public TrtGptModelTest +{ +protected: + TrtGptModelLoraTest() + : TrtGptModelTest(GPT_MODEL_PATH / GetModelSpec().getModelPath() / "tp1-pp1-cp1-gpu") + { + } + + static ModelSpec& GetModelSpec() + { + static ModelSpec modelSpec{"input_tokens.npy", nvinfer1::DataType::kHALF}; + modelSpec.useGptAttentionPlugin().usePackedInput().setKVCacheType(KVCacheType::kPAGED).useLoraPlugin(); + return modelSpec; + } +}; + +TEST_F(TrtGptModelTest, Forward) +{ + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 4; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{1, 2, 3, 4}); + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + + RequestList requestList{llmRequest}; + + auto& manager = *mManager; + std::vector<int32_t> newTokensHost(mMaxNumRequests, 5); + TensorPtr const fakeNewTokens + = manager.copyFrom(newTokensHost, ITensor::makeShape({mMaxNumRequests, 1}), MemoryType::kGPU); + + std::vector<bool> finished(mMaxNumRequests, false); + + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig(executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kMAX_UTILIZATION}); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + // Generate one token for the requests in request_table + // We need to sync with decoder + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + + EXPECT_EQ(requestList.size(), 1); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_IN_PROGRESS); + EXPECT_EQ(requestList.front()->getNumTokens(0), 5); + EXPECT_EQ(requestList.front()->getMaxNumGeneratedTokens(), 1); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(1, 2, 3, 4, 2)); +} + +TEST_F(TrtGptModelTest, ChangeBeamWidthClearsCudaGraphCache) +{ + if (mModelConfig.getMaxBeamWidth() < 2) + { + GTEST_SKIP() << "Engine was built with max_beam_width < 2; cannot exercise changeBeamWidth()."; + } + + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + // Configure the executor for max beam width = 2 so we can transition between + // operating beam widths 1 and 2. + executorConfig.setMaxBeamWidth(2); + executorConfig.setSchedulerConfig(executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kMAX_UTILIZATION}); + + auto extendedRuntimePerfKnobConfig = executor::ExtendedRuntimePerfKnobConfig{}; + extendedRuntimePerfKnobConfig.setCudaGraphMode(true); + extendedRuntimePerfKnobConfig.setCudaGraphCacheSize(8); + executorConfig.setExtendedRuntimePerfKnobConfig(extendedRuntimePerfKnobConfig); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + // Run a single beam=1 request to completion. After at least one generation step + // the model captures and caches a CUDA graph for the subsequent batch state. + SamplingConfig samplingConfig; + samplingConfig.beamWidth = 1; + samplingConfig.temperature = std::vector{1.0f}; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{1, 2, 3, 4}); + auto llmRequest = std::make_shared<LlmRequest>( + /*requestId=*/0, /*maxNewTokens=*/4, tokens, samplingConfig, /*isStreaming=*/false); + RequestList requestList{llmRequest}; + + forwardRequestsToCompletion(trtGptModel, requestList, /*maxNumIterations=*/8); + + // Cache must have been populated by the captured generation graph(s). + EXPECT_GT(trtGptModel->numCachedCudaGraphs(), 0) + << "Expected the CUDA graph executor cache to be populated after running a " + "beam=1 request to completion."; + + // Drop the completed request before changing beam width (changeBeamWidth requires + // no in-flight requests). + requestList.clear(); + + // Switch operating beam width via the fixture's friend-access helper. + changeBeamWidth(trtGptModel, 2); + + EXPECT_EQ(trtGptModel->numCachedCudaGraphs(), 0) + << "changeBeamWidth() must invalidate the CUDA graph executor cache. Stale " + "cudaGraphExec_t instances captured against the previous decoder state " + "would otherwise be replayed against freshly allocated memory."; +} + +TEST_F(TrtGptModelLoraTest, Forward) +{ + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 4; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{1, 2, 3, 4}); + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + + RequestList requestList{llmRequest}; + + auto& manager = *mManager; + std::vector<int32_t> newTokensHost(mMaxNumRequests, 5); + TensorPtr const fakeNewTokens + = manager.copyFrom(newTokensHost, ITensor::makeShape({mMaxNumRequests, 1}), MemoryType::kGPU); + + std::vector<bool> finished(mMaxNumRequests, false); + + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig(executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kMAX_UTILIZATION}); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + // Generate one token for the requests in request_table + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + + EXPECT_EQ(requestList.size(), 1); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_IN_PROGRESS); + EXPECT_EQ(requestList.front()->getNumTokens(0), 5); + EXPECT_EQ(requestList.front()->getMaxNumGeneratedTokens(), 1); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(1, 2, 3, 4, 2)); +} + +TEST_F(TrtGptModelTest, ForwardMaxNewTokens) +{ + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT}); + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setMaxTokens(10000); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 4; + auto tokens = std::make_shared<std::vector<int32_t>>(256); + std::iota(std::begin(*tokens), std::end(*tokens), 1); + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + + int correlationId2 = 2; + auto maxNewTokens2 = 8; + auto llmRequest2 = std::make_shared<LlmRequest>(correlationId2, maxNewTokens2, tokens, inSamplingConfig, false); + + RequestList requestList{llmRequest, llmRequest2}; + + auto& manager = *mManager; + std::vector<bool> finished(mMaxNumRequests, false); + + // Generate one token for the requests in request_table + // We call forward twice because the first call doesn't sync with decoder + SizeType32 maxNumIterations = 13; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + + for (auto& request : requestList) + { + auto outputTokens = request->getTokens(0); + if (request->mRequestId == correlationId) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + } + if (request->mRequestId == correlationId2) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens2); + } + } +} + +TEST_F(TrtGptModelTest, MaxNumTokensInChunked) +{ + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setEnableChunkedContext(true); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT}); + + auto modelConfig = mModelConfig; + mModelConfig.setMaxNumTokens(200); + + auto trtGptModelIfb = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + std::vector<std::shared_ptr<TrtGptModel>> trtGptModels{trtGptModelIfb}; + + for (auto trtGptModel : trtGptModels) + { + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 4; + auto tokens = std::make_shared<std::vector<int32_t>>(256); + std::iota(std::begin(*tokens), std::end(*tokens), 1); + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + + int correlationId2 = 2; + auto maxNewTokens2 = 8; + auto llmRequest2 = std::make_shared<LlmRequest>(correlationId2, maxNewTokens2, tokens, inSamplingConfig, false); + + RequestList requestList{llmRequest, llmRequest2}; + + auto& manager = *mManager; + std::vector<bool> finished(mMaxNumRequests, false); + + // Generate one token for the requests in request_table + // We call forward twice because the first call doesn't sync with decoder + SizeType32 maxNumIterations = 13; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + + for (auto& request : requestList) + { + auto outputTokens = request->getTokens(0); + if (request->mRequestId == correlationId) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + } + if (request->mRequestId == correlationId2) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens2); + } + } + } +} + +TEST_F(TrtGptModelTest, ForwardEndId) +{ + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT}); + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setMaxTokens(10000); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 4; + auto endId = 107; + auto tokens = std::make_shared<std::vector<int32_t>>(256); + std::iota(std::begin(*tokens), std::end(*tokens), 1); + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false, endId); + + int correlationId2 = 2; + auto maxNewTokens2 = 8; + auto llmRequest2 + = std::make_shared<LlmRequest>(correlationId2, maxNewTokens2, tokens, inSamplingConfig, false, endId); + + RequestList requestList{llmRequest, llmRequest2}; + + auto& manager = *mManager; + std::vector<bool> finished(mMaxNumRequests, false); + + // Generate one token for the requests in request_table + // We call forward twice because the first call doesn't sync with decoder + SizeType32 maxNumIterations = 13; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + + for (auto& request : requestList) + { + auto outputTokens = request->getTokens(0); + // endId token is generated at 2nd iteration, so expect 1 output token + if (request->mRequestId == correlationId) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + 1); + } + if (request->mRequestId == correlationId2) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + 1); + } + } +} + +TEST_F(TrtGptModelTest, ForwardNoEoS) +{ + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig(executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kSTATIC_BATCH}); + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setMaxTokens(10000); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + SamplingConfig inSamplingConfig; + inSamplingConfig.topP = {0.9}; + inSamplingConfig.temperature = {0.6}; + inSamplingConfig.minLength = {5}; + + auto tokens = std::make_shared<std::vector<int32_t>>(256); + std::iota(std::begin(*tokens), std::end(*tokens), 1); + + RequestList requestList; + for (auto requestIdx = 0; requestIdx < mMaxNumRequests; requestIdx++) + { + auto llmRequest = std::make_shared<LlmRequest>(requestIdx, 8, tokens, inSamplingConfig, false, -1); + requestList.push_back(llmRequest); + } + + auto& manager = *mManager; + std::vector<bool> finished(mMaxNumRequests, false); + + // Generate one token for the requests in request_table + // We call forward twice because the first call doesn't sync with decoder + SizeType32 maxNumIterations = 13; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); +} + +TEST_F(TrtGptModelTest, ForwardFinished) +{ + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 2; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{10, 9, 8, 7, 6}); + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + + RequestList requestList{llmRequest}; + + int mForwardCount = 0; + + auto& manager = *mManager; + std::vector<int32_t> newTokensHost(mMaxNumRequests, 5); + TensorPtr const fakeNewTokens + = manager.copyFrom(newTokensHost, ITensor::makeShape({mMaxNumRequests, 1}), MemoryType::kGPU); + + std::vector<int32_t> newTokensHost2(mMaxNumRequests, 4); + TensorPtr const fakeNewTokens2 + = manager.copyFrom(newTokensHost2, ITensor::makeShape({mMaxNumRequests, 1}), MemoryType::kGPU); + + // Below are only used if beam > 1 + // So we are just returning tensors with the correct shape, content is not important + std::vector<int32_t> outputIdsHost(mMaxNumRequests * (5 + 2), 5); + TensorPtr const fakeOutputIds + = manager.copyFrom(outputIdsHost, ITensor::makeShape({mMaxNumRequests, 1, 5 + 2}), MemoryType::kGPU); + + std::vector<bool> finishedFalse(mMaxNumRequests, false); + std::vector<bool> finishedTrue(mMaxNumRequests, true); + + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig(executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kMAX_UTILIZATION}); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + // Generate one token for the requests in request_table + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + + EXPECT_EQ(requestList.size(), 1); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_IN_PROGRESS); + EXPECT_EQ(requestList.front()->getNumTokens(0), 6); + EXPECT_EQ(requestList.front()->getMaxNumGeneratedTokens(), 1); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10)); + + // Generate one more token + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + + EXPECT_EQ(requestList.size(), 1); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + EXPECT_EQ(requestList.front()->getNumTokens(0), 7); + EXPECT_EQ(requestList.front()->getMaxNumGeneratedTokens(), 2); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6)); +} + +TEST_F(TrtGptModelTest, ForwardStopWords) +{ + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT}); + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setMaxTokens(10000); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 4; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{10, 9, 8, 7, 6}); + std::optional<SizeType32> endId(std::nullopt); + std::optional<SizeType32> padId(std::nullopt); + std::optional<TensorPtr> embeddingBias(std::nullopt); + std::optional<TensorPtr> badWordsList(std::nullopt); + + auto& manager = *mManager; + // No stop words + { + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + RequestList requestList{llmRequest}; + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6, 10, 6)); + } + // With stop words + { + TensorPtr stopWordsList = manager.cpu(ITensor::makeShape({1, 2, 3}), nvinfer1::DataType::kINT32); + auto stopWordsPtr = bufferCast<int32_t>(*stopWordsList); + // make 10, 6 10 the tokens for the stop word: + stopWordsPtr[0] = 10; + stopWordsPtr[1] = 6; + stopWordsPtr[2] = 10; + stopWordsPtr[3] = 3; + stopWordsPtr[4] = -1; + stopWordsPtr[5] = -1; + + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false, + endId, padId, embeddingBias, badWordsList, stopWordsList); + RequestList requestList{llmRequest}; + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6, 10)); + } + + // With stop words + { + TensorPtr stopWordsList = manager.cpu(ITensor::makeShape({1, 2, 1}), nvinfer1::DataType::kINT32); + auto stopWordsPtr = bufferCast<int32_t>(*stopWordsList); + // make 10 is the token for the stop word: + stopWordsPtr[0] = 10; + stopWordsPtr[1] = 1; + + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false, + endId, padId, embeddingBias, badWordsList, stopWordsList); + RequestList requestList{llmRequest}; + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10)); + } + + // Multiple requests, each with different stop words + { + // Request w/o stop words + auto llmRequest = std::make_shared<LlmRequest>(1, maxNewTokens, tokens, inSamplingConfig, false); + + TensorPtr stopWordsList2 = manager.cpu(ITensor::makeShape({1, 2, 1}), nvinfer1::DataType::kINT32); + { + auto stopWordsPtr = bufferCast<int32_t>(*stopWordsList2); + stopWordsPtr[0] = 10; + stopWordsPtr[1] = 1; + } + auto llmRequest2 = std::make_shared<LlmRequest>(2, maxNewTokens, tokens, inSamplingConfig, false, endId, padId, + embeddingBias, badWordsList, stopWordsList2); + + TensorPtr stopWordsList3 = manager.cpu(ITensor::makeShape({1, 2, 3}), nvinfer1::DataType::kINT32); + { + auto stopWordsPtr = bufferCast<int32_t>(*stopWordsList3); + stopWordsPtr[0] = 10; + stopWordsPtr[1] = 6; + stopWordsPtr[2] = 10; + stopWordsPtr[3] = 3; + stopWordsPtr[4] = -1; + stopWordsPtr[5] = -1; + } + auto llmRequest3 = std::make_shared<LlmRequest>(3, maxNewTokens, tokens, inSamplingConfig, false, endId, padId, + embeddingBias, badWordsList, stopWordsList3); + + RequestList requestList{llmRequest, llmRequest2, llmRequest3}; + + SizeType32 maxNumIterations(5); + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + + for (auto& request : requestList) + { + auto outputTokens = request->getTokens(0); + if (request->mRequestId == 1) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + EXPECT_THAT(request->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6, 10, 6)); + } + if (request->mRequestId == 2) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + 1); + EXPECT_THAT(request->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10)); + } + if (request->mRequestId == 3) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + 3); + EXPECT_THAT(request->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6, 10)); + } + } + } +} + +TEST_F(TrtGptModelTest, ForwardBadWords) +{ + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT}); + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setMaxTokens(10000); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 4; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{10, 9, 8, 7, 6}); + std::optional<SizeType32> endId(std::nullopt); + std::optional<SizeType32> padId(std::nullopt); + std::optional<TensorPtr> embeddingBias(std::nullopt); + std::optional<TensorPtr> stopWordsList(std::nullopt); + + auto& manager = *mManager; + // No bad words + { + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + RequestList requestList{llmRequest}; + + SizeType32 maxNumIterations = 5; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6, 10, 6)); + } + // With bad words, multiple tokens + { + TensorPtr badWordsList = manager.cpu(ITensor::makeShape({1, 2, 3}), nvinfer1::DataType::kINT32); + auto badWordsPtr = bufferCast<int32_t>(*badWordsList); + // make 10, 6 10 the tokens for the bad word: + badWordsPtr[0] = 10; + badWordsPtr[1] = 6; + badWordsPtr[2] = 10; + badWordsPtr[3] = 3; + badWordsPtr[4] = -1; + badWordsPtr[5] = -1; + + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false, + endId, padId, embeddingBias, badWordsList, stopWordsList); + RequestList requestList{llmRequest}; + SizeType32 maxNumIterations = 5; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + // Token at position 7 should be different than 10 + EXPECT_NE(requestList.front()->getTokens(0).at(7), 10); + } + + // With bad words single token + { + TensorPtr badWordsList = manager.cpu(ITensor::makeShape({1, 2, 1}), nvinfer1::DataType::kINT32); + auto badWordsPtr = bufferCast<int32_t>(*badWordsList); + // make 10 is the token for the bad word: + badWordsPtr[0] = 10; + badWordsPtr[1] = 1; + + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false, + endId, padId, embeddingBias, badWordsList, stopWordsList); + RequestList requestList{llmRequest}; + SizeType32 maxNumIterations = 5; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + EXPECT_NE(requestList.front()->getTokens(0).at(5), 10); + } + + // Multiple requests, each with different bad words + { + // Request w/o bad words + auto llmRequest = std::make_shared<LlmRequest>(1, maxNewTokens, tokens, inSamplingConfig, false); + + TensorPtr badWordsList2 = manager.cpu(ITensor::makeShape({1, 2, 1}), nvinfer1::DataType::kINT32); + { + auto badWordsPtr = bufferCast<int32_t>(*badWordsList2); + badWordsPtr[0] = 10; + badWordsPtr[1] = 1; + } + auto llmRequest2 = std::make_shared<LlmRequest>(2, maxNewTokens, tokens, inSamplingConfig, false, endId, padId, + embeddingBias, badWordsList2, stopWordsList); + + TensorPtr badWordsList3 = manager.cpu(ITensor::makeShape({1, 2, 3}), nvinfer1::DataType::kINT32); + { + auto badWordsPtr = bufferCast<int32_t>(*badWordsList3); + badWordsPtr[0] = 10; + badWordsPtr[1] = 6; + badWordsPtr[2] = 10; + badWordsPtr[3] = 3; + badWordsPtr[4] = -1; + badWordsPtr[5] = -1; + } + auto llmRequest3 = std::make_shared<LlmRequest>(3, maxNewTokens, tokens, inSamplingConfig, false, endId, padId, + embeddingBias, badWordsList3, stopWordsList); + + RequestList requestList{llmRequest, llmRequest2, llmRequest3}; + + SizeType32 maxNumIterations(6); + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + + for (auto& request : requestList) + { + auto outputTokens = request->getTokens(0); + if (request->mRequestId == 1) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + EXPECT_THAT(request->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6, 10, 6)); + } + if (request->mRequestId == 2) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + EXPECT_NE(request->getTokens(0).at(5), 10); + } + if (request->mRequestId == 3) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + EXPECT_NE(request->getTokens(0).at(7), 10); + } + } + } +} + +TEST_F(TrtGptModelTest, ForwardEmbeddingBias) +{ + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT}); + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setMaxTokens(10000); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto trtGptModelIfb = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + std::vector<std::shared_ptr<TrtGptModel>> trtGptModels{trtGptModelIfb}; + + for (auto& trtGptModel : trtGptModels) + { + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 4; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{10, 9, 8, 7, 6}); + std::optional<SizeType32> endId(std::nullopt); + std::optional<SizeType32> padId(std::nullopt); + std::optional<TensorPtr> badWordsList(std::nullopt); + std::optional<TensorPtr> stopWordsList(std::nullopt); + + auto& manager = *mManager; + // No bad words + { + auto llmRequest + = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + RequestList requestList{llmRequest}; + + SizeType32 maxNumIterations = 5; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6, 10, 6)); + } + // With embedding bias + { + TensorPtr embeddingBias + = manager.cpu(ITensor::makeShape({1, mVocabSizePadded}), nvinfer1::DataType::kFLOAT); + auto embeddingBiasPtr = bufferCast<float>(*embeddingBias); + for (SizeType32 vi = 0; vi < mVocabSizePadded; ++vi) + { + embeddingBiasPtr[vi] = 0.f; + } + // bias all words to the 10th token + embeddingBiasPtr[10] = std::numeric_limits<float>::max(); + + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false, + endId, padId, embeddingBias, badWordsList, stopWordsList); + RequestList requestList{llmRequest}; + SizeType32 maxNumIterations = 5; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + // All tokens should become 10 after applying bias + EXPECT_EQ(requestList.front()->getTokens(0).at(5), 10); + EXPECT_EQ(requestList.front()->getTokens(0).at(6), 10); + EXPECT_EQ(requestList.front()->getTokens(0).at(7), 10); + EXPECT_EQ(requestList.front()->getTokens(0).at(8), 10); + } + + // Multiple requests, each with different bias + { + // Request w/o bias + auto llmRequest = std::make_shared<LlmRequest>(1, maxNewTokens, tokens, inSamplingConfig, false); + + TensorPtr embeddingBias1 + = manager.cpu(ITensor::makeShape({1, mVocabSizePadded}), nvinfer1::DataType::kFLOAT); + auto embeddingBias1Ptr = bufferCast<float>(*embeddingBias1); + for (SizeType32 vi = 0; vi < mVocabSizePadded; ++vi) + { + embeddingBias1Ptr[vi] = 0.f; + } + // bias all words to the 10th token + embeddingBias1Ptr[10] = std::numeric_limits<float>::max(); + + auto llmRequest2 = std::make_shared<LlmRequest>(2, maxNewTokens, tokens, inSamplingConfig, false, endId, + padId, embeddingBias1, badWordsList, stopWordsList); + + TensorPtr embeddingBias2 + = manager.cpu(ITensor::makeShape({1, mVocabSizePadded}), nvinfer1::DataType::kFLOAT); + auto embeddingBias2Ptr = bufferCast<float>(*embeddingBias2); + for (SizeType32 vi = 0; vi < mVocabSizePadded; ++vi) + { + embeddingBias2Ptr[vi] = 0.f; + } + // bias all words to the 100th token + embeddingBias2Ptr[100] = std::numeric_limits<float>::max(); + + auto llmRequest3 = std::make_shared<LlmRequest>(3, maxNewTokens, tokens, inSamplingConfig, false, endId, + padId, embeddingBias2, badWordsList, stopWordsList); + + RequestList requestList{llmRequest, llmRequest2, llmRequest3}; + + SizeType32 maxNumIterations(6); + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + + for (auto& request : requestList) + { + auto outputTokens = request->getTokens(0); + if (request->mRequestId == 1) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + EXPECT_THAT(request->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 6, 10, 6)); + } + if (request->mRequestId == 2) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + EXPECT_THAT(request->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 10, 10, 10, 10)); + } + if (request->mRequestId == 3) + { + EXPECT_EQ(outputTokens.size(), tokens->size() + maxNewTokens); + EXPECT_THAT(request->getTokens(0), ElementsAre(10, 9, 8, 7, 6, 100, 100, 100, 100)); + } + } + } + } +} + +class TrtGptModelIfbHelper : public TrtGptModelInflightBatching +{ +public: + using TrtGptModelInflightBatching::TrtGptModelInflightBatching; + + [[nodiscard]] std::shared_ptr<kv_cache_manager::BaseKVCacheManager const> getKVCacheManager() const + { + return TrtGptModelInflightBatching::getKVCacheManager(); + } + + [[nodiscard]] SizeType32 getMaxAttentionWindow() const + { + return TrtGptModelInflightBatching::getMaxAttentionWindow(); + } +}; + +TEST_F(TrtGptModelTest, KVCacheReuseChunked) +{ + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setEnableChunkedContext(true); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT}); + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setEnableBlockReuse(true); + executorConfig.setKvCacheConfig(kvCacheConfig); + + mModelConfig.setMaxNumTokens(384); + + for (int const numBlocksExpectedReused : {1, 2}) + { + auto trtGptModelIfb = std::make_shared<TrtGptModelIfbHelper>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + auto const cacheManager = trtGptModelIfb->getKVCacheManager(); + auto const tokensPerBlock = cacheManager->getTokensPerBlock(); + constexpr int numPrefillBlocks = 2; + + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + constexpr int correlationId = 0; + constexpr int maxNewTokens = 4; + + auto tokens = std::make_shared<std::vector<int32_t>>(tokensPerBlock * numPrefillBlocks); + std::iota(std::begin(*tokens), std::end(*tokens), 1); + auto subTokens = std::make_shared<std::vector<int32_t>>( + tokens->begin(), tokens->begin() + numBlocksExpectedReused * tokensPerBlock); + // Add new token to "start" a new block. + subTokens->push_back(0); + { + auto llmRequest + = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + RequestList requests{llmRequest}; + forwardRequestsToCompletion(trtGptModelIfb, requests, 6); + EXPECT_EQ(llmRequest->isGenerationCompleteState(), true); + } + for (size_t i = 1; i <= 2; ++i) + { + auto llmRequest + = std::make_shared<LlmRequest>(correlationId, maxNewTokens, subTokens, inSamplingConfig, false); + RequestList req{llmRequest}; + forwardRequestsToCompletion(trtGptModelIfb, req, 5); + EXPECT_EQ(cacheManager->getBlockManager().getNumReusedBlocks(), i * numBlocksExpectedReused); + } + } +} + +TEST_F(TrtGptModelTest, PauseRequestStats) +{ + SamplingConfig inSamplingConfig; + inSamplingConfig.temperature = std::vector{2.0f}; + int correlationId = 0; + auto maxNewTokens = 3; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{1, 2, 3, 4}); + auto llmRequest = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false, + std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, + std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, + std::nullopt, std::nullopt, std::nullopt, std::nullopt, false, false, false, std::nullopt, std::nullopt, false, + std::nullopt, false, std::nullopt, false, std::nullopt, executor::Request::kDefaultPriority, std::nullopt, + std::nullopt, std::nullopt, LlmRequestType::LLMREQUEST_TYPE_CONTEXT_AND_GENERATION, std::nullopt, 1, + std::nullopt, std::nullopt, true /* returnPerfMetrics */); + + RequestList requestList{llmRequest}; + + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setSchedulerConfig(executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kMAX_UTILIZATION}); + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + // Generate one token for the requests in request_table + // We need to sync with decoder + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + + EXPECT_EQ(requestList.size(), 1); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_IN_PROGRESS); + EXPECT_EQ(requestList.front()->getNumTokens(0), 5); + EXPECT_EQ(requestList.front()->getMaxNumGeneratedTokens(), 1); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(1, 2, 3, 4, 2)); + + auto perfMetrics = requestList.front()->getPerfMetrics(); + auto zero = executor::RequestPerfMetrics::TimePoint{}; + + EXPECT_NE(perfMetrics.timingMetrics.arrivalTime, zero); + EXPECT_NE(perfMetrics.timingMetrics.firstScheduledTime, zero); + EXPECT_NE(perfMetrics.timingMetrics.firstTokenTime, zero); + EXPECT_EQ(perfMetrics.timingMetrics.lastTokenTime, zero); + EXPECT_EQ(perfMetrics.firstIter, 0); + EXPECT_EQ(perfMetrics.iter, 0); + EXPECT_EQ(perfMetrics.lastIter, std::nullopt); + + // Pause the request + trtGptModel->terminateRequest(llmRequest, true); + + // Resume work + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + + // Generate one more token + EXPECT_EQ(requestList.size(), 1); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_IN_PROGRESS); + EXPECT_EQ(requestList.front()->getNumTokens(0), 6); + EXPECT_EQ(requestList.front()->getMaxNumGeneratedTokens(), 1); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(1, 2, 3, 4, 2, 4)); + + auto newPerfMetrics = requestList.front()->getPerfMetrics(); + EXPECT_EQ(newPerfMetrics.firstIter, 0); + EXPECT_EQ(newPerfMetrics.iter, 1); + EXPECT_EQ(newPerfMetrics.lastIter, std::nullopt); + + // Check that firstScheduledTime and firstTokenTime are the same + EXPECT_EQ(perfMetrics.timingMetrics.firstScheduledTime, newPerfMetrics.timingMetrics.firstScheduledTime); + EXPECT_EQ(perfMetrics.timingMetrics.firstTokenTime, newPerfMetrics.timingMetrics.firstTokenTime); + + // Pause the request + trtGptModel->terminateRequest(llmRequest, true); + + // Resume work + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + + // Generate last token + EXPECT_EQ(requestList.size(), 1); + EXPECT_EQ(requestList.front()->getState(), LlmRequestState::kGENERATION_COMPLETE); + EXPECT_EQ(requestList.front()->getNumTokens(0), 7); + EXPECT_EQ(requestList.front()->getMaxNumGeneratedTokens(), 1); + EXPECT_THAT(requestList.front()->getTokens(0), ElementsAre(1, 2, 3, 4, 2, 4, 2)); + + auto endPerfMetrics = requestList.front()->getPerfMetrics(); + EXPECT_EQ(endPerfMetrics.firstIter, 0); + EXPECT_EQ(endPerfMetrics.iter, 2); + EXPECT_EQ(endPerfMetrics.lastIter, 2); + + // Check that firstScheduledTime and firstTokenTime are the same + EXPECT_EQ(perfMetrics.timingMetrics.firstScheduledTime, endPerfMetrics.timingMetrics.firstScheduledTime); + EXPECT_EQ(perfMetrics.timingMetrics.firstTokenTime, endPerfMetrics.timingMetrics.firstTokenTime); +} + +class TrtGptModelLogitsTest : public TrtGptModelTest +{ +protected: + TrtGptModelLogitsTest() + : TrtGptModelTest(GPT_MODEL_PATH / GetModelSpec().getModelPath() / "tp1-pp1-cp1-gpu") + { + } + + static ModelSpec& GetModelSpec() + { + static ModelSpec modelSpec{"input_tokens.npy", nvinfer1::DataType::kHALF}; + modelSpec.useGptAttentionPlugin().usePackedInput().setKVCacheType(KVCacheType::kPAGED).gatherLogits(); + return modelSpec; + } +}; + +TEST_F(TrtGptModelLogitsTest, ReturnContextLogitsWithChunkedContext) +{ + // General config + int correlationId = 0; + auto maxNewTokens = 4; + int const worldSize = 1; + auto const vocabSizePadded = mModelConfig.getVocabSizePadded(worldSize); + + SamplingConfig inSamplingConfig; + + // Different prompt length + for (int const promptLength : {10, 128, 200, 250, 256}) + { + RequestList finishList; + for (bool enableChunkedContext : {false, true}) + { + auto modelConfig = mModelConfig; + if (enableChunkedContext) + { + modelConfig.setMaxNumTokens(128); + } + + executor::ExecutorConfig executorConfig; + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(mBeamWidth); + executorConfig.setEnableChunkedContext(enableChunkedContext); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT}); + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setEnableBlockReuse(true); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto trtGptModelIfb = std::make_shared<TrtGptModelIfbHelper>( + mLogger, modelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + // Prepare input tokens + std::vector<int32_t> input_ids; + for (int i = 1; i <= promptLength; i++) + { + input_ids.push_back(i); + } + auto tokens = std::make_shared<std::vector<int32_t>>(input_ids); + + auto llmRequest + = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + TensorPtr contextLogitsHost = BufferManager::cpu( + ITensor::makeShape({llmRequest->mPromptLen, vocabSizePadded}), nvinfer1::DataType::kFLOAT); + + llmRequest->setContextLogitsHost(contextLogitsHost); + llmRequest->setReturnContextLogits(true); + + RequestList requestList{llmRequest}; + forwardRequestsToCompletion(trtGptModelIfb, requestList, 6); + + finishList.push_back(llmRequest); + } + EXPECT_EQ(finishList.size(), 2); + + float const* const disableChunkedContextLogits + = bufferCast<float>(*(finishList.front()->getContextLogitsHost())); + float const* const enableChunkedContextLogits = bufferCast<float>(*(finishList.back()->getContextLogitsHost())); + + for (int tokenIdx = 0; tokenIdx < promptLength; tokenIdx++) + { + for (int vocabIdx = 0; vocabIdx < vocabSizePadded; vocabIdx++) + { + size_t idx = tokenIdx * vocabSizePadded + vocabIdx; + EXPECT_NEAR(disableChunkedContextLogits[idx], enableChunkedContextLogits[idx], 1e-0) + << "tokenIdx=" << tokenIdx << " vocabIdx=" << vocabIdx; + } + } + finishList.clear(); + } +} + +class LlamaModelLADTest : public TrtGptModelTest +{ +protected: + LlamaModelLADTest() + : TrtGptModelTest(LLAMA_MODEL_PATH / GetModelSpec().getModelPath() / "tp1-pp1-cp1-gpu") + { + } + + static ModelSpec& GetModelSpec() + { + static ModelSpec modelSpec = ModelSpec{"input_tokens.npy", nvinfer1::DataType::kHALF} + .useGptAttentionPlugin() + .usePackedInput() + .setKVCacheType(KVCacheType::kPAGED) + .useLookaheadDecoding(); + return modelSpec; + } +}; + +TEST_F(LlamaModelLADTest, SeamlessLookaheadDecoding) +{ + GTEST_SKIP() << "Will enable this test when we have a force LAD support."; + SizeType32 requestId = 0; + for (bool const initLADConfig : {true, false}) + { + RequestList requestList{}; + for (SizeType32 i = 0; i < 8; ++i) + { + SamplingConfig inSamplingConfig; + int correlationId = requestId; + auto maxNewTokens = 8; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{1, 2, 3, 4}); + auto llmRequest + = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + requestList.emplace_back(std::move(llmRequest)); + requestId += 1; + } + + executor::ExecutorConfig executorConfig; + executorConfig.setEnableChunkedContext(false); + executorConfig.setEnableTrtOverlap(false); + executorConfig.setMaxBeamWidth(1); + executorConfig.setSchedulerConfig( + executor::SchedulerConfig{executor::CapacitySchedulerPolicy::kMAX_UTILIZATION}); + if (initLADConfig) + { + executor::DecodingConfig decodingConfig; + decodingConfig.setLookaheadDecodingConfig(executor::LookaheadDecodingConfig(5, 5, 5)); + executorConfig.setDecodingConfig(decodingConfig); + } + + auto trtGptModel = std::make_shared<TrtGptModelInflightBatching>( + mLogger, mModelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + + // Generate tokens for the requests in request_table + // We need to sync with decoder + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + EXPECT_EQ(trtGptModel->getSpeculativeDecodingMode().isLookaheadDecoding(), true); + + // Add new requests + for (SizeType32 i = 0; i < 4; ++i) + { + SamplingConfig inSamplingConfig; + int correlationId = requestId; + auto maxNewTokens = 8; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{1, 2, 3, 4}); + auto llmRequest + = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + requestList.emplace_back(std::move(llmRequest)); + requestId += 1; + } + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + EXPECT_EQ(trtGptModel->getSpeculativeDecodingMode().isLookaheadDecoding(), false); + + // Complete all of the requests + SizeType32 maxNumIterations = 8; + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + + // Run new requests with lookahead + requestList.clear(); + for (SizeType32 i = 0; i < 4; ++i) + { + SamplingConfig inSamplingConfig; + int correlationId = requestId; + auto maxNewTokens = 8; + auto tokens = std::make_shared<std::vector<int32_t>>(std::initializer_list<int32_t>{1, 2, 3, 4}); + auto llmRequest + = std::make_shared<LlmRequest>(correlationId, maxNewTokens, tokens, inSamplingConfig, false); + requestList.emplace_back(std::move(llmRequest)); + requestId += 1; + } + trtGptModel->forwardAsync(requestList); + trtGptModel->forwardSync(); + EXPECT_EQ(trtGptModel->getSpeculativeDecodingMode().isLookaheadDecoding(), true); + forwardRequestsToCompletion(trtGptModel, requestList, maxNumIterations); + requestList.clear(); + } +} + +TEST_F(TrtGptModelTest, ClampSeqLenToAttentionWindow) +{ + auto constexpr maxAttentionWindow = 65536; + auto constexpr maxSequenceLen = maxAttentionWindow + 1; + + executor::KvCacheConfig kvCacheConfig; + kvCacheConfig.setMaxAttentionWindowVec(std::vector<SizeType32>{maxAttentionWindow}); + kvCacheConfig.setFreeGpuMemoryFraction(0.0001); // minuscule amount of memory to force a clamp + + executor::ExecutorConfig executorConfig; + executorConfig.setKvCacheConfig(kvCacheConfig); + executorConfig.setMaxBeamWidth(mBeamWidth); + + auto modelConfig = mModelConfig; + modelConfig.setMaxSequenceLen(maxSequenceLen); + + auto trtGptModel = std::make_shared<TrtGptModelIfbHelper>( + mLogger, modelConfig, mWorldConfig, *mRawEngine, true, executorConfig, false); + EXPECT_LT(trtGptModel->getMaxAttentionWindow(), maxAttentionWindow); + EXPECT_EQ(trtGptModel->getMaxSequenceLen(), trtGptModel->getMaxAttentionWindow()); +} + +} // namespace tensorrt_llm::batch_manager diff --git a/cpp/tests/e2e_tests/executor/CMakeLists.txt b/cpp/tests/e2e_tests/executor/CMakeLists.txt new file mode 100644 index 000000000000..4813c92584fc --- /dev/null +++ b/cpp/tests/e2e_tests/executor/CMakeLists.txt @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +add_gtest(executorMockTest executorMockTest.cpp) +add_gtest(executorTest executorTest.cpp) +target_link_libraries(executorTest PRIVATE testingUtils) +add_gtest(encDecTest encDecTest.cpp) +target_link_libraries(encDecTest PRIVATE testingUtils) +add_gtest(disaggExecutorTest disaggExecutorTest.cpp) +target_link_libraries(disaggExecutorTest PRIVATE testingUtils) diff --git a/cpp/tests/e2e_tests/executor/disaggExecutor.h b/cpp/tests/e2e_tests/executor/disaggExecutor.h new file mode 100644 index 000000000000..6b3a529ca16e --- /dev/null +++ b/cpp/tests/e2e_tests/executor/disaggExecutor.h @@ -0,0 +1,840 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/stringUtils.h" +#include "tensorrt_llm/common/utils.h" +#include "tensorrt_llm/executor/dataTransceiverState.h" +#include "tensorrt_llm/executor/disaggServerUtil.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/requestWithId.h" +#include "tensorrt_llm/executor/serializeUtils.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include <gmock/gmock.h> +#include <gtest/gtest.h> +#include <nlohmann/json.hpp> + +#include <algorithm> +#include <chrono> +#include <cstddef> +#include <cstdint> +#include <memory> +#include <mutex> +#include <queue> +#include <string> +#include <thread> +#include <unordered_map> +#include <vector> + +using namespace tensorrt_llm::executor; +using namespace tensorrt_llm::executor::disagg_executor; +namespace su = tensorrt_llm::executor::serialize_utils; + +namespace tensorrt_llm::testing::disaggexecutor +{ + +constexpr int32_t kM_INSTANCE_ID_TAG{12024}; +constexpr int32_t kM_CONTROLLER_ID_TAG{22024}; +constexpr int32_t kM_INSTANCE_DATA_TAG{32024}; +constexpr int32_t kM_CONTROLLER_DATA_TAG{42024}; + +enum class MessageID : uint64_t +{ + PENDING_CONTEXT_REQUEST = 1, + PENDING_GENERATION_REQUEST = 2, + PENDING_FULL_REQUEST = 3, + CONTEXT_RESPONSE = 4, + GENERATION_RESPONSE = 5, + + TERMINATION = 6, +}; + +enum DisaggRole : uint32_t +{ + DISAGG_CONTEXT = 1, + DISAGG_GENERATION = 2, + DISAGG_MIXED = DISAGG_CONTEXT | DISAGG_GENERATION, + DISAGG_LEADER = 4, + DISAGG_CONTROLLER = 8, +}; + +struct RequestsData +{ + std::vector<RequestWithId> requests; +}; + +static std::vector<char> serializeResponseWithIds(std::vector<ResponseWithId> const& responseWithIds) +{ + size_t totalSize = 0; + totalSize += sizeof(size_t); + for (auto const& responseWithId : responseWithIds) + { + totalSize += su::serializedSize(responseWithId.gid); + totalSize += su::serializedSize(responseWithId.response); + } + + std::vector<char> buffer(totalSize); + std::stringbuf strbuf{std::ios_base::out | std::ios_base::in}; + strbuf.pubsetbuf(buffer.data(), static_cast<std::streamsize>(buffer.size())); + std::ostream ostream{&strbuf}; + + su::serialize(responseWithIds.size(), ostream); + for (auto const& responseWithId : responseWithIds) + { + su::serialize(responseWithId.gid, ostream); + su::serialize(responseWithId.response, ostream); + } + return buffer; +} + +static std::vector<ResponseWithId> deserializeResponseWithIds(std::vector<char>& buffer) +{ + std::vector<ResponseWithId> responseWithIds; + su::VectorWrapBuf<char> strbuf{buffer}; + std::istream istream{&strbuf}; + auto numReq = su::deserialize<std::int64_t>(istream); + for (int64_t req = 0; req < numReq; ++req) + { + auto const id = su::deserialize<std::uint64_t>(istream); + responseWithIds.emplace_back(ResponseWithId{Serialization::deserializeResponse(istream), id}); + } + return responseWithIds; +} + +struct ResponsesData +{ + std::vector<ResponseWithId> response; +}; + +using MessageData = std::variant<RequestsData, ResponsesData>; + +struct Message +{ + MessageID id; + MessageData data; +}; + +class MessageQueue +{ +public: + void push(Message&& message) + { + std::lock_guard<std::mutex> lock(mMutex); + mQueue.push(std::move(message)); + mCv.notify_one(); + } + + Message pop() + { + std::unique_lock<std::mutex> lock(mMutex); + mCv.wait(lock, [this] { return !mQueue.empty(); }); + Message message = std::move(mQueue.front()); + mQueue.pop(); + return message; + } + +private: + std::queue<Message> mQueue; + std::mutex mMutex; + std::condition_variable mCv; +}; + +class DisaggExecutorLeader +{ +public: + DisaggExecutorLeader(std::filesystem::path const& modelPath, ModelType modelType, + ExecutorConfig const& executorConfig, bool isController, bool isContext, bool isGeneration, int numRequests, + std::vector<int>& participatIds, std::vector<int> const& participantDeviceIdsThisInstance, int worldRank) + : mNumRequests(numRequests) + , mWorldRanksInstances(participatIds) + , mDeviceIdsThisInstance(participantDeviceIdsThisInstance) + , mWorldRank(worldRank) + , mShutdown(false) + , mWorldComm(tensorrt_llm::mpi::MpiComm::world()) + + { + +#if ENABLE_MULTI_DEVICE + + auto world_size = mWorldComm.getSize(); + mRolesPerRank.resize(world_size); + + if (isContext) + { + mRole |= DisaggRole::DISAGG_CONTEXT; + } + if (isGeneration) + { + mRole |= DisaggRole::DISAGG_GENERATION; + } + + if (!mWorldRanksInstances.empty() && mWorldRank == mWorldRanksInstances.front()) + { + mRole |= DisaggRole::DISAGG_LEADER; + } + + if (isController) + { + mRole |= DisaggRole::DISAGG_CONTROLLER; + } + + bool needExecutor = (std::find(mWorldRanksInstances.begin(), mWorldRanksInstances.end(), worldRank) + != mWorldRanksInstances.end()); + if (needExecutor) + { + ExecutorConfig executorConfigC = executorConfig; + + auto parallelConfig = executorConfigC.getParallelConfig().value_or(ParallelConfig{}); + std::vector<int> participantIds = mWorldRanksInstances; + + parallelConfig.setParticipantIds(participantIds); + TLLM_CHECK(parallelConfig.getCommunicationMode() == tensorrt_llm::executor::CommunicationMode::kLEADER); + parallelConfig.setCommunicationType(tensorrt_llm::executor::CommunicationType::kMPI); + parallelConfig.setDeviceIds(mDeviceIdsThisInstance); + executorConfigC.setParallelConfig(parallelConfig); + + mExecutor = std::make_unique<Executor>(modelPath, modelType, executorConfigC); + } + + TLLM_CHECK(mWorldRanksInstances.size() == mDeviceIdsThisInstance.size()); + + mWorldComm.allgather(&mRole, mRolesPerRank.data(), 1, tensorrt_llm::mpi::MpiType::kUINT32); + + generateRoles(); + + if (isController) + { + mControllerSendThread = std::thread(&DisaggExecutorLeader::ControllerSendThread, this); + mControllerRecvThread = std::thread(&DisaggExecutorLeader::ControllerRecvThread, this); + } + if (isLeaderInstance()) + { + mInstanceRecvThread = std::thread(&DisaggExecutorLeader::InstanceLeaderRecvThread, this); + mInstanceSendThread = std::thread(&DisaggExecutorLeader::InstanceLeaderSendThread, this); + mInstanceLoopThread = std::thread(&DisaggExecutorLeader::InstanceLeaderLoopThread, this); + } +#else + TLLM_THROW("DisaggExecutor only support being compiled with ENABLE_MULTI_DEVICE"); + +#endif + } + + bool isControllerRank() const + { + return mRole & DISAGG_CONTROLLER; + } + + bool isContextRank() const + { + return mRole & DISAGG_CONTEXT; + } + + bool isGenerationRank() const + { + return mRole & DISAGG_GENERATION; + } + + bool isLeaderInstance() const + { + return mRole & DISAGG_LEADER; + } + + std::vector<IdType> enqueueRequests(std::vector<Request> const& llmRequests) + + { + if (!isControllerRank()) + { + return {}; + } + + std::vector<RequestWithId> requestWithIds; + std::vector<RequestWithId> requestWithIdsFull; // full request, not disaggregated + std::vector<IdType> reqIds; + for (auto const& req : llmRequests) + { + IdType id = generatedControlId(); + reqIds.push_back(id); + + RequestWithId reqWithId{req, id}; + if (req.getRequestType() == RequestType::REQUEST_TYPE_CONTEXT_ONLY) + { + requestWithIds.push_back(std::move(reqWithId)); + } + else + { + TLLM_CHECK(req.getRequestType() == RequestType::REQUEST_TYPE_CONTEXT_AND_GENERATION); + requestWithIdsFull.push_back(std::move(reqWithId)); + } + + mRequestMap.insert(std::make_pair(id, req)); + } + + if (!requestWithIds.empty()) + { + Message message{MessageID::PENDING_CONTEXT_REQUEST, MessageData{RequestsData{requestWithIds}}}; + mControllerSendQueue.push(std::move(message)); + } + if (!requestWithIdsFull.empty()) + { + Message message{MessageID::PENDING_FULL_REQUEST, MessageData{RequestsData{requestWithIdsFull}}}; + mControllerSendQueue.push(std::move(message)); + } + + return reqIds; + } + + std::vector<Response> awaitResponses(std::optional<std::chrono::milliseconds> const& timeout) + { + // wait for responseQueue , modify reqid- + std::vector<Response> responses; + std::unique_lock<std::mutex> lck(mResponsesMtx); + auto pred = [&mShutdown = mShutdown, &resp = this->mResponses]() -> bool { return !resp.empty() || mShutdown; }; + auto storeResponses = [this, &resp = this->mResponses, &responses]() + { + for (auto it = resp.cbegin(); it != resp.cend();) + { + responses.insert(responses.end(), it->second.begin(), it->second.end()); + resp.erase(it++); + } + }; + + if (timeout) + { + if (mResponsesCv.wait_for(lck, timeout.value(), pred)) + { + storeResponses(); + } + } + else + { + mResponsesCv.wait(lck, pred); + storeResponses(); + } + return responses; + } + + std::deque<RequestStatsPerIteration> getLatestRequestStats() + { + if (mExecutor && mExecutor->canEnqueueRequests()) + { + return mExecutor->getLatestRequestStats(); + } + return {}; + } + + void shutDown() + { + if (mShutdown) + { + return; + } + + if (isControllerRank()) + { + std::call_once(mHasSendTerminFlag, + [&]() + { + MessageID terminationMessage = MessageID::TERMINATION; + std::vector<bool> isSend(mWorldComm.getSize(), false); + for (auto&& leaderRanks : {mContextLeaderRanks, mGenerationLeaderRanks}) + { + for (auto&& leaderRank : leaderRanks) + { + if (isSend[leaderRank]) + { + continue; + } + mWorldComm.sendRawTag(&terminationMessage, 1, tensorrt_llm::mpi::MpiType::kUINT64, + leaderRank, kM_CONTROLLER_ID_TAG); + isSend[leaderRank] = true; + } + } + + mWorldComm.sendRawTag(&terminationMessage, 1, tensorrt_llm::mpi::MpiType::kUINT64, mControllerRank, + kM_INSTANCE_ID_TAG); + }); + // end recv thread; + } + mShutdown = true; + + // end send thread + if (isControllerRank()) + { + mControllerSendQueue.push({MessageID::TERMINATION, {}}); + } + mInstanceSendQueue.push({MessageID::TERMINATION, {}}); + } + + ~DisaggExecutorLeader() + { + + if (isControllerRank()) + { + shutDown(); + } + + if (isLeaderInstance()) + { + if (mInstanceSendThread.joinable()) + { + mInstanceSendThread.join(); + } + if (mInstanceRecvThread.joinable()) + { + mInstanceRecvThread.join(); + } + if (mInstanceLoopThread.joinable()) + { + mInstanceLoopThread.join(); + } + } + + if (isControllerRank()) + { + if (mControllerSendThread.joinable()) + { + mControllerSendThread.join(); + } + if (mControllerRecvThread.joinable()) + { + mControllerRecvThread.join(); + } + } + + if (!isControllerRank()) + { + mExecutor->shutdown(); + } + if (isControllerRank() && isLeaderInstance()) + { + mExecutor->shutdown(); + } + + shutDown(); + } + +private: + tensorrt_llm::mpi::MpiComm const& mWorldComm; + std::unique_ptr<Executor> mExecutor; + std::thread mInstanceSendThread; + std::thread mInstanceRecvThread; + std::thread mInstanceLoopThread; + std::thread mControllerSendThread; + std::thread mControllerRecvThread; + int mNumRequests; + std::map<std::uint64_t, Request> mRequestMap; + std::map<IdType, DataTransceiverState> mGenIdToContextPhase; + std::unordered_map<IdType, IdType> mInstanceIdToGlobalId; + std::mutex mIdToGlbalMutex; + + std::vector<int> mWorldRanksInstances; + + int mWorldRank; + int mControllerRank = 0; + uint32_t mRole = 0; + std::vector<uint32_t> mRolesPerRank; + std::vector<int> mContextLeaderRanks; + std::vector<int> mGenerationLeaderRanks; + + IdType mLastId = 1; + MessageQueue mControllerSendQueue; + MessageQueue mInstanceSendQueue; + + std::atomic<bool> mShutdown; + + // Ready responses + std::unordered_map<IdType, std::vector<Response>> mResponses; + mutable std::mutex mResponsesMtx; + std::condition_variable mResponsesCv; + + std::vector<int> mDeviceIdsThisInstance; + std::once_flag mHasSendTerminFlag; + + void appendNewResponses(std::vector<ResponseWithId>& newResponses) + { + { + std::scoped_lock<std::mutex> lck(mResponsesMtx); + for (auto& responseWithId : newResponses) + { + // global id to Result + responseWithId.response = Response(responseWithId.gid, responseWithId.response.getResult()); + + mResponses[responseWithId.gid].emplace_back(responseWithId.response); + } + } + mResponsesCv.notify_all(); + } + + void generateRoles() + { + int contextNum = 0; + int genrationNum = 0; + int controllerNum = 0; + for (int rank = 0; rank < mRolesPerRank.size(); rank++) + { + uint32_t role = mRolesPerRank[rank]; + if (role & DISAGG_LEADER) + { + if (role & DISAGG_CONTEXT) + { + contextNum++; + mContextLeaderRanks.push_back(rank); + } + if (role & DISAGG_GENERATION) + { + genrationNum++; + mGenerationLeaderRanks.push_back(rank); + } + } + if (role & DISAGG_CONTROLLER) + { + controllerNum++; + mControllerRank = rank; + } + } + TLLM_CHECK_WITH_INFO(controllerNum == 1, "only one rank is controller but get %d controllerNum", controllerNum); + TLLM_LOG_INFO("leader ctx: %s, gen: %s", common::vec2str(mContextLeaderRanks).c_str(), + common::vec2str(mGenerationLeaderRanks).c_str()); + } + + IdType generatedControlId() + { + return (mLastId++ % UINT64_MAX); + } + + int selectContextLeaderRank() + { + static int leaderRank = 0; + leaderRank = (leaderRank + 1) % mContextLeaderRanks.size(); + return mContextLeaderRanks[leaderRank]; + } + + int selectGenerationLeaderRank() + { + + // TODO: for same reqId , need select specific generationLeader + static int leaderRank = 0; + leaderRank = (leaderRank + 1) % mGenerationLeaderRanks.size(); + return mGenerationLeaderRanks[leaderRank]; + } + + void ControllerSendThread() + { + // send request to context reqid + // and send context pahse to generation + + TLLM_CUDA_CHECK( + cudaSetDevice(mDeviceIdsThisInstance.at(COMM_SESSION.getRank() % (mDeviceIdsThisInstance.size())))); + tensorrt_llm::common::setThreadName("ControllerSendThread"); + + while (!mShutdown) + { + auto message = mControllerSendQueue.pop(); + if (message.id == MessageID::TERMINATION) + { + + TLLM_LOG_DEBUG("controller get termination message in sendQueue"); + break; + } + if (message.id == MessageID::PENDING_CONTEXT_REQUEST) + { + + auto& reqWithIds = std::get<RequestsData>(message.data); + auto packed = RequestWithId::serializeReqWithIds(reqWithIds.requests); + int contextRank = selectContextLeaderRank(); + + mWorldComm.sendRawTag( + &message.id, 1, tensorrt_llm::mpi::MpiType::kUINT64, contextRank, kM_CONTROLLER_ID_TAG); + + mWorldComm.sendRawTag(packed.data(), packed.size(), tensorrt_llm::mpi::MpiType::kCHAR, contextRank, + kM_CONTROLLER_DATA_TAG); + } + else if (message.id == MessageID::PENDING_GENERATION_REQUEST + || message.id == MessageID::PENDING_FULL_REQUEST) + { + + auto& reqWithIds = std::get<RequestsData>(message.data); + auto packed = RequestWithId::serializeReqWithIds(reqWithIds.requests); + int generationRank = selectGenerationLeaderRank(); + + mWorldComm.sendRawTag( + &message.id, 1, tensorrt_llm::mpi::MpiType::kUINT64, generationRank, kM_CONTROLLER_ID_TAG); + + mWorldComm.sendRawTag(packed.data(), packed.size(), tensorrt_llm::mpi::MpiType::kCHAR, generationRank, + kM_CONTROLLER_DATA_TAG); + } + else + { + TLLM_THROW("rank:%d, size:%d controller send Invalid message id:%ld", mWorldComm.getRank(), + mWorldComm.getSize(), static_cast<uint64_t>(message.id)); + } + } + } + + void ControllerRecvThread() + { +#if ENABLE_MULTI_DEVICE + tensorrt_llm::common::setThreadName("ControllerRecvThread"); + + // recv response from context and push to sendQueue + // recv response from generation and push to responseQueue and notify awaitResponse + TLLM_CUDA_CHECK( + cudaSetDevice(mDeviceIdsThisInstance.at(COMM_SESSION.getRank() % (mDeviceIdsThisInstance.size())))); + + while (!mShutdown) + { + + MPI_Message msg = nullptr; + MPI_Status status; + + mWorldComm.mprobeRawTag(MPI_ANY_SOURCE, kM_INSTANCE_ID_TAG, &msg, &status); + + auto sourceRank{status.MPI_SOURCE}; + int32_t count = 0; + MPICHECK(MPI_Get_count(&status, MPI_UINT64_T, &count)); + TLLM_CHECK(count == 1); + + MessageID messageId; + MPICHECK(MPI_Mrecv(&messageId, count, MPI_UINT64_T, &msg, &status)); + + if (messageId == MessageID::TERMINATION) + { + TLLM_LOG_DEBUG("controller received termination message***************\n"); + break; + } + if (messageId == MessageID::CONTEXT_RESPONSE) + { + mWorldComm.mprobeRawTag(sourceRank, kM_INSTANCE_DATA_TAG, &msg, &status); + MPICHECK(MPI_Get_count(&status, MPI_CHAR, &count)); + std::vector<char> buffer(count); + MPICHECK(MPI_Mrecv(buffer.data(), count, MPI_CHAR, &msg, &status)); + auto responseWithIds = deserializeResponseWithIds(buffer); + // enqueueTo sendQueue like enqueuRequest. . modify requestType and set ContextPhaseParams + // and push to sendQueue. + std::vector<RequestWithId> requestWithIds; + for (auto&& responseWithId : responseWithIds) + { + auto reqId = responseWithId.gid; + auto& request = mRequestMap.at(reqId); + + request.setRequestType(RequestType::REQUEST_TYPE_GENERATION_ONLY); + request.setContextPhaseParams(responseWithId.response.getResult().contextPhaseParams.value()); + requestWithIds.push_back(RequestWithId{request, reqId}); + } + mControllerSendQueue.push({MessageID::PENDING_GENERATION_REQUEST, RequestsData{requestWithIds}}); + } + + else if (messageId == MessageID::GENERATION_RESPONSE) + { + + mWorldComm.mprobeRawTag(sourceRank, kM_INSTANCE_DATA_TAG, &msg, &status); + MPICHECK(MPI_Get_count(&status, MPI_CHAR, &count)); + std::vector<char> buffer(count); + MPICHECK(MPI_Mrecv(buffer.data(), count, MPI_CHAR, &msg, &status)); + + auto responseWithIds = deserializeResponseWithIds(buffer); + appendNewResponses(responseWithIds); + } + else + { + TLLM_THROW("rank:%d, size:%d controller recv Invalid message id:%ld", mWorldComm.getRank(), + mWorldComm.getSize(), static_cast<uint64_t>(messageId)); + } + } +#endif + } + + void InstanceLeaderSendThread() + { + tensorrt_llm::common::setThreadName("InstanceLeaderSendThread"); + + TLLM_CUDA_CHECK( + cudaSetDevice(mDeviceIdsThisInstance.at(COMM_SESSION.getRank() % (mDeviceIdsThisInstance.size())))); + + // pop senQueue and send response to controller + + while (!mShutdown) + { + auto message = mInstanceSendQueue.pop(); + if (message.id == MessageID::CONTEXT_RESPONSE || message.id == MessageID::GENERATION_RESPONSE) + { + auto& responseWithIds = std::get<ResponsesData>(message.data); + auto packed = serializeResponseWithIds(responseWithIds.response); + + mWorldComm.sendRawTag( + &message.id, 1, tensorrt_llm::mpi::MpiType::kUINT64, mControllerRank, kM_INSTANCE_ID_TAG); + mWorldComm.sendRawTag(packed.data(), packed.size(), tensorrt_llm::mpi::MpiType::kCHAR, mControllerRank, + kM_INSTANCE_DATA_TAG); + } + else if (message.id == MessageID::TERMINATION) + { + // break; no send + TLLM_LOG_DEBUG( + "ranK:%d ,size:%d ,isContext:%d... Context or Generation leader get termination message in " + "sendQueue***************\n", + mWorldComm.getRank(), mWorldComm.getSize(), int(isContextRank())); + break; + } + else + { + TLLM_THROW("rank:%d, size:%d InstanceLeaderSendThread send Invalid message id:%ld", + mWorldComm.getRank(), mWorldComm.getSize(), static_cast<uint64_t>(message.id)); + } + } + } + + void InstanceLeaderRecvThread() + { + +#if ENABLE_MULTI_DEVICE + tensorrt_llm::common::setThreadName("InstanceLeaderRecvThread"); + + TLLM_CUDA_CHECK( + cudaSetDevice(mDeviceIdsThisInstance.at(COMM_SESSION.getRank() % (mDeviceIdsThisInstance.size())))); + + // recv request from controller and enqueRequest to executor + while (!mShutdown) + { + MPI_Message msg; + MPI_Status status; + auto sourceRank{mControllerRank}; + mWorldComm.mprobeRawTag(sourceRank, kM_CONTROLLER_ID_TAG, &msg, &status); + + int32_t count; + MPICHECK(MPI_Get_count(&status, MPI_UINT64_T, &count)); + TLLM_CHECK(count == 1); + + MessageID messageId; + MPICHECK(MPI_Mrecv(&messageId, count, MPI_UINT64_T, &msg, &status)); + + if (messageId == MessageID::TERMINATION) + { + TLLM_LOG_DEBUG( + "ranK:%d ,size:%d ,isContext:%d ... Context or Generation leader recv termination message in " + "InstanceLeaderRecvThread***************\n", + mWorldComm.getRank(), mWorldComm.getSize(), int(isContextRank())); + shutDown(); + break; + } + if (messageId == MessageID::PENDING_CONTEXT_REQUEST || messageId == MessageID::PENDING_GENERATION_REQUEST + || messageId == MessageID::PENDING_FULL_REQUEST) + { + mWorldComm.mprobeRawTag(sourceRank, kM_CONTROLLER_DATA_TAG, &msg, &status); + MPICHECK(MPI_Get_count(&status, MPI_CHAR, &count)); + std::vector<char> buffer(count); + MPICHECK(MPI_Mrecv(buffer.data(), count, MPI_CHAR, &msg, &status)); + auto requestWithIds = RequestWithId::deserializeReqWithIds(buffer); + for (auto&& requestWithId : requestWithIds) + { + + auto globalReqId = requestWithId.id; + if (isContextRank() && messageId == MessageID::PENDING_CONTEXT_REQUEST) + { + TLLM_CHECK(requestWithId.req.getRequestType() == RequestType::REQUEST_TYPE_CONTEXT_ONLY); + } + else if (isGenerationRank() + && (messageId == MessageID::PENDING_GENERATION_REQUEST + || messageId == MessageID::PENDING_FULL_REQUEST)) + { + if (messageId == MessageID::PENDING_GENERATION_REQUEST) + { + TLLM_CHECK(requestWithId.req.getRequestType() == RequestType::REQUEST_TYPE_GENERATION_ONLY); + } + else // PENDING_FULL_REQUEST + { + TLLM_CHECK( + requestWithId.req.getRequestType() == RequestType::REQUEST_TYPE_CONTEXT_AND_GENERATION); + } + } + else + { + TLLM_THROW("rank:%d, size:%d InstanceLeaderRecvThread recv Invalid message id:%ld", + mWorldComm.getRank(), mWorldComm.getSize(), static_cast<uint64_t>(messageId)); + } + auto reqId = mExecutor->enqueueRequest(requestWithId.req); + { + std::scoped_lock<std::mutex> lock{mIdToGlbalMutex}; + mInstanceIdToGlobalId[reqId] = globalReqId; + } + } + } + else + { + TLLM_THROW("rank:%d, size:%d InstanceLeaderRecvThread send Invalid message id:%ld", + mWorldComm.getRank(), mWorldComm.getSize(), static_cast<uint64_t>(messageId)); + } + } +#endif + } + + void InstanceLeaderLoopThread() + { + + tensorrt_llm::common::setThreadName("InstanceLeaderLoopThread"); + + TLLM_CUDA_CHECK( + cudaSetDevice(mDeviceIdsThisInstance.at(COMM_SESSION.getRank() % (mDeviceIdsThisInstance.size())))); + + // loop awaitResponse and enqueue into sendQueue + while (!mShutdown) + { + std::chrono::milliseconds waitTime(1); + + auto responses = mExecutor->awaitResponses(waitTime); + if (responses.empty()) + { + continue; + } + std::vector<ResponseWithId> responseWithIdsContext; + std::vector<ResponseWithId> responseWithIdsGeneration; + for (auto&& response : responses) + { + auto reqId = response.getRequestId(); + IdType globalId{0}; + { + std::scoped_lock<std::mutex> lock{mIdToGlbalMutex}; + globalId = mInstanceIdToGlobalId[reqId]; + } + TLLM_CHECK(globalId != 0); + auto const& result = response.getResult(); + if (result.contextPhaseParams.has_value()) + { + responseWithIdsContext.emplace_back(response, globalId); + } + else + { + responseWithIdsGeneration.emplace_back(response, globalId); + } + } + + if (isContextRank()) + { + mInstanceSendQueue.push({MessageID::CONTEXT_RESPONSE, ResponsesData{responseWithIdsContext}}); + } + if (isGenerationRank()) + { + mInstanceSendQueue.push({MessageID::GENERATION_RESPONSE, ResponsesData{responseWithIdsGeneration}}); + } + } + } +}; +} // namespace tensorrt_llm::testing::disaggexecutor diff --git a/cpp/tests/e2e_tests/executor/disaggExecutorTest.cpp b/cpp/tests/e2e_tests/executor/disaggExecutorTest.cpp new file mode 100644 index 000000000000..0eb05d2cc807 --- /dev/null +++ b/cpp/tests/e2e_tests/executor/disaggExecutorTest.cpp @@ -0,0 +1,1437 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "disaggExecutor.h" +#include "executorTest.h" +#include "tensorrt_llm/common/envUtils.h" +#include "tensorrt_llm/runtime/utils/numpyUtils.h" +#include "tests/utils/common.h" + +#include <cstddef> +#include <unordered_set> + +namespace tr = tensorrt_llm::runtime; + +using namespace tensorrt_llm::testing; + +namespace +{ +auto constexpr LLAMA_INPUT_FILE = "input_tokens_llama.npy"; +auto constexpr LLAMA_VOCAB_SIZE_PADDED = 128256; +auto constexpr LLAMA_END_ID = 128001; +auto constexpr LLAMA_PAD_ID = 128001; + +using CondDisaggParamsType = std::tuple<std::string>; // modelName + +enum class InstanceRole : int +{ + kCONTEXT = 1, + kGENERATION = 0, + kMIXED = 2 +}; + +using DisaggParamsType = std::tuple< // + int, // processNum + std::vector<std::string>, // modelNames + std::vector<std::vector<int>>, // participantIdsEachInstance + std::vector<std::vector<int>>, // participantDeviceIdsEachInstance + std::vector<InstanceRole>, // instanceRoles + int // controllerRank + >; + +std::string convertToString(std::vector<std::vector<int>> const& vec) +{ + std::ostringstream oss; + oss << "XX"; + + for (size_t i = 0; i < vec.size(); ++i) + { + for (size_t j = 0; j < vec[i].size(); ++j) + { + oss << vec[i][j]; + if (j < vec[i].size() - 1) + { + oss << "_"; + } + } + if (i < vec.size() - 1) + { + oss << "X_X"; + } + } + + oss << "XX"; + return oss.str(); +}; + +std::string convertToString(std::vector<InstanceRole> const& vec) +{ + std::ostringstream oss; + oss << "XX"; + + for (size_t j = 0; j < vec.size(); ++j) + { + oss << static_cast<int>(vec[j]); + if (j < vec.size() - 1) + { + oss << "_"; + } + } + + oss << "XX"; + return oss.str(); +}; + +std::string generateTestNameDisaggParams(testing::TestParamInfo<DisaggParamsType> const& info) +{ + auto const processNum = std::get<0>(info.param); + auto const modelNames = std::get<1>(info.param); + auto const participantIdsEachInstance = std::get<2>(info.param); // std::vector<std::vector<int>> + auto const participantDeviceIdsEachInstance = std::get<3>(info.param); // std::vector<std::vector<int>>; + auto const instanceRoles = std::get<4>(info.param); // std::vector<int> ; //1 is context , 0 is generation + auto const controllerRank = std::get<5>(info.param); + + std::string name = "DisaggExecutorTest_"; + + name.append("ProcessNum_" + std::to_string(processNum)); + // name.append("_contextModel_" + contextModel + "_genModel_" + genModel); + name.append("_modelNames_"); + for (auto&& modelName : modelNames) + { + name.append(modelName).append("_"); + } + + name.append("_controllerRank_" + std::to_string(controllerRank)); + + name.append("_ranks_").append(convertToString(participantIdsEachInstance)); + name.append("_devices_").append(convertToString(participantDeviceIdsEachInstance)); + name.append("_roles_").append(convertToString(instanceRoles)); + name.append("_controllerRank_" + std::to_string(controllerRank)); + + return name; +} + +std::string generateTestNameCondDisaggParams(testing::TestParamInfo<CondDisaggParamsType> const& info) +{ + auto const modelName = std::get<0>(info.param); + return "Model_" + modelName; +} + +class DisaggParamsTest : public GptExecutorTest, public ::testing::WithParamInterface<DisaggParamsType> +{ +}; + +class DisaggOrchestratorParamsTest : public GptExecutorTest, public ::testing::WithParamInterface<DisaggParamsType> +{ +}; + +class ConditionalDisaggParamsTest : public GptExecutorTest, public ::testing::WithParamInterface<CondDisaggParamsType> +{ +}; + +void verifyGenerateDistStats(std::deque<RequestStatsPerIteration> const& iterationStats) +{ + for (auto const& iteration : iterationStats) + { + for (auto const& requestStats : iteration.requestStats) + { + // exclude context only requests for mixed server + if (requestStats.stage == RequestStage::kGENERATION_COMPLETE && requestStats.numGeneratedTokens > 1) + { + EXPECT_TRUE(requestStats.disServingStats.has_value()); + EXPECT_GT(requestStats.disServingStats.value().kvCacheTransferMS, 0.0); + } + if (requestStats.stage != RequestStage::kQUEUED) + { + EXPECT_TRUE(requestStats.disServingStats.has_value()); + } + else + { + EXPECT_FALSE(requestStats.disServingStats.has_value()); + } + } + } +} +} // namespace + +void runDisaggTest(tensorrt_llm::testing::disaggexecutor::DisaggExecutorLeader& executor, + tensorrt_llm::runtime::BufferManager& manager, ITensor const& givenInput, ModelIds const& modelIds, + FlakyTestInfo const& flakyTestInfo, bool streaming, SizeType32 const vocabSizePadded, BeamResult const& beamResult, + OutputConfig const& outConfig, bool isSpeculativeDecoding, int maxWaitMs, BatchingType batchingType, + bool returnAllGeneratedTokens) +{ + + auto& comm = tensorrt_llm::mpi::MpiComm::world(); + auto const worldRank = comm.getRank(); + auto const worldSize = comm.getSize(); + auto const beamWidth = beamResult.beamWidth; + + std::unordered_map<IdType, SizeType32> reqIdToBatchId; + std::unordered_map<SizeType32, std::vector<BeamTokens>> tokens; + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(givenInput, modelIds.padId); + auto const* const givenInputData = tr::bufferCast<TokenIdType const>(givenInput); + + auto const& inputShape = givenInput.getShape(); + ASSERT_EQ(inputShape.nbDims, 2); + ASSERT_GT(inputShape.d[0], 0); + + // Load expected outputs for each beam width value + auto testData = TestData::loadTestData(beamResult, givenInput, beamWidth, manager, outConfig, modelIds); + auto const maxSeqLen = testData.maxSeqLen; + + // Load expected outputs and inputs + SizeType32 numRequests = static_cast<SizeType32>(givenInputLengths.size()); + SizeType32 maxRequests = numRequests; + std::vector<Request> requests; + std::vector<SizeType32> reqMaxNewTokens; + SizeType32 const numReturnSequences = 1; + + for (SizeType32 req = 0; req < maxRequests; ++req) + { + SizeType32 inputLen = givenInputLengths.at(req); + auto maxNewTokens = maxSeqLen - maxInputLength; + reqMaxNewTokens.push_back(maxNewTokens); + SizeType32 endId = -1; + auto const* const seqBegin = givenInputData + req * maxInputLength; + VecTokens tokens(seqBegin, seqBegin + inputLen); + auto samplingConfig = tensorrt_llm::executor::SamplingConfig(beamWidth); + samplingConfig.setNumReturnSequences(numReturnSequences); + auto request = Request( + VecTokens(seqBegin, seqBegin + inputLen), maxNewTokens, streaming, samplingConfig, outConfig, endId); + request.setReturnAllGeneratedTokens(returnAllGeneratedTokens); + request.setRequestType(RequestType::REQUEST_TYPE_CONTEXT_ONLY); + requests.emplace_back(std::move(request)); + } + + if (executor.isControllerRank()) + { + std::vector<IdType> reqIds; + + for (int i = 0; i < requests.size(); ++i) + { + std::vector<BeamTokens> resultTokens; + resultTokens.reserve(numReturnSequences); + for (SizeType32 seqIdx = 0; seqIdx < numReturnSequences; ++seqIdx) + { + resultTokens.emplace_back(beamWidth); + } + auto retReqId = executor.enqueueRequests({requests[i]}); + reqIds.push_back(retReqId.front()); + tokens[i] = std::move(resultTokens); + reqIdToBatchId[retReqId.front()] = i; + } + + // Get the new tokens for each requests + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + while (numFinished < maxRequests && iter < maxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + auto batchId = reqIdToBatchId.at(response.getRequestId()); + auto seqIdx = result.sequenceIndex; + + auto& contextLogits = result.contextLogits; + auto& genLogits = result.generationLogits; + auto& outputTokenIds = result.outputTokenIds; + + EXPECT_EQ(result.finishReasons.size(), beamWidth); + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto& newTokens = outputTokenIds.at(beam); + auto& reqTokens = tokens.at(batchId).at(seqIdx).at(beam); + + reqTokens.insert(reqTokens.end(), newTokens.begin(), newTokens.end()); + // FinishReason is only supported for bw=1 and inflight batching. + if (beamWidth == 1 && batchingType == BatchingType::kINFLIGHT) + { + EXPECT_EQ(result.finishReasons.at(beam), + result.isFinal ? FinishReason::kLENGTH : FinishReason::kNOT_FINISHED); + } + } + + auto& cumLogProbs = result.cumLogProbs; + auto& logProbs = result.logProbs; + auto& beamTokens = tokens.at(batchId).at(seqIdx); + testData.verifyLogProbs(outConfig.returnLogProbs, streaming, outConfig.excludeInputFromOutput, + givenInputLengths.at(batchId), beamWidth, beamTokens, cumLogProbs, logProbs, batchId, + flakyTestInfo); + + testData.validateContextLogits(outConfig.returnContextLogits, givenInputLengths.at(batchId), + beamWidth, contextLogits, vocabSizePadded, batchId); + testData.validateGenerationLogits(outConfig.returnGenerationLogits, result.isFinal, streaming, + outConfig.excludeInputFromOutput, givenInputLengths.at(batchId), reqMaxNewTokens.at(batchId), + beamWidth, beamTokens, genLogits, vocabSizePadded, batchId, returnAllGeneratedTokens); + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + ++iter; + } + EXPECT_LT(iter, maxWaitMs); + testData.verifyOutput(tokens, givenInputLengths, streaming, outConfig.excludeInputFromOutput, flakyTestInfo, + isSpeculativeDecoding, beamWidth, numReturnSequences, false); + } + comm.barrier(); + if (executor.isGenerationRank()) + { + verifyGenerateDistStats(executor.getLatestRequestStats()); + } +} + +void runDisaggTest(DisaggExecutorOrchestrator& executor, tensorrt_llm::runtime::BufferManager& manager, + ITensor const& givenInput, ModelIds const& modelIds, FlakyTestInfo const& flakyTestInfo, bool streaming, + SizeType32 const vocabSizePadded, BeamResult const& beamResult, OutputConfig const& outConfig, + bool isSpeculativeDecoding, int maxWaitMs, BatchingType batchingType, bool returnAllGeneratedTokens) +{ + + auto& comm = tensorrt_llm::mpi::MpiComm::world(); + auto const worldRank = comm.getRank(); + auto const worldSize = comm.getSize(); + auto const beamWidth = beamResult.beamWidth; + + std::unordered_map<IdType, SizeType32> reqIdToBatchId; + std::unordered_map<SizeType32, std::vector<BeamTokens>> tokens; + // std::unordered_map<IdType, IdType> gGenIdIdTogContextId; + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(givenInput, modelIds.padId); + auto const* const givenInputData = tr::bufferCast<TokenIdType const>(givenInput); + + auto const& inputShape = givenInput.getShape(); + ASSERT_EQ(inputShape.nbDims, 2); + ASSERT_GT(inputShape.d[0], 0); + + // Load expected outputs for each beam width value + auto testData = TestData::loadTestData(beamResult, givenInput, beamWidth, manager, outConfig, modelIds); + auto const maxSeqLen = testData.maxSeqLen; + + // Load expected outputs and inputs + SizeType32 numRequests = static_cast<SizeType32>(givenInputLengths.size()); + SizeType32 maxRequests = numRequests; + std::vector<Request> requests; + std::vector<SizeType32> reqMaxNewTokens; + SizeType32 const numReturnSequences = 1; + + for (SizeType32 req = 0; req < maxRequests; ++req) + { + SizeType32 inputLen = givenInputLengths.at(req); + auto maxNewTokens = maxSeqLen - maxInputLength; + reqMaxNewTokens.push_back(maxNewTokens); + SizeType32 endId = -1; + auto const* const seqBegin = givenInputData + req * maxInputLength; + VecTokens tokens(seqBegin, seqBegin + inputLen); + auto samplingConfig = tensorrt_llm::executor::SamplingConfig(beamWidth); + samplingConfig.setNumReturnSequences(numReturnSequences); + auto request = Request( + VecTokens(seqBegin, seqBegin + inputLen), maxNewTokens, streaming, samplingConfig, outConfig, endId); + request.setReturnAllGeneratedTokens(returnAllGeneratedTokens); + request.setRequestType(RequestType::REQUEST_TYPE_CONTEXT_ONLY); + requests.emplace_back(std::move(request)); + } + + if (worldRank == 0) + { + std::vector<IdType> reqIds; + + for (int i = 0; i < requests.size(); ++i) + { + std::vector<BeamTokens> resultTokens; + resultTokens.reserve(numReturnSequences); + for (SizeType32 seqIdx = 0; seqIdx < numReturnSequences; ++seqIdx) + { + resultTokens.emplace_back(beamWidth); + } + auto retReqId = executor.enqueueContext({requests[i]}, std::nullopt); + reqIds.push_back(retReqId.front()); + tokens[i] = std::move(resultTokens); + reqIdToBatchId[retReqId.front()] = i; + } + + int32_t numContextFinished = 0; + int contextIter = 0; + while (numContextFinished < maxRequests && contextIter < maxWaitMs) + { + std::chrono::milliseconds waitTime(1); + + auto contextResponses = executor.awaitContextResponses(waitTime); + contextIter++; + numContextFinished += contextResponses.size(); + + for (auto&& responseWithId : contextResponses) + { + auto contextGid = responseWithId.gid; + int batchId = reqIdToBatchId[contextGid]; + auto&& request = requests[batchId]; + request.setRequestType(RequestType::REQUEST_TYPE_GENERATION_ONLY); + request.setContextPhaseParams(responseWithId.response.getResult().contextPhaseParams.value()); + executor.enqueueGeneration({request}, {responseWithId.gid}, std::nullopt); + } + } + // Get the new tokens for each requests + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + while (numFinished < maxRequests && iter < maxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitGenerationResponses(waitTime); + for (auto& responseWithId : responses) + { + numResponses++; + if (!responseWithId.response.hasError()) + { + auto result = responseWithId.response.getResult(); + numFinished += result.isFinal; + auto batchId = reqIdToBatchId.at(responseWithId.gid); + auto seqIdx = result.sequenceIndex; + + auto& contextLogits = result.contextLogits; + auto& genLogits = result.generationLogits; + auto& outputTokenIds = result.outputTokenIds; + + EXPECT_EQ(result.finishReasons.size(), beamWidth); + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto& newTokens = outputTokenIds.at(beam); + auto& reqTokens = tokens.at(batchId).at(seqIdx).at(beam); + + reqTokens.insert(reqTokens.end(), newTokens.begin(), newTokens.end()); + // FinishReason is only supported for bw=1 and inflight batching. + if (beamWidth == 1 && batchingType == BatchingType::kINFLIGHT) + { + EXPECT_EQ(result.finishReasons.at(beam), + result.isFinal ? FinishReason::kLENGTH : FinishReason::kNOT_FINISHED); + } + } + + auto& cumLogProbs = result.cumLogProbs; + auto& logProbs = result.logProbs; + auto& beamTokens = tokens.at(batchId).at(seqIdx); + testData.verifyLogProbs(outConfig.returnLogProbs, streaming, outConfig.excludeInputFromOutput, + givenInputLengths.at(batchId), beamWidth, beamTokens, cumLogProbs, logProbs, batchId, + flakyTestInfo); + + testData.validateContextLogits(outConfig.returnContextLogits, givenInputLengths.at(batchId), + beamWidth, contextLogits, vocabSizePadded, batchId); + testData.validateGenerationLogits(outConfig.returnGenerationLogits, result.isFinal, streaming, + outConfig.excludeInputFromOutput, givenInputLengths.at(batchId), reqMaxNewTokens.at(batchId), + beamWidth, beamTokens, genLogits, vocabSizePadded, batchId, returnAllGeneratedTokens); + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(responseWithId.gid) + + " has already been processed and was terminated."; + EXPECT_EQ(responseWithId.response.getErrorMsg(), err); + } + } + ++iter; + } + EXPECT_LT(iter, maxWaitMs); + testData.verifyOutput(tokens, givenInputLengths, streaming, outConfig.excludeInputFromOutput, flakyTestInfo, + isSpeculativeDecoding, beamWidth, numReturnSequences, false); + } + comm.barrier(); +} + +TEST_P(DisaggParamsTest, DisaggTokenComparison) +{ + +#if ENABLE_MULTI_DEVICE + + if (!(tensorrt_llm::common::getEnvUseUCXKvCache())) + { + setenv("UCX_TLS", "^cuda_ipc", 1); // disable cuda_ipc for testing for mpi + } + else + { + setenv("UCX_TCP_CM_REUSEADDR", "y", + 1); // tests creates and destroies ucxCacheCommunicatoers frequently, so listener ports must be reused + } + auto const processNum = std::get<0>(GetParam()); + auto const modelNames = std::get<1>(GetParam()); + auto const participantIdsEachInstance = std::get<2>(GetParam()); // std::vector<std::vector<int>> + auto const participantDeviceIdsEachInstance = std::get<3>(GetParam()); // std::vector<std::vector<int>>; + auto const instanceRoles + = std::get<4>(GetParam()); // std::vector<int> ; //1 is context , 0 is generation, 2 is mixed + auto const controllerRank = std::get<5>(GetParam()); + + // params_check + auto const& world_comm = tensorrt_llm::mpi::MpiComm::world(); + int const commRank = world_comm.getRank(); + int const commSize = world_comm.getSize(); + if (commSize != processNum) + { + GTEST_SKIP() << " need " << processNum << " processes but got " << commSize << " mpi processes, skip test."; + } + ASSERT_EQ(participantIdsEachInstance.size(), participantDeviceIdsEachInstance.size()); + SizeType32 instanceNum = participantIdsEachInstance.size(); + ASSERT_EQ(instanceNum, instanceRoles.size()); + ASSERT_EQ(instanceNum, modelNames.size()); + + std::unordered_set<int> deviceIdsSet; + for (auto const& ids : participantDeviceIdsEachInstance) + { + for (auto const& id : ids) + { + deviceIdsSet.insert(id); + } + } + if (mDeviceCount < deviceIdsSet.size()) + { + GTEST_SKIP() << " need " << deviceIdsSet.size() << " devices but got " << mDeviceCount + << " devices, skip test."; + } + + ASSERT_GE(controllerRank, 0); + ASSERT_LT(controllerRank, commSize); + int ranksNum = 0; + std::unordered_map<SizeType32, SizeType32> rankCounter; + std::unordered_map<SizeType32, SizeType32> deviceCounter; + SizeType32 deviceRuseNum = 1; + bool isContext = false; + bool isGeneration = false; + std::vector<int> participatntIds; + std::vector<int> deviceIds; + std::string modelName; + bool isController = (commRank == controllerRank); + for (SizeType32 i = 0; i < instanceNum; i++) + { + auto const& ranksThisInstance = participantIdsEachInstance[i]; + auto const& devicesThisInstance = participantDeviceIdsEachInstance[i]; + + ASSERT_EQ(ranksThisInstance.size(), devicesThisInstance.size()); + SizeType32 rankNumThisInstance = ranksThisInstance.size(); + ASSERT_GT(rankNumThisInstance, 0); + ranksNum += rankNumThisInstance; + for (SizeType32 j = 0; j < rankNumThisInstance; j++) + { + rankCounter[ranksThisInstance[j]]++; + deviceCounter[devicesThisInstance[j]]++; + ASSERT_GE(rankCounter[ranksThisInstance[j]], 1); + deviceRuseNum = std::max(deviceCounter[devicesThisInstance[j]], deviceRuseNum); + ASSERT_GE(ranksThisInstance[j], 0); + ASSERT_LT(ranksThisInstance[j], commSize); + + if (commRank == ranksThisInstance[j]) + { + participatntIds = ranksThisInstance; + deviceIds = devicesThisInstance; + isContext = instanceRoles[i] == InstanceRole::kCONTEXT || instanceRoles[i] == InstanceRole::kMIXED; + isGeneration + = instanceRoles[i] == InstanceRole::kGENERATION || instanceRoles[i] == InstanceRole::kMIXED; + // modelName = isContext ? contextModel : genModel; + modelName = modelNames[i]; + } + } + } + ASSERT_GE(ranksNum, commSize); + + OutputConfig outConfig; + int const beamWidth = 1; + BeamResult beamResult{beamWidth}; + + bool streaming = false; + int const maxBeamWidth = 1; + ASSERT_TRUE(fs::exists(DATA_PATH)); + + fs::path modelPath; + // set defaults and adjust if needed by different models + fs::path inputPath = DATA_PATH / "input_tokens.npy"; + ModelIds modelIds{50256, 50256}; + SizeType32 vocabSizePadded{50257}; // gpt vocabSizePadded + bool isSpeculativeDecoding{false}; + + // NOTE: This can be used to disable checks for certain prompt batch entries + FlakyTestInfo flakyTestInfo; + + if (modelName == "gpt") + { + auto const resultsPath + = GPT_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + if (outConfig.returnContextLogits || outConfig.returnGenerationLogits) + { + modelPath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR() / "tp1-pp1-cp1-gpu"; + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_RESULT_FILE(); + beamResult.contextLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_FILE(); + beamResult.genLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_FILE(); + if (outConfig.returnLogProbs) + { + beamResult.cumLogProbsFile + = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_CUM_LOG_PROBS_FILE(); + beamResult.logProbsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_LOG_PROBS_FILE(); + } + } + else + { + modelPath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_FILE(); + if (outConfig.returnLogProbs) + { + beamResult.cumLogProbsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_CUM_LOG_PROBS_FILE(); + beamResult.logProbsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_LOG_PROBS_FILE(); + } + } + } + else if (modelName == "llama_tp4_pp1_cp1" || modelName == "llama_tp1_pp4_cp1" || modelName == "llama_tp2_pp2_cp1" + || modelName == "llama_tp1_pp2_cp1" || modelName == "llama_tp2_pp1_cp1" || modelName == "llama_tp1_pp1_cp1") + { + inputPath = DATA_PATH / LLAMA_INPUT_FILE; + vocabSizePadded = LLAMA_VOCAB_SIZE_PADDED; + + auto const resultsPath + = LLAMA_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + modelIds.padId = LLAMA_PAD_ID; + modelIds.endId = LLAMA_END_ID; + if (modelName == "llama_tp4_pp1_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP4_PP1_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp4_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP4_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp2_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP2_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp2-cp1-gpu"; + } + else if (modelName == "llama_tp2_pp1_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP1_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp2_pp2_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP2_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp1_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP2_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + } + } + else + { + TLLM_THROW("Unrecognized modelName"); + } + + // Warning: This should be the last check before running the test. + // It will initialize MPI which can take significant time. + if (modelName == "llama_tp4_pp1_cp1" || modelName == "llama_tp1_pp4_cp1" || modelName == "llama_tp2_pp2_cp1" + || modelName == "llama_tp1_pp2_cp1" || modelName == "llama_tp2_pp1_cp1") + { + if (outConfig.returnLogProbs || outConfig.returnContextLogits || outConfig.returnGenerationLogits) + { + GTEST_SKIP() << "Skipping logits and log probs tests for mpi runs"; + } + } + + // Returning logits will bring higher latency + if (streaming && (outConfig.returnContextLogits || outConfig.returnGenerationLogits)) + { + mMaxWaitMs = 20000; + } + + auto executorConfig = ExecutorConfig(maxBeamWidth); + FloatType freeGpuMemoryFraction = 0.9f / (deviceRuseNum); // context and gen instance run on same device + KvCacheConfig kvCacheConfig{true, std::nullopt, std::nullopt, std::nullopt, freeGpuMemoryFraction}; + executorConfig.setKvCacheConfig(kvCacheConfig); + executorConfig.setRequestStatsMaxIterations(1000); + executorConfig.setCacheTransceiverConfig( + texec::CacheTransceiverConfig(texec::CacheTransceiverConfig::BackendType::DEFAULT)); + auto manager = tr::BufferManager(std::make_shared<tr::CudaStream>()); + auto const& givenInput = tr::utils::loadNpy(manager, inputPath.string(), tr::MemoryType::kCPU); + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(*givenInput, modelIds.padId); + world_comm.barrier(); + auto disaggExecutor = tensorrt_llm::testing::disaggexecutor::DisaggExecutorLeader(modelPath, + ModelType::kDECODER_ONLY, executorConfig, isController, isContext, isGeneration, givenInputLengths.size(), + participatntIds, deviceIds, commRank); + + runDisaggTest(disaggExecutor, manager, *givenInput, modelIds, flakyTestInfo, streaming, vocabSizePadded, beamResult, + outConfig, isSpeculativeDecoding, mMaxWaitMs, executorConfig.getBatchingType(), false); + +#else + + GTEST_SKIP() << "Skipping DisaggExecutor Test"; + +#endif +} + +TEST_P(DisaggOrchestratorParamsTest, DisaggTokenComparison) +{ + +#if ENABLE_MULTI_DEVICE + + if (!(tensorrt_llm::common::getEnvUseUCXKvCache())) + { + setenv("UCX_TLS", "^cuda_ipc", 1); // disable cuda_ipc for testing for mpi + } + else + { + setenv("UCX_TCP_CM_REUSEADDR", "y", + 1); // tests creates and destroies ucxCacheCommunicatoers frequently, so listener ports must be reused + } + auto const processNum = std::get<0>(GetParam()); + auto const modelNames = std::get<1>(GetParam()); + auto const participantIdsEachInstance = std::get<2>(GetParam()); // std::vector<std::vector<int>> + auto const participantDeviceIdsEachInstance = std::get<3>(GetParam()); // std::vector<std::vector<int>>; + auto const instanceRoles = std::get<4>(GetParam()); // std::vector<int> ; //1 is context , 0 is generation + auto const controllerRank = std::get<5>(GetParam()); + + // params_check + auto const& world_comm = tensorrt_llm::mpi::MpiComm::world(); + int const commRank = world_comm.getRank(); + int const commSize = world_comm.getSize(); + if (commSize != processNum) + { + GTEST_SKIP() << " need " << processNum << " processes but got " << commSize << " mpi processes, skip test."; + } + + bool spawnProcess = false; + if (commSize == 1) + { + spawnProcess = true; + if (mDeviceCount < 4) + { + GTEST_SKIP() << "DisaggExecutorTest requires at least 4 GPUs"; + } + ASSERT_TRUE(tensorrt_llm::common::getEnvUseUCXKvCache() || tensorrt_llm::common::getEnvUseNixlKvCache()); + } + + ASSERT_EQ(participantIdsEachInstance.size(), participantDeviceIdsEachInstance.size()); + SizeType32 instanceNum = participantIdsEachInstance.size(); + ASSERT_EQ(instanceNum, instanceRoles.size()); + ASSERT_EQ(instanceNum, modelNames.size()); + + std::unordered_set<int> deviceIdsSet; + for (auto const& ids : participantDeviceIdsEachInstance) + { + for (auto const& id : ids) + { + deviceIdsSet.insert(id); + } + } + if (mDeviceCount < deviceIdsSet.size()) + { + GTEST_SKIP() << " need " << deviceIdsSet.size() << " devices but got " << mDeviceCount + << " devices, skip test."; + } + + ASSERT_GE(controllerRank, 0); + ASSERT_LT(controllerRank, commSize); + std::string modelName = modelNames[0]; + bool isController = (commRank == controllerRank); + std::vector<fs::path> contextModels; + std::vector<fs::path> genModels; + + auto getModelPath = [=](std::string modelNN) + { + fs::path retPath; + if (modelNN == "llama_tp4_pp1") + { + retPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + } + else if (modelNN == "llama_tp1_pp4") + { + retPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + } + else if (modelNN == "llama_tp1_pp2") + { + retPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp2-cp1-gpu"; + } + else if (modelNN == "llama_tp2_pp1") + { + retPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp1-cp1-gpu"; + } + else if (modelNN == "llama_tp2_pp2") + { + retPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + } + else if (modelNN == "llama_tp1_pp1") + { + retPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + } + return retPath; + }; + for (SizeType32 i = 0; i < instanceNum; i++) + { + if (instanceRoles[i] == InstanceRole::kCONTEXT) + { + contextModels.push_back(getModelPath(modelNames[i])); + } + else + { + genModels.push_back(getModelPath(modelNames[i])); + } + } + + OutputConfig outConfig; + int const beamWidth = 1; + BeamResult beamResult{beamWidth}; + + bool streaming = false; + int const maxBeamWidth = 1; + ASSERT_TRUE(fs::exists(DATA_PATH)); + + fs::path modelPath; + // set defaults and adjust if needed by different models + fs::path inputPath = DATA_PATH / "input_tokens.npy"; + ModelIds modelIds{50256, 50256}; + SizeType32 vocabSizePadded{50257}; // gpt vocabSizePadded + bool isSpeculativeDecoding{false}; + + // NOTE: This can be used to disable checks for certain prompt batch entries + FlakyTestInfo flakyTestInfo; + if (modelName == "llama_tp4_pp1" || modelName == "llama_tp1_pp4" || modelName == "llama_tp2_pp2" + || modelName == "llama_tp1_pp2" || modelName == "llama_tp2_pp1" || modelName == "llama_tp1_pp1") + { + inputPath = DATA_PATH / LLAMA_INPUT_FILE; + vocabSizePadded = LLAMA_VOCAB_SIZE_PADDED; + + auto const resultsPath + = LLAMA_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + modelIds.padId = LLAMA_PAD_ID; + modelIds.endId = LLAMA_END_ID; + if (modelName == "llama_tp4_pp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP4_PP1_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp4") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP4_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp2") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP2_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp2-cp1-gpu"; + } + else if (modelName == "llama_tp2_pp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP1_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp2_pp2") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP2_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP2_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + } + } + + else + { + TLLM_THROW("Unrecognized modelName"); + } + + // Warning: This should be the last check before running the test. + // It will initialize MPI which can take significant time. + if (modelName == "llama_tp4_pp1" || modelName == "llama_tp1_pp4" || modelName == "llama_tp2_pp2" + || modelName == "llama_tp1_pp2" || modelName == "llama_tp2_pp1") + { + if (outConfig.returnLogProbs || outConfig.returnContextLogits || outConfig.returnGenerationLogits) + { + GTEST_SKIP() << "Skipping logits and log probs tests for mpi runs"; + } + } + + // Returning logits will bring higher latency + if (streaming && (outConfig.returnContextLogits || outConfig.returnGenerationLogits)) + { + mMaxWaitMs = 20000; + } + + auto manager = tr::BufferManager(std::make_shared<tr::CudaStream>()); + auto const& givenInput = tr::utils::loadNpy(manager, inputPath.string(), tr::MemoryType::kCPU); + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(*givenInput, modelIds.padId); + world_comm.barrier(); + auto contextNum = contextModels.size(); + auto genNum = genModels.size(); + // int deviceCount = -1; + // TLLM_CUDA_CHECK(cudaGetDeviceCount(&deviceCount)); + bool isOrchestrator = commRank == 0; + std::vector<ExecutorConfig> ctxExecutorConfigs; + std::vector<ExecutorConfig> genExecutorConfigs; + for (int in = 0; in < instanceNum; in++) + { + tensorrt_llm::executor::SchedulerConfig schedulerConfig(CapacitySchedulerPolicy::kMAX_UTILIZATION); + KvCacheConfig kvCacheConfig{true, std::nullopt, std::nullopt, std::nullopt, 0.2}; + + tensorrt_llm::executor::ExecutorConfig executorConfig(maxBeamWidth, schedulerConfig, kvCacheConfig); + tensorrt_llm::executor::OrchestratorConfig orchestratorConfig{ + isOrchestrator, PathUtil::EXECUTOR_WORKER_PATH(), nullptr, spawnProcess}; + + tensorrt_llm::executor::ParallelConfig parallelConfig{tensorrt_llm::executor::CommunicationType::kMPI, + tensorrt_llm::executor::CommunicationMode::kORCHESTRATOR, participantDeviceIdsEachInstance.at(in), + spawnProcess ? std::nullopt : std::optional<std::vector<SizeType32>>(participantIdsEachInstance.at(in)), + orchestratorConfig}; + executorConfig.setParallelConfig(parallelConfig); + executorConfig.setCacheTransceiverConfig( + texec::CacheTransceiverConfig(texec::CacheTransceiverConfig::BackendType::DEFAULT)); + if (in < contextNum) + { + ctxExecutorConfigs.push_back(executorConfig); + } + else + { + genExecutorConfigs.push_back(executorConfig); + } + } + auto disaggExecutor + = DisaggExecutorOrchestrator(contextModels, genModels, ctxExecutorConfigs, genExecutorConfigs, true, true); + + runDisaggTest(disaggExecutor, manager, *givenInput, modelIds, flakyTestInfo, streaming, vocabSizePadded, beamResult, + outConfig, isSpeculativeDecoding, mMaxWaitMs, BatchingType::kINFLIGHT, false); + +#else + + GTEST_SKIP() << "Skipping DisaggExecutor Test"; + +#endif +} + +TEST_P(ConditionalDisaggParamsTest, DisaggTokenComparison) +{ +#if ENABLE_MULTI_DEVICE + if (!tensorrt_llm::common::getEnvUseUCXKvCache()) + { + setenv("UCX_TLS", "^cuda_ipc", 1); // disable cuda_ipc for testing for mpi + } + auto constexpr processNum = 2; + auto constexpr deviceNum = 2; + auto const& modelName = std::get<0>(GetParam()); + auto constexpr controllerRank = 0; + + // params_check + auto const& world_comm = tensorrt_llm::mpi::MpiComm::world(); + int const commRank = world_comm.getRank(); + int const commSize = world_comm.getSize(); + if (commSize != processNum) + { + GTEST_SKIP() << " need " << processNum << " processes but got " << commSize << " mpi processes, skip test."; + } + if (mDeviceCount < deviceNum) + { + GTEST_SKIP() << " need " << deviceNum << " devices but got " << mDeviceCount << " devices, skip test."; + } + + bool isContext = commRank == 0; + bool isGeneration = commRank == 1; + std::vector<int> participatntIds = {commRank}; + std::vector<int> deviceIds = {commRank}; + bool isController = (commRank == controllerRank); + + OutputConfig outConfig(false, false, false, false, false, false); + int const beamWidth = 1; + BeamResult beamResult{beamWidth}; + + bool streaming = false; + int const maxBeamWidth = 1; + ASSERT_TRUE(fs::exists(DATA_PATH)); + + fs::path modelPath; + // set defaults and adjust if needed by different models + fs::path inputPath = DATA_PATH / "input_tokens.npy"; + ModelIds modelIds{50256, 50256}; + SizeType32 vocabSizePadded{50257}; // gpt vocabSizePadded + bool isSpeculativeDecoding{false}; + + // NOTE: This can be used to disable checks for certain prompt batch entries + FlakyTestInfo flakyTestInfo; + + if (modelName == "gpt") + { + auto const resultsPath + = GPT_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + modelPath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_FILE(); + } + else if (modelName == "llama_tp1_pp1_cp1") + { + inputPath = DATA_PATH / LLAMA_INPUT_FILE; + vocabSizePadded = LLAMA_VOCAB_SIZE_PADDED; + + auto const resultsPath + = LLAMA_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + modelIds.padId = LLAMA_PAD_ID; + modelIds.endId = LLAMA_END_ID; + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP1_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + } + else + { + TLLM_THROW("Unrecognized modelName"); + } + + auto executorConfig = ExecutorConfig(maxBeamWidth); + FloatType freeGpuMemoryFraction = 0.9f; + KvCacheConfig kvCacheConfig{true, std::nullopt, std::nullopt, std::nullopt, freeGpuMemoryFraction}; + executorConfig.setKvCacheConfig(kvCacheConfig); + executorConfig.setRequestStatsMaxIterations(1000); + executorConfig.setCacheTransceiverConfig( + texec::CacheTransceiverConfig(CacheTransceiverConfig::BackendType::DEFAULT)); + auto manager = tr::BufferManager(std::make_shared<tr::CudaStream>()); + auto const& givenInput = tr::utils::loadNpy(manager, inputPath.string(), tr::MemoryType::kCPU); + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(*givenInput, modelIds.padId); + world_comm.barrier(); + auto executor = tensorrt_llm::testing::disaggexecutor::DisaggExecutorLeader(modelPath, ModelType::kDECODER_ONLY, + executorConfig, isController, isContext, isGeneration, givenInputLengths.size(), participatntIds, deviceIds, + commRank); + + std::unordered_map<IdType, SizeType32> reqIdToBatchId; + std::unordered_map<SizeType32, std::vector<BeamTokens>> tokens; + auto const* const givenInputData = tr::bufferCast<TokenIdType const>(*givenInput); + + auto const& inputShape = givenInput->getShape(); + ASSERT_EQ(inputShape.nbDims, 2); + ASSERT_GT(inputShape.d[0], 0); + + // Load expected outputs for each beam width value + auto testData = TestData::loadTestData(beamResult, *givenInput, beamWidth, manager, outConfig, modelIds); + auto const maxSeqLen = testData.maxSeqLen; + + // Load expected outputs and inputs + SizeType32 numRequests = static_cast<SizeType32>(givenInputLengths.size()); + SizeType32 maxRequests = numRequests; + std::vector<Request> requests; + std::vector<SizeType32> reqMaxNewTokens; + SizeType32 const numReturnSequences = 1; + + for (SizeType32 req = 0; req < maxRequests; ++req) + { + SizeType32 inputLen = givenInputLengths.at(req); + auto maxNewTokens = maxSeqLen - maxInputLength; + reqMaxNewTokens.push_back(maxNewTokens); + SizeType32 endId = -1; + auto const* const seqBegin = givenInputData + req * maxInputLength; + VecTokens tokens(seqBegin, seqBegin + inputLen); + auto samplingConfig = tensorrt_llm::executor::SamplingConfig(beamWidth); + samplingConfig.setNumReturnSequences(numReturnSequences); + auto request = Request( + VecTokens(seqBegin, seqBegin + inputLen), maxNewTokens, streaming, samplingConfig, outConfig, endId); + request.setReturnAllGeneratedTokens(false); + // setting request type to context/full by condition + if (req % 2 == 0) + { + request.setRequestType(RequestType::REQUEST_TYPE_CONTEXT_ONLY); + } + else + { + request.setRequestType(RequestType::REQUEST_TYPE_CONTEXT_AND_GENERATION); + } + requests.emplace_back(std::move(request)); + } + + if (isController) + { + std::vector<IdType> reqIds; + + for (int i = 0; i < requests.size(); ++i) + { + std::vector<BeamTokens> resultTokens; + resultTokens.reserve(numReturnSequences); + for (SizeType32 seqIdx = 0; seqIdx < numReturnSequences; ++seqIdx) + { + resultTokens.emplace_back(beamWidth); + } + auto retReqId = executor.enqueueRequests({requests[i]}); + reqIds.push_back(retReqId.front()); + tokens[i] = std::move(resultTokens); + reqIdToBatchId[retReqId.front()] = i; + } + + // Get the new tokens for each requests + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + while (numFinished < maxRequests && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + auto batchId = reqIdToBatchId.at(response.getRequestId()); + auto seqIdx = result.sequenceIndex; + + auto& outputTokenIds = result.outputTokenIds; + + EXPECT_EQ(result.finishReasons.size(), beamWidth); + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto& newTokens = outputTokenIds.at(beam); + auto& reqTokens = tokens.at(batchId).at(seqIdx).at(beam); + + reqTokens.insert(reqTokens.end(), newTokens.begin(), newTokens.end()); + // FinishReason is only supported for bw=1 and inflight batching. + if (beamWidth == 1 && executorConfig.getBatchingType() == BatchingType::kINFLIGHT) + { + EXPECT_EQ(result.finishReasons.at(beam), + result.isFinal ? FinishReason::kLENGTH : FinishReason::kNOT_FINISHED); + } + } + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + testData.verifyOutput(tokens, givenInputLengths, streaming, outConfig.excludeInputFromOutput, flakyTestInfo, + isSpeculativeDecoding, beamWidth, numReturnSequences, false); + } + world_comm.barrier(); +#else + GTEST_SKIP() << "Skipping DisaggExecutor Test"; +#endif +} + +INSTANTIATE_TEST_SUITE_P(GptDisaggSymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(2), // processNum + testing::Values(std::vector<std::string>{"gpt", "gpt"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0}, {1}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0, 1) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(GptDisaggSymmetricExecutorMixedTest, DisaggParamsTest, + testing::Combine( // + testing::Values(2), // processNum + testing::Values(std::vector<std::string>{"gpt", "gpt"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0}, {1}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kMIXED, InstanceRole::kMIXED}), // instanceRoles + testing::Values(1) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(GptSingleDeviceDisaggSymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(2), // processNum + testing::Values(std::vector<std::string>{"gpt", "gpt"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0}, {1}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {0}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(GptSingleDeviceDisaggSymmetricExecutorMixedTest, DisaggParamsTest, + testing::Combine( // + testing::Values(2), // processNum + testing::Values(std::vector<std::string>{"gpt", "gpt"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0}, {1}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {0}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kMIXED, InstanceRole::kMIXED}), // instanceRoles + testing::Values(1) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(GptConditionalDisaggSymmetricExecutorTest, ConditionalDisaggParamsTest, + testing::Combine(testing::Values("gpt")), generateTestNameCondDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaConditionalDisaggSymmetricExecutorTest, ConditionalDisaggParamsTest, + testing::Combine(testing::Values("llama_tp1_pp1_cp1")), generateTestNameCondDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaTP2DisaggSymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(4), // processNum + testing::Values(std::vector<std::string>{"llama_tp2_pp1_cp1", "llama_tp2_pp1_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0, 1}, {2, 3}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0, 1}, {2, 3}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaPP2DisaggSymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(4), // processNum + testing::Values(std::vector<std::string>{"llama_tp1_pp2_cp1", "llama_tp1_pp2_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0, 1}, {2, 3}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{1, 0}, {3, 2}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaTP2DisaggSymmetricExecutorMixedTest, DisaggParamsTest, + testing::Combine( // + testing::Values(2), // processNum + testing::Values(std::vector<std::string>{"llama_tp2_pp1_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0, 1}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0, 1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kMIXED}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaPP2DisaggSymmetricExecutorMixedTest, DisaggParamsTest, + testing::Combine( // + testing::Values(2), // processNum + testing::Values(std::vector<std::string>{"llama_tp1_pp2_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0, 1}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0, 1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kMIXED}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaTP2PP2DisaggSymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(8), // processNum + testing::Values(std::vector<std::string>{"llama_tp2_pp2_cp1", "llama_tp2_pp2_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0, 1, 2, 3}, {4, 5, 6, 7}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{2, 3, 0, 1}, {2, 3, 0, 1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaConPP2GenTP2DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(4), // processNum + testing::Values(std::vector<std::string>{"llama_tp1_pp2_cp1", "llama_tp2_pp1_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0, 1}, {2, 3}}), // (1,0) (2,3) // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{1, 0}, {2, 3}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaConTP2GenPP2DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(4), // processNum + testing::Values(std::vector<std::string>{"llama_tp2_pp1_cp1", "llama_tp1_pp2_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0, 1}, {2, 3}}), // (0,1), (3,2)// participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0, 1}, {3, 2}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaConTP2PP2GenPP2DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(6), // processNum + testing::Values(std::vector<std::string>{"llama_tp2_pp2_cp1", "llama_tp1_pp2_cp1"}), // modelNames + testing::Values( + std::vector<std::vector<int>>{{0, 1, 2, 3}, {4, 5}}), // (2,3,0,1) , (5,4)// participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{2, 3, 0, 1}, {1, 0}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaConTP2PP2GenTP2DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(6), // processNum + testing::Values(std::vector<std::string>{"llama_tp2_pp2_cp1", "llama_tp2_pp1_cp1"}), // modelNames + testing::Values( + std::vector<std::vector<int>>{{0, 1, 2, 3}, {4, 5}}), // (2,3,0,1), (4,5)// participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{2, 3, 0, 1}, {0, 1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); +INSTANTIATE_TEST_SUITE_P(LlamaConTP2PP1GenTP2PP2DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(6), // processNum + testing::Values(std::vector<std::string>{"llama_tp2_pp1_cp1", "llama_tp2_pp2_cp1"}), // modelNames + testing::Values( + std::vector<std::vector<int>>{{0, 1}, {2, 3, 4, 5}}), // (0,1) , (4,5,2,3)%4// participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0, 1}, {0, 1, 2, 3}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaConTP2GenPP4DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(6), // processNum + testing::Values(std::vector<std::string>{"llama_tp2_pp1_cp1", "llama_tp1_pp4_cp1"}), // modelNames + testing::Values( + std::vector<std::vector<int>>{{4, 5}, {0, 1, 2, 3}}), // (4,5) ,(3,2,1,0)// participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0, 1}, {3, 2, 1, 0}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon4TP1Gen1TP4DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(8), // processNum + testing::Values(std::vector<std::string>{"llama_tp1_pp1_cp1", "llama_tp1_pp1_cp1", "llama_tp1_pp1_cp1", + "llama_tp1_pp1_cp1", "llama_tp4_pp1_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {2}, {3}, {4, 5, 6, 7}}), // participantIdsEachInstance + testing::Values( + std::vector<std::vector<int>>{{0}, {1}, {2}, {3}, {0, 1, 2, 3}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(4) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon2TP1Gen2TP2AndPP2DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(6), // processNum + testing::Values(std::vector<std::string>{ + "llama_tp1_pp1_cp1", "llama_tp1_pp1_cp1", "llama_tp2_pp1_cp1", "llama_tp1_pp2_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {2, 3}, {4, 5}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {2, 3}, {1, 0}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kGENERATION, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon2TP1Gen2PP2DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(6), // processNum + testing::Values(std::vector<std::string>{ + "llama_tp1_pp1_cp1", "llama_tp1_pp1_cp1", "llama_tp1_pp2_cp1", "llama_tp1_pp2_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {2, 3}, {4, 5}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {3, 2}, {1, 0}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kGENERATION, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon4TP1Gen1TP2PP2DisaggAsymmetricExecutorTest, DisaggParamsTest, + testing::Combine( // + testing::Values(8), // processNum + testing::Values(std::vector<std::string>{"llama_tp1_pp1_cp1", "llama_tp1_pp1_cp1", "llama_tp1_pp1_cp1", + "llama_tp1_pp1_cp1", "llama_tp2_pp2_cp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {2}, {3}, {4, 5, 6, 7}}), // participantIdsEachInstance + testing::Values( + std::vector<std::vector<int>>{{0}, {1}, {2}, {3}, {2, 3, 0, 1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(4) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon2TP1Gen2TP2DisaaggOrchestrator, DisaggOrchestratorParamsTest, + testing::Combine( // + testing::Values(7), // processNum + testing::Values( + std::vector<std::string>{"llama_tp1_pp1", "llama_tp1_pp1", "llama_tp2_pp1", "llama_tp2_pp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{1}, {2}, {3, 4}, {5, 6}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {2, 3}, {0, 1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kGENERATION, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); +// for disaggOrchestrator 1->0, 2->1, 3->2, 4->3, 5->0, 6->1 + +INSTANTIATE_TEST_SUITE_P(LlamaCon2TP2Gen2TP1DisaaggOrchestrator, DisaggOrchestratorParamsTest, + testing::Combine( // + testing::Values(7), // processNum + testing::Values( + std::vector<std::string>{"llama_tp2_pp1", "llama_tp2_pp1", "llama_tp1_pp1", "llama_tp1_pp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{1, 2}, {3, 4}, {5}, {6}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0, 1}, {2, 3}, {0}, {1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kGENERATION, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon2TP1Gen2PP2DisaaggOrchestrator, DisaggOrchestratorParamsTest, + testing::Combine( // + testing::Values(7), // processNum + testing::Values( + std::vector<std::string>{"llama_tp1_pp1", "llama_tp1_pp1", "llama_tp1_pp2", "llama_tp1_pp2"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{1}, {2}, {3, 4}, {5, 6}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {3, 2}, {1, 0}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kGENERATION, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon2TP1Gen1TP2PP2DisaaggOrchestrator, DisaggOrchestratorParamsTest, + testing::Combine( // + testing::Values(7), // processNum + testing::Values(std::vector<std::string>{"llama_tp1_pp1", "llama_tp1_pp1", "llama_tp2_pp2"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{1}, {2}, {3, 4, 5, 6}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {0, 1, 2, 3}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{ + InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon2TP2Gen2TP1DisaggSpawnOrchestrator, DisaggOrchestratorParamsTest, + testing::Combine( // + testing::Values(1), // processNum + testing::Values( + std::vector<std::string>{"llama_tp2_pp1", "llama_tp2_pp1", "llama_tp1_pp1", "llama_tp1_pp1"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{1, 2}, {3, 4}, {5}, {6}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0, 1}, {2, 3}, {0}, {1}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kGENERATION, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); + +INSTANTIATE_TEST_SUITE_P(LlamaCon2TP1Gen2PP2DisaggSpawnOrchestrator, DisaggOrchestratorParamsTest, + testing::Combine( // + testing::Values(1), // processNum + testing::Values( + std::vector<std::string>{"llama_tp1_pp1", "llama_tp1_pp1", "llama_tp1_pp2", "llama_tp1_pp2"}), // modelNames + testing::Values(std::vector<std::vector<int>>{{1}, {2}, {3, 4}, {5, 6}}), // participantIdsEachInstance + testing::Values(std::vector<std::vector<int>>{{0}, {1}, {3, 2}, {1, 0}}), // participantDeviceIdsEachInstance + testing::Values(std::vector<InstanceRole>{InstanceRole::kCONTEXT, InstanceRole::kCONTEXT, + InstanceRole::kGENERATION, InstanceRole::kGENERATION}), // instanceRoles + testing::Values(0) // controllerRank + ), + generateTestNameDisaggParams); diff --git a/cpp/tests/e2e_tests/executor/encDecTest.cpp b/cpp/tests/e2e_tests/executor/encDecTest.cpp new file mode 100644 index 000000000000..0095ae30ad40 --- /dev/null +++ b/cpp/tests/e2e_tests/executor/encDecTest.cpp @@ -0,0 +1,387 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "executorTest.h" + +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tensorrt_llm/runtime/utils/numpyUtils.h" +#include "tensorrt_llm/testing/modelSpec.h" +#include "tests/utils/common.h" + +#include <gmock/gmock.h> +#include <gtest/gtest.h> +#include <nlohmann/json.hpp> + +#include <algorithm> +#include <chrono> +#include <memory> +#include <string> +#include <vector> + +namespace tr = tensorrt_llm::runtime; + +using namespace tensorrt_llm::testing; +using namespace tensorrt_llm::executor; +using namespace std::chrono_literals; +using tensorrt_llm::testing::KVCacheType; + +namespace +{ + +std::string getEncDecEnginePath(std::string const& modelName, SizeType32 tp, SizeType32 pp, SizeType32 cp) +{ + return modelName + '/' + std::to_string(tp * pp * cp) + "-gpu/float16"; +} + +TokenIdType getDecTokenFromJsonConfig(std::filesystem::path decEnginePath, std::string const& token_name) +{ + TokenIdType tokenId = 0; + try + { + std::ifstream decoderJsonConfigPath(decEnginePath / "config.json"); + auto const decoderPretrainedConfig + = nlohmann::json::parse(decoderJsonConfigPath, nullptr, true, true).at("pretrained_config"); + tokenId = decoderPretrainedConfig.at(token_name).template get<int32_t>(); + } + catch (nlohmann::json::out_of_range& e) + { + TLLM_LOG_ERROR( + "Parameter %s cannot be found from decoder config.json in pretrained_config. Using default id 0.", + token_name.c_str()); + } + catch (nlohmann::json::type_error const& e) + { + TLLM_LOG_ERROR( + "Parameter %s has a different type from decoder config.json in pretrained_config. Using default id 0.", + token_name.c_str()); + } + return tokenId; +} + +} // namespace + +using EncDecParamsType = std::tuple<std::string, SizeType32, SizeType32, SizeType32, SizeType32, SizeType32, SizeType32, + std::vector<SizeType32>>; + +std::string generateTestNameEncDec(testing::TestParamInfo<EncDecParamsType> const& info) +{ + auto modelName = std::get<0>(info.param); + auto const beamWidth = std::get<1>(info.param); + auto const maxNewTokens = std::get<2>(info.param); + auto const tp = std::get<3>(info.param); + auto const pp = std::get<4>(info.param); + + // GTEST does not allow '-' in its test name + for (auto& c : modelName) + { + if (c == '-') + { + c = '_'; + } + } + + std::string name = "EncDecTest"; + name.append("_" + modelName); + name.append("_BeamWidth" + std::to_string(beamWidth)); + name.append("_MaxNewTokens" + std::to_string(maxNewTokens)); + name.append("_TP" + std::to_string(tp)); + name.append("_PP" + std::to_string(pp)); + return name; +} + +bool isLanguageAdapterName(std::string const& modelName) +{ + return modelName == LANGUAGE_ADAPTER_NAME; +} + +class EncDecParamsTest : public GptExecutorTest, public ::testing::WithParamInterface<EncDecParamsType> +{ +}; + +TEST_P(EncDecParamsTest, validEncDecCtor) +{ + auto const modelName = std::get<0>(GetParam()); + SizeType32 const beamWidth = std::get<1>(GetParam()); + SizeType32 const maxNewTokens = std::get<2>(GetParam()); + SizeType32 const tp = std::get<3>(GetParam()); + SizeType32 const pp = std::get<4>(GetParam()); + SizeType32 const cp = std::get<5>(GetParam()); + + auto const enginePathName = getEncDecEnginePath(modelName, tp, pp, cp); + std::filesystem::path encEnginePath = ENC_DEC_ENGINE_BASE / enginePathName / "encoder"; + std::filesystem::path decEnginePath = ENC_DEC_ENGINE_BASE / enginePathName / "decoder"; + ExecutorConfig executorConfig{}; + FloatType freeGpuMemoryFraction = 0.4f; + FloatType crossKvCacheFraction = 0.4f; + KvCacheConfig kvCacheConfig{false, std::nullopt, std::nullopt, std::nullopt, freeGpuMemoryFraction}; + kvCacheConfig.setCrossKvCacheFraction(crossKvCacheFraction); + executorConfig.setKvCacheConfig(kvCacheConfig); + auto executor = Executor(encEnginePath, decEnginePath, ModelType::kENCODER_DECODER, executorConfig); +} + +TEST_P(EncDecParamsTest, Forward) +{ + bool constexpr VERBOSE = false; + auto const modelName = std::get<0>(GetParam()); + SizeType32 const beamWidth = std::get<1>(GetParam()); + SizeType32 const maxNewTokens = std::get<2>(GetParam()); + SizeType32 const tp = std::get<3>(GetParam()); + SizeType32 const pp = std::get<4>(GetParam()); + SizeType32 const cp = std::get<5>(GetParam()); + + // Parameters for language adapter test + SizeType32 const numLanguages = std::get<6>(GetParam()); + std::vector<SizeType32> languageAdapterUids = std::get<7>(GetParam()); + + bool const streaming = false; + + auto const enginePathName = getEncDecEnginePath(modelName, tp, pp, cp); + std::filesystem::path encEnginePath = ENC_DEC_ENGINE_BASE / enginePathName / "encoder"; + std::filesystem::path decEnginePath = ENC_DEC_ENGINE_BASE / enginePathName / "decoder"; + + // load ground truth input & output data + auto manager = tr::BufferManager(std::make_shared<tr::CudaStream>()); + auto inputsIdsHost + = tr::utils::loadNpy(manager, (ENC_DEC_DATA_BASE / "input_ids.npy").string(), tr::MemoryType::kCPU); + auto inputsIdsPtr = tr::bufferCast<TokenIdType>(*inputsIdsHost); + auto inputLengthsHost + = tr::utils::loadNpy(manager, (ENC_DEC_DATA_BASE / "input_lengths.npy").string(), tr::MemoryType::kCPU); + auto inputLengthsPtr = tr::bufferCast<SizeType32>(*inputLengthsHost); + auto encoderOutputHost + = tr::utils::loadNpy(manager, (ENC_DEC_DATA_BASE / "encoder_output.npy").string(), tr::MemoryType::kCPU); + auto encoderOutputPtr = tr::bufferCast<half>(*encoderOutputHost); + auto decoderOutputHost = tr::utils::loadNpy(manager, + (ENC_DEC_DATA_BASE / "output_ids_beam").string() + std::to_string(beamWidth) + ".npy", tr::MemoryType::kCPU); + auto decoderOutputPtr = tr::bufferCast<TokenIdType>(*decoderOutputHost); + + // Rank and size info + auto& comm = tensorrt_llm::mpi::MpiComm::world(); + auto const worldRank = comm.getRank(); + auto const worldSize = comm.getSize(); + + // create executor + BatchingType const batchingType = BatchingType::kINFLIGHT; + FloatType freeGpuMemoryFraction = 0.5f; + FloatType crossKvCacheFraction = 0.5f; + KvCacheConfig kvCacheConfig{false, std::nullopt, std::nullopt, std::nullopt, freeGpuMemoryFraction}; + kvCacheConfig.setCrossKvCacheFraction(crossKvCacheFraction); + + ExecutorConfig executorConfig{beamWidth}; + executorConfig.setBatchingType(batchingType); + executorConfig.setKvCacheConfig(kvCacheConfig); + executorConfig.setNormalizeLogProbs(false); + + // TODO: OrchestratorMode test does not pass + bool const useOrchestratorMode = (tp * pp) > worldSize; + std::optional<OrchestratorConfig> orchestratorConfig = std::nullopt; + if (useOrchestratorMode) + { + orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + } + auto parallelConfig = ParallelConfig(CommunicationType::kMPI, + useOrchestratorMode ? CommunicationMode::kORCHESTRATOR : CommunicationMode::kLEADER, std::nullopt, std::nullopt, + orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + auto executor = Executor(encEnginePath, decEnginePath, ModelType::kENCODER_DECODER, executorConfig); + + OutputConfig outConfig; + outConfig.excludeInputFromOutput = false; + outConfig.returnLogProbs = false; + outConfig.returnGenerationLogits = false; + outConfig.returnContextLogits = false; + outConfig.returnEncoderOutput = false; + + TokenIdType bosId = getDecTokenFromJsonConfig(decEnginePath, "bos_token_id"); + TokenIdType padId = getDecTokenFromJsonConfig(decEnginePath, "pad_token_id"); + TokenIdType eosId = getDecTokenFromJsonConfig(decEnginePath, "eos_token_id"); + TokenIdType decoderStartTokenId = getDecTokenFromJsonConfig(decEnginePath, "decoder_start_token_id"); + + bool const isLanguageAdapterTest = isLanguageAdapterName(modelName); + // create requests + SizeType32 const nbRequests = inputLengthsHost->getShape().d[0]; + std::vector<Request> requests; + for (int i = 0, cumInputLen = 0; i < nbRequests; i++) + { + auto encoderInput = VecTokens(&inputsIdsPtr[cumInputLen], + &inputsIdsPtr[cumInputLen] + inputLengthsPtr[i]); // assume inputIds is flattened / no-padding + cumInputLen += inputLengthsPtr[i]; + auto decoderInput = VecTokens{decoderStartTokenId}; + Request req(decoderInput, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig, + eosId, padId); + req.setEncoderInputTokenIds(encoderInput); + if (isLanguageAdapterTest) + { + req.setLanguageAdapterUid(languageAdapterUids[i]); + } + requests.emplace_back(req); + } + + using namespace std::chrono; + + // enqueue requests + if (worldRank == 0) + { + auto tik = high_resolution_clock::now(); + std::vector<IdType> reqIds = executor.enqueueRequests(std::move(requests)); + + // get responses + milliseconds waitTime(5000); + auto responsesAll = executor.awaitResponses(reqIds, waitTime); + auto tok = high_resolution_clock::now(); + TLLM_LOG_DEBUG("TRT-LLM C++ E2E time %d ms", duration_cast<milliseconds>(tok - tik).count()); + TLLM_LOG_DEBUG("Number of responses: %d", responsesAll.size()); + + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + std::unordered_map<IdType, std::vector<VecTokens>> outputTokens; + for_each(reqIds.begin(), reqIds.end(), + [&outputTokens, &beamWidth](auto const& id) + { + TLLM_LOG_DEBUG("Request IDs: %d", id); + outputTokens[id] = {}; + for (int i = 0; i < beamWidth; i++) + { + outputTokens[id].emplace_back(VecTokens{}); + } + }); + for (int i = 0; i < reqIds.size(); i++) + { + auto& responses = responsesAll[i]; + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + for (int beam = 0; beam < beamWidth; beam++) + { + auto& resTokens = result.outputTokenIds.at(beam); + auto& outTokens = outputTokens.at(response.getRequestId()).at(beam); + outTokens.insert(outTokens.end(), std::make_move_iterator(resTokens.begin()), + std::make_move_iterator(resTokens.end())); + } + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + } + + // print output & check correctness with ground truth + for (auto const& [reqId, tokens] : outputTokens) + { + SizeType32 gtMaxLength = decoderOutputHost->getShape().d[1]; + auto gtOutput = decoderOutputPtr + (reqId - 1) * gtMaxLength; + + if constexpr (VERBOSE) + { + std::cout << ">>> Request ID: " << reqId << std::endl; + for (int beam = 0; beam < beamWidth; beam++) + { + std::cout << "output tokens, beam " << beam << ", output length " << tokens[beam].size() << ": " + << std::endl; + for_each(tokens[beam].begin(), tokens[beam].end(), + [](auto const& token) { std::cout << token << ", "; }); + std::cout << std::endl; + } + std::cout << "ground truth tokens: " << std::endl; + + SizeType32 gtLength = 0; + for (int i = 0; i < gtMaxLength; i++) + { + if (gtOutput[i] != eosId) + { + std::cout << gtOutput[i] << ", "; + gtLength++; + } + } + std::cout << std::endl; + std::cout << "ground truth length: " << gtLength << std::endl; + } + + // check token-by-token match between beam 0 & ground truth + ASSERT_TRUE(tokens.size() <= gtMaxLength) + << "Request ID " << reqId << "'s generated length is longer than ground truth length " << gtMaxLength; + for (int i = 0; i < gtMaxLength; i++) + { + if (outConfig.excludeInputFromOutput) + { + // if results exclude decoder start token, skip it in ground truth too + continue; + } + if (i < tokens[0].size()) + { + ASSERT_EQ(tokens[0][i], gtOutput[i]) + << "Generated token id: " << tokens[0][i] << " v.s. ground truth: " << gtOutput[i]; + } + else + { + ASSERT_EQ(gtOutput[i], eosId) << "Request ID " << reqId << "'s generated length " << tokens.size() + << " is shorter than ground truth length " << gtMaxLength; + } + } + } + } +} + +INSTANTIATE_TEST_SUITE_P(T5BasicTest, EncDecParamsTest, + testing::Combine(testing::Values(T5_NAME), testing::Values(1), testing::Values(64), testing::Values(1), + testing::Values(1), testing::Values(1), testing::Values(0), testing::Values(std::vector<SizeType32>{})), + generateTestNameEncDec); + +INSTANTIATE_TEST_SUITE_P(T5Beam2Test, EncDecParamsTest, + testing::Combine(testing::Values(T5_NAME), testing::Values(2), testing::Values(64), testing::Values(1), + testing::Values(1), testing::Values(1), testing::Values(0), testing::Values(std::vector<SizeType32>{})), + generateTestNameEncDec); + +INSTANTIATE_TEST_SUITE_P(T5MultiGPUTest, EncDecParamsTest, + testing::Combine(testing::Values(T5_NAME), testing::Values(1), testing::Values(64), testing::Values(4), + testing::Values(1), testing::Values(1), testing::Values(0), testing::Values(std::vector<SizeType32>{})), + generateTestNameEncDec); + +INSTANTIATE_TEST_SUITE_P(BartBasicTest, EncDecParamsTest, + testing::Combine(testing::Values(BART_NAME), testing::Values(1), testing::Values(64), testing::Values(1), + testing::Values(1), testing::Values(1), testing::Values(0), testing::Values(std::vector<SizeType32>{})), + generateTestNameEncDec); + +INSTANTIATE_TEST_SUITE_P(BartBeam2Test, EncDecParamsTest, + testing::Combine(testing::Values(BART_NAME), testing::Values(2), testing::Values(64), testing::Values(1), + testing::Values(1), testing::Values(1), testing::Values(0), testing::Values(std::vector<SizeType32>{})), + generateTestNameEncDec); + +INSTANTIATE_TEST_SUITE_P(BartMultiGPUTest, EncDecParamsTest, + testing::Combine(testing::Values(BART_NAME), testing::Values(1), testing::Values(64), testing::Values(4), + testing::Values(1), testing::Values(1), testing::Values(0), testing::Values(std::vector<SizeType32>{})), + generateTestNameEncDec); + +INSTANTIATE_TEST_SUITE_P(LanguageAdapterBasicTest, EncDecParamsTest, + testing::Combine(testing::Values(LANGUAGE_ADAPTER_NAME), testing::Values(1), testing::Values(64), + testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(4), + testing::Values(std::vector<SizeType32>{2, 3})), + generateTestNameEncDec); diff --git a/cpp/tests/e2e_tests/executor/executorMockTest.cpp b/cpp/tests/e2e_tests/executor/executorMockTest.cpp new file mode 100644 index 000000000000..0f0176f1ed6b --- /dev/null +++ b/cpp/tests/e2e_tests/executor/executorMockTest.cpp @@ -0,0 +1,1028 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "executorTest.h" + +#include "tensorrt_llm/batch_manager/trtGptModel.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tensorrt_llm/testing/modelSpec.h" +#include "tests/utils/common.h" + +#include <gmock/gmock.h> +#include <gtest/gtest.h> +#include <nlohmann/json.hpp> + +#include <algorithm> +#include <chrono> +#include <memory> +#include <numeric> +#include <string> +#include <thread> +#include <vector> + +using ::testing::_; +using ::testing::Invoke; + +namespace tr = tensorrt_llm::runtime; +namespace tb = tensorrt_llm::batch_manager; + +using namespace tensorrt_llm::testing; +using namespace tensorrt_llm::executor; +using namespace std::chrono_literals; +using tensorrt_llm::testing::KVCacheType; + +class MockedModel : public Model +{ + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + +public: + MOCK_METHOD(void, forwardSync, (), ()); + MOCK_METHOD(void, forwardAsync, (RequestList const&), ()); + MOCK_METHOD(void, terminateRequest, (std::shared_ptr<tb::LlmRequest> const& llmRequest, bool pause), ()); + MOCK_METHOD( + void, terminateRequestSync, (std::shared_ptr<tb::LlmRequest> const& llmRequest, FinishReason finishReason), ()); + MOCK_METHOD(SizeType32, getMaxNumSequences, (), (const)); + MOCK_METHOD(SizeType32, getMaxInputLen, (), (const)); + MOCK_METHOD(SizeType32, getHiddenSize, (), (const)); + MOCK_METHOD(SizeType32, getMaxSequenceLen, (), (const)); + MOCK_METHOD(SizeType32, getVocabSizePadded, (), (const)); + MOCK_METHOD(SizeType32, getMaxDraftLen, (), (const)); + MOCK_METHOD(SizeType32, getNumMicroBatches, (), (const)); + MOCK_METHOD(SizeType32, getOperatingBeamWidth, (), (const)); + MOCK_METHOD(nvinfer1::DataType, getLogitDataType, (), (const)); + MOCK_METHOD(nvinfer1::DataType, getTensorDataType, (std::string const&), (const)); + MOCK_METHOD(nvinfer1::Dims, getTensorShape, (std::string const&), (const)); + MOCK_METHOD(void, getCurrentIterationStats, (IterationStats&), (const)); + MOCK_METHOD(void, getCurrentRequestStats, (RequestStatsPerIteration&), (const)); + MOCK_METHOD(DebugTensorsPerIteration, getCurrentDebugTensors, (), (const)); + MOCK_METHOD(tr::WorldConfig const&, getWorldConfig, (), (const)); + MOCK_METHOD(tr::ModelConfig const&, getModelConfig, (), (const)); + MOCK_METHOD(tr::BufferManager const&, getBufferManager, (), (const)); + MOCK_METHOD(tr::BufferManager::CudaStreamPtr, getRuntimeStreamPtr, (), (const)); + MOCK_METHOD(IterationType, getIterCounter, (), (const, noexcept)); + MOCK_METHOD(bool, hasSpeculativeDecodingFastLogits, (), (const, noexcept)); + MOCK_METHOD(bool, getGatherGenerationLogits, (), (const)); + MOCK_METHOD(void, updatePeftCache, (LlmRequestPtr const& llmReqeust), ()); + MOCK_METHOD(void, setLogitsPostProcessorBatched, (std::optional<LogitsPostProcessorBatched>), ()); + MOCK_METHOD(void, setReplicateLogitsPostProcessor, (bool), ()); + MOCK_METHOD(bool, getReplicateLogitsPostProcessor, (), (const)); + MOCK_METHOD(bool, hasGuidedDecoder, (), (const, noexcept)); + MOCK_METHOD(void, resetIterationStats, (), ()); + MOCK_METHOD( + std::shared_ptr<tensorrt_llm::batch_manager::kv_cache_manager::BaseKVCacheManager>, getKVCacheManager, (), ()); + MOCK_METHOD(std::shared_ptr<tensorrt_llm::batch_manager::kv_cache_manager::BaseKVCacheManager const>, + getKVCacheManager, (), (const)); + MOCK_METHOD(SizeType32, getMaxCapacityBatchSize, (SizeType32, SizeType32), (const)); +}; + +using ParamType = std::tuple<bool, bool, int>; + +std::string generateTestName(testing::TestParamInfo<ParamType> const& info) +{ + auto const streaming = std::get<0>(info.param); + auto const excludeInputFromOutput = std::get<1>(info.param); + auto const beamWidth = std::get<2>(info.param); + std::string name = "ExecutorTest"; + if (streaming) + { + name += "Streaming"; + } + if (excludeInputFromOutput) + { + name += "ExclInput"; + } + name.append("BW" + std::to_string(beamWidth)); + return name; +} + +class ParamTest : public GptExecutorTest, public ::testing::WithParamInterface<ParamType> +{ +}; + +TEST_P(ParamTest, MockedModel) +{ + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + bool const streaming = std::get<0>(GetParam()); + bool const excludeInputFromOutput = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + auto model = std::make_shared<MockedModel>(); + + EXPECT_CALL(*model, terminateRequest(_, _)).Times(0); + EXPECT_CALL(*model, getVocabSizePadded()).Times(0); + EXPECT_CALL(*model, getLogitDataType()).Times(0); + tr::ModelConfig dummyModelConfig(0, 0, 0, 0, 1, 0, nvinfer1::DataType::kHALF); + EXPECT_CALL(*model, getModelConfig()) + .WillRepeatedly(Invoke([&]() -> tr::ModelConfig const& { return dummyModelConfig; })); + SizeType32 callCount = 0; + EXPECT_CALL(*model, forwardAsync(_)) + .WillRepeatedly(Invoke( + [&](RequestList const& requestList) + { + for (auto const& llmReq : requestList) + { + // Don't add any tokens to simulate no output tokens + llmReq->addNewTokens(VecTokens(beamWidth, 1)); + llmReq->setState(tb::LlmRequestState::kGENERATION_IN_PROGRESS); + if (llmReq->getMaxNumGeneratedTokens() >= llmReq->mMaxNewTokens) + { + llmReq->setState(tb::LlmRequestState::kGENERATION_COMPLETE); + } + } + callCount++; + })); + + EXPECT_CALL(*model, getMaxNumSequences()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxInputLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxSequenceLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxDraftLen()).WillRepeatedly(Invoke([&]() { return 0; })); + EXPECT_CALL(*model, getVocabSizePadded()).WillRepeatedly(Invoke([&]() { return 80000; })); + tr::WorldConfig const dummyWorldConfig; + EXPECT_CALL(*model, getWorldConfig()) + .WillRepeatedly(Invoke([&]() -> tr::WorldConfig const& { return dummyWorldConfig; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& /*stats*/) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& /*stats*/) { return; })); + + ExecutorConfig const executorConfig(beamWidth); + auto executor = Executor(model, executorConfig); + + // Create the request + constexpr SizeType32 maxNewTokens = 5; + VecTokens const inputTokens{1, 2, 3, 4}; + auto request + = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + + // Enqueue the request + auto requestId = executor.enqueueRequest(request); + + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + auto const& result = response.getResult(); + done = result.isFinal; + } + ++iter; + } + + EXPECT_LT(iter, mMaxWaitMs); + EXPECT_EQ(callCount, maxNewTokens); +} + +TEST_F(GptExecutorTest, MockedModelMaxQueueSize) +{ + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + auto model = std::make_shared<MockedModel>(); + + EXPECT_CALL(*model, terminateRequest(_, _)).Times(0); + EXPECT_CALL(*model, terminateRequestSync(_, _)).Times(0); + EXPECT_CALL(*model, getVocabSizePadded()).Times(0); + EXPECT_CALL(*model, getLogitDataType()).Times(0); + + SizeType32 callCount = 0; + EXPECT_CALL(*model, forwardAsync(_)) + .WillRepeatedly(Invoke( + [&](RequestList const& requestList) + { + for (auto const& llmReq : requestList) + { + // Sleep to allow queue to fill up + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + // Don't add any tokens to simulate no output tokens + llmReq->addNewTokens({1}); + llmReq->setState(tb::LlmRequestState::kGENERATION_IN_PROGRESS); + if (llmReq->getMaxNumGeneratedTokens() >= llmReq->mMaxNewTokens) + { + llmReq->setState(tb::LlmRequestState::kGENERATION_COMPLETE); + } + } + callCount++; + })); + + EXPECT_CALL(*model, getMaxNumSequences()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxInputLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxSequenceLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxDraftLen()).WillRepeatedly(Invoke([&]() { return 0; })); + EXPECT_CALL(*model, getVocabSizePadded()).WillRepeatedly(Invoke([&]() { return 80000; })); + tr::WorldConfig const dummyWorldConfig; + EXPECT_CALL(*model, getWorldConfig()) + .WillRepeatedly(Invoke([&]() -> tr::WorldConfig const& { return dummyWorldConfig; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& /*stats*/) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& /*stats*/) { return; })); + tr::ModelConfig dummyModelConfig(0, 0, 0, 0, 1, 0, nvinfer1::DataType::kHALF); + EXPECT_CALL(*model, getModelConfig()) + .WillRepeatedly(Invoke([&]() -> tr::ModelConfig const& { return dummyModelConfig; })); + SizeType32 maxQueueSize = 6; + ExecutorConfig executorConfig; + executorConfig.setMaxQueueSize(maxQueueSize); + + auto executor = Executor(model, executorConfig); + + // Create the request + SizeType32 const maxNewTokens = 5; + VecTokens const inputTokens{1, 2, 3, 4}; + auto request = Request(inputTokens, maxNewTokens); + + // Enqueue as many requests as the queue can manage + for (int i = 0; i < maxQueueSize; i++) + { + auto requestId = executor.enqueueRequest(request); + } + try + { + auto requestId = executor.enqueueRequest(request); + + FAIL() << "Expected TllmException"; + } + catch (std::exception const& e) + { + EXPECT_THAT(e.what(), testing::HasSubstr("Maximum queue size of 6 has been reached, please try again later")); + } + + // Wait for requests to get scheduled to free up space in queue + std::this_thread::sleep_for(std::chrono::milliseconds(maxQueueSize * 200)); + auto requestId = executor.enqueueRequest(request); + + try + { + auto samplingConfig = SamplingConfig(1); + samplingConfig.setNumReturnSequences(maxQueueSize); + auto request = Request(inputTokens, maxNewTokens, false, samplingConfig); + auto requestId = executor.enqueueRequest(request); + FAIL() << "Expected TllmException"; + } + catch (std::exception const& e) + { + EXPECT_THAT(e.what(), testing::HasSubstr("Maximum queue size of 6 has been reached, please try again later")); + } +} + +TEST_F(GptExecutorTest, MockedModelReqStatsBug) +{ + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + auto model = std::make_shared<MockedModel>(); + + EXPECT_CALL(*model, terminateRequest(_, _)).Times(0); + EXPECT_CALL(*model, getVocabSizePadded()).Times(0); + EXPECT_CALL(*model, getLogitDataType()).Times(0); + EXPECT_CALL(*model, updatePeftCache(_)).WillRepeatedly(Invoke([&]() { return; })); + + SizeType32 callCount = 0; + RequestList currentReq; + EXPECT_CALL(*model, forwardAsync(_)) + .WillRepeatedly(Invoke( + [&](RequestList const& requestList) + { + currentReq = requestList; + for (auto const& llmReq : requestList) + { + // Don't add any tokens to simulate no output tokens + llmReq->addNewTokens({1}); + llmReq->setState(tb::LlmRequestState::kGENERATION_IN_PROGRESS); + } + callCount++; + })); + + EXPECT_CALL(*model, forwardSync()) + .WillRepeatedly(Invoke( + [&]() + { + for (auto const& llmReq : currentReq) + { + if (llmReq->getMaxNumGeneratedTokens() >= llmReq->mMaxNewTokens) + { + llmReq->setState(tb::LlmRequestState::kGENERATION_COMPLETE); + } + } + return; + })); + + EXPECT_CALL(*model, getMaxNumSequences()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxInputLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxSequenceLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxDraftLen()).WillRepeatedly(Invoke([&]() { return 0; })); + EXPECT_CALL(*model, getVocabSizePadded()).WillRepeatedly(Invoke([&]() { return 80000; })); + tr::WorldConfig const dummyWorldConfig; + EXPECT_CALL(*model, getWorldConfig()) + .WillRepeatedly(Invoke([&]() -> tr::WorldConfig const& { return dummyWorldConfig; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& stats) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& stats) { return; })); + + SizeType32 beamWidth = 1; + ExecutorConfig executorConfig(beamWidth); + executorConfig.setRequestStatsMaxIterations(1000); + auto executor = Executor(model, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + int numRequests = 10000; + auto request + = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + + auto done = std::atomic<bool>{false}; + auto statsThreadDone = false; + // Spawn a thread that continuously get stats + auto statsThread = std::thread( + [&executor, &done, &statsThreadDone]() + { + while (!done) + { + auto reqStats = executor.getLatestRequestStats(); + std::this_thread::sleep_for(std::chrono::microseconds(10)); + } + statsThreadDone = true; + }); + + // Spawn a thread that enqueues the requests + std::vector<IdType> requestIds; + auto enqueueThread = std::thread( + [&executor, &requestIds, &request, &done, numRequests]() + { + for (int i = 0; i < numRequests; ++i) + { + requestIds.push_back(executor.enqueueRequest(request)); + } + done = true; + }); + enqueueThread.join(); + ASSERT_EQ(requestIds.size(), numRequests); + + // Wait for stats thread to be done, fail otherwise + int iter = 0; + while (!statsThreadDone && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + std::this_thread::sleep_for(std::chrono::milliseconds(waitTime)); + iter++; + } + ASSERT_TRUE(statsThreadDone); + statsThread.join(); +} + +TEST_F(GptExecutorTest, MockedModelEvictRestartValidityTest) +{ + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + constexpr bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + auto model = std::make_shared<MockedModel>(); + + EXPECT_CALL(*model, terminateRequest(_, _)).Times(0); + EXPECT_CALL(*model, getVocabSizePadded()).Times(0); + EXPECT_CALL(*model, getLogitDataType()).Times(0); + EXPECT_CALL(*model, updatePeftCache(_)).WillRepeatedly(Invoke([&]() { return; })); + tr::ModelConfig dummyModelConfig(0, 0, 0, 0, 1, 0, nvinfer1::DataType::kHALF); + EXPECT_CALL(*model, getModelConfig()) + .WillRepeatedly(Invoke([&]() -> tr::ModelConfig const& { return dummyModelConfig; })); + SizeType32 callCount = 0; + RequestList currentReq; + EXPECT_CALL(*model, forwardAsync(_)) + .WillRepeatedly(Invoke( + [&](RequestList const& requestList) + { + currentReq = requestList; + for (auto const& llmReq : requestList) + { + // Don't add any tokens to simulate no output tokens + llmReq->addNewTokens({1}); + llmReq->setState(tb::LlmRequestState::kGENERATION_IN_PROGRESS); + } + callCount++; + })); + + EXPECT_CALL(*model, forwardSync()) + .WillRepeatedly(Invoke( + [&]() + { + for (auto const& llmReq : currentReq) + { + if (llmReq->getMaxNumGeneratedTokens() >= llmReq->mMaxNewTokens) + { + llmReq->setState(tb::LlmRequestState::kGENERATION_COMPLETE); + } + } + return; + })); + + EXPECT_CALL(*model, getMaxNumSequences()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxInputLen()).WillRepeatedly(Invoke([&]() { return 6; })); + EXPECT_CALL(*model, getMaxSequenceLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxDraftLen()).WillRepeatedly(Invoke([&]() { return 0; })); + EXPECT_CALL(*model, getVocabSizePadded()).WillRepeatedly(Invoke([&]() { return 80000; })); + tr::WorldConfig const dummyWorldConfig; + EXPECT_CALL(*model, getWorldConfig()) + .WillRepeatedly(Invoke([&]() -> tr::WorldConfig const& { return dummyWorldConfig; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& /*stats*/) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& /*stats*/) { return; })); + + SizeType32 const beamWidth = 1; + ExecutorConfig executorConfig(beamWidth, + SchedulerConfig(CapacitySchedulerPolicy::kMAX_UTILIZATION)); // Condition 1 : MAX_UTILIZATION scheduling policy + executorConfig.setEnableChunkedContext(false); // Condition 2 : Chunked context disabled + executorConfig.setRequestStatsMaxIterations(1000); + auto executor = Executor(model, executorConfig); + + // Create the request + constexpr bool streaming = true; // Condition 3 : Streaming enabled + SizeType32 const maxNewTokens = 5; + VecTokens const tooLongInputTokens{1, 2, 3, 4, 5}; // Condition 4 : prompt input len + maxNewTokens > MaxInputLen + auto tooLongRequest = Request( + tooLongInputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + + // Enqueue the request + auto longRequestId = executor.enqueueRequest(tooLongRequest); + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(longRequestId, waitTime); + for (auto& response : responses) + { + EXPECT_EQ(response.hasError(), true); + EXPECT_THAT(response.getErrorMsg(), + testing::HasSubstr("sequence length is potentially greater than max input length")); + done = true; + } + ++iter; + } +} + +#if ENABLE_MULTI_DEVICE +// This test can be run manually to test multiGPU execution +// mpirun --allow-run-as-root -n 5 ./executorTest --gtest_filter="*MockedModelMultiGpu/ExecutorTest" +// Number of MPI ranks can be greater than tp + +TEST_P(ParamTest, MockedModelMultiGpu) +{ + auto const& world = tensorrt_llm::mpi::MpiComm::world(); + auto const worldRank = world.getRank(); + auto const worldSize = world.getSize(); + + // In this test, allow worldSize to be greater than tp = 4 + // If so, set participant ids to be the last 4 ranks + SizeType32 const tp = std::min(4, worldSize); + + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + bool const streaming = std::get<0>(GetParam()); + bool const excludeInputFromOutput = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + auto model = std::make_shared<MockedModel>(); + + // Create the request + constexpr SizeType32 maxNewTokens = 5; + VecTokens const inputTokens{1, 2, 3, 4}; + auto request + = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + + EXPECT_CALL(*model, terminateRequest(_, _)).Times(0); + EXPECT_CALL(*model, getVocabSizePadded()).Times(0); + EXPECT_CALL(*model, getLogitDataType()).Times(0); + tr::ModelConfig dummyModelConfig(0, 0, 0, 0, 1, 0, nvinfer1::DataType::kHALF); + EXPECT_CALL(*model, getModelConfig()) + .WillRepeatedly(Invoke([&]() -> tr::ModelConfig const& { return dummyModelConfig; })); + SizeType32 callCount = 0; + SizeType32 reqCallCount = 0; + EXPECT_CALL(*model, forwardAsync(_)) + .WillRepeatedly(Invoke( + [&](RequestList const& requestList) + { + for (auto const& llmReq : requestList) + { + EXPECT_EQ(llmReq->getTokens().size(), beamWidth); + // Verify that all MPI ranks get the expected request, even though only rank 0 actually gets the + // request + if (reqCallCount == 0) + { + EXPECT_EQ(llmReq->getOrigPromptLen(), request.getInputTokenIds().size()); + for (int i = 0; i < llmReq->getOrigPromptLen(); ++i) + { + EXPECT_EQ(llmReq->getTokens(beamWidth - 1).at(i), request.getInputTokenIds().at(i)); + } + } + EXPECT_EQ(llmReq->isStreaming(), request.getStreaming()); + EXPECT_EQ(llmReq->mMaxNewTokens, request.getMaxTokens()); + EXPECT_EQ( + llmReq->getTokens(beamWidth - 1).size(), request.getInputTokenIds().size() + reqCallCount); + + SizeType32 tokenId = 1; + COMM_SESSION.bcastValue(tokenId, 0); + // Don't add any tokens to simulate no output tokens + // Simulate leader rank communicating with comm session + VecTokens const newTokens(beamWidth, tokenId); + llmReq->addNewTokens(newTokens); + llmReq->setState(tb::LlmRequestState::kGENERATION_IN_PROGRESS); + if (llmReq->getMaxNumGeneratedTokens() >= llmReq->mMaxNewTokens) + { + llmReq->setState(tb::LlmRequestState::kGENERATION_COMPLETE); + } + reqCallCount++; + } + callCount++; + })); + + EXPECT_CALL(*model, getMaxNumSequences()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxInputLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxSequenceLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getVocabSizePadded()).WillRepeatedly(Invoke([&]() { return 80000; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& /*stats*/) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& /*stats*/) { return; })); + + tr::WorldConfig dummyWorldConfig = tr::WorldConfig(tp, 1, 1, worldRank, tp); + EXPECT_CALL(*model, getWorldConfig()) + .WillRepeatedly(Invoke([&]() -> tr::WorldConfig const& { return dummyWorldConfig; })); + + ParallelConfig parallelConfig; + + // Set participant ids to be of size tp, starting at worldSize - 1 + std::vector<SizeType32> participantIds; + participantIds.reserve(tp); + for (int i = 0; i < tp; ++i) + { + participantIds.push_back(worldSize - tp + i); + } + bool const isLeader = (worldRank == participantIds.front()); + parallelConfig.setParticipantIds(participantIds); + + bool const isWorker = (std::find(participantIds.begin(), participantIds.end(), worldRank) != participantIds.end()); + + // Set device ids + std::vector<SizeType32> deviceIds(tp); + std::iota(deviceIds.begin(), deviceIds.end(), 0); + parallelConfig.setDeviceIds(deviceIds); + + ExecutorConfig executorConfig(beamWidth); + executorConfig.setParallelConfig(parallelConfig); + auto executor = Executor(model, executorConfig); + + EXPECT_EQ(isWorker, executor.isParticipant()); + + // Enqueue the request + IdType requestId = 0; + if (isLeader) + { + requestId = executor.enqueueRequest(request); + + SizeType32 numResponses{0}; + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + ++numResponses; + auto const& result = response.getResult(); + EXPECT_EQ(result.outputTokenIds.size(), beamWidth); + auto expectedSize = streaming ? (beamWidth > 1 ? numResponses : 1) + : (maxNewTokens + (excludeInputFromOutput ? 0 : inputTokens.size())); + EXPECT_EQ(result.outputTokenIds.at(beamWidth - 1).size(), expectedSize); + done = result.isFinal; + } + ++iter; + } + + EXPECT_LT(iter, mMaxWaitMs); + EXPECT_EQ(numResponses, streaming ? maxNewTokens : 1); + EXPECT_EQ(callCount, maxNewTokens); + } +} +#endif // ENABLE_MULTI_DEVICE + +TEST_F(GptExecutorTest, MockedModelWithError) +{ + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + struct MockedModelParams + { + SizeType32 maxInputLen; + SizeType32 maxSeqLen; + SizeType32 expectedTerminateCnt; + SizeType32 expectedForwardCnt; + bool computeGenLogits; + bool computeContextLogits; + std::string expectedError; + }; + + std::vector<MockedModelParams> mockedModelParams; + // Mocked error in forward call + mockedModelParams.emplace_back(MockedModelParams{10, 20, 1, 1, true, true, "mocked error"}); + // prompt longer than maxInputLen + mockedModelParams.emplace_back(MockedModelParams{1, 20, 0, 0, true, true, "exceeds maximum input length"}); + // Model doesn't support context logits output + mockedModelParams.emplace_back( + MockedModelParams{10, 20, 0, 0, false, true, "gather_generation_logits must be enabled"}); + // Model doesn't support gen logits output + mockedModelParams.emplace_back( + MockedModelParams{10, 20, 0, 0, true, false, "need to build engine with gather_context"}); + + for (auto const& mockedModelParam : mockedModelParams) + { + auto model = std::make_shared<MockedModel>(); + SizeType32 beamWidth = 1; + + // One request should be terminated + EXPECT_CALL(*model, terminateRequest(_, _)).Times(mockedModelParam.expectedTerminateCnt); + EXPECT_CALL(*model, getVocabSizePadded()).WillRepeatedly(Invoke([&]() { return 1024; })); + EXPECT_CALL(*model, getLogitDataType()).WillRepeatedly(Invoke([&]() { return nvinfer1::DataType::kFLOAT; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& stats) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& stats) { return; })); + + SizeType32 callCount = 0; + EXPECT_CALL(*model, forwardAsync(_)) + .WillRepeatedly(Invoke( + [&](RequestList const&) + { + callCount++; + // There was a bug where we were missing a notify call when errors were encountered + // and this test was not catching it, probably because the error was reported + // before the first call to awaitResponses. So we add a sleep here to make sure + // the awaitResponses is called before the error is thrown + std::this_thread::sleep_for(std::chrono::milliseconds(200)); + throw std::runtime_error("mocked error"); + })); + + EXPECT_CALL(*model, getMaxNumSequences()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxInputLen()).WillRepeatedly(Invoke([&]() { return mockedModelParam.maxInputLen; })); + EXPECT_CALL(*model, getMaxSequenceLen()).WillRepeatedly(Invoke([&]() { return mockedModelParam.maxSeqLen; })); + EXPECT_CALL(*model, getMaxDraftLen()).WillRepeatedly(Invoke([&]() { return 0; })); + tr::WorldConfig const dummyWorldConfig; + EXPECT_CALL(*model, getWorldConfig()) + .WillRepeatedly(Invoke([&]() -> tr::WorldConfig const& { return dummyWorldConfig; })); + tr::ModelConfig dummyModelConfig(0, 0, 0, 0, 1, 0, nvinfer1::DataType::kHALF); + dummyModelConfig.computeContextLogits(mockedModelParam.computeContextLogits); + dummyModelConfig.computeGenerationLogits(mockedModelParam.computeGenLogits); + EXPECT_CALL(*model, getModelConfig()) + .WillRepeatedly(Invoke([&]() -> tr::ModelConfig const& { return dummyModelConfig; })); + EXPECT_CALL(*model, getGatherGenerationLogits()) + .WillRepeatedly(Invoke([&]() -> bool { return mockedModelParam.computeGenLogits; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& stats) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& stats) { return; })); + EXPECT_CALL(*model, getIterCounter()).WillRepeatedly(Invoke([&]() -> IterationType { return 0; })); + + ExecutorConfig executorConfig(beamWidth); + auto executor = Executor(model, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + + OutputConfig outConfig; + outConfig.returnContextLogits = true; + outConfig.returnGenerationLogits = true; + + auto streaming = false; + auto request = Request( + inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + + // Enqueue the request + auto requestId = executor.enqueueRequest(std::move(request)); + + bool done = false; + auto responses = executor.awaitResponses(requestId); + for (auto& response : responses) + { + if (!response.hasError()) + { + FAIL() << "Expecting an error to be received"; + } + else + { + auto err = response.getErrorMsg(); + EXPECT_THAT(err, testing::HasSubstr(mockedModelParam.expectedError)); + done = true; + } + } + + EXPECT_TRUE(done); + EXPECT_EQ(callCount, mockedModelParam.expectedForwardCnt); + } +} + +TEST_F(GptExecutorTest, MockedModelCancelRequest) +{ + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + constexpr bool streaming = true; + auto model = std::make_shared<MockedModel>(); + + std::unordered_map<IdType, tensorrt_llm::executor::FinishReason> reqIdsToTerminate; + // Two requests with one child request (3 in total) should be terminated + EXPECT_CALL(*model, terminateRequestSync(_, _)) + .Times(3) + .WillRepeatedly(Invoke([&](LlmRequestPtr const& llmRequest, FinishReason finishReason) + { reqIdsToTerminate.try_emplace(llmRequest->mRequestId, finishReason); })); + EXPECT_CALL(*model, terminateRequest(_, _)).Times(3); + EXPECT_CALL(*model, getVocabSizePadded()).Times(0); + EXPECT_CALL(*model, getLogitDataType()).Times(0); + tr::WorldConfig const dummyWorldConfig; + EXPECT_CALL(*model, getWorldConfig()) + .WillRepeatedly(Invoke([&]() -> tr::WorldConfig const& { return dummyWorldConfig; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& /*stats*/) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& /*stats*/) { return; })); + tr::ModelConfig dummyModelConfig(0, 0, 0, 0, 1, 0, nvinfer1::DataType::kHALF); + EXPECT_CALL(*model, getModelConfig()) + .WillRepeatedly(Invoke([&]() -> tr::ModelConfig const& { return dummyModelConfig; })); + + SizeType32 callCount = 0; + std::unordered_map<IdType, SizeType32> callCountPerSeq; + EXPECT_CALL(*model, forwardAsync(_)) + .WillRepeatedly(Invoke( + [&](RequestList const& requestList) + { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + + for (auto const& llmReq : requestList) + { + if (llmReq->isGenerationCompleteState()) + { + continue; + } + // Don't add any tokens to simulate no output tokens + llmReq->addNewTokens({1}); + llmReq->setState(tb::LlmRequestState::kGENERATION_IN_PROGRESS); + if (llmReq->getMaxNumGeneratedTokens() >= llmReq->mMaxNewTokens) + { + llmReq->setState(tb::LlmRequestState::kGENERATION_COMPLETE); + } + if (callCountPerSeq.find(llmReq->mRequestId) != callCountPerSeq.end()) + { + callCountPerSeq[llmReq->mRequestId]++; + } + else + { + callCountPerSeq[llmReq->mRequestId] = 1; + } + + if (reqIdsToTerminate.count(llmReq->mRequestId) != 0U) + { + if (!llmReq->isGenerationToCompleteState()) + { + model->terminateRequest(llmReq, false); + llmReq->finishByReason(reqIdsToTerminate[llmReq->mRequestId]); + llmReq->clearGeneratedTokens(); + } + reqIdsToTerminate.erase(llmReq->mRequestId); + } + } + callCount++; + })); + + EXPECT_CALL(*model, getMaxNumSequences()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxInputLen()).WillRepeatedly(Invoke([&]() { return 100; })); + EXPECT_CALL(*model, getMaxSequenceLen()).WillRepeatedly(Invoke([&]() { return 200; })); + EXPECT_CALL(*model, getVocabSizePadded()).WillRepeatedly(Invoke([&]() { return 80000; })); + + SizeType32 const beamWidth = 1; + ExecutorConfig const executorConfig(beamWidth); + auto executor = Executor(model, executorConfig); + + // Create the request + SizeType32 const maxNewTokens = 150; + VecTokens const inputTokens{1, 2, 3, 4}; + auto request = Request(inputTokens, maxNewTokens, streaming); + + // Enqueue the request + auto requestId = executor.enqueueRequest(std::move(request)); + + // Cancel the request + std::this_thread::sleep_for(std::chrono::milliseconds(30)); + executor.cancelRequest(requestId); + + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + + if (response.hasError()) + { + FAIL() << "Not expecting an error to be received"; + } + + auto const& result = response.getResult(); + done = result.isFinal; + if (done) + { + for (SizeType32 beamIdx = 0; beamIdx < beamWidth; ++beamIdx) + { + EXPECT_EQ(result.finishReasons[beamIdx], FinishReason::kCANCELLED); + } + } + } + ++iter; + } + + EXPECT_LT(iter, mMaxWaitMs); + // Expecting to receiving fewer tokens than maxNewTokens + EXPECT_LT(callCount, maxNewTokens); + + // Create the request having child requests. + auto samplingConfig2 = SamplingConfig(1); + samplingConfig2.setNumReturnSequences(2); + auto request2 = Request(inputTokens, maxNewTokens, streaming, samplingConfig2); + + // Reset call count. + callCount = 0; + callCountPerSeq.clear(); + + // Enqueue the request + auto requestId2 = executor.enqueueRequest(request2); + + // Cancel the request + std::this_thread::sleep_for(std::chrono::milliseconds(30)); + executor.cancelRequest(requestId2); + + done = false; + iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId2, waitTime); + for (auto& response : responses) + { + + if (response.hasError()) + { + FAIL() << "Not expecting an error to be received"; + } + + auto const& result = response.getResult(); + done = result.isFinal; + if (done) + { + EXPECT_EQ(result.finishReasons[0], FinishReason::kCANCELLED); + } + } + ++iter; + } + + EXPECT_LT(iter, mMaxWaitMs); + for (auto& [reqId, count] : callCountPerSeq) + { + // Expecting to receiving fewer tokens than maxNewTokens + EXPECT_LT(count, maxNewTokens) << "Failed at request id: " << reqId; + } +} + +TEST_F(GptExecutorTest, MockedModelNumReturns) +{ + using LlmRequestPtr = std::shared_ptr<tb::LlmRequest>; + using RequestList = std::list<LlmRequestPtr>; + + SizeType32 const maxBeamWidth = 4; + OutputConfig const outConfig; + auto model = std::make_shared<MockedModel>(); + + EXPECT_CALL(*model, terminateRequest(_, _)).Times(0); + EXPECT_CALL(*model, getVocabSizePadded()).Times(0); + EXPECT_CALL(*model, getLogitDataType()).Times(0); + tr::WorldConfig const dummyWorldConfig; + EXPECT_CALL(*model, getWorldConfig()) + .WillRepeatedly(Invoke([&]() -> tr::WorldConfig const& { return dummyWorldConfig; })); + EXPECT_CALL(*model, getCurrentIterationStats(_)).WillRepeatedly(Invoke([&](IterationStats& /*stats*/) { return; })); + EXPECT_CALL(*model, getCurrentRequestStats(_)) + .WillRepeatedly(Invoke([&](RequestStatsPerIteration& /*stats*/) { return; })); + tr::ModelConfig dummyModelConfig(0, 0, 0, 0, 1, 0, nvinfer1::DataType::kHALF); + EXPECT_CALL(*model, getModelConfig()) + .WillRepeatedly(Invoke([&]() -> tr::ModelConfig const& { return dummyModelConfig; })); + SizeType32 callCount = 0; + EXPECT_CALL(*model, forwardAsync(_)) + .WillRepeatedly(Invoke( + [&](RequestList const& requestList) + { + for (auto const& llmReq : requestList) + { + // Don't add any tokens to simulate no output tokens + auto numBeams = llmReq->mSamplingConfig.getNumReturnBeams(); + llmReq->addNewTokens(VecTokens(numBeams, 1)); + llmReq->setState(tb::LlmRequestState::kGENERATION_IN_PROGRESS); + if (llmReq->getMaxNumGeneratedTokens() >= llmReq->mMaxNewTokens) + { + llmReq->setState(tb::LlmRequestState::kGENERATION_COMPLETE); + } + } + callCount++; + })); + + EXPECT_CALL(*model, getMaxNumSequences()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxInputLen()).WillRepeatedly(Invoke([&]() { return 10; })); + EXPECT_CALL(*model, getMaxSequenceLen()).WillRepeatedly(Invoke([&]() { return 20; })); + EXPECT_CALL(*model, getVocabSizePadded()).WillRepeatedly(Invoke([&]() { return 80000; })); + + ExecutorConfig const executorConfig(maxBeamWidth); + auto executor = Executor(model, executorConfig); + + // Create the request + SizeType32 const maxNewTokens = 5; + VecTokens const inputTokens{1, 2, 3, 4}; + constexpr bool streaming = false; + + auto samplingConfig1 = SamplingConfig(1); + samplingConfig1.setNumReturnSequences(3); + auto request1 = Request(inputTokens, maxNewTokens, streaming, samplingConfig1, outConfig); + auto samplingConfig2 = SamplingConfig(4); + auto request2 = Request(inputTokens, maxNewTokens, streaming, samplingConfig2, outConfig); + auto samplingConfig3 = SamplingConfig(4); + samplingConfig3.setNumReturnSequences(2); + auto request3 = Request(inputTokens, maxNewTokens, streaming, samplingConfig3, outConfig); + + // Enqueue the request + auto requestId1 = executor.enqueueRequest(request1); + auto requestId2 = executor.enqueueRequest(request2); + auto requestId3 = executor.enqueueRequest(request3); + + // Expecting one response in beam search. Instead, numReturnSequences limits the number of beams to return. + std::unordered_map<IdType, SizeType32> expectedNumResponses{{requestId1, 3}, {requestId2, 1}, {requestId3, 1}}; + std::unordered_map<IdType, SizeType32> const expectedNumBeams{{requestId1, 1}, {requestId2, 4}, {requestId3, 2}}; + + std::unordered_map<IdType, SizeType32> numResponses{{requestId1, 0}, {requestId2, 0}, {requestId3, 0}}; + std::unordered_map<IdType, SizeType32> numBeams{{requestId1, 0}, {requestId2, 0}, {requestId3, 0}}; + int numFinished = 0; + int iter = 0; + while (numFinished < 3 && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + auto const& result = response.getResult(); + auto reqId = response.getRequestId(); + numFinished += result.isFinal; + numResponses[reqId]++; + numBeams[reqId] = result.outputTokenIds.size(); + } + ++iter; + } + + EXPECT_LT(iter, mMaxWaitMs); + EXPECT_EQ(numFinished, 3); + for (auto& [reqId, numResp] : numResponses) + { + EXPECT_EQ(numResp, expectedNumResponses[reqId]); + } + for (auto& [reqId, numResp] : numResponses) + { + EXPECT_EQ(numResp, expectedNumResponses[reqId]); + } +} + +INSTANTIATE_TEST_SUITE_P(GptExecutorTest, ParamTest, + testing::Combine(testing::Values(false, true), // streaming + testing::Values(false, true), // excludeInputFromOutput + testing::Values(1, 2) // beamWidth + ), + generateTestName); diff --git a/cpp/tests/e2e_tests/executor/executorTest.cpp b/cpp/tests/e2e_tests/executor/executorTest.cpp new file mode 100644 index 000000000000..e1227970cb71 --- /dev/null +++ b/cpp/tests/e2e_tests/executor/executorTest.cpp @@ -0,0 +1,4671 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TOP_LEVEL_DIR +#error "Define TOP_LEVEL_DIR" +#endif + +#include "executorTest.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/executor/dataTransceiverState.h" +#include "tensorrt_llm/executor/requestWithId.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/executor/version.h" +#include "tensorrt_llm/runtime/gptJsonConfig.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include "tensorrt_llm/runtime/utils/numpyUtils.h" +#include "tensorrt_llm/testing/modelSpec.h" +#include "tests/utils/common.h" + +#include <gmock/gmock.h> +#include <gtest/gtest.h> +#include <nlohmann/json.hpp> + +#include <algorithm> +#include <chrono> +#include <cstddef> +#include <functional> +#include <memory> +#include <string> +#include <thread> +#include <vector> + +namespace tr = tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; + +using namespace tensorrt_llm::testing; +using namespace tensorrt_llm::executor; +using namespace std::chrono_literals; +namespace fs = std::filesystem; +using tensorrt_llm::testing::KVCacheType; +using tensorrt_llm::testing::ModelSpec; + +namespace +{ + +auto const LORA_DATA_PATH = DATA_PATH / "lora-test-weights-gpt2-tp1"; +auto const LORA_WEIGHTS_FILE = LORA_DATA_PATH / "source.npy"; +auto const LORA_CONFIG_FILE = LORA_DATA_PATH / "config.npy"; + +auto constexpr LLAMA_INPUT_FILE = "input_tokens_llama.npy"; +auto constexpr LLAMA_VOCAB_SIZE_PADDED = 128256; +auto constexpr LLAMA_PAD_ID = 128001; +auto constexpr LLAMA_END_ID = 128001; + +} // namespace + +void testInvalidCtor(std::filesystem::path const& enginePath, ModelType modelType, ExecutorConfig executorConfig, + std::string expectedErrMsg = "") +{ + try + { + auto executor = Executor(enginePath, modelType, executorConfig); + + FAIL() << "Expected TllmException"; + } + catch (std::exception const& e) + { + EXPECT_THAT(e.what(), testing::HasSubstr(expectedErrMsg)); + } +} + +TEST_F(GptExecutorTest, version) +{ + EXPECT_STRNE(kTensorRtLlmVersion, "@TRTLLM_VERSION@"); + EXPECT_STREQ(kTensorRtLlmVersion, version()); +} + +TEST_F(GptExecutorTest, validCtor) +{ + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); +} + +TEST_F(GptExecutorTest, invalidCtor) +{ + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + std::filesystem::path invalidPath{"Bla"}; + + // Invalid path + { + testInvalidCtor(invalidPath, ModelType::kDECODER_ONLY, executorConfig, "File does not exist"); + } +} + +TEST_F(GptExecutorTest, enqueueAfterShutdown) +{ + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + auto request = Request(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth)); + auto requestId = executor.enqueueRequest(request); + + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + FAIL(); + } + else + { + done = response.getResult().isFinal; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + executor.shutdown(); + + EXPECT_FALSE(executor.canEnqueueRequests()); + + std::string expErrMsg{"Shutdown called"}; + EXPECT_THAT([&]() { auto reqId = executor.enqueueRequest(request); }, + testing::Throws<tensorrt_llm::common::TllmException>( + testing::Property(&tensorrt_llm::common::TllmException::what, testing::HasSubstr(expErrMsg)))); + EXPECT_THAT([&]() { auto resp = executor.awaitResponses(); }, + testing::Throws<tensorrt_llm::common::TllmException>( + testing::Property(&tensorrt_llm::common::TllmException::what, testing::HasSubstr(expErrMsg)))); + EXPECT_THAT([&]() { auto stats = executor.getLatestIterationStats(); }, + testing::Throws<tensorrt_llm::common::TllmException>( + testing::Property(&tensorrt_llm::common::TllmException::what, testing::HasSubstr(expErrMsg)))); + EXPECT_THAT([&]() { auto stats = executor.getLatestRequestStats(); }, + testing::Throws<tensorrt_llm::common::TllmException>( + testing::Property(&tensorrt_llm::common::TllmException::what, testing::HasSubstr(expErrMsg)))); + EXPECT_THAT([&]() { executor.cancelRequest(requestId); }, + testing::Throws<tensorrt_llm::common::TllmException>( + testing::Property(&tensorrt_llm::common::TllmException::what, testing::HasSubstr(expErrMsg)))); +} + +TEST_F(GptExecutorTest, missingPeftTask) +{ + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_LORA_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + auto request = Request(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth)); + auto loraConfig = LoraConfig{10}; + request.setLoraConfig(loraConfig); + + auto requestId = executor.enqueueRequest(request); + + bool done = false; + std::chrono::milliseconds waitTime(mMaxWaitMs); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + auto err = response.getErrorMsg(); + EXPECT_EQ(err, std::string("LoRA task 10 not found in cache. Please send LoRA weights with request")); + done = true; + } + else + { + FAIL() << "Expects error due to missing Lora weights"; + } + } + EXPECT_TRUE(done); +} + +TEST_F(GptExecutorTest, ReturnAcceptedTokenLogits) +{ + SizeType32 constexpr beamWidth{1}; + SizeType32 constexpr vocabSizePadded{50257}; // gpt vocabSizePadded + + // Create executor config + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setGatherGenerationLogits(true); + + // Enable kv cache reuse of executorConfig + bool enableBlockReuse = true; + FloatType freeGpuMemoryFraction = 0.4; + auto kvCacheConfig + = KvCacheConfig(enableBlockReuse, std::nullopt, std::nullopt, std::nullopt, freeGpuMemoryFraction); + executorConfig.setKvCacheConfig(kvCacheConfig); + + // Create executor + auto trtEnginePath + = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DRAFT_TOKENS_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4, 5, 6, 7, 8}; + + std::vector<bool> streamingOptions{false, true}; + + for (auto streaming : streamingOptions) + { + auto request = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth)); + + // Set draft tokens + auto draftTokens = VecTokens{9, 10, 11, 12, 13}; // draft tokens + auto draftLength = draftTokens.size(); + FloatType const acceptanceThreshold = 0.00001f; // Ensure the draft token can be accepted + auto externalDraftTokensConfig = ExternalDraftTokensConfig(draftTokens, std::nullopt, acceptanceThreshold); + request.setExternalDraftTokensConfig(externalDraftTokensConfig); + + // Set return accepted token logits for this request + OutputConfig outConfig; + outConfig.returnGenerationLogits = true; + request.setOutputConfig(outConfig); + + // Enqueue this request + auto requestId = executor.enqueueRequest(request); + + bool done = false; + int iter = 0; + while (!done && iter < 5000) + { + std::chrono::milliseconds waitTime(mMaxWaitMs); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + FAIL(); + } + else + { + auto result = response.getResult(); + done = result.isFinal; + auto& genLogits = result.generationLogits; + EXPECT_TRUE(genLogits.has_value()); + + // Expected shape: (1, numAcceptedDraftToken, vocabSizePadded) + auto const& acceptedTokenLogitsShape = genLogits->getShape(); + EXPECT_EQ(acceptedTokenLogitsShape.size(), 3); + EXPECT_EQ(acceptedTokenLogitsShape[0], 1); + EXPECT_LE(acceptedTokenLogitsShape[1], draftLength); // number of accepted tokens + EXPECT_EQ(acceptedTokenLogitsShape[2], vocabSizePadded); // vocabSizePadded + } + } + ++iter; + } + } +} + +TEST_F(GptExecutorTest, GenerationLogitsEarlyStop) +{ + SizeType32 constexpr beamWidth{1}; + SizeType32 constexpr vocabSizePadded{50257}; // gpt vocabSizePadded + auto constexpr streaming = false; + + ExtendedRuntimePerfKnobConfig perfKnobConfig = ExtendedRuntimePerfKnobConfig(); + + // Create executor config + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setExtendedRuntimePerfKnobConfig(perfKnobConfig); + executorConfig.setGatherGenerationLogits(true); + + // Create executor + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + auto const inputPath = DATA_PATH / "input_tokens.npy"; + ModelIds modelIds{50256, 50256}; + + auto manager = tr::BufferManager(std::make_shared<tr::CudaStream>()); + auto const& givenInput = tr::utils::loadNpy(manager, inputPath.string(), tr::MemoryType::kCPU); + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(*givenInput, modelIds.padId); + auto const* const givenInputData = tr::bufferCast<TokenIdType const>(*givenInput); + + auto const& inputShape = givenInput->getShape(); + ASSERT_EQ(inputShape.nbDims, 2); + ASSERT_GT(inputShape.d[0], 0); + + BeamResult beamResult{beamWidth}; + auto const resultsPath + = GPT_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_RESULT_FILE(); + beamResult.contextLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_FILE(); + beamResult.genLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_FILE(); + + // Set return generation logits for this request + OutputConfig outConfig; + outConfig.returnGenerationLogits = true; + outConfig.excludeInputFromOutput = true; + + // Load expected outputs for each beam width value + auto testData = TestData::loadTestData(beamResult, *givenInput, beamWidth, manager, outConfig, modelIds); + auto const maxSeqLen = testData.maxSeqLen; + + // Load expected outputs and inputs + std::vector<Request> requests; + std::vector<SizeType32> reqMaxNewTokens; + + auto constexpr reqIdx = 0; + SizeType32 inputLen = givenInputLengths.at(reqIdx); + auto maxNewTokens = maxSeqLen - maxInputLength; + reqMaxNewTokens.push_back(maxNewTokens); + auto const* const seqBegin = givenInputData + reqIdx * maxInputLength; + + auto request = Request(VecTokens(seqBegin, seqBegin + inputLen), maxNewTokens, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig, modelIds.endId); + // copy request + auto request2 = request; + + auto const expectedOutputData = tr::BufferRange<TokenIdType const>(*testData.expectedOutputIds); + auto const expectedOutputLengths = testData.expectedOutputLengths; + auto const endPos = expectedOutputLengths[reqIdx] - 3; + auto const endIndex = tc::flat_index3(reqIdx, beamWidth - 1, endPos, beamWidth, maxSeqLen); + auto const endToken = expectedOutputData[endIndex]; + + // Set end id to stop early + request.setEndId(endToken); + requests.emplace_back(std::move(request)); + + // Set stop words to stop early + request2.setStopWords({{endToken}}); + requests.emplace_back(std::move(request2)); + + // Enqueue requests + auto requestIds = executor.enqueueRequests(requests); + + std::map<IdType, SizeType32> expectedNewTokens; + expectedNewTokens[requestIds.at(0)] = endPos - inputLen; + expectedNewTokens[requestIds.at(1)] = endPos - inputLen + 1; + + std::map<IdType, FinishReason> expectedFinishReason; + expectedFinishReason[requestIds.at(0)] = FinishReason::kEND_ID; + expectedFinishReason[requestIds.at(1)] = FinishReason::kSTOP_WORDS; + + std::map<IdType, bool> done; + std::for_each(requestIds.begin(), requestIds.end(), [&done](auto id) { done[id] = false; }); + int iter = 0; + while (!(std::all_of(done.begin(), done.end(), [](auto x) { return x.second; })) && iter < 5000) + { + std::chrono::milliseconds waitTime(mMaxWaitMs); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + FAIL(); + } + else + { + auto const reqId = response.getRequestId(); + auto const& result = response.getResult(); + EXPECT_TRUE(result.isFinal); + done.at(reqId) = result.isFinal; + + // only 1 beam + auto const& outputIds = result.outputTokenIds.at(0); + EXPECT_EQ(outputIds.size(), expectedNewTokens.at(reqId)) << "req " << reqId; + + auto const& finishReason = result.finishReasons.at(0); + EXPECT_EQ(finishReason, expectedFinishReason.at(reqId)) << "req " << reqId; + + auto const& genLogits = result.generationLogits; + EXPECT_TRUE(genLogits.has_value()); + + // Expected shape: (1, numAcceptedDraftToken, vocabSizePadded) + auto const& generationLogitsShape = genLogits->getShape(); + EXPECT_EQ(generationLogitsShape.size(), 3); + EXPECT_EQ(generationLogitsShape[0], 1); + EXPECT_LE(generationLogitsShape[1], maxNewTokens); + EXPECT_EQ(generationLogitsShape[2], vocabSizePadded); + + auto const genLogitsTensor = detail::toITensor(*genLogits); + genLogitsTensor->squeeze(0); // only 1 beam + + for (size_t outputIdx = 0; outputIdx < expectedNewTokens.at(reqId); ++outputIdx) + { + // logits argmax should be equal to tokenId + auto const genLogitsSlice = tr::ITensor::slice(genLogitsTensor, outputIdx, 1); + auto const genLogitsRange = tr::BufferRange<float>(*genLogitsSlice); + auto const* maxPos = std::max_element(genLogitsRange.begin(), genLogitsRange.end()); + auto const maxIdx = std::distance(genLogitsRange.begin(), maxPos); + + auto const tokenId = outputIds.at(outputIdx); + // Observed token mismatch at index 2 after building GPT engine with TRT builder optimization + // level 3. The testcase is sensitive to slight variation in kernel computation, so we skip checking + // for token id at index 2. + if (outputIdx != 2) + { + EXPECT_EQ(tokenId, maxIdx) << "req " << reqId << " outputIdx " << outputIdx; + } + } + } + } + ++iter; + } +} + +TEST_F(GptExecutorTest, GenerationChangeEndId) +{ + SizeType32 constexpr beamWidth{2}; + SizeType32 constexpr vocabSizePadded{50257}; // gpt vocabSizePadded + auto constexpr streaming = false; + + ExtendedRuntimePerfKnobConfig perfKnobConfig = ExtendedRuntimePerfKnobConfig(); + perfKnobConfig.setEnableContextFMHAFP32Acc(true); // use fmha fp32 acc for better accuracy + + // Create executor config + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setExtendedRuntimePerfKnobConfig(perfKnobConfig); + + // Create executor + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + auto const inputPath = DATA_PATH / "input_tokens.npy"; + ModelIds modelIds{50256, 50256}; + + auto manager = tr::BufferManager(std::make_shared<tr::CudaStream>()); + auto const& givenInput = tr::utils::loadNpy(manager, inputPath.string(), tr::MemoryType::kCPU); + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(*givenInput, modelIds.padId); + auto const* const givenInputData = tr::bufferCast<TokenIdType const>(*givenInput); + + auto const& inputShape = givenInput->getShape(); + ASSERT_EQ(inputShape.nbDims, 2); + ASSERT_GT(inputShape.d[0], 0); + + BeamResult beamResult{beamWidth}; + auto const resultsPath + = GPT_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_CONTEXTFMHAFP32ACC_RESULT_FILE(); + + // Just return tokens for check + OutputConfig outConfig; + outConfig.excludeInputFromOutput = true; + + // Load expected outputs for each beam width value + auto testData = TestData::loadTestData(beamResult, *givenInput, beamWidth, manager, outConfig, modelIds); + auto const maxSeqLen = testData.maxSeqLen; + + // Load expected outputs and inputs + std::vector<Request> requests; + std::vector<SizeType32> reqMaxNewTokens; + + // Only use the first request to test + auto constexpr reqIdx = 0; + SizeType32 inputLen = givenInputLengths.at(reqIdx); + auto maxNewTokens = maxSeqLen - maxInputLength; + reqMaxNewTokens.push_back(maxNewTokens); + auto const* const seqBegin = givenInputData + reqIdx * maxInputLength; + + // Use customized `EndId` to enqueue once + auto request = Request(VecTokens(seqBegin, seqBegin + inputLen), maxNewTokens, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig, modelIds.endId); + + TokenIdType customizedEndId = *(seqBegin + 1); // Use a token appeared in ground-truth + request.setEndId(customizedEndId); + requests.emplace_back(std::move(request)); + + auto requestIds = executor.enqueueRequests(requests); + std::chrono::milliseconds waitTime(mMaxWaitMs); + auto responses = executor.awaitResponses(waitTime); + if (responses.at(0).hasError()) + { + FAIL(); + } + requests.clear(); + + // Change back to default `EndId` to enqueue again, and check the output + request = Request(VecTokens(seqBegin, seqBegin + inputLen), maxNewTokens, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig, modelIds.endId); + + auto const expectedOutputData = tr::BufferRange<TokenIdType const>(*testData.expectedOutputIds); + auto const expectedOutputLengths = testData.expectedOutputLengths; + auto const endPos = expectedOutputLengths[reqIdx]; + auto const endIndex = tc::flat_index3(reqIdx, beamWidth, endPos, beamWidth, maxSeqLen); + auto const endToken = expectedOutputData[endIndex]; + + request.setEndId(endToken); + requests.emplace_back(std::move(request)); + requestIds = executor.enqueueRequests(requests); + auto const requestId = requestIds.at(0); + + std::map<IdType, SizeType32> expectedNewTokens; + expectedNewTokens[requestId] = endPos - inputLen; + + std::map<IdType, FinishReason> expectedFinishReason; + expectedFinishReason[requestId] = FinishReason::kLENGTH; + + std::map<IdType, bool> done; + std::for_each(requestIds.begin(), requestIds.end(), [&done](auto id) { done[id] = false; }); + int iter = 0; + while (!(std::all_of(done.begin(), done.end(), [](auto x) { return x.second; })) && iter < 5000) + { + std::chrono::milliseconds waitTime(mMaxWaitMs); + auto responses = executor.awaitResponses(waitTime); + auto& response = responses.at(0); + if (response.hasError()) + { + FAIL(); + } + else + { + auto const reqId = response.getRequestId(); + auto const& result = response.getResult(); + EXPECT_TRUE(result.isFinal); + done.at(reqId) = result.isFinal; + + bool anyMismatch = false; + for (int i = 0; i < result.outputTokenIds.size(); ++i) + { + auto const& outputIds = result.outputTokenIds.at(i); + EXPECT_EQ(outputIds.size(), expectedNewTokens.at(reqId)) << "req " << reqId; + anyMismatch |= outputIds.size() != expectedNewTokens.at(reqId); + + auto const& finishReason = result.finishReasons.at(i); + EXPECT_EQ(finishReason, expectedFinishReason.at(reqId)) << "req " << reqId; + anyMismatch |= finishReason != expectedFinishReason.at(reqId); + + if (anyMismatch) + { + break; + } + + for (int j = 0; j < outputIds.size(); ++j) + { + auto const resultToken = outputIds[j]; + auto const groundTruthToken = expectedOutputData[maxSeqLen * i + inputLen + j]; + EXPECT_EQ(resultToken, groundTruthToken); + anyMismatch |= resultToken != groundTruthToken; + } + } + EXPECT_FALSE(anyMismatch); + } + ++iter; + } +} + +// stream, excludeInputFromOutput, beamWidth +using ParamType = std::tuple<bool, bool, int>; +// useOrchestratorMode, beamWidth, modelName +using ParamCancelReqType = std::tuple<bool, int, std::string>; +// modelName +using LeaderApiUsageType = std::tuple<std::string>; +// iterStatsMaxIterations, useOrchestratorMode +using ParamStatsType = std::tuple<int, bool>; +// streaming, beamWidth, computeLogProbs, excludeInputInOutput, returnContextLogits, returnGenerationLogits, modelName, +// useOrchestratorMode, returnAllGeneratedTokens, numReturnSequences +using AllParamsType = std::tuple<bool, int, bool, bool, bool, bool, std::string, bool, bool, int>; +// modelName, batched, replicated +using LogitsProcParamsType = std::tuple<std::string, bool, bool>; +// modelName +using GuidedDecodingParamsType = std::tuple<std::string>; +// modelName, useOrchestratorMode, beamWidth +using TimeoutTestParamsType = std::tuple<std::string, bool, int>; + +std::string generateTestName(testing::TestParamInfo<ParamType> const& info) +{ + auto const streaming = std::get<0>(info.param); + auto const excludeInputFromOutput = std::get<1>(info.param); + auto const beamWidth = std::get<2>(info.param); + std::string name = "ExecutorTest"; + if (streaming) + { + name += "Streaming"; + } + if (excludeInputFromOutput) + { + name += "ExclInput"; + } + name.append("BW" + std::to_string(beamWidth)); + return name; +} + +std::string generateTestNameCancelReq(testing::TestParamInfo<ParamCancelReqType> const& info) +{ + auto const& useOrchestratorMode = std::get<0>(info.param); + auto const beamWidth = std::get<1>(info.param); + auto const modelName = std::get<2>(info.param); + std::string name = "ExecutorTest"; + name.append("BW" + std::to_string(beamWidth)); + name.append("_" + modelName + "_"); + + if (useOrchestratorMode) + { + name.append("OrchMode"); + } + else + { + name.append("LeaderMode"); + } + return name; +} + +std::string generateTestNameLeaderApiUsage(testing::TestParamInfo<LeaderApiUsageType> const& info) +{ + auto const modelName = std::get<0>(info.param); + std::string name = "ExecutorTest"; + name.append("_" + modelName); + return name; +} + +std::string generateTestNameLogitsProc(testing::TestParamInfo<LogitsProcParamsType> const& info) +{ + auto const modelName = std::get<0>(info.param); + bool const batched = std::get<1>(info.param); + bool const replicated = std::get<2>(info.param); + std::string name = "ExecutorTest"; + name.append("_" + modelName); + if (batched) + { + name.append("_Batched"); + } + if (replicated) + { + name.append("_Replicated"); + } + return name; +} + +std::string generateTestNameGuidedDecoding(testing::TestParamInfo<GuidedDecodingParamsType> const& info) +{ + auto const modelName = std::get<0>(info.param); + std::string name = "ExecutorTest"; + name.append("_" + modelName); + return name; +} + +std::string generateTestNameTimeoutTest(testing::TestParamInfo<TimeoutTestParamsType> const& info) +{ + auto const modelName = std::get<0>(info.param); + auto const& useOrchestratorMode = std::get<1>(info.param); + auto const beamWidth = std::get<2>(info.param); + + std::string name = "ExecutorTest"; + name.append("_" + modelName); + + if (useOrchestratorMode) + { + name.append("_OrchMode"); + } + else + { + name.append("_LeaderMode"); + } + name.append("_BW" + std::to_string(beamWidth)); + return name; +} + +std::string generateTestNameStats(testing::TestParamInfo<ParamStatsType> const& info) +{ + int iterStatsMaxIterations = std::get<0>(info.param); + auto const& useOrchestratorMode = std::get<1>(info.param); + std::string name = "ExecutorTest_"; + name.append(std::to_string(iterStatsMaxIterations) + "_"); + if (useOrchestratorMode) + { + name.append("OrchMode"); + } + else + { + name.append("LeaderMode"); + } + return name; +} + +std::string generateTestNameAllParams(testing::TestParamInfo<AllParamsType> const& info) +{ + auto const streaming = std::get<0>(info.param); + auto const& beamWidth = std::get<1>(info.param); + auto const& computeLogProbs = std::get<2>(info.param); + auto const& excludeInputInOutput = std::get<3>(info.param); + auto const& returnContextLogits = std::get<4>(info.param); + auto const& returnGenerationLogits = std::get<5>(info.param); + auto const modelName = std::get<6>(info.param); + auto const& useOrchestratorMode = std::get<7>(info.param); + auto const& returnAllGeneratedTokens = std::get<8>(info.param); + auto const& numReturnSequences = std::get<9>(info.param); + + std::string name = "ExecutorTest_"; + + if (streaming) + { + name += "Streaming"; + } + + name.append("_BW" + std::to_string(beamWidth)); + name.append("Nseq" + std::to_string(numReturnSequences)); + + if (computeLogProbs) + { + name.append("LogProbs"); + } + if (excludeInputInOutput) + { + name.append("ExcludeInput"); + } + if (returnContextLogits) + { + name.append("ContextLogits"); + } + if (returnGenerationLogits) + { + name.append("GenerationLogits"); + } + name.append("_" + modelName + "_"); + if (useOrchestratorMode) + { + name.append("OrchMode"); + } + else + { + name.append("LeaderMode"); + } + + if (returnAllGeneratedTokens) + { + name.append("returnAllGeneratedTokens"); + } + return name; +} + +class ParamTest : public GptExecutorTest, public ::testing::WithParamInterface<ParamType> +{ +}; + +class ParamStatsTest : public GptExecutorTest, public ::testing::WithParamInterface<ParamStatsType> +{ +}; + +class AllParamsTest : public GptExecutorTest, public ::testing::WithParamInterface<AllParamsType> +{ +}; + +class ParamCancelReqTest : public GptExecutorTest, public ::testing::WithParamInterface<ParamCancelReqType> +{ +}; + +class LeaderApiUsageTest : public GptExecutorTest, public ::testing::WithParamInterface<LeaderApiUsageType> +{ +}; + +class LogitsProcParamsTest : public GptExecutorTest, public ::testing::WithParamInterface<LogitsProcParamsType> +{ +}; + +class GuidedDecodingParamsTest : public GptExecutorTest, public ::testing::WithParamInterface<GuidedDecodingParamsType> +{ +}; + +class TimeoutTest : public GptExecutorTest, public ::testing::WithParamInterface<TimeoutTestParamsType> +{ +}; + +TEST_F(GptExecutorTest, GetLatestStats) +{ + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + auto request + = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + auto requestId = executor.enqueueRequest(std::move(request)); + + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + FAIL(); + } + else + { + done = response.getResult().isFinal; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + // Expect 6 non-empty iterations + auto stats = executor.getLatestIterationStats(); + EXPECT_EQ(stats.size(), 6); + uint64_t currentIter = 0; + for (auto const& stat : stats) + { + EXPECT_EQ(stat.timestamp.size(), 26); + EXPECT_EQ(stat.iter, currentIter); + if (currentIter != 5) + { + EXPECT_EQ(stat.numActiveRequests, 1); + } + else + { + // For the last iteration the number of active requests + // should be zero. + EXPECT_EQ(stat.numActiveRequests, 0); + } + EXPECT_EQ(stat.maxNumActiveRequests, 64); + // Very loose check to make sure the memory stats are valid + EXPECT_GT(stat.gpuMemUsage, 16); + EXPECT_GT(stat.cpuMemUsage, 16); + EXPECT_GT(stat.pinnedMemUsage, 16); + + // Stats for KV cache + EXPECT_TRUE(stat.kvCacheStats.has_value()); + KvCacheStats const& kvStats = stat.kvCacheStats.value(); + EXPECT_GT(kvStats.maxNumBlocks, 0); + EXPECT_GT(kvStats.freeNumBlocks, 0); + EXPECT_EQ(kvStats.usedNumBlocks, currentIter == maxNewTokens ? 0 : 1); + EXPECT_GT(kvStats.tokensPerBlock, 0); + EXPECT_GT(kvStats.allocTotalBlocks, 0); + EXPECT_GT(kvStats.allocNewBlocks, 0); + EXPECT_GE(kvStats.reusedBlocks, 0); + EXPECT_GE(kvStats.missedBlocks, 0); + EXPECT_GE(kvStats.cacheHitRate, 0); + + // Stats for inflight batching + EXPECT_TRUE(stat.inflightBatchingStats.has_value() && !stat.staticBatchingStats.has_value()); + InflightBatchingStats const& modelStats = stat.inflightBatchingStats.value(); + EXPECT_EQ(modelStats.numScheduledRequests, currentIter == maxNewTokens ? 0 : 1); + EXPECT_EQ(modelStats.numContextRequests, currentIter == 0 ? 1 : 0); + EXPECT_EQ(modelStats.numGenRequests, currentIter == 0 || currentIter == maxNewTokens ? 0 : 1); + EXPECT_EQ(modelStats.numPausedRequests, 0); + EXPECT_EQ(modelStats.numCtxTokens, currentIter == 0 ? inputTokens.size() : 0); + EXPECT_EQ(modelStats.microBatchId, 0); + EXPECT_NEAR( + modelStats.avgNumDecodedTokensPerIter, currentIter == 0 || currentIter == maxNewTokens ? 0.f : 1.f, 1e-9f); + + auto jsonStr = JsonSerialization::toJsonStr(stat); + EXPECT_THAT(jsonStr, testing::HasSubstr("\"iter\":" + std::to_string(currentIter))); + EXPECT_THAT(jsonStr, testing::HasSubstr("\"staticBatchingStats\":null")); + EXPECT_THAT(jsonStr, testing::HasSubstr("\"numCtxTokens\":" + std::to_string(modelStats.numCtxTokens))); + EXPECT_THAT(jsonStr, testing::HasSubstr("\"numGenRequests\":" + std::to_string(modelStats.numGenRequests))); + + ++currentIter; + } +} + +TEST_F(GptExecutorTest, GetLatestStatsWithMultipleRequests) +{ + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the requests + SizeType32 const numRequests = 2; + std::vector<SizeType32> maxNewTokens{3, 5}; + std::vector<VecTokens> inputTokens{{1, 2, 3, 4}, {5, 6, 7}}; + std::vector<IdType> reqIds; + for (SizeType32 ireq = 0; ireq < numRequests; ++ireq) + { + auto request = Request(inputTokens[ireq], maxNewTokens[ireq], streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + auto requestId = executor.enqueueRequest(std::move(request)); + reqIds.emplace_back(requestId); + // sleep for 10 ms before sending the next request + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + + for (SizeType32 ireq = 0; ireq < numRequests; ++ireq) + { + auto requestId = reqIds[ireq]; + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + FAIL(); + } + else + { + done = response.getResult().isFinal; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + } + + // NOTES: + // Expect at least max(maxNewTokens) i.e. 5 non-empty iterations + // 4th iteration should have numCompletedRequests to be 1. + // Depending on the timing, first iteration will either have: + // 2 active requests + // or + // 1 active requests and 1 queued requests + auto stats = executor.getLatestIterationStats(); + EXPECT_GT(stats.size(), 0); // make sure we have at least 1 stat before the accessing 0-th element + if (stats[0].numActiveRequests == 2) + { + // we cannot reliably check queue latency since both started in the same iteration + // there should be exactly 5 non-empty iterations + EXPECT_EQ(stats.size(), 5); + // only check numCompletedRequests in 4th iteration + EXPECT_EQ(stats[3].numCompletedRequests, 1); + // 1st iteration shall record all 2 requests queueing time; + EXPECT_EQ(stats[0].numNewActiveRequests, 2); + // all rest iterations shall not return any queueing time; + for (int i = 1; i < stats.size(); ++i) + { + EXPECT_EQ(stats[i].numNewActiveRequests, 0); + } + } + else + { + // there should be more than 5 non-empty iterations since 2nd request started after 1st iteration + EXPECT_GT(stats.size(), 5); + // 1st request's completion is at 4th iteration + EXPECT_EQ(stats[3].numCompletedRequests, 1); + // 1st iteration record 1 request's queueing time; + EXPECT_EQ(stats[0].numNewActiveRequests, 1); + // the iteration where 2nd request became active, queue latency must be > 0 + uint64_t currentIter = 0; + for (auto const& stat : stats) + { + // To check when 2nd request becomes active, we need to think about 2 cases: + // - it overlaps with first request + // => only check queue time in this case + // - it doesn't overlap with the first request (e.g. 1st request ended too fast) + // => little to no queue time, cannot check reliably + // so we only check for queue time when numActiveRequests > 1 i.e. overlap happened after first iteration + if (stat.numActiveRequests > 1) + { + EXPECT_GT(currentIter, 0); // it must be after 1st iteration + EXPECT_GT(stat.newActiveRequestsQueueLatencyMS, 0); + // 2nd request record queueing time in this iteration + EXPECT_EQ(stat.numNewActiveRequests, 1); + break; + } + ++currentIter; + } + } +} + +TEST_F(GptExecutorTest, GetLatestRequestStats) +{ + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setRequestStatsMaxIterations(1000); + executorConfig.setEnableChunkedContext(true); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the requests + std::vector<std::pair<SizeType32, VecTokens>> requestParams = { + // {maxNewTokens, inputTokens} + {5, {1, 2, 3, 4}}, {4, {1, 1, 2, 3, 5}}, {1, {1}}, + {8, VecTokens(383, 1)} // Long enough to be chunked into multiple iterations + }; + std::vector<Request> requests; + for (auto requestParam : requestParams) + { + requests.emplace_back(requestParam.second, requestParam.first, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + } + auto requestIdsVec = executor.enqueueRequests(std::move(requests)); + std::map<IdType, SizeType32> requestIdToIndex; + std::set<IdType> activeRequests; + for (SizeType32 i = 0; i < requestIdsVec.size(); ++i) + { + auto requestId = requestIdsVec[i]; + activeRequests.insert(requestId); + requestIdToIndex[requestId] = i; + } + + int iter = 0; + while (!activeRequests.empty() && iter < mMaxWaitMs) + { + for (auto i = activeRequests.begin(); i != activeRequests.end();) + { + auto requestId = *i; + bool thisDone = false; + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + else + { + thisDone = response.getResult().isFinal; + } + } + if (thisDone) + { + // Erase completed request and move to the next one + i = activeRequests.erase(i); + } + else + { + ++i; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + // Expect 5 non-empty iterations + // Note: The 6th iteration with the last finished request will be reported + // but might be unavailable when getLatestRequestStats is called since + // it could be updated after the final response has been sent. + auto stats = executor.getLatestRequestStats(); + EXPECT_GE(stats.size(), 5); + SizeType32 currentIter = 0; + auto invalidStart = std::numeric_limits<SizeType32>::max(); + std::vector<SizeType32> genStart(requestParams.size(), invalidStart); // The iteration index when generation started + std::set<IdType> completedRequests; + for (auto stat = stats.begin(); stat != stats.begin() + 5; ++stat) + { + auto jsonStrIter = JsonSerialization::toJsonStr(*stat); + EXPECT_EQ(stat->iter, currentIter); + EXPECT_THAT(jsonStrIter, testing::HasSubstr("\"iter\":" + std::to_string(currentIter))); + EXPECT_EQ(stat->requestStats.size() + completedRequests.size(), requestParams.size()); + for (auto rStat : stat->requestStats) + { + auto jsonStr = JsonSerialization::toJsonStr(rStat); + // Only a few requests here so all of them should be scheduled. A separate test + // GetLatestRequestStatsScheduling will target the scheduling stats. + if (rStat.stage != RequestStage::kGENERATION_COMPLETE) + { + EXPECT_TRUE(rStat.scheduled); + EXPECT_THAT(jsonStr, testing::HasSubstr("\"scheduled\":true")); + } + EXPECT_TRUE(!rStat.paused); + EXPECT_THAT(jsonStr, testing::HasSubstr("\"paused\":false")); + EXPECT_TRUE(requestIdToIndex.count(rStat.id)); + EXPECT_THAT(jsonStr, testing::HasSubstr("\"id\":" + std::to_string(rStat.id))); + auto requestIndex = requestIdToIndex[rStat.id]; + auto contextSize = requestParams[requestIndex].second.size(); + if (rStat.contextPrefillPosition == contextSize) // Check generation phase + { + bool firstIteration{false}; + // Context phase is done + EXPECT_TRUE(rStat.stage == RequestStage::kGENERATION_IN_PROGRESS + || rStat.stage == RequestStage::kGENERATION_COMPLETE); + EXPECT_THAT(jsonStr, testing::HasSubstr("\"stage\":\"GENERATION")); + if (genStart[requestIndex] == invalidStart) + { + // Just started generation + genStart[requestIndex] = currentIter; + firstIteration = true; + } + + // One token per iteration + EXPECT_TRUE(currentIter - genStart[requestIndex] == rStat.numGeneratedTokens); + EXPECT_NEAR(rStat.avgNumDecodedTokensPerIter, firstIteration ? 0.f : 1.0f, 1e-9); + if (rStat.stage == RequestStage::kGENERATION_COMPLETE) + { + EXPECT_TRUE(requestParams[requestIndex].first >= rStat.numGeneratedTokens); + completedRequests.insert(requestIndex); + } + else + { + EXPECT_FALSE(completedRequests.count(requestIndex)); + } + } + else if (rStat.contextPrefillPosition < contextSize) // Check context phase + { + // Must be chunked + SizeType32 const maxChunkSize = 128; + EXPECT_TRUE(rStat.contextPrefillPosition % maxChunkSize == 0); + // Context phase is on-going + EXPECT_TRUE(rStat.stage == RequestStage::kCONTEXT_IN_PROGRESS); + // No tokens are generated + EXPECT_TRUE(0 == rStat.numGeneratedTokens); + } + else + { + FAIL() << "Out-of-boundary contextPrefillPosition in stats: " << rStat.contextPrefillPosition + << " out of " << contextSize; + } + // Sanity check that disaggregated serving stats is not set in typical use case + EXPECT_FALSE(rStat.disServingStats.has_value()); + } + ++currentIter; + } + // We should have visited all requests. + // Take into consideration the last request has not been reported + EXPECT_EQ(completedRequests.size() + 1, requestParams.size()); +} + +TEST_F(GptExecutorTest, GetLatestRequestStatsScheduling) +{ + // Specifically test the case where there are too many requests to be scheduled for a iteration + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setRequestStatsMaxIterations(1000); + executorConfig.setEnableChunkedContext(true); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create 100 requests. Note the max batch size for this model is 64 so some requests won't be scheduled right away. + std::vector<std::pair<SizeType32, VecTokens>> requestParams(100, {5, {1, 2, 3, 4}}); + std::vector<Request> requests; + requests.reserve(requestParams.size()); + for (auto requestParam : requestParams) + { + requests.emplace_back(requestParam.second, requestParam.first, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + } + auto requestIdsVec = executor.enqueueRequests(std::move(requests)); + std::map<IdType, SizeType32> requestIdToIndex; + std::set<IdType> activeRequests; + for (SizeType32 i = 0; i < requestIdsVec.size(); ++i) + { + auto requestId = requestIdsVec[i]; + activeRequests.insert(requestId); + requestIdToIndex[requestId] = i; + } + + int iter = 0; + while (!activeRequests.empty() && iter < mMaxWaitMs) + { + for (auto i = activeRequests.begin(); i != activeRequests.end();) + { + auto requestId = *i; + bool thisDone = false; + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + else + { + thisDone = response.getResult().isFinal; + } + } + if (thisDone) + { + // Erase completed request and move to the next one + i = activeRequests.erase(i); + } + else + { + ++i; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + auto stats = executor.getLatestRequestStats(); + SizeType32 numFinished = 0; + SizeType32 const maxActiveSize = 64; // Decided by the model + + // The 6th iteration request stat may or may not be available when getLatestRequestStats + // is called. When there are no other active or inTransmission requests, there will be + // another request stats to properly reset all the statistics to zero. + for (auto stat = stats.begin(); stat != stats.begin() + 5; ++stat) + { + SizeType32 numReqs = 0; + SizeType32 numReqsActive = 0; + SizeType32 numReqsQueued = 0; + SizeType32 numReqsJustDone = 0; + for (auto rStat : stat->requestStats) + { + ++numReqs; + numReqsActive += rStat.scheduled ? 1 : 0; + numReqsQueued += rStat.stage == RequestStage::kQUEUED ? 1 : 0; + numReqsJustDone += rStat.stage == RequestStage::kGENERATION_COMPLETE ? 1 : 0; + } + EXPECT_EQ(numReqs, numReqsActive + numReqsQueued + numReqsJustDone); + EXPECT_EQ(numReqs + numFinished, requestParams.size()); // Should report all unfinished requests + EXPECT_TRUE(numReqsActive <= maxActiveSize); // Not all requests are active due to max active size limit. + numFinished += numReqsJustDone; + } +} + +TEST_F(GptExecutorTest, GetRequestStatsMultipleRequests) +{ + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setRequestStatsMaxIterations(1000); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + auto sendRequestWaitForResponseFn = [&]() + { + Request request({1, 2, 3}, 5); + auto requestId = executor.enqueueRequest(request); + bool isFinalResponse = false; + while (!isFinalResponse) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto response : responses) + { + if (response.getResult().isFinal) + { + isFinalResponse = true; + break; + } + } + } + return requestId; + }; + + std::unordered_map<IdType, size_t> requestIdToGenerationComplete; + auto updateStats = [&]() + { + auto stats = executor.getLatestRequestStats(); + for (auto& stat : stats) + { + for (auto const& request : stat.requestStats) + { + // only check and aggregate results when request is completed + if (request.stage == RequestStage::kGENERATION_COMPLETE) + { + requestIdToGenerationComplete[request.id] += 1; + } + } + } + }; + + auto requestId = sendRequestWaitForResponseFn(); + requestIdToGenerationComplete[requestId] = 0; + updateStats(); + + requestId = sendRequestWaitForResponseFn(); + requestIdToGenerationComplete[requestId] = 0; + updateStats(); + + for (auto [key, value] : requestIdToGenerationComplete) + { + EXPECT_EQ(value, 1); + } +} + +TEST_F(GptExecutorTest, BatchSizeTuning) +{ + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setRequestStatsMaxIterations(1000); + executorConfig.setEnableChunkedContext(true); + + DynamicBatchConfig dynamicBatchConfig(true, false, 1); // Set window size to 1 + SchedulerConfig schedulerConfig(CapacitySchedulerPolicy::kGUARANTEED_NO_EVICT, std::nullopt, dynamicBatchConfig); + executorConfig.setSchedulerConfig(schedulerConfig); + + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + std::vector<SizeType32> tunerRecommendedBatchSizes; + + for (size_t i = 0; i <= 8; ++i) + { + auto inputLength = 1 << i; // Note that for this model max input len is 383 + Request request( + VecTokens(inputLength, 2), 5, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + auto requestId = executor.enqueueRequest(std::move(request)); + // Wait for current request to finish + while (true) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + bool done = false; + if (responses.size() != 0) + { + EXPECT_TRUE(responses.size() == 1); + auto response = responses[0]; + EXPECT_FALSE(response.hasError()); + if (response.getResult().isFinal) + { + break; + } + } + } + auto reqStats = executor.getLatestIterationStats(); + EXPECT_TRUE(reqStats.size() > 0); + auto lastStat = reqStats.back(); + tunerRecommendedBatchSizes.push_back(lastStat.maxBatchSizeTunerRecommended); + } + + EXPECT_TRUE(tunerRecommendedBatchSizes.size() > 0); + // It's supposed to be decreasing when input length increases + EXPECT_TRUE(*tunerRecommendedBatchSizes.begin() > *tunerRecommendedBatchSizes.rbegin()); +} + +TEST_F(GptExecutorTest, GetLatestDebugTensors) +{ + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + SizeType32 maxNewTokens = 5; + + tensorrt_llm::executor::DebugConfig debugConfig; + debugConfig.setDebugTensorNames({{"sequence_length"}}); + debugConfig.setDebugTensorsMaxIterations(maxNewTokens); + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setDebugConfig(debugConfig); + + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + VecTokens inputTokens{1, 2, 3, 4}; + auto request + = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + auto requestId = executor.enqueueRequest(request); + + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + FAIL(); + } + else + { + done = response.getResult().isFinal; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + auto stream = std::make_shared<tr::CudaStream>(); + + // Expect 5 non-empty iterations + auto debugTensors = executor.getLatestDebugTensors(); + EXPECT_EQ(debugTensors.size(), 5); + uint64_t currentIter = 0; + for (auto const& debugIteration : debugTensors) + { + EXPECT_EQ(debugIteration.iter, currentIter); + EXPECT_EQ(debugIteration.debugTensors.size(), 2); + + { + auto it = debugIteration.debugTensors.find("request_ids"); + EXPECT_NE(it, debugIteration.debugTensors.end()); + auto const& tensor = it->second; + auto const& shape = tensor.getShape(); + EXPECT_EQ(shape.size(), 1); + EXPECT_EQ(shape[0], 1); + EXPECT_EQ(tensor.getSize(), 1); + auto const* dataPtr = static_cast<SizeType32 const*>(tensor.getData()); + EXPECT_EQ(dataPtr[0], 1) << "currentIter " << currentIter; + } + { + auto it = debugIteration.debugTensors.find("sequence_length"); + EXPECT_NE(it, debugIteration.debugTensors.end()); + auto const& tensor = it->second; + auto const& shape = tensor.getShape(); + EXPECT_EQ(shape.size(), 1); + EXPECT_EQ(tensor.getSize(), 1); + auto tensorHost = tensor.copyToCpu(stream); + auto const* dataPtr = static_cast<SizeType32 const*>(tensorHost.getData()); + EXPECT_EQ(dataPtr[0], inputTokens.size() + currentIter); + } + + ++currentIter; + } +} + +TEST_P(ParamTest, SingleRequestDemo) +{ + bool const streaming = std::get<0>(GetParam()); + bool const excludeInputFromOutput = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + auto request + = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + + // Enqueue the request + auto requestId = executor.enqueueRequest(request); + + // Get the new tokens + VecTokens tokens; + SizeType32 numResponses{0}; + bool done = false; + int iter = 0; + std::chrono::milliseconds waitTime(1); + while (!done && iter < mMaxWaitMs) + { + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + ++numResponses; + if (response.hasError()) + { + // This request failed for some reason, get error msg + std::string errStr + = "Request id " + std::to_string(requestId) + " failed with err " + response.getErrorMsg(); + FAIL(); + } + + auto result = response.getResult(); + done = result.isFinal; + auto& newTokens = result.outputTokenIds.at(beamWidth - 1); + auto const expectedSize = streaming ? (beamWidth > 1 ? numResponses : 1) + : (maxNewTokens + (excludeInputFromOutput ? 0 : inputTokens.size())); + EXPECT_EQ(newTokens.size(), expectedSize); + + if (streaming && beamWidth > 1) + { + // replace tokens + tokens = newTokens; + } + else + { + // Append tokens + tokens.insert(tokens.end(), newTokens.begin(), newTokens.end()); + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + EXPECT_EQ(numResponses, streaming ? maxNewTokens : 1); + EXPECT_EQ( + tokens.size(), streaming ? maxNewTokens : (excludeInputFromOutput ? 0 : inputTokens.size()) + maxNewTokens); + + // Expect awaitResponse to return error message because the request is already terminated (isFinal = True) + auto response = executor.awaitResponses(requestId, waitTime).at(0); + EXPECT_TRUE(response.hasError()); + std::string err + = "ReqId " + std::to_string(response.getRequestId()) + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); +} + +TEST_P(ParamTest, MultipleRequestDemo) +{ + bool const streaming = std::get<0>(GetParam()); + bool const excludeInputFromOutput = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + SizeType32 numRequests = 20; + + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 maxPromptLen = 20; + SizeType32 maxMaxNewTokens = 20; + + SizeType32 endId = -1; + // Enqueue the requests + std::unordered_map<IdType, VecTokens> tokens; + std::unordered_map<IdType, SizeType32> expectedNumTokens; + std::unordered_map<IdType, SizeType32> expectedNumResponses; + for (SizeType32 req = 0; req < numRequests; ++req) + { + SizeType32 promptLen = rand() % maxPromptLen + 1; + SizeType32 maxNewTokens = rand() % maxMaxNewTokens + 1; + + auto request = Request(VecTokens(promptLen, 1), maxNewTokens, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig, endId); + auto reqId = executor.enqueueRequest(std::move(request)); + tokens[reqId] = {}; + expectedNumTokens[reqId] = ((streaming || excludeInputFromOutput) ? 0 : promptLen) + maxNewTokens; + expectedNumResponses[reqId] = streaming ? maxNewTokens : 1; + } + + // Get the new tokens for each requests + int32_t numFinished = 0; + int iter = 0; + std::unordered_map<IdType, SizeType32> numResponses; + while (numFinished < numRequests && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + auto reqId = response.getRequestId(); + ++numResponses[reqId]; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + auto& newTokens = result.outputTokenIds.at(beamWidth - 1); + auto const expectedSize + = streaming ? (beamWidth > 1 ? numResponses[reqId] : 1) : expectedNumTokens[reqId]; + EXPECT_EQ(newTokens.size(), expectedSize); + + auto& reqTokens = tokens.at(response.getRequestId()); + if (streaming && beamWidth > 1) + { + reqTokens = newTokens; + } + else + { + reqTokens.insert(reqTokens.end(), newTokens.begin(), newTokens.end()); + } + + for (SizeType32 b = 0; b < beamWidth; ++b) + { + EXPECT_EQ(result.finishReasons.at(b), + result.isFinal ? FinishReason::kLENGTH : FinishReason::kNOT_FINISHED); + } + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + // Check that number of tokens matches expectations + for (auto const& [reqId, numTokens] : expectedNumTokens) + { + EXPECT_EQ(expectedNumResponses[reqId], numResponses[reqId]) << "reqId " << reqId; + EXPECT_EQ(expectedNumTokens[reqId], tokens[reqId].size()) << "reqId " << reqId; + } +} + +TEST_P(ParamStatsTest, MultipleRequestStats) +{ + bool streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + SizeType32 numRequests = 100; + auto iterStatsMaxIterations = std::get<0>(GetParam()); + bool useOrchestratorMode = std::get<1>(GetParam()); + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setIterStatsMaxIterations(iterStatsMaxIterations); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + + std::optional<OrchestratorConfig> orchestratorConfig = std::nullopt; + if (useOrchestratorMode) + { + orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + } + auto parallelConfig = ParallelConfig(CommunicationType::kMPI, + useOrchestratorMode ? CommunicationMode::kORCHESTRATOR : CommunicationMode::kLEADER, std::nullopt, std::nullopt, + orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 maxPromptLen = 20; + SizeType32 maxMaxNewTokens = 20; + + SizeType32 endId = -1; + // Enqueue the requests + std::unordered_map<IdType, VecTokens> tokens; + std::unordered_map<IdType, SizeType32> expectedNumTokens; + for (SizeType32 req = 0; req < numRequests; ++req) + { + SizeType32 promptLen = rand() % maxPromptLen + 1; + SizeType32 maxNewTokens = rand() % maxMaxNewTokens + 1; + + auto request = Request(VecTokens(promptLen, 1), maxNewTokens, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig, endId); + auto reqId = executor.enqueueRequest(std::move(request)); + tokens[reqId] = {}; + expectedNumTokens[reqId] = (streaming ? 0 : (excludeInputFromOutput ? 0 : promptLen)) + maxNewTokens; + } + + std::atomic<bool> statsThreadDone = false; + std::atomic<int32_t> numFinished = 0; + std::deque<IterationStats> iterStatsReceived; + // Spawn a thread that continuously get stats + auto statsThread = std::thread( + [&executor, &numFinished, numRequests, &iterStatsReceived, &statsThreadDone]() + { + while (numFinished < numRequests) + { + auto reqStats = executor.getLatestIterationStats(); + iterStatsReceived.insert(iterStatsReceived.end(), std::make_move_iterator(reqStats.begin()), + std::make_move_iterator(reqStats.end())); + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + statsThreadDone = true; + }); + + // Get the new tokens for each requests + int iter = 0; + SizeType32 numResponses = 0; + while (numFinished < numRequests && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + auto& newTokens = result.outputTokenIds.at(beamWidth - 1); + auto& reqTokens = tokens.at(response.getRequestId()); + reqTokens.insert(reqTokens.end(), std::make_move_iterator(newTokens.begin()), + std::make_move_iterator(newTokens.end())); + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + // Check that number of tokens matches expectations + for (auto const& [reqId, numTokens] : expectedNumTokens) + { + EXPECT_EQ(expectedNumTokens[reqId], tokens[reqId].size()) << "reqId " << reqId; + } + + // Wait for stats thread to be done, fail otherwise + iter = 0; + while (!statsThreadDone && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + std::this_thread::sleep_for(std::chrono::milliseconds(waitTime)); + iter++; + } + ASSERT_TRUE(statsThreadDone); + if (iterStatsMaxIterations > 0) + { + ASSERT_GT(iterStatsReceived.size(), 1); + + for (auto stats : iterStatsReceived) + { + EXPECT_GT(stats.numActiveRequests, 0); + TLLM_LOG_INFO("%d %d", stats.iter, stats.numActiveRequests); + + EXPECT_TRUE(stats.inflightBatchingStats.has_value()); + if (stats.inflightBatchingStats.has_value()) + { + EXPECT_GT(stats.inflightBatchingStats.value().numScheduledRequests, 0); + } + } + } + + statsThread.join(); +} + +TEST_P(ParamTest, MultipleRequestBatchResponses) +{ + bool const streaming = std::get<0>(GetParam()); + bool const excludeInputFromOutput = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + SizeType32 constexpr numRequests{20}; + + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 constexpr maxPromptLen{20}; + SizeType32 constexpr maxMaxNewTokens{20}; + + SizeType32 endId = -1; + // Enqueue the requests + std::unordered_map<IdType, VecTokens> tokens; + std::unordered_map<IdType, SizeType32> expectedNumTokens; + std::vector<IdType> requestIds; + for (SizeType32 req = 0; req < numRequests; ++req) + { + SizeType32 promptLen = rand() % maxPromptLen + 1; + SizeType32 maxNewTokens = rand() % maxMaxNewTokens + 1; + + auto request = Request(VecTokens(promptLen, 1), maxNewTokens, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig, endId); + auto reqId = executor.enqueueRequest(std::move(request)); + requestIds.push_back(reqId); + tokens[reqId] = {}; + expectedNumTokens[reqId] = (streaming ? 0 : (excludeInputFromOutput ? 0 : promptLen)) + maxNewTokens; + } + + // Get the new tokens for each requests + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + std::chrono::milliseconds waitTime(1); + while (numFinished < numRequests && iter < mMaxWaitMs) + { + auto idResponses = executor.awaitResponses(requestIds, waitTime); + for (unsigned i = 0; i < requestIds.size(); ++i) + { + auto& responses = idResponses[i]; + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + auto& newTokens = result.outputTokenIds.at(beamWidth - 1); + auto& reqTokens = tokens.at(response.getRequestId()); + if (streaming && beamWidth > 1) + { + reqTokens = newTokens; + } + else + { + reqTokens.insert(reqTokens.end(), newTokens.begin(), newTokens.end()); + } + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + // Rerun awaitResponses again and we expect to only see terminated request id error. + auto idResponses = executor.awaitResponses(requestIds, waitTime); + for (auto const& responses : idResponses) + { + for (auto& response : responses) + { + EXPECT_TRUE(response.hasError()); + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + + // Check that number of tokens matches expectations + for (auto const& [reqId, numTokens] : expectedNumTokens) + { + EXPECT_EQ(expectedNumTokens[reqId], tokens[reqId].size()) << "reqId " << reqId; + } +} + +TEST_P(ParamTest, GetNumResponsesReadyTest) +{ + bool const streaming = std::get<0>(GetParam()); + bool const excludeInputFromOutput = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 maxNumRequests = 50; + SizeType32 maxPromptLen = 20; + SizeType32 maxMaxNewTokens = 20; + + SizeType32 numRequests = rand() % maxNumRequests + 1; + SizeType32 numExpectedResponses = 0; + std::map<IdType, SizeType32> reqNumExpectedResponses; + std::vector<IdType> ids; + for (SizeType32 req = 0; req < numRequests; ++req) + { + SizeType32 promptLen = rand() % maxPromptLen + 1; + SizeType32 maxNewTokens = rand() % maxMaxNewTokens + 1; + + auto request = Request(VecTokens(promptLen, 1), maxNewTokens, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + auto id = executor.enqueueRequest(std::move(request)); + ids.emplace_back(id); + reqNumExpectedResponses[id] = streaming ? maxNewTokens : 1; + numExpectedResponses += reqNumExpectedResponses.at(id); + } + + SizeType32 iter = 0; + SizeType32 numReady = 0; + while (numReady < numExpectedResponses && iter < mMaxWaitMs) + { + numReady = 0; + for (auto id : ids) + { + numReady += executor.getNumResponsesReady(id); + } + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + // Expect one response per request + for (auto id : ids) + { + SizeType32 numReady = executor.getNumResponsesReady(id); + EXPECT_EQ(numReady, reqNumExpectedResponses.at(id)); + } + auto numResponsesReady = executor.getNumResponsesReady(); + EXPECT_EQ(numResponsesReady, numExpectedResponses); +} + +namespace +{ + +void runTest(Executor& executor, fs::path const& inputPath, ModelIds const& modelIds, + FlakyTestInfo const& flakyTestInfo, bool streaming, SizeType32 const vocabSizePadded, BeamResult const& beamResult, + OutputConfig const& outConfig, bool isSpeculativeDecoding, int maxWaitMs, bool returnAllGeneratedTokens, + SizeType32 const numReturnSequences, bool isNonGreedySampling, SizeType32 const modelParallelism) +{ + auto const beamWidth = beamResult.beamWidth; + + auto manager = tr::BufferManager(std::make_shared<tr::CudaStream>()); + auto const& givenInput = tr::utils::loadNpy(manager, inputPath.string(), tr::MemoryType::kCPU); + auto [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(*givenInput, modelIds.padId); + auto const* const givenInputData = tr::bufferCast<TokenIdType const>(*givenInput); + + auto const& inputShape = givenInput->getShape(); + ASSERT_EQ(inputShape.nbDims, 2); + ASSERT_GT(inputShape.d[0], 0); + + // Load expected outputs for each beam width value + auto testData = TestData::loadTestData(beamResult, *givenInput, beamWidth, manager, outConfig, modelIds); + auto const maxSeqLen = testData.maxSeqLen; + + // Load expected outputs and inputs + SizeType32 numRequests = static_cast<SizeType32>(givenInputLengths.size()); + SizeType32 maxRequests = numRequests; + std::vector<Request> requests; + std::vector<SizeType32> reqMaxNewTokens; + + auto samplingConfig = tensorrt_llm::executor::SamplingConfig(beamWidth); + // top-k will be set by a large number to test non-identical N sequences. + if (isNonGreedySampling) + { + samplingConfig.setTopK(32); + } + samplingConfig.setNumReturnSequences(numReturnSequences); + + for (SizeType32 req = 0; req < maxRequests; ++req) + { + SizeType32 inputLen = givenInputLengths.at(req); + auto maxNewTokens = maxSeqLen - maxInputLength; + reqMaxNewTokens.push_back(maxNewTokens); + SizeType32 endId = -1; + auto const* const seqBegin = givenInputData + req * maxInputLength; + VecTokens tokens(seqBegin, seqBegin + inputLen); + auto request = Request( + VecTokens(seqBegin, seqBegin + inputLen), maxNewTokens, streaming, samplingConfig, outConfig, endId); + request.setReturnAllGeneratedTokens(returnAllGeneratedTokens); + requests.emplace_back(std::move(request)); + } + + auto& comm = tensorrt_llm::mpi::MpiComm::world(); + auto const worldRank = comm.getRank(); + + // Expected return sizes. + auto const numSequences = beamWidth > 1 ? 1 : numReturnSequences; + auto const numReturnBeams = std::min(beamWidth, numReturnSequences); + + if (worldRank == 0) + { + auto const reqIds = executor.enqueueRequests(requests); + + std::unordered_map<SizeType32, std::vector<BeamTokens>> tokens; + std::unordered_map<IdType, SizeType32> reqIdToBatchId; + + for (SizeType32 req = 0; req < reqIds.size(); ++req) + { + std::vector<BeamTokens> resultTokens(numSequences, BeamTokens(numReturnBeams)); + tokens[req] = std::move(resultTokens); + reqIdToBatchId[reqIds.at(req)] = req; + } + + // Get the new tokens for each requests + int32_t numFinished = 0; + int iter = 0; + std::unordered_map<IdType, SizeType32> numResponses; + while (numFinished < maxRequests && iter < maxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + auto batchId = reqIdToBatchId.at(response.getRequestId()); + numResponses[batchId]++; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + auto seqIdx = result.sequenceIndex; + + auto const& contextLogits = result.contextLogits; + auto const& genLogits = result.generationLogits; + auto const& outputTokenIds = result.outputTokenIds; + + EXPECT_EQ(result.finishReasons.size(), numReturnBeams); + for (SizeType32 beam = 0; beam < numReturnBeams; ++beam) + { + auto const& newTokens = outputTokenIds.at(beam); + auto& reqTokens = tokens.at(batchId).at(seqIdx).at(beam); + + if (!returnAllGeneratedTokens) + { + reqTokens.insert(reqTokens.end(), newTokens.begin(), newTokens.end()); + } + else + { + EXPECT_EQ(newTokens.size(), + (numResponses.at(batchId) + numReturnSequences - 1) / numReturnSequences); + reqTokens = newTokens; + } + // FinishReason is only supported for bw=1 and inflight batching. + if (beamWidth == 1) + { + EXPECT_EQ(result.finishReasons.at(beam), + result.isSequenceFinal ? FinishReason::kLENGTH : FinishReason::kNOT_FINISHED); + } + } + + auto const& cumLogProbs = result.cumLogProbs; + auto const& logProbs = result.logProbs; + auto const& beamTokens = tokens.at(batchId).at(seqIdx); + EXPECT_EQ(beamTokens.size(), numReturnBeams); + + if (!isNonGreedySampling) + { + float const logitsAtol = modelParallelism > 1 ? 1e-1 : 1e-2; + float const logitsRtol = modelParallelism > 1 ? 1e-2 : 1e-3; + + testData.verifyLogProbs(outConfig.returnLogProbs, streaming, outConfig.excludeInputFromOutput, + givenInputLengths.at(batchId), beamWidth, beamTokens, cumLogProbs, logProbs, batchId, + flakyTestInfo); + testData.validateContextLogits(outConfig.returnContextLogits, givenInputLengths.at(batchId), + beamWidth, contextLogits, vocabSizePadded, batchId, logitsAtol, logitsRtol); + testData.validateGenerationLogits(outConfig.returnGenerationLogits, result.isSequenceFinal, + streaming, outConfig.excludeInputFromOutput, givenInputLengths.at(batchId), + reqMaxNewTokens.at(batchId), beamWidth, beamTokens, genLogits, vocabSizePadded, batchId, + returnAllGeneratedTokens, logitsAtol, logitsRtol); + } + + // Ignore first iteration as it doesn't use draft tokens + if (outConfig.returnPerfMetrics && isSpeculativeDecoding + && result.requestPerfMetrics.value().iter > 0) + { + auto& specDecMetrics = result.requestPerfMetrics.value().speculativeDecoding; + // 4 draft tokens are used per step + EXPECT_EQ(specDecMetrics.totalDraftTokens, result.requestPerfMetrics.value().iter.value() * 4); + EXPECT_EQ(specDecMetrics.acceptanceRate, + static_cast<float>(specDecMetrics.totalAcceptedDraftTokens) + / specDecMetrics.totalDraftTokens); + } + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + ++iter; + } + EXPECT_LT(iter, maxWaitMs); + testData.verifyOutput(tokens, givenInputLengths, streaming, outConfig.excludeInputFromOutput, flakyTestInfo, + isSpeculativeDecoding, beamWidth, numSequences, isNonGreedySampling); + } +} + +void runTest(fs::path const& modelPath, ExecutorConfig const& executorConfig, fs::path const& inputPath, + ModelIds const& modelIds, FlakyTestInfo const& flakyTestInfo, bool streaming, SizeType32 const vocabSizePadded, + BeamResult const& beamResult, OutputConfig const& outConfig, bool isSpeculativeDecoding, int maxWaitMs, + bool returnAllGeneratedTokens, SizeType32 const numReturnSequences, bool isNonGreedySampling, + SizeType32 const modelParallelism) +{ + auto executor = Executor{modelPath, ModelType::kDECODER_ONLY, executorConfig}; + + runTest(executor, inputPath, modelIds, flakyTestInfo, streaming, vocabSizePadded, beamResult, outConfig, + isSpeculativeDecoding, maxWaitMs, returnAllGeneratedTokens, numReturnSequences, isNonGreedySampling, + modelParallelism); +} + +ExecutorConfig createExecutorConfig(SizeType32 maxBeamWidth, bool useOrchestratorMode, bool gatherGenerationLogits, + std::optional<std::vector<SizeType32>> deviceIds = std::nullopt, + std::optional<std::vector<SizeType32>> participantIds = std::nullopt) +{ + // Note: we reduce memory fraction for cases that return context/generation logits which require more free + // memory + FloatType constexpr freeGpuMemoryFraction{0.5F}; + KvCacheConfig kvCacheConfig(false, std::nullopt, std::nullopt, std::nullopt, freeGpuMemoryFraction); + auto executorConfig = ExecutorConfig(maxBeamWidth); + executorConfig.setKvCacheConfig(kvCacheConfig); + executorConfig.setNormalizeLogProbs(false); + executorConfig.setGatherGenerationLogits(gatherGenerationLogits); + + std::optional<OrchestratorConfig> orchestratorConfig = std::nullopt; + if (useOrchestratorMode) + { + orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + } + auto parallelConfig = ParallelConfig(CommunicationType::kMPI, + useOrchestratorMode ? CommunicationMode::kORCHESTRATOR : CommunicationMode::kLEADER, std::move(deviceIds), + std::move(participantIds), orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + return executorConfig; +} + +} // namespace + +TEST_P(AllParamsTest, TokenComparison) +{ + auto const streaming = std::get<0>(GetParam()); + auto const& beamWidth = std::get<1>(GetParam()); + OutputConfig outConfig; + outConfig.returnLogProbs = std::get<2>(GetParam()); + outConfig.excludeInputFromOutput = std::get<3>(GetParam()); + outConfig.returnContextLogits = std::get<4>(GetParam()); + outConfig.returnGenerationLogits = std::get<5>(GetParam()); + auto const modelName = std::get<6>(GetParam()); + auto const useOrchestratorMode = std::get<7>(GetParam()); + auto const returnAllGeneratedTokens = std::get<8>(GetParam()); + auto const numReturnSequences = std::get<9>(GetParam()); + if (returnAllGeneratedTokens && !streaming) + { + GTEST_SKIP() << "Test does not support returnAllGeneratedTokens without streaming"; + } + + std::optional<std::vector<SizeType32>> participantIds = std::nullopt; + + BeamResult beamResult{beamWidth}; + + ASSERT_TRUE(fs::exists(DATA_PATH)); + + fs::path modelPath; + // set defaults and adjust if needed by different models + fs::path inputPath = DATA_PATH / "input_tokens.npy"; + ModelIds modelIds{50256, 50256}; + bool isSpeculativeDecoding{false}; + + SizeType32 vocabSizePadded = 50257; + + // NOTE: This can be used to disable checks for certain prompt batch entries + FlakyTestInfo flakyTestInfo; + + if (modelName == "gpt") + { + auto const resultsPath + = GPT_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + if (outConfig.returnContextLogits || outConfig.returnGenerationLogits) + { + modelPath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR() / "tp1-pp1-cp1-gpu"; + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_RESULT_FILE(); + beamResult.contextLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_FILE(); + beamResult.genLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_FILE(); + if (outConfig.returnLogProbs) + { + beamResult.cumLogProbsFile + = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_CUM_LOG_PROBS_FILE(); + beamResult.logProbsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_LOG_PROBS_FILE(); + } + } + else + { + modelPath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_FILE(); + if (outConfig.returnLogProbs) + { + beamResult.cumLogProbsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_CUM_LOG_PROBS_FILE(); + beamResult.logProbsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_LOG_PROBS_FILE(); + } + } + } + else if (modelName == "llama_tp4_pp1_cp1" || modelName == "llama_tp1_pp4_cp1" || modelName == "llama_tp2_pp2_cp1" + || modelName == "llama_tp1_pp2_cp1") + { + inputPath = DATA_PATH / LLAMA_INPUT_FILE; + modelIds.padId = LLAMA_PAD_ID; + modelIds.endId = LLAMA_END_ID; + + vocabSizePadded = LLAMA_VOCAB_SIZE_PADDED; + + auto const resultsPath + = LLAMA_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + if (modelName == "llama_tp4_pp1_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP4_PP1_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp4_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP4_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp2_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP2_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp2-cp1-gpu"; + } + else if (modelName == "llama_tp2_pp2_cp1") + { + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP2_FILE(); + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + } + beamResult.genLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_TP4_PP1_FILE(); + if (outConfig.returnLogProbs) + { + beamResult.cumLogProbsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_CUM_LOG_PROBS_TP4_PP1_FILE(); + beamResult.logProbsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_LOG_PROBS_TP4_PP1_FILE(); + } + } + else if (modelName == "medusa") + { + TLLM_CHECK_WITH_INFO(beamWidth == 1, "Medusa does not support beam search."); + auto const resultsPath = MEDUSA_DATA_PATH / "sampling"; + auto modelSpec = ModelSpec::getDefaultModelSpec() + .useMedusa() + .setInputFile("input_tokens_long.npy") + .setMaxOutputLength(128); + beamResult.resultsFile = resultsPath / modelSpec.getResultsFile(); + modelPath = MEDUSA_MODEL_PATH / modelSpec.getModelPath() / "tp1-pp1-cp1-gpu"; + + inputPath = DATA_PATH / "input_vicuna.npy"; + modelIds.padId = 2; + modelIds.endId = 2; + isSpeculativeDecoding = true; + outConfig.returnPerfMetrics = true; + } + else if (modelName == "chatglm" || modelName == "chatglm2" || modelName == "chatglm3" || modelName == "glm") + { + fs::path resultsPath; + if (modelName == "chatglm") + { + resultsPath = CHATGLM_DATA_PATH; + modelPath = CHATGLM_MODEL_PATH; + } + else if (modelName == "chatglm2") + { + resultsPath = CHATGLM2_DATA_PATH; + modelPath = CHATGLM2_MODEL_PATH; + } + else if (modelName == "chatglm3") + { + resultsPath = CHATGLM3_DATA_PATH; + modelPath = CHATGLM3_MODEL_PATH; + } + else if (modelName == "glm") + { + resultsPath = GLM_DATA_PATH; + modelPath = GLM_MODEL_PATH; + } + resultsPath /= (beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth); + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_FILE(); + modelPath = modelPath / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + + char versionChatglm{0}; + if (size_t index = modelPath.string().find("chatglm"); index != std::string::npos) + { + versionChatglm = modelPath.string()[index + 7]; + std::string const vChatglmString + = (versionChatglm == '-') ? std::string("") : std::string(1, versionChatglm); + inputPath = DATA_PATH / ("input_tokens_chatglm" + vChatglmString + "-6b.npy"); + modelIds.padId = (versionChatglm == '-') ? 3 : 0; + modelIds.endId = (versionChatglm == '-') ? 130005 : 2; + } + else if (size_t index = modelPath.string().find("glm-10b"); index != std::string::npos) + { + inputPath = DATA_PATH / "input_tokens_glm-10b.npy"; + modelIds.padId = 50256; + modelIds.endId = 50258; + } + + if (versionChatglm != 0) + { + flakyTestInfo.batchIdBeams.insert(std::make_pair(1, 0)); + } + } + else + { + TLLM_THROW("Unrecognized modelName"); + } + + if (streaming && beamWidth > 1) + { + GTEST_SKIP() << "Test does not support streaming with beam search"; + } + + // Warning: This should be the last check before running the test. + // It will initialize MPI which can take significant time. + if (modelName == "llama_tp4_pp1_cp1" || modelName == "llama_tp1_pp4_cp1" || modelName == "llama_tp2_pp2_cp1" + || modelName == "llama_tp1_pp2_cp1") + { + // For llama model, only run for multiple GPUs + // This is detected by setting an env variable when running the test + char const* val = getenv("RUN_LLAMA_MULTI_GPU"); + if (val == nullptr) + { + GTEST_SKIP() << "Skipping Llama test"; + } + + if (outConfig.returnContextLogits) + { + GTEST_SKIP() << "Skipping context logits tests for mpi runs"; + } + + // Check that it was launched with right number of MPI ranks + if (!useOrchestratorMode && COMM_SESSION.getSize() != 4) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Leader mode and world size is not equal to 4"; + } + if (useOrchestratorMode && COMM_SESSION.getSize() != 1) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Orchestrator mode and World size is not equal to 1"; + } + } + auto decoderJsonConfig = tensorrt_llm::runtime::GptJsonConfig::parse(modelPath / "config.json"); + + auto const modelTP = decoderJsonConfig.getTensorParallelism(); + auto const modelPP = decoderJsonConfig.getPipelineParallelism(); + auto const modelParallelism = modelTP * modelPP; + int deviceCount = -1; + TLLM_CUDA_CHECK(cudaGetDeviceCount(&deviceCount)); + std::optional<std::vector<SizeType32>> deviceIds = std::vector<SizeType32>(modelParallelism); + for (auto i = 0; i < deviceIds->size(); i++) + { + deviceIds->at(i) = i % deviceCount; + } + if (modelName == "llama_tp1_pp2_cp1") + { + auto const& session = tensorrt_llm::mpi::MpiComm::world(); + if (session.getSize() != 4) + { + FAIL() << "Llama-tp1-pp2 is intended solely for testing coexisting engines within the same MPI world," + " which requires a session size of 4. However, the current session size is " + << session.getSize() << " ."; + } + if (session.getRank() / 2 == 0) + { + participantIds = std::vector<SizeType32>{0, 1}; + deviceIds = std::vector<SizeType32>{0, 1}; + } + else + { + participantIds = std::vector<SizeType32>{2, 3}; + deviceIds = std::vector<SizeType32>{2, 3}; + } + } + + if (modelPP > 1) + { + std::reverse(deviceIds->begin(), deviceIds->end()); + if (modelTP > 1) + { + for (SizeType32 ppRank = 0; ppRank < modelPP; ppRank++) + { + std::reverse(deviceIds->begin() + ppRank * modelTP, deviceIds->begin() + (ppRank + 1) * modelPP); + } + } + } + + // Returning logits will bring higher latency + if (streaming && (outConfig.returnContextLogits || outConfig.returnGenerationLogits)) + { + mMaxWaitMs = 20000; + } + + auto executorConfig = createExecutorConfig(beamWidth, useOrchestratorMode, outConfig.returnGenerationLogits, + std::move(deviceIds), std::move(participantIds)); + + runTest(modelPath, executorConfig, inputPath, modelIds, flakyTestInfo, streaming, vocabSizePadded, beamResult, + outConfig, isSpeculativeDecoding, mMaxWaitMs, returnAllGeneratedTokens, numReturnSequences, false, + modelParallelism); +} + +TEST_F(GptExecutorTest, ChangeBeamWidth) +{ + SizeType32 constexpr maxBeamWidth{2}; + auto executorConfig = ExecutorConfig(maxBeamWidth); + + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 constexpr beamWidth1{1}; + SizeType32 constexpr beamWidth2{2}; + SizeType32 constexpr maxNewTokens{2}; + VecTokens inputTokens{1, 2, 3, 4}; + + // Create requests with different beam widths + std::vector<Request> requests; + requests.emplace_back(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth1)); + requests.emplace_back(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth1)); + requests.emplace_back(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth2)); + requests.emplace_back(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth1)); + + auto requestIds = executor.enqueueRequests(requests); + + int numFinished = 0; + int iter = 0; + while (numFinished < 4 && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + auto err = response.getErrorMsg(); + std::cout << "err:" << err << std::endl; + FAIL() << "Should not get a response with error"; + } + else + { + auto result = response.getResult(); + numFinished += static_cast<int>(result.isFinal); + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + + auto stats = executor.getLatestIterationStats(); + uint64_t currentIter = 0; + for (auto const& stat : stats) + { + // TODO: enable this check when stats are cleaned + // EXPECT_EQ(stat.iter, currentIter); + if (stat.iter < 2) + { + // req 1 and 2 run with same beam width + EXPECT_EQ(stat.numActiveRequests, 2); + } + else if (stat.numActiveRequests != 0) // TODO: remove this check when stats are cleaned + { + // req 3 or 4 run width different beam width + EXPECT_EQ(stat.numActiveRequests, 1); + } + + ++currentIter; + } +} + +void doTokenComparisonChangeBeamWidth(bool enableReuse, SizeType32 maxWaitMs) +{ + SizeType32 constexpr maxBeamWidth{2}; + SizeType32 constexpr vocabSizePadded{50257}; // gpt vocabSizePadded + auto constexpr streaming = false; + + // Create executor config + auto kvCacheConfig = KvCacheConfig(enableReuse); + auto executorConfig = ExecutorConfig(maxBeamWidth, SchedulerConfig(), kvCacheConfig); + + // Create executor + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + auto const inputPath = DATA_PATH / "input_tokens.npy"; + ModelIds modelIds{50256, 50256}; + + OutputConfig outConfig; + FlakyTestInfo flakyTestInfo; + bool constexpr isSpeculativeDecoding{false}; + + for (SizeType32 beamWidth : {1, 2}) + { + TLLM_LOG_INFO("Running beam width: %d", beamWidth); + BeamResult beamResult{beamWidth}; + auto const resultsPath + = GPT_DATA_PATH / ((beamWidth == 1) ? "sampling" : "beam_search_" + std::to_string(beamWidth)); + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_RESULT_FILE(); + beamResult.contextLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_FILE(); + beamResult.genLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_FILE(); + + auto const numReturnSequences = beamWidth; + + runTest(executor, inputPath, modelIds, flakyTestInfo, streaming, vocabSizePadded, beamResult, outConfig, + isSpeculativeDecoding, maxWaitMs, false, numReturnSequences, false, 1); + } +} + +TEST_F(GptExecutorTest, TokenComparisonChangeBeamWidth) +{ + doTokenComparisonChangeBeamWidth(false, mMaxWaitMs); +} + +TEST_F(GptExecutorTest, TokenComparisonChangeBeamWidthBlockReuse) +{ + doTokenComparisonChangeBeamWidth(true, mMaxWaitMs); +} + +TEST_F(GptExecutorTest, NReturnRandomness) +{ + SizeType32 constexpr maxBeamWidth{1}; + SizeType32 constexpr numReturnSequences{2}; + SizeType32 constexpr vocabSizePadded{50257}; // gpt vocabSizePadded + auto constexpr streaming = false; + + // Create executor config + auto executorConfig = ExecutorConfig(maxBeamWidth); + + // Create executor + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + auto const inputPath = DATA_PATH / "input_tokens.npy"; + ModelIds modelIds{50256, 50256}; + + OutputConfig outConfig; + FlakyTestInfo flakyTestInfo; + bool constexpr isSpeculativeDecoding{false}; + + BeamResult beamResult{maxBeamWidth}; + auto const resultsPath = GPT_DATA_PATH / "sampling"; + beamResult.resultsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_RESULT_FILE(); + beamResult.contextLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_FILE(); + beamResult.genLogitsFile = resultsPath / PathUtil::FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_FILE(); + + runTest(executor, inputPath, modelIds, flakyTestInfo, streaming, vocabSizePadded, beamResult, outConfig, + isSpeculativeDecoding, mMaxWaitMs, false, 1, true, 1); +} + +TEST_F(GptExecutorTest, TimedOut) +{ + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // No requests enqueued, expect no responses + auto numResponsesReady = executor.getNumResponsesReady(); + EXPECT_EQ(numResponsesReady, 0); + + std::chrono::milliseconds waitTime(10); + auto responses = executor.awaitResponses(waitTime); + EXPECT_EQ(responses.size(), 0); +} + +TEST_F(GptExecutorTest, MaxSeqIdleMicrosecondsError) +{ + auto executorConfig = ExecutorConfig(1); + // Request will time out + executorConfig.setMaxSeqIdleMicroseconds(1); + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 constexpr maxNewTokens{5}; + VecTokens inputTokens{1, 2, 3, 4}; + + std::vector<Request> requests; + requests.emplace_back(inputTokens, maxNewTokens, false); + + auto requestIds = executor.enqueueRequests(requests); + + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + auto err = response.getErrorMsg(); + std::cout << "err:" << err << std::endl; + EXPECT_THAT(err, testing::HasSubstr("Unable to get batch slot for request ID")); + done = true; + } + else + { + FAIL() << "Should get a response with error"; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); +} + +void logitsProcessorMixedReqsTest(std::string const& modelDir, SizeType32 worldRank, SizeType32 maxWaitMs, + bool replicated, std::optional<std::vector<SizeType32>> deviceIds); + +TEST_P(LogitsProcParamsTest, All) +{ + auto const modelName = std::get<0>(GetParam()); + auto const batched = std::get<1>(GetParam()); + auto const replicated = std::get<2>(GetParam()); + + std::string modelDir; + int tp_size = 1, pp_size = 1, cp_size = 1; + std::optional<std::vector<SizeType32>> deviceIds = std::nullopt; + + if (modelName == "llama_tp1_pp1_cp1") + { + modelDir = "tp1-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp4_pp1_cp1") + { + modelDir = "tp4-pp1-cp1-gpu"; + tp_size = 4; + } + else if (modelName == "llama_tp1_pp4_cp1") + { + modelDir = "tp1-pp4-cp1-gpu"; + pp_size = 4; + deviceIds = std::vector<SizeType32>{3, 2, 1, 0}; + } + else if (modelName == "llama_tp2_pp2_cp1") + { + modelDir = "tp2-pp2-cp1-gpu"; + tp_size = pp_size = 2; + deviceIds = std::vector<SizeType32>{2, 3, 0, 1}; + } + else + { + TLLM_THROW("Unrecognized modelName"); + } + std::filesystem::path modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / modelDir; + + auto& comm = tensorrt_llm::mpi::MpiComm::world(); + auto const worldRank = comm.getRank(); + auto const worldSize = comm.getSize(); + + if (tp_size * pp_size * cp_size != 1) + { + // Run multi GPU test only when env variable is set + char const* val = getenv("RUN_LLAMA_MULTI_GPU"); + if (val == NULL) + { + GTEST_SKIP() << "Skipping multi-gpu logits post processor test"; + } + + if (worldSize != 4) + { + FAIL() << "Leader mode and world size is not equal to 4"; + } + } + else + { + // This has no effect for single-GPU tests + if (replicated) + { + GTEST_SKIP() << "Skipping single-gpu replicated logits post processor test"; + } + } + + // Configuration options + bool const streaming = false; + bool excludeInputFromOutput = false; + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + SizeType32 numRequests = 20; + IdType const kClientId = 1234; + + SizeType32 beamWidth = 1; + SizeType32 maxPromptLen = 20; + SizeType32 maxMaxNewTokens = 20; + + SizeType32 constexpr endId{2}; + SizeType32 constexpr vocabSizePadded{32000}; // llama-7b vocabSizePadded + // We just use tokenIdCalculator to generate a token_id based on request index, output position and max new tokens. + // Then LogitsPostProcessor set all other logits except the generated token_id to large negative value. + // So the output token should be the generated token by tokenIdCalculator. + auto tokenIdCalculator = [endId, vocabSizePadded](IdType req, SizeType32 pos) + { + SizeType32 tokenId = (req * 1000 + pos) % vocabSizePadded; + if (tokenId == endId) + { + tokenId = 0; + } + return tokenId; + }; + + std::unordered_map<IdType, VecTokens> tokens; + std::unordered_map<IdType, SizeType32> expectedNumTokens; + std::unordered_map<IdType, VecTokens> expectedOutputTokens; + + // Enqueue the requests + auto enqueueRequests = [&](Executor& executor, std::optional<std::string const> logitsProcessorName, + std::optional<LogitsPostProcessor> logitsProcessor = std::nullopt) + { + tokens.clear(); + expectedNumTokens.clear(); + expectedOutputTokens.clear(); + + for (SizeType32 req = 0; req < numRequests; ++req) + { + SizeType32 promptLen = rand() % maxPromptLen + 1; + SizeType32 maxNewTokens = rand() % maxMaxNewTokens + 1; + + auto request = Request(VecTokens(promptLen, 1), maxNewTokens, streaming, + tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig, endId); + request.setClientId(kClientId); + if (logitsProcessorName) + { + request.setLogitsPostProcessorName(logitsProcessorName.value()); + } + else if (logitsProcessor) + { + request.setLogitsPostProcessor(logitsProcessor.value()); + } + auto reqId = executor.enqueueRequest(std::move(request)); + tokens[reqId] = {}; + expectedNumTokens[reqId] = (streaming ? 0 : (excludeInputFromOutput ? 0 : promptLen)) + maxNewTokens; + expectedOutputTokens[reqId] = {}; + if (!streaming && !excludeInputFromOutput) + { + expectedOutputTokens[reqId].resize(promptLen, 1); + } + for (SizeType32 outputPos = 0; outputPos < maxNewTokens; ++outputPos) + { + SizeType32 outputTokenId = tokenIdCalculator(reqId, outputPos + promptLen); + expectedOutputTokens[reqId].push_back(outputTokenId); + } + } + }; + + // Get the new tokens for each requests + auto collectResponses = [&](Executor& executor) + { + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + while (numFinished < numRequests && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + EXPECT_EQ(response.getClientId().value(), kClientId); + auto result = response.getResult(); + numFinished += result.isFinal; + auto& newTokens = result.outputTokenIds.at(beamWidth - 1); + auto& reqTokens = tokens.at(response.getRequestId()); + reqTokens.insert(reqTokens.end(), std::make_move_iterator(newTokens.begin()), + std::make_move_iterator(newTokens.end())); + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + }; + + // Check that tokens matches expectations + auto checkOutput = [&]() + { + for (auto const& [reqId, numTokens] : expectedNumTokens) + { + EXPECT_EQ(expectedNumTokens[reqId], tokens[reqId].size()) << "reqId " << reqId; + for (SizeType32 tokenPos = 0; + tokenPos < std::min<SizeType32>(expectedNumTokens[reqId], tokens[reqId].size()); ++tokenPos) + { + EXPECT_EQ(expectedOutputTokens[reqId][tokenPos], tokens[reqId][tokenPos]) + << "reqId=" << reqId << ", tokenPos=" << tokenPos; + } + } + }; + + // Test non-batched logits processor + std::string const logitsProcessorName = "SelectToken"; + + auto logitsPostProcessorFn = [&](IdType reqId, Tensor& logits, BeamTokens const& tokens, StreamPtr const& streamPtr, + std::optional<IdType> clientId) + { + if (replicated) + { + EXPECT_TRUE(worldRank <= tp_size - 1); + } + else + { + EXPECT_TRUE(worldRank == 0); + } + EXPECT_TRUE(clientId.value() == kClientId); + SizeType32 numTokens = tokens.at(0).size(); + SizeType32 pos = numTokens; + SizeType32 outputTokenId = tokenIdCalculator(reqId, pos); + auto logitsDataType = logits.getDataType(); + EXPECT_TRUE(logitsDataType == DataType::kFP16 || logitsDataType == DataType::kBF16 + || logitsDataType == DataType::kFP32); + // logits has shape [draftLength + 1, reqBeamWidth, vocabSize] + auto logitsCpu = tensorrt_llm::executor::Tensor::cpu(logitsDataType, logits.getShape()); + auto* dataPtr = logitsCpu.getData(); + auto eltSize = logitsCpu.getSizeInBytes() / logitsCpu.getSize(); + EXPECT_TRUE(eltSize == 2 || eltSize == 4); + if (eltSize == 2) + { + auto* dataPtrU16 = static_cast<uint16_t*>(dataPtr); + uint16_t hugeNegValue = logitsDataType == DataType::kFP16 ? 0xFBFF : 0xFF7F; // a huge negative value + for (size_t i = 0; i < logitsCpu.getSize(); ++i) + { + dataPtrU16[i] = hugeNegValue; + } + dataPtrU16[outputTokenId] = 0; + } + else + { + auto* dataPtrFloat = static_cast<float*>(dataPtr); + for (size_t i = 0; i < logitsCpu.getSize(); ++i) + { + dataPtrFloat[i] = -HUGE_VALF; + } + dataPtrFloat[outputTokenId] = 0.0f; + } + + logits.setFrom(logitsCpu, streamPtr); + }; + + if (!batched) + { + auto executorConfig = ExecutorConfig(beamWidth); + LogitsPostProcessorConfig logitsProcConfig{ + std::unordered_map<std::string, tensorrt_llm::executor::LogitsPostProcessor>{ + {logitsProcessorName, logitsPostProcessorFn}}, + std::nullopt, replicated}; + executorConfig.setLogitsPostProcessorConfig(logitsProcConfig); + if (deviceIds.has_value()) + { + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + parallelConfig.setDeviceIds(deviceIds.value()); + executorConfig.setParallelConfig(parallelConfig); + } + auto executor = Executor(modelPath, ModelType::kDECODER_ONLY, executorConfig); + + if (worldRank == 0) + { + enqueueRequests(executor, logitsProcessorName); + collectResponses(executor); + checkOutput(); + + if (!replicated || tp_size == 1) + { + // Dynamic logits postprocessor must be used with replicate=false or no tensor parallelism. + enqueueRequests(executor, std::nullopt, logitsPostProcessorFn); + collectResponses(executor); + checkOutput(); + } + } + } + + // Test batched logits processor + auto logitsPostProcessorBatchedFn + = [logitsPostProcessorFn](std::vector<IdType> const& reqIdBatch, std::vector<Tensor>& logitsBatch, + std::vector<std::reference_wrapper<BeamTokens const>> const& tokensBatch, StreamPtr const& streamPtr, + std::vector<std::optional<IdType>> const& clientIdBatch) + { + for (int sample = 0; sample < reqIdBatch.size(); sample++) + { + logitsPostProcessorFn( + reqIdBatch[sample], logitsBatch[sample], tokensBatch[sample], streamPtr, clientIdBatch[sample]); + } + }; + + if (batched) + { + auto batchedExecutorConfig = ExecutorConfig(beamWidth); + if (deviceIds.has_value()) + { + auto parallelConfig = batchedExecutorConfig.getParallelConfig().value_or(ParallelConfig()); + + parallelConfig.setDeviceIds(deviceIds.value()); + batchedExecutorConfig.setParallelConfig(parallelConfig); + } + LogitsPostProcessorConfig logitsProcConfig{std::nullopt, logitsPostProcessorBatchedFn, replicated}; + batchedExecutorConfig.setLogitsPostProcessorConfig(logitsProcConfig); + + auto batchedExecutor = Executor(modelPath, ModelType::kDECODER_ONLY, batchedExecutorConfig); + + if (worldRank == 0) + { + enqueueRequests(batchedExecutor, Request::kBatchedPostProcessorName); + collectResponses(batchedExecutor); + checkOutput(); + } + } + + if (!batched) + { + logitsProcessorMixedReqsTest(modelDir, worldRank, mMaxWaitMs, replicated, std::move(deviceIds)); + } +} + +// Test for mixing requests with and without logits processor. +void logitsProcessorMixedReqsTest(std::string const& modelDir, SizeType32 worldRank, SizeType32 maxWaitMs, + bool replicated, std::optional<std::vector<SizeType32>> deviceIds) +{ + std::string const logitsProcessorName = "dummy"; + auto logitsPostProcessorFn = [&](IdType reqId, Tensor& logits, BeamTokens const& tokens, StreamPtr const& streamPtr, + std::optional<IdType> clientId) + { + // Dummy callback that does not modify logits + assert(!clientId.has_value()); + }; + + LogitsPostProcessorConfig logitsProcConfig{ + std::unordered_map<std::string, tensorrt_llm::executor::LogitsPostProcessor>{ + {logitsProcessorName, logitsPostProcessorFn}}, + std::nullopt, replicated}; + + // Create executor + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + executorConfig.setLogitsPostProcessorConfig(logitsProcConfig); + if (deviceIds.has_value()) + { + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + + parallelConfig.setDeviceIds(deviceIds.value()); + executorConfig.setParallelConfig(parallelConfig); + } + std::filesystem::path modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / modelDir; + auto executor = Executor(modelPath, ModelType::kDECODER_ONLY, executorConfig); + + if (worldRank == 0) + { + SizeType32 numRequests = 2; + SizeType32 promptLen = 5; + + // First request with no LP and many output tokens + auto request1 = Request(VecTokens(promptLen, 1), 25); + // Second request with LP and few output tokens + auto request2 = Request(VecTokens(promptLen, 1), 5); + request2.setLogitsPostProcessorName(logitsProcessorName); + + // Enqueue requests + auto reqId1 = executor.enqueueRequest(request1); + auto reqId2 = executor.enqueueRequest(request2); + + // Wait for responses + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + while (numFinished < numRequests && iter < maxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + EXPECT_EQ(response.getErrorMsg(), err); + } + } + ++iter; + } + EXPECT_LT(iter, maxWaitMs); + } +} + +TEST_F(GptExecutorTest, LogitsPostProcessorThrow) +{ + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + std::string const logitsProcessorName = "UnExistProcessor"; + + auto request + = Request(VecTokens(10, 1), 10, false, tensorrt_llm::executor::SamplingConfig(beamWidth), OutputConfig()); + request.setLogitsPostProcessorName(logitsProcessorName); + EXPECT_THROW({ auto reqId = executor.enqueueRequest(std::move(request)); }, tensorrt_llm::common::TllmException); +} + +static Response executeDraftRequest(Executor& executor) +{ + OutputConfig outputConfig; + outputConfig.returnGenerationLogits = true; + + // Create the request + SizeType32 maxNewTokens = 4; + VecTokens inputTokens{1, 2, 3, 4}; + + Request request{std::move(inputTokens), maxNewTokens}; + request.setOutputConfig(outputConfig); + + // Enqueue the request + auto requestId = executor.enqueueRequest(std::move(request)); + + // Wait for the response + auto responses = executor.awaitResponses(requestId); + + return responses.at(0); +} + +static Response executeTargetRequest(Executor& executor, Result const& draftResult) +{ + // Create the request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + + Request request{std::move(inputTokens), maxNewTokens}; + + VecTokens const& outputTokenIds = draftResult.outputTokenIds.at(0); + VecTokens draftTokens(outputTokenIds.end() - 4, outputTokenIds.end()); + + auto const& logitsInfo = draftResult.specDecFastLogitsInfo.value(); + auto logitsTensor = logitsInfo.toTensor(); + + ExternalDraftTokensConfig draftTokensConfig( + std::move(draftTokens), logitsTensor, std::nullopt /* acceptance threshold */, true /* fastLogits */); + request.setExternalDraftTokensConfig(draftTokensConfig); + + // Enqueue the request + auto requestId = executor.enqueueRequest(std::move(request)); + + // Wait for the response + auto responses = executor.awaitResponses(requestId); + + return responses.at(0); +} + +class SpeculativeDecodingTest : public GptExecutorTest +{ +}; + +TEST_F(SpeculativeDecodingTest, SpecDecFastLogits) +{ + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtDraftEnginePath + = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR() / "tp1-pp1-cp1-gpu"; + auto trtEnginePath + = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DRAFT_TOKENS_DIR() / "tp1-pp1-cp1-gpu"; + + FloatType freeGpuMemoryFraction = 0.3; + auto kvCacheConfig + = KvCacheConfig(true /* enableBlockReuse */, std::nullopt, std::nullopt, std::nullopt, freeGpuMemoryFraction); + executorConfig.setKvCacheConfig(kvCacheConfig); + + tensorrt_llm::mpi::initialize(tensorrt_llm::mpi::MpiThreadSupport::THREAD_MULTIPLE); + int const worldSize = tensorrt_llm::mpi::MpiComm::world().getSize(); + ASSERT_EQ(worldSize, 3); + int const myRank = tensorrt_llm::mpi::MpiComm::world().getRank(); + bool const isOrchestrator = (myRank == 0); + + auto orchestratorConfig + = OrchestratorConfig(isOrchestrator, "" /* workerExecutablePath */, nullptr, false /* spawnPrcesses */); + auto parallelConfig = ParallelConfig( + CommunicationType::kMPI, CommunicationMode::kORCHESTRATOR, std::nullopt, std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + auto specDecConfig = SpeculativeDecodingConfig(true /* fastLogits */); + executorConfig.setSpecDecConfig(specDecConfig); + + std::unique_ptr<Executor> draftExecutor; + std::unique_ptr<Executor> targetExecutor; + + if (isOrchestrator) + { + auto executorConfigDraft = executorConfig; + parallelConfig.setParticipantIds({1}); + executorConfigDraft.setParallelConfig(parallelConfig); + + draftExecutor = std::make_unique<Executor>(trtDraftEnginePath, ModelType::kDECODER_ONLY, executorConfigDraft); + + parallelConfig.setParticipantIds({2}); + executorConfig.setParallelConfig(parallelConfig); + + targetExecutor = std::make_unique<Executor>(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + } + else if (myRank == 1) // draft model process + { + parallelConfig.setParticipantIds({1}); + parallelConfig.setDeviceIds({0}); + executorConfig.setParallelConfig(parallelConfig); + executorConfig.setGatherGenerationLogits(true); + draftExecutor = std::make_unique<Executor>(trtDraftEnginePath, ModelType::kDECODER_ONLY, executorConfig); + } + else if (myRank == 2) // target model process + { + parallelConfig.setParticipantIds({2}); + parallelConfig.setDeviceIds({0}); + executorConfig.setParallelConfig(parallelConfig); + draftExecutor = std::make_unique<Executor>(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + } + + if (isOrchestrator) + { + auto response = executeDraftRequest(*draftExecutor); + ASSERT_FALSE(response.hasError()); + response = executeTargetRequest(*targetExecutor, response.getResult()); + ASSERT_FALSE(response.hasError()); + } +} + +TEST_F(GptExecutorTest, OrchestratorMaxQueueSize) +{ + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + SizeType32 maxQueueSize = 6; + ExecutorConfig executorConfig; + executorConfig.setMaxQueueSize(maxQueueSize); + auto orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + auto parallelConfig = ParallelConfig( + CommunicationType::kMPI, CommunicationMode::kORCHESTRATOR, std::nullopt, std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 100; + VecTokens inputTokens{1, 2, 3, 4}; + auto request = Request(inputTokens, maxNewTokens); + std::vector<IdType> requestIds; + auto numberOfRequests = maxQueueSize * 5; + requestIds.reserve(numberOfRequests); + + // Enqueue more requests than the queue can manage + for (int i = 0; i < numberOfRequests; i++) + { + auto requestId = executor.enqueueRequest(request); + requestIds.emplace_back(requestId); + } + + auto responseVectors = executor.awaitResponses(std::move(requestIds)); + bool failedWithFullQueue = false; + for (auto& responseVector : responseVectors) + { + for (auto& response : responseVector) + { + if (response.hasError()) + { + EXPECT_THAT(response.getErrorMsg(), + testing::HasSubstr("Maximum queue size of 6 has been reached, please try again later")); + failedWithFullQueue = true; + } + } + } + EXPECT_TRUE(failedWithFullQueue) << "Expected requests to fail due to maximum queue size reached"; + + // Wait for requests to get scheduled to free up space in queue + std::this_thread::sleep_for(std::chrono::milliseconds(maxQueueSize * 200)); + auto requestId = executor.enqueueRequest(std::move(request)); + auto responses = executor.awaitResponses(requestId); + for (auto& response : responses) + { + EXPECT_FALSE(response.hasError()); + } +} + +TEST_F(GptExecutorTest, SingleRequestInvalidInputs) +{ + bool streaming = true; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + + std::vector<std::string> expectedErrMsgs; + std::vector<Request> requests; + + // Invalid embedding bias shape + { + requests.emplace_back(inputTokens, maxNewTokens, streaming); + auto embeddingBias = Tensor::cpu(DataType::kFP32, {1}); + requests.back().setEmbeddingBias(embeddingBias); + expectedErrMsgs.emplace_back("embedding bias shape is not as expected"); + } + + for (auto req = 0; req < requests.size(); ++req) + { + auto& request = requests.at(req); + auto const& expectedErrMsg = expectedErrMsgs.at(req); + + auto requestId = executor.enqueueRequest(std::move(request)); + + // Try to get the new tokens + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + + auto err = response.getErrorMsg(); + EXPECT_THAT(err, testing::HasSubstr(expectedErrMsg)); + done = true; + } + else + { + FAIL() << "Expected an err: " << expectedErrMsg; + } + } + ++iter; + } + EXPECT_EQ(done, true); + } +} + +TEST_F(GptExecutorTest, ExecutorKVCacheManager) +{ + + bool streaming = true; + int numRequests = 3; + + SizeType32 beamWidth = 1; + SizeType32 maxNewTokens = 5; + auto executorConfig = ExecutorConfig(beamWidth); + auto kvCacheConfig = KvCacheConfig(true, 128); + kvCacheConfig.setEventBufferMaxSize(1024); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + auto kvCacheManager = *executor.getKVCacheEventManager(); + + // Created event should be available before any requests. + auto events = kvCacheManager->getLatestEvents(std::chrono::seconds(1)); + EXPECT_EQ(events.size(), 1); + EXPECT_TRUE(std::holds_alternative<KVCacheCreatedData>(events.front().data)); + + // Create requests + std::vector<Request> requests; + for (int request = 0; request < 3; request++) + { + VecTokens inputTokens; + for (int i = 0; i < 63; i++) + { + inputTokens.emplace_back(i + request); + } + requests.emplace_back(inputTokens, maxNewTokens, streaming); + } + + for (auto req = 0; req < requests.size(); ++req) + { + auto& request = requests.at(req); + + auto requestId = executor.enqueueRequest(std::move(request)); + + // Get the new tokens + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + // This request failed for some reason, get error msg + std::string errStr + = "Request id " + std::to_string(requestId) + " failed with err " + response.getErrorMsg(); + FAIL(); + } + else + { + auto result = response.getResult(); + done = result.isFinal; + if (done) + { + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + auto events = kvCacheManager->getLatestEvents(std::chrono::milliseconds(100)); + if (req == 0) + { + EXPECT_EQ(events.size(), 3); + + // Store the first context block + EXPECT_EQ(std::get<KVCacheStoredData>(events.front().data).parentHash, std::nullopt); + EXPECT_EQ(std::get<KVCacheStoredData>(events.front().data).blocks.size(), 1); + events.pop_front(); + // Store the second (now completed) context block and the partial decode block. + EXPECT_EQ(std::get<KVCacheStoredData>(events.front().data).blocks.size(), 1); + EXPECT_EQ(std::get<KVCacheStoredData>(events.back().data).blocks.size(), 1); + EXPECT_EQ(std::get<KVCacheStoredData>(events.front().data).blocks[0].blockHash, + std::get<KVCacheStoredData>(events.back().data).parentHash); + } + else + { + EXPECT_EQ(events.size(), 5); + + // Remove a block to make room for the second context block. On the second request, we need + // to remove 2 blocks. + EXPECT_EQ(std::get<KVCacheRemovedData>(events.front().data).blockHashes.size(), req); + events.pop_front(); + // Store the first filled context block + EXPECT_EQ(std::get<KVCacheStoredData>(events.front().data).blocks.size(), 1); + events.pop_front(); + // Remove a block for the decode phase + EXPECT_EQ(std::get<KVCacheRemovedData>(events.front().data).blockHashes.size(), 1); + events.pop_front(); + // Store the final context block and the decode block + EXPECT_EQ(std::get<KVCacheStoredData>(events.front().data).blocks.size(), 1); + events.pop_front(); + EXPECT_EQ(std::get<KVCacheStoredData>(events.front().data).blocks.size(), 1); + } + } + } + } + iter++; + } + EXPECT_EQ(done, true); + } +} + +TEST_F(GptExecutorTest, SingleRequestLora) +{ + bool streaming = true; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Load lora weights, config + auto manager = tr::BufferManager(std::make_shared<tr::CudaStream>()); + auto loraWeightsTensor + = std::shared_ptr(tr::utils::loadNpy(manager, LORA_WEIGHTS_FILE.string(), tr::MemoryType::kCPU)); + auto loraConfigTensor + = std::shared_ptr(tr::utils::loadNpy(manager, LORA_CONFIG_FILE.string(), tr::MemoryType::kCPU)); + + // Create the request + SizeType32 maxNewTokens = 5; + VecTokens inputTokens{1, 2, 3, 4}; + auto request = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig()); + auto loraConfig = LoraConfig(0, detail::ofITensor(loraWeightsTensor), detail::ofITensor(loraConfigTensor)); + request.setLoraConfig(loraConfig); + + // Enqueue the request + auto requestId = executor.enqueueRequest(std::move(request)); + + // Get the new tokens + VecTokens tokens; + bool done = false; + int iter = 0; + std::chrono::milliseconds waitTime(1); + while (!done && iter < mMaxWaitMs) + { + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + // This request failed for some reason, get error msg + std::string errStr + = "Request id " + std::to_string(requestId) + " failed with err " + response.getErrorMsg(); + FAIL(); + } + else + { + auto result = response.getResult(); + done = result.isFinal; + // Append tokens + auto& newTokens = result.outputTokenIds.at(beamWidth - 1); + tokens.insert( + tokens.end(), std::make_move_iterator(newTokens.begin()), std::make_move_iterator(newTokens.end())); + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); + EXPECT_EQ(tokens.size(), maxNewTokens); +} + +TEST_P(GuidedDecodingParamsTest, All) +{ + auto const modelName = std::get<0>(GetParam()); + std::filesystem::path enginePath; + std::filesystem::path tokenizerInfoPath; + int tp_size = 1, pp_size = 1, cp_size = 1; + std::optional<std::vector<SizeType32>> deviceIds = std::nullopt; + + if (modelName == "gpt") + { + enginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + tokenizerInfoPath = GPT_XGRAMMAR_TOKENIZER_INFO_PATH; + } + else if (modelName == "llama_tp1_pp1_cp1") + { + enginePath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + tokenizerInfoPath = LLAMA_XGRAMMAR_TOKENIZER_INFO_PATH; + } + else if (modelName == "llama_tp4_pp1_cp1") + { + enginePath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + tokenizerInfoPath = LLAMA_XGRAMMAR_TOKENIZER_INFO_PATH; + tp_size = 4; + } + else if (modelName == "llama_tp1_pp4_cp1") + { + enginePath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + tokenizerInfoPath = LLAMA_XGRAMMAR_TOKENIZER_INFO_PATH; + pp_size = 4; + deviceIds = std::vector<SizeType32>{3, 2, 1, 0}; + } + else if (modelName == "llama_tp2_pp2_cp1") + { + enginePath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + tokenizerInfoPath = LLAMA_XGRAMMAR_TOKENIZER_INFO_PATH; + tp_size = 2; + pp_size = 2; + deviceIds = std::vector<SizeType32>{2, 3, 0, 1}; + } + else + { + TLLM_THROW("Unrecognized modelName"); + } + + auto& comm = tensorrt_llm::mpi::MpiComm::world(); + auto const worldRank = comm.getRank(); + auto const worldSize = comm.getSize(); + + if (tp_size * pp_size * cp_size > 1) + { + // Run multi GPU test only when env variable is set + char const* val = getenv("RUN_LLAMA_MULTI_GPU"); + if (val == NULL) + { + GTEST_SKIP() << "Skipping multi-gpu guided decoding test"; + } + else + { + if (worldSize != 4) + { + FAIL() << "Leader mode and world size is not equal to 4"; + } + } + } + + bool streaming = false; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + + auto const tokenizerInfo = nlohmann::json::parse(std::ifstream{tokenizerInfoPath}); + auto const encodedVocab = tokenizerInfo["encoded_vocab"].template get<std::vector<std::string>>(); + auto const tokenizerStr = tokenizerInfo["tokenizer_str"].template get<std::string>(); + auto const stopTokenIds = tokenizerInfo["stop_token_ids"].template get<std::vector<TokenIdType>>(); + GuidedDecodingConfig guidedDecodingConfig( + GuidedDecodingConfig::GuidedDecodingBackend::kXGRAMMAR, encodedVocab, tokenizerStr, stopTokenIds); + executorConfig.setGuidedDecodingConfig(guidedDecodingConfig); + + if (deviceIds.has_value()) + { + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + + parallelConfig.setDeviceIds(deviceIds.value()); + executorConfig.setParallelConfig(parallelConfig); + } + auto executor = Executor(enginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the requests + VecTokens inputTokens; + if (modelName == "gpt") + { + inputTokens = {2061, 318, 352, 10, 16, 30, 23998, 39559, 287, 257, 8633, 287, 33918, 5794, 25, 220}; + } + else // llama + { + inputTokens = { + 128000, 62, 3923, 7037, 62, 16, 10, 16, 30, 62, 16533, 87710, 1265, 4404, 5356, 1265, 9643, 9132, 25, 62}; + } + SizeType32 maxNewTokens = 10; + SamplingConfig samplingConfig{}; + OutputConfig outputConfig{false, false, false, true}; + + std::vector<Request> requests; + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outputConfig, stopTokenIds[0]); + + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outputConfig, stopTokenIds[0]); + requests.back().setGuidedDecodingParams(GuidedDecodingParams(GuidedDecodingParams::GuideType::kJSON)); + + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outputConfig, stopTokenIds[0]); + std::string jsonSchema{ + R"({"properties": {"answer": {"title": "Answer", "type": "integer"}}, "required": ["answer"], "title": "Answer", "type": "object"})"}; + requests.back().setGuidedDecodingParams( + GuidedDecodingParams(GuidedDecodingParams::GuideType::kJSON_SCHEMA, jsonSchema)); + + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outputConfig, stopTokenIds[0]); + std::string regex{R"(\d+)"}; + requests.back().setGuidedDecodingParams(GuidedDecodingParams(GuidedDecodingParams::GuideType::kREGEX, regex)); + + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outputConfig, stopTokenIds[0]); + std::string ebnfGrammar{R"(root ::= [0-9]+)"}; + requests.back().setGuidedDecodingParams( + GuidedDecodingParams(GuidedDecodingParams::GuideType::kEBNF_GRAMMAR, ebnfGrammar)); + + std::vector<VecTokens> expectedOutputTokens; + if (modelName == "gpt") + { + expectedOutputTokens.push_back({1849, 7, 16, 10, 16, 8, 198, 16, 10, 16}); + expectedOutputTokens.push_back({90, 366, 3672, 1298, 366, 7554, 31780, 1600, 366, 12888}); + expectedOutputTokens.push_back({90, 366, 64, 77, 2032, 68, 81, 1, 1058, 352}); + expectedOutputTokens.push_back({25645, 25645, 25645, 25645, 25645, 25645, 25645, 25645, 25645, 25645}); + expectedOutputTokens.push_back({25645, 25645, 25645, 25645, 25645, 25645, 25645, 25645, 25645, 25645}); + } + else // llama + { + expectedOutputTokens.push_back({16, 10, 16, 28, 17, 198, 62, 3923, 7037, 62}); + expectedOutputTokens.push_back({5018, 16, 794, 330, 16, 498, 330, 17, 794, 330}); + expectedOutputTokens.push_back({5018, 9399, 794, 16, 92}); + expectedOutputTokens.push_back({16}); + expectedOutputTokens.push_back({16}); + } + + if (executor.canEnqueueRequests()) + { + // Enqueue the requests + auto reqIds = executor.enqueueRequests(std::move(requests)); + + // Get the responses + int numFinished = 0; + int iter = 0; + while (numFinished < 5 && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + auto reqId = response.getRequestId(); + if (response.hasError()) + { + // This request failed for some reason, get error msg + std::string errStr + = "Request id " + std::to_string(reqId) + " failed with err " + response.getErrorMsg(); + FAIL(); + } + else + { + auto result = response.getResult(); + auto& newTokens = result.outputTokenIds.at(0); + + int reqIdx = std::find(reqIds.begin(), reqIds.end(), reqId) - reqIds.begin(); + EXPECT_THAT(newTokens, ::testing::ElementsAreArray(expectedOutputTokens[reqIdx])); + } + numFinished++; + } + } + EXPECT_LT(iter, mMaxWaitMs); + EXPECT_EQ(numFinished, 5); + } +} + +TEST_F(GptExecutorTest, GuidedDecodingFailure) +{ + bool streaming = false; + + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + + std::vector<int> stopTokenIds{50256}; + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the requests + SizeType32 maxNewTokens = 10; + SamplingConfig samplingConfig{}; + OutputConfig outputConfig{false, false, false, true}; + VecTokens inputTokens{2061, 318, 352, 10, 16, 30, 23998, 39559, 287, 257, 8633, 287, 33918, 5794, 25, 220}; + + std::vector<Request> requests; + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outputConfig, stopTokenIds[0]); + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outputConfig, stopTokenIds[0]); + requests.back().setGuidedDecodingParams(GuidedDecodingParams(GuidedDecodingParams::GuideType::kJSON)); + + // Enqueue the requests + auto reqIds = executor.enqueueRequests(std::move(requests)); + + // Get the responses + int numFinished = 0; + int iter = 0; + while (numFinished < 2 && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + auto reqId = response.getRequestId(); + int reqIdx = std::find(reqIds.begin(), reqIds.end(), reqId) - reqIds.begin(); + if (reqIdx == 0) + { + EXPECT_FALSE(response.hasError()); + } + else + { + EXPECT_TRUE(response.hasError()); + } + numFinished++; + } + } + EXPECT_LT(iter, mMaxWaitMs); + EXPECT_EQ(numFinished, 2); +} + +TEST_P(ParamTest, SingleRequestCancelRequest) +{ + bool const streaming = std::get<0>(GetParam()); + bool const excludeInputFromOutput = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + OutputConfig outConfig; + outConfig.excludeInputFromOutput = excludeInputFromOutput; + + auto executorConfig = ExecutorConfig(beamWidth); + auto trtEnginePath = GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 300; + VecTokens inputTokens{1, 2, 3, 4}; + auto request + = Request(inputTokens, maxNewTokens, streaming, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + + auto requestId = executor.enqueueRequest(std::move(request)); + + std::this_thread::sleep_for(std::chrono::milliseconds(100)); + executor.cancelRequest(requestId); + + // Try to get the new tokens + bool done = false; + int iter = 0; + VecTokens tokens; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(requestId, waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + FAIL() << "Did not expect errors"; + } + else + { + auto result = response.getResult(); + done = result.isFinal; + // Append tokens + auto& newTokens = result.outputTokenIds.at(beamWidth - 1); + if (done) + { + for (SizeType32 beamIdx = 0; beamIdx < beamWidth; ++beamIdx) + { + EXPECT_EQ(result.finishReasons[beamIdx], FinishReason::kCANCELLED); + } + } + + if (streaming && beamWidth > 1) + { + tokens = newTokens; + } + else + { + tokens.insert(tokens.end(), newTokens.begin(), newTokens.end()); + } + } + } + ++iter; + } + EXPECT_EQ(done, true); + EXPECT_LT(iter, mMaxWaitMs); + auto expectedNumTokens + = streaming ? maxNewTokens : (excludeInputFromOutput ? 0 : inputTokens.size()) + maxNewTokens; + TLLM_LOG_INFO("num tokens: %d, expected %d", tokens.size(), expectedNumTokens); + EXPECT_LT(tokens.size(), expectedNumTokens); +} + +TEST_F(GptExecutorTest, orchModeFetchNewReqErr) +{ + SizeType32 beamWidth = 1; + auto executorConfig = ExecutorConfig(beamWidth); + + auto orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + auto parallelConfig = ParallelConfig( + CommunicationType::kMPI, CommunicationMode::kORCHESTRATOR, std::nullopt, std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Create a req with invalid parameters + SizeType32 maxNewTokens = 5; + // Create very long prompt which should result in error during request validate + VecTokens inputTokens(10000000); + + auto request = Request(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth)); + auto requestId = executor.enqueueRequest(request); + auto requestId2 = executor.enqueueRequest(request); + + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + auto err = response.getErrorMsg(); + EXPECT_THAT(err, testing::HasSubstr("exceeds maximum input length")); + EXPECT_THAT(err, testing::HasSubstr("Encountered an error when fetching new request:")); + done = true; + } + else + { + FAIL() << "Should get a response with error"; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); +} + +TEST_F(GptExecutorTest, orchModeForwardError) +{ + SizeType32 constexpr maxBeamWidth{1}; + auto executorConfig = ExecutorConfig(maxBeamWidth); + + auto orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + auto parallelConfig = ParallelConfig( + CommunicationType::kMPI, CommunicationMode::kORCHESTRATOR, std::nullopt, std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + + // Setting request beam width to 2 which should cause failure + SizeType32 constexpr beamWidth{2}; + SizeType32 constexpr maxNewTokens{5}; + VecTokens inputTokens{1, 2, 3, 4}; + + auto request = Request(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth)); + auto requestId = executor.enqueueRequest(request); + auto requestId2 = executor.enqueueRequest(request); + + bool done = false; + int iter = 0; + while (!done && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + if (response.hasError()) + { + auto err = response.getErrorMsg(); + std::cout << "err:" << err << std::endl; + EXPECT_THAT( + err, testing::HasSubstr("Requested beam width 2 is larger than configured max beam width 1")); + done = true; + } + else + { + FAIL() << "Should get a response with error"; + } + } + ++iter; + } + EXPECT_LT(iter, mMaxWaitMs); +} + +TEST_P(ParamCancelReqTest, MultipleRequestsMultiGpuCancelRequest) +{ + auto const useOrchestratorMode = std::get<0>(GetParam()); + auto const beamWidth = std::get<1>(GetParam()); + auto const modelName = std::get<2>(GetParam()); + + std::optional<std::vector<SizeType32>> deviceIds = std::nullopt; + + OutputConfig outConfig; + + auto executorConfig = ExecutorConfig(beamWidth); + std::filesystem::path modelPath; + if (modelName == "llama_tp4_pp1_cp1" || modelName == "llama_tp1_pp4_cp1" || modelName == "llama_tp2_pp2_cp1") + { + if (modelName == "llama_tp4_pp1_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp4_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + deviceIds = std::vector<SizeType32>{3, 2, 1, 0}; + } + else if (modelName == "llama_tp2_pp2_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + deviceIds = std::vector<SizeType32>{2, 3, 0, 1}; + } + } + + // For llama model, only run for multiple GPUs + // This is detected by setting an env variable when running the test + char const* val = getenv("RUN_LLAMA_MULTI_GPU"); + if (val == NULL) + { + GTEST_SKIP() << "Skipping Llama test"; + } + else + { + // Check that it was launched with right number of MPI ranks + if (!useOrchestratorMode && COMM_SESSION.getSize() != 4) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Leader mode and world size is not equal to 4"; + } + else if (useOrchestratorMode && COMM_SESSION.getSize() != 1) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Orchestrator mode and World size is not equal to 1"; + } + } + + if (useOrchestratorMode) + { + auto orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + auto parallelConfig = ParallelConfig(CommunicationType::kMPI, + useOrchestratorMode ? CommunicationMode::kORCHESTRATOR : CommunicationMode::kLEADER, std::nullopt, + std::nullopt, orchestratorConfig); + if (deviceIds.has_value()) + { + parallelConfig.setDeviceIds(deviceIds.value()); + } + executorConfig.setParallelConfig(parallelConfig); + } + else + { + if (deviceIds.has_value()) + { + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + parallelConfig.setDeviceIds(deviceIds.value()); + executorConfig.setParallelConfig(parallelConfig); + } + } + + auto executor = Executor(modelPath, ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + SizeType32 maxNewTokens = 50; + VecTokens inputTokens{1, 2, 3, 4}; + + std::vector<Request> requests; + for (auto streaming : {false, true}) + { + // Add two requests with numReturnSequences = 1 + auto samplingConfig = tensorrt_llm::executor::SamplingConfig(beamWidth); + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outConfig); + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig, outConfig); + // Add a request with numReturnSequences > 1 + auto samplingConfig2 = tensorrt_llm::executor::SamplingConfig(beamWidth); + auto constexpr numReturnSequences = 2; + samplingConfig2.setNumReturnSequences(numReturnSequences); + requests.emplace_back(inputTokens, maxNewTokens, streaming, samplingConfig2, outConfig); + } + std::vector<bool> cancelRequests{true, false, true, true, false, true}; + + if (executor.canEnqueueRequests()) + { + auto const requestIds = executor.enqueueRequests(requests); + + // Cancel the first and third requests + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + for (SizeType32 i = 0; i < requests.size(); i++) + { + if (cancelRequests.at(i)) + { + executor.cancelRequest(requestIds.at(i)); + } + } + + std::unordered_map<IdType, bool> isStreaming; + std::unordered_map<IdType, SizeType32> expectedNumTokens; + SizeType32 expectedNumResponses = 0; + for (SizeType32 i = 0; i < requests.size(); i++) + { + auto const& request = requests.at(i); + auto requestId = requestIds.at(i); + isStreaming[requestId] = request.getStreaming(); + expectedNumTokens[requestId] = (request.getStreaming() ? 0 : inputTokens.size()) + maxNewTokens; + auto const numResponses = request.getStreaming() ? expectedNumTokens[requestId] : 1; + auto const numReturnSequences = request.getSamplingConfig().getBeamWidth() > 1 + ? 1 + : request.getSamplingConfig().getNumReturnSequences().value_or(1); + expectedNumResponses += numResponses * numReturnSequences; + } + + std::unordered_map<IdType, std::unordered_map<SizeType32, VecTokens>> tokens; + + // Get the new tokens for each requests + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + while (numFinished < requests.size() && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + auto requestId = response.getRequestId(); + auto result = response.getResult(); + numFinished += result.isFinal; + auto seqIdx = result.sequenceIndex; + auto numSequences = result.outputTokenIds.size(); + auto& newTokens = result.outputTokenIds.at(numSequences - 1); + auto& reqResults = tokens[response.getRequestId()]; + auto& reqTokens = reqResults[seqIdx]; + if (isStreaming.at(requestId) && beamWidth > 1) + { + reqTokens = newTokens; + } + else + { + reqTokens.insert(reqTokens.end(), newTokens.begin(), newTokens.end()); + } + } + else + { + FAIL() << "Did not expect errors"; + } + } + ++iter; + } + + EXPECT_LE(numResponses, expectedNumResponses); + EXPECT_EQ(numFinished, requests.size()); + EXPECT_LT(iter, mMaxWaitMs); + + for (auto requestIdx = 0; requestIdx < requests.size(); requestIdx++) + { + auto const requestId = requestIds.at(requestIdx); + for (auto seqIdx = 0; seqIdx < tokens.at(requestId).size(); seqIdx++) + { + auto const& seqTokens = tokens.at(requestId).at(seqIdx); + if (cancelRequests.at(requestIdx)) + { + EXPECT_LT(seqTokens.size(), expectedNumTokens.at(requestId)); + } + else + { + EXPECT_EQ(seqTokens.size(), expectedNumTokens.at(requestId)); + } + } + } + } +} + +TEST_P(LeaderApiUsageTest, LeaderModeTest) +{ + auto const modelName = std::get<0>(GetParam()); + + SizeType32 beamWidth = 2; + OutputConfig outConfig; + std::optional<std::vector<SizeType32>> deviceIds = std::nullopt; + + auto executorConfig = ExecutorConfig(beamWidth); + std::filesystem::path modelPath; + if (modelName == "llama_tp4_pp1_cp1" || modelName == "llama_tp1_pp4_cp1" || modelName == "llama_tp2_pp2_cp1") + { + if (modelName == "llama_tp4_pp1_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp4_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + deviceIds = std::vector<SizeType32>{3, 2, 1, 0}; + } + else if (modelName == "llama_tp2_pp2_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + deviceIds = std::vector<SizeType32>{2, 3, 0, 1}; + } + } + + // For llama model, only run for multiple GPUs + // This is detected by setting an env variable when running the test + char const* val = getenv("RUN_LLAMA_MULTI_GPU"); + if (val == NULL) + { + GTEST_SKIP() << "Skipping Llama test"; + } + else + { + // Check that it was launched with right number of MPI ranks + if (COMM_SESSION.getSize() != 4) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Leader mode and world size is not equal to 4"; + } + } + + if (deviceIds.has_value()) + { + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + + parallelConfig.setDeviceIds(deviceIds.value()); + executorConfig.setParallelConfig(parallelConfig); + } + auto executor = Executor(modelPath, ModelType::kDECODER_ONLY, executorConfig); + + // Since this is leader mode, all ranks should participate + EXPECT_TRUE(executor.isParticipant()); + + // Create the request + SizeType32 maxNewTokens = 50; + VecTokens inputTokens{1, 2, 3, 4}; + auto request + = Request(inputTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + auto requestStreaming + = Request(inputTokens, maxNewTokens, true, tensorrt_llm::executor::SamplingConfig(beamWidth), outConfig); + + // Leader enqueues requests and wait for responses + if (executor.canEnqueueRequests()) + { + auto requestId = executor.enqueueRequest(request); + auto requestId2 = executor.enqueueRequest(request); + auto requestId3 = executor.enqueueRequest(requestStreaming); + auto requestId4 = executor.enqueueRequest(requestStreaming); + + int32_t numFinished = 0; + int iter = 0; + SizeType32 numResponses = 0; + while (numFinished < 4 && iter < mMaxWaitMs) + { + std::chrono::milliseconds waitTime(1); + auto responses = executor.awaitResponses(waitTime); + for (auto& response : responses) + { + numResponses++; + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + } + else + { + FAIL() << "Did not expect errors"; + } + } + ++iter; + } + EXPECT_EQ(numFinished, 4); + EXPECT_LT(iter, mMaxWaitMs); + } + else + { + // Check that non-leader cannot enqueue requests + EXPECT_THROW({ auto reqId = executor.enqueueRequest(request); }, tensorrt_llm::common::TllmException); + EXPECT_THROW({ auto responses = executor.awaitResponses(); }, tensorrt_llm::common::TllmException); + EXPECT_THROW({ auto numResp = executor.getNumResponsesReady(); }, tensorrt_llm::common::TllmException); + EXPECT_THROW({ executor.cancelRequest(1); }, tensorrt_llm::common::TllmException); + EXPECT_THROW({ auto stats = executor.getLatestIterationStats(); }, tensorrt_llm::common::TllmException); + EXPECT_THROW({ auto stats = executor.getLatestRequestStats(); }, tensorrt_llm::common::TllmException); + } +} + +TEST_F(GptExecutorTest, validateParallelConfig) +{ + + auto trtEnginePath = (GPT_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"); + { + auto executorConfig = ExecutorConfig(); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + } + + { + std::string expectedErrMsg = "OrchestratorConfig must be set"; + try + { + auto executorConfig = ExecutorConfig(); + auto parallelConfig = ParallelConfig(CommunicationType::kMPI, CommunicationMode::kORCHESTRATOR); + executorConfig.setParallelConfig(parallelConfig); + auto executor = Executor(trtEnginePath, ModelType::kDECODER_ONLY, executorConfig); + FAIL() << "Expected TllmException"; + } + catch (tc::TllmException& e) + { + EXPECT_THAT(e.what(), testing::HasSubstr(expectedErrMsg)); + } + catch (std::exception const& e) + { + FAIL() << "Expected TllmException"; + } + } +} + +TEST_P(TimeoutTest, TimeoutStreamingTest) +{ + auto const modelName = std::get<0>(GetParam()); + auto const useOrchestratorMode = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + + auto executorConfig = ExecutorConfig(beamWidth); + std::filesystem::path modelPath; + bool isMultiGpu{false}; + std::optional<std::vector<SizeType32>> deviceIds = std::nullopt; + + if (modelName == "llama_tp4_pp1_cp1" || modelName == "llama_tp1_pp4_cp1" || modelName == "llama_tp2_pp2_cp1") + { + isMultiGpu = true; + if (modelName == "llama_tp4_pp1_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp4_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + deviceIds = std::vector<SizeType32>{3, 2, 1, 0}; + } + else if (modelName == "llama_tp2_pp2_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + deviceIds = std::vector<SizeType32>{2, 3, 0, 1}; + } + } + if (modelName == "llama_tp1_pp1_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + } + // For llama model, only run for multiple GPUs + // This is detected by setting an env variable when running the test + char const* val = getenv("RUN_LLAMA_MULTI_GPU"); + if (val == NULL && isMultiGpu) + { + GTEST_SKIP() << "Skipping MultiGpu tests"; + } + if (val != NULL && !isMultiGpu) + { + GTEST_SKIP() << "Skipping SingleGpu tests"; + } + if (val != NULL && isMultiGpu) + { + // Check that it was launched with right number of MPI ranks + if (!useOrchestratorMode && COMM_SESSION.getSize() != 4) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Leader mode and world size is not equal to 4"; + } + if (useOrchestratorMode && COMM_SESSION.getSize() != 1) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Orchestrator mode and World size is not equal to 1"; + } + } + + if (useOrchestratorMode) + { + auto orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + auto parallelConfig = ParallelConfig(CommunicationType::kMPI, + useOrchestratorMode ? CommunicationMode::kORCHESTRATOR : CommunicationMode::kLEADER, std::nullopt, + std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + if (deviceIds.has_value()) + { + parallelConfig.setDeviceIds(deviceIds.value()); + } + executorConfig.setParallelConfig(parallelConfig); + } + else + { + if (deviceIds.has_value()) + { + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + parallelConfig.setDeviceIds(deviceIds.value()); + executorConfig.setParallelConfig(parallelConfig); + } + } + auto executor = Executor(modelPath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 constexpr maxNewTokens = 10; + // create 1 request that times out immediately + // momentarily we don't cancel requests before forwardAsync so it will get scheduled for at least 1 forward + VecTokens immediateCancelTokens{1, 2, 3, 4}; + auto immediateCancelRequest + = Request(immediateCancelTokens, maxNewTokens, true, tensorrt_llm::executor::SamplingConfig(beamWidth)); + immediateCancelRequest.setReturnAllGeneratedTokens(true); + immediateCancelRequest.setAllottedTimeMs(std::chrono::milliseconds(0)); + SizeType32 constexpr immediateCancelMinLength = 0; + SizeType32 constexpr immediateCancelMaxLength = 1; + + // create 1 request that times out during the first forward + VecTokens oneForwardTokens{11, 12, 13, 14}; + auto oneForwardRequest + = Request(oneForwardTokens, maxNewTokens, true, tensorrt_llm::executor::SamplingConfig(beamWidth)); + oneForwardRequest.setReturnAllGeneratedTokens(true); + oneForwardRequest.setAllottedTimeMs(std::chrono::milliseconds(1)); + SizeType32 constexpr oneForwardlMinLength = 0; + SizeType32 constexpr oneForwardlMaxLength = 1; + + // Create the request that finishes by the number of tokens + VecTokens finishedTokens{101, 102, 103, 104}; + auto finishedRequest + = Request(finishedTokens, maxNewTokens, true, tensorrt_llm::executor::SamplingConfig(beamWidth)); + finishedRequest.setReturnAllGeneratedTokens(true); + finishedRequest.setAllottedTimeMs(std::chrono::milliseconds(5000)); + SizeType32 constexpr finishedMinLength = 5; + SizeType32 constexpr finishedMaxLength = maxNewTokens; + + std::vector<FinishReason> referenceFinishReasons + = {FinishReason::kTIMED_OUT, FinishReason::kTIMED_OUT, FinishReason::kLENGTH}; + std::vector<SizeType32> minLengths = {immediateCancelMinLength, oneForwardlMinLength, finishedMinLength}; + std::vector<SizeType32> maxLengths = {immediateCancelMaxLength, oneForwardlMaxLength, finishedMaxLength}; + // workaround because the last response will be empty, but we want to have at least *some* responses surpass the + // minLength + std::vector<SizeType32> achievedLength = {0, 0, 0}; + SizeType32 itNr{0}; + + if (executor.canEnqueueRequests()) + { + + std::vector<Request> requests = {immediateCancelRequest, oneForwardRequest, finishedRequest}; + auto requestIds = executor.enqueueRequests(requests); + + auto numFinished = 0; + + while (numFinished < static_cast<SizeType32>(requests.size())) + { + itNr++; + std::chrono::milliseconds waitTime(mMaxWaitMs); + auto responses = executor.awaitResponses(requestIds, waitTime); + for (auto const& response : responses) + { + for (auto const& responseIt : response) + { + auto const reqId = responseIt.getRequestId(); + if (responseIt.hasError()) + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err + = "ReqId " + std::to_string(reqId) + " has already been processed and was terminated."; + if (responseIt.getErrorMsg() != err) + { + TLLM_THROW("Request id %lu encountered error: %s", reqId, responseIt.getErrorMsg().c_str()); + } + continue; + } + + auto const& result = responseIt.getResult(); + if (result.isFinal) + { + requestIds.erase(std::remove(requestIds.begin(), requestIds.end(), reqId), requestIds.end()); + numFinished++; + } + + auto const finishReason = result.finishReasons; + auto const actualResponse = result.outputTokenIds; + TLLM_LOG_DEBUG("reqId %d finished %d", reqId, result.isFinal); + TLLM_LOG_DEBUG("actual response:"); + + for (auto const& beam : actualResponse) + { + std::string tokenStr; + for (auto tok : beam) + { + tokenStr += std::to_string(tok) + " "; + } + TLLM_LOG_DEBUG("%s", tokenStr.c_str()); + } + + TLLM_LOG_DEBUG( + "beams' length must be in range [%d, %d]", minLengths[reqId - 1], maxLengths[reqId - 1]); + + if (result.isFinal) + { + TLLM_LOG_DEBUG("finishReason"); + std::string reasonStr; + for (auto const reason : finishReason) + { + // cast for easier visibility during debugging + EXPECT_EQ(static_cast<int>(reason), static_cast<int>(referenceFinishReasons[reqId - 1])); + reasonStr += std::to_string(static_cast<int>(reason)) + " "; + } + TLLM_LOG_DEBUG("%s", reasonStr.c_str()); + } + + EXPECT_EQ(beamWidth, actualResponse.size()); + for (int beam = 0; beam < beamWidth; beam++) + { + EXPECT_LE(actualResponse.at(beam).size(), maxLengths[reqId - 1]) << "for request " << reqId; + achievedLength[reqId - 1] = std::max( + achievedLength[reqId - 1], static_cast<SizeType32>(actualResponse.at(beam).size())); + } + } + } + } + + for (int reqIt = 0; reqIt < achievedLength.size(); ++reqIt) + { + EXPECT_GE(achievedLength[reqIt], minLengths[reqIt]) + << "request " << reqIt + 1 << " has not achieved min lengths"; + } + } +} + +TEST_P(TimeoutTest, TimeoutNonstreamingTest) +{ + auto const modelName = std::get<0>(GetParam()); + auto const useOrchestratorMode = std::get<1>(GetParam()); + auto const beamWidth = std::get<2>(GetParam()); + + std::optional<std::vector<SizeType32>> deviceIds = std::nullopt; + + auto executorConfig = ExecutorConfig(beamWidth); + std::filesystem::path modelPath; + bool isMultiGpu{false}; + if (modelName == "llama_tp4_pp1_cp1" || modelName == "llama_tp1_pp4_cp1" || modelName == "llama_tp2_pp2_cp1") + { + isMultiGpu = true; + if (modelName == "llama_tp4_pp1_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp4-pp1-cp1-gpu"; + } + else if (modelName == "llama_tp1_pp4_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp4-cp1-gpu"; + deviceIds = std::vector<SizeType32>{3, 2, 1, 0}; + } + else if (modelName == "llama_tp2_pp2_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp2-pp2-cp1-gpu"; + deviceIds = std::vector<SizeType32>{2, 3, 0, 1}; + } + } + if (modelName == "llama_tp1_pp1_cp1") + { + modelPath = LLAMA_MODEL_PATH / PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() / "tp1-pp1-cp1-gpu"; + } + // For llama model, only run for multiple GPUs + // This is detected by setting an env variable when running the test + char const* val = getenv("RUN_LLAMA_MULTI_GPU"); + if (val == NULL && isMultiGpu) + { + GTEST_SKIP() << "Skipping MultiGpu tests"; + } + if (val != NULL && !isMultiGpu) + { + GTEST_SKIP() << "Skipping SingleGpu tests"; + } + if (val != NULL && isMultiGpu) + { + // Check that it was launched with right number of MPI ranks + if (!useOrchestratorMode && COMM_SESSION.getSize() != 4) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Leader mode and world size is not equal to 4"; + } + if (useOrchestratorMode && COMM_SESSION.getSize() != 1) + { + // No orchestrator, need worldSize to match TP*PP + FAIL() << "Orchestrator mode and World size is not equal to 1"; + } + } + + if (useOrchestratorMode) + { + auto orchestratorConfig = OrchestratorConfig(true, PathUtil::EXECUTOR_WORKER_PATH()); + auto parallelConfig = ParallelConfig(CommunicationType::kMPI, + useOrchestratorMode ? CommunicationMode::kORCHESTRATOR : CommunicationMode::kLEADER, std::nullopt, + std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + if (deviceIds.has_value()) + { + parallelConfig.setDeviceIds(deviceIds.value()); + } + executorConfig.setParallelConfig(parallelConfig); + } + else + { + if (deviceIds.has_value()) + { + auto parallelConfig = executorConfig.getParallelConfig().value_or(ParallelConfig()); + parallelConfig.setDeviceIds(deviceIds.value()); + executorConfig.setParallelConfig(parallelConfig); + } + } + auto executor = Executor(modelPath, ModelType::kDECODER_ONLY, executorConfig); + + SizeType32 constexpr maxNewTokens = 5; + // create 1 request that times out immediately + // momentarily we don't cancel requests before forwardAsync so it will get scheduled for at least 1 forward + VecTokens immediateCancelTokens{1, 2, 3, 4}; + auto immediateCancelRequest + = Request(immediateCancelTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth)); + immediateCancelRequest.setAllottedTimeMs(std::chrono::milliseconds(0)); + std::vector<std::vector<int>> immediateCancelResponse = {immediateCancelTokens, immediateCancelTokens}; + + // create 1 request that times out during the first forward + VecTokens oneForwardTokens{11, 12, 13, 14}; + auto oneForwardRequest + = Request(oneForwardTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth)); + oneForwardRequest.setAllottedTimeMs(std::chrono::milliseconds(1)); + std::vector<std::vector<int>> oneForwardResponse = {oneForwardTokens, oneForwardTokens}; + + // Create the request that finishes by the number of tokens + VecTokens finishedTokens{101, 102, 103, 104}; + auto finishedRequest + = Request(finishedTokens, maxNewTokens, false, tensorrt_llm::executor::SamplingConfig(beamWidth)); + finishedRequest.setAllottedTimeMs(std::chrono::milliseconds(6000)); + std::vector<std::vector<int>> finishedReponse + = {{101, 102, 103, 104, 49849, 225, 49849, 232, 55742}, {101, 102, 103, 104, 49849, 225, 49849, 232, 29082}}; + + // assume responses will come in FIFO order + std::vector<BeamTokens> refResponses = {immediateCancelResponse, oneForwardResponse, finishedReponse}; + std::vector<FinishReason> referenceFinishReasons + = {FinishReason::kTIMED_OUT, FinishReason::kTIMED_OUT, FinishReason::kLENGTH}; + if (executor.canEnqueueRequests()) + { + + std::vector<Request> requests = {immediateCancelRequest, oneForwardRequest, finishedRequest}; + auto requestIds = executor.enqueueRequests(requests); + + std::chrono::milliseconds waitTime(mMaxWaitMs); + auto responses = executor.awaitResponses(requestIds, waitTime); + for (auto const& response : responses) + { + for (auto const& responseIt : response) + { + auto const reqId = responseIt.getRequestId(); + if (responseIt.hasError()) + { + TLLM_THROW("Request id %lu encountered error: %s", reqId, responseIt.getErrorMsg().c_str()); + } + + auto const& result = responseIt.getResult(); + + auto const finishReason = result.finishReasons; + auto const actualResponse = result.outputTokenIds; + TLLM_LOG_DEBUG("reqId %d finished %d", reqId, result.isFinal); + TLLM_LOG_DEBUG("actual response:"); + + for (auto const& beam : actualResponse) + { + std::string tokenStr; + for (auto tok : beam) + { + tokenStr += std::to_string(tok) + " "; + } + TLLM_LOG_DEBUG("%s", tokenStr.c_str()); + } + + TLLM_LOG_DEBUG("reference:"); + auto referenceResponse = refResponses[reqId - 1]; + for (auto const& beam : referenceResponse) + { + std::string tokenStr; + for (auto tok : beam) + { + tokenStr += std::to_string(tok) + " "; + } + TLLM_LOG_DEBUG("%s", tokenStr.c_str()); + } + + if (result.isFinal) + { + TLLM_LOG_DEBUG("finishReason"); + std::string reasonStr; + for (auto const reason : finishReason) + { + // cast for easier visibility during debugging + EXPECT_EQ(static_cast<int>(reason), static_cast<int>(referenceFinishReasons[reqId - 1])); + reasonStr += std::to_string(static_cast<int>(reason)) + " "; + } + TLLM_LOG_DEBUG("%s", reasonStr.c_str()); + } + + EXPECT_EQ(beamWidth, actualResponse.size()); + for (int beam = 0; beam < beamWidth; beam++) + { + EXPECT_EQ(referenceResponse.at(beam).size(), actualResponse.at(beam).size()); + EXPECT_THAT(actualResponse.at(beam), testing::ElementsAreArray(referenceResponse.at(beam))); + } + } + } + } +} + +INSTANTIATE_TEST_SUITE_P(GptExecutorTest, ParamTest, + testing::Combine( // + testing::Values(false, true), // streaming + testing::Values(false, true), // excludeInputFromOutput + testing::Values(1, 2) // beamWidth + ), + generateTestName); + +INSTANTIATE_TEST_SUITE_P(GptExecutorTest, ParamStatsTest, + testing::Combine( // + testing::Values(0, 1000), // iterStatsMaxIterations + testing::Values(false, true) // useOrchestratorMode + ), + generateTestNameStats); + +INSTANTIATE_TEST_SUITE_P(LlamaExecutorTest, ParamCancelReqTest, + testing::Combine( // + testing::Values(false, true), // useOrchestratorMode + testing::Values(1, 2), // beamWidth + testing::Values("llama_tp1_pp4_cp1", "llama_tp4_pp1_cp1", "llama_tp2_pp2_cp1") // modelName + ), + generateTestNameCancelReq); + +INSTANTIATE_TEST_SUITE_P(LlamaExecutorTest, TimeoutTest, + testing::Combine( // + testing::Values("llama_tp1_pp4_cp1", "llama_tp4_pp1_cp1", "llama_tp1_pp1_cp1"), // modelName + testing::Values(false, true), // useOrchestratorMode + testing::Values(2) // beamWidth + ), + generateTestNameTimeoutTest); + +INSTANTIATE_TEST_SUITE_P(LlamaExecutorTest, LeaderApiUsageTest, + testing::Combine( // + testing::Values("llama_tp1_pp4_cp1", "llama_tp4_pp1_cp1", "llama_tp2_pp2_cp1") // modelName + ), + generateTestNameLeaderApiUsage); + +INSTANTIATE_TEST_SUITE_P(GptExecutorTest, AllParamsTest, + testing::Combine( // + testing::Values(false, true), // streaming + testing::Values(1, 2), // beamWidth + testing::Values(true), // computeLogProbs + testing::Values(false, true), // excludeInputInOutput + testing::Values(true), // returnContextLogits + testing::Values(true), // returnGenerationLogits + testing::Values("gpt"), // modelName + testing::Values(false, true), // useOrchestratorMode + testing::Values(false, true), // returnAllGeneratedTokens + testing::Values(1, 2) // numReturnSequences + ), + generateTestNameAllParams); + +INSTANTIATE_TEST_SUITE_P(LlamaExecutorTest, AllParamsTest, + testing::Combine( // + testing::Values(false, true), // streaming + testing::Values(1, 2), // beamWidth + testing::Values(true), // computeLogProbs + testing::Values(false, true), // excludeInputInOutput + testing::Values(false), // returnContextLogits + testing::Values(true), // returnGenerationLogits + testing::Values("llama_tp1_pp4_cp1", "llama_tp4_pp1_cp1", "llama_tp2_pp2_cp1"), // modelName + testing::Values(false, true), // useOrchestratorMode + testing::Values(false), // returnAllGeneratedTokens + testing::Values(1) // numReturnSequences + ), + generateTestNameAllParams); + +INSTANTIATE_TEST_SUITE_P(LlamaMultiExecutorTest, AllParamsTest, + testing::Combine( // + testing::Values(false, true), // streaming + testing::Values(1, 2), // beamWidth + testing::Values(false), // computeLogProbs + testing::Values(false, true), // excludeInputInOutput + testing::Values(false), // returnContextLogits + testing::Values(false), // returnGenerationLogits + testing::Values("llama_tp1_pp2_cp1"), // modelName + testing::Values(false), // useOrchestratorMode + testing::Values(false), // returnAllGeneratedTokens + testing::Values(1) // numReturnSequences + ), + generateTestNameAllParams); + +INSTANTIATE_TEST_SUITE_P(MedusaExecutorTest, AllParamsTest, + testing::Combine( // + testing::Values(false, true), // streaming + testing::Values(1), // beamWidth + testing::Values(false), // computeLogProbs + testing::Values(false, true), // excludeInputInOutput + testing::Values(false), // returnContextLogits + testing::Values(false), // returnGenerationLogits + testing::Values("medusa"), // modelName + testing::Values(false, true), // useOrchestratorMode + testing::Values(false), // returnAllGeneratedTokens + testing::Values(1) // numReturnSequences + ), + generateTestNameAllParams); + +// Disable some of ChatGLM's tests since they are the same as gpt's. +INSTANTIATE_TEST_SUITE_P(ChatGlmExecutorTest, AllParamsTest, + testing::Combine( // + testing::Values(false), // streaming + testing::Values(1, 2), // beamWidth + testing::Values(false), // computeLogProbs + testing::Values(false), // excludeInputInOutput + testing::Values(false), // returnContextLogits + testing::Values(false), // returnGenerationLogits + testing::Values("chatglm"), // modelName + testing::Values(false), // useOrchestratorMode + testing::Values(false), // returnAllGeneratedTokens + testing::Values(1, 2) // numReturnSequences + ), + generateTestNameAllParams); + +// ChatGlm0 Test is for glm-10b. +INSTANTIATE_TEST_SUITE_P(ChatGlm0ExecutorTest, AllParamsTest, + testing::Combine( // + testing::Values(false), // streaming + testing::Values(1), // beamWidth + testing::Values(false), // computeLogProbs + testing::Values(false), // excludeInputInOutput + testing::Values(false), // returnContextLogits + testing::Values(false), // returnGenerationLogits + testing::Values("glm"), // modelName + testing::Values(false), // useOrchestratorMode + testing::Values(false), // returnAllGeneratedTokens + testing::Values(1) // numReturnSequences + ), + generateTestNameAllParams); + +INSTANTIATE_TEST_SUITE_P(ChatGlm2ExecutorTest, AllParamsTest, + testing::Combine( // + testing::Values(false), // streaming + testing::Values(1), // beamWidth + testing::Values(false), // computeLogProbs + testing::Values(false), // excludeInputInOutput + testing::Values(false), // returnContextLogits + testing::Values(false), // returnGenerationLogits + testing::Values("chatglm2"), // modelName + testing::Values(false), // useOrchestratorMode + testing::Values(false), // returnAllGeneratedTokens + testing::Values(1) // numReturnSequences + ), + generateTestNameAllParams); + +INSTANTIATE_TEST_SUITE_P(ChatGlm3ExecutorTest, AllParamsTest, + testing::Combine( // + testing::Values(false), // streaming + testing::Values(1), // beamWidth + testing::Values(false), // computeLogProbs + testing::Values(false), // excludeInputInOutput + testing::Values(false), // returnContextLogits + testing::Values(false), // returnGenerationLogits + testing::Values("chatglm3"), // modelName + testing::Values(false), // useOrchestratorMode + testing::Values(false), // returnAllGeneratedTokens + testing::Values(1) // numReturnSequences + ), + generateTestNameAllParams); + +INSTANTIATE_TEST_SUITE_P(LlamaExecutorTest, LogitsProcParamsTest, + testing::Combine( // + testing::Values( + "llama_tp1_pp1_cp1", "llama_tp4_pp1_cp1", "llama_tp2_pp2_cp1", "llama_tp1_pp4_cp1"), // modelName + testing::Values(false, true), // batched + testing::Values(false, true) // replicated + ), + generateTestNameLogitsProc); + +INSTANTIATE_TEST_SUITE_P(GptExecutorGuidedDecodingTest, GuidedDecodingParamsTest, + testing::Combine(testing::Values("gpt")), generateTestNameGuidedDecoding); + +INSTANTIATE_TEST_SUITE_P(LlamaExecutorGuidedDecodingTest, GuidedDecodingParamsTest, + testing::Combine( + testing::Values("llama_tp1_pp1_cp1", "llama_tp4_pp1_cp1", "llama_tp2_pp2_cp1", "llama_tp1_pp4_cp1")), + generateTestNameGuidedDecoding); diff --git a/cpp/tests/e2e_tests/executor/executorTest.h b/cpp/tests/e2e_tests/executor/executorTest.h new file mode 100644 index 000000000000..7866a6992266 --- /dev/null +++ b/cpp/tests/e2e_tests/executor/executorTest.h @@ -0,0 +1,58 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tests/utils/common.h" + +#include <gmock/gmock.h> +#include <gtest/gtest.h> + +#include <memory> + +namespace tensorrt_llm::testing +{ + +class GptExecutorTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type-member-init) +{ +public: + using SizeType32 = tensorrt_llm::testing::SizeType32; + +protected: + void SetUp() override + { + mDeviceCount = tensorrt_llm::common::getDeviceCount(); + if (mDeviceCount == 0) + { + GTEST_SKIP() << "No GPUs found"; + } + + mLogger = std::make_shared<tensorrt_llm::runtime::TllmLogger>(); + initTrtLlmPlugins(mLogger.get()); + } + + void TearDown() override {} + + int mDeviceCount{}; + std::shared_ptr<nvinfer1::ILogger> mLogger{}; + SizeType32 mMaxWaitMs = 300000; + SizeType32 mTrigWarnMs = 10000; +}; + +} // namespace tensorrt_llm::testing diff --git a/cpp/tests/resources/scripts/build_chatglm_engines.py b/cpp/tests/resources/scripts/build_chatglm_engines.py new file mode 100644 index 000000000000..abe187307604 --- /dev/null +++ b/cpp/tests/resources/scripts/build_chatglm_engines.py @@ -0,0 +1,193 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import os +import platform +import shutil +import sys +import typing +from pathlib import Path +from typing import Optional + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + +resources_dir = Path(__file__).parent.resolve().parent +model_dir = resources_dir / "models" +chatglm_example_dir = Path("examples/chatglm") +bCopyModel = True # "False" to remove redundant copy of model from model_cache + + +def convert_ckpt(model_dir: str, output_dir: str, world_size: int): + if os.path.exists(output_dir): + print('Skip ckpt convert - output already exists') + return + + convert_cmd = [ + sys.executable, + str(chatglm_example_dir / "convert_checkpoint.py"), "--dtype=float16", + f"--model_dir={model_dir}", f"--output_dir={output_dir}", + f"--tp_size={world_size}" + ] + run_command(convert_cmd) + + +def build_engine(ckpt_dir: str, + engine_dir: str, + is_ifb: bool = False, + is_chatglm_6b_or_glm_10b: bool = False): + if os.path.exists(engine_dir): + print('Skip engine build - output already exists') + return + + build_cmd = [ + "trtllm-build", + f"--checkpoint_dir={ckpt_dir}", + f"--output_dir={engine_dir}", + "--log_level=error", + "--max_batch_size=8", + "--max_beam_width=2", + "--max_input_len=256", + "--max_seq_len=384", + "--gpt_attention_plugin=float16", + "--gemm_plugin=float16", + ] + if is_ifb: + build_cmd.extend([ + "--remove_input_padding=enable", + "--paged_kv_cache=enable", + "--context_fmha=enable", + "--use_paged_context_fmha=enable", + ]) + else: + build_cmd.extend([ + "--remove_input_padding=disable", + "--paged_kv_cache=disable", + ]) + + if is_chatglm_6b_or_glm_10b: + print("Disable Context FMHA for ChatGLM-6B and GLM-10B") + build_cmd.extend(["--context_fmha=disable"]) + + run_command(build_cmd) + + +def build_engines(model_cache: typing.Optional[str] = None, + world_size: int = 1, + clean: Optional[bool] = False): + + for model_name in [ + "chatglm-6b", "chatglm2-6b", "chatglm3-6b", "glm-10b", "glm-4-9b", + "chatglm3-6b-32k" + ]: + is_chatglm_6b_or_glm_10b = model_name in ["chatglm-6b", "glm-10b"] + if model_cache and (Path(model_cache) / model_name).is_dir(): + model_cache_dir = Path(model_cache) / model_name + if bCopyModel or model_name == "chatglm-6b": + print("Copy model from model_cache") + hf_dir = model_dir / model_name + if platform.system() == "Windows": + wincopy(source=str(model_cache_dir), + dest=model_name, + isdir=True, + cwd=model_dir) + else: + run_command(["rsync", "-rlptD", + str(model_cache_dir), "."], + cwd=model_dir) + else: + print("Use model from model_cache directly except ChatGLM-6B") + hf_dir = Path(model_cache) + + else: + hf_dir = model_dir / model_name + if not hf_dir.is_dir(): + print("Clone model from HF") + run_command( + [ + "git", "clone", + f"https://huggingface.co/THUDM/{model_name}", model_name + ], + cwd=model_dir, + ) + + # Build engines + print(f"Building {model_name}") + ckpt_dir = Path(model_dir) / "c-model" / model_name + if clean: + print('clean up ckpt folder ', ckpt_dir) + if ckpt_dir.is_dir(): + shutil.rmtree(ckpt_dir, ignore_errors=True) + + # Fix HF error for ChatGLM-6B / GLM-4-9B / ChatGLM2-6B / ChatGLM3-6B-32K, hope to remove this in the future + if model_name in [ + "chatglm-6b", "glm-4-9b", "chatglm2-6b", "chatglm3-6b-32k" + ]: + shutil.copy( + chatglm_example_dir / f"{model_name}/tokenization_chatglm.py", + hf_dir, + ) + + convert_ckpt(hf_dir, ckpt_dir, world_size) + + model_spec_obj = ModelSpec('input_tokens.npy', _tb.DataType.HALF) + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.CONTINUOUS) + model_spec_obj.use_gpt_plugin() + engine_dir = Path( + model_dir + ) / "rt_engine" / model_name / model_spec_obj.get_model_path( + ) / "tp1-pp1-cp1-gpu" + if clean: + print('clean up engine folder ', engine_dir) + if engine_dir.is_dir(): + shutil.rmtree(engine_dir, ignore_errors=True) + build_engine(ckpt_dir, engine_dir, False, is_chatglm_6b_or_glm_10b) + + model_spec_obj.use_packed_input() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + engine_dir = Path( + model_dir + ) / "rt_engine" / model_name / model_spec_obj.get_model_path( + ) / "tp1-pp1-cp1-gpu" + if clean: + print('clean up engine folder ', engine_dir) + if engine_dir.is_dir(): + shutil.rmtree(engine_dir, ignore_errors=True) + build_engine(ckpt_dir, engine_dir, True, is_chatglm_6b_or_glm_10b) + + print("Done") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + + parser.add_argument('--world_size', + type=int, + default=1, + help='world size, only support tensor parallelism now') + + parser.add_argument('--clean', + action='store_true', + default=False, + help='Clean target folders before building engines') + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/build_eagle_engines.py b/cpp/tests/resources/scripts/build_eagle_engines.py new file mode 100755 index 000000000000..8b10698a603b --- /dev/null +++ b/cpp/tests/resources/scripts/build_eagle_engines.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import pathlib as _pl +import platform as _pf +import sys as _sys + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def build_engine(base_model_dir: _pl.Path, eagle_model_dir: _pl.Path, + engine_dir: _pl.Path, build_base_model: bool, *args): + + if build_base_model: + checkpoint_path = "examples/models/core/llama/convert_checkpoint.py" + else: + checkpoint_path = "examples/eagle/convert_checkpoint.py" + + covert_cmd = [_sys.executable, checkpoint_path] + ( + ['--model_dir', str(base_model_dir)] if base_model_dir else []) + [ + '--output_dir', str(engine_dir), '--dtype=float16' + ] + list(args) + + if not build_base_model: + covert_cmd += [ + '--eagle_model_dir', + str(eagle_model_dir), '--num_eagle_layers=4', '--max_draft_len=63' + ] + + run_command(covert_cmd) + + build_args = ["trtllm-build"] + ( + ['--checkpoint_dir', str(engine_dir)] if engine_dir else []) + [ + '--output_dir', + str(engine_dir), + '--gemm_plugin=float16', + '--max_batch_size=8', + '--max_input_len=12', + '--max_seq_len=140', + '--log_level=error', + '--paged_kv_cache=enable', + '--remove_input_padding=enable', + '--use_paged_context_fmha=enable', + ] + + if not build_base_model: + build_args += ['--speculative_decoding_mode=eagle'] + + run_command(build_args) + + +def build_engines(model_cache: str): + resources_dir = _pl.Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_name = 'vicuna-7b-eagle' + base_model_name = 'vicuna-7b-v1.3' + eagle_model_name = 'EAGLE-Vicuna-7B-v1.3' + + if model_cache: + print(f"Copy model from {model_cache}") + base_model_cache_dir = _pl.Path(model_cache) / base_model_name + eagle_cache_dir = _pl.Path(model_cache) / eagle_model_name + assert base_model_cache_dir.is_dir(), base_model_cache_dir + assert eagle_cache_dir.is_dir(), eagle_cache_dir + + if _pf.system() == "Windows": + wincopy(source=str(base_model_cache_dir), + dest=base_model_name, + isdir=True, + cwd=models_dir) + wincopy(source=str(eagle_cache_dir), + dest=eagle_model_name, + isdir=True, + cwd=models_dir) + else: + run_command(["rsync", "-rlptD", + str(base_model_cache_dir), "."], + cwd=models_dir) + run_command(["rsync", "-rlptD", + str(eagle_cache_dir), "."], + cwd=models_dir) + + base_model_dir = models_dir / base_model_name + eagle_model_dir = models_dir / eagle_model_name + assert base_model_dir.is_dir() + assert eagle_model_dir.is_dir() + + eagle_engine_dir = models_dir / 'rt_engine' / model_name + base_engine_dir = models_dir / 'rt_engine' / base_model_name + + model_spec_obj = ModelSpec('input_tokens.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + + base_full_engine_path = base_engine_dir / model_spec_obj.get_model_path( + ) / 'tp1-pp1-cp1-gpu' + print(f"\nBuilding fp16 engine at {str(base_full_engine_path)}") + build_engine(base_model_dir, + eagle_model_dir, + base_full_engine_path, + build_base_model=True) + + model_spec_obj = ModelSpec('input_tokens.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + model_spec_obj.use_eagle() + eagle_full_engine_path = eagle_engine_dir / model_spec_obj.get_model_path( + ) / 'tp1-pp1-cp1-gpu' + print(f"\nBuilding fp16 engine at {str(eagle_full_engine_path)}") + build_engine(base_model_dir, + eagle_model_dir, + eagle_full_engine_path, + build_base_model=False) + + print("Done.") + + +if __name__ == "__main__": + parser = _arg.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/build_enc_dec_engines.py b/cpp/tests/resources/scripts/build_enc_dec_engines.py new file mode 100644 index 000000000000..7079916267d9 --- /dev/null +++ b/cpp/tests/resources/scripts/build_enc_dec_engines.py @@ -0,0 +1,187 @@ +import os.path +from argparse import ArgumentParser +from dataclasses import dataclass, fields +from subprocess import run +from sys import stderr, stdout +from typing import List, Literal, Union + +split = os.path.split +join = os.path.join +dirname = os.path.dirname + + +@dataclass +class Arguments: + download: bool = False + dtype: Literal['float16', 'float32', 'bfloat16'] = 'float16' + + hf_repo_name: Literal[ + 'facebook/bart-large-cnn', 't5-small', + 'language_adapter-enc_dec_language_adapter'] = 'facebook/bart-large-cnn' + + model_cache: str = '/llm-models' + + tp: int = 1 + pp: int = 1 + + beams: str = '1' + gpus_per_node: int = 4 + debug: bool = False + + rm_pad: bool = True + gemm: bool = True + + max_new_tokens: int = 64 + + @property + def beams_tuple(self): + return eval(f'tuple([{self.beams}])') + + @property + def max_beam(self): + return max(self.beams_tuple) + + @property + def ckpt(self): + return self.hf_repo_name.split('/')[-1] + + @property + def base_dir(self): + return dirname(dirname(__file__)) + + @property + def data_dir(self): + return join(self.base_dir, 'data/enc_dec') + + @property + def models_dir(self): + return join(self.base_dir, 'models/enc_dec') + + @property + def hf_models_dir(self): + return join(self.model_cache, self.ckpt) + + @property + def trt_models_dir(self): + return join(self.models_dir, 'trt_models', self.ckpt) + + @property + def engines_dir(self): + return join(self.models_dir, 'trt_engines', self.ckpt, + f'{self.tp * self.pp}-gpu', self.dtype) + + @property + def model_type(self): + return self.ckpt.split('-')[0] + + def __post_init__(self): + parser = ArgumentParser() + for k in fields(self): + k = k.name + v = getattr(self, k) + if isinstance(v, bool): + parser.add_argument(f'--{k}', action='store_true') + else: + parser.add_argument(f'--{k}', default=v, type=type(v)) + + args = parser.parse_args() + for k, v in args._get_kwargs(): + setattr(self, k, v) + + +@dataclass +class RunCMDMixin: + args: Arguments + + def command(self) -> Union[str, List[str]]: + raise NotImplementedError + + def run(self): + cmd = self.command() + if cmd: + cmd = ' '.join(cmd) if isinstance(cmd, list) else cmd + print('+ ' + cmd) + run(cmd, shell='bash', stdout=stdout, stderr=stderr, check=True) + + +class DownloadHF(RunCMDMixin): + + def command(self): + args = self.args + return [ + 'git', 'clone', f'https://huggingface.co/{args.hf_repo_name}', + args.hf_models_dir + ] if args.download and args.model_type != 'language_adapter' else '' + + +class Convert(RunCMDMixin): + + def command(self): + args = self.args + return [ + f'python examples/models/core/enc_dec/convert_checkpoint.py', + f'--model_type {args.model_type}', + f'--model_dir {args.hf_models_dir}', + f'--output_dir {args.trt_models_dir}', + f'--tp_size {args.tp} --pp_size {args.pp}' + ] + + +class Build(RunCMDMixin): + + def command(self): + args = self.args + engine_dir = args.engines_dir + weight_dir = args.trt_models_dir + encoder_build = [ + f"trtllm-build --checkpoint_dir {join(weight_dir, 'encoder')}", + f"--output_dir {join(engine_dir, 'encoder')}", + f'--paged_kv_cache disable', + f'--max_beam_width {args.max_beam}', + f'--max_batch_size 8', + f'--max_input_len 512', + f'--gemm_plugin {args.dtype}', + f'--bert_attention_plugin {args.dtype}', + f'--gpt_attention_plugin {args.dtype}', + f'--remove_input_padding enable', + ] + + decoder_build = [ + f"trtllm-build --checkpoint_dir {join(weight_dir, 'decoder')}", + f"--output_dir {join(engine_dir, 'decoder')}", + f'--paged_kv_cache enable', + f'--max_beam_width {args.max_beam}', + f'--max_batch_size 8', + f'--max_seq_len 201', + f'--max_encoder_input_len 512', + f'--gemm_plugin {args.dtype}', + f'--bert_attention_plugin {args.dtype}', + f'--gpt_attention_plugin {args.dtype}', + f'--remove_input_padding enable', + '--max_input_len 1', + ] + + # t5 model with relative attention cannot use context_fmha + encoder_build.append(f'--context_fmha disable') + decoder_build.append(f'--context_fmha disable') + + # language adapter plugin leverages MOE plugin for static expert selection + if args.model_type == 'language_adapter': + encoder_build.append(f'--moe_plugin auto') + decoder_build.append(f'--moe_plugin auto') + else: + encoder_build.append(f'--moe_plugin disable') + decoder_build.append(f'--moe_plugin disable') + + encoder_build = ' '.join(encoder_build) + decoder_build = ' '.join(decoder_build) + ret = ' && '.join((encoder_build, decoder_build)) + return ret + + +if __name__ == "__main__": + # TODO: add support for more models / setup + args = Arguments() + DownloadHF(args).run() + Convert(args).run() + Build(args).run() diff --git a/cpp/tests/resources/scripts/build_engines_utils.py b/cpp/tests/resources/scripts/build_engines_utils.py new file mode 100644 index 000000000000..ad8525217e3a --- /dev/null +++ b/cpp/tests/resources/scripts/build_engines_utils.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging as _log +import os as _os +import pathlib as _pl +import subprocess as _sp +import typing as _tp + + +def run_command(command: _tp.Sequence[str], + *, + cwd=None, + timeout=None, + **kwargs) -> None: + _log.info("Running: cd %s && %s", str(cwd), " ".join(command)) + override_timeout = int(_os.environ.get("CPP_TEST_TIMEOUT_OVERRIDDEN", "-1")) + if override_timeout > 0 and (timeout is None or override_timeout > timeout): + _log.info("Overriding the command timeout: %s (before) and %s (after)", + timeout, override_timeout) + timeout = override_timeout + _sp.check_call(command, cwd=cwd, timeout=timeout, **kwargs) + + +# We can't use run_command() because robocopy (Robust Copy, rsync equivalent on Windows) +# for some reason uses nonzero return codes even on *successful* copies, so we need to check it manually. +# Also, robocopy only accepts dirs, not individual files, so we need a separate command for the +# single-file case. +def wincopy(source: str, dest: str, isdir: bool, cwd=None) -> None: + if not isdir: # Single-file copy + run_command(["cmd", "/c", "copy", + str(_pl.Path(source)), f".\\{dest}"], + cwd=cwd) + else: # Directory sync + copy_cmd = ["robocopy", source, f"./{dest}", "/mir", "/e"] + print(f"Running: cd %s && %s" % + (str(cwd or _pl.Path.cwd()), " ".join(copy_cmd))) + + # Run the command from the specified directory + result = _sp.run(copy_cmd, cwd=cwd) + + # Check for valid exit code + if result.returncode < 8: + print("ROBOCOPY completed successfully.") + else: + print( + "ROBOCOPY failure. Displaying error. See https://ss64.com/nt/robocopy-exit.html for exit code info." + ) + raise _sp.CalledProcessError(returncode=result.returncode, + cmd=copy_cmd, + output=result.stderr) diff --git a/cpp/tests/resources/scripts/build_gpt_engines.py b/cpp/tests/resources/scripts/build_gpt_engines.py new file mode 100755 index 000000000000..fa089d773dc0 --- /dev/null +++ b/cpp/tests/resources/scripts/build_gpt_engines.py @@ -0,0 +1,292 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import os +import platform +import shutil +import sys +from pathlib import Path +from typing import Optional + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec, QuantMethod + + +def convert_ckpt(model_dir: str, + output_dir: str, + *args, + world_size: int = 1, + dtype: str = 'float16'): + convert_cmd = [ + sys.executable, "examples/models/core/gpt/convert_checkpoint.py", + f"--model_dir={model_dir}", f"--output_dir={output_dir}", + f"--dtype={dtype}", f"--tp_size={world_size}" + ] + list(args) + run_command(convert_cmd) + + +def build_engine( + checkpoint_dir: str, + engine_dir: str, + *args, + max_input_len: int = 256, + max_seq_len: int = 384, +): + + build_cmd = [ + "trtllm-build", + '--log_level=error', + f'--checkpoint_dir={checkpoint_dir}', + f'--output_dir={engine_dir}', + '--max_batch_size=64', + f'--max_input_len={max_input_len}', + f'--max_seq_len={max_seq_len}', + '--max_beam_width=2', + '--kv_cache_type=continuous', + ] + legacy_args = [ + "--gpt_attention_plugin=disable", + "--context_fmha=disable", + "--remove_input_padding=disable", + ] + build_cmd = build_cmd + legacy_args + list(args) + run_command(build_cmd) + + +def build_engines(model_cache: Optional[str] = None, + world_size: int = 1, + clean: Optional[bool] = False): + # TODO add support of Pipeline parallelism to GPT + tp_size = world_size + pp_size = 1 + cp_size = 1 + + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_name = 'gpt2' + + # Clone or update the model directory without lfs + hf_dir = models_dir / model_name + if hf_dir.exists(): + assert hf_dir.is_dir() + run_command(["git", "pull"], cwd=hf_dir) + else: + if platform.system() == "Windows": + url_prefix = "" + else: + url_prefix = "file://" + + model_url = url_prefix + str( + Path(model_cache) / + model_name) if model_cache else "https://huggingface.co/gpt2" + run_command([ + "git", "clone", model_url, "--single-branch", "--no-local", + model_name + ], + cwd=hf_dir.parent, + env={ + **os.environ, "GIT_LFS_SKIP_SMUDGE": "1" + }) + + assert hf_dir.is_dir() + + # Download the model file + model_file_name = "pytorch_model.bin" + if model_cache: + if platform.system() == "Windows": + wincopy(source=str( + Path(model_cache) / model_name / model_file_name), + dest=model_file_name, + isdir=False, + cwd=hf_dir) + else: + run_command([ + "rsync", "-rlptD", + str(Path(model_cache) / model_name / model_file_name), "." + ], + cwd=hf_dir) + else: + run_command(["git", "lfs", "pull", "--include", model_file_name], + cwd=hf_dir) + + safetensor_file = hf_dir / "model.safetensors" + has_safetensor = safetensor_file.exists() + if has_safetensor: + safetensor_file.rename(str(safetensor_file) + ".bak") + + assert (hf_dir / model_file_name).is_file() + + ckpt_dir = models_dir / 'c-model' / model_name + engine_dir = models_dir / 'rt_engine' / model_name + + if clean: + target_dir = Path(engine_dir) + print('clean up target folder ', target_dir) + if target_dir.is_dir(): + shutil.rmtree(target_dir, ignore_errors=True) + + tp_pp_cp_dir = f"tp{tp_size}-pp{pp_size}-cp{cp_size}-gpu" + tp_dir = f"{world_size}-gpu" + + print("\nConverting to fp16") + fp16_ckpt_dir = ckpt_dir / 'fp16' / tp_dir + convert_ckpt(str(hf_dir), + str(fp16_ckpt_dir), + world_size=tp_size, + dtype='float16') + + print("\nBuilding fp16 engines") + + input_file = 'input_tokens.npy' + # this engine can be use for in-flight batching + ifb_base_args = [ + '--gpt_attention_plugin=float16', + '--remove_input_padding=enable', + '--context_fmha=enable', + '--max_num_tokens=10000', + '--use_paged_context_fmha=enable', + ] + + paged_kv_cache_args = ['--kv_cache_type=paged'] + + no_kv_cache_args = ['--kv_cache_type=disabled'] + + def get_ifb_args(kv_cache_type): + if kv_cache_type == _tb.KVCacheType.DISABLED: + return ifb_base_args + no_kv_cache_args + elif kv_cache_type == _tb.KVCacheType.PAGED: + return ifb_base_args + paged_kv_cache_args + else: + assert False, f"Unsupported kv_cache_type: {kv_cache_type}" + + model_spec_obj = ModelSpec(input_file, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + + model_spec_current = model_spec_obj.__copy__() + + for kv_cache_type in [_tb.KVCacheType.DISABLED, _tb.KVCacheType.PAGED]: + model_spec_current.set_kv_cache_type(kv_cache_type) + build_engine( + str(fp16_ckpt_dir), + str(engine_dir / model_spec_current.get_model_path() / + tp_pp_cp_dir), *get_ifb_args(kv_cache_type)) + + model_spec_current = model_spec_obj.__copy__() + max_draft_tokens = 5 + model_spec_current.use_draft_tokens_external_decoding() + model_spec_current.set_draft_tokens(max_draft_tokens) + + build_engine( + str(fp16_ckpt_dir), + str(engine_dir / model_spec_current.get_model_path() / tp_pp_cp_dir), + f'--max_draft_len={max_draft_tokens}', + '--speculative_decoding_mode=draft_tokens_external', + *get_ifb_args(_tb.KVCacheType.PAGED)) + + model_spec_current = model_spec_obj.__copy__() + model_spec_current.use_multiple_profiles() + + build_engine( + str(fp16_ckpt_dir), + str(engine_dir / model_spec_current.get_model_path() / tp_pp_cp_dir), + '--multiple_profiles=enable', *get_ifb_args(_tb.KVCacheType.PAGED)) + + model_spec_current = model_spec_obj.__copy__() + max_input_len = 128 + model_spec_current.set_max_input_length(max_input_len) + + build_engine(str(fp16_ckpt_dir), + str(engine_dir / model_spec_current.get_model_path() / + tp_pp_cp_dir), + *get_ifb_args(_tb.KVCacheType.PAGED), + max_input_len=max_input_len) + + # We build almost the same engine twice. But this engine has gather_context_logits + # to extract logits from python runtime and uses context FMHA for generation to match draft model executions, + # which uses context FMHA for draft tokens prediction. + # Currently the gather_context_logits is not supported with target model of speculative decoding + model_spec_current = model_spec_obj.__copy__() + model_spec_current.gather_logits() + + build_engine( + str(fp16_ckpt_dir), + str(engine_dir / model_spec_current.get_model_path() / tp_pp_cp_dir), + '--gather_context_logits', *get_ifb_args(_tb.KVCacheType.PAGED)) + + # build engine with lora enabled + model_spec_current = model_spec_obj.__copy__() + model_spec_current.use_lora_plugin() + build_engine( + str(fp16_ckpt_dir), + str(engine_dir / model_spec_current.get_model_path() / tp_pp_cp_dir), + "--lora_target_modules=attn_qkv", '--lora_plugin=float16', + *get_ifb_args(_tb.KVCacheType.PAGED)) + + if model_cache: + llm_datasets_root = Path(model_cache) / "datasets" + calib_dataset = llm_datasets_root / "cimec/lambada/" + else: + calib_dataset = "lambada" + print("\nConverting to fp16 SQ") + fp16_sq_ckpt_dir = ckpt_dir / 'fp16-sq' / tp_dir + convert_ckpt(str(hf_dir), + str(fp16_sq_ckpt_dir), + "--smoothquant=0.5", + f"--calib_dataset={calib_dataset}", + world_size=tp_size, + dtype='float16') + + print("\nBuilding fp16 SQ engines") + model_spec_current = ModelSpec(input_file, _tb.DataType.HALF) + model_spec_current.use_gpt_plugin() + model_spec_current.use_packed_input() + model_spec_current.set_quant_method(QuantMethod.SMOOTH_QUANT) + + for kv_cache_type in [_tb.KVCacheType.DISABLED, _tb.KVCacheType.PAGED]: + model_spec_current.set_kv_cache_type(kv_cache_type) + build_engine( + str(fp16_sq_ckpt_dir), + str(engine_dir / model_spec_current.get_model_path() / + tp_pp_cp_dir), *get_ifb_args(kv_cache_type)) + + if has_safetensor: + Path(str(safetensor_file) + ".bak").rename(safetensor_file) + + print("Done.") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + + parser.add_argument('--world_size', + type=int, + default=1, + help='World size, only support tensor parallelism now') + + parser.add_argument('--clean', + action='store_true', + default=False, + help='Clean target folders before building engines') + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/build_gptj_engines.py b/cpp/tests/resources/scripts/build_gptj_engines.py new file mode 100755 index 000000000000..bfab97e0ec11 --- /dev/null +++ b/cpp/tests/resources/scripts/build_gptj_engines.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import os as _os +import pathlib as _pl +import platform as _pf +import sys as _sys +import typing as _tp + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def get_ckpt_without_quatization(model_dir, output_dir): + build_args = [ + _sys.executable, "examples/models/contrib/gpt/convert_checkpoint.py" + ] + [ + '--model_dir={}'.format(model_dir), + '--output_dir={}'.format(output_dir), + ] + run_command(build_args) + + +def get_ckpt_with_modelopt_quant(model_dir, output_dir, model_cache): + build_args = [_sys.executable, "examples/quantization/quantize.py"] + [ + '--model_dir={}'.format(model_dir), + '--output_dir={}'.format(output_dir), '--qformat=fp8', + '--kv_cache_dtype=fp8', + f'--calib_dataset={model_cache}/datasets/cnn_dailymail' + ] + run_command(build_args) + + +def build_engine(checkpoint_dir: _pl.Path, engine_dir: _pl.Path, *args): + build_args = ["trtllm-build"] + ( + ['--checkpoint_dir', str(checkpoint_dir)] if checkpoint_dir else []) + [ + '--output_dir', + str(engine_dir), + '--logits_dtype=float16', + '--gemm_plugin=float16', + '--max_batch_size=32', + '--max_input_len=40', + '--max_seq_len=60', + '--max_beam_width=2', + '--log_level=error', + ] + list(args) + run_command(build_args) + + +def build_engines(model_cache: _tp.Optional[str] = None, only_fp8=False): + resources_dir = _pl.Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_name = 'gpt-j-6b' + + # Clone or update the model directory without lfs + hf_dir = models_dir / model_name + if hf_dir.exists(): + assert hf_dir.is_dir() + run_command(["git", "pull"], cwd=hf_dir) + else: + if _pf.system() == "Windows": + url_prefix = "" + else: + url_prefix = "file://" + model_url = url_prefix + str( + _pl.Path(model_cache) / model_name + ) if model_cache else "https://huggingface.co/EleutherAI/gpt-j-6b" + run_command([ + "git", "clone", model_url, "--single-branch", "--no-local", + model_name + ], + cwd=hf_dir.parent, + env={ + **_os.environ, "GIT_LFS_SKIP_SMUDGE": "1" + }) + + assert (hf_dir.is_dir()) + + # Download the model file + model_file_name = "pytorch_model.bin" + if model_cache: + if _pf.system() == "Windows": + wincopy(source=str( + _pl.Path(model_cache) / model_name / model_file_name), + dest=model_file_name, + isdir=False, + cwd=hf_dir) + else: + run_command([ + "rsync", "-rlptD", + str(_pl.Path(model_cache) / model_name / model_file_name), "." + ], + cwd=hf_dir) + else: + run_command(["git", "lfs", "pull", "--include", model_file_name], + cwd=hf_dir) + + assert ((hf_dir / model_file_name).is_file()) + + engine_dir = models_dir / 'rt_engine' / model_name + + # TODO add Tensor and Pipeline parallelism to GPT-J + tp_size = 1 + pp_size = 1 + cp_size = 1 + tp_pp_cp_dir = f"tp{tp_size}-pp{pp_size}-cp{cp_size}-gpu" + input_file = 'input_tokens.npy' + + if only_fp8: + # with ifb, new plugin + print( + "\nBuilding fp8-plugin engine using gpt_attention_plugin with inflight-batching, packed" + ) + # TODO: use dummy scales atm; to re-enable when data is uploaded to the model cache + # quantized_fp8_model_arg = '--quantized_fp8_model_path=' + \ + # str(_pl.Path(model_cache) / 'fp8-quantized-modelopt' / 'gptj_tp1_rank0.npz') + fp8_ckpt_path = engine_dir / 'fp8' / tp_pp_cp_dir + get_ckpt_with_modelopt_quant(hf_dir, fp8_ckpt_path, model_cache) + model_spec_obj = ModelSpec(input_file, _tb.DataType.FP8) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + build_engine( + fp8_ckpt_path, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--gpt_attention_plugin=float16', + '--paged_kv_cache=enable', + '--remove_input_padding=enable', + '--use_paged_context_fmha=enable', + ) + else: + fp16_ckpt_path = engine_dir / 'fp16' / tp_pp_cp_dir + get_ckpt_without_quatization(hf_dir, fp16_ckpt_path) + print("\nBuilding fp16-plugin engine") + model_spec_obj = ModelSpec(input_file, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.CONTINUOUS) + + build_engine( + fp16_ckpt_path, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--gpt_attention_plugin=float16', '--paged_kv_cache=disable', + '--remove_input_padding=disable', "--context_fmha=disable") + + print("\nBuilding fp16-plugin-packed engine") + model_spec_obj.use_packed_input() + build_engine( + fp16_ckpt_path, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--gpt_attention_plugin=float16', '--paged_kv_cache=disable', + '--remove_input_padding=enable', "--context_fmha=disable") + + print("\nBuilding fp16-plugin-packed-paged engine") + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + build_engine( + fp16_ckpt_path, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--gpt_attention_plugin=float16', '--paged_kv_cache=enable', + '--remove_input_padding=enable', "--context_fmha=disable") + print("Done.") + + +if __name__ == "__main__": + parser = _arg.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + parser.add_argument( + "--only_fp8", + action="store_true", + help="Build engines for only FP8 tests. Implemented for H100 runners.") + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/build_llama_engines.py b/cpp/tests/resources/scripts/build_llama_engines.py new file mode 100644 index 000000000000..dbac12621c73 --- /dev/null +++ b/cpp/tests/resources/scripts/build_llama_engines.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import pathlib as _pl +import platform as _pf +import sys as _sys +import time + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def build_engine(weight_dir: _pl.Path, engine_dir: _pl.Path, convert_extra_args, + build_extra_args): + + ckpt_dir = engine_dir / 'ckpt' + + convert_cmd = [ + _sys.executable, "examples/models/core/llama/convert_checkpoint.py" + ] + ([f'--model_dir={weight_dir}'] if weight_dir else []) + [ + f'--output_dir={ckpt_dir}', + '--dtype=float16', + ] + convert_extra_args + + run_command(convert_cmd) + + build_args = [ + 'trtllm-build', + f'--checkpoint_dir={ckpt_dir}', + f'--output_dir={engine_dir}', + '--gpt_attention_plugin=float16', + '--gemm_plugin=float16', + '--max_batch_size=32', + '--max_input_len=40', + '--max_seq_len=60', + '--max_beam_width=2', + '--log_level=error', + '--paged_kv_cache=enable', + '--remove_input_padding=enable', + ] + build_extra_args + + run_command(build_args) + + +def build_engines(model_cache: str, only_multi_gpu: bool): + resources_dir = _pl.Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_name = 'Llama-3.2-1B' + + if model_cache: + print("Copy model from model_cache") + model_cache_dir = _pl.Path( + model_cache) / 'llama-3.2-models' / model_name + assert (model_cache_dir.is_dir()), model_cache_dir + + if _pf.system() == "Windows": + wincopy(source=str(model_cache_dir), + dest=model_name, + isdir=True, + cwd=models_dir) + else: + run_command(["rsync", "-rlptD", + str(model_cache_dir), "."], + cwd=models_dir) + + hf_dir = models_dir / model_name + assert hf_dir.is_dir(), f"testing {hf_dir}" + + engine_dir = models_dir / 'rt_engine' / model_name + + model_spec_obj = ModelSpec('input_tokens_llama.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + + tp_pp_cp_sizes = [(1, 1, 1)] + if only_multi_gpu: + tp_pp_cp_sizes = [(1, 4, 1), (4, 1, 1), (1, 2, 1), (2, 2, 1), (2, 1, 1), + (1, 1, 2), (2, 1, 2)] + for tp_size, pp_size, cp_size in tp_pp_cp_sizes: + print(f"\nBuilding fp16 tp{tp_size} pp{pp_size} cp{cp_size} engine") + start_time = time.time() + + tp_pp_cp_dir = f"tp{tp_size}-pp{pp_size}-cp{cp_size}-gpu" + model_spec_obj.use_tensor_parallelism(tp_size) + model_spec_obj.use_pipeline_parallelism(pp_size) + model_spec_obj.use_context_parallelism(cp_size) + + build_engine( + hf_dir, engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + [ + f'--tp_size={tp_size}', f'--pp_size={pp_size}', + f'--cp_size={cp_size}' + ], ['--use_paged_context_fmha=disable']) + + duration = time.time() - start_time + print( + f"Building fp16 tp{tp_size} pp{pp_size} cp{cp_size} engine took {duration} seconds" + ) + + if not only_multi_gpu: + print(f"\nBuilding lookahead engine") + start_time = time.time() + + model_spec_obj.use_tensor_parallelism(1) + model_spec_obj.use_pipeline_parallelism(1) + model_spec_obj.use_context_parallelism(1) + model_spec_obj.use_lookahead_decoding() + build_engine( + hf_dir, + engine_dir / model_spec_obj.get_model_path() / 'tp1-pp1-cp1-gpu', + [], [ + '--max_draft_len=39', + '--speculative_decoding_mode=lookahead_decoding' + ]) + + duration = time.time() - start_time + print(f"Building lookahead engine took {duration} seconds") + + print("Done.") + + +if __name__ == "__main__": + parser = _arg.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + parser.add_argument( + "--only_multi_gpu", + action="store_true", + help="Flag to build only for Tensor and Pipeline parallelism") + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/build_mamba_engines.py b/cpp/tests/resources/scripts/build_mamba_engines.py new file mode 100644 index 000000000000..6b10a5b03531 --- /dev/null +++ b/cpp/tests/resources/scripts/build_mamba_engines.py @@ -0,0 +1,154 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import os as _os +import pathlib as _pl +import platform as _pf +import sys as _sys +import typing as _tp + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def build_engine(weight_dir: _pl.Path, ckpt_dir: _pl.Path, engine_dir: _pl.Path, + *args): + convert_args = [ + _sys.executable, "examples/models/core/mamba/convert_checkpoint.py" + ] + (['--model_dir', str(weight_dir)] if weight_dir else []) + [ + '--output_dir', + str(ckpt_dir), + '--dtype=float16', + ] + run_command(convert_args) + build_args = ["trtllm-build"] + ['--checkpoint_dir', + str(ckpt_dir)] + [ + '--output_dir', + str(engine_dir), + '--gpt_attention_plugin=disable', + '--paged_kv_cache=disable', + '--gemm_plugin=disable', + '--max_batch_size=8', + '--max_input_len=924', + '--max_seq_len=1024', + '--max_beam_width=1', + ] + list(args) + run_command(build_args) + + +def build_engines(model_cache: _tp.Optional[str] = None): + resources_dir = _pl.Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_name = 'mamba-2.8b-hf' + + if model_cache: + print("Copy model from model_cache") + model_cache_dir = _pl.Path(model_cache) / 'mamba' / model_name + if _pf.system() == "Windows": + wincopy(source=str(model_cache_dir), + dest=model_name, + isdir=True, + cwd=models_dir) + else: + run_command(["rsync", "-rlptD", + str(model_cache_dir), "."], + cwd=models_dir) + else: + print("Clone model from HF") + hf_dir = _pl.Path(models_dir) / model_name + run_command( + [ + "git", "clone", + "https://huggingface.co/state-spaces/mamba-2.8b-hf", model_name + ], + cwd=models_dir, + ) + hf_dir = models_dir / model_name + assert (hf_dir.is_dir()) + + # Clone or update the tokenizer directory without lfs + tokenizer_name = 'gpt-neox-20b' + tokenizer_hf_dir = models_dir / tokenizer_name + if tokenizer_hf_dir.exists(): + assert tokenizer_hf_dir.is_dir() + run_command(["git", "pull"], cwd=tokenizer_hf_dir) + else: + if _pf.system() == "Windows": + url_prefix = "" + else: + url_prefix = "file://" + tokenizer_url = url_prefix + str( + _pl.Path(model_cache) / tokenizer_name + ) if model_cache else "https://huggingface.co/EleutherAI/gpt-neox-20b" + run_command([ + "git", "clone", tokenizer_url, "--single-branch", "--no-local", + tokenizer_name + ], + cwd=tokenizer_hf_dir.parent, + env={ + **_os.environ, "GIT_LFS_SKIP_SMUDGE": "1" + }) + + tp_size = 1 + pp_size = 1 + cp_size = 1 + tp_pp_cp_dir = f"tp{tp_size}-pp{pp_size}-cp{cp_size}-gpu" + + ckpt_dir = models_dir / 'rt_ckpt' / model_name + engine_dir = models_dir / 'rt_engine' / model_name + model_spec_obj = ModelSpec('input_tokens.npy', _tb.DataType.HALF) + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.CONTINUOUS) + model_spec_obj.use_tensor_parallelism(tp_size) + model_spec_obj.use_pipeline_parallelism(pp_size) + model_spec_obj.use_context_parallelism(cp_size) + + print("\nBuilding fp16 engine") + build_engine(hf_dir, + ckpt_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--remove_input_padding=disable', '--paged_state=disable', + '--mamba_conv1d_plugin=disable') + print("\nBuilding fp16-plugin engine") + model_spec_obj.use_mamba_plugin() + build_engine(hf_dir, + ckpt_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--remove_input_padding=disable', '--paged_state=disable') + print("\nBuilding fp16-plugin-packed engine") + model_spec_obj.use_packed_input() + build_engine(hf_dir, + ckpt_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--remove_input_padding=enable', '--paged_state=disable') + print("\nBuilding fp16-plugin-packed-paged engine") + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + build_engine(hf_dir, + ckpt_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--remove_input_padding=enable', '--paged_state=enable') + print("Done.") + + +if __name__ == "__main__": + parser = _arg.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/build_medusa_engines.py b/cpp/tests/resources/scripts/build_medusa_engines.py new file mode 100755 index 000000000000..cf9c74f8779f --- /dev/null +++ b/cpp/tests/resources/scripts/build_medusa_engines.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import pathlib as _pl +import platform as _pf +import sys as _sys + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def build_engine(base_model_dir: _pl.Path, medusa_model_dir: _pl.Path, + engine_dir: _pl.Path, *args): + + covert_cmd = [_sys.executable, "examples/medusa/convert_checkpoint.py"] + ( + ['--model_dir', str(base_model_dir)] if base_model_dir else []) + [ + '--medusa_model_dir', str(medusa_model_dir), \ + '--output_dir', str(engine_dir), '--dtype=float16', '--num_medusa_heads=4' + ] + list(args) + + run_command(covert_cmd) + + build_args = ["trtllm-build"] + ( + ['--checkpoint_dir', str(engine_dir)] if engine_dir else []) + [ + '--output_dir', + str(engine_dir), + '--gemm_plugin=float16', + '--max_batch_size=8', + '--max_input_len=12', + '--max_seq_len=140', + '--log_level=error', + '--paged_kv_cache=enable', + '--use_paged_context_fmha=enable', + '--remove_input_padding=enable', + '--speculative_decoding_mode=medusa', + ] + + run_command(build_args) + + +def build_engines(model_cache: str): + resources_dir = _pl.Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_name = 'vicuna-7b-medusa' + base_model_name = 'vicuna-7b-v1.3' + medusa_model_name = 'medusa-vicuna-7b-v1.3' + + if model_cache: + print(f"Copy model from {model_cache}") + base_model_cache_dir = _pl.Path(model_cache) / base_model_name + medusa_head_cache_dir = _pl.Path(model_cache) / medusa_model_name + assert base_model_cache_dir.is_dir(), base_model_cache_dir + assert medusa_head_cache_dir.is_dir(), medusa_head_cache_dir + + if _pf.system() == "Windows": + wincopy(source=str(base_model_cache_dir), + dest=base_model_name, + isdir=True, + cwd=models_dir) + wincopy(source=str(medusa_head_cache_dir), + dest=medusa_model_name, + isdir=True, + cwd=models_dir) + else: + run_command(["rsync", "-rlptD", + str(base_model_cache_dir), "."], + cwd=models_dir) + run_command(["rsync", "-rlptD", + str(medusa_head_cache_dir), "."], + cwd=models_dir) + + base_model_dir = models_dir / base_model_name + medusa_model_dir = models_dir / medusa_model_name + assert base_model_dir.is_dir() + assert medusa_model_dir.is_dir() + + engine_dir = models_dir / 'rt_engine' / model_name + + model_spec_obj = ModelSpec('input_tokens.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + model_spec_obj.use_medusa() + + full_engine_path = engine_dir / model_spec_obj.get_model_path( + ) / 'tp1-pp1-cp1-gpu' + print(f"\nBuilding fp16 engine at {str(full_engine_path)}") + build_engine(base_model_dir, medusa_model_dir, full_engine_path) + + print("Done.") + + +if __name__ == "__main__": + parser = _arg.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/build_recurrentgemma_engines.py b/cpp/tests/resources/scripts/build_recurrentgemma_engines.py new file mode 100644 index 000000000000..293aab101d38 --- /dev/null +++ b/cpp/tests/resources/scripts/build_recurrentgemma_engines.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import os as _os +import pathlib as _pl +import platform as _pf +import sys as _sys +import typing as _tp + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def build_engine(weight_dir: _pl.Path, ckpt_dir: _pl.Path, engine_dir: _pl.Path, + *args): + convert_args = [ + _sys.executable, + "examples/models/core/recurrentgemma/convert_checkpoint.py" + ] + (['--model_dir', str(weight_dir)] if weight_dir else []) + [ + '--output_dir', + str(ckpt_dir), + '--ckpt_type=hf', + '--dtype=float16', + ] + run_command(convert_args) + build_args = ["trtllm-build"] + ['--checkpoint_dir', + str(ckpt_dir)] + [ + '--output_dir', + str(engine_dir), + '--gpt_attention_plugin=float16', + '--paged_kv_cache=enable', + '--gemm_plugin=float16', + '--max_batch_size=8', + '--max_input_len=924', + '--max_seq_len=1024', + '--max_beam_width=1', + ] + list(args) + run_command(build_args) + + +def build_engines(model_cache: _tp.Optional[str] = None): + resources_dir = _pl.Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_name = 'recurrentgemma-2b' + hf_dir = models_dir / model_name + + # Clone or update the model directory without lfs + if model_cache: + print("Copy model from model_cache") + model_cache_dir = _pl.Path(model_cache) / 'recurrentgemma' / model_name + print(model_cache_dir) + assert (model_cache_dir.is_dir()) + if _pf.system() == "Windows": + wincopy(source=str(model_cache_dir), + dest=model_name, + isdir=True, + cwd=models_dir) + else: + run_command(["rsync", "-rlptD", + str(model_cache_dir), "."], + cwd=models_dir) + else: + if not hf_dir.is_dir(): + if _pf.system() == "Windows": + url_prefix = "" + else: + url_prefix = "file://" + model_url = "https://huggingface.co/google/recurrentgemma-2b" + run_command([ + "git", "clone", model_url, "--single-branch", "--no-local", + model_name + ], + cwd=models_dir, + env={ + **_os.environ, "GIT_LFS_SKIP_SMUDGE": "1" + }) + + assert (hf_dir.is_dir()) + + # Download the model file + model_file_name = "*" + if not model_cache: + run_command(["git", "lfs", "pull", "--include", model_file_name], + cwd=hf_dir) + + tp_size = 1 + pp_size = 1 + cp_size = 1 + tp_pp_cp_dir = f"tp{tp_size}-pp{pp_size}-cp{cp_size}-gpu" + + ckpt_dir = models_dir / 'rt_ckpt' / model_name + engine_dir = models_dir / 'rt_engine' / model_name + + python_exe = _sys.executable + run_command([python_exe, "-m", "pip", "install", "transformers>=4.40.0"], + env=_os.environ, + timeout=300) + input_file = 'input_tokens.npy' + model_spec_obj = ModelSpec(input_file, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.use_packed_input() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + + print("\nBuilding fp16-plugin-packed-paged engine") + build_engine(hf_dir, + ckpt_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + engine_dir / model_spec_obj.get_model_path() / tp_pp_cp_dir, + '--remove_input_padding=enable', '--paged_state=enable') + + print("Done.") + + +if __name__ == "__main__": + parser = _arg.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/build_redrafter_engines.py b/cpp/tests/resources/scripts/build_redrafter_engines.py new file mode 100755 index 000000000000..cdf3e889ac35 --- /dev/null +++ b/cpp/tests/resources/scripts/build_redrafter_engines.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import pathlib as _pl +import platform as _pf +import sys as _sys + +from build_engines_utils import run_command, wincopy + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def build_engine(base_model_dir: _pl.Path, drafter_model_dir: _pl.Path, + engine_dir: _pl.Path, *args): + + base_ckpt_dir = f'{base_model_dir}-ckpt' + covert_cmd_base = [ + _sys.executable, "examples/models/core/llama/convert_checkpoint.py" + ] + (['--model_dir', str(base_model_dir)] if base_model_dir else []) + [ + '--output_dir', str(base_ckpt_dir), '--dtype=float16' + ] + list(args) + + run_command(covert_cmd_base) + + covert_cmd = [ + _sys.executable, "examples/redrafter/convert_checkpoint.py"] + ( + ['--base_model_checkpoint_dir', str(base_ckpt_dir)] if base_model_dir else []) + [ + '--drafter_model_dir', str(drafter_model_dir), \ + '--output_dir', str(engine_dir), '--dtype=float16', + '--redrafter_num_beams=5', '--redrafter_draft_len_per_beam=5' + ] + list(args) + + run_command(covert_cmd) + + build_args = ["trtllm-build"] + ( + ['--checkpoint_dir', str(engine_dir)] if engine_dir else []) + [ + '--output_dir', + str(engine_dir), + '--gemm_plugin=float16', + '--max_batch_size=8', + '--max_input_len=64', + '--max_seq_len=1024', + '--log_level=error', + '--paged_kv_cache=enable', + '--use_paged_context_fmha=enable', + '--remove_input_padding=enable', + '--speculative_decoding_mode=explicit_draft_tokens', + ] + + run_command(build_args) + + +def build_engines(model_cache: str): + resources_dir = _pl.Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_name = 'vicuna-7b-redrafter' + base_model_name = 'vicuna-7b-v1.3' + drafter_model_name = 'redrafter-vicuna-7b-v1.3' + + if model_cache: + print(f"Copy model from {model_cache}") + base_model_cache_dir = _pl.Path(model_cache) / base_model_name + drafter_cache_dir = _pl.Path(model_cache) / drafter_model_name + assert base_model_cache_dir.is_dir(), base_model_cache_dir + assert drafter_cache_dir.is_dir(), drafter_cache_dir + + if _pf.system() == "Windows": + wincopy(source=str(base_model_cache_dir), + dest=base_model_name, + isdir=True, + cwd=models_dir) + wincopy(source=str(drafter_cache_dir), + dest=drafter_model_name, + isdir=True, + cwd=models_dir) + else: + run_command(["rsync", "-rlptD", + str(base_model_cache_dir), "."], + cwd=models_dir) + run_command(["rsync", "-rlptD", + str(drafter_cache_dir), "."], + cwd=models_dir) + + base_model_dir = models_dir / base_model_name + drafter_model_dir = models_dir / drafter_model_name + assert base_model_dir.is_dir() + assert drafter_model_dir.is_dir() + + engine_dir = models_dir / 'rt_engine' / model_name + + model_spec_obj = ModelSpec('input_tokens.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + model_spec_obj.use_explicit_draft_tokens_decoding() + + full_engine_path = engine_dir / model_spec_obj.get_model_path( + ) / 'tp1-pp1-cp1-gpu' + print(f"\nBuilding fp16 engine at {str(full_engine_path)}") + build_engine(base_model_dir, drafter_model_dir, full_engine_path) + + print("Done.") + + +if __name__ == "__main__": + parser = _arg.ArgumentParser() + parser.add_argument("--model_cache", + type=str, + help="Directory where models are stored") + + build_engines(**vars(parser.parse_args())) diff --git a/cpp/tests/resources/scripts/generate_expected_chatglm_output.py b/cpp/tests/resources/scripts/generate_expected_chatglm_output.py new file mode 100755 index 000000000000..416f76938700 --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_chatglm_output.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +from pathlib import Path + +import numpy as np + +# isort: off +import run +# isort: on + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + +resources_dir = Path(__file__).parent.resolve().parent +model_path = resources_dir / "models" + + +def generate_output( + model_name: str = "", + num_beams: int = 1, + max_output_len: int = 8, + output_logits: bool = False, + output_cum_log_probs: bool = False, + output_log_probs: bool = False, +): + hf_path = model_path / model_name + tp_size = 1 + pp_size = 1 + cp_size = 1 + tp_pp_cp_dir = f"tp{tp_size}-pp{pp_size}-cp{cp_size}-gpu/" + input_file = f"input_tokens_{model_name}.npy" + + data_input_file_name = resources_dir / "data" / input_file + if num_beams == 1: + output_dir = resources_dir / "data" / model_name / "sampling" + else: + output_dir = resources_dir / "data" / model_name / f"beam_search_{num_beams}" + output_dir.mkdir(exist_ok=True, parents=True) + + model_spec_obj_list = [ + ModelSpec(input_file, + _tb.DataType.HALF).use_gpt_plugin().set_kv_cache_type( + _tb.KVCacheType.CONTINUOUS), + ModelSpec(input_file, _tb.DataType.HALF).use_gpt_plugin(). + use_packed_input().set_kv_cache_type(_tb.KVCacheType.PAGED), + ] + + for model_spec_obj in model_spec_obj_list: + engine_dir = model_path / 'rt_engine' / model_name / model_spec_obj.get_model_path( + ) / tp_pp_cp_dir + base_output_name = os.path.splitext( + model_spec_obj.get_results_file())[0] + output_npy_file_name = output_dir / f'{base_output_name}.npy' + output_csv_file_name = output_dir / f'{base_output_name}.csv' + + args_list = [ + '--engine_dir', + str(engine_dir), + '--tokenizer_dir', + str(hf_path), + '--input_file', + str(data_input_file_name), + '--output_npy', + str(output_npy_file_name), + '--output_csv', + str(output_csv_file_name), + '--max_output_len', + str(max_output_len), + '--num_beams', + str(num_beams), + '--use_py_session', + ] + + if output_logits: + file_name = str(output_npy_file_name)[:-4] + "_logits.npy" + args_list.extend(['--output_logits_npy', file_name]) + + if output_cum_log_probs: + file_name = str(output_npy_file_name)[:-4] + "_cum_log_probs.npy" + args_list.extend(['--output_cum_log_probs_npy', file_name]) + + if output_log_probs: + file_name = str(output_npy_file_name)[:-4] + "_log_probs.npy" + args_list.extend(['--output_log_probs_npy', file_name]) + + args = run.parse_arguments(args_list) + run.main(args) + + # Convert pad_id to end_id in .npy out put file + data = np.load(str(output_npy_file_name)) + if model_name == 'chatglm-6b': + data[data == 3] = 130005 + elif model_name == 'chatglm2-6b' or model_name == 'chatglm3-6b': + data[data == 0] = 2 + elif model_name == 'glm-10b': + data[data == 50256] = 50258 + else: + raise NameError('bad model name') + + np.save(str(output_npy_file_name), data) + + +if __name__ == '__main__': + generate_output(model_name='chatglm-6b', num_beams=1) + generate_output(model_name='chatglm-6b', num_beams=2) + generate_output(model_name='chatglm2-6b', num_beams=1) + generate_output(model_name='chatglm2-6b', num_beams=2) + generate_output(model_name='chatglm3-6b', num_beams=1) + generate_output(model_name='chatglm3-6b', num_beams=2) + generate_output(model_name='glm-10b', num_beams=1) + print("Done") diff --git a/cpp/tests/resources/scripts/generate_expected_eagle_output.py b/cpp/tests/resources/scripts/generate_expected_eagle_output.py new file mode 100755 index 000000000000..253a98beaf4e --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_eagle_output.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import os +from pathlib import Path + +# isort: off +import run +# isort: on + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def generate_output(engine: str, + model_spec_obj: ModelSpec, + max_output_len: int = 8): + + model = 'vicuna-7b-v1.3' + model_eagle = 'vicuna-7b-eagle' + hf_model = 'vicuna-7b-v1.3' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + hf_dir = models_dir / hf_model + tp_pp_cp_dir = 'tp1-pp1-cp1-gpu/' + engine_dir = models_dir / 'rt_engine' / model / engine / tp_pp_cp_dir + + data_dir = resources_dir / 'data' + input_file = data_dir / 'input_vicuna.npy' + model_data_dir = data_dir / model_eagle + output_dir = model_data_dir / 'sampling' + + base_output_name = os.path.splitext(model_spec_obj.get_results_file())[0] + + args = run.parse_arguments([ + '--engine_dir', + str(engine_dir), '--input_file', + str(input_file), '--tokenizer_dir', + str(hf_dir), '--output_npy', + str(output_dir / (base_output_name + '.npy')), '--output_csv', + str(output_dir / (base_output_name + '.csv')), '--max_output_len', + str(max_output_len), '--use_py_session', '--temperature', '1.0' + ]) + run.main(args) + print(f"Output saved at {str(output_dir / base_output_name)}.[npy|csv]") + + +def generate_outputs(): + print(f'Generating outputs for Vicuna 7B v1.3 FP16') + max_output_len = 128 + model_spec_obj = ModelSpec('input_tokens_long.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_max_output_length(max_output_len) + model_spec_obj.use_packed_input() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + + generate_output(engine=model_spec_obj.get_model_path(), + model_spec_obj=model_spec_obj, + max_output_len=max_output_len) + + +if __name__ == '__main__': + parser = _arg.ArgumentParser() + parser.add_argument( + "--only_multi_gpu", + action="store_true", + help="Generate data with Pipeline and Tensor Parallelism") + + args = parser.parse_args() + + generate_outputs() + print("Done") diff --git a/cpp/tests/resources/scripts/generate_expected_enc_dec_output.py b/cpp/tests/resources/scripts/generate_expected_enc_dec_output.py new file mode 100644 index 000000000000..fc3dc615c918 --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_enc_dec_output.py @@ -0,0 +1,30 @@ +from build_enc_dec_engines import Arguments, RunCMDMixin + + +class Run(RunCMDMixin): + + def command(self): + args = self.args + world_size = args.tp * args.pp + mpi_run = f'mpirun --allow-run-as-root -np {world_size}' if world_size > 1 else '' + ret = [] + for beam in args.beams_tuple: + ret.append(( + mpi_run, + f'python3 examples/models/core/enc_dec/run.py --engine_dir {args.engines_dir}', + f'--engine_name {args.ckpt}', + f'--model_name "{args.hf_models_dir}"', + f'--max_new_tokens={args.max_new_tokens}', + f'--num_beams={beam}', + f'--compare_hf_fp32', + f'--output_npy={args.data_dir}', + "--debug_mode" if args.debug else "", + )) + ret = [' '.join(x) for x in ret] + ret = ' && '.join(ret) + return ret + + +if __name__ == '__main__': + args = Arguments() + Run(args).run() diff --git a/cpp/tests/resources/scripts/generate_expected_gpt_output.py b/cpp/tests/resources/scripts/generate_expected_gpt_output.py new file mode 100755 index 000000000000..16fa5cc8db64 --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_gpt_output.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +from pathlib import Path + +# isort: off +import run +# isort: on + +import os +import shutil + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec, QuantMethod + + +def get_model_data_dir(): + resources_dir = Path(__file__).parent.resolve().parent + data_dir = resources_dir / 'data' + return data_dir / 'gpt2' + + +def generate_output(engine: str, + num_beams: int, + input_name: str, + model_spec_obj: ModelSpec, + max_output_len: int = 8, + output_logits: bool = False, + output_cum_log_probs: bool = False, + output_log_probs: bool = False): + tp_size = 1 + pp_size = 1 + cp_size = 1 + model = 'gpt2' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + tp_pp_cp_dir = 'tp' + str(tp_size) + '-pp' + str(pp_size) + '-cp' + str( + cp_size) + '-gpu/' + engine_dir = models_dir / 'rt_engine' / model / engine / tp_pp_cp_dir + + data_dir = resources_dir / 'data' + input_file = data_dir / input_name + model_data_dir = get_model_data_dir() + if num_beams <= 1: + output_dir = model_data_dir / 'sampling' + else: + output_dir = model_data_dir / ('beam_search_' + str(num_beams)) + + model_spec_obj.use_tensor_parallelism(tp_size).use_pipeline_parallelism( + pp_size).use_context_parallelism(cp_size) + + base_output_name = os.path.splitext(model_spec_obj.get_results_file())[0] + + args_list = [ + f'--engine_dir={engine_dir}', + f'--input_file={input_file}', + f'--tokenizer_dir={models_dir / model}', + f'--output_npy={output_dir / (base_output_name + ".npy")}', + f'--output_csv={output_dir / (base_output_name + ".csv")}', + f'--max_output_len={max_output_len}', + f'--num_beams={num_beams}', + '--use_py_session', + ] + + if output_logits: + args_list.extend([ + f'--output_logits_npy={output_dir / (base_output_name + "_logits.npy")}', + '--output_generation_logits', + ]) + + # Generate context_fmha_fp32_acc enabled results for GptExecutorTest.GenerationLogitsEarlyStop + if model_spec_obj.get_enable_context_fmha_fp32_acc(): + args_list.extend(["--enable_context_fmha_fp32_acc"]) + + if output_cum_log_probs: + args_list.extend([ + f'--output_cum_log_probs_npy={output_dir / model_spec_obj.get_cum_log_probs_file()}' + ]) + + if output_log_probs: + args_list.extend([ + f'--output_log_probs_npy={output_dir / model_spec_obj.get_log_probs_file()}' + ]) + + args = run.parse_arguments(args_list) + run.main(args) + + +def generate_outputs(num_beams): + input_name = 'input_tokens.npy' + input_name_long = 'input_tokens_long.npy' + + print('Generating GPT2 FP16 outputs') + model_spec_obj = ModelSpec(input_name, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.use_packed_input() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.gather_logits() + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj, + output_logits=True, + output_log_probs=True, + output_cum_log_probs=True) + # GptExecutorTest.GenerationLogitsEarlyStop and several tests require to use context_fmha_fp32_acc flag in runtime + model_spec_obj.enable_context_fmha_fp32_acc() + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj, + output_logits=True, + output_log_probs=True, + output_cum_log_probs=True) + + model_spec_obj = ModelSpec(input_name, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj, + output_logits=False, + output_log_probs=True, + output_cum_log_probs=True) + model_spec_obj.enable_context_fmha_fp32_acc() + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj, + output_logits=False, + output_log_probs=True, + output_cum_log_probs=True) + model_spec_obj.set_max_output_length(128) + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj, + output_logits=False, + max_output_len=128) + + model_spec_obj = ModelSpec(input_name_long, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.use_packed_input() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name_long, + model_spec_obj=model_spec_obj, + output_logits=False) + + model_spec_obj = ModelSpec(input_name, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.use_packed_input() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.set_quant_method(QuantMethod.SMOOTH_QUANT) + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj, + output_logits=False) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument('--clean', + action='store_true', + default=False, + help='Clean target folders before building engines') + args = parser.parse_args() + if args.clean: + model_data_dir = get_model_data_dir() + print(f'Cleaning target folder {model_data_dir}') + shutil.rmtree(model_data_dir, ignore_errors=True) + generate_outputs(num_beams=1) + generate_outputs(num_beams=2) diff --git a/cpp/tests/resources/scripts/generate_expected_gptj_output.py b/cpp/tests/resources/scripts/generate_expected_gptj_output.py new file mode 100755 index 000000000000..8d650d6bfc31 --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_gptj_output.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import os +from pathlib import Path + +# isort: off +import run +# isort: on + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def generate_output(engine: str, + num_beams: int, + model_spec_obj: ModelSpec, + max_output_len: int = 4): + + tp_size = 1 + pp_size = 1 + cp_size = 1 + model = 'gpt-j-6b' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + hf_dir = models_dir / model + tp_pp_cp_dir = 'tp' + str(tp_size) + '-pp' + str(pp_size) + '-cp' + str( + cp_size) + '-gpu/' + engine_dir = models_dir / 'rt_engine' / model / engine / tp_pp_cp_dir + + data_dir = resources_dir / 'data' + input_file = data_dir / 'input_tokens.npy' + model_data_dir = data_dir / model + if num_beams <= 1: + output_dir = model_data_dir / 'sampling' + else: + output_dir = model_data_dir / ('beam_search_' + str(num_beams)) + + base_output_name = os.path.splitext(model_spec_obj.get_results_file())[0] + + args = run.parse_arguments([ + '--engine_dir', + str(engine_dir), '--input_file', + str(input_file), '--tokenizer_dir', + str(hf_dir), '--output_npy', + str(output_dir / (base_output_name + '.npy')), '--output_csv', + str(output_dir / (base_output_name + '.csv')), '--max_output_len', + str(max_output_len), '--num_beams', + str(num_beams), '--use_py_session' + ]) + run.main(args) + + +def generate_outputs(only_fp8, num_beams): + input_file = 'input_tokens.npy' + if only_fp8 and num_beams == 1: + model_spec_obj = ModelSpec(input_file, _tb.DataType.FP8) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + + print('Generating GPT-J FP8-kv-cache outputs') + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + model_spec_obj=model_spec_obj) + elif not only_fp8: + print('Generating GPT-J FP16 outputs') + model_spec_obj = ModelSpec(input_file, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.CONTINUOUS) + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + model_spec_obj=model_spec_obj) + + model_spec_obj.use_packed_input() + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + model_spec_obj=model_spec_obj) + + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + model_spec_obj=model_spec_obj) + + +if __name__ == '__main__': + parser = _arg.ArgumentParser() + parser.add_argument( + "--only_fp8", + action="store_true", + help="Generate data for only FP8 tests. Implemented for H100 runners.") + + generate_outputs(**vars(parser.parse_args()), num_beams=1) + generate_outputs(**vars(parser.parse_args()), num_beams=2) diff --git a/cpp/tests/resources/scripts/generate_expected_llama_output.py b/cpp/tests/resources/scripts/generate_expected_llama_output.py new file mode 100644 index 000000000000..74916e77d053 --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_llama_output.py @@ -0,0 +1,151 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import os +import time +from pathlib import Path + +from mpi4py.MPI import COMM_WORLD + +# isort: off +import run +# isort: on + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def generate_output(engine: str, + num_beams: int, + model_spec_obj: ModelSpec, + tp_size: int = 1, + pp_size: int = 1, + cp_size: int = 1, + max_output_len: int = 8, + output_logits: bool = False, + output_cum_log_probs: bool = False, + output_log_probs: bool = False): + + model = 'Llama-3.2-1B' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + tp_pp_cp_dir = 'tp' + str(tp_size) + '-pp' + str(pp_size) + '-cp' + str( + cp_size) + '-gpu/' + engine_dir = models_dir / 'rt_engine' / model / engine / tp_pp_cp_dir + + data_dir = resources_dir / 'data' + input_file = data_dir / 'input_tokens_llama.npy' + model_data_dir = data_dir / model + if num_beams <= 1: + output_dir = model_data_dir / 'sampling' + else: + output_dir = model_data_dir / ('beam_search_' + str(num_beams)) + + base_output_name = os.path.splitext(model_spec_obj.get_results_file())[0] + + args_list = [ + f'--engine_dir={engine_dir}', + f'--input_file={input_file}', + f'--tokenizer_dir={models_dir / model}', + f'--output_npy={output_dir / (base_output_name + ".npy")}', + f'--output_csv={output_dir / (base_output_name + ".csv")}', + f'--max_output_len={max_output_len}', + f'--num_beams={num_beams}', + '--use_py_session', + ] + + if output_logits: + args_list.extend([ + f'--output_logits_npy={output_dir / (base_output_name + "_logits.npy")}', + '--output_generation_logits', + ]) + + if output_cum_log_probs: + args_list.extend([ + f'--output_cum_log_probs_npy={output_dir / model_spec_obj.get_cum_log_probs_file()}' + ]) + + if output_log_probs: + args_list.extend([ + f'--output_log_probs_npy={output_dir / model_spec_obj.get_log_probs_file()}' + ]) + + args = run.parse_arguments(args_list) + run.main(args) + + +def generate_outputs(num_beams, only_multi_gpu=False): + if not only_multi_gpu: + tp_pp_cp_sizes = [(1, 1, 1)] + elif COMM_WORLD.size == 4: + tp_pp_cp_sizes = [(4, 1, 1), (2, 2, 1), (1, 4, 1)] + elif COMM_WORLD.size == 2: + tp_pp_cp_sizes = [(1, 2, 1), (2, 1, 1)] + else: + raise RuntimeError( + f"The world size of MPI {COMM_WORLD.size} is not equal to 1, 2, or 4." + ) + model_spec_obj = ModelSpec('input_tokens_llama.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + + for tp_size, pp_size, cp_size in tp_pp_cp_sizes: + print( + f'Generating outputs for Llama FP16 with TP={tp_size}, PP={pp_size}, CP={cp_size}, BW={num_beams}' + ) + start_time = time.time() + + output_logits = False + output_log_probs = False + output_cum_log_probs = False + if tp_size == 4 and pp_size == 1: + output_logits = True + output_log_probs = True + output_cum_log_probs = True + + model_spec_obj.use_tensor_parallelism(tp_size) + model_spec_obj.use_pipeline_parallelism(pp_size) + model_spec_obj.use_context_parallelism(cp_size) + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + tp_size=tp_size, + pp_size=pp_size, + cp_size=cp_size, + model_spec_obj=model_spec_obj, + output_logits=output_logits, + output_log_probs=output_log_probs, + output_cum_log_probs=output_cum_log_probs) + + duration = time.time() - start_time + print( + f"Generating outputs for Llama FP16 with TP={tp_size}, PP={pp_size}, CP={cp_size}, BW={num_beams} took {duration} seconds" + ) + + +if __name__ == '__main__': + parser = _arg.ArgumentParser() + parser.add_argument( + "--only_multi_gpu", + action="store_true", + help="Generate data with Pipeline and Tensor Parallelism") + + args = parser.parse_args() + + generate_outputs(num_beams=1, only_multi_gpu=args.only_multi_gpu) + generate_outputs(num_beams=2, only_multi_gpu=args.only_multi_gpu) + print("Done") diff --git a/cpp/tests/resources/scripts/generate_expected_mamba_output.py b/cpp/tests/resources/scripts/generate_expected_mamba_output.py new file mode 100644 index 000000000000..16779c434775 --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_mamba_output.py @@ -0,0 +1,107 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +from pathlib import Path + +# isort: off +import run +# isort: on + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def generate_output(engine: str, + num_beams: int, + input_name: str, + model_spec_obj: ModelSpec, + max_output_len: int = 8, + output_logits: bool = False): + tp_size = 1 + pp_size = 1 + cp_size = 1 + model = 'mamba-2.8b-hf' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + tp_pp_cp_dir = 'tp' + str(tp_size) + '-pp' + str(pp_size) + '-cp' + str( + cp_size) + '-gpu/' + engine_dir = models_dir / 'rt_engine' / model / engine / tp_pp_cp_dir + + data_dir = resources_dir / 'data' + input_file = data_dir / input_name + model_data_dir = data_dir / model + if num_beams <= 1: + output_dir = model_data_dir / 'sampling' + else: + output_dir = model_data_dir / ('beam_search_' + str(num_beams)) + + base_output_name = os.path.splitext(model_spec_obj.get_results_file())[0] + + output_logits_npy = None + if output_logits: + output_logits_npy = str(output_dir / + (base_output_name + '_logits' + '.npy')) + + args = run.parse_arguments([ + '--engine_dir', + str(engine_dir), '--input_file', + str(input_file), '--tokenizer_dir', + str(models_dir / 'gpt-neox-20b'), '--output_npy', + str(output_dir / (base_output_name + '.npy')), '--output_csv', + str(output_dir / (base_output_name + '.csv')), '--max_output_len', + str(max_output_len), '--num_beams', + str(num_beams), '--output_logits_npy', + str(output_logits_npy), '--use_py_session' + ]) + run.main(args) + + +def generate_outputs(num_beams): + print('Generating Mamba FP16 outputs') + input_name = 'input_tokens.npy' + model_spec_obj = ModelSpec(input_name, _tb.DataType.HALF) + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.CONTINUOUS) + + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj) + + print('Generating Mamba FP16-plugin outputs') + model_spec_obj.use_gpt_plugin() + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj) + + print('Generating Mamba FP16-plugin-packed outputs') + model_spec_obj.use_packed_input() + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj) + + print('Generating Mamba FP16-plugin-packed-paged outputs') + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_name, + model_spec_obj=model_spec_obj) + + +if __name__ == '__main__': + generate_outputs(num_beams=1) diff --git a/cpp/tests/resources/scripts/generate_expected_medusa_output.py b/cpp/tests/resources/scripts/generate_expected_medusa_output.py new file mode 100755 index 000000000000..e1cbc20c051b --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_medusa_output.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import os +from pathlib import Path + +# isort: off +import run +# isort: on + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def generate_output(engine: str, + model_spec_obj: ModelSpec, + max_output_len: int = 8): + + model = 'vicuna-7b-medusa' + hf_model = 'vicuna-7b-v1.3' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + hf_dir = models_dir / hf_model + tp_pp_dir = 'tp1-pp1-cp1-gpu/' + engine_dir = models_dir / 'rt_engine' / model / engine / tp_pp_dir + + data_dir = resources_dir / 'data' + input_file = data_dir / 'input_vicuna.npy' + model_data_dir = data_dir / model + output_dir = model_data_dir / 'sampling' + + base_output_name = os.path.splitext(model_spec_obj.get_results_file())[0] + + args = run.parse_arguments([ + '--engine_dir', + str(engine_dir), '--input_file', + str(input_file), '--tokenizer_dir', + str(hf_dir), '--output_npy', + str(output_dir / (base_output_name + '.npy')), '--output_csv', + str(output_dir / (base_output_name + '.csv')), '--max_output_len', + str(max_output_len), '--use_py_session', + '--medusa_choices=[[0], [0, 0], [1], [0, 1], [2], [0, 0, 0], [1, 0], [0, 2], [3], [0, 3], [4], [0, 4], [2, 0], [0, 5], [0, 0, 1], [5], [0, 6], [6], [0, 7], [0, 1, 0], [1, 1], [7], [0, 8], [0, 0, 2], [3, 0], [0, 9], [8], [9], [1, 0, 0], [0, 2, 0], [1, 2], [0, 0, 3], [4, 0], [2, 1], [0, 0, 4], [0, 0, 5], [0, 0, 0, 0], [0, 1, 1], [0, 0, 6], [0, 3, 0], [5, 0], [1, 3], [0, 0, 7], [0, 0, 8], [0, 0, 9], [6, 0], [0, 4, 0], [1, 4], [7, 0], [0, 1, 2], [2, 0, 0], [3, 1], [2, 2], [8, 0], [0, 5, 0], [1, 5], [1, 0, 1], [0, 2, 1], [9, 0], [0, 6, 0], [0, 0, 0, 1], [1, 6], [0, 7, 0]]', + '--temperature', '1.0' + ]) + run.main(args) + print(f"Output saved at {str(output_dir / base_output_name)}.[npy|csv]") + + +def generate_outputs(): + print(f'Generating outputs for Medusa FP16') + max_output_len = 128 + model_spec_obj = ModelSpec('input_tokens_long.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_max_output_length(max_output_len) + model_spec_obj.use_packed_input() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_medusa() + + generate_output(engine=model_spec_obj.get_model_path(), + model_spec_obj=model_spec_obj, + max_output_len=max_output_len) + + +if __name__ == '__main__': + parser = _arg.ArgumentParser() + parser.add_argument( + "--only_multi_gpu", + action="store_true", + help="Generate data with Pipeline and Tensor Parallelism") + + args = parser.parse_args() + + generate_outputs() + print("Done") diff --git a/cpp/tests/resources/scripts/generate_expected_recurrentgemma_output.py b/cpp/tests/resources/scripts/generate_expected_recurrentgemma_output.py new file mode 100644 index 000000000000..0ef4cc4509fd --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_recurrentgemma_output.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +from pathlib import Path + +# isort: off +import run +# isort: on + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def generate_output(engine: str, + num_beams: int, + input_name: str, + model_spec_obj: ModelSpec, + max_output_len: int = 8, + output_logits: bool = False): + tp_size = 1 + pp_size = 1 + cp_size = 1 + model = 'recurrentgemma-2b' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + tp_pp_cp_dir = 'tp' + str(tp_size) + '-pp' + str(pp_size) + '-cp' + str( + cp_size) + '-gpu/' + engine_dir = models_dir / 'rt_engine' / model / engine / tp_pp_cp_dir + + data_dir = resources_dir / 'data' + input_file = data_dir / input_name + model_data_dir = data_dir / model + if num_beams <= 1: + output_dir = model_data_dir / 'sampling' + else: + output_dir = model_data_dir / ('beam_search_' + str(num_beams)) + + base_output_name = os.path.splitext(model_spec_obj.get_results_file())[0] + + output_logits_npy = None + if output_logits: + output_logits_npy = str(output_dir / + (base_output_name + '_logits' + '.npy')) + + args = run.parse_arguments([ + '--engine_dir', + str(engine_dir), '--input_file', + str(input_file), '--tokenizer_dir', + str(models_dir / model), '--output_npy', + str(output_dir / (base_output_name + '.npy')), '--output_csv', + str(output_dir / (base_output_name + '.csv')), '--max_output_len', + str(max_output_len), '--num_beams', + str(num_beams), '--output_logits_npy', + str(output_logits_npy), '--use_py_session' + ]) + run.main(args) + + +def generate_outputs(num_beams): + input_file = 'input_tokens.npy' + model_spec_obj = ModelSpec(input_file, _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_packed_input() + + print('Generating RecurrentGemma FP16-plugin-packed-paged outputs') + generate_output(engine=model_spec_obj.get_model_path(), + num_beams=num_beams, + input_name=input_file, + model_spec_obj=model_spec_obj) + + +if __name__ == '__main__': + generate_outputs(num_beams=1) diff --git a/cpp/tests/resources/scripts/generate_expected_redrafter_output.py b/cpp/tests/resources/scripts/generate_expected_redrafter_output.py new file mode 100644 index 000000000000..989e029a5ab1 --- /dev/null +++ b/cpp/tests/resources/scripts/generate_expected_redrafter_output.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse as _arg +import os +from pathlib import Path + +# isort: off +import run +# isort: on + +import tensorrt_llm.bindings as _tb +from tensorrt_llm.bindings.internal.testing import ModelSpec + + +def generate_output(engine: str, + model_spec_obj: ModelSpec, + max_output_len: int = 8): + + model = 'vicuna-7b-redrafter' + hf_model = 'vicuna-7b-v1.3' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + hf_dir = models_dir / hf_model + tp_pp_dir = 'tp1-pp1-cp1-gpu/' + engine_dir = models_dir / 'rt_engine' / model / engine / tp_pp_dir + + data_dir = resources_dir / 'data' + input_filename = model_spec_obj.get_input_file() + input_file = data_dir / input_filename + model_data_dir = data_dir / model + output_dir = model_data_dir / 'sampling' + + base_output_name = os.path.splitext(model_spec_obj.get_results_file())[0] + + args = run.parse_arguments([ + '--engine_dir', + str(engine_dir), + '--input_file', + str(input_file), + '--tokenizer_dir', + str(hf_dir), + '--output_npy', + str(output_dir / (base_output_name + '.npy')), + '--output_csv', + str(output_dir / (base_output_name + '.csv')), + '--max_output_len', + str(max_output_len), + '--use_py_session', + ]) + run.main(args) + print(f"Output saved at {str(output_dir / base_output_name)}.[npy|csv]") + + +def generate_outputs(): + print(f'Generating outputs for ReDrafter FP16') + max_output_len = 128 + model_spec_obj = ModelSpec('input_vicuna.npy', _tb.DataType.HALF) + model_spec_obj.use_gpt_plugin() + model_spec_obj.set_max_output_length(max_output_len) + model_spec_obj.use_packed_input() + model_spec_obj.set_kv_cache_type(_tb.KVCacheType.PAGED) + model_spec_obj.use_explicit_draft_tokens_decoding() + + generate_output(engine=model_spec_obj.get_model_path(), + model_spec_obj=model_spec_obj, + max_output_len=max_output_len) + + +if __name__ == '__main__': + parser = _arg.ArgumentParser() + args = parser.parse_args() + + generate_outputs() + print("Done") diff --git a/cpp/tests/resources/scripts/generate_hf_gpt_output.py b/cpp/tests/resources/scripts/generate_hf_gpt_output.py new file mode 100755 index 000000000000..a40ada8cb455 --- /dev/null +++ b/cpp/tests/resources/scripts/generate_hf_gpt_output.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path + +import run_hf + + +def generate_hf_output(data_type: str, + output_name: str, + max_output_len: int = 8): + + model = 'gpt2' + resources_dir = Path(__file__).parent.resolve().parent + models_dir = resources_dir / 'models' + model_dir = models_dir / model + + data_dir = resources_dir / 'data' + input_file = data_dir / 'input_tokens.npy' + output_dir = data_dir / model / 'huggingface' + + run_hf.generate(model_dir=str(model_dir), + data_type=data_type, + input_file=str(input_file), + output_npy=str(output_dir / (output_name + '.npy')), + output_csv=str(output_dir / (output_name + '.csv')), + max_output_len=max_output_len) + + +def generate_hf_outputs(): + generate_hf_output(data_type='fp32', + output_name='output_tokens_fp32_huggingface') + generate_hf_output(data_type='fp16', + output_name='output_tokens_fp16_huggingface') + + +if __name__ == '__main__': + generate_hf_outputs() diff --git a/cpp/tests/resources/scripts/io_converter.py b/cpp/tests/resources/scripts/io_converter.py new file mode 100755 index 000000000000..0ed6413c5ac4 --- /dev/null +++ b/cpp/tests/resources/scripts/io_converter.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import csv +import os + +import numpy as np + + +def csv_to_npy(input_file, output_file, pad_id, verbose): + data = [] + with open(input_file, newline='') as csvfile: + csv_reader = csv.reader(csvfile, delimiter=',') + for line in csv_reader: + data.append([int(e) for e in line]) + max_input_length = max([len(x) for x in data]) + data = [row + [pad_id] * (max_input_length - len(row)) for row in data] + data = np.array(data, dtype='int32') + if (verbose): + print(data, data.dtype) + np.save(output_file, data) + + +def npy_to_csv(input_file, output_file, verbose): + data = np.load(input_file) + if (verbose): + print(data, data.dtype) + np.savetxt(output_file, data, delimiter=",", fmt='%i') + + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser.add_argument( + 'input_file', + type=str, + help='Read token ids from this file. Must be csv or npy.') + parser.add_argument('output_file', + type=str, + help='Write token ids this file. Must be csv or npy.') + parser.add_argument( + '-p', + '--pad_id', + type=int, + help= + 'Token id used for padding csv input with different sequence lengths.', + default=-1) + parser.add_argument('-v', '--verbose', action="store_true") + args = parser.parse_args() + + _, input_ext = os.path.splitext(args.input_file) + _, output_ext = os.path.splitext(args.output_file) + + if (input_ext == '.csv' and output_ext == '.npy'): + print('Converting csv to npy') + csv_to_npy(args.input_file, args.output_file, args.pad_id, args.verbose) + elif (input_ext == '.npy' and output_ext == '.csv'): + print('Converting npy to csv') + npy_to_csv(args.input_file, args.output_file, args.verbose) + else: + print('unknown file extensions') diff --git a/cpp/tests/unit_tests/CMakeLists.txt b/cpp/tests/unit_tests/CMakeLists.txt index 9d22bd03b52a..034de457bb78 100644 --- a/cpp/tests/unit_tests/CMakeLists.txt +++ b/cpp/tests/unit_tests/CMakeLists.txt @@ -27,3 +27,4 @@ add_subdirectory(multi_gpu) add_subdirectory(layers) add_subdirectory(runtime) add_subdirectory(thop) +add_subdirectory(utils) diff --git a/cpp/tests/unit_tests/batch_manager/CMakeLists.txt b/cpp/tests/unit_tests/batch_manager/CMakeLists.txt index 75329d192a8c..9425d5d13c2a 100644 --- a/cpp/tests/unit_tests/batch_manager/CMakeLists.txt +++ b/cpp/tests/unit_tests/batch_manager/CMakeLists.txt @@ -30,5 +30,7 @@ add_gtest(microBatchSchedulerTest microBatchSchedulerTest.cpp) add_gtest(peftCacheManagerTest peftCacheManagerTest.cpp) add_gtest(staticThreadPoolTest staticThreadPoolTest.cpp) add_gtest(rnnCacheFormatterTest rnnCacheFormatterTest.cpp) +add_gtest(cudaGraphExecutorCacheTest cudaGraphExecutorCacheTest.cpp) add_gtest(agentTreeTest agentTreeTest.cpp) add_gtest(truncateBlocksTest truncateBlocksTest.cpp) +add_gtest(encDecBeamSearchTest encDecBeamSearchTest.cpp) diff --git a/cpp/tests/unit_tests/batch_manager/bufferIndexHolderTest.cpp b/cpp/tests/unit_tests/batch_manager/bufferIndexHolderTest.cpp index 31400831a5bb..d4d561f55d07 100644 --- a/cpp/tests/unit_tests/batch_manager/bufferIndexHolderTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/bufferIndexHolderTest.cpp @@ -18,7 +18,7 @@ #include "tensorrt_llm/batch_manager/baseTransBuffer.h" #include "tensorrt_llm/batch_manager/cacheTransBuffer.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <gtest/gtest.h> #include <memory> #include <optional> @@ -48,16 +48,6 @@ class ObservableTransBufferManager : public CacheTransBufferManager { return mConcurrenceRecvResource.mConcurrence.load(); } - - void configureIndexPoolsForTest(size_t count) - { - ASSERT_EQ(mConcurrenceSendResource.mConcurrence.load(), 0); - ASSERT_EQ(mConcurrenceRecvResource.mConcurrence.load(), 0); - mSendBufferCount = count; - mRecvBufferCount = count; - mConcurrenceSendResource.mBufferIndexFlag.assign(count, 0); - mConcurrenceRecvResource.mBufferIndexFlag.assign(count, 0); - } }; } // namespace @@ -79,6 +69,8 @@ class BufferIndexHolderLifecycleTest : public ::testing::TestWithParam<HolderCas protected: void SetUp() override { + setenv("TRTLLM_USE_UCX_KVCACHE", "1", 1); + int constexpr numLayers = 2; int constexpr numHeads = 2; int constexpr sizePerHead = 8; @@ -95,13 +87,10 @@ class BufferIndexHolderLifecycleTest : public ::testing::TestWithParam<HolderCas mKv = std::make_unique<KVCacheManager>(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, maxBeamWidth, std::vector<BlockManager::SizeType32>{kvMaxNumTokens}, - tensorrt_llm::DataType::kFLOAT, sinkTokenLength, stream, kvMaxNumTokens, kvMaxNumTokens, + nvinfer1::DataType::kFLOAT, sinkTokenLength, stream, kvMaxNumTokens, kvMaxNumTokens, /*enableBlockReuse=*/true, CacheType::kSELF, std::nullopt, nullptr, true); mKv->allocatePools(false); mTrans = std::make_unique<ObservableTransBufferManager>(mKv.get(), std::optional<size_t>{kvMaxNumTokens}); - // envUtils caches process-wide settings, so set up the index-only test - // pools directly instead of relying on test-order-sensitive setenv calls. - mTrans->configureIndexPoolsForTest(2); } void TearDown() override @@ -168,44 +157,6 @@ TEST_P(BufferIndexHolderLifecycleTest, NulloptIndexReleasesNothing) EXPECT_EQ(inUse(), before); } -// A nullopt holder owns no concrete slot but retains the manager binding needed -// to poison a dynamic buffer after an unquiesced transfer exit. -TEST_P(BufferIndexHolderLifecycleTest, NulloptIndexCanPoisonManager) -{ - int const before = inUse(); - BufferIndexHolder holder{mgr(), std::nullopt, isRecv()}; - EXPECT_FALSE(holder.held()); - - holder.poison(); - - EXPECT_FALSE(holder.held()); - EXPECT_TRUE(mgr().hasPoisonedBuffer()); - EXPECT_EQ(inUse(), before); -} - -// Poisoning a concrete slot fails the entire direction closed: the slot stays -// reserved and no later request can acquire another buffer from that pool. -TEST_P(BufferIndexHolderLifecycleTest, ValidIndexPoisonPreventsReuse) -{ - int const before = inUse(); - auto idx = acquire(); - ASSERT_TRUE(idx.has_value()); - BufferIndexHolder holder{mgr(), idx, isRecv()}; - EXPECT_EQ(inUse(), before + 1); - - holder.poison(); - - EXPECT_FALSE(holder.held()); - EXPECT_TRUE(mgr().hasPoisonedBuffer()); - EXPECT_EQ(inUse(), before + 1); - EXPECT_THROW((void) acquire(), std::exception); - - // poison() disarms the holder; neither an explicit release nor its - // destructor may return the quarantined slot to the pool. - holder.release(); - EXPECT_EQ(inUse(), before + 1); -} - // RAII: a held slot is released when the holder goes out of scope. TEST_P(BufferIndexHolderLifecycleTest, ValidIndexReleasedOnDestruction) { @@ -298,7 +249,6 @@ TEST_P(BufferIndexHolderLifecycleTest, MoveConstructTransfersOwnership) TEST_P(BufferIndexHolderLifecycleTest, MoveAssignReleasesPriorThenTransfers) { int const before = inUse(); - ASSERT_GE(isRecv() ? mgr().getRecvBufferCount() : mgr().getSendBufferCount(), 2); auto firstIdx = acquire(); auto secondIdx = acquire(); ASSERT_TRUE(firstIdx.has_value()); diff --git a/cpp/tests/unit_tests/batch_manager/cacheTransBufferTest.cpp b/cpp/tests/unit_tests/batch_manager/cacheTransBufferTest.cpp index 2fa0477d2352..8150c6fa5406 100644 --- a/cpp/tests/unit_tests/batch_manager/cacheTransBufferTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/cacheTransBufferTest.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/batch_manager/cacheTransBuffer.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/common/envUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -52,7 +51,7 @@ class CacheTransBufferTest : public ::testing::Test auto constexpr blocksInSecondaryPool = 0; auto constexpr enableBlockReuse = true; - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; using BlocksPerWindow = std::map<SizeType32, std::tuple<SizeType32, SizeType32>>; auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {totalNumBlocks, blocksInSecondaryPool}}}; diff --git a/cpp/tests/unit_tests/batch_manager/capacitySchedulerTest.cpp b/cpp/tests/unit_tests/batch_manager/capacitySchedulerTest.cpp index 2c9757b847b2..7bb87c91e361 100644 --- a/cpp/tests/unit_tests/batch_manager/capacitySchedulerTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/capacitySchedulerTest.cpp @@ -31,7 +31,7 @@ #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/testing/kvCacheManagerTestUtil.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferPlugin.h> #include <cstdlib> #include <functional> @@ -132,7 +132,7 @@ class CapacitySchedulerTest : public ::testing::Test // NOLINT(cppcoreguidelines auto const nbKvHeads = 10; auto constexpr sizePerHead = 1; auto const maxNumBlocks = tc::divUp(maxNumTokens, tokensPerBlock); - auto const kvDtype = tensorrt_llm::DataType::kHALF; + auto const kvDtype = nvinfer1::DataType::kHALF; CudaStreamPtr streamPtr = std::make_shared<tensorrt_llm::runtime::CudaStream>(); using BlocksPerWindow = std::map<SizeType32, std::tuple<SizeType32, SizeType32>>; diff --git a/cpp/tests/unit_tests/batch_manager/cudaGraphExecutorCacheTest.cpp b/cpp/tests/unit_tests/batch_manager/cudaGraphExecutorCacheTest.cpp new file mode 100644 index 000000000000..f8ae0a9db64f --- /dev/null +++ b/cpp/tests/unit_tests/batch_manager/cudaGraphExecutorCacheTest.cpp @@ -0,0 +1,161 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/batch_manager/utils/inflightBatchingUtils.h" + +#include <gtest/gtest.h> + +#include <memory> + +namespace tb = tensorrt_llm::batch_manager; +namespace tbu = tensorrt_llm::batch_manager::utils; +using SizeType32 = tensorrt_llm::runtime::SizeType32; + +namespace +{ +// A default-constructed CudaGraphExecutor holds mInstance == nullptr, so its destructor +// is a no-op and these tests do not require an active CUDA context. +std::shared_ptr<tbu::CudaGraphExecutor> makeDummyExecutor() +{ + return std::make_shared<tbu::CudaGraphExecutor>(); +} + +tb::BatchState makeBatchState(SizeType32 numTokens) +{ + return tb::BatchState{/*numCtxRequests=*/0, /*numGenRequests=*/1, numTokens, /*maxKvCacheLength=*/256}; +} +} // namespace + +class CudaGraphExecutorCacheTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type-member-init) +{ +}; + +TEST_F(CudaGraphExecutorCacheTest, EmptyByDefault) +{ + tbu::CudaGraphExecutorCache cache(/*capacity=*/4); + EXPECT_EQ(cache.size(), 0); + EXPECT_FALSE(cache.get(makeBatchState(1)).has_value()); +} + +TEST_F(CudaGraphExecutorCacheTest, PutAndGetReturnsSameInstance) +{ + tbu::CudaGraphExecutorCache cache(/*capacity=*/4); + + auto bs = makeBatchState(1); + auto exec = makeDummyExecutor(); + cache.put(bs, exec); + + ASSERT_EQ(cache.size(), 1); + auto got = cache.get(bs); + ASSERT_TRUE(got.has_value()); + EXPECT_EQ(got->get(), exec.get()); +} + +TEST_F(CudaGraphExecutorCacheTest, PutWithExistingKeyReplaces) +{ + tbu::CudaGraphExecutorCache cache(/*capacity=*/4); + + auto bs = makeBatchState(1); + auto first = makeDummyExecutor(); + auto second = makeDummyExecutor(); + + cache.put(bs, first); + cache.put(bs, second); + + EXPECT_EQ(cache.size(), 1); + auto got = cache.get(bs); + ASSERT_TRUE(got.has_value()); + EXPECT_EQ(got->get(), second.get()); +} + +TEST_F(CudaGraphExecutorCacheTest, EvictsLeastRecentlyUsedAtCapacity) +{ + tbu::CudaGraphExecutorCache cache(/*capacity=*/2); + + auto bsA = makeBatchState(1); + auto bsB = makeBatchState(2); + auto bsC = makeBatchState(3); + + auto execA = makeDummyExecutor(); + auto execB = makeDummyExecutor(); + auto execC = makeDummyExecutor(); + + cache.put(bsA, execA); + cache.put(bsB, execB); + ASSERT_EQ(cache.size(), 2); + + // Access A so that B becomes the LRU entry. + EXPECT_TRUE(cache.get(bsA).has_value()); + + // Inserting C must evict B (the LRU), not A (just touched). + cache.put(bsC, execC); + EXPECT_EQ(cache.size(), 2); + EXPECT_TRUE(cache.get(bsA).has_value()); + EXPECT_FALSE(cache.get(bsB).has_value()); + EXPECT_TRUE(cache.get(bsC).has_value()); +} + +TEST_F(CudaGraphExecutorCacheTest, ClearDropsAllEntries) +{ + tbu::CudaGraphExecutorCache cache(/*capacity=*/4); + + auto bsA = makeBatchState(1); + auto bsB = makeBatchState(2); + auto bsC = makeBatchState(3); + + cache.put(bsA, makeDummyExecutor()); + cache.put(bsB, makeDummyExecutor()); + cache.put(bsC, makeDummyExecutor()); + ASSERT_EQ(cache.size(), 3); + + cache.clear(); + + EXPECT_EQ(cache.size(), 0); + EXPECT_FALSE(cache.get(bsA).has_value()); + EXPECT_FALSE(cache.get(bsB).has_value()); + EXPECT_FALSE(cache.get(bsC).has_value()); + + // After clearing, the cache must remain functional (i.e. clear() must not + // leave it in a broken state). + auto execA2 = makeDummyExecutor(); + cache.put(bsA, execA2); + EXPECT_EQ(cache.size(), 1); + auto got = cache.get(bsA); + ASSERT_TRUE(got.has_value()); + EXPECT_EQ(got->get(), execA2.get()); +} + +TEST_F(CudaGraphExecutorCacheTest, ClearReleasesExecutorOwnership) +{ + tbu::CudaGraphExecutorCache cache(/*capacity=*/4); + + auto exec = makeDummyExecutor(); + std::weak_ptr<tbu::CudaGraphExecutor> weak = exec; + + cache.put(makeBatchState(1), exec); + exec.reset(); + // The cache still owns one strong reference at this point. + ASSERT_FALSE(weak.expired()); + + cache.clear(); + + // After clear(), no strong references should remain. This guarantees that + // ~CudaGraphExecutor (which calls cudaGraphExecDestroy) actually runs for + // every cached entry - exactly what changeBeamWidth() relies on. + EXPECT_TRUE(weak.expired()); +} diff --git a/cpp/tests/unit_tests/batch_manager/encDecBeamSearchTest.cpp b/cpp/tests/unit_tests/batch_manager/encDecBeamSearchTest.cpp new file mode 100644 index 000000000000..8e96c6a52ee3 --- /dev/null +++ b/cpp/tests/unit_tests/batch_manager/encDecBeamSearchTest.cpp @@ -0,0 +1,154 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/kvCacheManager.h" +#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/runtimeBuffers.h" +#include "tensorrt_llm/batch_manager/utils/inflightBatchingUtils.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/kernels/kvCacheIndex.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/cudaStream.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/samplingConfig.h" +#include "gtest/gtest.h" +#include <memory> + +using namespace tensorrt_llm::batch_manager; +using namespace tensorrt_llm::batch_manager::kv_cache_manager; +namespace tr = tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; +namespace tk = tensorrt_llm::kernels; +using SizeType32 = tr::SizeType32; + +// Verify that copyGenerationLogits correctly assembles the host logits buffer +// using the real kernel merge path, and that two back-to-back calls (simulating +// two requests flushing in the same batch) use distinct fragmentPointerDevice +// slots so their pointer arrays do not clobber each other. +TEST(CopyGenerationLogitsTest, KernelMergePathProducesCorrectHostLayoutAndSlotsAreIsolated) +{ + SizeType32 constexpr beamWidth = 2; + SizeType32 constexpr numSteps = RuntimeBuffers::GenerationLogitsCache::kCACHE_LENGTH; // full flush + SizeType32 constexpr vocabSize = 8; + SizeType32 constexpr promptLen = 1; + SizeType32 constexpr maxBatchSize = 4; // must be >= 2 to test slot isolation + + auto stream = std::make_shared<tr::CudaStream>(); + tr::BufferManager bufferMgr{stream}; + + // Build a real GenerationLogitsCache so that transposedLogits, + // fragmentPointerDevice and fragmentPointerHost are all properly allocated. + // cache.logits uses pinned memory so the test can fill it from the CPU while + // the GPU kernel can still read from it via DMA. + RuntimeBuffers::GenerationLogitsCache cache; + cache.logits = tr::BufferManager::pinnedPool( + tr::ITensor::makeShape({numSteps, maxBatchSize * beamWidth, vocabSize}), nvinfer1::DataType::kFLOAT); + cache.transposedLogits + = bufferMgr.gpu(tr::ITensor::makeShape({beamWidth, numSteps, vocabSize}), nvinfer1::DataType::kFLOAT); + cache.fragmentPointerDevice + = bufferMgr.gpu(tr::ITensor::makeShape({maxBatchSize, numSteps}), nvinfer1::DataType::kINT64); + cache.fragmentPointerHost + = tr::BufferManager::pinnedPool(tr::ITensor::makeShape({maxBatchSize, numSteps}), nvinfer1::DataType::kINT64); + + // Helper: build one LlmRequest that has numSteps fragments pointing into + // cache.logits[0..numSteps-1][logitsIndex:logitsIndex+beamWidth]. + // Each fragment is filled with sentinel value (step*100 + beam + reqOffset). + auto makeRequest = [&](RequestIdType reqId, SizeType32 logitsIndex, float reqOffset) -> std::shared_ptr<LlmRequest> + { + auto tokens = std::make_shared<VecTokens>(promptLen, 0); + tr::SamplingConfig sc{beamWidth}; + auto req = std::make_shared<LlmRequest>(reqId, numSteps, tokens, sc, false); + + LlmRequest::BeamTokens gen(beamWidth, VecTokens(numSteps, 1)); + req->setGeneratedTokens(gen); + req->allocGenerationLogitsHost(vocabSize, nvinfer1::DataType::kFLOAT); + + // Write known values into the logits cache slots for this request and + // create matching fragment slice views. + for (SizeType32 step = 0; step < numSteps; ++step) + { + // cache.logits shape: [numSteps, maxBatchSize*beamWidth, vocabSize] + // Slice to [1, maxBS*bw, vocab], squeeze to [maxBS*bw, vocab]. + tr::ITensor::SharedPtr slot = tr::ITensor::slice(cache.logits, step, 1); + slot->squeeze(0); // [maxBS*bw, vocab] + auto* slotPtr = tr::bufferCast<float>(*slot); + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + float const val = reqOffset + static_cast<float>(step * 100 + beam); + for (SizeType32 v = 0; v < vocabSize; ++v) + { + slotPtr[(logitsIndex + beam) * vocabSize + v] = val; + } + } + + // Fragment matches HandleGenerationLogits: slice [logitsIndex:logitsIndex+beamWidth] + // from the step slot, then unsqueeze(0) → [1, beamWidth, vocab]. + tr::ITensor::SharedPtr fragView = tr::ITensor::slice(slot, logitsIndex, beamWidth); + fragView->unsqueeze(0); // [1, beamWidth, vocab] + req->addGenerationLogitsFragment(fragView); + } + return req; + }; + + // Request 0 occupies logitsIndex=0 in the batch slot. + auto req0 = makeRequest(1, /*logitsIndex=*/0, /*reqOffset=*/0.0f); + // Request 1 occupies logitsIndex=beamWidth in the batch slot. + auto req1 = makeRequest(2, /*logitsIndex=*/beamWidth, /*reqOffset=*/1000.0f); + + // Flush request 0 — uses workIdx=0. + utils::copyGenerationLogits(cache, bufferMgr, *req0, /*beforeDecoder=*/false, {}); + // Flush request 1 — uses workIdx=1 (different slot → no pointer clobbering). + utils::copyGenerationLogits(cache, bufferMgr, *req1, /*beforeDecoder=*/false, {}); + + ASSERT_EQ(cudaStreamSynchronize(stream->get()), cudaSuccess); + + // Verify req0 host buffer: host[beam, step, v] == step*100 + beam + auto const* host0 = tr::bufferCast<float>(*req0->getGenerationLogitsHost()); + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + for (SizeType32 step = 0; step < numSteps; ++step) + { + float const expected = static_cast<float>(step * 100 + beam); + for (SizeType32 v = 0; v < vocabSize; ++v) + { + SizeType32 const idx = (beam * numSteps + step) * vocabSize + v; + EXPECT_FLOAT_EQ(host0[idx], expected) << "req0 host[beam=" << beam << ",step=" << step << ",v=" << v + << "]=" << host0[idx] << " expected " << expected; + } + } + } + + // Verify req1 host buffer: host[beam, step, v] == 1000 + step*100 + beam + auto const* host1 = tr::bufferCast<float>(*req1->getGenerationLogitsHost()); + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + for (SizeType32 step = 0; step < numSteps; ++step) + { + float const expected = 1000.0f + static_cast<float>(step * 100 + beam); + for (SizeType32 v = 0; v < vocabSize; ++v) + { + SizeType32 const idx = (beam * numSteps + step) * vocabSize + v; + EXPECT_FLOAT_EQ(host1[idx], expected) << "req1 host[beam=" << beam << ",step=" << step << ",v=" << v + << "]=" << host1[idx] << " expected " << expected; + } + } + } + + // Both requests must have had their fragments cleared. + EXPECT_EQ(req0->getGenerationLogitsFragmentsSize(), 0); + EXPECT_EQ(req1->getGenerationLogitsFragmentsSize(), 0); +} diff --git a/cpp/tests/unit_tests/batch_manager/kvCacheManagerFabricMemoryTest.cpp b/cpp/tests/unit_tests/batch_manager/kvCacheManagerFabricMemoryTest.cpp index 4d41d751476c..d23cbabe4144 100644 --- a/cpp/tests/unit_tests/batch_manager/kvCacheManagerFabricMemoryTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/kvCacheManagerFabricMemoryTest.cpp @@ -20,7 +20,6 @@ #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/batch_manager/llmRequest.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/samplingConfig.h" #include "tensorrt_llm/testing/kvCacheManagerTestUtil.h" @@ -109,7 +108,7 @@ TEST_F(KVCacheManagerFabricMemoryTest, AllocatePoolsFallbackWhenFabricUnsupporte BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, 0); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, 0); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -148,7 +147,7 @@ TEST_F(KVCacheManagerFabricMemoryTest, AllocatePoolsWithFabricMemory) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, 0); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, 0); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getMaxNumBlocks(), blocksInPrimaryPool); @@ -192,7 +191,7 @@ TEST_F(KVCacheManagerFabricMemoryTest, OffloadOnboardRoundTripWithFabricPrimary) BlockManager blockManager(std::vector<BlockManager::SizeType32>(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, 0); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, 0); blockManager.allocatePools(false); auto primaryPoolPtr = blockManager.getPrimaryPool(0); @@ -293,7 +292,7 @@ TEST_F(KVCacheManagerFabricMemoryTest, ReleasePoolsClearsFabricMemory) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, 0); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, 0); size_t freeBefore = 0; size_t freeAfterAlloc = 0; diff --git a/cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp b/cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp index 7bacaf9a2575..b1c91ae09d4c 100644 --- a/cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/kvCacheManagerTest.cpp @@ -26,7 +26,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/transferAgent.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/kernels/kvCacheIndex.h" @@ -175,8 +174,7 @@ TEST_F(KVCacheManagerTest, BlockManagerTest) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, - maxAttentionWindow); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, maxAttentionWindow); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -221,11 +219,6 @@ TEST_F(KVCacheManagerTest, BlockManagerTest) EXPECT_EQ(idSet.size(), occupiedBlocks); blockManager.releaseBlocks(seq0); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool); - for (auto const blockId : idSet) - { - auto const block = blockManager.getBlockById(blockId, maxAttentionWindow); - EXPECT_EQ(block->getPrevBlockInSeq(), nullptr); - } // Test: last block shared (inputLength aligned to tokensPerBlock) auto inputTokensAligned = makeInputTokens(numTokens); @@ -312,7 +305,7 @@ void writePatternToOffloadedBlocksGDS( } } -template <typename T, tensorrt_llm::DataType type, int mask, KvCacheTransferMode transferMode> +template <typename T, nvinfer1::DataType type, int mask, KvCacheTransferMode transferMode> void runPartialCopyTest() { auto constexpr numLayers = 12; @@ -331,12 +324,12 @@ void runPartialCopyTest() auto constexpr maxAttentionWindowAllLayer = 4096; auto constexpr sinkTokenLen = 0; auto constexpr canUseOneMoreBlock = true; - std::string directory; + std::string directory = ""; + static int file_num = 0; if constexpr (transferMode == KvCacheTransferMode::GDS) { - auto const filename - = std::string("test_copy_") + std::to_string(::getpid()) + "_" + std::to_string(static_cast<int>(type)); + std::string filename = std::string("test_copy") + std::to_string(file_num++); auto dirPath = fs::absolute(filename); fs::create_directories(dirPath); directory = dirPath.string(); @@ -444,7 +437,7 @@ void runPartialCopyTest() llmRequest1->setPrepopulatedPromptLen(prepopulatedPromptLen1, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest1->getContextCurrentPosition(), 16); auto cacheBlockIds1 = seq1.getCacheBlockIds(maxAttentionWindow).at(beamIdx); - EXPECT_THAT(cacheBlockIds1, ::testing::ElementsAreArray({0, 1, 2})); + EXPECT_THAT(cacheBlockIds1, ::testing::ElementsAreArray({0, 1, 6})); // store blocks 0, 1 ([0,1,2,3,4,5,6,7], [8,9,10,11,12,13,14,15]) tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest1); blockManager.storeContextBlocks(seq1, *llmRequest1); @@ -452,12 +445,11 @@ void runPartialCopyTest() // Add sequence [0,1,2,3,4,5,6,7,8,9,10,11] again. // Reuse blocks 0 and 1(pc). Block 1 is partially reused, but already referenced by seq1 so must be partial copied - // into new block 6. Clear block 6 so we can see what was partial copied. - auto constexpr partialCopyBlockId = 6; - auto partialCopyBlock = blockManager.getBlockById(partialCopyBlockId, maxAttentionWindow); - auto memoryPoolIndex = partialCopyBlock->getMemoryPoolBlockIndex(); - auto partialCopyBlockPtr{tr::ITensor::slice(primaryPoolPtr, memoryPoolIndex, 1)}; - EXPECT_EQ(cudaMemset(partialCopyBlockPtr->data(), 0, blockSize * sizeof(T)), cudaSuccess); + // into new block 2. Clear block 2 so we can see what was partial copied. + auto block2 = blockManager.getBlockById(2, maxAttentionWindow); + auto memoryPoolIndex2 = block2->getMemoryPoolBlockIndex(); + auto block2Ptr{tr::ITensor::slice(primaryPoolPtr, memoryPoolIndex2, 1)}; + EXPECT_EQ(cudaMemset(block2Ptr->data(), 0, blockSize * sizeof(T)), cudaSuccess); auto inputTokens2 = inputTokens; auto constexpr partiallyReusedTokens = 3; inputTokens2->resize(8 + partiallyReusedTokens + 1); @@ -475,22 +467,22 @@ void runPartialCopyTest() llmRequest2->setPrepopulatedPromptLen(prepopulatedPromptLen2, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest2->getContextCurrentPosition(), 11); auto cacheBlockIds2 = seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx); - EXPECT_THAT(cacheBlockIds2, ::testing::ElementsAreArray({0, partialCopyBlockId})); + EXPECT_THAT(cacheBlockIds2, ::testing::ElementsAreArray({0, 2})); EXPECT_EQ(cudaDeviceSynchronize(), cudaSuccess); - // Verify the partially copied block. + // Verify partial copied block 2 // Block has shape [2, numLayers, numKvHeads, tokensPerBlock, sizePerHead] - blockManager.offloadBlock(partialCopyBlock, maxAttentionWindow); - EXPECT_FALSE(partialCopyBlock->isPrimary()); + blockManager.offloadBlock(block2, maxAttentionWindow); + EXPECT_FALSE(block2->isPrimary()); // need to sync so D2H transfer is done before accessing blocks EXPECT_EQ(cudaDeviceSynchronize(), cudaSuccess); - memoryPoolIndex = partialCopyBlock->getMemoryPoolBlockIndex(); - partialCopyBlockPtr = tr::ITensor::slice(secondaryPoolPtr, memoryPoolIndex, 1); - T const* rawPtr = reinterpret_cast<T*>(partialCopyBlockPtr->data()); + memoryPoolIndex2 = block2->getMemoryPoolBlockIndex(); + block2Ptr = tr::ITensor::slice(secondaryPoolPtr, memoryPoolIndex2, 1); + T const* rawPtr2 = reinterpret_cast<T*>(block2Ptr->data()); int numBad = 0; for (int i = 0; i < blockSize && numBad < 10; ++i) { - T value = rawPtr[i]; + T value = rawPtr2[i]; int kOrV = i / (numLayers * numKvHeads * tokensPerBlock * sizePerHead); int j = i - kOrV * (numLayers * numKvHeads * tokensPerBlock * sizePerHead); int layer = j / (numKvHeads * tokensPerBlock * sizePerHead); @@ -502,14 +494,14 @@ void runPartialCopyTest() T expectedValue = (token < partiallyReusedTokens) ? i & mask : 0; if (value != expectedValue) { - TLLM_LOG_WARNING("partialCopyBlock[%d,%d,%d,%d,%d] - expected %d, actual %d", kOrV, layer, head, token, j, - expectedValue, value); + TLLM_LOG_WARNING( + "block2[%d,%d,%d,%d,%d] - expected %d, actual %d", kOrV, layer, head, token, j, expectedValue, value); ++numBad; } } EXPECT_EQ(numBad, 0); - blockManager.onboardBlock(seq2, partialCopyBlock, maxAttentionWindow, KvCacheTransferMode::DRAM); - EXPECT_TRUE(partialCopyBlock->isPrimary()); + blockManager.onboardBlock(seq2, block2, maxAttentionWindow, transferMode, directory); + EXPECT_TRUE(block2->isPrimary()); EXPECT_EQ(cudaDeviceSynchronize(), cudaSuccess); tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest1); @@ -523,59 +515,59 @@ void runPartialCopyTest() TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyINT64) { - runPartialCopyTest<std::uint64_t, tensorrt_llm::DataType::kINT64, -1, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint64_t, tensorrt_llm::DataType::kINT64, -1, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint64_t, nvinfer1::DataType::kINT64, -1, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint64_t, nvinfer1::DataType::kINT64, -1, KvCacheTransferMode::GDS>(); } TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyINT32) { - runPartialCopyTest<std::uint32_t, tensorrt_llm::DataType::kINT32, -1, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint32_t, tensorrt_llm::DataType::kINT32, -1, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint32_t, nvinfer1::DataType::kINT32, -1, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint32_t, nvinfer1::DataType::kINT32, -1, KvCacheTransferMode::GDS>(); } TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyFLOAT) { - runPartialCopyTest<std::uint32_t, tensorrt_llm::DataType::kFLOAT, -1, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint32_t, tensorrt_llm::DataType::kFLOAT, -1, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint32_t, nvinfer1::DataType::kFLOAT, -1, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint32_t, nvinfer1::DataType::kFLOAT, -1, KvCacheTransferMode::GDS>(); } #ifdef ENABLE_BF16 TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyBF16) { - runPartialCopyTest<std::uint16_t, tensorrt_llm::DataType::kBF16, 65535, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint16_t, tensorrt_llm::DataType::kBF16, 65535, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint16_t, nvinfer1::DataType::kBF16, 65535, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint16_t, nvinfer1::DataType::kBF16, 65535, KvCacheTransferMode::GDS>(); } #endif TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyHALF) { - runPartialCopyTest<std::uint16_t, tensorrt_llm::DataType::kHALF, 65535, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint16_t, tensorrt_llm::DataType::kHALF, 65535, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint16_t, nvinfer1::DataType::kHALF, 65535, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint16_t, nvinfer1::DataType::kHALF, 65535, KvCacheTransferMode::GDS>(); } TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyBOOL) { - runPartialCopyTest<std::uint8_t, tensorrt_llm::DataType::kBOOL, 255, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint8_t, tensorrt_llm::DataType::kBOOL, 255, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint8_t, nvinfer1::DataType::kBOOL, 255, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint8_t, nvinfer1::DataType::kBOOL, 255, KvCacheTransferMode::GDS>(); } TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyUINT8) { - runPartialCopyTest<std::uint8_t, tensorrt_llm::DataType::kUINT8, 255, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint8_t, tensorrt_llm::DataType::kUINT8, 255, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint8_t, nvinfer1::DataType::kUINT8, 255, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint8_t, nvinfer1::DataType::kUINT8, 255, KvCacheTransferMode::GDS>(); } TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyINT8) { - runPartialCopyTest<std::uint8_t, tensorrt_llm::DataType::kINT8, 255, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint8_t, tensorrt_llm::DataType::kINT8, 255, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint8_t, nvinfer1::DataType::kINT8, 255, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint8_t, nvinfer1::DataType::kINT8, 255, KvCacheTransferMode::GDS>(); } #ifdef ENABLE_FP8 TEST_F(KVCacheManagerTest, BlockManagerTestPartialCopyFP8) { - runPartialCopyTest<std::uint8_t, tensorrt_llm::DataType::kFP8, 255, KvCacheTransferMode::DRAM>(); - runPartialCopyTest<std::uint8_t, tensorrt_llm::DataType::kFP8, 255, KvCacheTransferMode::GDS>(); + runPartialCopyTest<std::uint8_t, nvinfer1::DataType::kFP8, 255, KvCacheTransferMode::DRAM>(); + runPartialCopyTest<std::uint8_t, nvinfer1::DataType::kFP8, 255, KvCacheTransferMode::GDS>(); } #endif @@ -733,8 +725,8 @@ TEST_F(KVCacheManagerTest, FindBlocksInReuseTreeByBlockKeysTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numKvHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, false, - stream, maxAttentionWindow, maxAttentionWindow, true); + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, false, stream, + maxAttentionWindow, maxAttentionWindow, true); // Add sequence [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] (17 tokens, three blocks) auto inputTokens = std::make_shared<VecTokens>(VecTokens{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}); @@ -785,8 +777,8 @@ TEST_F(KVCacheManagerTest, FP4BlockScaleManagementTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kFP4, false, - stream, maxAttentionWindow, maxAttentionWindow, true); + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kFP4, false, stream, + maxAttentionWindow, maxAttentionWindow, true); kvCacheManager.allocatePools(/*useUvm=*/false); @@ -801,90 +793,6 @@ TEST_F(KVCacheManagerTest, FP4BlockScaleManagementTest) // The expected block size of pool 1 should be the number of FP4 elements / vectorSize. EXPECT_EQ(blockManager.getBlockSize(0) * numFp4EltsPerContainer / vectorSize, blockManager.getBlockSize(1)); } - -TEST_F(KVCacheManagerTest, FP4AttentionWithHalfRecurrentStatesPoolTest) -{ - auto constexpr numKvHeads = 2; - auto constexpr sizePerHead = 16; - auto constexpr tokensPerBlock = 4; - auto constexpr blocksInPrimaryPool = 4; - auto constexpr blocksInSecondaryPool = 0; - auto constexpr maxNumSequences = 2; - auto constexpr maxBeamWidth = 1; - auto constexpr maxAttentionWindow = 16; - auto constexpr recurrentStatesBytes = 64; - SizeType32 constexpr recurrentStatesWindow = LinearAttentionMetadata::LinearCacheType::kRecurrentStates; - - LinearAttentionMetadata const linearAttentionMetadata{ - .linearLayerIndices = {0}, - .cacheType = recurrentStatesWindow, - .allRecurrentStatesBytes = recurrentStatesBytes, - }; - auto const blocksPerWindow = BlocksPerWindow{ - {recurrentStatesWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}, - {maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}, - }; - auto const poolConfigurations = std::vector<PoolConfiguration>{ - {recurrentStatesWindow, sizePerHead, tensorrt_llm::DataType::kHALF}, - {maxAttentionWindow, sizePerHead, tensorrt_llm::DataType::kFP4}, - }; - auto const stream = std::make_shared<tr::CudaStream>(); - - KVCacheManager kvCacheManager(std::vector<SizeType32>{0, numKvHeads}, sizePerHead, tokensPerBlock, blocksPerWindow, - maxNumSequences, maxBeamWidth, std::vector<SizeType32>{recurrentStatesWindow, maxAttentionWindow}, - tensorrt_llm::DataType::kFP4, - /*sinkTokenLength=*/0, stream, maxAttentionWindow, /*chunkSize=*/0, /*enableBlockReuse=*/false, - CacheType::kSELF, std::nullopt, nullptr, /*enablePartialReuse=*/false, /*copyOnPartialReuse=*/true, nullptr, - /*enableIndexerKCache=*/false, /*indexerKCacheQuantBlockSize=*/128, /*indexerKCacheIndexHeadDim=*/0, - /*indexerKCacheUseFp4=*/false, linearAttentionMetadata, poolConfigurations); - kvCacheManager.allocatePools(/*useUvm=*/false); - auto const& blockManager = kvCacheManager.getBlockManager(); - - EXPECT_EQ(blockManager.getDataTypeForWindow(recurrentStatesWindow), tensorrt_llm::DataType::kHALF); - EXPECT_EQ(blockManager.getDataTypeForWindow(maxAttentionWindow), tensorrt_llm::DataType::kFP4); - - auto const& recurrentStatesPool = blockManager.getRecurrentStatesPool(); - ASSERT_NE(recurrentStatesPool.primaryPtr, nullptr); - EXPECT_EQ(recurrentStatesPool.primaryPtr->getDataType(), tensorrt_llm::DataType::kHALF); - auto const recurrentStatesElementsPerBlock = recurrentStatesBytes / tc::getDTypeSize(tensorrt_llm::DataType::kHALF); - EXPECT_EQ(recurrentStatesPool.blockSize, recurrentStatesElementsPerBlock); - - SizeType32 numRecurrentScalePools = 0; - SizeType32 numAttentionScalePools = 0; - for (SizeType32 poolIdx = 0; poolIdx < blockManager.getNumPools(); ++poolIdx) - { - if (!blockManager.containsBlockScales(poolIdx)) - { - continue; - } - if (blockManager.getPoolWindowSize(poolIdx) == recurrentStatesWindow) - { - ++numRecurrentScalePools; - } - else if (blockManager.getPoolWindowSize(poolIdx) == maxAttentionWindow) - { - ++numAttentionScalePools; - } - } - EXPECT_EQ(numRecurrentScalePools, 0); - EXPECT_EQ(numAttentionScalePools, 1); - - auto const blockPoolPointers = kvCacheManager.getBlockPoolPointers(); - auto const blockScalePoolPointers = kvCacheManager.getBlockScalePoolPointers(); - ASSERT_NE(blockPoolPointers, nullptr); - ASSERT_NE(blockScalePoolPointers, nullptr); - EXPECT_EQ(blockPoolPointers->getShape().d[0], 2); - EXPECT_EQ(blockScalePoolPointers->getShape().d[0], 1); - auto const blockScalePoolPointersRange = tr::BufferRange<void*>(*blockScalePoolPointers); - EXPECT_NE(blockScalePoolPointersRange[0], nullptr); - EXPECT_EQ(blockScalePoolPointersRange[1], nullptr); - - auto const layerToPoolMapping = kvCacheManager.getLayerToPoolMapping(); - ASSERT_NE(layerToPoolMapping, nullptr); - auto const layerToPoolMappingRange = tr::BufferRange<SizeType32>(*layerToPoolMapping); - EXPECT_EQ(layerToPoolMappingRange[0], 0); - EXPECT_EQ(layerToPoolMappingRange[2], 1); -} #endif TEST_F(KVCacheManagerTest, BlockManagerReuseTest) @@ -907,8 +815,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, - maxAttentionWindow); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, maxAttentionWindow); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -1004,7 +911,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) // note that seq0_dup is holding blocks 0, 1 and 2 until releaseBlocks is called // input tokens [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] - // reuse blocks 0, 1 ([0, 1, 2, 3], [4, 5, 6, 7]) and get recycled block 3 + // reuse blocks 0, 1 ([0, 1, 2, 3], [4, 5, 6, 7]) and get new block 4 auto inputTokens1 = std::make_shared<VecTokens>(llmRequest1->getTokens(0)); GenerationRequest seq1_dup{11, inputLength, beamWidth, blockManager.getWindowSizesMetadata()}; llmRequest1 = std::make_shared<LlmRequest>( @@ -1017,8 +924,8 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) .prepopulatedLen; llmRequest1->setPrepopulatedPromptLen(prepopulatedPromptLen1, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest1->getContextCurrentPosition(), 2 * tokensPerBlock); - EXPECT_THAT(seq1_dup.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 3})); - llmRequest1->addNewToken(10, beamIdx); // block 3 contains [8, 9, 10] + EXPECT_THAT(seq1_dup.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 4})); + llmRequest1->addNewToken(10, beamIdx); // block 4 contains [8, 9, 10] EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks + 1); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks - 1); @@ -1027,7 +934,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) blockManager.releaseBlocks(seq0_dup, llmRequest0); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks); - // block 3 is stored for reuse (block contains [8, 9]). nb! Last token of last block is never stored + // block 4 is stored for reuse (block contains [8, 9]). nb! Last token of last block is never stored tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest1); blockManager.releaseBlocks(seq1_dup, llmRequest1); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), 0); @@ -1044,7 +951,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) numTokens = llmRequest2->getNumTokens(beamIdx); GenerationRequest seq2{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // reuse block 0 ([0, 1, 2, 3]), get new block 4 + // reuse block 0 ([0, 1, 2, 3]), get new block 5 auto promptLen2 = llmRequest2->getNumTokens(beamIdx); auto numContextBlocks2 = tc::ceilDiv(promptLen2, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen2 = blockManager @@ -1053,8 +960,8 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) .prepopulatedLen; llmRequest2->setPrepopulatedPromptLen(prepopulatedPromptLen2, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest2->getContextCurrentPosition(), tokensPerBlock); - EXPECT_THAT(seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 4})); - llmRequest2->addNewToken(5, beamIdx); // block 4 contains [4] + EXPECT_THAT(seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 5})); + llmRequest2->addNewToken(5, beamIdx); // block 5 contains [4] numTokens = llmRequest2->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); @@ -1069,7 +976,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) numTokens = llmRequest3->getNumTokens(beamIdx); GenerationRequest seq3{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // reuse blocks 0, 1, 3(p) ([0, 1, 2, 3], [4, 5, 6, 7], [8, 9]) + // reuse blocks 0, 1, 4(p) ([0, 1, 2, 3], [4, 5, 6, 7], [8, 9]) auto promptLen3 = llmRequest3->getNumTokens(beamIdx); auto numContextBlocks3 = tc::ceilDiv(promptLen3, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen3 = blockManager @@ -1078,18 +985,18 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) .prepopulatedLen; llmRequest3->setPrepopulatedPromptLen(prepopulatedPromptLen3, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest3->getContextCurrentPosition(), numTokens - 1); - EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 3})); - llmRequest3->addNewToken(11, beamIdx); // block 3 contains [8, 9, 11] + EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 4})); + llmRequest3->addNewToken(11, beamIdx); // block 4 contains [8, 9, 11] numTokens = llmRequest3->getNumTokens(beamIdx); // one block used by both seq2 and seq3 numBlocks += tc::ceilDiv(numTokens, tokensPerBlock) - 1; EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks); - // block 4 is not stored since it is last block and has only one token + // block 5 is not stored since it is last block and has only one token tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest2); blockManager.releaseBlocks(seq2, llmRequest2); - // block 3 is stored for reuse (block contains [8, 9]). nb! Last token of last block not stored + // block 4 is stored for reuse (block contains [8, 9]). nb! Last token of last block not stored tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest3); blockManager.releaseBlocks(seq3, llmRequest3); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), 0); @@ -1105,7 +1012,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) numTokens = llmRequest4->getNumTokens(beamIdx); GenerationRequest seq4{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // reuse blocks 0, 1, 3(p) ([0, 1, 2, 3], [4, 5, 6, 7], [8,9]) + // reuse blocks 0, 1, 4(p) ([0, 1, 2, 3], [4, 5, 6, 7], [8,9]) auto promptLen4 = llmRequest4->getNumTokens(beamIdx); auto numContextBlocks4 = tc::ceilDiv(promptLen4, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen4 = blockManager @@ -1114,7 +1021,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) .prepopulatedLen; llmRequest4->setPrepopulatedPromptLen(prepopulatedPromptLen4, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest4->getContextCurrentPosition(), promptLen4 - 1); - EXPECT_THAT(seq4.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 3})); + EXPECT_THAT(seq4.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 4})); numTokens = llmRequest4->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); @@ -1125,7 +1032,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseTest) // llmRequest4Short tokens [0, 1, 2, 3, 4, 5, 6, 7, 8] // blocks 0 and 1 ([0, 1, 2, 3], [4, 5, 6, 7]) are already stored, - // block 3 is freed + // block 4 is freed tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest4Short); blockManager.releaseBlocks(seq4, llmRequest4Short); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), 0); @@ -1243,8 +1150,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithExtraIdTest) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, - maxAttentionWindow); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, maxAttentionWindow); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -1331,7 +1237,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithExtraIdTest) /////////////////////////////////////////////////////////////////////////// // add both requests again and then remove them - // reuse blocks 0, 1 and get recycled block 3 + // reuse blocks 0, 1 and get new block 4 GenerationRequest seq0_dup{10, inputLength, beamWidth, blockManager.getWindowSizesMetadata()}; llmRequest0 = std::make_shared<LlmRequest>(seq0_dup.getRequestId(), maxNewTokens, inputTokens, samplingConfig, isStreaming, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, @@ -1349,7 +1255,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithExtraIdTest) llmRequest0->setPrepopulatedPromptLen(prepopulatedPromptLen0, blockManager.getTokensPerBlock()); llmRequest0->addNewToken(3, beamIdx); EXPECT_EQ(llmRequest0->getContextCurrentPosition(), 2 * tokensPerBlock); - EXPECT_THAT(seq0_dup.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 3})); + EXPECT_THAT(seq0_dup.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 4})); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks); @@ -1402,7 +1308,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithExtraIdTest) numTokens = llmRequest2->getNumTokens(beamIdx); GenerationRequest seq2{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // no reuse, get new blocks 4, 5, 6 + // no reuse, get new block 5, 6, 7 auto promptLen2 = llmRequest2->getNumTokens(beamIdx); auto numContextBlocks2 = tc::ceilDiv(promptLen2, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen2 = blockManager @@ -1412,7 +1318,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithExtraIdTest) .prepopulatedLen; llmRequest2->setPrepopulatedPromptLen(prepopulatedPromptLen2, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest2->getContextCurrentPosition(), 0); - EXPECT_THAT(seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({4, 5, 6})); + EXPECT_THAT(seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({5, 6, 7})); llmRequest2->addNewToken(3, beamIdx); numTokens = llmRequest2->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); @@ -1432,7 +1338,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithExtraIdTest) numTokens = llmRequest3->getNumTokens(beamIdx); GenerationRequest seq3{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // reuse block 0, get new blocks 7 and 8 + // reuse block 0, get new block 8, 9 auto promptLen3 = llmRequest3->getNumTokens(beamIdx); auto numContextBlocks3 = tc::ceilDiv(promptLen3, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen3 = blockManager @@ -1442,7 +1348,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithExtraIdTest) .prepopulatedLen; llmRequest3->setPrepopulatedPromptLen(prepopulatedPromptLen3, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest3->getContextCurrentPosition(), tokensPerBlock); - EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 7, 8})); + EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 8, 9})); llmRequest3->addNewToken(3, beamIdx); numTokens = llmRequest3->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); @@ -1479,8 +1385,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithMultimodalHashTest) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, - maxAttentionWindow); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, maxAttentionWindow); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -1596,7 +1501,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithMultimodalHashTest) numReturnSequences); GenerationRequest seq2{requestId, inputLength, beamWidth, blockManager.getWindowSizesMetadata()}; - // no reuse, get new blocks 3, 4, 5 + // no reuse, get new blocks 4, 5, 6 auto promptLen2 = llmRequest2->getNumTokens(beamIdx); auto numContextBlocks2 = tc::ceilDiv(promptLen2, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen2 = blockManager @@ -1606,7 +1511,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithMultimodalHashTest) .prepopulatedLen; llmRequest2->setPrepopulatedPromptLen(prepopulatedPromptLen2, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest2->getContextCurrentPosition(), 0); - EXPECT_THAT(seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({3, 4, 5})); + EXPECT_THAT(seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({4, 5, 6})); llmRequest2->addNewToken(9, beamIdx); numTokens = llmRequest2->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); @@ -1634,7 +1539,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithMultimodalHashTest) std::nullopt, std::nullopt, LlmRequestType::LLMREQUEST_TYPE_CONTEXT_AND_GENERATION, std::nullopt, numReturnSequences); GenerationRequest seq3{requestId, inputLength, beamWidth, blockManager.getWindowSizesMetadata()}; - // reuse block 0, get new blocks 6 and 7 + // reuse block 0, get new blocks 7, 8 auto promptLen3 = llmRequest3->getNumTokens(beamIdx); auto numContextBlocks3 = tc::ceilDiv(promptLen3, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen3 = blockManager @@ -1645,7 +1550,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithMultimodalHashTest) llmRequest3->setPrepopulatedPromptLen(prepopulatedPromptLen3, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest3->getContextCurrentPosition(), tokensPerBlock); // only reuse block 0 [100, 101, 102, 103] with same hash/offset - EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 6, 7})); + EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 7, 8})); llmRequest3->addNewToken(11, beamIdx); numTokens = llmRequest3->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); @@ -1684,8 +1589,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, - maxAttentionWindow); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, maxAttentionWindow); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -1768,7 +1672,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) /////////////////////////////////////////////////////////////////////////// // add both requests again and then remove them // inputTokens = (0, 1, 2, 3, 4, 5, 6, 7, 8) - // reuse blocks 0, 1 and get recycled block 3 + // reuse blocks 0, 1 and get new block 4 GenerationRequest seq0_dup{10, inputLength, beamWidth, blockManager.getWindowSizesMetadata()}; llmRequest0 = std::make_shared<LlmRequest>(seq0_dup.getRequestId(), maxNewTokens, inputTokens, samplingConfig, isStreaming, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, std::nullopt, @@ -1789,7 +1693,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) // but promptLen is number of input tokens. llmRequest0->addNewToken(9, beamIdx); EXPECT_EQ(llmRequest0->getContextCurrentPosition(), 2 * tokensPerBlock); - EXPECT_THAT(seq0_dup.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 3})); + EXPECT_THAT(seq0_dup.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 4})); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks); @@ -1815,7 +1719,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks + 1); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks - 1); - // store block 3 for reuse ([8]) + // store block 4 for reuse ([8]) tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest0); blockManager.releaseBlocks(seq0_dup, llmRequest0); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); @@ -1836,7 +1740,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) numTokens = llmRequest2->getNumTokens(beamIdx); GenerationRequest seq2{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // no reuse, get new blocks 4, 5, 6 + // no reuse, get new block 5, 6, 7 auto promptLen2 = llmRequest2->getNumTokens(beamIdx); auto numContextBlocks2 = tc::ceilDiv(promptLen2, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen2 = blockManager @@ -1847,13 +1751,13 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) llmRequest2->setPrepopulatedPromptLen(prepopulatedPromptLen2, blockManager.getTokensPerBlock()); // no reuse expected. Input tokens match blocks 0 and 1, but lora task id differs. EXPECT_EQ(llmRequest2->getContextCurrentPosition(), 0); - EXPECT_THAT(seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({4, 5, 6})); + EXPECT_THAT(seq2.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({5, 6, 7})); llmRequest2->addNewToken(9, beamIdx); numTokens = llmRequest2->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks); - // store blocks 4, 5, 6 for reuse ([0,1,2,3], [4,5,6,7], [8]) with loraTaskId 1 + // store blocks 5, 6, 7 for reuse ([0,1,2,3], [4,5,6,7], [8]) with loraTaskId 1 tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest2); blockManager.releaseBlocks(seq2, llmRequest2); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), 0); @@ -1869,7 +1773,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) numTokens = llmRequest3->getNumTokens(beamIdx); GenerationRequest seq3{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // reuse blocks 4, 5, 6(p) ([0,1,2,3], [4,5,6,7], [8]) + // reuse blocks 5, 6, 7(p) ([0,1,2,3], [4,5,6,7], [8]) auto promptLen3 = llmRequest3->getNumTokens(beamIdx); auto numContextBlocks3 = tc::ceilDiv(promptLen3, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen3 = blockManager @@ -1879,13 +1783,13 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) .prepopulatedLen; llmRequest3->setPrepopulatedPromptLen(prepopulatedPromptLen3, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest3->getContextCurrentPosition(), promptLen3 - 2); - EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({4, 5, 6})); + EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({5, 6, 7})); llmRequest3->addNewToken(11, beamIdx); numTokens = llmRequest3->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks); - // store block 6 for reuse ([8,9]) with loraTaskId 1 + // store block 7 for reuse ([8,9]) with loraTaskId 1 tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest3); blockManager.releaseBlocks(seq3, llmRequest3); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), 0); @@ -1903,7 +1807,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) numTokens = llmRequest4->getNumTokens(beamIdx); GenerationRequest seq4{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // reuse block 0, get new block 7 + // reuse blocks 0, get new block 8 auto promptLen4 = llmRequest4->getNumTokens(beamIdx); auto numContextBlocks4 = tc::ceilDiv(promptLen4, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen4 = blockManager @@ -1913,13 +1817,13 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) .prepopulatedLen; llmRequest4->setPrepopulatedPromptLen(prepopulatedPromptLen4, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest4->getContextCurrentPosition(), tokensPerBlock); - EXPECT_THAT(seq4.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 7})); + EXPECT_THAT(seq4.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 8})); llmRequest4->addNewToken(5, beamIdx); numTokens = llmRequest4->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks); - // block 7 is stored with [4] and loraTaskId 0 + // blocks 8 is stored with [4] and loraTaskId 0 tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest4); blockManager.releaseBlocks(seq4, llmRequest4); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), 0); @@ -1932,7 +1836,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) numTokens = llmRequest5->getNumTokens(beamIdx); GenerationRequest seq5{requestId, numTokens, beamWidth, blockManager.getWindowSizesMetadata()}; - // no reuse, get new blocks 8, 9, 10 + // no reuse, get new block 9, 10, 11 auto promptLen5 = llmRequest5->getNumTokens(beamIdx); auto numContextBlocks5 = tc::ceilDiv(promptLen5, blockManager.getTokensPerBlock()); auto prepopulatedPromptLen5 = blockManager @@ -1942,13 +1846,13 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithLoraTaskIdTest) .prepopulatedLen; llmRequest5->setPrepopulatedPromptLen(prepopulatedPromptLen5, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest5->getContextCurrentPosition(), 0); - EXPECT_THAT(seq5.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({8, 9, 10})); + EXPECT_THAT(seq5.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({9, 10, 11})); llmRequest5->addNewToken(9, beamIdx); numTokens = llmRequest5->getNumTokens(beamIdx); numBlocks = tc::ceilDiv(numTokens, tokensPerBlock); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), numBlocks); EXPECT_EQ(blockManager.getNumFreeBlocks(), blocksInPrimaryPool - numBlocks); - // blocks 8, 9, 10 are stored without loraTaskId + // blocks 9, 10, 11 are stored without loraTaskId tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest5); blockManager.releaseBlocks(seq5, llmRequest5); EXPECT_EQ(blockManager.getNumAllocatedBlocks(), 0); @@ -1978,8 +1882,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithExtraIdAndLoraTaskIdTest) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, - maxAttentionWindow); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, maxAttentionWindow); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -2248,8 +2151,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithCacheSaltTest) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, - maxAttentionWindow); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, maxAttentionWindow); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -2409,7 +2311,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithCacheSaltTest) .prepopulatedLen; llmRequest3->setPrepopulatedPromptLen(prepopulatedPromptLen3, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest3->getContextCurrentPosition(), 0); // No reuse - EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({6, 7, 8})); + EXPECT_THAT(seq3.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({7, 8, 9})); llmRequest3->addNewToken(5, beamIdx); llmRequest3->addNewToken(6, beamIdx); @@ -2440,7 +2342,7 @@ TEST_F(KVCacheManagerTest, BlockManagerReuseWithCacheSaltTest) .prepopulatedLen; llmRequest4->setPrepopulatedPromptLen(prepopulatedPromptLen4, blockManager.getTokensPerBlock()); EXPECT_EQ(llmRequest4->getContextCurrentPosition(), 2 * tokensPerBlock); // Reuse blocks 0,1 - EXPECT_THAT(seq4.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 9})); + EXPECT_THAT(seq4.getCacheBlockIds(maxAttentionWindow).at(beamIdx), ::testing::ElementsAreArray({0, 1, 10})); llmRequest4->addNewToken(7, beamIdx); numTokens = llmRequest4->getNumTokens(beamIdx); @@ -2481,7 +2383,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerPerRequestStatsTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength, true); kvCacheManager.allocatePools(false); @@ -2538,8 +2440,7 @@ TEST_F(KVCacheManagerTest, BlockManagerBlockPriorityTest) BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, - maxAttentionWindow); + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, maxAttentionWindow); blockManager.allocatePools(false); EXPECT_EQ(blockManager.getTokensPerBlock(), tokensPerBlock); @@ -2677,7 +2578,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerDecodeBlockPriorityTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength, true); kvCacheManager.allocatePools(false); @@ -2784,7 +2685,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerTimedEvictionTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength, true); kvCacheManager.allocatePools(false); @@ -2856,7 +2757,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerDecodeTimedEvictionTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength, true); kvCacheManager.allocatePools(false); { @@ -2949,7 +2850,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerSecondaryBlockPrimaryChildTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength, true); kvCacheManager.allocatePools(false); @@ -3046,7 +2947,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerStoreContextBlocksUsesMaterializedConte auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, /*chunkSize*/ 0, true); kvCacheManager.allocatePools(false); @@ -3090,7 +2991,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerReleaseBlocksUsesMaterializedContextExt auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, /*chunkSize*/ 0, true); kvCacheManager.allocatePools(false); @@ -3132,7 +3033,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerLeafBlockTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength, true); kvCacheManager.allocatePools(false); @@ -3157,8 +3058,8 @@ TEST_F(KVCacheManagerTest, KVCacheManagerLeafBlockTest) GenerationRequest const& seq1 = kvCacheManager.getSequence(1); EXPECT_EQ(llmRequest1->getContextCurrentPosition(), 0); - // Detached block 1 is recycled without reusing its KV contents. - EXPECT_THAT(seq1.getCacheBlockIds(maxAttentionWindow).at(0), ::testing::ElementsAreArray({1})); + // Block 1 should NOT be reused. It was not freed even if partial. + EXPECT_THAT(seq1.getCacheBlockIds(maxAttentionWindow).at(0), ::testing::ElementsAreArray({2})); // Allocate the remaining 3 blocks in primary auto inputTokens2 = std::make_shared<VecTokens>(VecTokens{2, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}); @@ -3217,7 +3118,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerLeafBlockWithDependentTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength, true); kvCacheManager.allocatePools(false); @@ -3321,7 +3222,7 @@ TEST_P(KVCacheManagerTest, DISABLED_KVCacheManagerAllocationTest) auto constexpr maxNumSequences = 8; auto constexpr maxBeamWidth = 4; auto constexpr sinkTokenLength = 0; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr maxSequenceLength = tokensPerBlock * maxBlocksPerSeq; @@ -3344,12 +3245,11 @@ TEST_P(KVCacheManagerTest, DISABLED_KVCacheManagerAllocationTest) KVCacheManager kvCacheManager = homogeneousLayers ? KVCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse) : KVCacheManager(std::vector<KVCacheManager::SizeType32>(numLayers, numHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, - tensorrt_llm::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, - enableBlockReuse); + nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse); auto const& blockManager = kvCacheManager.getBlockManager(); auto const& bufferManager = blockManager.getBufferManager(theOnlyWindowSize(kvCacheManager)); @@ -3420,10 +3320,10 @@ TEST_P(KVCacheManagerTest, KVCacheManagerTest) KVCacheManager kvCacheManager = homogeneousLayers ? KVCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse) : KVCacheManager(numHeadsPerLayer, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, maxBeamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, sinkTokenLength, + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse); kvCacheManager.allocatePools(false); @@ -3590,12 +3490,11 @@ TEST_P(KVCacheManagerTest, KVCacheManagerRewindTokensTest) KVCacheManager kvCacheManager = homogeneousLayers ? KVCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse) : KVCacheManager(std::vector<KVCacheManager::SizeType32>(numLayers, numHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, - tensorrt_llm::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, - enableBlockReuse); + nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse); kvCacheManager.allocatePools(false); EXPECT_EQ(kvCacheManager.getTokensPerBlock(), tokensPerBlock); @@ -3699,10 +3598,10 @@ TEST_P(KVCacheManagerTest, KVCacheManagerMaxAttentionWindowTest) KVCacheManager kvCacheManager = homogeneousLayers ? KVCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse) : KVCacheManager(numHeadsPerLayer, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, maxBeamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, sinkTokenLength, + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse); kvCacheManager.allocatePools(false); @@ -3823,7 +3722,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerMaxAttentionWindowSmallerThanBlockSizeT auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse); kvCacheManager.allocatePools(false); @@ -3917,7 +3816,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventStream) auto constexpr blocksInPrimaryPool = 8; auto constexpr blocksInSecondaryPool = 2; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr beamWidth = 1; @@ -4033,40 +3932,39 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventStream) events = getEvents(kvCacheManager); - // Replacing block 1 removes its detached child block 2 from the search tree. Block 2 is then recycled before a - // reusable block needs to be offloaded, producing one offload, one onboard, and one remove event. + // Onboard block 0, in replace, offload block 7 + // Offload block 6, and write content of [1,1,1,1] to block 1 + // Upon freeing up block 1, its child block 2, will be removed from the search tree, + // which is a remove event. + // Offload block 5, in replace onboard block 7, and write content of [0] to block 7. + // In total, there are 2 offloads, 1 onboard, 1 removed, total of 4 events. + // FIXME: For better improvement, when block 1 is overwritten, child blocks + // are removed from the search tree and no longer reusable. Therefore these blocks + // should be the first to be called upon when we want a new block. auto onboardedBlocks = 0; auto offloadedBlocks = 0; auto removedBlocks = 0; - ASSERT_EQ(events.size(), 3); + EXPECT_EQ(events.size(), 4); - for (int i = 0; i < 3; i++) + for (int i = 0; i < 4; i++) { if (std::holds_alternative<tle::KVCacheUpdatedData>(events.front().data)) { if (std::get<tle::KVCacheUpdatedData>(events.front().data).cacheLevel->oldValue == 0) - { offloadedBlocks++; - } else - { onboardedBlocks++; - } } else if (std::holds_alternative<tle::KVCacheRemovedData>(events.front().data)) - { removedBlocks++; - } else - { FAIL(); - } events.pop_front(); } EXPECT_EQ(onboardedBlocks, 1); - EXPECT_EQ(offloadedBlocks, 1); + EXPECT_EQ(offloadedBlocks, 2); EXPECT_EQ(removedBlocks, 1); tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest4); @@ -4101,7 +3999,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerMaxAttentionWindowWithReuseTest) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, /*chunkSize=*/tokensPerBlock, enableBlockReuse); kvCacheManager.allocatePools(false); @@ -4182,7 +4080,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerMaxAttentionWindowWithReuseTest) /////////////////////////////////////////////////////////////////////////// // add a medium request and then remove it - // reuse first 3 blocks {0, 1, 2(p)} in first request, copying block 2 to block 16 + // reuse first 3 blocks {0, 1, 2(p)} in first request, copying block 2 to a new block 8 requestId = 2; inputLength = 10; inputTokens->resize(inputLength); @@ -4191,14 +4089,14 @@ TEST_F(KVCacheManagerTest, KVCacheManagerMaxAttentionWindowWithReuseTest) kvCacheManager.addSequenceBatch({{{requestId, inputLength, beamWidth}}}, {std::ref(*llmRequest)}); GenerationRequest const& seq2 = kvCacheManager.getSequence(requestId); EXPECT_EQ(llmRequest->getContextCurrentPosition(), 9); - EXPECT_THAT(seq2.getCacheBlockIds(onlyWindowSize).at(beamIdx), ::testing::ElementsAreArray({0, 1, 16})); + EXPECT_THAT(seq2.getCacheBlockIds(onlyWindowSize).at(beamIdx), ::testing::ElementsAreArray({0, 1, 8})); tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest); EXPECT_NO_THROW(static_cast<void>(kvCacheManager.removeSequence(requestId, llmRequest))); /////////////////////////////////////////////////////////////////////////// // add a longer request within attention window and try to reuse - // reuse blocks {0, 1, 2, 3(p)}, copying block 3 to recycled block 7 - // then upon reaching attention window, get block 8 + // reuse blocks {0, 1, 2, 3(p)}, copying block 3 to a new block 9 + // then upon reaching attention window, get new block 10 requestId = 3; inputLength = 15; inputTokens->resize(inputLength); @@ -4207,14 +4105,14 @@ TEST_F(KVCacheManagerTest, KVCacheManagerMaxAttentionWindowWithReuseTest) kvCacheManager.addSequenceBatch({{{requestId, inputLength, beamWidth}}}, {std::ref(*llmRequest)}); GenerationRequest const& seq3 = kvCacheManager.getSequence(requestId); EXPECT_EQ(llmRequest->getContextCurrentPosition(), 14); - EXPECT_THAT(seq3.getCacheBlockIds(onlyWindowSize).at(beamIdx), ::testing::ElementsAreArray({0, 1, 2, 7})); + EXPECT_THAT(seq3.getCacheBlockIds(onlyWindowSize).at(beamIdx), ::testing::ElementsAreArray({0, 1, 2, 9})); // add new tokens to allocate another block, but not enough to detach block llmRequest->addNewToken(1015, beamIdx); kvCacheManager.addToken(requestId); llmRequest->addNewToken(1016, beamIdx); kvCacheManager.addToken(requestId); - EXPECT_THAT(seq3.getCacheBlockIds(onlyWindowSize).at(beamIdx), ::testing::ElementsAreArray({0, 1, 2, 7, 8})); + EXPECT_THAT(seq3.getCacheBlockIds(onlyWindowSize).at(beamIdx), ::testing::ElementsAreArray({0, 1, 2, 9, 10})); tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*llmRequest); EXPECT_NO_THROW(static_cast<void>(kvCacheManager.removeSequence(requestId, llmRequest))); } @@ -4228,7 +4126,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerSWAInvalidateReuseTest) auto constexpr maxNumSequences = 8; auto constexpr maxBeamWidth = 1; auto constexpr sinkTokenLength = 0; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr maxSequenceLength = 128; SizeType32 constexpr maxNewTokens = 40; @@ -4311,7 +4209,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerVariableWindowAttentionWithReuseTest) auto constexpr maxNumSequences = 8; auto constexpr maxBeamWidth = 1; auto constexpr sinkTokenLength = 0; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr maxSequenceLength = 128; @@ -4437,7 +4335,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventStreamOverflow) auto constexpr blocksInPrimaryPool = 8; auto constexpr blocksInSecondaryPool = 2; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr beamWidth = 1; @@ -4497,7 +4395,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventStreamPriority) auto constexpr blocksInPrimaryPool = 8; auto constexpr blocksInSecondaryPool = 2; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr beamWidth = 1; @@ -4574,7 +4472,7 @@ TEST_F(KVCacheManagerTest, GetPriorityByBlockId) auto constexpr maxAttentionWindow = 32; auto constexpr maxNumSequences = 4; auto constexpr beamWidth = 1; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); SizeType32 constexpr maxNewTokens = 4; tr::SamplingConfig const samplingConfig{beamWidth}; @@ -4640,7 +4538,7 @@ TEST_F(KVCacheManagerTest, CommitAndGetBlockHashesForRequest) auto constexpr maxNumSequences = 4; auto constexpr beamWidth = 1; auto constexpr beamIdx = 0; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); SizeType32 constexpr maxNewTokens = 8; tr::SamplingConfig const samplingConfig{beamWidth}; @@ -4766,7 +4664,7 @@ TEST_F(KVCacheManagerTest, CommitAndGetBlockHashesFrontRunsTrailingFullBlock) auto constexpr maxNumSequences = 4; auto constexpr beamWidth = 1; auto constexpr beamIdx = 0; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); SizeType32 constexpr maxNewTokens = 8; tr::SamplingConfig const samplingConfig{beamWidth}; @@ -4882,7 +4780,7 @@ TEST_F(KVCacheManagerTest, PinAndUnpinBlocksById) BlocksPerWindow const blocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numKvHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxAttentionWindow, maxAttentionWindow, true); kvCacheManager.allocatePools(false); @@ -4934,7 +4832,7 @@ TEST_F(KVCacheManagerTest, StoreBlocksForReuseWithPinDoesNotCreateGhostFreeBlock BlocksPerWindow const blocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numKvHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxAttentionWindow, maxAttentionWindow, true /* enableBlockReuse */); kvCacheManager.allocatePools(false); @@ -5006,7 +4904,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventStreamBlocking) auto constexpr blocksInPrimaryPool = 8; auto constexpr blocksInSecondaryPool = 2; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr beamWidth = 1; @@ -5026,7 +4924,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventStreamBlocking) EXPECT_EQ(getEvents(kvCacheManagerTest).size(), 0); KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength, true, CacheType::kSELF, std::nullopt, std::make_unique<tlk::KVCacheEventManager>(1024)); @@ -5061,7 +4959,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventStreamWindowSize) auto blocksInPool = std::vector<SizeType32>{8, 2}; auto blocksInSlidingWindowPool = std::vector<SizeType32>{4, 2}; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr beamWidth = 1; @@ -5120,11 +5018,10 @@ TEST_F(KVCacheManagerTest, KVCacheTransferManagerConcurrencyTest) auto pool = KVCacheBlockPool(0, 2, 0, 0, 0); - pool.primaryPtr = bufferManager.gpu(tr::ITensor::makeShape({1, blockSize}), tensorrt_llm::DataType::kFLOAT); + pool.primaryPtr = bufferManager.gpu(tr::ITensor::makeShape({1, blockSize}), nvinfer1::DataType::kFLOAT); bufferManager.setZero(*pool.primaryPtr); - pool.secondaryPtr - = tr::BufferManager::pinned(tr::ITensor::makeShape({1, blockSize}), tensorrt_llm::DataType::kFLOAT); + pool.secondaryPtr = tr::BufferManager::pinned(tr::ITensor::makeShape({1, blockSize}), nvinfer1::DataType::kFLOAT); // Write some specific data into the cpu blocks. for (int i = 0; i < blockSize; i++) @@ -5161,11 +5058,11 @@ TEST_F(KVCacheManagerTest, KVCacheTransferManagerPendingTransfersDistinguishPrim auto pool = KVCacheBlockPool(0, 2, 0, 0, 0); pool.primaryPtr - = bufferManager.gpu(tr::ITensor::makeShape({kNumSlotsPerPool, kBlockSize}), tensorrt_llm::DataType::kFLOAT); + = bufferManager.gpu(tr::ITensor::makeShape({kNumSlotsPerPool, kBlockSize}), nvinfer1::DataType::kFLOAT); bufferManager.setZero(*pool.primaryPtr); - pool.secondaryPtr = tr::BufferManager::pinned( - tr::ITensor::makeShape({kNumSlotsPerPool, kBlockSize}), tensorrt_llm::DataType::kFLOAT); + pool.secondaryPtr + = tr::BufferManager::pinned(tr::ITensor::makeShape({kNumSlotsPerPool, kBlockSize}), nvinfer1::DataType::kFLOAT); auto primarySlot0 = std::make_shared<KVCacheBlock>(0, tk::KVCacheIndex(0, false)); auto primarySlot1 = std::make_shared<KVCacheBlock>(1, tk::KVCacheIndex(1, false)); @@ -5252,10 +5149,10 @@ TEST_P(KVCacheManagerTest, DISABLED_KVCacheManagerSinkTokenLengthTest) auto const maxSequenceLength = tokensPerBlock * maxBlocksPerSeq; KVCacheManager kvCacheManager = homogeneousLayers ? KVCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse) : KVCacheManager(numHeadsPerLayer, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, maxBeamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, sinkTokenLength, + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse); kvCacheManager.allocatePools(false); @@ -5414,10 +5311,10 @@ TEST_P(KVCacheManagerTest, KVCacheManagerBatchTest) KVCacheManager kvCacheManager = homogeneousLayers ? KVCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse) : KVCacheManager(numHeadsPerLayer, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, maxBeamWidth, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, sinkTokenLength, + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, maxSequenceLength, enableBlockReuse); kvCacheManager.allocatePools(false); @@ -5554,12 +5451,12 @@ void testNeededBlocksOneStep(bool kv_cache_block_reuse, int beamWidth, int draft KVCacheManager kvCacheManager = homogeneousLayers ? KVCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, - tensorrt_llm::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, + sinkTokenLength, stream, maxSequenceLength, /*chunkSize=*/tokensPerBlock, kv_cache_block_reuse) : KVCacheManager(numHeadsPerLayer, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, - tensorrt_llm::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, + maxBeamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, + sinkTokenLength, stream, maxSequenceLength, /*chunkSize=*/tokensPerBlock, kv_cache_block_reuse); kvCacheManager.allocatePools(false); @@ -5762,7 +5659,7 @@ struct KvCacheManagerInstantiationParameters SizeType32 maxNumTokens; bool kvCacheBlockReuse; std::vector<SizeType32> maxAttentionWindowVec = {maxAttentionWindow}; - tensorrt_llm::DataType dtype = tensorrt_llm::DataType::kFLOAT; + nvinfer1::DataType dtype = nvinfer1::DataType::kFLOAT; }; BlocksPerWindow blocksAndWindow(SizeType32 numPrimaryBlocks, SizeType32 windowSize) @@ -6961,8 +6858,8 @@ TEST(KVCacheManagerReuseAccountingTest, CountReusableBlocksPartialMatch) = kvCacheManager->getNeededBlocksOneStep(req1, /*twoStepsLookAhead=*/false, onlyWindowSize); EXPECT_EQ(neededOneStep, promptLength / tokensPerBlock); // All 4 context blocks - // Free-but-cached blocks still consume free-pool capacity, but their tokens do not require recomputation. - EXPECT_EQ(req1.getEstimatedReusableTokens(), summaryShared.reusableBlocksAll * tokensPerBlock); + // Blocks are free (released via removeSequence), so onlyAllocated=true yields 0 reusable blocks. + EXPECT_EQ(req1.getEstimatedReusableTokens(), 0); } TEST(KVCacheManagerReuseAccountingTest, GetRemainingBlocksToCompletionWithPartialReuse) @@ -7079,7 +6976,7 @@ TEST(KVCacheManagerReuseAccountingTest, GetNeededBlocksOneStepWithFullReuse) tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(req0); kvCacheManager->removeSequence(req0.mRequestId, req0); - // Second request with identical tokens - the recoverable cached prefix should be reusable + // Second request with identical tokens - all context blocks should be reusable auto req1 = LlmRequest{ 1, maxNewTokens, @@ -7092,12 +6989,11 @@ TEST(KVCacheManagerReuseAccountingTest, GetNeededBlocksOneStepWithFullReuse) // getNeededBlocksOneStep must NOT subtract free reusable blocks. auto const neededOneStep = kvCacheManager->getNeededBlocksOneStep(req1, /*twoStepsLookAhead=*/false, onlyWindowSize); - auto const numContextBlocks = promptLength / tokensPerBlock; // 3 blocks - EXPECT_EQ(neededOneStep, numContextBlocks); // All 3 context blocks + auto const numSharedBlocks = promptLength / tokensPerBlock; // 3 blocks + EXPECT_EQ(neededOneStep, numSharedBlocks); // All 3 context blocks - // Sequence insertion omits the final prompt token, so only the preceding full blocks are recoverable. - auto const expectedReusableBlocks = (promptLength - 1) / tokensPerBlock; - EXPECT_EQ(req1.getEstimatedReusableTokens(), expectedReusableBlocks * tokensPerBlock); + // Blocks are free (released via removeSequence), so onlyAllocated=true yields 0 reusable blocks. + EXPECT_EQ(req1.getEstimatedReusableTokens(), 0); } TEST(KVCacheManagerReuseAccountingTest, ReuseDisabledReturnsFullBlockCount) @@ -7235,8 +7131,8 @@ TEST(KVCacheManagerReuseAccountingTest, MultipleRequestsWithSharedPrefix) = kvCacheManager->getNeededBlocksOneStep(req1, /*twoStepsLookAhead=*/false, onlyWindowSize); EXPECT_EQ(neededOneStep, promptLength / tokensPerBlock); // All 4 context blocks - // Free-but-cached blocks still consume free-pool capacity, but their tokens do not require recomputation. - EXPECT_EQ(req1.getEstimatedReusableTokens(), summaryPrefix.reusableBlocksAll * tokensPerBlock); + // Blocks are free (released via removeSequence), so onlyAllocated=true yields 0 reusable blocks. + EXPECT_EQ(req1.getEstimatedReusableTokens(), 0); // getRemainingBlocksToCompletion: 4 context + 1 gen = 5 blocks (no subtraction; blocks are free) auto const remaining = kvCacheManager->getRemainingBlocksToCompletion(req1, onlyWindowSize); @@ -7258,7 +7154,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventRemovedBatchedWithinWindow) auto constexpr maxNumSequences = 4; auto constexpr maxAttentionWindow = 32; auto constexpr beamWidth = 1; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); SizeType32 constexpr maxNewTokens{0}; tr::SamplingConfig const samplingConfig{beamWidth}; @@ -7336,7 +7232,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventRemovedOrderedBeforeStore) auto constexpr maxNumSequences = 4; auto constexpr maxAttentionWindow = 32; auto constexpr beamWidth = 1; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); SizeType32 constexpr maxNewTokens{0}; tr::SamplingConfig const samplingConfig{beamWidth}; @@ -7430,7 +7326,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerEventStoreForDifferentWindowDoesNotFlus auto constexpr blocksInSecondaryPool = 0; auto constexpr maxNumSequences = 4; auto constexpr beamWidth = 1; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); SizeType32 constexpr maxNewTokens{0}; tr::SamplingConfig const samplingConfig{beamWidth}; @@ -7543,8 +7439,7 @@ void testBlockManagerLinearAttention_ContextNoReuse(int beamWidth, int numTokens BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{linearWindowSizeCode, maxAttentionWindow}, tensorrt_llm::DataType::kHALF, - 0, + std::vector<BlockManager::SizeType32>{linearWindowSizeCode, maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, /*chunkSize*/ 0, CacheType::kSELF, std::nullopt, nullptr, false, true, nullptr, std::nullopt, false, 128, 0, false, linearAttentionMetadata); blockManager.allocatePools(false); @@ -7689,8 +7584,7 @@ void testBlockManagerLinearAttention_ContextReuse(int beamWidth, int numTokens0, BlockManager blockManager(std::vector(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, maxAttentionWindow, beamWidth, - std::vector<BlockManager::SizeType32>{linearWindowSizeCode, maxAttentionWindow}, tensorrt_llm::DataType::kHALF, - 0, + std::vector<BlockManager::SizeType32>{linearWindowSizeCode, maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, /*chunkSize*/ 0, CacheType::kSELF, std::nullopt, nullptr, false, true, nullptr, std::nullopt, false, 128, 0, false, linearAttentionMetadata); blockManager.allocatePools(false); @@ -7914,7 +7808,7 @@ void testKVCacheManagerLinearAttention_DecodingBlockGrowth( {linearWindowSizeCode, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numKvHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, beamWidth, std::vector<BlockManager::SizeType32>{linearWindowSizeCode}, - /*dtype*/ tensorrt_llm::DataType::kHALF, + /*dtype*/ nvinfer1::DataType::kHALF, /*sinkTokenLen*/ sinkTokenLen, /*stream*/ stream, /*maxSequenceLength*/ maxAttentionWindow, @@ -8026,7 +7920,7 @@ void testKVCacheManagerLinearAttention_BlockCopying( {linearWindowSizeCode, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numKvHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, beamWidth, std::vector<BlockManager::SizeType32>{linearWindowSizeCode, maxAttentionWindow}, - tensorrt_llm::DataType::kHALF, sinkTokenLen, stream, maxAttentionWindow, /*chunkSize*/ 0, enableContextReuse, + nvinfer1::DataType::kHALF, sinkTokenLen, stream, maxAttentionWindow, /*chunkSize*/ 0, enableContextReuse, CacheType::kSELF, std::nullopt, nullptr, false, true, nullptr, false, 128, 0, false, linearAttentionMetadata); kvCacheManager.allocatePools(false); @@ -8352,7 +8246,7 @@ TEST_F(KVCacheManagerTest, StaticLinearHybridAllocationTest) // Static-hybrid path requires block reuse to be disabled. tle::KvCacheConfig const kvCacheConfigDisabledReuse{/*enableBlockReuse=*/false}; auto const blocksPerWindow - = KVCacheManager::calculateMaxNumBlocks(kvCacheConfigDisabledReuse, tensorrt_llm::DataType::kHALF, + = KVCacheManager::calculateMaxNumBlocks(kvCacheConfigDisabledReuse, nvinfer1::DataType::kHALF, numKvHeadsPerLayer, sizePerHead, tokensPerBlock, worldConfig, windowSizeToLayers, allottedPrimaryMemBytes, allottedSecondaryMemBytes, extraCostMemory, kvFactor, maxBatchSize, linearAttentionMetadata); @@ -8371,7 +8265,7 @@ TEST_F(KVCacheManagerTest, StaticLinearHybridAllocationTest) // so the linear pool falls back to memory-budget-based sizing rather than maxBatchSize. tle::KvCacheConfig const kvCacheConfigEnabledReuse{/*enableBlockReuse=*/true}; auto const dynamicBlocksPerWindow - = KVCacheManager::calculateMaxNumBlocks(kvCacheConfigEnabledReuse, tensorrt_llm::DataType::kHALF, + = KVCacheManager::calculateMaxNumBlocks(kvCacheConfigEnabledReuse, nvinfer1::DataType::kHALF, numKvHeadsPerLayer, sizePerHead, tokensPerBlock, worldConfig, windowSizeToLayers, allottedPrimaryMemBytes, allottedSecondaryMemBytes, extraCostMemory, kvFactor, maxBatchSize, linearAttentionMetadata); EXPECT_NE(std::get<0>(dynamicBlocksPerWindow.at(linearWindowSizeCode)), maxBatchSize); @@ -8404,7 +8298,7 @@ static auto makeBatchTestKVCacheManager(std::shared_ptr<tensorrt_llm::runtime::C auto mgr = std::make_unique<KVCacheManager>(numLayers, numKvHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, - tensorrt_llm::DataType::kHALF, 0, stream, maxAttentionWindow, /*chunkSize=*/maxAttentionWindow, + nvinfer1::DataType::kHALF, 0, stream, maxAttentionWindow, /*chunkSize=*/maxAttentionWindow, /*enableBlockReuse=*/true, CacheType::kSELF, /*secondaryOffloadMinPriority=*/std::nullopt, /*eventManager=*/nullptr, @@ -8733,7 +8627,7 @@ TEST_F(KVCacheManagerTest, BatchAddSequence_NonLeafCopySourceTightPool) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numKvHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxAttentionWindow, /*chunkSize=*/maxAttentionWindow, /*enableBlockReuse=*/true, CacheType::kSELF, /*secondaryOffloadMinPriority=*/std::nullopt, /*eventManager=*/nullptr, @@ -8890,7 +8784,7 @@ std::unique_ptr<KVCacheManager> makePriorityEvictionManager( auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, 0}}}; auto mgr = std::make_unique<KVCacheManager>(kPE_NUM_LAYERS, kPE_NUM_HEADS, kPE_SIZE_PER_HEAD, kPE_TOKENS_PER_BLOCK, blocksPerWindow, kPE_MAX_NUM_SEQUENCES, kPE_BEAM_WIDTH, - std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxAttentionWindow, /*chunkSize=*/maxAttentionWindow, /*enableBlockReuse=*/true); mgr->allocatePools(false); return mgr; @@ -9224,7 +9118,7 @@ std::unique_ptr<KVCacheManager> makeVSWAManager( { auto const blocksPerWindow = BlocksPerWindow{{kVSWA_ATTENTION_WINDOW, {blocksInPrimaryPool, 0}}}; auto mgr = std::make_unique<KVCacheManager>(2, 2, 64, kVSWA_TOKENS_PER_BLOCK, blocksPerWindow, 8, kVSWA_BEAM_WIDTH, - std::vector<SizeType32>{kVSWA_ATTENTION_WINDOW}, tensorrt_llm::DataType::kHALF, 0, stream, + std::vector<SizeType32>{kVSWA_ATTENTION_WINDOW}, nvinfer1::DataType::kHALF, 0, stream, kVSWA_MAX_SEQUENCE_LENGTH, /*chunkSize=*/kVSWA_MAX_SEQUENCE_LENGTH, enableBlockReuse); mgr->allocatePools(false); return mgr; @@ -9243,7 +9137,7 @@ std::unique_ptr<KVCacheManager> makeSmallWindowManager( SizeType32 constexpr kSmallMaxSeqLen = 128; auto const blocksPerWindow = BlocksPerWindow{{kSmallWindow, {blocksInPrimaryPool, 0}}}; auto mgr = std::make_unique<KVCacheManager>(2, 2, 64, kSmallTpb, blocksPerWindow, 8, kVSWA_BEAM_WIDTH, - std::vector<SizeType32>{kSmallWindow}, tensorrt_llm::DataType::kHALF, 0, stream, kSmallMaxSeqLen, + std::vector<SizeType32>{kSmallWindow}, nvinfer1::DataType::kHALF, 0, stream, kSmallMaxSeqLen, /*chunkSize=*/kSmallMaxSeqLen, /*enableBlockReuse=*/true); mgr->allocatePools(false); return mgr; @@ -9966,7 +9860,7 @@ TEST_F(KVCacheManagerTest, VSWAEvictedPlaceholderAnchorAllowsTrailingReuse) auto const blocksPerWindow = BlocksPerWindow{{window, {blocksInPrimaryPool, 0}}}; KVCacheManager kvCacheManager(2, 2, 64, tpb, blocksPerWindow, 8, kVSWA_BEAM_WIDTH, std::vector<SizeType32>{window}, - tensorrt_llm::DataType::kHALF, 0, stream, + nvinfer1::DataType::kHALF, 0, stream, /*maxSequenceLength=*/128, /*chunkSize=*/128, /*enableBlockReuse=*/true); kvCacheManager.allocatePools(false); auto const& blockManager = kvCacheManager.getBlockManager(); @@ -10102,7 +9996,7 @@ std::unique_ptr<KVCacheManager> makeConnectorTestKVCacheManager( auto mgr = std::make_unique<KVCacheManager>(std::vector<SizeType32>(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, - /*dtype*/ tensorrt_llm::DataType::kHALF, + /*dtype*/ nvinfer1::DataType::kHALF, /*sinkTokenLength*/ 0, stream, /*maxSequenceLength*/ maxAttentionWindow, /*chunkSize*/ maxAttentionWindow, @@ -10296,7 +10190,7 @@ TEST_F(KVCacheManagerTest, BlockManagerTestPerWindowFallback) auto constexpr maxBeamWidth = 1; auto constexpr smallWindow = 1024; auto constexpr largeWindow = 4096; - auto constexpr scalarDtype = tensorrt_llm::DataType::kHALF; + auto constexpr scalarDtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto const maxAttentionWindowVec = std::vector<SizeType32>{smallWindow, largeWindow}; auto const blocksPerWindow = BlocksPerWindow{ @@ -10344,7 +10238,7 @@ TEST(BaseKVCacheManagerCalculateMaxNumBlocks, PerWindowOverrideDivergesByteBudge uint64_t const allottedPrimaryMemBytes = static_cast<uint64_t>(1) << 30; // 1 GiB uint64_t const allottedSecondaryMemBytes = static_cast<uint64_t>(1) << 30; size_t const extraCostMemory = 0; - auto const dtype = tensorrt_llm::DataType::kHALF; + auto const dtype = nvinfer1::DataType::kHALF; tensorrt_llm::executor::KvCacheConfig const config{}; tensorrt_llm::runtime::WorldConfig const worldConfig{}; @@ -10398,7 +10292,7 @@ TEST_F(KVCacheManagerTest, KVCacheManagerSWAEvictionCountPerWindow) auto constexpr maxNumSequences = 4; auto constexpr maxBeamWidth = 1; auto constexpr sinkTokenLength = 0; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto const stream = std::make_shared<tr::CudaStream>(); auto constexpr maxSequenceLength = 128; auto constexpr maxNewTokens = 40; @@ -10488,7 +10382,7 @@ TEST_F(KVCacheManagerTest, GenerationRequestClearCacheBlocksPerWindowResetsOnlyT BlockManager blockManager(std::vector<BlockManager::SizeType32>(numLayers, numKvHeads), sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, stream, /*maxSequenceLength=*/fullWindow, maxBeamWidth, maxAttentionWindowVec, - tensorrt_llm::DataType::kHALF, /*sinkBubbleLength=*/0, /*chunkSize=*/0); + nvinfer1::DataType::kHALF, /*sinkBubbleLength=*/0, /*chunkSize=*/0); blockManager.allocatePools(/*useUvm=*/false); auto constexpr requestId = 7; @@ -10541,7 +10435,7 @@ TEST_F(KVCacheManagerTest, VswaMixedHeadDimReuseSmoke) auto constexpr smallSizePerHead = 256; auto constexpr largeSizePerHead = 512; auto constexpr sinkTokenLength = 0; - auto constexpr dtype = tensorrt_llm::DataType::kHALF; + auto constexpr dtype = nvinfer1::DataType::kHALF; auto constexpr maxSequenceLength = 64; auto constexpr maxNewTokens = 0; auto const stream = std::make_shared<tr::CudaStream>(); @@ -10619,58 +10513,12 @@ TEST_F(KVCacheManagerTest, VswaMixedHeadDimReuseSmoke) } } -TEST_F(KVCacheManagerTest, HybridDisaggUsesAttentionPoolDtype) -{ - auto constexpr numKvHeads = 2; - auto constexpr sizePerHead = 16; - auto constexpr tokensPerBlock = 4; - auto constexpr blocksInPrimaryPool = 4; - auto constexpr blocksInSecondaryPool = 0; - auto constexpr maxNumSequences = 2; - auto constexpr maxBeamWidth = 1; - auto constexpr maxAttentionWindow = 16; - auto constexpr recurrentStatesBytes = 64; - SizeType32 constexpr recurrentStatesWindow = LinearAttentionMetadata::LinearCacheType::kRecurrentStates; - - LinearAttentionMetadata const linearAttentionMetadata{ - .linearLayerIndices = {0}, - .cacheType = recurrentStatesWindow, - .allRecurrentStatesBytes = recurrentStatesBytes, - }; - auto const blocksPerWindow = BlocksPerWindow{ - {recurrentStatesWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}, - {maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}, - }; - auto const poolConfigurations = std::vector<PoolConfiguration>{ - {recurrentStatesWindow, sizePerHead, tensorrt_llm::DataType::kHALF}, - {maxAttentionWindow, sizePerHead, tensorrt_llm::DataType::kFP8}, - }; - auto const stream = std::make_shared<tr::CudaStream>(); - - auto kvCacheManager = std::make_unique<KVCacheManager>(std::vector<SizeType32>{0, numKvHeads}, sizePerHead, - tokensPerBlock, blocksPerWindow, maxNumSequences, maxBeamWidth, - std::vector<SizeType32>{recurrentStatesWindow, maxAttentionWindow}, tensorrt_llm::DataType::kFP8, - /*sinkTokenLength=*/0, stream, maxAttentionWindow, /*chunkSize=*/0, /*enableBlockReuse=*/false, - CacheType::kSELF, std::nullopt, nullptr, /*enablePartialReuse=*/false, /*copyOnPartialReuse=*/true, nullptr, - /*enableIndexerKCache=*/false, /*indexerKCacheQuantBlockSize=*/128, /*indexerKCacheIndexHeadDim=*/0, - /*indexerKCacheUseFp4=*/false, linearAttentionMetadata, poolConfigurations); - kvCacheManager->allocatePools(/*useUvm=*/false); - - CacheTransBufferManager cacheTransBufferManager(kvCacheManager.get(), /*maxNumTokens=*/tokensPerBlock); - EXPECT_EQ(cacheTransBufferManager.getDataType(), tensorrt_llm::DataType::kFP8); - - auto const bufferId = cacheTransBufferManager.assignBufferIndexForSend(); - ASSERT_TRUE(bufferId.has_value()); - EXPECT_EQ(cacheTransBufferManager.getSendBuffer(bufferId)->getDataType(), tensorrt_llm::DataType::kFP8); - cacheTransBufferManager.freeBufferIndexForSend(bufferId); -} - // A6: VSWA + disagg dtype mismatch must fire the A4 guard. // -// CacheTransBufferManager uses a single dtype for the wire transport. When a -// KVCacheManager hosts attention pools with differing dtypes (mixed-precision -// per-window), that silent coercion would corrupt the wire format. The guard in -// cacheTransBuffer.cpp must throw at construction time. +// The constructor of CacheTransBufferManager picks pool 0's dtype as canonical for +// the wire transport. When a KVCacheManager hosts pools with differing dtypes +// (mixed-precision per-window), that silent coercion would corrupt the wire format. +// The guard added in cacheTransBuffer.cpp must throw at construction time. // // This test only exercises the helper / construction path that runs the guard; it // does not stand up a full disaggregated transfer (out of scope at unit-test @@ -10693,12 +10541,11 @@ TEST_F(KVCacheManagerTest, VswaDisaggDtypeMismatchTriggersGuard) auto const maxAttentionWindowVec = std::vector<SizeType32>{smallWindow, largeWindow}; auto const blocksPerWindow = BlocksPerWindow{ {smallWindow, {blocksInPrimary, blocksInSecondary}}, {largeWindow, {blocksInPrimary, blocksInSecondary}}}; - auto const poolConfigurations - = std::vector<PoolConfiguration>{{smallWindow, sizePerHead, tensorrt_llm::DataType::kHALF}, - {largeWindow, sizePerHead, tensorrt_llm::DataType::kBF16}}; + auto const poolConfigurations = std::vector<PoolConfiguration>{ + {smallWindow, sizePerHead, nvinfer1::DataType::kHALF}, {largeWindow, sizePerHead, nvinfer1::DataType::kBF16}}; auto kvCacheManager = std::make_unique<KVCacheManager>(numLayers, numKvHeads, sizePerHead, tokensPerBlock, - blocksPerWindow, maxNumSequences, maxBeamWidth, maxAttentionWindowVec, /*dtype=*/tensorrt_llm::DataType::kHALF, + blocksPerWindow, maxNumSequences, maxBeamWidth, maxAttentionWindowVec, /*dtype=*/nvinfer1::DataType::kHALF, sinkTokenLength, stream, maxSequenceLength, /*chunkSize=*/0, /*enableBlockReuse=*/false, CacheType::kSELF, /*secondaryOffloadMinPriority=*/std::nullopt, /*eventManager=*/nullptr, @@ -10710,23 +10557,14 @@ TEST_F(KVCacheManagerTest, VswaDisaggDtypeMismatchTriggersGuard) kvCacheManager->allocatePools(/*useUvm=*/false); // Sanity: the manager really does host KV pools with two different dtypes. - auto const& blockManager = kvCacheManager->getBlockManager(); - ASSERT_GE(blockManager.getNumPools(/*includeBlockScalePools=*/false, /*includeIndexerKCachePools=*/false), 2); - std::optional<tensorrt_llm::DataType> dtype0; + auto const numKvPools = kvCacheManager->getBlockManager().getNumPools( + /*includeBlockScalePools=*/false, /*includeIndexerKCachePools=*/false); + ASSERT_GE(numKvPools, 2); + auto const dtype0 = kvCacheManager->getPrimaryPool(0)->getDataType(); bool foundMismatch = false; - for (SizeType32 poolIdx = 0; poolIdx < blockManager.getNumPools(); ++poolIdx) + for (SizeType32 i = 1; i < numKvPools; ++i) { - auto const& pool = blockManager.getPool(poolIdx); - if (pool.containsBlockScales || pool.containsIndexerKCache) - { - continue; - } - auto const dataType = blockManager.getPrimaryPool(poolIdx)->getDataType(); - if (!dtype0.has_value()) - { - dtype0 = dataType; - } - else if (dataType != dtype0.value()) + if (kvCacheManager->getPrimaryPool(i)->getDataType() != dtype0) { foundMismatch = true; break; diff --git a/cpp/tests/unit_tests/batch_manager/kvCacheUtilsTest.cpp b/cpp/tests/unit_tests/batch_manager/kvCacheUtilsTest.cpp index a7b15e5ae8e0..2047a885d98c 100644 --- a/cpp/tests/unit_tests/batch_manager/kvCacheUtilsTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/kvCacheUtilsTest.cpp @@ -21,7 +21,6 @@ #include <gtest/gtest.h> #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" namespace tc = tensorrt_llm::common; namespace tr = tensorrt_llm::runtime; @@ -49,7 +48,7 @@ TEST_F(BlockIteratorTest, BasicTest) auto constexpr mNumLayers = 5; auto constexpr mBlockSize = 32; auto const cacheShape = tr::ITensor::makeShape({mNumPrimaryBlocks, mNumLayers, 2, mBlockSize}); - constexpr tensorrt_llm::DataType dtype{tr::TRTDataType<DataType>::value}; + constexpr nvinfer1::DataType dtype{tr::TRTDataType<DataType>::value}; tr::ITensor::SharedPtr pool = tr::BufferManager::cpu(cacheShape, dtype); std::vector<SizeType32> blockIds(mNumPrimaryBlocks); std::iota(blockIds.begin(), blockIds.end(), 0); @@ -76,7 +75,7 @@ TEST_F(BlockIteratorTest, BasicTest) TEST_F(BlockIteratorTest, CacheManagerTest) { - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; auto constexpr numLayers = 12; auto constexpr numKvHeads = 6; auto constexpr sizePerHead = 16; diff --git a/cpp/tests/unit_tests/batch_manager/llmRequestTest.cpp b/cpp/tests/unit_tests/batch_manager/llmRequestTest.cpp index 418178ff5023..120263b01af1 100644 --- a/cpp/tests/unit_tests/batch_manager/llmRequestTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/llmRequestTest.cpp @@ -16,7 +16,6 @@ */ #include "tensorrt_llm/batch_manager/llmRequest.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/executor/types.h" @@ -98,7 +97,7 @@ TEST_F(LlmRequestTest, fromExecutorRequest) EXPECT_TRUE(llmReq.getStopWordsList().has_value()); { auto badWordsTensor = llmReq.getBadWordsList().value(); - EXPECT_EQ(badWordsTensor->getDataType(), tensorrt_llm::DataType::kINT32); + EXPECT_EQ(badWordsTensor->getDataType(), nvinfer1::DataType::kINT32); EXPECT_EQ(badWordsTensor->getShape().nbDims, 3); EXPECT_EQ(badWordsTensor->getShape().d[0], 1); EXPECT_EQ(badWordsTensor->getShape().d[1], 2); @@ -120,7 +119,7 @@ TEST_F(LlmRequestTest, fromExecutorRequest) { auto stopWordsTensor = llmReq.getStopWordsList().value(); - EXPECT_EQ(stopWordsTensor->getDataType(), tensorrt_llm::DataType::kINT32); + EXPECT_EQ(stopWordsTensor->getDataType(), nvinfer1::DataType::kINT32); EXPECT_EQ(stopWordsTensor->getShape().nbDims, 3); EXPECT_EQ(stopWordsTensor->getShape().d[0], 1); EXPECT_EQ(stopWordsTensor->getShape().d[1], 2); @@ -152,7 +151,7 @@ TEST_F(LlmRequestTest, fromExecutorRequest) EXPECT_EQ(llmReq.getPromptEmbeddingTable().value()->getShape().d[0], 1); EXPECT_EQ(llmReq.getPromptEmbeddingTable().value()->getShape().d[1], vocabSize); EXPECT_EQ(llmReq.getPromptEmbeddingTable().value()->getShape().d[2], hiddenSize); - EXPECT_EQ(llmReq.getPromptEmbeddingTable().value()->getDataType(), tensorrt_llm::DataType::kFLOAT); + EXPECT_EQ(llmReq.getPromptEmbeddingTable().value()->getDataType(), nvinfer1::DataType::kFLOAT); EXPECT_EQ(llmReq.getPromptVocabSize().value(), vocabSize); VecUniqueTokens uniqueTokens; for (size_t i = 0; i < inputTokens.size(); ++i) @@ -374,7 +373,7 @@ TEST_F(LlmRequestTest, testAllocateLogitsBuffer) EXPECT_EQ(llmReq.mPromptLen, 5); SizeType32 vocabSizePadded = 32000; - tensorrt_llm::DataType logitsDataType = tensorrt_llm::DataType::kFLOAT; + nvinfer1::DataType logitsDataType = nvinfer1::DataType::kFLOAT; // Test the allocation of context logits EXPECT_EQ(llmReq.getContextLogitsHost(), nullptr); @@ -463,7 +462,7 @@ TEST_F(LlmRequestTest, testCreateRequests) SizeType32 maxNewTokens{60}; tb::LlmRequest::RequestIdType requestId{77}; SizeType32 vocabSize{32}; - tensorrt_llm::DataType dtype{tensorrt_llm::DataType::kHALF}; + nvinfer1::DataType dtype{nvinfer1::DataType::kHALF}; tr::SamplingConfig samplingConfig(1); samplingConfig.randomSeed = std::vector<texec::RandomSeedType>{7}; diff --git a/cpp/tests/unit_tests/batch_manager/microBatchSchedulerTest.cpp b/cpp/tests/unit_tests/batch_manager/microBatchSchedulerTest.cpp index c392b05dccf7..c2c308b37dd9 100644 --- a/cpp/tests/unit_tests/batch_manager/microBatchSchedulerTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/microBatchSchedulerTest.cpp @@ -23,7 +23,6 @@ #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/batch_manager/llmRequest.h" #include "tensorrt_llm/batch_manager/microBatchScheduler.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/testing/kvCacheManagerTestUtil.h" #include <numeric> @@ -60,7 +59,7 @@ class MicroBatchSchedulerTest : public ::testing::Test // NOLINT(cppcoreguidelin { draftTokens = std::make_shared<std::vector<int32_t>>(draftTokensLen, 2); draftLogits = BufferManager::cpu( - ITensor::makeShape({draftTokensLen, /* vocabSizePadded*/ 42}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({draftTokensLen, /* vocabSizePadded*/ 42}), nvinfer1::DataType::kFLOAT); } return std::make_shared<LlmRequest>(reqId, maxNewTokens, inputTokens, samplingConfig, /*isStreaming=*/false, @@ -1247,7 +1246,7 @@ class CombinedSchedulerTest : public ::testing::Test return std::make_shared<kv_cache_manager::KVCacheManager>( /*numLayers=*/10, /*nbKvHeads=*/10, /*sizePerHead=*/1, tokensPerBlock, blocksPerWindow, maxNumRequests, - /*maxBeamWidth=*/1, std::vector<SizeType32>{maxNumTokensPerSeq}, tensorrt_llm::DataType::kHALF, + /*maxBeamWidth=*/1, std::vector<SizeType32>{maxNumTokensPerSeq}, nvinfer1::DataType::kHALF, /*sinkTokenLength=*/0, stream, maxNumTokensPerSeq, /*chunkSize=*/maxNumTokensPerSeq, enableReuse); } @@ -2054,19 +2053,19 @@ class ForceChunkTest : public MicroBatchSchedulerTest } }; -TEST_F(ForceChunkTest, NoSnapshotPointsUsesRemainingContext) +TEST_F(ForceChunkTest, Basic) { - // A request without snapshot points is not split at chunkUnitSize. + // A single request with prompt_len > chunk_unit_size is chunked to unit_size. auto reqs = initRequests({30}); MicroBatchScheduler::setCtxRequestsChunkSize(reqs, Policy::kFORCE_CHUNK, /*ctxTokensCapacity=*/std::nullopt, /*chunkUnitSize=*/10, /*maxContextLength=*/std::nullopt); - EXPECT_EQ(reqs[0]->getContextChunkSize(), 30); + EXPECT_EQ(reqs[0]->getContextChunkSize(), 10); } TEST_F(ForceChunkTest, PromptSmallerThanUnit) { - // Without snapshot points, a short prompt is consumed in full. + // When prompt_len < chunk_unit_size, chunk_size = prompt_len (min). auto reqs = initRequests({8}); MicroBatchScheduler::setCtxRequestsChunkSize(reqs, Policy::kFORCE_CHUNK, std::nullopt, 20, std::nullopt); @@ -2075,7 +2074,7 @@ TEST_F(ForceChunkTest, PromptSmallerThanUnit) TEST_F(ForceChunkTest, ExactUnitSize) { - // Without snapshot points, an exact-unit prompt is consumed in full. + // When prompt_len == chunk_unit_size, chunk_size = prompt_len. auto reqs = initRequests({10}); MicroBatchScheduler::setCtxRequestsChunkSize(reqs, Policy::kFORCE_CHUNK, std::nullopt, 10, std::nullopt); @@ -2084,36 +2083,31 @@ TEST_F(ForceChunkTest, ExactUnitSize) TEST_F(ForceChunkTest, MultipleRequests) { - // Requests without snapshot points independently consume their remaining contexts. + // Each request independently gets min(remaining, unit_size). auto reqs = initRequests({25, 15, 5}); MicroBatchScheduler::setCtxRequestsChunkSize(reqs, Policy::kFORCE_CHUNK, std::nullopt, 10, std::nullopt); - EXPECT_EQ(reqs[0]->getContextChunkSize(), 25); - EXPECT_EQ(reqs[1]->getContextChunkSize(), 15); - EXPECT_EQ(reqs[2]->getContextChunkSize(), 5); + EXPECT_EQ(reqs[0]->getContextChunkSize(), 10); + EXPECT_EQ(reqs[1]->getContextChunkSize(), 10); + EXPECT_EQ(reqs[2]->getContextChunkSize(), 5); // min(5, 10) = 5 } TEST_F(ForceChunkTest, CapacityLimits) { - // Budget truncation is unit-aligned; later requests with less than one - // unit available are delayed. + // When capacity is limited, later requests get chunk_size=0. auto reqs = initRequests({30, 30}); MicroBatchScheduler::setCtxRequestsChunkSize( reqs, Policy::kFORCE_CHUNK, /*ctxTokensCapacity=*/15, /*chunkUnitSize=*/10, std::nullopt); - // req0 is budget-truncated to 10; only 5 remain, so req1 gets 0. + // req0 gets 10, req1 would push total to 20 > 15 → 0 EXPECT_EQ(reqs[0]->getContextChunkSize(), 10); EXPECT_EQ(reqs[1]->getContextChunkSize(), 0); } TEST_F(ForceChunkTest, CapacityExactFit) { - // Capacity exactly accommodates both requested snapshot chunks. + // When capacity exactly accommodates all chunks. auto reqs = initRequests({30, 30}); - for (auto const& req : reqs) - { - req->setExpectedSnapshotPoints({10}); - } MicroBatchScheduler::setCtxRequestsChunkSize( reqs, Policy::kFORCE_CHUNK, /*ctxTokensCapacity=*/20, /*chunkUnitSize=*/10, std::nullopt); @@ -2121,38 +2115,11 @@ TEST_F(ForceChunkTest, CapacityExactFit) EXPECT_EQ(reqs[1]->getContextChunkSize(), 10); } -TEST_F(ForceChunkTest, ExpectedChunkingPoints) -{ - // Expected snapshot points are absolute context positions. - auto reqs = initRequests({30}); - reqs[0]->setExpectedSnapshotPoints({12, 25}); - - chunkIteration(reqs, 10); - expectPositions(reqs, {12}, "iter 1"); - - chunkIteration(reqs, 10); - expectPositions(reqs, {25}, "iter 2"); - - chunkIteration(reqs, 10); - expectPositions(reqs, {30}, "iter 3"); -} - -TEST_F(ForceChunkTest, CapacityRoundsExpectedChunkDownToUnit) -{ - auto reqs = initRequests({50}); - reqs[0]->setExpectedSnapshotPoints({30}); - - MicroBatchScheduler::setCtxRequestsChunkSize( - reqs, Policy::kFORCE_CHUNK, /*ctxTokensCapacity=*/25, /*chunkUnitSize=*/10, std::nullopt); - - EXPECT_EQ(reqs[0]->getContextChunkSize(), 20); -} - TEST_F(ForceChunkTest, MultiIteration) { - // Snapshot points at 10 and 20 split a 25-token prompt into three iterations. + // A request with prompt_len=25 and chunk_unit_size=10 processes in 3 iterations: + // chunk 1: 10, chunk 2: 10, chunk 3: 5. auto reqs = initRequests({25}); - reqs[0]->setExpectedSnapshotPoints({10, 20}); // Iteration 1 chunkIteration(reqs, 10); @@ -2170,10 +2137,8 @@ TEST_F(ForceChunkTest, MultiIteration) TEST_F(ForceChunkTest, MultiRequestMultiIteration) { // Two requests with different lengths processed over multiple iterations. - // Their expected snapshot points determine each boundary. + // prompt_len={25, 12}, chunk_unit_size=10. auto reqs = initRequests({25, 12}); - reqs[0]->setExpectedSnapshotPoints({10, 20}); - reqs[1]->setExpectedSnapshotPoints({10}); // Iteration 1: both get 10 chunkIteration(reqs, 10); @@ -2191,12 +2156,8 @@ TEST_F(ForceChunkTest, MultiRequestMultiIteration) TEST_F(ForceChunkTest, CapacityAcrossIterations) { // With limited capacity, some requests may be delayed to later iterations. - // Both requests have snapshot points at 10 and 20; capacity is 15. + // prompt_len={25, 25}, chunk_unit_size=10, capacity=15. auto reqs = initRequests({25, 25}); - for (auto const& req : reqs) - { - req->setExpectedSnapshotPoints({10, 20}); - } // Iteration 1: req0=10, req1=0 (10+10=20 > 15) chunkIteration(reqs, 10, /*ctxTokensCapacity=*/15); @@ -2219,10 +2180,10 @@ TEST_F(ForceChunkTest, CapacityAcrossIterations) expectPositions(reqs, {25, 25}, "iter 5"); } -TEST_F(ForceChunkTest, FullSchedulerWithoutSnapshotPoints) +TEST_F(ForceChunkTest, FullSchedulerPath) { - // Test via MicroBatchScheduler::operator(): without snapshot points, a - // context that fits is not split at chunkUnitSize. + // Test via MicroBatchScheduler::operator() — FORCE_CHUNK always re-chunks + // even when all contexts fit within the token budget. batch_scheduler::ContextChunkingConfig chunkConfig; chunkConfig.chunkingPolicy = Policy::kFORCE_CHUNK; chunkConfig.chunkUnitSize = 10; @@ -2239,33 +2200,9 @@ TEST_F(ForceChunkTest, FullSchedulerWithoutSnapshotPoints) auto const [contextRequests, genRequests] = (*scheduler)(activeRequests, inflightReqIds, maxBatchSize, maxNumTokens); + // Despite budget=100 >> prompt=30, FORCE_CHUNK limits chunk to unit_size=10. ASSERT_EQ(contextRequests.size(), 1); - EXPECT_EQ(contextRequests[0]->getContextChunkSize(), 30); - EXPECT_EQ(genRequests.size(), 0); -} - -TEST_F(ForceChunkTest, FullSchedulerUsesExpectedChunkingPoints) -{ - batch_scheduler::ContextChunkingConfig chunkConfig; - chunkConfig.chunkingPolicy = Policy::kFORCE_CHUNK; - chunkConfig.chunkUnitSize = 10; - - auto scheduler = std::make_shared<MicroBatchScheduler>(chunkConfig); - - constexpr SizeType32 maxBatchSize = 4; - constexpr SizeType32 maxNumTokens = 100; - - RequestVector activeRequests; - auto request = createRequest(/*promptLen=*/30, /*maxNewTokens=*/1, /*reqId=*/0); - request->setExpectedSnapshotPoints({12, 25}); - activeRequests.push_back(request); - - ReqIdsSet inflightReqIds; - auto const [contextRequests, genRequests] - = (*scheduler)(activeRequests, inflightReqIds, maxBatchSize, maxNumTokens); - - ASSERT_EQ(contextRequests.size(), 1); - EXPECT_EQ(contextRequests[0]->getContextChunkSize(), 12); + EXPECT_EQ(contextRequests[0]->getContextChunkSize(), 10); EXPECT_EQ(genRequests.size(), 0); } @@ -2297,8 +2234,8 @@ TEST_F(ForceChunkTest, FullSchedulerMultipleRequests) { chunks[req->mRequestId] = req->getContextChunkSize(); } - EXPECT_EQ(chunks[0], 25); - EXPECT_EQ(chunks[1], 15); + EXPECT_EQ(chunks[0], 10); + EXPECT_EQ(chunks[1], 10); EXPECT_EQ(chunks[2], 5); } @@ -2330,7 +2267,6 @@ TEST_F(ForceChunkTest, FullSchedulerWithGeneration) EXPECT_EQ(genRequests.size(), 1); ASSERT_EQ(contextRequests.size(), 1); - // Budget remaining is 14, so the context is rounded down to one - // 10-token chunk-unit boundary. + // Budget remaining = 15 - 1 (gen) = 14; chunk = min(30, 10) = 10 EXPECT_EQ(contextRequests[0]->getContextChunkSize(), 10); } diff --git a/cpp/tests/unit_tests/batch_manager/peftCacheManagerTest.cpp b/cpp/tests/unit_tests/batch_manager/peftCacheManagerTest.cpp index ef513894bfb6..49adfe5a6cb6 100644 --- a/cpp/tests/unit_tests/batch_manager/peftCacheManagerTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/peftCacheManagerTest.cpp @@ -32,7 +32,7 @@ #include "tensorrt_llm/runtime/utils/numpyUtils.h" #include "tensorrt_llm/runtime/worldConfig.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <cuda_runtime.h> #include <gmock/gmock-matchers.h> @@ -78,7 +78,7 @@ class PeftCacheManagerTest : public ::testing::Test // NOLINT(cppcoreguidelines- void SetUp() override { - mModelConfig = std::make_unique<ModelConfig>(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + mModelConfig = std::make_unique<ModelConfig>(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); mModelConfig->setMlpHiddenSize(32); mWorldConfig = std::make_unique<WorldConfig>(2, 1, 1, 0); std::vector<LoraModule> modules{ @@ -285,7 +285,7 @@ TEST_F(PeftCacheManagerTest, gptManagerSim) auto peftManager = std::make_unique<PeftCacheManager>(config, *mModelConfig, *mWorldConfig, *mManager); auto pageConfig = LoraCachePageManagerConfig( - runtime::MemoryType::kCPU, tensorrt_llm::DataType::kFLOAT, 128, 128, 2 * 8 * 64, 4 * 16, 1); + runtime::MemoryType::kCPU, nvinfer1::DataType::kFLOAT, 128, 128, 2 * 8 * 64, 4 * 16, 1); auto loraCache = std::make_unique<LoraCache>(pageConfig, *mModelConfig, *mWorldConfig, *mManager); std::map<uint64_t, std::pair<TensorPtr, TensorPtr>> loras; @@ -505,7 +505,7 @@ TEST_F(PeftCacheManagerTest, getMaxNumSlots) config.numHostModuleLayer = 8192 * 8; config.numDeviceModuleLayer = 8292 * 2; auto [hostSlots, deviceSlots] - = PeftCacheManager::getMaxNumSlots(config, tensorrt_llm::DataType::kHALF, 256, 4 * 256, *mManager); + = PeftCacheManager::getMaxNumSlots(config, nvinfer1::DataType::kHALF, 256, 4 * 256, *mManager); EXPECT_EQ(262144, hostSlots); EXPECT_EQ(66336, deviceSlots); @@ -516,13 +516,13 @@ TEST_F(PeftCacheManagerTest, getMaxNumSlots) config.maxPagesPerBlockDevice = 8; std::tie(hostSlots, deviceSlots) - = PeftCacheManager::getMaxNumSlots(config, tensorrt_llm::DataType::kHALF, 256, 4 * 256, *mManager); + = PeftCacheManager::getMaxNumSlots(config, nvinfer1::DataType::kHALF, 256, 4 * 256, *mManager); EXPECT_EQ(195, hostSlots); EXPECT_EQ(66336, deviceSlots); std::tie(hostSlots, deviceSlots) - = PeftCacheManager::getMaxNumSlots(config, tensorrt_llm::DataType::kFLOAT, 384, 4 * 1024, *mManager); + = PeftCacheManager::getMaxNumSlots(config, nvinfer1::DataType::kFLOAT, 384, 4 * 1024, *mManager); config.hostCacheSize = 100000000; config.numHostModuleLayer = 8291 * 2; @@ -539,7 +539,7 @@ TEST_F(PeftCacheManagerTest, getPageManagerConfig) auto [hostCfg, deviceCfg] = PeftCacheManager::getPageManagerConfig(config, *mModelConfig, *mWorldConfig, *mManager); EXPECT_EQ(runtime::MemoryType::kCPU, hostCfg.getMemoryType()); - EXPECT_EQ(tensorrt_llm::DataType::kFLOAT, hostCfg.getDataType()); + EXPECT_EQ(nvinfer1::DataType::kFLOAT, hostCfg.getDataType()); EXPECT_EQ(456, hostCfg.getTotalNumPages()); EXPECT_EQ(24, hostCfg.getMaxPagesPerBlock()); EXPECT_EQ(288, hostCfg.getSlotsPerPage()); @@ -547,7 +547,7 @@ TEST_F(PeftCacheManagerTest, getPageManagerConfig) EXPECT_FALSE(hostCfg.getInitToZero()); EXPECT_EQ(runtime::MemoryType::kGPU, deviceCfg.getMemoryType()); - EXPECT_EQ(tensorrt_llm::DataType::kFLOAT, deviceCfg.getDataType()); + EXPECT_EQ(nvinfer1::DataType::kFLOAT, deviceCfg.getDataType()); EXPECT_EQ(116, deviceCfg.getTotalNumPages()); EXPECT_EQ(8, deviceCfg.getMaxPagesPerBlock()); EXPECT_EQ(288, deviceCfg.getSlotsPerPage()); @@ -563,7 +563,7 @@ TEST_F(PeftCacheManagerTest, getPageManagerConfig) = PeftCacheManager::getPageManagerConfig(config, *mModelConfig, *mWorldConfig, *mManager); EXPECT_EQ(runtime::MemoryType::kCPU, hostCfg.getMemoryType()); - EXPECT_EQ(tensorrt_llm::DataType::kFLOAT, hostCfg.getDataType()); + EXPECT_EQ(nvinfer1::DataType::kFLOAT, hostCfg.getDataType()); EXPECT_EQ(3617, hostCfg.getTotalNumPages()); EXPECT_EQ(4, hostCfg.getMaxPagesPerBlock()); EXPECT_EQ(288, hostCfg.getSlotsPerPage()); @@ -571,7 +571,7 @@ TEST_F(PeftCacheManagerTest, getPageManagerConfig) EXPECT_FALSE(hostCfg.getInitToZero()); EXPECT_EQ(runtime::MemoryType::kGPU, deviceCfg.getMemoryType()); - EXPECT_EQ(tensorrt_llm::DataType::kFLOAT, deviceCfg.getDataType()); + EXPECT_EQ(nvinfer1::DataType::kFLOAT, deviceCfg.getDataType()); EXPECT_EQ(116, deviceCfg.getTotalNumPages()); EXPECT_EQ(8, deviceCfg.getMaxPagesPerBlock()); EXPECT_EQ(288, deviceCfg.getSlotsPerPage()); @@ -586,7 +586,7 @@ class PeftCacheManagerPrefetchTest : public ::testing::Test // NOLINT(cppcoregui void SetUp() override { - mModelConfig = std::make_unique<ModelConfig>(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + mModelConfig = std::make_unique<ModelConfig>(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); mModelConfig->setMlpHiddenSize(32); mWorldConfig = std::make_unique<WorldConfig>(2, 1, 1, 0); std::vector<LoraModule> modules{ diff --git a/cpp/tests/unit_tests/batch_manager/rnnCacheFormatterTest.cpp b/cpp/tests/unit_tests/batch_manager/rnnCacheFormatterTest.cpp index 7dc4cea6e9af..282475bc007c 100644 --- a/cpp/tests/unit_tests/batch_manager/rnnCacheFormatterTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/rnnCacheFormatterTest.cpp @@ -7,7 +7,6 @@ #include "tensorrt_llm/batch_manager/cacheFormatter.h" #include "tensorrt_llm/batch_manager/rnnCacheFormatter.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include <random> @@ -28,12 +27,12 @@ class RnnTargetIRanksTest : public ::testing::Test std::vector<SizeType32> kvLayersPerPP(pp, 0); // No attention layers auto state = texec::kv_cache::CacheState( /*nbAttentionLayers=*/0, /*nbKvHeads=*/1, /*sizePerHead=*/64, /*tokensPerBlock=*/32, tp, pp, - /*contextParallelism=*/1, kvLayersPerPP, tensorrt_llm::DataType::kFLOAT); + /*contextParallelism=*/1, kvLayersPerPP, nvinfer1::DataType::kFLOAT); texec::kv_cache::CacheState::RnnModelConfig rnnModelConfig{/*mDState=*/16, /*mDConv=*/4, /*mHiddenSize=*/256, /*mHeadDim=*/64, /*mConvDimSize=*/128, /*mNGroups=*/1, /*mNumLayers=*/numLayers, /*mNumHeads=*/4}; - state.setRnnConfig(rnnModelConfig, layersPerPP, tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kFLOAT); + state.setRnnConfig(rnnModelConfig, layersPerPP, nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kFLOAT); return state; } }; @@ -195,7 +194,7 @@ TEST_F(RnnTargetIRanksTest, inquireSupport) // Use reinterpret_cast to pass a non-null dummy pointer (formatter only stores it, doesn't use it in // inquireSupport) - tbm::RnnCacheFormatter formatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter formatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); // Same TP, different PP -> should be supported @@ -217,7 +216,7 @@ class HybridModelCounterpartsTest : public ::testing::Test SizeType32 tokensPerBlock = 32) { return texec::kv_cache::CacheState(numLayers, numHeads, sizePerHead, tokensPerBlock, tp, pp, - /*contextParallelism=*/1, layersPerPP, tensorrt_llm::DataType::kFLOAT, + /*contextParallelism=*/1, layersPerPP, nvinfer1::DataType::kFLOAT, texec::kv_cache::CacheState::AttentionType::kDEFAULT, /*kvFactor=*/2, /*enableAttentionDP=*/false, /*DPrank=*/0, /*DPsize=*/1); } @@ -229,8 +228,7 @@ class HybridModelCounterpartsTest : public ::testing::Test { auto state = makeKvCacheState(kvNumLayers, tp, pp, kvLayersPerPP, numHeads, sizePerHead, tokensPerBlock); texec::kv_cache::CacheState::RnnModelConfig rnnModelConfig{16, 4, 256, 64, 128, 1, rnnNumLayers, 4}; - state.setRnnConfig( - rnnModelConfig, rnnLayersPerPP, tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kFLOAT); + state.setRnnConfig(rnnModelConfig, rnnLayersPerPP, nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kFLOAT); return state; } @@ -274,7 +272,7 @@ TEST_F(HybridModelCounterpartsTest, DifferentPPDistributionKvRnn) auto genState = makeHybridState(/*kvNumLayers=*/10, /*rnnNumLayers=*/6, tp, /*pp=*/2, {5, 5}, {3, 3}); // Use dummy formatter pointers (we only need them to call getCounterparts) - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); // ============= Test from Context rank 0 (PP=0, TP=0) ============= @@ -339,7 +337,7 @@ TEST_F(HybridModelCounterpartsTest, AsymmetricKvRnnDistribution) auto contextState = makeHybridState(/*kvNumLayers=*/8, /*rnnNumLayers=*/4, tp, /*pp=*/1, {8}, {4}); auto genState = makeHybridState(/*kvNumLayers=*/8, /*rnnNumLayers=*/4, tp, /*pp=*/4, {2, 2, 2, 2}, {2, 2, 0, 0}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); // ============= Test from Context rank 0 (PP=0, TP=0) ============= @@ -422,7 +420,7 @@ TEST_F(HybridModelCounterpartsTest, DisjointKvRnnCounterparts) auto genState = makeHybridState( /*kvNumLayers=*/4, /*rnnNumLayers=*/4, tp, /*pp=*/4, {2, 2, 0, 0}, {0, 0, 2, 2}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); // ============= Test from Context rank 0 (PP=0, TP=0) - has KV only ============= @@ -473,7 +471,7 @@ class AttentionOnlyModelTest : public ::testing::Test SizeType32 tokensPerBlock = 32) { return texec::kv_cache::CacheState(numLayers, numHeads, sizePerHead, tokensPerBlock, tp, pp, - /*contextParallelism=*/1, layersPerPP, tensorrt_llm::DataType::kFLOAT, + /*contextParallelism=*/1, layersPerPP, nvinfer1::DataType::kFLOAT, texec::kv_cache::CacheState::AttentionType::kDEFAULT, /*kvFactor=*/2, /*enableAttentionDP=*/false, /*DPrank=*/0, /*DPsize=*/1); } @@ -588,7 +586,7 @@ TEST_F(HybridModelCounterpartsTest, InterleavedLayers) auto contextState = makeHybridState(/*kvNumLayers=*/8, /*rnnNumLayers=*/8, tp, /*pp=*/1, {8}, {8}); auto genState = makeHybridState(/*kvNumLayers=*/8, /*rnnNumLayers=*/8, tp, /*pp=*/2, {4, 4}, {4, 4}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); SizeType32 contextRank0 = 0; @@ -614,7 +612,7 @@ TEST_F(HybridModelCounterpartsTest, RnnMorePPThanKv) auto contextState = makeHybridState(/*kvNumLayers=*/4, /*rnnNumLayers=*/8, tp, /*pp=*/1, {4}, {8}); auto genState = makeHybridState(/*kvNumLayers=*/4, /*rnnNumLayers=*/8, tp, /*pp=*/4, {2, 2, 0, 0}, {2, 2, 2, 2}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); SizeType32 contextRank0 = 0; @@ -641,7 +639,7 @@ TEST_F(HybridModelCounterpartsTest, KvMorePPThanRnn) auto contextState = makeHybridState(/*kvNumLayers=*/8, /*rnnNumLayers=*/4, tp, /*pp=*/1, {8}, {4}); auto genState = makeHybridState(/*kvNumLayers=*/8, /*rnnNumLayers=*/4, tp, /*pp=*/4, {2, 2, 2, 2}, {2, 2, 0, 0}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); SizeType32 contextRank0 = 0; @@ -674,7 +672,7 @@ TEST_F(HybridModelCounterpartsTest, RnnOnlyModel) auto contextState = makeHybridState(/*kvNumLayers=*/0, /*rnnNumLayers=*/8, tp, /*pp=*/1, {0}, {8}); auto genState = makeHybridState(/*kvNumLayers=*/0, /*rnnNumLayers=*/8, tp, /*pp=*/2, {0, 0}, {4, 4}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); SizeType32 contextRank0 = 0; @@ -704,7 +702,7 @@ TEST_F(HybridModelCounterpartsTest, LargeScaleMixedLayers) auto contextState = makeHybridState(/*kvNumLayers=*/32, /*rnnNumLayers=*/16, tp, /*pp=*/1, {32}, {16}); auto genState = makeHybridState(/*kvNumLayers=*/32, /*rnnNumLayers=*/16, tp, /*pp=*/4, {8, 8, 8, 8}, {8, 8, 0, 0}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); // Context rank 0 (PP=0, TP=0) @@ -756,7 +754,7 @@ TEST_F(HybridModelCounterpartsTest, ContextPPGreaterThanGenPP) auto genState = makeHybridState( /*kvNumLayers=*/16, /*rnnNumLayers=*/8, tp, /*pp=*/1, {16}, {8}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); // ============= Test from Context rank 0 (PP=0, TP=0) ============= @@ -814,7 +812,7 @@ TEST_F(HybridModelCounterpartsTest, AsymmetricContextPPGreaterThanGenPP) auto genState = makeHybridState( /*kvNumLayers=*/8, /*rnnNumLayers=*/6, tp, /*pp=*/2, {4, 4}, {3, 3}); - tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::kv_cache_manager::BaseKVCacheManager*>(0x1), + tbm::RnnCacheFormatter rnnFormatter(reinterpret_cast<tbm::rnn_state_manager::RnnStateManager*>(0x1), reinterpret_cast<tbm::rnn_state_manager::RnnCacheTransBufferManager*>(0x2)); // ============= Test from Context rank 0 (PP=0, TP=0) - KV only ============= diff --git a/cpp/tests/unit_tests/batch_manager/truncateBlocksTest.cpp b/cpp/tests/unit_tests/batch_manager/truncateBlocksTest.cpp index 85d4ac112244..d0dce8eb71c5 100644 --- a/cpp/tests/unit_tests/batch_manager/truncateBlocksTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/truncateBlocksTest.cpp @@ -13,7 +13,6 @@ #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/batch_manager/llmRequest.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/samplingConfig.h" #include "tensorrt_llm/testing/kvCacheManagerTestUtil.h" @@ -90,7 +89,7 @@ TEST_F(TruncateBlocksTest, MultiTurnConversationTruncation) // Create KVCacheManager with block reuse enabled KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength /* chunkSize */, true /* enableBlockReuse */); kvCacheManager.allocatePools(false); @@ -220,7 +219,7 @@ TEST_F(TruncateBlocksTest, SharedPrefixTruncation) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength /* chunkSize */, true /* enableBlockReuse */); kvCacheManager.allocatePools(false); @@ -318,7 +317,7 @@ TEST_F(TruncateBlocksTest, CompleteTruncation) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength /* chunkSize */, true /* enableBlockReuse */); kvCacheManager.allocatePools(false); @@ -378,7 +377,7 @@ TEST_F(TruncateBlocksTest, NonExistentTokensTruncation) auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {blocksInPrimaryPool, blocksInSecondaryPool}}}; KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength /* chunkSize */, true /* enableBlockReuse */); kvCacheManager.allocatePools(false); @@ -454,7 +453,7 @@ TEST_F(TruncateBlocksTest, ComplexMultiTurnConversationTruncation) // Create KVCacheManager with block reuse enabled KVCacheManager kvCacheManager(numLayers, numHeads, sizePerHead, tokensPerBlock, blocksPerWindow, maxNumSequences, - beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, tensorrt_llm::DataType::kHALF, 0, stream, + beamWidth, std::vector<BlockManager::SizeType32>{maxAttentionWindow}, nvinfer1::DataType::kHALF, 0, stream, maxSequenceLength, maxSequenceLength /* chunkSize */, true /* enableBlockReuse */); kvCacheManager.allocatePools(false); diff --git a/cpp/tests/unit_tests/common/loggerTest.cpp b/cpp/tests/unit_tests/common/loggerTest.cpp index 6fa28a69b5f8..8aaea6863fb9 100644 --- a/cpp/tests/unit_tests/common/loggerTest.cpp +++ b/cpp/tests/unit_tests/common/loggerTest.cpp @@ -26,7 +26,7 @@ using namespace tensorrt_llm::common; TEST(LoggerModuleTest, FormatModuleNoTrailingSpaces) { for (auto const* raw : {"batch_manager", "common", "cutlass_extensions", "deep_ep", "deep_gemm", "executor", - "flash_mla", "kernels", "layers", "nanobind", "runtime", "testing", "thop"}) + "executor_worker", "flash_mla", "kernels", "layers", "nanobind", "plugins", "runtime", "testing", "thop"}) { auto const fmt = formatModule(raw); EXPECT_FALSE(fmt.empty()); diff --git a/cpp/tests/unit_tests/executor/CMakeLists.txt b/cpp/tests/unit_tests/executor/CMakeLists.txt index 2275683879a4..a51baa6ed00f 100644 --- a/cpp/tests/unit_tests/executor/CMakeLists.txt +++ b/cpp/tests/unit_tests/executor/CMakeLists.txt @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2023-2026 NVIDIA CORPORATION & +# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & # AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); you may not @@ -19,12 +19,21 @@ add_gtest(decodingConfigTest decodingConfigTest.cpp) add_gtest(requestTest requestTest.cpp) add_gtest(responseTest responseTest.cpp) +add_gtest(executorTestSmall executorTestSmall.cpp) +target_link_libraries(executorTestSmall PRIVATE testingUtils) + +add_gtest(executorTestSmallArbitraryOutputTensors + executorTestSmallArbitraryOutputTensors.cpp) +target_link_libraries(executorTestSmallArbitraryOutputTensors + PRIVATE testingUtils) + add_gtest(executorConfigTest executorConfigTest.cpp) add_gtest(executorTensorTest tensorTest.cpp) add_gtest(serializeUtilsTest serializeUtilsTest.cpp) add_gtest(requestWithIdTest requestWithIdTest.cpp) add_gtest(loraConfigTest loraConfigTest.cpp) -add_gtest(coalesceTest coalesceTest.cpp) +add_gtest(intervalSetTest intervalSetTest.cpp) +add_gtest(dynamicBatchTunerTest dynamicBatchTunerTest.cpp) add_gtest(genUniqueAgentNameTest genUniqueAgentNameTest.cpp) target_link_libraries(genUniqueAgentNameTest PRIVATE ${Python3_LIBRARIES}) add_gtest(ucxCommTest ucxCommTest.cpp) @@ -47,6 +56,10 @@ if(NIXL_ROOT OR (MOONCAKE_ROOT AND NOT IS_ROCKY8)) ${Python3_LIBRARIES}) target_compile_definitions(transferAgentTest PRIVATE TEST_NIXL_BACKEND=1) target_compile_definitions(agentCommTest PRIVATE TEST_NIXL_BACKEND=1) + + add_gtest(coalesceTest coalesceTest.cpp) + target_link_libraries(coalesceTest PRIVATE tensorrt_llm_nixl_wrapper + NIXL::nixl) endif() if(MOONCAKE_ROOT) diff --git a/cpp/tests/unit_tests/executor/agentCommTest.cpp b/cpp/tests/unit_tests/executor/agentCommTest.cpp index 89488ba373e8..194d5267c6b3 100644 --- a/cpp/tests/unit_tests/executor/agentCommTest.cpp +++ b/cpp/tests/unit_tests/executor/agentCommTest.cpp @@ -15,7 +15,6 @@ * limitations under the License. */ -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/cache_transmission/agent_utils/connection.h" #include <gtest/gtest.h> @@ -121,7 +120,7 @@ class AgentCommTest : public ::testing::TestWithParam<std::string> auto constexpr blocksInSecondaryPool = 0; auto constexpr enableBlockReuse = true; - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; using BlocksPerWindow = std::map<SizeType32, std::tuple<SizeType32, SizeType32>>; BlocksPerWindow const blocksPerWindow diff --git a/cpp/tests/unit_tests/executor/coalesceTest.cpp b/cpp/tests/unit_tests/executor/coalesceTest.cpp index f115d0fd06db..8675d93fc3b4 100644 --- a/cpp/tests/unit_tests/executor/coalesceTest.cpp +++ b/cpp/tests/unit_tests/executor/coalesceTest.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -15,60 +15,154 @@ * limitations under the License. */ -#include "tensorrt_llm/executor/transferAgent.h" +#include "tensorrt_llm/executor/cache_transmission/nixl_utils/transferAgent.h" #include <gtest/gtest.h> using namespace tensorrt_llm::executor::kv_cache; -namespace +// ==================== coalesceMemoryDescs tests ==================== + +TEST(CoalesceMemoryDescsTest, EmptyInput) +{ + MemoryDescs descs{MemoryType::kVRAM, {}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + EXPECT_EQ(result.getDescs().size(), 0); +} + +TEST(CoalesceMemoryDescsTest, SingleEntry) +{ + MemoryDescs descs{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 1); + EXPECT_EQ(result.getDescs()[0].getAddr(), 0x1000); + EXPECT_EQ(result.getDescs()[0].getLen(), 256); + EXPECT_EQ(result.getDescs()[0].getDeviceId(), 0); +} + +TEST(CoalesceMemoryDescsTest, TwoContiguous) +{ + // [0x1000, 256) then [0x1100, 256) — adjacent, should merge into one + MemoryDescs descs{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 1); + EXPECT_EQ(result.getDescs()[0].getAddr(), 0x1000); + EXPECT_EQ(result.getDescs()[0].getLen(), 512); + EXPECT_EQ(result.getDescs()[0].getDeviceId(), 0); +} + +TEST(CoalesceMemoryDescsTest, TwoNonContiguous) { -VramRegionMap const kEmptyMap; + // Gap between blocks — should stay as two + MemoryDescs descs{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x2000, 256, 0}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 2); + EXPECT_EQ(result.getDescs()[0].getAddr(), 0x1000); + EXPECT_EQ(result.getDescs()[0].getLen(), 256); + EXPECT_EQ(result.getDescs()[1].getAddr(), 0x2000); + EXPECT_EQ(result.getDescs()[1].getLen(), 256); +} -// Merging requires region metadata: an address that misses its region map is never coalesced. -// Tests exercising merge behavior therefore provide maps covering their addresses; a flat -// region (chunkSize=0) imposes no chunk boundaries within it. -VramRegionMap flatRegion(uintptr_t base, size_t len) +TEST(CoalesceMemoryDescsTest, ThreeContiguous) { - VramRegionMap map; - map[base] = {len, 0}; - return map; + MemoryDescs descs{ + MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}, MemoryDesc{0x1200, 256, 0}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 1); + EXPECT_EQ(result.getDescs()[0].getAddr(), 0x1000); + EXPECT_EQ(result.getDescs()[0].getLen(), 768); } -std::pair<MemoryDescs, MemoryDescs> run(TransferDescs const& src, TransferDescs const& dst, - VramRegionMap const& localMap = kEmptyMap, VramRegionMap const& remoteMap = kEmptyMap) +TEST(CoalesceMemoryDescsTest, UnsortedInput) { - return VmmDescSplitter::splitAndCoalesceTransferDescs(src, dst, localMap, remoteMap); + // Same three contiguous blocks but in reverse order — sorting should fix it + MemoryDescs descs{ + MemoryType::kVRAM, {MemoryDesc{0x1200, 256, 0}, MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 1); + EXPECT_EQ(result.getDescs()[0].getAddr(), 0x1000); + EXPECT_EQ(result.getDescs()[0].getLen(), 768); } -} // namespace -// ==================== coalescing within flat regions (chunkSize=0) ==================== +TEST(CoalesceMemoryDescsTest, DifferentDevices) +{ + // Contiguous addresses but different devices — should NOT merge + MemoryDescs descs{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 1}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 2); +} -TEST(SplitAndCoalesceTest, EmptyInput) +TEST(CoalesceMemoryDescsTest, MixedContiguousAndGaps) +{ + // First two are contiguous, then a gap before the third + MemoryDescs descs{ + MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}, MemoryDesc{0x3000, 128, 0}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 2); + EXPECT_EQ(result.getDescs()[0].getAddr(), 0x1000); + EXPECT_EQ(result.getDescs()[0].getLen(), 512); + EXPECT_EQ(result.getDescs()[1].getAddr(), 0x3000); + EXPECT_EQ(result.getDescs()[1].getLen(), 128); +} + +TEST(CoalesceMemoryDescsTest, MultipleDevicesEachContiguous) +{ + // Two contiguous on device 0, two contiguous on device 1 + MemoryDescs descs{MemoryType::kVRAM, + {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}, MemoryDesc{0x2000, 128, 1}, + MemoryDesc{0x2080, 128, 1}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 2); + EXPECT_EQ(result.getDescs()[0].getAddr(), 0x1000); + EXPECT_EQ(result.getDescs()[0].getLen(), 512); + EXPECT_EQ(result.getDescs()[0].getDeviceId(), 0); + EXPECT_EQ(result.getDescs()[1].getAddr(), 0x2000); + EXPECT_EQ(result.getDescs()[1].getLen(), 256); + EXPECT_EQ(result.getDescs()[1].getDeviceId(), 1); +} + +TEST(CoalesceMemoryDescsTest, PreservesMemoryType) +{ + MemoryDescs descs{MemoryType::kDRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + EXPECT_EQ(result.getType(), MemoryType::kDRAM); +} + +TEST(CoalesceMemoryDescsTest, AllSeparate) +{ + // Nothing can be merged — all have gaps + MemoryDescs descs{MemoryType::kVRAM, + {MemoryDesc{0x1000, 100, 0}, MemoryDesc{0x2000, 100, 0}, MemoryDesc{0x3000, 100, 0}, + MemoryDesc{0x4000, 100, 0}}}; + auto result = NixlHelper::coalesceMemoryDescs(descs); + ASSERT_EQ(result.getDescs().size(), 4); +} + +// ==================== coalesceTransferDescs tests ==================== + +TEST(CoalesceTransferDescsTest, EmptyInput) { TransferDescs src{MemoryType::kVRAM, {}}; TransferDescs dst{MemoryType::kVRAM, {}}; - auto [resSrc, resDst] = run(src, dst); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); EXPECT_EQ(resSrc.getDescs().size(), 0); EXPECT_EQ(resDst.getDescs().size(), 0); } -TEST(SplitAndCoalesceTest, SinglePair) +TEST(CoalesceTransferDescsTest, SinglePair) { TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}}}; TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 1); ASSERT_EQ(resDst.getDescs().size(), 1); - EXPECT_EQ(resSrc.getDescs()[0].getAddr(), 0x1000); - EXPECT_EQ(resDst.getDescs()[0].getAddr(), 0x5000); } -TEST(SplitAndCoalesceTest, BothSidesContiguous) +TEST(CoalesceTransferDescsTest, BothSidesContiguous) { - // src contiguous AND dst contiguous within known regions — should merge into one transfer + // src contiguous AND dst contiguous — should merge into one transfer TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}, MemoryDesc{0x5100, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x1000), flatRegion(0x5000, 0x1000)); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 1); ASSERT_EQ(resDst.getDescs().size(), 1); EXPECT_EQ(resSrc.getDescs()[0].getAddr(), 0x1000); @@ -77,79 +171,98 @@ TEST(SplitAndCoalesceTest, BothSidesContiguous) EXPECT_EQ(resDst.getDescs()[0].getLen(), 512); } -TEST(SplitAndCoalesceTest, SrcContiguousDstNot) +TEST(CoalesceTransferDescsTest, SrcContiguousDstNot) { + // src is contiguous but dst has a gap — can't merge TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}, MemoryDesc{0x6000, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x1000), flatRegion(0x5000, 0x2000)); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 2); ASSERT_EQ(resDst.getDescs().size(), 2); } -TEST(SplitAndCoalesceTest, DstContiguousSrcNot) +TEST(CoalesceTransferDescsTest, DstContiguousSrcNot) { + // dst is contiguous but src has a gap — can't merge TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x2000, 256, 0}}}; TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}, MemoryDesc{0x5100, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x2000), flatRegion(0x5000, 0x1000)); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 2); ASSERT_EQ(resDst.getDescs().size(), 2); } -TEST(SplitAndCoalesceTest, DifferentDevicesOnSrc) +TEST(CoalesceTransferDescsTest, DifferentDevicesOnSrc) { + // src addresses look contiguous but are on different devices — can't merge TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 1}}}; TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 0}, MemoryDesc{0x5100, 256, 0}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x1000), flatRegion(0x5000, 0x1000)); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 2); ASSERT_EQ(resDst.getDescs().size(), 2); } -TEST(SplitAndCoalesceTest, DifferentDevicesOnDst) +TEST(CoalesceTransferDescsTest, DifferentDevicesOnDst) { + // dst addresses look contiguous but are on different devices — can't merge TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 0}, MemoryDesc{0x5100, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x1000), flatRegion(0x5000, 0x1000)); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 2); ASSERT_EQ(resDst.getDescs().size(), 2); } -TEST(SplitAndCoalesceTest, ThreePairsAllContiguous) +TEST(CoalesceTransferDescsTest, MismatchedSizes) +{ + // src has 2 entries, dst has 1 — sizes don't match, return as-is + TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; + TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}}}; + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); + ASSERT_EQ(resSrc.getDescs().size(), 2); + ASSERT_EQ(resDst.getDescs().size(), 1); +} + +TEST(CoalesceTransferDescsTest, ThreePairsAllContiguous) { TransferDescs src{ MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}, MemoryDesc{0x1200, 256, 0}}}; TransferDescs dst{ MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}, MemoryDesc{0x5100, 256, 1}, MemoryDesc{0x5200, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x1000), flatRegion(0x5000, 0x1000)); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 1); ASSERT_EQ(resDst.getDescs().size(), 1); EXPECT_EQ(resSrc.getDescs()[0].getLen(), 768); EXPECT_EQ(resDst.getDescs()[0].getLen(), 768); } -TEST(SplitAndCoalesceTest, PartialMerge) +TEST(CoalesceTransferDescsTest, PartialMerge) { - // First two pairs merge; third pair's dst has a gap — stays separate + // First two pairs: both sides contiguous — merge + // Third pair: src contiguous but dst has gap — stays separate TransferDescs src{ MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}, MemoryDesc{0x1200, 256, 0}}}; TransferDescs dst{ MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}, MemoryDesc{0x5100, 256, 1}, MemoryDesc{0x9000, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x1000), flatRegion(0x5000, 0x8000)); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 2); ASSERT_EQ(resDst.getDescs().size(), 2); + // Merged pair EXPECT_EQ(resSrc.getDescs()[0].getAddr(), 0x1000); EXPECT_EQ(resSrc.getDescs()[0].getLen(), 512); EXPECT_EQ(resDst.getDescs()[0].getAddr(), 0x5000); EXPECT_EQ(resDst.getDescs()[0].getLen(), 512); + // Separate pair EXPECT_EQ(resSrc.getDescs()[1].getAddr(), 0x1200); + EXPECT_EQ(resSrc.getDescs()[1].getLen(), 256); EXPECT_EQ(resDst.getDescs()[1].getAddr(), 0x9000); + EXPECT_EQ(resDst.getDescs()[1].getLen(), 256); } -TEST(SplitAndCoalesceTest, UnsortedInput) +TEST(CoalesceTransferDescsTest, UnsortedInput) { - // Same as BothSidesContiguous but in reverse order — sorting by src addr should fix it + // Same as BothSidesContiguous but in reverse order — sorting should fix it TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1100, 256, 0}, MemoryDesc{0x1000, 256, 0}}}; TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5100, 256, 1}, MemoryDesc{0x5000, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x1000), flatRegion(0x5000, 0x1000)); + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); ASSERT_EQ(resSrc.getDescs().size(), 1); ASSERT_EQ(resDst.getDescs().size(), 1); EXPECT_EQ(resSrc.getDescs()[0].getAddr(), 0x1000); @@ -158,259 +271,11 @@ TEST(SplitAndCoalesceTest, UnsortedInput) EXPECT_EQ(resDst.getDescs()[0].getLen(), 512); } -TEST(SplitAndCoalesceTest, NonVramPassthrough) +TEST(CoalesceTransferDescsTest, PreservesMemoryType) { - // Non-kVRAM descs pass through unchanged: no region info exists to bound a merge - TransferDescs src{MemoryType::kDRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; - TransferDescs dst{MemoryType::kDRAM, {MemoryDesc{0x5000, 256, 0}, MemoryDesc{0x5100, 256, 0}}}; - auto [resSrc, resDst] = run(src, dst); - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); + TransferDescs src{MemoryType::kDRAM, {MemoryDesc{0x1000, 256, 0}}}; + TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 0}}}; + auto [resSrc, resDst] = NixlHelper::coalesceTransferDescs(src, dst); EXPECT_EQ(resSrc.getType(), MemoryType::kDRAM); -} - -// ==================== unknown regions never merge ==================== - -TEST(SplitAndCoalesceTest, NoMergeWithoutRegionMetadata) -{ - // Contiguous on both sides, but neither map covers the addresses: two unknown regions are - // indistinguishable (both look up as a miss), so merging is disabled and pairs stay separate. - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}, MemoryDesc{0x5100, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst); - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); -} - -TEST(SplitAndCoalesceTest, NoMergeWhenRemoteRegionUnknown) -{ - // Local map covers src, but the remote side sent no region info (e.g. an older peer): - // dst lookups miss, so nothing merges even though both sides are contiguous. - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}, MemoryDesc{0x5100, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, flatRegion(0x1000, 0x1000)); - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); -} - -TEST(SplitAndCoalesceTest, NoMergeWhenLocalRegionUnknown) -{ - // Remote map covers dst, but src addresses miss the local map: no merging. - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 256, 0}, MemoryDesc{0x1100, 256, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x5000, 256, 1}, MemoryDesc{0x5100, 256, 1}}}; - auto [resSrc, resDst] = run(src, dst, kEmptyMap, flatRegion(0x5000, 0x1000)); - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); -} - -// ==================== chunk-boundary-constrained coalescing ==================== - -TEST(SplitAndCoalesceTest, MergeStopsAtSrcChunkBoundary) -{ - // Local VMM region: base=0x100000, 4MB total, 2MB chunks → boundary at 0x300000. - VramRegionMap localMap; - localMap[0x100000] = {0x400000, 0x200000}; - - // Four contiguous 1MB pairs covering 4MB on both sides; dst is one flat remote region. - std::vector<MemoryDesc> srcVec, dstVec; - for (size_t i = 0; i < 4; ++i) - { - srcVec.emplace_back(0x100000 + i * 0x100000, 0x100000, 0); - dstVec.emplace_back(0x900000 + i * 0x100000, 0x100000, 1); - } - TransferDescs src{MemoryType::kVRAM, srcVec}; - TransferDescs dst{MemoryType::kVRAM, dstVec}; - - auto [resSrc, resDst] = run(src, dst, localMap, flatRegion(0x900000, 0x400000)); - // Merged per src chunk: two 2MB transfers, split exactly at the chunk boundary. - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); - EXPECT_EQ(resSrc.getDescs()[0].getAddr(), 0x100000); - EXPECT_EQ(resSrc.getDescs()[0].getLen(), 0x200000); - EXPECT_EQ(resSrc.getDescs()[1].getAddr(), 0x300000); - EXPECT_EQ(resSrc.getDescs()[1].getLen(), 0x200000); - EXPECT_EQ(resDst.getDescs()[0].getAddr(), 0x900000); - EXPECT_EQ(resDst.getDescs()[0].getLen(), 0x200000); - EXPECT_EQ(resDst.getDescs()[1].getAddr(), 0xB00000); - EXPECT_EQ(resDst.getDescs()[1].getLen(), 0x200000); -} - -TEST(SplitAndCoalesceTest, MergeStopsAtDstChunkBoundary) -{ - // Remote VMM region: base=0x800000, 1MB chunks → boundary at 0x900000. - VramRegionMap remoteMap; - remoteMap[0x800000] = {0x400000, 0x100000}; - - // Two contiguous pairs whose dst junction sits exactly on the remote chunk boundary. - auto localMap = flatRegion(0x1000, 0x100000); - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x1000, 0x80000, 0}, MemoryDesc{0x81000, 0x80000, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x880000, 0x80000, 1}, MemoryDesc{0x900000, 0x80000, 1}}}; - - // With a flat remote region they would merge into one transfer... - { - auto [resSrc, resDst] = run(src, dst, localMap, flatRegion(0x800000, 0x400000)); - ASSERT_EQ(resSrc.getDescs().size(), 1); - ASSERT_EQ(resDst.getDescs().size(), 1); - } - // ...but the dst chunk boundary must block the merge. - auto [resSrc, resDst] = run(src, dst, localMap, remoteMap); - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); - EXPECT_EQ(resDst.getDescs()[0].getAddr(), 0x880000); - EXPECT_EQ(resDst.getDescs()[1].getAddr(), 0x900000); -} - -TEST(SplitAndCoalesceTest, SplitPiecesAreNotRemerged) -{ - // A single pair spanning two src chunks stays split even though the pieces are contiguous. - VramRegionMap localMap; - localMap[0x100000] = {0x400000, 0x100000}; - - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x100000, 0x200000, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x900000, 0x200000, 1}}}; - - auto [resSrc, resDst] = run(src, dst, localMap, flatRegion(0x900000, 0x400000)); - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); - EXPECT_EQ(resSrc.getDescs()[0].getLen(), 0x100000); - EXPECT_EQ(resSrc.getDescs()[1].getLen(), 0x100000); -} - -TEST(SplitAndCoalesceTest, UnalignedRegionBaseBoundary) -{ - // Chunk boundaries are relative to the region base, not absolute alignment. - // base=0x180000, 1MB chunks → boundaries at 0x280000, 0x380000, ... - VramRegionMap localMap; - localMap[0x180000] = {0x300000, 0x100000}; - - // Three contiguous 512KB pairs: first two share the first chunk, third starts a new chunk. - std::vector<MemoryDesc> srcVec, dstVec; - for (size_t i = 0; i < 3; ++i) - { - srcVec.emplace_back(0x180000 + i * 0x80000, 0x80000, 0); - dstVec.emplace_back(0x900000 + i * 0x80000, 0x80000, 1); - } - TransferDescs src{MemoryType::kVRAM, srcVec}; - TransferDescs dst{MemoryType::kVRAM, dstVec}; - - auto [resSrc, resDst] = run(src, dst, localMap, flatRegion(0x900000, 0x300000)); - ASSERT_EQ(resSrc.getDescs().size(), 2); - EXPECT_EQ(resSrc.getDescs()[0].getAddr(), 0x180000); - EXPECT_EQ(resSrc.getDescs()[0].getLen(), 0x100000); - EXPECT_EQ(resSrc.getDescs()[1].getAddr(), 0x280000); - EXPECT_EQ(resSrc.getDescs()[1].getLen(), 0x80000); -} - -TEST(SplitAndCoalesceTest, NoMergeAcrossRegions) -{ - // Two VA-adjacent but distinct local regions (cudaMalloc-style, chunkSize=0): - // contiguous descs must not merge across the region boundary. - VramRegionMap localMap; - localMap[0x100000] = {0x100000, 0}; - localMap[0x200000] = {0x100000, 0}; - - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x100000, 0x100000, 0}, MemoryDesc{0x200000, 0x100000, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x900000, 0x100000, 1}, MemoryDesc{0xA00000, 0x100000, 1}}}; - - auto [resSrc, resDst] = run(src, dst, localMap, flatRegion(0x900000, 0x200000)); - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); -} - -TEST(SplitAndCoalesceTest, MergeWithinSingleRegion) -{ - // Control for NoMergeAcrossRegions: same layout as one region (chunkSize=0) merges freely. - VramRegionMap localMap; - localMap[0x100000] = {0x200000, 0}; - - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x100000, 0x100000, 0}, MemoryDesc{0x200000, 0x100000, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x900000, 0x100000, 1}, MemoryDesc{0xA00000, 0x100000, 1}}}; - - auto [resSrc, resDst] = run(src, dst, localMap, flatRegion(0x900000, 0x200000)); - ASSERT_EQ(resSrc.getDescs().size(), 1); - EXPECT_EQ(resSrc.getDescs()[0].getLen(), 0x200000); - EXPECT_EQ(resDst.getDescs()[0].getLen(), 0x200000); -} - -TEST(SplitAndCoalesceTest, NoMergeAcrossRemoteRegions) -{ - // The remote side registered two discrete but VA-adjacent buffers (chunkSize=0 each): - // contiguous dst descs must not merge across the remote registration boundary. - VramRegionMap remoteMap; - remoteMap[0x900000] = {0x100000, 0}; - remoteMap[0xA00000] = {0x100000, 0}; - - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x100000, 0x100000, 0}, MemoryDesc{0x200000, 0x100000, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x900000, 0x100000, 1}, MemoryDesc{0xA00000, 0x100000, 1}}}; - - auto [resSrc, resDst] = run(src, dst, flatRegion(0x100000, 0x200000), remoteMap); - ASSERT_EQ(resSrc.getDescs().size(), 2); - ASSERT_EQ(resDst.getDescs().size(), 2); -} - -TEST(SplitAndCoalesceTest, CoalesceDisabledSplitsOnly) -{ - // With coalescing disabled, contiguous pairs stay separate and chunk splitting still applies. - VramRegionMap localMap; - localMap[0x100000] = {0x400000, 0x100000}; - - // Two contiguous 512KB pairs within one chunk plus one pair spanning a chunk boundary. - TransferDescs src{MemoryType::kVRAM, - {MemoryDesc{0x100000, 0x80000, 0}, MemoryDesc{0x180000, 0x80000, 0}, MemoryDesc{0x200000, 0x200000, 0}}}; - TransferDescs dst{MemoryType::kVRAM, - {MemoryDesc{0x900000, 0x80000, 1}, MemoryDesc{0x980000, 0x80000, 1}, MemoryDesc{0xA00000, 0x200000, 1}}}; - - auto [resSrc, resDst] - = VmmDescSplitter::splitAndCoalesceTransferDescs(src, dst, localMap, kEmptyMap, /*enableCoalesce=*/false); - // No merging: pair 1, pair 2, and pair 3 split into two chunk pieces → 4 descs. - ASSERT_EQ(resSrc.getDescs().size(), 4); - ASSERT_EQ(resDst.getDescs().size(), 4); - EXPECT_EQ(resSrc.getDescs()[0].getLen(), 0x80000); - EXPECT_EQ(resSrc.getDescs()[1].getLen(), 0x80000); - EXPECT_EQ(resSrc.getDescs()[2].getLen(), 0x100000); - EXPECT_EQ(resSrc.getDescs()[3].getLen(), 0x100000); -} - -TEST(SplitAndCoalesceTest, CoalesceDisabledPreservesInputOrder) -{ - // With coalescing disabled there is no sorting either: descs come out in input order, - // matching the historical split-only behavior. - TransferDescs src{MemoryType::kVRAM, {MemoryDesc{0x2000, 256, 0}, MemoryDesc{0x1000, 256, 0}}}; - TransferDescs dst{MemoryType::kVRAM, {MemoryDesc{0x6000, 256, 1}, MemoryDesc{0x5000, 256, 1}}}; - - auto [resSrc, resDst] - = VmmDescSplitter::splitAndCoalesceTransferDescs(src, dst, kEmptyMap, kEmptyMap, /*enableCoalesce=*/false); - ASSERT_EQ(resSrc.getDescs().size(), 2); - EXPECT_EQ(resSrc.getDescs()[0].getAddr(), 0x2000); - EXPECT_EQ(resSrc.getDescs()[1].getAddr(), 0x1000); - EXPECT_EQ(resDst.getDescs()[0].getAddr(), 0x6000); - EXPECT_EQ(resDst.getDescs()[1].getAddr(), 0x5000); -} - -TEST(SplitAndCoalesceTest, SplitAndMergeScatteredBlocks) -{ - // Scattered input order + chunked src region: blocks are sorted, merged per chunk. - // base=0x100000, 1MB chunks; four 512KB blocks given out of order. - VramRegionMap localMap; - localMap[0x100000] = {0x400000, 0x100000}; - - std::vector<size_t> perm{2, 0, 3, 1}; - std::vector<MemoryDesc> srcVec, dstVec; - for (size_t i : perm) - { - srcVec.emplace_back(0x100000 + i * 0x80000, 0x80000, 0); - dstVec.emplace_back(0x900000 + i * 0x80000, 0x80000, 1); - } - TransferDescs src{MemoryType::kVRAM, srcVec}; - TransferDescs dst{MemoryType::kVRAM, dstVec}; - - auto [resSrc, resDst] = run(src, dst, localMap, flatRegion(0x900000, 0x400000)); - // 2MB of contiguous data over two 1MB chunks → one transfer per chunk. - ASSERT_EQ(resSrc.getDescs().size(), 2); - EXPECT_EQ(resSrc.getDescs()[0].getAddr(), 0x100000); - EXPECT_EQ(resSrc.getDescs()[0].getLen(), 0x100000); - EXPECT_EQ(resSrc.getDescs()[1].getAddr(), 0x200000); - EXPECT_EQ(resSrc.getDescs()[1].getLen(), 0x100000); - EXPECT_EQ(resDst.getDescs()[0].getAddr(), 0x900000); - EXPECT_EQ(resDst.getDescs()[1].getAddr(), 0xA00000); + EXPECT_EQ(resDst.getType(), MemoryType::kVRAM); } diff --git a/cpp/tests/unit_tests/executor/dynamicBatchTunerTest.cpp b/cpp/tests/unit_tests/executor/dynamicBatchTunerTest.cpp new file mode 100644 index 000000000000..04ce10393b67 --- /dev/null +++ b/cpp/tests/unit_tests/executor/dynamicBatchTunerTest.cpp @@ -0,0 +1,99 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/executor/dynamicBatchTuner.h" +#include "tensorrt_llm/common/tllmException.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include <gmock/gmock.h> +#include <gtest/gtest.h> + +using ::testing::_; +using ::testing::Invoke; + +using namespace tensorrt_llm::executor; +using namespace tensorrt_llm::common; + +TEST(DynamicBatchTunerTest, Stats) +{ + // moving average window size is 3 + DynamicBatchConfig dynamicBatchConfig(true, true, 3); + DynamicBatchTuner dynamicBatchTuner(dynamicBatchConfig); + + // check no division by zero issue + EXPECT_EQ(dynamicBatchTuner.getAverageInputLength(), 0); + EXPECT_EQ(dynamicBatchTuner.getAverageOutputLength(), 0); + + dynamicBatchTuner.updateStats(1, 2); + EXPECT_EQ(dynamicBatchTuner.getAverageInputLength(), 1); + EXPECT_EQ(dynamicBatchTuner.getAverageOutputLength(), 2); + + dynamicBatchTuner.updateStats(2, 3); + EXPECT_EQ(dynamicBatchTuner.getAverageInputLength(), 1.5); + EXPECT_EQ(dynamicBatchTuner.getAverageOutputLength(), 2.5); + + dynamicBatchTuner.updateStats(3, 4); + EXPECT_EQ(dynamicBatchTuner.getAverageInputLength(), 2); + EXPECT_EQ(dynamicBatchTuner.getAverageOutputLength(), 3); + + // check that the first element is removed from the moving average window + dynamicBatchTuner.updateStats(4, 5); + EXPECT_EQ(dynamicBatchTuner.getAverageInputLength(), 3); + EXPECT_EQ(dynamicBatchTuner.getAverageOutputLength(), 4); +} + +TEST(DynamicBatchConfig, RuntimeBatchSize) +{ + // moving average window size is 3 + DynamicBatchConfig dynamicBatchConfig(true, true, 3); + DynamicBatchTuner dynamicBatchTuner(dynamicBatchConfig); + // check runtime batch size computation + EXPECT_EQ(dynamicBatchTuner.getRuntimeBatchSize(143), 128); + EXPECT_EQ(dynamicBatchTuner.getRuntimeBatchSize(335), 256); + EXPECT_EQ(dynamicBatchTuner.getRuntimeBatchSize(671), 512); + EXPECT_EQ(dynamicBatchTuner.getRuntimeBatchSize(831), 768); + EXPECT_EQ(dynamicBatchTuner.getRuntimeBatchSize(1279), 1024); + EXPECT_EQ(dynamicBatchTuner.getRuntimeBatchSize(1663), 1536); + // fall back + EXPECT_EQ(dynamicBatchTuner.getRuntimeBatchSize(2049), 2048); + EXPECT_EQ(dynamicBatchTuner.getRuntimeBatchSize(1665), 1665); +} + +TEST(DynamicBatchConfig, RuntimeMaxNumTokens) +{ + // moving average window size is 1 + DynamicBatchConfig dynamicBatchConfig(true, true, 1); + DynamicBatchTuner dynamicBatchTuner(dynamicBatchConfig); + + // context heavy + dynamicBatchTuner.updateStats(100, 2); + EXPECT_EQ(dynamicBatchTuner.getRuntimeMaxNumTokens(1), 8192); + // context heavy fall back + EXPECT_EQ(dynamicBatchTuner.getRuntimeMaxNumTokens(256), 16384); + + // balanced + dynamicBatchTuner.updateStats(100, 100); + EXPECT_EQ(dynamicBatchTuner.getRuntimeMaxNumTokens(1), 4096); + // balanced fall back + EXPECT_EQ(dynamicBatchTuner.getRuntimeMaxNumTokens(4000), 8192); + + // gen heavy + dynamicBatchTuner.updateStats(2, 256); + EXPECT_EQ(dynamicBatchTuner.getRuntimeMaxNumTokens(1), 2048); + // gen heavy fall back + EXPECT_EQ(dynamicBatchTuner.getRuntimeMaxNumTokens(4000), 4096); +} diff --git a/cpp/tests/unit_tests/executor/executorTestSmall.cpp b/cpp/tests/unit_tests/executor/executorTestSmall.cpp new file mode 100644 index 000000000000..2987509f16ac --- /dev/null +++ b/cpp/tests/unit_tests/executor/executorTestSmall.cpp @@ -0,0 +1,289 @@ +#include "tensorrt_llm/batch_manager/trtGptModelInflightBatching.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tests/utils/common.h" +#include "tests/utils/engines.h" +#include "tests/utils/executorUtils.h" + +#include "gtest/gtest.h" + +#include <random> +#include <tuple> +#include <unordered_map> + +namespace tensorrt_llm::testing +{ + +struct TrivialConstantDecoderTestParameters +{ + using TupleT = std::tuple<runtime::SizeType32, runtime::SizeType32, runtime::SizeType32, runtime::SizeType32, + runtime::SizeType32, runtime::SizeType32, runtime::SizeType32, runtime::SizeType32>; + runtime::SizeType32 randomSeed; + runtime::SizeType32 vocabSize; + runtime::SizeType32 maxNumTokens; + runtime::SizeType32 maxBeamWidth; + runtime::SizeType32 maxBatchSize; + runtime::SizeType32 numRequests; + runtime::SizeType32 promptLength; + runtime::SizeType32 maxOutputLength; + + // Constructor that takes a tuple + TrivialConstantDecoderTestParameters( // NOLINT: implicit to allow gtest to convert from tuple generated by + // 'combine' + TupleT t) + : randomSeed(std::get<0>(t)) + , vocabSize(std::get<1>(t)) + , maxNumTokens(std::get<2>(t)) + , maxBeamWidth(std::get<3>(t)) + , maxBatchSize(std::get<4>(t)) + , numRequests(std::get<5>(t)) + , promptLength(std::get<6>(t)) + , maxOutputLength(std::get<7>(t)) + { + } +}; + +template <typename TLogits> +struct DecoderTestShared +{ + static constexpr runtime::SizeType32 kNumTokensPerBlock = 64; + static constexpr runtime::SizeType32 kKvCacheMaxTokens = 2048 * 8; + + DecoderTestShared(std::shared_ptr<runtime::TllmLogger> logger, std::mt19937 rng, + std::shared_ptr<executor::Executor> executor, std::vector<TLogits> randomLogits) + : logger(std::move(logger)) + , rng(rng) + , executor(std::move(executor)) + , randomLogits(std::move(randomLogits)){}; + std::shared_ptr<runtime::TllmLogger> logger; + std::mt19937 rng; + std::shared_ptr<executor::Executor> executor; + std::vector<TLogits> randomLogits; +}; + +template <typename TLogits> +std::unique_ptr<DecoderTestShared<TLogits>> SetupDecoderTest(TrivialConstantDecoderTestParameters const& params) +{ + auto logger = std::make_shared<runtime::TllmLogger>(); + auto rng = std::mt19937(params.randomSeed); + auto randomLogits = tensorrt_llm::testing::randomLogits<std::mt19937, TLogits>(params.vocabSize, &rng); + auto const decoderParameters = tensorrt_llm::testing::utils::engines::ConstantTrivialDecoderParameters<TLogits>{ + tensorrt_llm::testing::utils::engines::TrivialDecoderParameters{params.vocabSize, params.maxBatchSize, + params.maxNumTokens, DecoderTestShared<TLogits>::kNumTokensPerBlock, params.maxBeamWidth, false}, + randomLogits}; + auto engineHostMemory + = tensorrt_llm::testing::utils::engines::createConstantTrivialDecoder<TLogits>(decoderParameters, logger); + auto const engine = runtime::RawEngine(engineHostMemory.release()); + auto const dtype = runtime::TRTDataType<TLogits>::value; + auto modelConfig = runtime::ModelConfig(params.vocabSize, 1, 1, 0, 1, 1, dtype); + modelConfig.useGptAttentionPlugin(true); + modelConfig.setModelVariant(runtime::ModelConfig::ModelVariant::kGpt); + modelConfig.usePackedInput(true); + modelConfig.setKVCacheType(runtime::ModelConfig::KVCacheType::kPAGED); + modelConfig.setMaxNumTokens(params.maxNumTokens); + modelConfig.setMaxBatchSize(params.maxBatchSize); + modelConfig.setMaxBeamWidth(params.maxBeamWidth); + modelConfig.setMaxSequenceLen(params.maxNumTokens); + modelConfig.setMaxInputLen(params.maxNumTokens); + modelConfig.setLayerTypes({runtime::ModelConfig::LayerType::kATTENTION}); + modelConfig.setTokensPerBlock(DecoderTestShared<TLogits>::kNumTokensPerBlock); + modelConfig.setPagedContextFMHA(true); + + auto const worldConfig = runtime::WorldConfig(); + auto kvCacheConfig = executor::KvCacheConfig{}; + kvCacheConfig.setMaxTokens(DecoderTestShared<TLogits>::kKvCacheMaxTokens); + + auto const executorConfig + = tensorrt_llm::executor::ExecutorConfig(params.maxBeamWidth, executor::SchedulerConfig(), kvCacheConfig, true, + true, 1, 1, executor::BatchingType::kINFLIGHT, params.maxBatchSize, params.maxNumTokens, std::nullopt, + std::nullopt, std::nullopt, std::nullopt, false, 1, std::nullopt, executor::ExtendedRuntimePerfKnobConfig(), + std::nullopt, 0, executor::ExecutorConfig::kDefaultMaxSeqIdleMicroseconds, std::nullopt, std::nullopt); + + auto model = std::make_shared<batch_manager::TrtGptModelInflightBatching>( + logger, modelConfig, worldConfig, engine, false, executorConfig, false); + + return std::make_unique<DecoderTestShared<TLogits>>( + logger, rng, std::make_shared<executor::Executor>(model, executorConfig), randomLogits); +} + +template <typename TLogits> +class DecoderTest : public ::testing::Test, public ::testing::WithParamInterface<TrivialConstantDecoderTestParameters> +{ +protected: + std::unique_ptr<DecoderTestShared<TLogits>> state; + + DecoderTest() + { + auto const params = GetParam(); + state = SetupDecoderTest<TLogits>(params); + } + + void runDecoderTest(TrivialConstantDecoderTestParameters const& parameters) + { + auto const requestTokens = createConsecutiveTokenSequence(parameters.promptLength, parameters.vocabSize, 0); + auto requests = std::vector<executor::Request>{}; + requests.reserve(static_cast<std::size_t>(parameters.numRequests)); + for (auto i = 0; i < parameters.numRequests; i++) + { + requests.emplace_back(requestTokens, parameters.maxOutputLength, false, executor::SamplingConfig{}, + executor::OutputConfig{false, false, false, true, false, false}); + } + auto const accumulatedResponses + = runThroughRequests(*state->executor, requests, std::chrono::duration<float, std::milli>(3600000)); + ASSERT_EQ(accumulatedResponses.size(), parameters.numRequests); + + std::sort(state->randomLogits.begin(), state->randomLogits.end()); + std::reverse(state->randomLogits.begin(), state->randomLogits.end()); + for (auto const& [requestId, responses] : accumulatedResponses) + { + for (auto const& response : responses) + { + ASSERT_FALSE(response.hasError()); + auto const& tokensByBeam = response.getResult().outputTokenIds; + ASSERT_EQ(tokensByBeam.size(), 1); + for (auto const& tokensForBeam : tokensByBeam) + { + ASSERT_EQ(tokensForBeam.size(), parameters.maxOutputLength); + } + } + } + } +}; + +namespace +{ +constexpr runtime::SizeType32 kRandomSeed1 = 45; +auto const randomSeeds = ::testing::Values(kRandomSeed1); + +constexpr runtime::SizeType32 kMinVocabSize = 16; +auto const vocabSizes = ::testing::Values(kMinVocabSize); + +constexpr runtime::SizeType32 kMinMaxNumTokens = 2048; +auto const maxNumTokenses = ::testing::Values(kMinMaxNumTokens); + +constexpr runtime::SizeType32 kMinBeamWidth = 1; +auto const beamWidths = ::testing::Values(kMinBeamWidth); + +constexpr runtime::SizeType32 kMinMaxBatchSize = 2048; +auto const maxBatchSizes = ::testing::Values(kMinMaxBatchSize); + +constexpr runtime::SizeType32 kMinNumRequests = 64; +auto const numRequestses = ::testing::Values(kMinNumRequests); + +constexpr runtime::SizeType32 kMinPromptLength = 32; +auto const promptLengths = ::testing::Values(kMinPromptLength); + +constexpr runtime::SizeType32 kMinMaxOutputLength = 16; +auto const maxOutputLengths = ::testing::Values(kMinMaxOutputLength); + +auto const paramGenerator + = ::testing::ConvertGenerator<TrivialConstantDecoderTestParameters::TupleT>(::testing::Combine(randomSeeds, + vocabSizes, maxNumTokenses, beamWidths, maxBatchSizes, numRequestses, promptLengths, maxOutputLengths)); +} // namespace + +using DecoderFloatTest = DecoderTest<float>; + +TEST_P(DecoderFloatTest, TestSizeAndValues) +{ + runDecoderTest(GetParam()); +} + +INSTANTIATE_TEST_SUITE_P(Float, DecoderFloatTest, paramGenerator, + [](::testing::TestParamInfo<TrivialConstantDecoderTestParameters> const& info) -> std::string + { + std::stringstream nameStringStream; + nameStringStream << "_maxBatchSize_" << info.param.maxBatchSize << "_vocabSize_" << info.param.vocabSize + << "_maxBeamWidth_" << info.param.maxBeamWidth << "_maxNumTokens_" << info.param.maxNumTokens + << "_maxOutputLength_" << info.param.maxOutputLength << "_numRequests_" + << info.param.numRequests << "_promptLength_" << info.param.promptLength << "_randomSeed_" + << info.param.randomSeed; + return nameStringStream.str(); + }); + +// Helper function to test calculateCacheSizePerToken with given parameters. +std::map<runtime::SizeType32, runtime::SizeType32> calculateCacheSizePerTokenHelper( + std::vector<runtime::SizeType32> const& maxAttentionWindowVec, runtime::SizeType32 kvFactor = 2, + runtime::SizeType32 vocabSize = 32, runtime::SizeType32 nbLayers = 4, runtime::SizeType32 nbAttentionLayers = 4, + runtime::SizeType32 nbRnnLayers = 0, runtime::SizeType32 nbHeads = 8, runtime::SizeType32 hiddenSize = 512, + bool isCrossAttention = false) +{ + // Create minimal ModelConfig for testing. + auto modelConfig = runtime::ModelConfig( + vocabSize, nbLayers, nbAttentionLayers, nbRnnLayers, nbHeads, hiddenSize, nvinfer1::DataType::kFLOAT); + modelConfig.useGptAttentionPlugin(true); + modelConfig.setModelVariant(runtime::ModelConfig::ModelVariant::kGpt); + modelConfig.setKVCacheType(runtime::ModelConfig::KVCacheType::kPAGED); + + auto const worldConfig = runtime::WorldConfig(); + + return batch_manager::TrtGptModelInflightBatching::calculateCacheSizePerTokenForDisagg( + modelConfig, worldConfig, maxAttentionWindowVec, isCrossAttention, kvFactor); +} + +// Test for TrtGptModelInflightBatching::calculateCacheSizePerToken function with different layer types. +TEST(TrtInflightBatchingTest, CalculateCacheSizePerTokenForDisagg) +{ + // Common parameters. + constexpr runtime::SizeType32 nbLayers = 5; + constexpr runtime::SizeType32 hiddenSize = 512; + constexpr runtime::SizeType32 kvFactor = 2; + constexpr runtime::SizeType32 vocabSize = 32; + constexpr runtime::SizeType32 nbHeads = 8; + // Test case 1: Single attention window size - attention layers only. + { + std::vector<runtime::SizeType32> maxAttentionWindowVec = {128}; + constexpr runtime::SizeType32 nbAttentionLayers = 5; + constexpr runtime::SizeType32 numBytesPerFloatElement = 4; + constexpr runtime::SizeType32 nbRnnLayers = 0; + auto result = calculateCacheSizePerTokenHelper(maxAttentionWindowVec, kvFactor, vocabSize, nbLayers, + nbAttentionLayers, nbRnnLayers, nbHeads, hiddenSize, false); + EXPECT_EQ(result.size(), 1); + EXPECT_EQ(result.at(128), nbAttentionLayers * kvFactor * hiddenSize * numBytesPerFloatElement); + } + + // Test case 2: Multiple attention window sizes - attention layers only. + { + std::vector<runtime::SizeType32> maxAttentionWindowVec = {128, 256}; + constexpr runtime::SizeType32 nbAttentionLayers = 5; + constexpr runtime::SizeType32 numBytesPerFloatElement = 4; + constexpr runtime::SizeType32 nbRnnLayers = 0; + auto result = calculateCacheSizePerTokenHelper(maxAttentionWindowVec, kvFactor, vocabSize, nbLayers, + nbAttentionLayers, nbRnnLayers, nbHeads, hiddenSize, false); + EXPECT_EQ(result.size(), 2); + auto const nbAttentionLayersIn128Window = 3; + auto const nbAttentionLayersIn256Window = 2; + EXPECT_EQ(result.at(128), nbAttentionLayersIn128Window * kvFactor * hiddenSize * numBytesPerFloatElement); + EXPECT_EQ(result.at(256), nbAttentionLayersIn256Window * kvFactor * hiddenSize * numBytesPerFloatElement); + } + + // Test case 3: Single attention window size - attention and rnn layers. + { + std::vector<runtime::SizeType32> maxAttentionWindowVec = {128}; + constexpr runtime::SizeType32 nbAttentionLayers = 3; + constexpr runtime::SizeType32 numBytesPerFloatElement = 4; + constexpr runtime::SizeType32 nbRnnLayers = 2; + auto result = calculateCacheSizePerTokenHelper(maxAttentionWindowVec, kvFactor, vocabSize, nbLayers, + nbAttentionLayers, nbRnnLayers, nbHeads, hiddenSize, false); + EXPECT_EQ(result.size(), 1); + EXPECT_EQ(result.at(128), nbAttentionLayers * kvFactor * hiddenSize * numBytesPerFloatElement); + } + + // Test case 4: Multiple attention window sizes - attention and rnn layers. + { + std::vector<runtime::SizeType32> maxAttentionWindowVec = {128, 256}; + constexpr runtime::SizeType32 nbAttentionLayers = 3; + constexpr runtime::SizeType32 numBytesPerFloatElement = 4; + constexpr runtime::SizeType32 nbRnnLayers = 2; + auto result = calculateCacheSizePerTokenHelper(maxAttentionWindowVec, kvFactor, vocabSize, nbLayers, + nbAttentionLayers, nbRnnLayers, nbHeads, hiddenSize, false); + EXPECT_EQ(result.size(), 2); + auto const nbAttentionLayersIn128Window = 2; + auto const nbAttentionLayersIn256Window = 1; + EXPECT_EQ(result.at(128), nbAttentionLayersIn128Window * kvFactor * hiddenSize * numBytesPerFloatElement); + EXPECT_EQ(result.at(256), nbAttentionLayersIn256Window * kvFactor * hiddenSize * numBytesPerFloatElement); + } +} + +} // namespace tensorrt_llm::testing diff --git a/cpp/tests/unit_tests/executor/executorTestSmallArbitraryOutputTensors.cpp b/cpp/tests/unit_tests/executor/executorTestSmallArbitraryOutputTensors.cpp new file mode 100644 index 000000000000..b64bd775fe30 --- /dev/null +++ b/cpp/tests/unit_tests/executor/executorTestSmallArbitraryOutputTensors.cpp @@ -0,0 +1,491 @@ +#include "include/tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/batch_manager/trtGptModelInflightBatching.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/modelConfig.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/worldConfig.h" +#include "tests/utils/common.h" +#include "tests/utils/engines.h" +#include "tests/utils/executorUtils.h" + +#include "gtest/gtest.h" +#include <NvInfer.h> +#include <NvInferRuntime.h> +#include <NvInferRuntimeBase.h> +#include <gmock/gmock.h> + +#include <algorithm> +#include <chrono> +#include <cstddef> +#include <memory> +#include <optional> +#include <random> +#include <ratio> +#include <utility> +#include <vector> + +namespace tensorrt_llm::testing +{ + +struct TrivialConstantDecoderWithTopKLogitsTestParameters +{ + using TupleT = std::tuple<runtime::SizeType32, runtime::SizeType32, runtime::SizeType32, runtime::SizeType32, + runtime::SizeType32, runtime::SizeType32, runtime::SizeType32, runtime::SizeType32, runtime::SizeType32, bool>; + runtime::SizeType32 randomSeed; + runtime::SizeType32 vocabSize; + runtime::SizeType32 maxNumTokens; + runtime::SizeType32 maxBeamWidth; + runtime::SizeType32 maxBatchSize; + runtime::SizeType32 numTopKLogits; + runtime::SizeType32 numRequests; + runtime::SizeType32 promptLength; + runtime::SizeType32 maxOutputLength; + bool gatherContext; + + // Constructor that takes a tuple + TrivialConstantDecoderWithTopKLogitsTestParameters( // NOLINT: implicit to allow gtest to convert from tuple + // generated by 'combine' + TupleT t) + : randomSeed(std::get<0>(t)) + , vocabSize(std::get<1>(t)) + , maxNumTokens(std::get<2>(t)) + , maxBeamWidth(std::get<3>(t)) + , maxBatchSize(std::get<4>(t)) + , numTopKLogits(std::get<5>(t)) + , numRequests(std::get<6>(t)) + , promptLength(std::get<7>(t)) + , maxOutputLength(std::get<8>(t)) + , gatherContext(std::get<9>(t)) + { + } +}; + +template <typename TLogits> +struct DecoderTestShared +{ + static constexpr runtime::SizeType32 kNumTokensPerBlock = 64; + static constexpr runtime::SizeType32 kKvCacheMaxTokens = 2048 * 8; + static constexpr auto kTopKTensorName = "topKLogits"; + + DecoderTestShared(std::shared_ptr<runtime::TllmLogger> logger, std::mt19937 rng, + std::shared_ptr<executor::Executor> executor, std::vector<TLogits> randomLogits) + : logger(std::move(logger)) + , rng(rng) + , executor(std::move(executor)) + , randomLogits(std::move(randomLogits)){}; + std::shared_ptr<runtime::TllmLogger> logger; + std::mt19937 rng; + std::shared_ptr<executor::Executor> executor; + std::vector<TLogits> randomLogits; +}; + +template <typename TLogits> +std::unique_ptr<DecoderTestShared<TLogits>> SetupDecoderTest( + TrivialConstantDecoderWithTopKLogitsTestParameters const& params) +{ + auto logger = std::make_shared<runtime::TllmLogger>(); + auto rng = std::mt19937(params.randomSeed); + auto randomLogits = tensorrt_llm::testing::randomLogits<std::mt19937, TLogits>(params.vocabSize, &rng); + auto const decoderParameters = tensorrt_llm::testing::utils::engines::ConstantTrivialDecoderParameters<TLogits>{ + tensorrt_llm::testing::utils::engines::TrivialDecoderParameters{params.vocabSize, params.maxBatchSize, + params.maxNumTokens, DecoderTestShared<TLogits>::kNumTokensPerBlock, params.maxBeamWidth, + params.gatherContext}, + randomLogits}; + auto engineHostMemory = tensorrt_llm::testing::utils::engines::createConstantTrivialDecoderWithTopKLogits<TLogits>( + decoderParameters, params.numTopKLogits, DecoderTestShared<TLogits>::kTopKTensorName, logger); + auto const engine = runtime::RawEngine(engineHostMemory.release()); + + auto const dtype = runtime::TRTDataType<TLogits>::value; + auto modelConfig = runtime::ModelConfig(params.vocabSize, 1, 1, 0, 1, 1, dtype); + modelConfig.useGptAttentionPlugin(true); + modelConfig.setModelVariant(runtime::ModelConfig::ModelVariant::kGpt); + modelConfig.usePackedInput(true); + modelConfig.setKVCacheType(runtime::ModelConfig::KVCacheType::kPAGED); + modelConfig.setMaxNumTokens(params.maxNumTokens); + modelConfig.setMaxBatchSize(params.maxBatchSize); + modelConfig.setMaxBeamWidth(params.maxBeamWidth); + modelConfig.setMaxSequenceLen(params.maxNumTokens); + modelConfig.setMaxInputLen(params.maxNumTokens); + modelConfig.setLayerTypes({runtime::ModelConfig::LayerType::kATTENTION}); + modelConfig.setTokensPerBlock(DecoderTestShared<TLogits>::kNumTokensPerBlock); + modelConfig.setPagedContextFMHA(true); + modelConfig.computeContextLogits(params.gatherContext); + + auto const worldConfig = runtime::WorldConfig(); + + auto kvCacheConfig = executor::KvCacheConfig{}; + kvCacheConfig.setMaxTokens(DecoderTestShared<TLogits>::kKvCacheMaxTokens); + + auto const executorConfig + = executor::ExecutorConfig(params.maxBeamWidth, executor::SchedulerConfig(), kvCacheConfig, true, true, 1, 1, + executor::BatchingType::kINFLIGHT, params.maxBatchSize, params.maxNumTokens, std::nullopt, std::nullopt, + std::nullopt, std::nullopt, false, 1, std::nullopt, executor::ExtendedRuntimePerfKnobConfig(), std::nullopt, + 0, executor::ExecutorConfig::kDefaultMaxSeqIdleMicroseconds, std::nullopt, std::nullopt, + std::vector<executor::AdditionalModelOutput>{ + executor::AdditionalModelOutput{DecoderTestShared<TLogits>::kTopKTensorName, params.gatherContext}}); + + auto model = std::make_shared<batch_manager::TrtGptModelInflightBatching>( + logger, modelConfig, worldConfig, engine, false, executorConfig, false); + + return std::make_unique<DecoderTestShared<TLogits>>( + logger, rng, std::make_shared<executor::Executor>(model, executorConfig), randomLogits); +} + +template <typename TLogits> +class DecoderTopKGenerationLogitsTest + : public ::testing::Test, + public ::testing::WithParamInterface<TrivialConstantDecoderWithTopKLogitsTestParameters> +{ +protected: + std::unique_ptr<DecoderTestShared<TLogits>> state; + + DecoderTopKGenerationLogitsTest() + { + auto const params = GetParam(); + state = SetupDecoderTest<TLogits>(params); + } + + void runTopKGenerationLogitsTest(TrivialConstantDecoderWithTopKLogitsTestParameters const& parameters) + { + auto const requestTokens = createConsecutiveTokenSequence(parameters.promptLength, parameters.vocabSize, 0); + auto requests = std::vector<executor::Request>{}; + requests.reserve(static_cast<std::size_t>(parameters.numRequests)); + for (auto i = 0; i < parameters.numRequests; i++) + { + std::vector<executor::AdditionalModelOutput> additionalOutputs{ + executor::AdditionalModelOutput{DecoderTestShared<TLogits>::kTopKTensorName}}; + requests.emplace_back(requestTokens, parameters.maxOutputLength, false, executor::SamplingConfig{}, + executor::OutputConfig{false, false, false, true, false, false, additionalOutputs}); + } + auto const accumulatedResponses + = runThroughRequests(*state->executor, requests, std::chrono::duration<float, std::milli>(100000)); + ASSERT_EQ(accumulatedResponses.size(), parameters.numRequests); + + std::sort(state->randomLogits.begin(), state->randomLogits.end()); + std::reverse(state->randomLogits.begin(), state->randomLogits.end()); + for (auto const& [requestId, responses] : accumulatedResponses) + { + for (auto const& response : responses) + { + ASSERT_FALSE(response.hasError()); + auto const& tokensByBeam = response.getResult().outputTokenIds; + auto const& additionalOutputs = response.getResult().additionalOutputs; + ASSERT_EQ(additionalOutputs.size(), 1); + auto const& topKLogits = additionalOutputs.front(); + auto const expectedOutputSize = parameters.maxOutputLength * parameters.numTopKLogits; + ASSERT_EQ(topKLogits.output.getSize(), expectedOutputSize); + auto const* topKLogitsData = reinterpret_cast<TLogits const*>(topKLogits.output.getData()); + for (auto i = 0; i < parameters.numTopKLogits; i++) + { + EXPECT_TRUE(almostEqual(topKLogitsData[i], state->randomLogits[i], 1e-5)) + << "requestId " << requestId << " i " << i << ": " << topKLogitsData[i] + << " != " << state->randomLogits[i]; + } + ASSERT_EQ(tokensByBeam.size(), 1); + for (auto const& tokensForBeam : tokensByBeam) + { + ASSERT_EQ(tokensForBeam.size(), parameters.maxOutputLength); + } + } + } + } +}; + +template <typename TLogits> +class DecoderTopKGenerationLogitsStreamingTest + : public ::testing::Test, + public ::testing::WithParamInterface<TrivialConstantDecoderWithTopKLogitsTestParameters> +{ +protected: + std::unique_ptr<DecoderTestShared<TLogits>> state; + + DecoderTopKGenerationLogitsStreamingTest() + { + auto const params = GetParam(); + state = SetupDecoderTest<TLogits>(params); + } + + void runTopKGenerationLogitsStreamingTest(TrivialConstantDecoderWithTopKLogitsTestParameters const& parameters) + { + auto const requestTokens = createConsecutiveTokenSequence(parameters.promptLength, parameters.vocabSize, 0); + auto requests = std::vector<executor::Request>{}; + requests.reserve(static_cast<std::size_t>(parameters.numRequests)); + for (auto i = 0; i < parameters.numRequests; i++) + { + std::vector<executor::AdditionalModelOutput> additionalOutputs{ + executor::AdditionalModelOutput{DecoderTestShared<TLogits>::kTopKTensorName}}; + requests.emplace_back(requestTokens, parameters.maxOutputLength, true, executor::SamplingConfig{}, + executor::OutputConfig{false, false, false, true, false, false, additionalOutputs}); + } + auto const accumulatedResponses + = runThroughRequests(*state->executor, requests, std::chrono::duration<float, std::milli>(100000)); + ASSERT_EQ(accumulatedResponses.size(), parameters.numRequests); + + std::sort(state->randomLogits.begin(), state->randomLogits.end()); + std::reverse(state->randomLogits.begin(), state->randomLogits.end()); + for (auto const& idResponsesKvp : accumulatedResponses) + { + auto const& [requestId, responses] = idResponsesKvp; + auto numTokensForRequest = 0; + for (auto const& response : responses) + { + ASSERT_FALSE(response.hasError()); + auto const& tokensByBeam = response.getResult().outputTokenIds; + auto const& additionalOutputs = response.getResult().additionalOutputs; + ASSERT_EQ(additionalOutputs.size(), 1); + auto const& topKLogits = additionalOutputs.front(); + auto const expectedOutputSize = parameters.maxOutputLength * parameters.numTopKLogits; + ASSERT_EQ(topKLogits.output.getSize(), expectedOutputSize); + auto const* topKLogitsData = reinterpret_cast<TLogits const*>(topKLogits.output.getData()); + for (auto i = 0; i < parameters.numTopKLogits; i++) + { + EXPECT_TRUE(almostEqual(topKLogitsData[i], state->randomLogits[i], 1e-5)) + << "requestId " << requestId << " i " << i << ": " << topKLogitsData[i] + << " != " << state->randomLogits[i]; + } + ASSERT_EQ(tokensByBeam.size(), 1); + for (auto const& tokensForBeam : tokensByBeam) + { + numTokensForRequest += tokensForBeam.size(); + } + } + ASSERT_EQ(numTokensForRequest, parameters.maxOutputLength); + } + } +}; + +template <typename TLogits> +class DecoderTopKContextLogitsStreamingTest + : public ::testing::Test, + public ::testing::WithParamInterface<TrivialConstantDecoderWithTopKLogitsTestParameters> +{ +protected: + std::unique_ptr<DecoderTestShared<TLogits>> state; + + DecoderTopKContextLogitsStreamingTest() + { + auto const params = GetParam(); + state = SetupDecoderTest<TLogits>(params); + } + + void runTopKContextLogitsTest(TrivialConstantDecoderWithTopKLogitsTestParameters const& parameters) + { + auto requests = std::vector<executor::Request>{}; + requests.reserve(static_cast<std::size_t>(parameters.numRequests)); + for (auto i = 0; i < parameters.numRequests; i++) + { + // create different sequence for each request to avoid KV cache reuse + auto const requestTokens = createConsecutiveTokenSequence(parameters.promptLength, parameters.vocabSize, i); + std::vector<executor::AdditionalModelOutput> additionalOutputs{ + executor::AdditionalModelOutput{DecoderTestShared<TLogits>::kTopKTensorName, true}}; + requests.emplace_back(requestTokens, parameters.maxOutputLength, true, executor::SamplingConfig{}, + executor::OutputConfig{false, false, false, true, false, false, additionalOutputs}); + } + auto const& accumulatedResponses + = runThroughRequests(*state->executor, requests, std::chrono::duration<float, std::milli>(100000)); + ASSERT_EQ(accumulatedResponses.size(), parameters.numRequests); + + std::sort(state->randomLogits.begin(), state->randomLogits.end()); + std::reverse(state->randomLogits.begin(), state->randomLogits.end()); + std::string const expectedAdditionalOutputName + = std::string("context_") + DecoderTestShared<TLogits>::kTopKTensorName; + for (auto const& idResponsesKvp : accumulatedResponses) + { + auto const& [requestId, responses] = idResponsesKvp; + std::size_t numTokensForRequest{0}; + for (auto const& response : responses) + { + ASSERT_FALSE(response.hasError()); + auto const& tokensByBeam = response.getResult().outputTokenIds; + auto const& additionalOutputs = response.getResult().additionalOutputs; + ASSERT_EQ(additionalOutputs.size(), 2); + auto const contextTopKLogitsPtr = std::find_if(additionalOutputs.cbegin(), additionalOutputs.cend(), + [&expectedAdditionalOutputName](auto const& ao) + { return ao.name == expectedAdditionalOutputName; }); + auto const expectedOutputSize = parameters.promptLength * parameters.numTopKLogits; + ASSERT_EQ(contextTopKLogitsPtr->output.getSize(), expectedOutputSize); + auto const* topKLogitsData = reinterpret_cast<TLogits const*>(contextTopKLogitsPtr->output.getData()); + for (auto i = 0; i < parameters.numTopKLogits; i++) + { + EXPECT_TRUE(almostEqual(topKLogitsData[i], state->randomLogits[i], 1e-5)) + << "requestId " << requestId << " i " << i << ": " << topKLogitsData[i] + << " != " << state->randomLogits[i]; + } + ASSERT_EQ(tokensByBeam.size(), 1); + for (auto const& tokensForBeam : tokensByBeam) + { + numTokensForRequest += static_cast<std::size_t>(tokensForBeam.size()); + } + } + ASSERT_EQ(numTokensForRequest, parameters.maxOutputLength); + } + } +}; + +template <typename TLogits> +class DecoderTopKContextLogitsTest + : public ::testing::Test, + public ::testing::WithParamInterface<TrivialConstantDecoderWithTopKLogitsTestParameters> +{ +protected: + std::unique_ptr<DecoderTestShared<TLogits>> state; + + DecoderTopKContextLogitsTest() + { + auto const params = GetParam(); + state = SetupDecoderTest<TLogits>(params); + } + + void runTopKContextLogitsTest(TrivialConstantDecoderWithTopKLogitsTestParameters const& parameters) + { + auto requests = std::vector<executor::Request>{}; + requests.reserve(static_cast<std::size_t>(parameters.numRequests)); + for (auto i = 0; i < parameters.numRequests; i++) + { + // create different sequence for each request to avoid KV cache reuse + auto const requestTokens = createConsecutiveTokenSequence(parameters.promptLength, parameters.vocabSize, i); + std::vector<executor::AdditionalModelOutput> additionalOutputs{ + executor::AdditionalModelOutput{DecoderTestShared<TLogits>::kTopKTensorName, true}}; + requests.emplace_back(requestTokens, parameters.maxOutputLength, false, executor::SamplingConfig{}, + executor::OutputConfig{false, false, false, true, false, false, additionalOutputs}); + } + auto const accumulatedResponses + = runThroughRequests(*state->executor, requests, std::chrono::duration<float, std::milli>(100000)); + ASSERT_EQ(accumulatedResponses.size(), parameters.numRequests); + + std::sort(state->randomLogits.begin(), state->randomLogits.end()); + std::reverse(state->randomLogits.begin(), state->randomLogits.end()); + std::string const expectedAdditionalOutputName + = std::string("context_") + DecoderTestShared<TLogits>::kTopKTensorName; + for (auto const& idResponsesKvp : accumulatedResponses) + { + auto const& [requestId, responses] = idResponsesKvp; + for (auto const& response : responses) + { + ASSERT_FALSE(response.hasError()); + auto const& tokensByBeam = response.getResult().outputTokenIds; + auto const& additionalOutputs = response.getResult().additionalOutputs; + ASSERT_EQ(additionalOutputs.size(), 2); + auto const contextTopKLogitsPtr = std::find_if(additionalOutputs.cbegin(), additionalOutputs.cend(), + [&expectedAdditionalOutputName](auto const& ao) + { return ao.name == expectedAdditionalOutputName; }); + auto const expectedOutputSize = parameters.promptLength * parameters.numTopKLogits; + ASSERT_EQ(contextTopKLogitsPtr->output.getSize(), expectedOutputSize); + auto const* topKLogitsData = reinterpret_cast<TLogits const*>(contextTopKLogitsPtr->output.getData()); + for (auto i = 0; i < parameters.numTopKLogits; i++) + { + EXPECT_TRUE(almostEqual(topKLogitsData[i], state->randomLogits[i], 1e-5)) + << "requestId " << requestId << " i " << i << ": " << topKLogitsData[i] + << " != " << state->randomLogits[i]; + } + ASSERT_EQ(tokensByBeam.size(), 1); + for (auto const& tokensForBeam : tokensByBeam) + { + ASSERT_EQ(tokensForBeam.size(), parameters.maxOutputLength); + } + } + } + } +}; + +namespace +{ +constexpr runtime::SizeType32 kRandomSeed1 = 45; +auto const randomSeeds = ::testing::Values(kRandomSeed1); + +constexpr runtime::SizeType32 kMinVocabSize = 64; +constexpr runtime::SizeType32 kMaxVocabSize = 2048; +auto const vocabSizes = ::testing::Values(kMinVocabSize); + +constexpr runtime::SizeType32 kMinMaxNumTokens = 2048; +auto const maxNumTokenses = ::testing::Values(kMinMaxNumTokens); + +constexpr runtime::SizeType32 kMinBeamWidth = 1; +auto const beamWidths = ::testing::Values(kMinBeamWidth); + +constexpr runtime::SizeType32 kMinMaxBatchSize = 2048; +auto const batchSizes = ::testing::Values(kMinMaxBatchSize); + +constexpr runtime::SizeType32 kMinNumTopKLogits = 4; +constexpr runtime::SizeType32 kMaxNumTopKLogits = 32; +auto const numTopKLogitses = ::testing::Values(kMinNumTopKLogits, kMaxNumTopKLogits); + +constexpr runtime::SizeType32 kMinNumRequests = 16; +constexpr runtime::SizeType32 kMaxNumRequests = 2048; +auto const numRequestses = ::testing::Values(kMinNumRequests); + +constexpr runtime::SizeType32 kMinPromptLength = 4; +constexpr runtime::SizeType32 kMaxPromptLength = 512; +auto const promptLengths = ::testing::Values(kMinPromptLength, kMaxPromptLength); + +constexpr runtime::SizeType32 kMinMaxOutputLength = 4; +constexpr runtime::SizeType32 kMaxMaxOutputLength = 256; +auto const maxOutputLengths = ::testing::Values(kMinMaxOutputLength, kMaxMaxOutputLength); + +auto const gatherContext = ::testing::Values(false, true); +auto const alwaysGatherContext = ::testing::Values(true); + +auto const paramGenerator = ::testing::ConvertGenerator<TrivialConstantDecoderWithTopKLogitsTestParameters::TupleT>( + ::testing::Combine(randomSeeds, vocabSizes, maxNumTokenses, beamWidths, batchSizes, numTopKLogitses, numRequestses, + promptLengths, maxOutputLengths, gatherContext)); + +auto const paramGeneratorGatherContext + = ::testing::ConvertGenerator<TrivialConstantDecoderWithTopKLogitsTestParameters::TupleT>( + ::testing::Combine(randomSeeds, vocabSizes, maxNumTokenses, beamWidths, batchSizes, numTopKLogitses, + numRequestses, promptLengths, maxOutputLengths, alwaysGatherContext)); + +auto const nameSuffixGenerator + = [](::testing::TestParamInfo<TrivialConstantDecoderWithTopKLogitsTestParameters> const& info) -> std::string +{ + std::stringstream nameStringStream; + nameStringStream << "gatherContext_" << info.param.gatherContext << "_maxBatchSize_" << info.param.maxBatchSize + << "_vocabSize_" << info.param.vocabSize << "_maxBeamWidth_" << info.param.maxBeamWidth + << "_maxNumTokens_" << info.param.maxNumTokens << "_maxOutputLength_" << info.param.maxOutputLength + << "_numRequests_" << info.param.numRequests << "_numTopKLogits_" << info.param.numTopKLogits + << "_promptLength_" << info.param.promptLength << "_randomSeed_" << info.param.randomSeed; + return nameStringStream.str(); +}; + +} // namespace + +using DecoderTopKGenerationLogitsFloatTest = DecoderTopKGenerationLogitsTest<float>; + +TEST_P(DecoderTopKGenerationLogitsFloatTest, TestSizeAndValues) +{ + runTopKGenerationLogitsTest(GetParam()); +} + +INSTANTIATE_TEST_SUITE_P(Float, DecoderTopKGenerationLogitsFloatTest, paramGenerator, nameSuffixGenerator); + +using DecoderTopKGenerationLogitsStreamingFloatTest = DecoderTopKGenerationLogitsStreamingTest<float>; + +TEST_P(DecoderTopKGenerationLogitsStreamingFloatTest, TestSizeAndValues) +{ + runTopKGenerationLogitsStreamingTest(GetParam()); +} + +INSTANTIATE_TEST_SUITE_P(Float, DecoderTopKGenerationLogitsStreamingFloatTest, paramGenerator, nameSuffixGenerator); + +using DecoderTopKContextLogitsStreamingFloatTest = DecoderTopKContextLogitsStreamingTest<float>; + +TEST_P(DecoderTopKContextLogitsStreamingFloatTest, TestSizeAndValues) +{ + runTopKContextLogitsTest(GetParam()); +} + +INSTANTIATE_TEST_SUITE_P( + Float, DecoderTopKContextLogitsStreamingFloatTest, paramGeneratorGatherContext, nameSuffixGenerator); + +using DecoderTopKContextLogitsFloatTest = DecoderTopKContextLogitsTest<float>; + +TEST_P(DecoderTopKContextLogitsFloatTest, TestSizeAndValues) +{ + runTopKContextLogitsTest(GetParam()); +} + +INSTANTIATE_TEST_SUITE_P(Float, DecoderTopKContextLogitsFloatTest, paramGeneratorGatherContext, nameSuffixGenerator); + +} // namespace tensorrt_llm::testing diff --git a/cpp/tests/unit_tests/executor/intervalSetTest.cpp b/cpp/tests/unit_tests/executor/intervalSetTest.cpp new file mode 100644 index 000000000000..a2bb0a8f7532 --- /dev/null +++ b/cpp/tests/unit_tests/executor/intervalSetTest.cpp @@ -0,0 +1,224 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/executor/intervalSet.h" + +#include <gtest/gtest.h> + +using tensorrt_llm::executor::IntervalSet; +using tensorrt_llm::executor::IdType; + +class IntervalSetTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type-member-init) +{ +protected: + void SetUp() override {} + + void TearDown() override {} + + IntervalSet<IdType> mIntervalSet; +}; + +namespace +{ + +TEST_F(IntervalSetTest, testPublicAPI) +{ + EXPECT_FALSE(mIntervalSet.contains(0)); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); + mIntervalSet.insert(0); + mIntervalSet.insert(1); + mIntervalSet.insert(4); + mIntervalSet.insert(6); + EXPECT_TRUE(mIntervalSet.contains(0)); + EXPECT_TRUE(mIntervalSet.contains(4)); + EXPECT_FALSE(mIntervalSet.contains(2125)); + EXPECT_EQ(mIntervalSet.getNumElements(), 4); + mIntervalSet.insert(6); + mIntervalSet.insert(4); + mIntervalSet.insert(1); + mIntervalSet.insert(0); + EXPECT_EQ(mIntervalSet.getNumElements(), 4); + EXPECT_TRUE(mIntervalSet.contains(0)); + EXPECT_TRUE(mIntervalSet.contains(4)); + EXPECT_FALSE(mIntervalSet.contains(9)); + EXPECT_FALSE(mIntervalSet.contains(3)); + EXPECT_FALSE(mIntervalSet.contains(11)); + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); +} + +TEST_F(IntervalSetTest, testClear) +{ + for (int i = 0; i < 100; i++) + { + if (i % 2 == 0) + { + EXPECT_FALSE(mIntervalSet.contains(i)); + mIntervalSet.insert(i); + EXPECT_TRUE(mIntervalSet.contains(i)); + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); + } + EXPECT_FALSE(mIntervalSet.contains(i)); + } + for (int i = 0; i < 100; i++) + { + EXPECT_FALSE(mIntervalSet.contains(i)); + } + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); + + for (int i = 19; i >= 10; i--) + { + if (i % 2 == 0) + { + EXPECT_FALSE(mIntervalSet.contains(i)); + mIntervalSet.insert(i); + EXPECT_TRUE(mIntervalSet.contains(i)); + } + else + { + EXPECT_FALSE(mIntervalSet.contains(i)); + } + } + EXPECT_EQ(mIntervalSet.getNumElements(), 5); + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); +} + +TEST_F(IntervalSetTest, testRandomInsert) +{ + mIntervalSet.insert(4); + mIntervalSet.insert(8); + EXPECT_EQ(mIntervalSet.getNumElements(), 2); + std::vector<int> idToAdd{9, 7, 5, 1, 6, 0, 2}; + for (auto id : idToAdd) + { + mIntervalSet.insert(id); + } + for (int i = 0; i < 10; i++) + { + if (i != 3) + { + EXPECT_TRUE(mIntervalSet.contains(i)); + } + else + { + EXPECT_FALSE(mIntervalSet.contains(i)); + } + } + EXPECT_EQ(mIntervalSet.getNumElements(), 9); + mIntervalSet.insert(3); + for (int i = 0; i < 10; i++) + { + EXPECT_TRUE(mIntervalSet.contains(i)); + } + EXPECT_EQ(mIntervalSet.getNumElements(), 10); + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); +} + +TEST_F(IntervalSetTest, testTerminatedReqIdIntervals) +{ + mIntervalSet.insert(4); + mIntervalSet.insert(8); + EXPECT_EQ(mIntervalSet.getNumElements(), 2); + // terminatedReqIdIntervals is [[4, 4], [8, 8]] + EXPECT_EQ(mIntervalSet.getIntervals().size(), 2); + mIntervalSet.insert(3); + mIntervalSet.insert(5); + // terminatedReqIdIntervals is [[3, 5], [8, 8]] + EXPECT_EQ(mIntervalSet.getNumElements(), 4); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 2); + mIntervalSet.insert(9); + mIntervalSet.insert(7); + // terminatedReqIdIntervals is [[3, 5], [7, 9]] + EXPECT_EQ(mIntervalSet.getNumElements(), 6); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 2); + mIntervalSet.insert(6); + // terminatedReqIdIntervals is [[3, 9]] + EXPECT_EQ(mIntervalSet.getNumElements(), 7); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 1); + mIntervalSet.insert(1); + // terminatedReqIdIntervals is [[1, 1], [3, 9]] + EXPECT_EQ(mIntervalSet.getNumElements(), 8); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 2); + mIntervalSet.insert(0); + // terminatedReqIdIntervals is [[0, 1], [3, 9]] + EXPECT_EQ(mIntervalSet.getNumElements(), 9); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 2); + mIntervalSet.insert(2); + // terminatedReqIdIntervals is [[0, 9]] + EXPECT_EQ(mIntervalSet.getNumElements(), 10); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 1); + for (int i = 0; i < 10; i++) + { + mIntervalSet.insert(i); + // terminatedReqIdIntervals is always [[0, 9]] + EXPECT_EQ(mIntervalSet.getNumElements(), 10); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 1); + } + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 0); + + // Insert continuous decreasing numbers. Interval size is always one. + for (int i = 19; i >= 10; i--) + { + mIntervalSet.insert(i); + EXPECT_TRUE(mIntervalSet.contains(i)); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 1); + } + mIntervalSet.clear(); + EXPECT_EQ(mIntervalSet.getNumElements(), 0); + EXPECT_EQ(mIntervalSet.getIntervals().size(), 0); + + // Insert 50 disjoint even numbers + for (int i = 0; i < 100; i++) + { + if (i % 2 == 0) + { + mIntervalSet.insert(i); + EXPECT_EQ(mIntervalSet.getNumElements(), (i / 2) + 1); + EXPECT_EQ(mIntervalSet.getIntervals().size(), (i / 2) + 1); + } + } + + // Insert 50 disjoint odd numbers. Interval size should go down as the intervals are merged. + for (int i = 0; i < 100; i++) + { + if (i % 2 != 0) + { + mIntervalSet.insert(i); + EXPECT_EQ(mIntervalSet.getNumElements(), 50 + (i + 1) / 2); + if (i != 99) + { + EXPECT_EQ(mIntervalSet.getIntervals().size(), 50 - (i + 1) / 2); + } + else + { + EXPECT_EQ(mIntervalSet.getIntervals().size(), 1); + } + } + } +} + +} // namespace diff --git a/cpp/tests/unit_tests/executor/requestTest.cpp b/cpp/tests/unit_tests/executor/requestTest.cpp index ec43d088657f..f44f55b5141d 100644 --- a/cpp/tests/unit_tests/executor/requestTest.cpp +++ b/cpp/tests/unit_tests/executor/requestTest.cpp @@ -26,9 +26,7 @@ using ::testing::_; using ::testing::Invoke; using namespace tensorrt_llm::executor; -// Not a namespace-wide import: common also exports DataType/Dims, which would -// make the unqualified DataType (= executor::DataType) below ambiguous. -using tensorrt_llm::common::TllmException; +using namespace tensorrt_llm::common; TEST(RequestTest, validInputs) { diff --git a/cpp/tests/unit_tests/executor/serializeUtilsTest.cpp b/cpp/tests/unit_tests/executor/serializeUtilsTest.cpp index 3c78659c87c4..a39756bf7243 100644 --- a/cpp/tests/unit_tests/executor/serializeUtilsTest.cpp +++ b/cpp/tests/unit_tests/executor/serializeUtilsTest.cpp @@ -18,7 +18,6 @@ #include "tensorrt_llm/executor/serializeUtils.h" #include "tensorrt_llm/batch_manager/kvCacheManager.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/cache_transmission/agent_utils/connection.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/executor/executor.h" @@ -752,8 +751,7 @@ TEST(SerializeUtilsTest, ContextPhaseParams) { auto state = std::make_unique<texec::DataTransceiverState>(); state->setCommState(texec::kv_cache::CommState{12, "127.0.0.1"}); - state->setCacheState( - texec::kv_cache::CacheState{10, 12, 128, 128, 8, 8, 8, {4}, tensorrt_llm::DataType::kFLOAT}); + state->setCacheState(texec::kv_cache::CacheState{10, 12, 128, 128, 8, 8, 8, {4}, nvinfer1::DataType::kFLOAT}); auto stats = texec::ContextPhaseParams({10, 20, 30, 40, 50, 60}, 0, state.release(), VecTokens{10, 20}); auto stats2 = serializeDeserialize(stats); EXPECT_EQ(stats, stats2); @@ -1555,7 +1553,7 @@ TEST(SerializeUtilsTest, CacheStateIndexerKCache) texec::SizeType32 pp = 1; texec::SizeType32 cp = 1; std::vector<texec::SizeType32> attentionLayerNumPerPP{static_cast<texec::SizeType32>(nbKvHeadsPerLayer.size())}; - auto dataType = tensorrt_llm::DataType::kFLOAT; + auto dataType = nvinfer1::DataType::kFLOAT; auto attentionType = CacheState::AttentionType::kDEFAULT; int kvFactor = 2; bool enableAttentionDP = false; diff --git a/cpp/tests/unit_tests/executor/transferAgentTest.cpp b/cpp/tests/unit_tests/executor/transferAgentTest.cpp index dcf6dd5a6e1e..b915ec3bb9c4 100644 --- a/cpp/tests/unit_tests/executor/transferAgentTest.cpp +++ b/cpp/tests/unit_tests/executor/transferAgentTest.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2023-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -702,7 +702,8 @@ TEST(VmmDescSplitterTest, SplitTransferDescsDifferentChunkSizes) MemoryDescs srcInput{MemoryType::kVRAM, srcDescs}; MemoryDescs dstInput{MemoryType::kVRAM, dstDescs}; - auto [splitSrc, splitDst] = VmmDescSplitter::splitAndCoalesceTransferDescs(srcInput, dstInput, localMap, remoteMap); + auto [splitSrc, splitDst] + = VmmDescSplitter::splitTransferDescsWithRegionMaps(srcInput, dstInput, localMap, remoteMap); // dst has smaller chunks (512KB), so we get 4 pieces: 512K, 512K, 512K, 512K ASSERT_EQ(splitSrc.getDescs().size(), 4); @@ -734,7 +735,8 @@ TEST(VmmDescSplitterTest, SplitTransferDescsUnalignedBothSides) MemoryDescs srcInput{MemoryType::kVRAM, srcDescs}; MemoryDescs dstInput{MemoryType::kVRAM, dstDescs}; - auto [splitSrc, splitDst] = VmmDescSplitter::splitAndCoalesceTransferDescs(srcInput, dstInput, localMap, remoteMap); + auto [splitSrc, splitDst] + = VmmDescSplitter::splitTransferDescsWithRegionMaps(srcInput, dstInput, localMap, remoteMap); // src has 4MB chunks starting at srcBase → 1 piece from src side // dst has 2MB chunks starting at dstBase → 2 pieces from dst side @@ -758,7 +760,7 @@ TEST(VmmDescSplitterTest, SplitTransferDescsNoDstRegion) MemoryDescs dstInput{MemoryType::kVRAM, dstDescs}; auto [splitSrc, splitDst] - = VmmDescSplitter::splitAndCoalesceTransferDescs(srcInput, dstInput, localMap, emptyRemoteMap); + = VmmDescSplitter::splitTransferDescsWithRegionMaps(srcInput, dstInput, localMap, emptyRemoteMap); // Only src boundaries: 2 pieces of 1MB each ASSERT_EQ(splitSrc.getDescs().size(), 2); diff --git a/cpp/tests/unit_tests/executor/ucxCommTest.cpp b/cpp/tests/unit_tests/executor/ucxCommTest.cpp index 5d2bfc6e772f..51d1d84d2676 100644 --- a/cpp/tests/unit_tests/executor/ucxCommTest.cpp +++ b/cpp/tests/unit_tests/executor/ucxCommTest.cpp @@ -36,7 +36,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/envUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/cache_transmission/mpi_utils/connection.h" #include "tensorrt_llm/executor/dataTransceiverState.h" #include "tensorrt_llm/executor/executor.h" @@ -133,11 +132,11 @@ TEST_F(UcxCommTest, Basic) tensorrt_llm::runtime::BufferManager bufferManager{std::make_shared<tensorrt_llm::runtime::CudaStream>()}; // Create and fill source CUDA buffer with random data - auto srcBuffer = bufferManager.gpu(buffer.size(), tensorrt_llm::DataType::kINT8); + auto srcBuffer = bufferManager.gpu(buffer.size(), nvinfer1::DataType::kINT8); bufferManager.copy(buffer.data(), *srcBuffer); bufferManager.getStream().synchronize(); - auto dstBuffer = bufferManager.gpu(buffer.size(), tensorrt_llm::DataType::kINT8); + auto dstBuffer = bufferManager.gpu(buffer.size(), nvinfer1::DataType::kINT8); // Send CUDA buffer using connection1 connection1->send(DataContext{0x75}, srcBuffer->data(), srcBuffer->getSizeInBytes()); @@ -205,14 +204,14 @@ TEST_F(UcxCommTest, multiSend) tensorrt_llm::runtime::BufferManager bufferManager{std::make_shared<tensorrt_llm::runtime::CudaStream>()}; - auto srcBuffer1 = bufferManager.gpu(buffer1.size(), tensorrt_llm::DataType::kINT8); - auto srcBuffer2 = bufferManager.gpu(buffer2.size(), tensorrt_llm::DataType::kINT8); + auto srcBuffer1 = bufferManager.gpu(buffer1.size(), nvinfer1::DataType::kINT8); + auto srcBuffer2 = bufferManager.gpu(buffer2.size(), nvinfer1::DataType::kINT8); bufferManager.copy(buffer1.data(), *srcBuffer1); bufferManager.copy(buffer2.data(), *srcBuffer2); bufferManager.getStream().synchronize(); - auto dstBuffer1 = bufferManager.gpu(buffer1.size(), tensorrt_llm::DataType::kINT8); - auto dstBuffer2 = bufferManager.gpu(buffer2.size(), tensorrt_llm::DataType::kINT8); + auto dstBuffer1 = bufferManager.gpu(buffer1.size(), nvinfer1::DataType::kINT8); + auto dstBuffer2 = bufferManager.gpu(buffer2.size(), nvinfer1::DataType::kINT8); connection1Peer->send(DataContext{0x75}, srcBuffer1->data(), srcBuffer1->getSizeInBytes()); connection2Peer->send(DataContext{0x75}, srcBuffer2->data(), srcBuffer2->getSizeInBytes()); diff --git a/cpp/tests/unit_tests/kernels/CMakeLists.txt b/cpp/tests/unit_tests/kernels/CMakeLists.txt index 5f121d2406a9..e593ce4b76b8 100644 --- a/cpp/tests/unit_tests/kernels/CMakeLists.txt +++ b/cpp/tests/unit_tests/kernels/CMakeLists.txt @@ -1,4 +1,4 @@ -# SPDX-FileCopyrightText: Copyright (c) 2023-2026 NVIDIA CORPORATION & +# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & # AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); you may not @@ -35,14 +35,9 @@ if(USING_OSS_CUTLASS_MOE_GEMM) target_compile_definitions(mixtureOfExpertsTest PUBLIC USING_OSS_CUTLASS_MOE_GEMM) - # The internal-path variant includes headers (quantization.h) that only ship - # with the internal cutlass kernels sources; the prebuilt tarball provides the - # library and a reduced header set only. - if(INTERNAL_CUTLASS_KERNELS_PATH) - add_gtest(mixtureOfExpertsInternalTest mixtureOfExpertsTest.cu) - remove_compile_definition(mixtureOfExpertsInternalTest - USING_OSS_CUTLASS_MOE_GEMM) - endif() + add_gtest(mixtureOfExpertsInternalTest mixtureOfExpertsTest.cu) + remove_compile_definition(mixtureOfExpertsInternalTest + USING_OSS_CUTLASS_MOE_GEMM) endif() add_gtest(ropeTest ropeTest.cu) diff --git a/cpp/tests/unit_tests/kernels/banRepeatNGramsKernelsTest.cpp b/cpp/tests/unit_tests/kernels/banRepeatNGramsKernelsTest.cpp index 83e94ea2d6ad..567cb95e8e44 100644 --- a/cpp/tests/unit_tests/kernels/banRepeatNGramsKernelsTest.cpp +++ b/cpp/tests/unit_tests/kernels/banRepeatNGramsKernelsTest.cpp @@ -15,7 +15,6 @@ */ #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/banRepeatNgram.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/runtimeKernels.h" @@ -53,25 +52,24 @@ class BanRepeatNgramKernelsTest : public testing::Test SizeType32 const batchSize = outputIds.size(); auto const maxBatchSize = 2 * batchSize; - mLogits - = BufferManager::pinned(ITensor::makeShape({batchSize, mVocabSizePadded}), tensorrt_llm::DataType::kFLOAT); + mLogits = BufferManager::pinned(ITensor::makeShape({batchSize, mVocabSizePadded}), nvinfer1::DataType::kFLOAT); mSequenceLengths - = BufferManager::pinned(ITensor::makeShape({maxBatchSize, mBeamWidth}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize, mBeamWidth}), nvinfer1::DataType::kINT32); mFinished = BufferManager::pinned( ITensor::makeShape({maxBatchSize, mBeamWidth}), TRTDataType<tk::FinishedState::UnderlyingType>::value); mOutputIds = BufferManager::pinned( - ITensor::makeShape({maxBatchSize, mBeamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize, mBeamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); mOutputIdsPtr = BufferManager::pinned(ITensor::makeShape({maxBatchSize, mBeamWidth}), ptrType); mParentIds = BufferManager::pinned( - ITensor::makeShape({maxBatchSize, mBeamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize, mBeamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); mParentIdsPtr = BufferManager::pinned(ITensor::makeShape({maxBatchSize, mBeamWidth}), ptrType); - mNGramSizes = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + mNGramSizes = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); auto batchSlotsPtr = bufferCast<int32_t>(*mBatchSlots); for (SizeType32 bi = 0; bi < batchSize; ++bi) diff --git a/cpp/tests/unit_tests/kernels/cudaCoreGemm/cudaCoreGemmKernelTest.cpp b/cpp/tests/unit_tests/kernels/cudaCoreGemm/cudaCoreGemmKernelTest.cpp index f7dabf98a93e..05247e2d27a4 100644 --- a/cpp/tests/unit_tests/kernels/cudaCoreGemm/cudaCoreGemmKernelTest.cpp +++ b/cpp/tests/unit_tests/kernels/cudaCoreGemm/cudaCoreGemmKernelTest.cpp @@ -1,3 +1,4 @@ +#include <NvInferRuntime.h> #include <cublasLt.h> #include <cuda_fp8.h> #include <cuda_profiler_api.h> diff --git a/cpp/tests/unit_tests/kernels/decodingKernelTest.cpp b/cpp/tests/unit_tests/kernels/decodingKernelTest.cpp index a458f783ef65..4b94e67cb5a1 100644 --- a/cpp/tests/unit_tests/kernels/decodingKernelTest.cpp +++ b/cpp/tests/unit_tests/kernels/decodingKernelTest.cpp @@ -15,7 +15,6 @@ */ #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/decodingCommon.h" #include "tensorrt_llm/kernels/decodingKernels.h" #include "tensorrt_llm/kernels/speculativeDecoding/externalDraftTokensKernels.h" @@ -217,15 +216,13 @@ class TestBeamHypothesesCopy : public ::testing::Test srcBeams.empty(*mBufferManager); srcBeams.reshape(batchSize, beamWidth, maxSeqLen); - mSrcCumLogProbs - = mBufferManager->gpu(ITensor::makeShape({batchSize, beamWidth}), tensorrt_llm::DataType::kFLOAT); + mSrcCumLogProbs = mBufferManager->gpu(ITensor::makeShape({batchSize, beamWidth}), nvinfer1::DataType::kFLOAT); setBuffers(srcBeams, mSrcCumLogProbs, 2); dstBeams.empty(*mBufferManager); dstBeams.reshape(batchSize, beamWidth, maxSeqLen); - mDstCumLogProbs - = mBufferManager->gpu(ITensor::makeShape({batchSize, beamWidth}), tensorrt_llm::DataType::kFLOAT); + mDstCumLogProbs = mBufferManager->gpu(ITensor::makeShape({batchSize, beamWidth}), nvinfer1::DataType::kFLOAT); setBuffers(dstBeams, mDstCumLogProbs, 1); } @@ -547,7 +544,7 @@ class TestGatherTree : public ::testing::Test SizeType32 constexpr nbRnnLayers{0}; SizeType32 constexpr nbHeads{16}; SizeType32 constexpr hiddenSize{1024}; - tensorrt_llm::DataType constexpr dtype{tensorrt_llm::DataType::kFLOAT}; + nvinfer1::DataType constexpr dtype{nvinfer1::DataType::kFLOAT}; ModelConfig modelConfig{ vocabSize, nbAttentionLayers + nbRnnLayers, nbAttentionLayers, nbRnnLayers, nbHeads, hiddenSize, dtype}; @@ -1142,35 +1139,32 @@ class DecodingKernelsTest : public testing::Test auto const ptrType = TRTDataType<T*>::value; mDraftTokens = mBufferManager->pinnedPool( - ITensor::makeShape({mMaxBatchSize, mMaxDraftSeqlen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mMaxBatchSize, mMaxDraftSeqlen}), nvinfer1::DataType::kINT32); mTargetTokens = mBufferManager->pinnedPool( - ITensor::makeShape({mMaxBatchSize, mMaxTargetSeqlen}), tensorrt_llm::DataType::kINT32); - mOutputTokens = mBufferManager->pinnedPool( - ITensor::makeShape({mMaxBatchSize, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mMaxBatchSize, mMaxTargetSeqlen}), nvinfer1::DataType::kINT32); + mOutputTokens + = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize, mMaxSeqLen}), nvinfer1::DataType::kINT32); mNumsDraftTokens = mBufferManager->pinnedPool( - ITensor::makeShape({mMaxBatchSize, mMaxDraftSeqPerStep}), tensorrt_llm::DataType::kINT32); - mSequenceLengths - = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mAcceptedLengths - = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mContextLengths - = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mMaxBatchSize, mMaxDraftSeqPerStep}), nvinfer1::DataType::kINT32); + mSequenceLengths = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mAcceptedLengths = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mContextLengths = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mFinishedSteps = mBufferManager->pinnedPool(ITensor::makeShape({mMaxDraftTokens + 1, mMaxBatchSize}), TRTDataType<tk::FinishedState::UnderlyingType>::value); mFinishedFinal = mBufferManager->pinnedPool( ITensor::makeShape({mMaxBatchSize}), TRTDataType<tk::FinishedState::UnderlyingType>::value); - mFinishedSum = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mFinishedSum = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mPaths = mBufferManager->pinnedPool( - ITensor::makeShape({mMaxBatchSize, mMaxDraftSeqPerStep, mMaxDraftTokens}), tensorrt_llm::DataType::kINT32); - mEndIds = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mMaxBatchSize, mMaxDraftSeqPerStep, mMaxDraftTokens}), nvinfer1::DataType::kINT32); + mEndIds = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mBatchSlots = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); auto batchSlotsRange = BufferRange<SizeType32>(*mBatchSlots); std::iota(batchSlotsRange.begin(), batchSlotsRange.end(), 0); mCurandStates = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, sizeof(curandState_t)}), tensorrt_llm::DataType::kINT8); + ITensor::makeShape({mMaxBatchSize, sizeof(curandState_t)}), nvinfer1::DataType::kINT8); mAcceptedLen.resize(mMaxBatchSize); mOutputLen.resize(mMaxBatchSize); @@ -1200,9 +1194,8 @@ class DecodingKernelsTest : public testing::Test mMedusaInputLogitsPtrs = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize, mMaxNumHeads}), ptrType); mTokensPerStep - = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mBestPaths - = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mBestPaths = mBufferManager->pinnedPool(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); } } diff --git a/cpp/tests/unit_tests/kernels/eaglePackDataTest.cpp b/cpp/tests/unit_tests/kernels/eaglePackDataTest.cpp index 8ce24f813e65..bdf74efb59be 100644 --- a/cpp/tests/unit_tests/kernels/eaglePackDataTest.cpp +++ b/cpp/tests/unit_tests/kernels/eaglePackDataTest.cpp @@ -25,8 +25,9 @@ #include "tensorrt_llm/runtime/iBuffer.h" #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/runtimeKernels.h" +#include "tensorrt_llm/runtime/tllmLogger.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntimeBase.h> #include <algorithm> #include <cstdint> @@ -131,81 +132,81 @@ class EaglePackDataTest : public ::testing::Test { // inputs mBatchSlots = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mInputTemperatures = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kFLOAT); mInputRandomDataSample = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kFLOAT); mInputRandomDataValidation = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kFLOAT); + nvinfer1::DataType::kFLOAT); mInputNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingDraftTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mInputNextDraftPaths = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mInputSpecDecodingGenerationLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mInputSpecDecodingPositionOffsets = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); auto const numPackedMasks = static_cast<SizeType32>(tensorrt_llm::common::divUp(mSamplingParams.getMaxDecodingTokens(), 32)); mInputSpecDecodingPackedMasks = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens(), numPackedMasks}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); // outputs mOutputTemperatures = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kFLOAT); mOutputRandomDataSample = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kFLOAT); mOutputRandomDataValidation = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kFLOAT); + nvinfer1::DataType::kFLOAT); mOutputNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingDraftTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mOutputNextDraftLens = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mOutputNextDraftPaths = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mOutputSpecDecodingGenerationLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mOutputSpecDecodingPositionOffsets = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mOutputSpecDecodingPackedMasks = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens(), numPackedMasks}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); // workspace - mMaxGenerationLength = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + mMaxGenerationLength = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); mCumSumGenerationLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize() + 1}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getBatchSize() + 1}), nvinfer1::DataType::kINT32); mScanReduceTempStorageBytes = tksd::invokeScanReduceGenerationLengths( mSamplingParams.getBatchSize(), nullptr, nullptr, 0, nullptr, nullptr, mStream->get()); diff --git a/cpp/tests/unit_tests/kernels/mixtureOfExpertsTest.cu b/cpp/tests/unit_tests/kernels/mixtureOfExpertsTest.cu index 61c182dfd1ea..01cd1c4d792d 100644 --- a/cpp/tests/unit_tests/kernels/mixtureOfExpertsTest.cu +++ b/cpp/tests/unit_tests/kernels/mixtureOfExpertsTest.cu @@ -32,7 +32,6 @@ #endif #include "tensorrt_llm/kernels/cutlass_kernels/include/cutlass_kernel_selector.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include <tensorrt_llm/kernels/cutlass_kernels/cutlass_type_conversion.h> @@ -2509,31 +2508,31 @@ constexpr static auto typeToDtypeID() { if constexpr (std::is_same_v<T, SafeFP8>) { - return tensorrt_llm::DataType::kFP8; + return nvinfer1::DataType::kFP8; } else if constexpr (std::is_same_v<T, SafeFP4>) { - return tensorrt_llm::DataType::kFP4; + return nvinfer1::DataType::kFP4; } else if constexpr (std::is_same_v<T, uint8_t>) { - return tensorrt_llm::DataType::kINT8; + return nvinfer1::DataType::kINT8; } else if constexpr (std::is_same_v<T, cutlass::uint4b_t>) { - return tensorrt_llm::DataType::kINT4; + return nvinfer1::DataType::kINT4; } else if constexpr (std::is_same_v<T, nv_bfloat16>) { - return tensorrt_llm::DataType::kBF16; + return nvinfer1::DataType::kBF16; } else if constexpr (std::is_same_v<T, half>) { - return tensorrt_llm::DataType::kHALF; + return nvinfer1::DataType::kHALF; } else if constexpr (std::is_same_v<T, float>) { - return tensorrt_llm::DataType::kFLOAT; + return nvinfer1::DataType::kFLOAT; } else { @@ -2603,16 +2602,14 @@ TEST_F(MixtureOfExpertsProfilerTest, TestGeneratedProfilerDistribution) for (int ep : {1, 4, 8}) { #ifdef USING_OSS_CUTLASS_MOE_GEMM - backend.init(this->mMoERunner, GemmProfilerBackend::GemmToProfile::GEMM_1, tensorrt_llm::DataType::kHALF, - tensorrt_llm::DataType::kHALF, tensorrt_llm::DataType::kHALF, num_experts, k, 1024, 1024, 4096, - mGroupSize, {}, false, mUseLora, /*min_latency_mode=*/false, /*need_weights=*/true, - MOEParallelismConfig{1, 0, ep, 0}, + backend.init(this->mMoERunner, GemmProfilerBackend::GemmToProfile::GEMM_1, nvinfer1::DataType::kHALF, + nvinfer1::DataType::kHALF, nvinfer1::DataType::kHALF, num_experts, k, 1024, 1024, 4096, mGroupSize, {}, + false, mUseLora, /*min_latency_mode=*/false, /*need_weights=*/true, MOEParallelismConfig{1, 0, ep, 0}, /*enable_alltoall=*/false); #else - backend.init(this->mMoERunner, GemmProfilerBackend::GemmToProfile::GEMM_1, tensorrt_llm::DataType::kHALF, - tensorrt_llm::DataType::kHALF, tensorrt_llm::DataType::kHALF, num_experts, k, 1024, 4096, mGroupSize, - {}, false, mUseLora, /*min_latency_mode=*/false, /*need_weights=*/true, - MOEParallelismConfig{1, 0, ep, ep - 1}); + backend.init(this->mMoERunner, GemmProfilerBackend::GemmToProfile::GEMM_1, nvinfer1::DataType::kHALF, + nvinfer1::DataType::kHALF, nvinfer1::DataType::kHALF, num_experts, k, 1024, 4096, mGroupSize, {}, false, + mUseLora, /*min_latency_mode=*/false, /*need_weights=*/true, MOEParallelismConfig{1, 0, ep, ep - 1}); #endif auto ws_size = backend.getWorkspaceSize(num_tokens); diff --git a/cpp/tests/unit_tests/kernels/mlaChunkedPrefillTest.cu b/cpp/tests/unit_tests/kernels/mlaChunkedPrefillTest.cu index c1fa77239729..3e4e9a1da0a3 100644 --- a/cpp/tests/unit_tests/kernels/mlaChunkedPrefillTest.cu +++ b/cpp/tests/unit_tests/kernels/mlaChunkedPrefillTest.cu @@ -8,7 +8,6 @@ #include "tensorrt_llm/kernels/kvCacheUtils.h" #include "tensorrt_llm/runtime/bufferManager.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/mlaChunkedPrefill.cuh" #include "tensorrt_llm/runtime/cudaStream.h" #include <cstring> @@ -430,18 +429,18 @@ protected: using tensorrt_llm::runtime::ITensor; using tensorrt_llm::runtime::bufferCast; - auto dtype = tensorrt_llm::DataType::kHALF; + auto dtype = nvinfer1::DataType::kHALF; if constexpr (std::is_same_v<DataType, float>) { - dtype = tensorrt_llm::DataType::kFLOAT; + dtype = nvinfer1::DataType::kFLOAT; } else if constexpr (std::is_same_v<DataType, half>) { - dtype = tensorrt_llm::DataType::kHALF; + dtype = nvinfer1::DataType::kHALF; } else if constexpr (std::is_same_v<DataType, __nv_bfloat16>) { - dtype = tensorrt_llm::DataType::kBF16; + dtype = nvinfer1::DataType::kBF16; } else { @@ -450,11 +449,11 @@ protected: auto cacheType = dtype; if constexpr (std::is_same_v<TCache, __nv_fp8_e4m3>) { - cacheType = tensorrt_llm::DataType::kFP8; + cacheType = nvinfer1::DataType::kFP8; this->h_kv_scale_quant_orig - = tensorrt_llm::runtime::BufferManager::pinned(ITensor::makeShape({1}), tensorrt_llm::DataType::kFLOAT); - this->d_kv_scale_quant_orig = tensorrt_llm::runtime::BufferManager::gpuSync( - ITensor::makeShape({1}), tensorrt_llm::DataType::kFLOAT); + = tensorrt_llm::runtime::BufferManager::pinned(ITensor::makeShape({1}), nvinfer1::DataType::kFLOAT); + this->d_kv_scale_quant_orig + = tensorrt_llm::runtime::BufferManager::gpuSync(ITensor::makeShape({1}), nvinfer1::DataType::kFLOAT); auto* kv_scale_quant_orig_ptr = bufferCast<float>(*(this->h_kv_scale_quant_orig)); float kv_scale_orig_quant = 2.0F; kv_scale_quant_orig_ptr[0] = 1.0 / kv_scale_orig_quant; @@ -464,13 +463,13 @@ protected: // cu lens this->h_cu_kv_seq_lens = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({this->mBatchSize + 1}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({this->mBatchSize + 1}), nvinfer1::DataType::kINT64); this->h_cu_q_seq_lens = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({this->mBatchSize + 1}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({this->mBatchSize + 1}), nvinfer1::DataType::kINT64); this->d_cu_kv_seq_lens = tensorrt_llm::runtime::BufferManager::gpuSync( - this->h_cu_kv_seq_lens->getShape(), tensorrt_llm::DataType::kINT64); + this->h_cu_kv_seq_lens->getShape(), nvinfer1::DataType::kINT64); this->d_cu_q_seq_lens = tensorrt_llm::runtime::BufferManager::gpuSync( - this->h_cu_q_seq_lens->getShape(), tensorrt_llm::DataType::kINT64); + this->h_cu_q_seq_lens->getShape(), nvinfer1::DataType::kINT64); { this->mMaxSeqLen = 0; this->mMaxQSeqLen = 0; @@ -513,14 +512,14 @@ protected: int const total_cached_kv_len = this->mTotalKVLen - this->mTotalQLen; int const chunked_loop_num = (total_cached_kv_len + total_chunk_size - 1) / total_chunk_size; this->h_cu_chunk_lens = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({chunked_loop_num + 1, this->mBatchSize + 1}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({chunked_loop_num + 1, this->mBatchSize + 1}), nvinfer1::DataType::kINT64); this->h_chunked_ld_global_offset = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({chunked_loop_num + 1, this->mBatchSize}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({chunked_loop_num + 1, this->mBatchSize}), nvinfer1::DataType::kINT64); this->memsetZeroHost(this->h_chunked_ld_global_offset); this->d_cu_chunk_lens = tensorrt_llm::runtime::BufferManager::gpuSync( - this->h_cu_chunk_lens->getShape(), tensorrt_llm::DataType::kINT64); + this->h_cu_chunk_lens->getShape(), nvinfer1::DataType::kINT64); this->d_chunked_ld_global_offset = tensorrt_llm::runtime::BufferManager::gpuSync( - this->h_chunked_ld_global_offset->getShape(), tensorrt_llm::DataType::kINT64); + this->h_chunked_ld_global_offset->getShape(), nvinfer1::DataType::kINT64); // kv cache this->mMaxBlockPerSeq = (this->mMaxSeqLen + this->mTokensPerBlock - 1) / this->mTokensPerBlock; @@ -540,13 +539,13 @@ protected: this->mLoraSize + this->mRopeSize}), cacheType); this->h_compressed_offset_tensor = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({this->mBatchSize, 2, this->mMaxBlockPerSeq + 1}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({this->mBatchSize, 2, this->mMaxBlockPerSeq + 1}), nvinfer1::DataType::kINT32); this->d_kv_cache_tensor = tensorrt_llm::runtime::BufferManager::gpuSync(this->h_kv_cache_tensor->getShape(), dtype); this->d_compressed_kv_cache_tensor = tensorrt_llm::runtime::BufferManager::gpuSync(this->h_compressed_kv_cache_tensor->getShape(), cacheType); this->d_compressed_offset_tensor = tensorrt_llm::runtime::BufferManager::gpuSync( - this->h_compressed_offset_tensor->getShape(), tensorrt_llm::DataType::kINT32); + this->h_compressed_offset_tensor->getShape(), nvinfer1::DataType::kINT32); { auto* compressed_kv_cache_ptr = bufferCast<TCache>(*(this->h_compressed_kv_cache_tensor)); @@ -602,15 +601,15 @@ protected: this->m_h_output_tensor = tensorrt_llm::runtime::BufferManager::pinned( ITensor::makeShape({this->mTotalQLen, this->mNumHeads, this->mNopeSize}), dtype); this->m_h_softmax_sum_tensor = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({2, this->mTotalQLen, this->mNumHeads}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({2, this->mTotalQLen, this->mNumHeads}), nvinfer1::DataType::kFLOAT); this->m_h_softmax_sum_accum_tensor = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({2, this->mTotalQLen, this->mNumHeads}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({2, this->mTotalQLen, this->mNumHeads}), nvinfer1::DataType::kFLOAT); this->m_h_output_tensor_ref = tensorrt_llm::runtime::BufferManager::pinned( ITensor::makeShape({this->mTotalQLen, this->mNumHeads, this->mNopeSize}), dtype); this->m_h_output_tensor_accum = tensorrt_llm::runtime::BufferManager::pinned( ITensor::makeShape({this->mTotalQLen, this->mNumHeads, this->mNopeSize}), dtype); this->m_h_merge_op = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({chunked_loop_num + 1, this->mBatchSize}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({chunked_loop_num + 1, this->mBatchSize}), nvinfer1::DataType::kINT64); this->m_d_q_tensor = tensorrt_llm::runtime::BufferManager::gpuSync(this->m_h_q_tensor->getShape(), dtype); this->m_d_kv_full_tensor = tensorrt_llm::runtime::BufferManager::gpuSync(this->m_h_kv_full_tensor->getShape(), dtype); @@ -619,13 +618,13 @@ protected: this->m_d_output_tensor = tensorrt_llm::runtime::BufferManager::gpuSync(this->m_h_output_tensor->getShape(), dtype); this->m_d_softmax_sum_tensor = tensorrt_llm::runtime::BufferManager::gpuSync( - this->m_h_softmax_sum_tensor->getShape(), tensorrt_llm::DataType::kFLOAT); + this->m_h_softmax_sum_tensor->getShape(), nvinfer1::DataType::kFLOAT); this->m_d_softmax_sum_accum_tensor = tensorrt_llm::runtime::BufferManager::gpuSync( - this->m_h_softmax_sum_accum_tensor->getShape(), tensorrt_llm::DataType::kFLOAT); + this->m_h_softmax_sum_accum_tensor->getShape(), nvinfer1::DataType::kFLOAT); this->m_d_output_tensor_accum = tensorrt_llm::runtime::BufferManager::gpuSync(this->m_h_output_tensor_accum->getShape(), dtype); - this->m_d_merge_op = tensorrt_llm::runtime::BufferManager::gpuSync( - this->m_h_merge_op->getShape(), tensorrt_llm::DataType::kINT64); + this->m_d_merge_op + = tensorrt_llm::runtime::BufferManager::gpuSync(this->m_h_merge_op->getShape(), nvinfer1::DataType::kINT64); { auto* q_ptr = bufferCast<DataType>(*(this->m_h_q_tensor)); diff --git a/cpp/tests/unit_tests/kernels/mlaPreprocessTest.cu b/cpp/tests/unit_tests/kernels/mlaPreprocessTest.cu index 3fc249a2f24a..f2c0863779bc 100644 --- a/cpp/tests/unit_tests/kernels/mlaPreprocessTest.cu +++ b/cpp/tests/unit_tests/kernels/mlaPreprocessTest.cu @@ -23,7 +23,6 @@ #include "tensorrt_llm/kernels/kvCacheUtils.h" #include "tensorrt_llm/runtime/bufferManager.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/mlaKernels.h" #include <random> @@ -233,18 +232,18 @@ protected: using tensorrt_llm::runtime::ITensor; using tensorrt_llm::runtime::bufferCast; - auto dtype = tensorrt_llm::DataType::kHALF; + auto dtype = nvinfer1::DataType::kHALF; if constexpr (std::is_same_v<DataType, float>) { - dtype = tensorrt_llm::DataType::kFLOAT; + dtype = nvinfer1::DataType::kFLOAT; } else if constexpr (std::is_same_v<DataType, half>) { - dtype = tensorrt_llm::DataType::kHALF; + dtype = nvinfer1::DataType::kHALF; } else if constexpr (std::is_same_v<DataType, __nv_bfloat16>) { - dtype = tensorrt_llm::DataType::kBF16; + dtype = nvinfer1::DataType::kBF16; } else { @@ -253,15 +252,15 @@ protected: auto cache_dtype = dtype; if constexpr (std::is_same_v<TCache, __nv_fp8_e4m3>) { - cache_dtype = tensorrt_llm::DataType::kFP8; + cache_dtype = nvinfer1::DataType::kFP8; this->h_kv_scale_orig_quant - = tensorrt_llm::runtime::BufferManager::pinned(ITensor::makeShape({1}), tensorrt_llm::DataType::kFLOAT); - this->d_kv_scale_orig_quant = tensorrt_llm::runtime::BufferManager::gpuSync( - ITensor::makeShape({1}), tensorrt_llm::DataType::kFLOAT); + = tensorrt_llm::runtime::BufferManager::pinned(ITensor::makeShape({1}), nvinfer1::DataType::kFLOAT); + this->d_kv_scale_orig_quant + = tensorrt_llm::runtime::BufferManager::gpuSync(ITensor::makeShape({1}), nvinfer1::DataType::kFLOAT); this->h_kv_scale_quant_orig - = tensorrt_llm::runtime::BufferManager::pinned(ITensor::makeShape({1}), tensorrt_llm::DataType::kFLOAT); - this->d_kv_scale_quant_orig = tensorrt_llm::runtime::BufferManager::gpuSync( - ITensor::makeShape({1}), tensorrt_llm::DataType::kFLOAT); + = tensorrt_llm::runtime::BufferManager::pinned(ITensor::makeShape({1}), nvinfer1::DataType::kFLOAT); + this->d_kv_scale_quant_orig + = tensorrt_llm::runtime::BufferManager::gpuSync(ITensor::makeShape({1}), nvinfer1::DataType::kFLOAT); auto* kv_scale_orig_quant_ptr = bufferCast<float>(*(this->h_kv_scale_orig_quant)); auto* kv_scale_quant_orig_ptr = bufferCast<float>(*(this->h_kv_scale_quant_orig)); float kv_scale_orig_quant = 2.0f; @@ -277,13 +276,13 @@ protected: static_assert(std::is_same_v<DataType, TCache>, "TCache must be the same type as DataType"); } this->h_cu_seq_lens = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({this->mNumRequests + 1}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({this->mNumRequests + 1}), nvinfer1::DataType::kINT64); this->h_cu_ctx_cached_kv_lens = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({this->mNumRequests + 1}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({this->mNumRequests + 1}), nvinfer1::DataType::kINT64); this->d_cu_seq_lens = tensorrt_llm::runtime::BufferManager::gpuSync( - ITensor::makeShape({this->mNumRequests + 1}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({this->mNumRequests + 1}), nvinfer1::DataType::kINT64); this->d_cu_ctx_cached_kv_lens = tensorrt_llm::runtime::BufferManager::gpuSync( - ITensor::makeShape({this->mNumRequests + 1}), tensorrt_llm::DataType::kINT64); + ITensor::makeShape({this->mNumRequests + 1}), nvinfer1::DataType::kINT64); { // set random sequence length auto* cu_seq_lens_temp_ptr = bufferCast<int64_t>(*(this->h_cu_seq_lens)); @@ -334,9 +333,9 @@ protected: this->mTokensPerBlock, this->mLoraSize + this->mRopeSize}), cache_dtype); this->h_offset_tensor = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq}), nvinfer1::DataType::kINT32); this->h_compressed_offset_tensor = tensorrt_llm::runtime::BufferManager::pinned( - ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq}), nvinfer1::DataType::kINT32); this->d_kv_cache_tensor = tensorrt_llm::runtime::BufferManager::gpuSync( ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq, this->mNumHeadsUncompressed, this->mTokensPerBlock, this->mUncompressedHeadSize + this->mRopeSize}), @@ -350,9 +349,9 @@ protected: this->mTokensPerBlock, this->mLoraSize + this->mRopeSize}), cache_dtype); this->d_offset_tensor = tensorrt_llm::runtime::BufferManager::gpuSync( - ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq}), nvinfer1::DataType::kINT32); this->d_compressed_offset_tensor = tensorrt_llm::runtime::BufferManager::gpuSync( - ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({this->mNumRequests, 2, this->mMaxBlockPerSeq}), nvinfer1::DataType::kINT32); { auto* kv_cache_ptr = bufferCast<DataType>(*(this->h_kv_cache_tensor)); auto* kv_cache_ref_ptr = bufferCast<DataType>(*(this->h_kv_cache_tensor_ref)); diff --git a/cpp/tests/unit_tests/kernels/prepareCustomMaskTest.cpp b/cpp/tests/unit_tests/kernels/prepareCustomMaskTest.cpp index 6616c9c47668..61617934f236 100644 --- a/cpp/tests/unit_tests/kernels/prepareCustomMaskTest.cpp +++ b/cpp/tests/unit_tests/kernels/prepareCustomMaskTest.cpp @@ -25,7 +25,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaKernels.h" #include "tensorrt_llm/kernels/trtllmGenKernels/fmha/fmhaRunnerParams.h" #include "tensorrt_llm/kernels/trtllmGenKernels/fmha/prepareCustomMask.h" @@ -294,8 +293,8 @@ class PrepareCustomMaskTest : public ::testing::Test int64_t totalMaskSize = static_cast<int64_t>(batchSize) * maxNumTilesQ * maxNumCustomMaskTilesKv * numInstsQ * numInstsKv * (tileSizeQ * tileSizeKvPadded) / 32; - auto customMaskOffsetsDevice = mBufferManager->gpu(batchSize, tensorrt_llm::DataType::kINT64); - auto customMaskDevice = mBufferManager->gpu(totalMaskSize, tensorrt_llm::DataType::kINT32); + auto customMaskOffsetsDevice = mBufferManager->gpu(batchSize, nvinfer1::DataType::kINT64); + auto customMaskDevice = mBufferManager->gpu(totalMaskSize, nvinfer1::DataType::kINT32); // Clear GPU buffers to ensure no stale data from previous tests cudaMemsetAsync(bufferCast<int64_t>(*customMaskOffsetsDevice), 0, batchSize * sizeof(int64_t), mStream->get()); diff --git a/cpp/tests/unit_tests/kernels/ropeTest.cu b/cpp/tests/unit_tests/kernels/ropeTest.cu index 36c91481722f..517b006e4fde 100644 --- a/cpp/tests/unit_tests/kernels/ropeTest.cu +++ b/cpp/tests/unit_tests/kernels/ropeTest.cu @@ -15,13 +15,11 @@ */ #include <gtest/gtest.h> -#include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/quantization.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/decodingCommon.h" -#include "tensorrt_llm/kernels/gptKernels.h" #include "tensorrt_llm/kernels/kvCacheUtils.h" #include "tensorrt_llm/kernels/unfusedAttentionKernels.h" +#include "tensorrt_llm/plugins/gptAttentionCommon/gptAttentionCommon.h" #include "tensorrt_llm/runtime/bufferManager.h" #include <random> @@ -31,7 +29,6 @@ #include <cuda_fp4.h> #endif -using namespace tensorrt_llm::common; using namespace tensorrt_llm::runtime; using namespace tensorrt_llm::kernels; @@ -505,27 +502,26 @@ protected: { auto const cu_seqlens_size = batch_size + 1; - cu_q_seqlens_tensor - = mBufferManager->pinned(ITensor::makeShape({cu_seqlens_size}), tensorrt_llm::DataType::kINT32); + cu_q_seqlens_tensor = mBufferManager->pinned(ITensor::makeShape({cu_seqlens_size}), nvinfer1::DataType::kINT32); cu_kv_seqlens_tensor - = mBufferManager->pinned(ITensor::makeShape({cu_seqlens_size}), tensorrt_llm::DataType::kINT32); - padding_offset_tensor = mBufferManager->pinned( - ITensor::makeShape({batch_size, input_seq_length}), tensorrt_llm::DataType::kINT32); - encoder_padding_offset_tensor = mBufferManager->pinned( - ITensor::makeShape({batch_size, cross_qkv_length}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({cu_seqlens_size}), nvinfer1::DataType::kINT32); + padding_offset_tensor + = mBufferManager->pinned(ITensor::makeShape({batch_size, input_seq_length}), nvinfer1::DataType::kINT32); + encoder_padding_offset_tensor + = mBufferManager->pinned(ITensor::makeShape({batch_size, cross_qkv_length}), nvinfer1::DataType::kINT32); fmha_tile_counter_ptr_tensor - = mBufferManager->pinned(ITensor::makeShape({mEnableContextFMHA ? 1 : 0}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({mEnableContextFMHA ? 1 : 0}), nvinfer1::DataType::kINT32); rotary_inv_freq_buf_tensor = mBufferManager->pinned( - ITensor::makeShape({batch_size, mRotaryEmbeddingDim / 2}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({batch_size, mRotaryEmbeddingDim / 2}), nvinfer1::DataType::kFLOAT); int const max_num_tokens = batch_size * input_seq_length; tokens_info_tensor - = mBufferManager->pinned(ITensor::makeShape({max_num_tokens, 2}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({max_num_tokens, 2}), nvinfer1::DataType::kINT32); #ifdef ENABLE_FP4 if constexpr (std::is_same_v<KVCacheType, __nv_fp4_e2m1>) { - global_scale_tensor = mBufferManager->pinned(ITensor::makeShape({2}), tensorrt_llm::DataType::kFLOAT); + global_scale_tensor = mBufferManager->pinned(ITensor::makeShape({2}), nvinfer1::DataType::kFLOAT); } #endif } @@ -588,7 +584,7 @@ protected: // // Rotary cos sin cache buffer to avoid re-computing. SizeType32 maxOutputSize{generateRandomSizeSmallerThan(1024)}; rotary_cos_sin_tensor = this->mBufferManager->pinned( - ITensor::makeShape({mRotaryEmbeddingMaxPositions, mRotaryEmbeddingDim}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({mRotaryEmbeddingMaxPositions, mRotaryEmbeddingDim}), nvinfer1::DataType::kFLOAT); rotary_fill_help = bufferCast<float>(*(rotary_cos_sin_tensor)); // createCosSinBuf(rotary_fill_help, mRotaryEmbeddingMaxPositions, mRotaryEmbeddingDim); //currently broken // fillWithOnesAndZerosInterleaved(rotary_fill_help, mRotaryEmbeddingMaxPositions* @@ -598,7 +594,7 @@ protected: batch_size = generateRandomSizeSmallerThan(12); - q_seq_lengths_tensor = mBufferManager->pinned(ITensor::makeShape({batch_size}), tensorrt_llm::DataType::kINT32); + q_seq_lengths_tensor = mBufferManager->pinned(ITensor::makeShape({batch_size}), nvinfer1::DataType::kINT32); q_seq_lengths = bufferCast<int32_t>(*(q_seq_lengths_tensor)); for (SizeType32 ii = 0; ii < batch_size; ++ii) diff --git a/cpp/tests/unit_tests/kernels/routing/routingDeepSeekTest.cpp b/cpp/tests/unit_tests/kernels/routing/routingDeepSeekTest.cpp index be5c8f3c48d1..78598fa4c417 100644 --- a/cpp/tests/unit_tests/kernels/routing/routingDeepSeekTest.cpp +++ b/cpp/tests/unit_tests/kernels/routing/routingDeepSeekTest.cpp @@ -14,7 +14,6 @@ * limitations under the License. */ -#include "tensorrt_llm/common/tllmDataType.h" #include "tests/unit_tests/kernels/routing/routingTest.h" namespace tk = tensorrt_llm::kernels; @@ -157,8 +156,8 @@ class RoutingDeepSeekKernelTest : public RoutingKernelTest<T> { RoutingKernelTest<T>::allocateBuffers(param); int64_t scoresSize = param.numTokens * param.numExperts; - this->mPtrScoresHost = mBufferManager->pinned(ITensor::makeShape({scoresSize}), tensorrt_llm::DataType::kFLOAT); - this->mPtrScoresDevice = mBufferManager->gpu(ITensor::makeShape({scoresSize}), tensorrt_llm::DataType::kFLOAT); + this->mPtrScoresHost = mBufferManager->pinned(ITensor::makeShape({scoresSize}), nvinfer1::DataType::kFLOAT); + this->mPtrScoresDevice = mBufferManager->gpu(ITensor::makeShape({scoresSize}), nvinfer1::DataType::kFLOAT); this->mPtrRoutingBiasHost = mBufferManager->pinned(ITensor::makeShape({param.numExperts}), TRTDataType<T>::value); @@ -431,9 +430,9 @@ TYPED_TEST(RoutingDeepSeekKernelTest, ClusterLevelWithFloat32Bias) // the GPU kernel (using fp32 bias) and the host reference (using T-typed bias) // observe numerically equivalent inputs. auto float32BiasHost - = this->mBufferManager->pinned(ITensor::makeShape({param.numExperts}), tensorrt_llm::DataType::kFLOAT); + = this->mBufferManager->pinned(ITensor::makeShape({param.numExperts}), nvinfer1::DataType::kFLOAT); auto float32BiasDevice - = this->mBufferManager->gpu(ITensor::makeShape({param.numExperts}), tensorrt_llm::DataType::kFLOAT); + = this->mBufferManager->gpu(ITensor::makeShape({param.numExperts}), nvinfer1::DataType::kFLOAT); auto fp32BiasPtr = bufferCast<float>(*float32BiasHost); auto tBiasPtr = bufferCast<TypeParam>(*this->mPtrRoutingBiasHost); for (int i = 0; i < param.numExperts; i++) diff --git a/cpp/tests/unit_tests/kernels/routing/routingTest.cpp b/cpp/tests/unit_tests/kernels/routing/routingTest.cpp index c71510dc319c..ba5c020ade9e 100644 --- a/cpp/tests/unit_tests/kernels/routing/routingTest.cpp +++ b/cpp/tests/unit_tests/kernels/routing/routingTest.cpp @@ -14,7 +14,6 @@ * limitations under the License. */ #include "tests/unit_tests/kernels/routing/routingTest.h" -#include "tensorrt_llm/common/tllmDataType.h" namespace tensorrt_llm::tests::kernels::routing { @@ -55,25 +54,25 @@ void RoutingKernelTest<T>::allocateBuffers(RoutingKernelTestParam const& param) { countsSize = 2 * 256; } - mPtrExpertCountsHost = mBufferManager->pinned(ITensor::makeShape({countsSize}), tensorrt_llm::DataType::kINT32); - mPtrExpertCountsDevice = mBufferManager->gpu(ITensor::makeShape({countsSize}), tensorrt_llm::DataType::kINT32); + mPtrExpertCountsHost = mBufferManager->pinned(ITensor::makeShape({countsSize}), nvinfer1::DataType::kINT32); + mPtrExpertCountsDevice = mBufferManager->gpu(ITensor::makeShape({countsSize}), nvinfer1::DataType::kINT32); int64_t permIdxSize = 1; - mPtrPermutedIdxSizeHost = mBufferManager->pinned(ITensor::makeShape({permIdxSize}), tensorrt_llm::DataType::kINT32); - mPtrPermutedIdxSizeDevice = mBufferManager->gpu(ITensor::makeShape({permIdxSize}), tensorrt_llm::DataType::kINT32); + mPtrPermutedIdxSizeHost = mBufferManager->pinned(ITensor::makeShape({permIdxSize}), nvinfer1::DataType::kINT32); + mPtrPermutedIdxSizeDevice = mBufferManager->gpu(ITensor::makeShape({permIdxSize}), nvinfer1::DataType::kINT32); int64_t expIdxToPermIdxSize = numTokens * topK; mPtrExpandedIdxToPermutedIdxHost - = mBufferManager->pinned(ITensor::makeShape({expIdxToPermIdxSize}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({expIdxToPermIdxSize}), nvinfer1::DataType::kINT32); mPtrExpandedIdxToPermutedIdxDevice - = mBufferManager->gpu(ITensor::makeShape({expIdxToPermIdxSize}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({expIdxToPermIdxSize}), nvinfer1::DataType::kINT32); // int64_t permIdxToTokenIdxSize = (numTokens * topK + (numExperts << paddingLog2) - numExperts); int64_t permIdxToTokenIdxSize = (numTokens * topK + (numExperts * tileTokensDim) - numExperts); mPtrPermutedIdxToTokenIdxHost - = mBufferManager->pinned(ITensor::makeShape({permIdxToTokenIdxSize}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({permIdxToTokenIdxSize}), nvinfer1::DataType::kINT32); mPtrPermutedIdxToTokenIdxDevice - = mBufferManager->gpu(ITensor::makeShape({permIdxToTokenIdxSize}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({permIdxToTokenIdxSize}), nvinfer1::DataType::kINT32); int64_t expWeightsSize = numTokens * topK; mPtrTopKWeightsHost = mBufferManager->pinned(ITensor::makeShape({expWeightsSize}), TRTDataType<T>::value); @@ -82,8 +81,8 @@ void RoutingKernelTest<T>::allocateBuffers(RoutingKernelTestParam const& param) if (useTopKAsInput) { int64_t topKIdsSize = numTokens * topK; - mPtrTopKIdsHost = mBufferManager->pinned(ITensor::makeShape({topKIdsSize}), tensorrt_llm::DataType::kINT32); - mPtrTopKIdsDevice = mBufferManager->gpu(ITensor::makeShape({topKIdsSize}), tensorrt_llm::DataType::kINT32); + mPtrTopKIdsHost = mBufferManager->pinned(ITensor::makeShape({topKIdsSize}), nvinfer1::DataType::kINT32); + mPtrTopKIdsDevice = mBufferManager->gpu(ITensor::makeShape({topKIdsSize}), nvinfer1::DataType::kINT32); } else { @@ -92,26 +91,23 @@ void RoutingKernelTest<T>::allocateBuffers(RoutingKernelTestParam const& param) } int64_t ctaIdxSize = numTokens * topK; - mPtrCtaIdxXyToBatchIdxHost - = mBufferManager->pinned(ITensor::makeShape({ctaIdxSize}), tensorrt_llm::DataType::kINT32); - mPtrCtaIdxXyToBatchIdxDevice - = mBufferManager->gpu(ITensor::makeShape({ctaIdxSize}), tensorrt_llm::DataType::kINT32); + mPtrCtaIdxXyToBatchIdxHost = mBufferManager->pinned(ITensor::makeShape({ctaIdxSize}), nvinfer1::DataType::kINT32); + mPtrCtaIdxXyToBatchIdxDevice = mBufferManager->gpu(ITensor::makeShape({ctaIdxSize}), nvinfer1::DataType::kINT32); - mPtrCtaIdxXyToMnLimitHost - = mBufferManager->pinned(ITensor::makeShape({ctaIdxSize}), tensorrt_llm::DataType::kINT32); - mPtrCtaIdxXyToMnLimitDevice = mBufferManager->gpu(ITensor::makeShape({ctaIdxSize}), tensorrt_llm::DataType::kINT32); + mPtrCtaIdxXyToMnLimitHost = mBufferManager->pinned(ITensor::makeShape({ctaIdxSize}), nvinfer1::DataType::kINT32); + mPtrCtaIdxXyToMnLimitDevice = mBufferManager->gpu(ITensor::makeShape({ctaIdxSize}), nvinfer1::DataType::kINT32); int64_t numNonExitingCtasSize = 1; mPtrNumNonExitingCtasHost - = mBufferManager->pinned(ITensor::makeShape({numNonExitingCtasSize}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({numNonExitingCtasSize}), nvinfer1::DataType::kINT32); mPtrNumNonExitingCtasDevice - = mBufferManager->gpu(ITensor::makeShape({numNonExitingCtasSize}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({numNonExitingCtasSize}), nvinfer1::DataType::kINT32); int64_t idxSize = numTokens * topK * sizeof(PackedType); - mPtrTopKPackedHost = mBufferManager->pinned(ITensor::makeShape({idxSize}), tensorrt_llm::DataType::kINT8); - mPtrTopKPackedDevice = mBufferManager->gpu(ITensor::makeShape({idxSize}), tensorrt_llm::DataType::kINT8); + mPtrTopKPackedHost = mBufferManager->pinned(ITensor::makeShape({idxSize}), nvinfer1::DataType::kINT8); + mPtrTopKPackedDevice = mBufferManager->gpu(ITensor::makeShape({idxSize}), nvinfer1::DataType::kINT8); mCurandStatesDevice - = mBufferManager->gpu(ITensor::makeShape({numTokens, sizeof(curandState_t)}), tensorrt_llm::DataType::kINT8); + = mBufferManager->gpu(ITensor::makeShape({numTokens, sizeof(curandState_t)}), nvinfer1::DataType::kINT8); } template <typename T> @@ -131,19 +127,19 @@ void RoutingKernelTest<T>::computePermutation(RoutingKernelTestParam const& para PackedType* expIdxHostPtr = reinterpret_cast<PackedType*>(bufferCast<int8_t>(*this->mPtrTopKPackedHost)); auto tokenToExpertHost - = mBufferManager->pinned(ITensor::makeShape({param.numTokens * param.topK}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({param.numTokens * param.topK}), nvinfer1::DataType::kINT32); auto tokenToExpertHostPtr = bufferCast<int32_t>(*tokenToExpertHost); auto tokenToIdxInExpertHost - = mBufferManager->pinned(ITensor::makeShape({param.numTokens * param.topK}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({param.numTokens * param.topK}), nvinfer1::DataType::kINT32); auto tokenToIdxInExpertHostPtr = bufferCast<int32_t>(*tokenToIdxInExpertHost); auto expertScanCountsHost - = mBufferManager->pinned(ITensor::makeShape({param.numExperts + 1}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({param.numExperts + 1}), nvinfer1::DataType::kINT32); auto expertScanCountsHostPtr = bufferCast<int32_t>(*expertScanCountsHost); auto ctaScanCountsHost - = mBufferManager->pinned(ITensor::makeShape({param.numExperts + 1}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({param.numExperts + 1}), nvinfer1::DataType::kINT32); auto ctaScanCountsHostPtr = bufferCast<int32_t>(*ctaScanCountsHost); for (int ie = 0; ie < param.numExperts + 1; ++ie) @@ -411,7 +407,7 @@ void RoutingKernelTest<T>::runTest(RoutingKernelTestParam const& param) // Retrieve the workspace size of the routing kernel. auto const workspaceSize = getDeviceWorkspaceSize(param); TensorPtr workspaceDevice - = mBufferManager->gpu(ITensor::makeShape({static_cast<int64_t>(workspaceSize)}), tensorrt_llm::DataType::kINT8); + = mBufferManager->gpu(ITensor::makeShape({static_cast<int64_t>(workspaceSize)}), nvinfer1::DataType::kINT8); // Call tested function routing callTestedFunction(param, workspaceDevice); // Verify results diff --git a/cpp/tests/unit_tests/kernels/routing/routingTest.h b/cpp/tests/unit_tests/kernels/routing/routingTest.h index 8b24ee3aa24e..630cd72a5fcb 100644 --- a/cpp/tests/unit_tests/kernels/routing/routingTest.h +++ b/cpp/tests/unit_tests/kernels/routing/routingTest.h @@ -23,6 +23,7 @@ #include "tensorrt_llm/runtime/cudaStream.h" #include "tensorrt_llm/runtime/iBuffer.h" #include "tensorrt_llm/runtime/runtimeKernels.h" +#include "tensorrt_llm/runtime/tllmLogger.h" #include <chrono> #include <cmath> #include <memory> //@todo check the usage of this diff --git a/cpp/tests/unit_tests/kernels/sampling/samplingPenaltyTest.cpp b/cpp/tests/unit_tests/kernels/sampling/samplingPenaltyTest.cpp index a188abf6bc31..8896dd005cf7 100644 --- a/cpp/tests/unit_tests/kernels/sampling/samplingPenaltyTest.cpp +++ b/cpp/tests/unit_tests/kernels/sampling/samplingPenaltyTest.cpp @@ -14,7 +14,6 @@ * limitations under the License. */ -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/penaltyTypes.h" #include "tests/unit_tests/kernels/sampling/samplingTest.h" @@ -162,14 +161,14 @@ class TemperaturePenaltyTest : public SamplingKernelTest<T> mLogitsPtrs = BufferManager::pinned(ITensor::makeShape({mBatchSize}), ptrType); mPenaltyWorkspaceDevice = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, mMaxTokensPerStep, mVocabSize * 2}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mMaxBatchSize, mMaxTokensPerStep, mVocabSize * 2}), nvinfer1::DataType::kINT32); - mTokensPerStep = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mTokensPerStep = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mBiasHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize, mVocabSizePadded}), dataType); mBiasDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mVocabSizePadded}), dataType); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), nvinfer1::DataType::kINT32); trk::invokeFill(*mLogitsRefHost, T{0.0f}, *mStream); trk::invokeFill(*mOutLogitsDevice, T{0.0f}, *mStream); @@ -205,7 +204,7 @@ class TemperaturePenaltyTest : public SamplingKernelTest<T> ASSERT_EQ(param.temperaturesSize, mMaxBatchSize) << "Invalid test configuration."; mTemperaturesDevice - = mBufferManager->gpu(ITensor::makeShape({param.temperaturesSize}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({param.temperaturesSize}), nvinfer1::DataType::kFLOAT); mBufferManager->copy(*param.temperatures, *mTemperaturesDevice); } @@ -282,8 +281,7 @@ TYPED_TEST(TemperaturePenaltyTest, NoPenalty) { int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; - TensorPtr temperaturesHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + TensorPtr temperaturesHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*temperaturesHost)[i] = 1.0f; @@ -299,8 +297,7 @@ TYPED_TEST(TemperaturePenaltyTest, LessThanOne) { int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; - TensorPtr temperaturesHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + TensorPtr temperaturesHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*temperaturesHost)[i] = 0.53f; @@ -316,8 +313,7 @@ TYPED_TEST(TemperaturePenaltyTest, GreaterThaneOne) { int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; - TensorPtr temperaturesHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + TensorPtr temperaturesHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*temperaturesHost)[i] = 2.01f; @@ -333,8 +329,7 @@ TYPED_TEST(TemperaturePenaltyTest, Mixed) { int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; - TensorPtr temperaturesHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + TensorPtr temperaturesHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*temperaturesHost)[i] = 0.53f + 0.2f * i; @@ -350,8 +345,7 @@ TYPED_TEST(TemperaturePenaltyTest, LargeVocab) { int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; - TensorPtr temperaturesHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + TensorPtr temperaturesHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*temperaturesHost)[i] = 0.53f + 0.2f * i; @@ -367,8 +361,7 @@ TYPED_TEST(TemperaturePenaltyTest, LargeVocabTokensPerStep) { int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; - TensorPtr temperaturesHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + TensorPtr temperaturesHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*temperaturesHost)[i] = 1.f; // 0.53f + 0.2f * i; @@ -548,25 +541,25 @@ class RepetitionPenaltyTest : public SamplingKernelTest<T> mLogitsPtrs = BufferManager::pinned(ITensor::makeShape({mBatchSize}), ptrType); mPenaltyWorkspaceDevice = mBufferManager->gpu( - ITensor::makeShape({mBatchSize, mMaxTokensPerStep, mVocabSize * 2}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mBatchSize, mMaxTokensPerStep, mVocabSize * 2}), nvinfer1::DataType::kINT32); - mTokensPerStep = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mTokensPerStep = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mOutputIdsHost = BufferManager::pinned( - ITensor::makeShape({mMaxBatchSize, mSequenceLength}), tensorrt_llm::DataType::kINT32); + mOutputIdsHost + = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize, mSequenceLength}), nvinfer1::DataType::kINT32); mOutputIdsDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mSequenceLength}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mSequenceLength}), nvinfer1::DataType::kINT32); - mSeqLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mSeqLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mSeqLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mSeqLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mContextLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mContextLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mIdsPtrHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), ptrType); mIdsPtrDevice = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), ptrType); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), nvinfer1::DataType::kINT32); auto batchSlotsPtr = bufferCast<int32_t>(*mBatchSlots); for (SizeType32 bi = 0; bi < mBatchSize; ++bi) @@ -613,13 +606,13 @@ class RepetitionPenaltyTest : public SamplingKernelTest<T> ASSERT_EQ(param.frequencyPenaltiesSize, mMaxBatchSize) << "Invalid test configuration."; ASSERT_EQ(param.promptIgnoreLengthsSize, mMaxBatchSize) << "Invalid test configuration."; mRepetitionPenaltiesDevice - = mBufferManager->gpu(ITensor::makeShape({param.repetitionPenaltiesSize}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({param.repetitionPenaltiesSize}), nvinfer1::DataType::kFLOAT); mPresencePenaltiesDevice - = mBufferManager->gpu(ITensor::makeShape({param.presencePenaltiesSize}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({param.presencePenaltiesSize}), nvinfer1::DataType::kFLOAT); mFrequencyPenaltiesDevice - = mBufferManager->gpu(ITensor::makeShape({param.frequencyPenaltiesSize}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({param.frequencyPenaltiesSize}), nvinfer1::DataType::kFLOAT); mPromptIgnoreLengthsDevice - = mBufferManager->gpu(ITensor::makeShape({param.promptIgnoreLengthsSize}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({param.promptIgnoreLengthsSize}), nvinfer1::DataType::kINT32); mBufferManager->copy(*param.repetitionPenalties, *mRepetitionPenaltiesDevice); mBufferManager->copy(*param.presencePenalties, *mPresencePenaltiesDevice); mBufferManager->copy(*param.frequencyPenalties, *mFrequencyPenaltiesDevice); @@ -747,13 +740,13 @@ TYPED_TEST(RepetitionPenaltyTest, BatchNoPenalty) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 1.0f; @@ -780,13 +773,13 @@ TYPED_TEST(RepetitionPenaltyTest, BatchRepetitionLessThanOne) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 0.53f; @@ -813,13 +806,13 @@ TYPED_TEST(RepetitionPenaltyTest, BatchRepetitionGreaterThaneOne) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 2.01f; @@ -846,13 +839,13 @@ TYPED_TEST(RepetitionPenaltyTest, BatchRepetitionMixed) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 0.53 + i * 0.2f; @@ -879,13 +872,13 @@ TYPED_TEST(RepetitionPenaltyTest, BatchPresenceMixed) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 1.0f; @@ -912,13 +905,13 @@ TYPED_TEST(RepetitionPenaltyTest, BatchPresenceHasDefaultValueZero2) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 1.0f; @@ -945,13 +938,13 @@ TYPED_TEST(RepetitionPenaltyTest, BatchFrequencyMixed) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 1.0f; @@ -978,13 +971,13 @@ TYPED_TEST(RepetitionPenaltyTest, BatchFrequencyHasDefaultValueZero2) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 1.0f; @@ -1011,13 +1004,13 @@ TYPED_TEST(RepetitionPenaltyTest, PenaltyTypeRepetitionPresence) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 0.53 + i * 0.2f; @@ -1044,13 +1037,13 @@ TYPED_TEST(RepetitionPenaltyTest, PenaltyTypeRepetitionFrequency) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 0.53 + i * 0.2f; @@ -1077,13 +1070,13 @@ TYPED_TEST(RepetitionPenaltyTest, PenaltyTypePresenceFrequency) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 1.0f; @@ -1110,13 +1103,13 @@ TYPED_TEST(RepetitionPenaltyTest, PenaltyTypeFull) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 0.53 + i * 0.2f; @@ -1143,13 +1136,13 @@ TYPED_TEST(RepetitionPenaltyTest, PenaltyTypeFullTokensPerStep) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 0.53 + i * 0.2f; @@ -1177,13 +1170,13 @@ TYPED_TEST(RepetitionPenaltyTest, PenaltyTypeFullWithPartialPromptIgnore) int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 0.53 + i * 0.2f; @@ -1210,13 +1203,13 @@ TYPED_TEST(RepetitionPenaltyTest, PenaltyTypeFullTokensPerStepWithFullPromptIgno int32_t batchSize = 6; int32_t maxBatchSize = 2 * batchSize; TensorPtr repetitionPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr presencePenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr frequencyPenaltyHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); TensorPtr promptIgnoreLengthsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); for (int32_t i = 0; i < maxBatchSize; ++i) { bufferCast<float>(*repetitionPenaltyHost)[i] = 0.53 + i * 0.2f; @@ -1340,23 +1333,23 @@ class MinLengthPenaltyTest : public SamplingKernelTest<T> mLogitsPtrs = BufferManager::pinned(ITensor::makeShape({mBatchSize}), ptrType); mPenaltyWorkspaceDevice = mBufferManager->gpu( - ITensor::makeShape({mBatchSize, mMaxTokensPerStep, mVocabSize}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mBatchSize, mMaxTokensPerStep, mVocabSize}), nvinfer1::DataType::kINT32); - mTokensPerStep = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mTokensPerStep = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mSeqLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mSeqLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mSeqLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mSeqLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mContextLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mContextLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mMinLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mMinLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mMinLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mMinLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mEndIdsHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mEndIdsHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), nvinfer1::DataType::kINT32); auto batchSlotsPtr = bufferCast<int32_t>(*mBatchSlots); for (SizeType32 bi = 0; bi < mBatchSize; ++bi) @@ -1558,30 +1551,30 @@ class MinLengthPenaltyOOBSafetyTest : public SamplingKernelTest<T> } // Defines currentStep. - mSeqLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mSeqLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); initConstant<int>(bufferCast<int32_t>(*mSeqLengthHost), mMaxBatchSize, 3); - mSeqLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mSeqLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mBufferManager->copy(*mSeqLengthHost, *mSeqLengthDevice); // Defines inputLength. - mContextLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mContextLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); initConstant<int>(bufferCast<int32_t>(*mContextLengthHost), mMaxBatchSize, 2); - mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mBufferManager->copy(*mContextLengthHost, *mContextLengthDevice); // Defines minLength. - mMinLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mMinLengthHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); initConstant<int>(bufferCast<int32_t>(*mMinLengthHost), mMaxBatchSize, 10); - mMinLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mMinLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mBufferManager->copy(*mMinLengthHost, *mMinLengthDevice); // Defines endIds. - mEndIdsHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mEndIdsHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); initConstant<int>(bufferCast<int32_t>(*mEndIdsHost), mMaxBatchSize, -1); - mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mBufferManager->copy(*mEndIdsHost, *mEndIdsDevice); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), nvinfer1::DataType::kINT32); auto batchSlotsPtr = bufferCast<int32_t>(*mBatchSlots); for (SizeType32 bi = 0; bi < mBatchSize; ++bi) { diff --git a/cpp/tests/unit_tests/kernels/sampling/samplingTest.cpp b/cpp/tests/unit_tests/kernels/sampling/samplingTest.cpp index 1d4f19c45ad2..90da247f5203 100644 --- a/cpp/tests/unit_tests/kernels/sampling/samplingTest.cpp +++ b/cpp/tests/unit_tests/kernels/sampling/samplingTest.cpp @@ -14,7 +14,6 @@ * limitations under the License. */ #include "tests/unit_tests/kernels/sampling/samplingTest.h" -#include "tensorrt_llm/common/tllmDataType.h" namespace tensorrt_llm::tests::kernels::sampling { @@ -52,78 +51,75 @@ void SamplingKernelTest<T>::allocateBuffers(SamplingKernelTestParam const& param auto const ptrType = TRTDataType<T*>::value; // Allocate GPU data - mSeqLengthsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); - mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + mSeqLengthsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); mFinishedHost = BufferManager::pinned( ITensor::makeShape({maxBatchSize}), TRTDataType<tk::FinishedState::UnderlyingType>::value); mFinishedDevice = mBufferManager->gpu( ITensor::makeShape({maxBatchSize}), TRTDataType<tk::FinishedState::UnderlyingType>::value); - mOutputIdsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); - mOutputIdsDevice - = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + mOutputIdsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize, mMaxSeqLen}), nvinfer1::DataType::kINT32); + mOutputIdsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, mMaxSeqLen}), nvinfer1::DataType::kINT32); mProbsHost = BufferManager::pinned(ITensor::makeShape({batchSize, maxTokensPerStep, vocabSize}), dataType); mProbsDevice = mBufferManager->gpu(ITensor::makeShape({batchSize, maxTokensPerStep, vocabSize}), dataType); mProbsPtrsDevice - = BufferManager::pinned(ITensor::makeShape({batchSize, maxTokensPerStep}), tensorrt_llm::DataType::kINT64); + = BufferManager::pinned(ITensor::makeShape({batchSize, maxTokensPerStep}), nvinfer1::DataType::kINT64); - mCumLogProbsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + mCumLogProbsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); if (param.returnAllSelectedTokens) { SizeType32 maxTopK = param.topK == 0 ? vocabSize : param.topK; mOutputLogProbsDevice - = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, maxTopK}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, maxTopK}), nvinfer1::DataType::kFLOAT); } else { mOutputLogProbsDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxSeqLen, maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({mMaxSeqLen, maxBatchSize}), nvinfer1::DataType::kFLOAT); } mZeroParentIdsDevice - = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, maxTokensPerStep}), nvinfer1::DataType::kINT32); mLogitsHost = BufferManager::pinned(ITensor::makeShape({batchSize, maxTokensPerStep, vocabSize}), dataType); mLogProbsHost = BufferManager::pinned(ITensor::makeShape({batchSize, maxTokensPerStep, vocabSize}), dataType); mIdsPtrHost = BufferManager::pinned(ITensor::makeShape({2 * maxBatchSize}), ptrType); - mEndIdsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); - mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + mEndIdsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); - mTopPsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); - mTopPsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + mTopPsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); + mTopPsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); - mTopKsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); - mTopKsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + mTopKsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + mTopKsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); - mSkipDecodeHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kBOOL); - mSkipDecodeDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kBOOL); + mSkipDecodeHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kBOOL); + mSkipDecodeDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kBOOL); - mTokensPerStep = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + mTokensPerStep = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); - mExpectedCumLogProbsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + mExpectedCumLogProbsHost = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); if (param.returnAllSelectedTokens) { SizeType32 maxTopK = param.topK == 0 ? vocabSize : param.topK; mExpectedLogProbsHost - = BufferManager::pinned(ITensor::makeShape({maxBatchSize, maxTopK}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize, maxTopK}), nvinfer1::DataType::kFLOAT); } else { mExpectedLogProbsHost - = BufferManager::pinned(ITensor::makeShape({mMaxSeqLen, maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({mMaxSeqLen, maxBatchSize}), nvinfer1::DataType::kFLOAT); } mCurandStatesDevice - = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, sizeof(curandState_t)}), tensorrt_llm::DataType::kINT8); + = mBufferManager->gpu(ITensor::makeShape({maxBatchSize, sizeof(curandState_t)}), nvinfer1::DataType::kINT8); } template <typename T> @@ -498,7 +494,7 @@ void SamplingKernelTest<T>::runTest(SamplingKernelTestParam const& param) // Retrieve the workspace size of the sampling kernel. auto const workspaceSize = getWorkspaceSize(param); TensorPtr workspaceDevice - = mBufferManager->gpu(ITensor::makeShape({static_cast<int32_t>(workspaceSize)}), tensorrt_llm::DataType::kINT8); + = mBufferManager->gpu(ITensor::makeShape({static_cast<int32_t>(workspaceSize)}), nvinfer1::DataType::kINT8); // Call tested function sampling callTestedFunction(param, workspaceDevice); diff --git a/cpp/tests/unit_tests/kernels/sampling/samplingTest.h b/cpp/tests/unit_tests/kernels/sampling/samplingTest.h index 74268bee092c..0c7f52ba369e 100644 --- a/cpp/tests/unit_tests/kernels/sampling/samplingTest.h +++ b/cpp/tests/unit_tests/kernels/sampling/samplingTest.h @@ -27,6 +27,7 @@ #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaStream.h" #include "tensorrt_llm/runtime/runtimeKernels.h" +#include "tensorrt_llm/runtime/tllmLogger.h" namespace tensorrt_llm::tests::kernels::sampling { diff --git a/cpp/tests/unit_tests/kernels/sampling/samplingUtilsTest.cu b/cpp/tests/unit_tests/kernels/sampling/samplingUtilsTest.cu index f0f7690257e7..71a6d767171c 100644 --- a/cpp/tests/unit_tests/kernels/sampling/samplingUtilsTest.cu +++ b/cpp/tests/unit_tests/kernels/sampling/samplingUtilsTest.cu @@ -14,7 +14,6 @@ * limitations under the License. */ -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/gptDecoder.h" #include "tests/unit_tests/kernels/sampling/samplingTest.h" #include <random> @@ -58,8 +57,7 @@ TEST_F(SamplingUtilsKernelTest, CurandInitialize) sync_check_cuda_error(this->mStream->get()); // Generate random numbers using initialized curand states.MemoryType - auto randValsDevice - = this->mBufferManager->gpu(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + auto randValsDevice = this->mBufferManager->gpu(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); generateRandomNumber<<<1, batchSize, 0, this->mStream->get()>>>( bufferCast<int32_t>(*randValsDevice), batchSlotsPtr, curandStates, batchSize); auto randValsHost = this->mBufferManager->copyFrom(*randValsDevice, MemoryType::kCPU); @@ -99,7 +97,7 @@ TEST_F(SamplingUtilsKernelTest, CurandBatchInitialize) curandState_t* curandStates; cudaMalloc(&curandStates, sizeof(curandState_t) * 2 * batchSize); - auto randomSeedsHost = mBufferManager->pinnedPool(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT64); + auto randomSeedsHost = mBufferManager->pinnedPool(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT64); auto randomSeedsHostPtr = bufferCast<int64_t>(*randomSeedsHost); size_t const periodSize = 3; for (size_t i = 0; i < batchSize; ++i) @@ -108,7 +106,7 @@ TEST_F(SamplingUtilsKernelTest, CurandBatchInitialize) } auto randomSeedsDevice = mBufferManager->copyFrom(*randomSeedsHost, MemoryType::kGPU); - auto batchSlots = mBufferManager->pinnedPool(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + auto batchSlots = mBufferManager->pinnedPool(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); auto batchSlotsPtr = bufferCast<SizeType32>(*batchSlots); for (SizeType32 bi = 0; bi < batchSize; ++bi) @@ -122,7 +120,7 @@ TEST_F(SamplingUtilsKernelTest, CurandBatchInitialize) sync_check_cuda_error(mStream->get()); // Generate random numbers using initialized curand states. - auto randValsDevice = mBufferManager->gpu(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + auto randValsDevice = mBufferManager->gpu(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); generateRandomNumber<<<1, batchSize, 0, this->mStream->get()>>>( bufferCast<SizeType32>(*randValsDevice), batchSlotsPtr, curandStates, batchSize); auto const randValsHost = mBufferManager->copyFrom(*randValsDevice, MemoryType::kCPU); @@ -168,26 +166,25 @@ public: ITensor::makeShape({batchSize, maxBeamWidth, vocabSizePadded}), dataType); ITensor::SharedPtr logitsHostPtrs = this->mBufferManager->pinnedPool(ITensor::makeShape({batchSize}), ptrType); auto refLogitsHost = this->mBufferManager->pinnedPool( - ITensor::makeShape({batchSize, maxBeamWidth, vocabSizePadded}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({batchSize, maxBeamWidth, vocabSizePadded}), nvinfer1::DataType::kFLOAT); auto refEntropyHost = this->mBufferManager->pinnedPool( - ITensor::makeShape({maxBatchSize, maxBeamWidth}), tensorrt_llm::DataType::kFLOAT); - auto entropyDevice = this->mBufferManager->gpu( - ITensor::makeShape({maxBatchSize, maxBeamWidth}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({maxBatchSize, maxBeamWidth}), nvinfer1::DataType::kFLOAT); + auto entropyDevice + = this->mBufferManager->gpu(ITensor::makeShape({maxBatchSize, maxBeamWidth}), nvinfer1::DataType::kFLOAT); auto biasHost = this->mBufferManager->pinnedPool(ITensor::makeShape({vocabSize}), dataType); auto temperatureHost - = this->mBufferManager->pinnedPool(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = this->mBufferManager->pinnedPool(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kFLOAT); auto endIdsHost - = this->mBufferManager->pinnedPool(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = this->mBufferManager->pinnedPool(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); auto beamWidthsHost - = this->mBufferManager->pinnedPool(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = this->mBufferManager->pinnedPool(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); ITensor::SharedPtr finishedHost = this->mBufferManager->pinnedPool( ITensor::makeShape({maxBeamWidth, maxBatchSize}), TRTDataType<tk::FinishedState::UnderlyingType>::value); - auto batchSlots - = this->mBufferManager->pinnedPool(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + auto batchSlots = this->mBufferManager->pinnedPool(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); auto batchSlotsPtr = bufferCast<int32_t>(*batchSlots); auto beamWidthsHostPtr = bufferCast<SizeType32>(*beamWidthsHost); diff --git a/cpp/tests/unit_tests/kernels/shiftKCacheKernelTest.cu b/cpp/tests/unit_tests/kernels/shiftKCacheKernelTest.cu index 75fede666434..b1b3bd6234bf 100644 --- a/cpp/tests/unit_tests/kernels/shiftKCacheKernelTest.cu +++ b/cpp/tests/unit_tests/kernels/shiftKCacheKernelTest.cu @@ -2,7 +2,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/decoderMaskedMultiheadAttentionUtils.h" #include "tensorrt_llm/kernels/gptKernels.h" #include "tensorrt_llm/kernels/kvCacheUtils.h" @@ -198,37 +197,37 @@ public: std::vector<int32_t> const& tokenSeqIdxs) { // allocate buffer - mSeqLengthsHost = mBufferManager->pinned(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); - mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + mSeqLengthsHost = mBufferManager->pinned(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); + mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); - mInputLengthsHost = mBufferManager->pinned(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); - mInputLengthsDevice = mBufferManager->gpu(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + mInputLengthsHost = mBufferManager->pinned(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); + mInputLengthsDevice = mBufferManager->gpu(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); - mKScaleQuantOrigDevice = mBufferManager->gpu(ITensor::makeShape({1}), tensorrt_llm::DataType::kFLOAT); + mKScaleQuantOrigDevice = mBufferManager->gpu(ITensor::makeShape({1}), nvinfer1::DataType::kFLOAT); mTokenReadIdxsHost = mBufferManager->pinned( - ITensor::makeShape({static_cast<int>(tokenReadIdxs.size())}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({static_cast<int>(tokenReadIdxs.size())}), nvinfer1::DataType::kINT32); mTokenReadIdxsDevice = mBufferManager->gpu( - ITensor::makeShape({static_cast<int>(tokenReadIdxs.size())}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({static_cast<int>(tokenReadIdxs.size())}), nvinfer1::DataType::kINT32); mTokenWriteIdxsHost = mBufferManager->pinned( - ITensor::makeShape({static_cast<int>(tokenWriteIdxs.size())}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({static_cast<int>(tokenWriteIdxs.size())}), nvinfer1::DataType::kINT32); mTokenWriteIdxsDevice = mBufferManager->gpu( - ITensor::makeShape({static_cast<int>(tokenWriteIdxs.size())}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({static_cast<int>(tokenWriteIdxs.size())}), nvinfer1::DataType::kINT32); mTokenPosIdxsHost = mBufferManager->pinned( - ITensor::makeShape({static_cast<int>(tokenPosIdxs.size())}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({static_cast<int>(tokenPosIdxs.size())}), nvinfer1::DataType::kINT32); mTokenPosIdxsDevice = mBufferManager->gpu( - ITensor::makeShape({static_cast<int>(tokenPosIdxs.size())}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({static_cast<int>(tokenPosIdxs.size())}), nvinfer1::DataType::kINT32); mTokenSeqIdxsHost = mBufferManager->pinned( - ITensor::makeShape({static_cast<int>(tokenSeqIdxs.size())}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({static_cast<int>(tokenSeqIdxs.size())}), nvinfer1::DataType::kINT32); mTokenSeqIdxsDevice = mBufferManager->gpu( - ITensor::makeShape({static_cast<int>(tokenSeqIdxs.size())}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({static_cast<int>(tokenSeqIdxs.size())}), nvinfer1::DataType::kINT32); - // tensorrt_llm::DataType dataType = tensorrt_llm::DataType::kHALF - // tensorrt_llm::DataType::kHALF - // tensorrt_llm::DataType::kBF16 + // nvinfer1::DataType dataType = nvinfer1::DataType::kHALF + // nvinfer1::DataType::kHALF + // nvinfer1::DataType::kBF16 int32_t batchBeam = batchSize * beamWidth; if (pagedKvCache) { diff --git a/cpp/tests/unit_tests/kernels/sparseAttentionKernelsTest.cpp b/cpp/tests/unit_tests/kernels/sparseAttentionKernelsTest.cpp index 14aa58f04df5..508d157e1f61 100644 --- a/cpp/tests/unit_tests/kernels/sparseAttentionKernelsTest.cpp +++ b/cpp/tests/unit_tests/kernels/sparseAttentionKernelsTest.cpp @@ -1,6 +1,5 @@ #include <gtest/gtest.h> -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/sparseAttentionKernels.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaStream.h" @@ -41,29 +40,28 @@ TEST_F(sparseAttentionKernelsTest, GatherKvPageOffsetsKernelTest) constexpr int total_sparse_tokens = 14; // Create input buffers - auto kv_page_offsets = mBufferManager->gpu( - ITensor::makeShape({batch_size, 2, max_num_pages_per_seq}), tensorrt_llm::DataType::kINT32); - auto seq_lengths = mBufferManager->gpu(ITensor::makeShape({batch_size}), tensorrt_llm::DataType::kINT32); + auto kv_page_offsets + = mBufferManager->gpu(ITensor::makeShape({batch_size, 2, max_num_pages_per_seq}), nvinfer1::DataType::kINT32); + auto seq_lengths = mBufferManager->gpu(ITensor::makeShape({batch_size}), nvinfer1::DataType::kINT32); // Shape: [num_head_kv, total_sparse_tokens] - flattened across all batches auto sparse_indices - = mBufferManager->gpu(ITensor::makeShape({num_head_kv, total_sparse_tokens}), tensorrt_llm::DataType::kINT32); - auto sparse_indices_offsets - = mBufferManager->gpu(ITensor::makeShape({batch_size + 1}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({num_head_kv, total_sparse_tokens}), nvinfer1::DataType::kINT32); + auto sparse_indices_offsets = mBufferManager->gpu(ITensor::makeShape({batch_size + 1}), nvinfer1::DataType::kINT32); // Create output buffers auto output_kv_page_offsets = mBufferManager->gpu( - ITensor::makeShape({num_head_kv, batch_size, 2, max_num_pages_per_seq}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({num_head_kv, batch_size, 2, max_num_pages_per_seq}), nvinfer1::DataType::kINT32); auto output_seq_lengths - = mBufferManager->gpu(ITensor::makeShape({num_head_kv, batch_size}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({num_head_kv, batch_size}), nvinfer1::DataType::kINT32); // Create pinned host buffers for data initialization auto kv_page_offsets_host = mBufferManager->pinned( - ITensor::makeShape({batch_size, 2, max_num_pages_per_seq}), tensorrt_llm::DataType::kINT32); - auto seq_lengths_host = mBufferManager->pinned(ITensor::makeShape({batch_size}), tensorrt_llm::DataType::kINT32); - auto sparse_indices_host = mBufferManager->pinned( - ITensor::makeShape({num_head_kv, total_sparse_tokens}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({batch_size, 2, max_num_pages_per_seq}), nvinfer1::DataType::kINT32); + auto seq_lengths_host = mBufferManager->pinned(ITensor::makeShape({batch_size}), nvinfer1::DataType::kINT32); + auto sparse_indices_host + = mBufferManager->pinned(ITensor::makeShape({num_head_kv, total_sparse_tokens}), nvinfer1::DataType::kINT32); auto sparse_indices_offsets_host - = mBufferManager->pinned(ITensor::makeShape({batch_size + 1}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({batch_size + 1}), nvinfer1::DataType::kINT32); // Initialize test data auto kv_page_offsets_ptr = bufferCast<int32_t>(*kv_page_offsets_host); @@ -145,9 +143,9 @@ TEST_F(sparseAttentionKernelsTest, GatherKvPageOffsetsKernelTest) // Copy results back to host for verification auto output_kv_page_offsets_host = mBufferManager->pinned( - ITensor::makeShape({num_head_kv, batch_size, 2, max_num_pages_per_seq}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({num_head_kv, batch_size, 2, max_num_pages_per_seq}), nvinfer1::DataType::kINT32); auto output_seq_lengths_host - = mBufferManager->pinned(ITensor::makeShape({num_head_kv, batch_size}), tensorrt_llm::DataType::kINT32); + = mBufferManager->pinned(ITensor::makeShape({num_head_kv, batch_size}), nvinfer1::DataType::kINT32); mBufferManager->copy(*output_kv_page_offsets, *output_kv_page_offsets_host); mBufferManager->copy(*output_seq_lengths, *output_seq_lengths_host); diff --git a/cpp/tests/unit_tests/kernels/stopCriteriaKernelsTest.cpp b/cpp/tests/unit_tests/kernels/stopCriteriaKernelsTest.cpp index 9fe4737e82c0..2fefae39552e 100644 --- a/cpp/tests/unit_tests/kernels/stopCriteriaKernelsTest.cpp +++ b/cpp/tests/unit_tests/kernels/stopCriteriaKernelsTest.cpp @@ -16,7 +16,6 @@ #include "tensorrt_llm/kernels/stopCriteriaKernels.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/decodingCommon.h" #include "tensorrt_llm/runtime/bufferManager.h" @@ -61,35 +60,34 @@ class StopCriteriaKernelsTest : public testing::Test std::uniform_int_distribution<SizeType32> tokensPerStepDistr(1, mMaxTokensPerStep); mSequenceLengths - = BufferManager::pinned(ITensor::makeShape({maxBatchSize, beamWidth}), tensorrt_llm::DataType::kINT32); - mSequenceLengthLimits - = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize, beamWidth}), nvinfer1::DataType::kINT32); + mSequenceLengthLimits = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); mFinished = BufferManager::pinned( ITensor::makeShape({maxBatchSize, beamWidth}), TRTDataType<tk::FinishedState::UnderlyingType>::value); - mFinishedSum = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + mFinishedSum = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); mOutputIds = BufferManager::pinned( - ITensor::makeShape({maxBatchSize, beamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize, beamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); mOutputIdsPtr - = BufferManager::pinned(ITensor::makeShape({maxBatchSize, beamWidth}), tensorrt_llm::DataType::kINT64); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize, beamWidth}), nvinfer1::DataType::kINT64); mParentIds = BufferManager::pinned( - ITensor::makeShape({maxBatchSize, beamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize, beamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); mParentIdsPtr - = BufferManager::pinned(ITensor::makeShape({maxBatchSize, beamWidth}), tensorrt_llm::DataType::kINT64); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize, beamWidth}), nvinfer1::DataType::kINT64); mRefOutputIds = BufferManager::pinned( - ITensor::makeShape({maxBatchSize, beamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize, beamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); - mStopWords = BufferManager::pinned( - ITensor::makeShape({maxBatchSize, 2, maxStopWordsLen}), tensorrt_llm::DataType::kINT32); - mStopWordsPtr = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT64); - mStopWordsLen = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + mStopWords + = BufferManager::pinned(ITensor::makeShape({maxBatchSize, 2, maxStopWordsLen}), nvinfer1::DataType::kINT32); + mStopWordsPtr = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT64); + mStopWordsLen = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); - mEndIds = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); - mTokensPerStep = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), tensorrt_llm::DataType::kINT32); + mEndIds = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); + mTokensPerStep = BufferManager::pinned(ITensor::makeShape({maxBatchSize}), nvinfer1::DataType::kINT32); auto batchSlotsPtr = bufferCast<SizeType32>(*mBatchSlots); for (SizeType32 bi = 0; bi < batchSize; ++bi) diff --git a/cpp/tests/unit_tests/layers/baseSamplingLayerTest.cpp b/cpp/tests/unit_tests/layers/baseSamplingLayerTest.cpp index 7886a6b54e2b..8bdacb2e9f6c 100644 --- a/cpp/tests/unit_tests/layers/baseSamplingLayerTest.cpp +++ b/cpp/tests/unit_tests/layers/baseSamplingLayerTest.cpp @@ -15,7 +15,6 @@ */ #include "tests/unit_tests/layers/baseSamplingLayerTest.h" -#include "tensorrt_llm/common/tllmDataType.h" namespace tensorrt_llm::tests::layers::sampling { @@ -68,26 +67,26 @@ void BaseSamplingLayerTest<T>::setup(uint64_t seed, TestSamplingParams const& pa BaseSamplingLayerTest::mMaxOutputLen * params.beamWidth, mVocabSize); } - mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize()}), tensorrt_llm::DataType::kINT32); - mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize()}), tensorrt_llm::DataType::kINT32); + mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize()}), nvinfer1::DataType::kINT32); + mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize()}), nvinfer1::DataType::kINT32); mFinishedDevice = params.isExternalDraftTokensLayerTest ? mBufferManager->gpu(ITensor::makeShape({mMaxTokensPerEngineStep, maxBatchSize()}), TRTDataType<tk::FinishedState::UnderlyingType>::value) : mBufferManager->gpu( ITensor::makeShape({maxBatchSize()}), TRTDataType<tk::FinishedState::UnderlyingType>::value); - mOutputIdsDevice = mBufferManager->gpu( - ITensor::makeShape({maxBatchSize(), mBeamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); - mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize()}), tensorrt_llm::DataType::kINT32); + mOutputIdsDevice + = mBufferManager->gpu(ITensor::makeShape({maxBatchSize(), mBeamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); + mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize()}), nvinfer1::DataType::kINT32); mIdsPtrHost = mBufferManager->pinned(ITensor::makeShape({maxBatchSize()}), ptrType); - mCumLogProbsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize()}), tensorrt_llm::DataType::kFLOAT); + mCumLogProbsDevice = mBufferManager->gpu(ITensor::makeShape({maxBatchSize()}), nvinfer1::DataType::kFLOAT); mOutputLogProbsDevice - = mBufferManager->gpu(ITensor::makeShape({maxBatchSize(), mMaxSeqLen}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({maxBatchSize(), mMaxSeqLen}), nvinfer1::DataType::kFLOAT); mBatchSlots - = mBufferManager->pinned(ITensor::makeShape({mBatchSize + mBatchSizeBadPad}), tensorrt_llm::DataType::kINT32); - mCurandStatesDevice = mBufferManager->gpu( - ITensor::makeShape({maxBatchSize(), sizeof(curandState_t)}), tensorrt_llm::DataType::kINT8); + = mBufferManager->pinned(ITensor::makeShape({mBatchSize + mBatchSizeBadPad}), nvinfer1::DataType::kINT32); + mCurandStatesDevice + = mBufferManager->gpu(ITensor::makeShape({maxBatchSize(), sizeof(curandState_t)}), nvinfer1::DataType::kINT8); auto const workspaceSize = mSamplingLayer->getWorkspaceSize(); @@ -153,11 +152,11 @@ void BaseSamplingLayerTest<T>::setup(uint64_t seed, TestSamplingParams const& pa setupParams = samplingSetupParams; mSrcCacheIndirection = mBufferManager->gpu( - ITensor::makeShape({maxBatchSize(), mBeamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize(), mBeamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); mTgtCacheIndirection = mBufferManager->gpu( - ITensor::makeShape({maxBatchSize(), mBeamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize(), mBeamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); mParentIds = mBufferManager->gpu( - ITensor::makeShape({maxBatchSize(), mBeamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize(), mBeamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); auto constexpr nvTokenIdType = TRTDataType<TokenIdType>::value; auto constexpr nvSizeType = TRTDataType<SizeType32>::value; diff --git a/cpp/tests/unit_tests/layers/baseSamplingLayerTest.h b/cpp/tests/unit_tests/layers/baseSamplingLayerTest.h index 2bf782a5f5e8..5a375000a5e8 100644 --- a/cpp/tests/unit_tests/layers/baseSamplingLayerTest.h +++ b/cpp/tests/unit_tests/layers/baseSamplingLayerTest.h @@ -34,6 +34,7 @@ #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaStream.h" #include "tensorrt_llm/runtime/runtimeKernels.h" +#include "tensorrt_llm/runtime/tllmLogger.h" #include "tensorrt_llm/common/tllmException.h" diff --git a/cpp/tests/unit_tests/layers/dynamicDecodeLayerTest.cpp b/cpp/tests/unit_tests/layers/dynamicDecodeLayerTest.cpp index cc3b9c411f5b..a3c2d56de16e 100644 --- a/cpp/tests/unit_tests/layers/dynamicDecodeLayerTest.cpp +++ b/cpp/tests/unit_tests/layers/dynamicDecodeLayerTest.cpp @@ -15,7 +15,6 @@ */ #include "tests/unit_tests/layers/dynamicDecodeLayerTest.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/runtime/runtimeKernels.h" #include <algorithm> @@ -151,43 +150,43 @@ void DynamicDecodeLayerTest<T>::allocateData(TestSamplingParams const& params, T mRuntimeLogitsHost = BufferManager::pinned(ITensor::makeShape({mBatchSize, mBeamWidth, mVocabSizePadded}), dataType); - mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mContextLengthDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mFinishedDevice = mBufferManager->gpu( ITensor::makeShape({mMaxBatchSize}), TRTDataType<tk::FinishedState::UnderlyingType>::value); - mFinishedSumDevice = BufferManager::pinned(ITensor::makeShape({1}), tensorrt_llm::DataType::kFLOAT); - mOutputIdsDevice = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, mBeamWidth, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + mFinishedSumDevice = BufferManager::pinned(ITensor::makeShape({1}), nvinfer1::DataType::kFLOAT); + mOutputIdsDevice + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mBeamWidth, mMaxSeqLen}), nvinfer1::DataType::kINT32); mNewTokens - = BufferManager::pinned(ITensor::makeShape({mMaxTokensPerStep, mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({mMaxTokensPerStep, mMaxBatchSize}), nvinfer1::DataType::kINT32); + mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mEmbeddingBiasHost = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize, mVocabSizePadded}), dataType); mEmbeddingBiasDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mVocabSizePadded}), dataType); mRefLogProbsHost - = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize, mMaxSeqLen}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize, mMaxSeqLen}), nvinfer1::DataType::kFLOAT); mOutputLogProbsDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mMaxSeqLen}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mMaxSeqLen}), nvinfer1::DataType::kFLOAT); mOutputLogProbsTiledDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxSeqLen, mMaxBatchSize}), tensorrt_llm::DataType::kFLOAT); + = mBufferManager->gpu(ITensor::makeShape({mMaxSeqLen, mMaxBatchSize}), nvinfer1::DataType::kFLOAT); - mCumLogProbsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kFLOAT); + mCumLogProbsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kFLOAT); mMaxBadWordsLen = getMaxWordsLen(params.badWords); mMaxStopWordsLen = getMaxWordsLen(params.stopWords); - mBadWords = BufferManager::pinned( - ITensor::makeShape({mMaxBatchSize, 2, mMaxBadWordsLen}), tensorrt_llm::DataType::kINT32); - mBadWordsLens = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mBadWordsPtrs = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT64); + mBadWords + = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize, 2, mMaxBadWordsLen}), nvinfer1::DataType::kINT32); + mBadWordsLens = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mBadWordsPtrs = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT64); - mStopWords = BufferManager::pinned( - ITensor::makeShape({mMaxBatchSize, 2, mMaxStopWordsLen}), tensorrt_llm::DataType::kINT32); - mStopWordsLens = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mStopWordsPtrs = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT64); + mStopWords + = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize, 2, mMaxStopWordsLen}), nvinfer1::DataType::kINT32); + mStopWordsLens = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mStopWordsPtrs = BufferManager::pinned(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT64); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), nvinfer1::DataType::kINT32); if (mDecodingMode.isMedusa()) { @@ -205,19 +204,19 @@ void DynamicDecodeLayerTest<T>::allocateMedusaData(TestSamplingParams const& par auto const dataType = TRTDataType<T>::value; mMaxMedusaHeads = params.maxNumMedusaHeads.value(); mPathsDevice = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, mMaxTokensPerStep, mMaxMedusaHeads + 1}), tensorrt_llm::DataType::kINT32); - mAcceptedLengths = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mMaxBatchSize, mMaxTokensPerStep, mMaxMedusaHeads + 1}), nvinfer1::DataType::kINT32); + mAcceptedLengths = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mMedusaLogitsDevice = BufferManager::pinned( ITensor::makeShape({mMaxMedusaHeads, mMaxBatchSize, mMaxTokensPerStep, mVocabSizePadded}), dataType); - mNextDraftTokensDevice = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, mMaxTokensPerStep - 1}), tensorrt_llm::DataType::kINT32); - mTokensPerStepDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); - mTreeIdsDevice = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, mMaxTokensPerStep - 1}), tensorrt_llm::DataType::kINT32); + mNextDraftTokensDevice + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mMaxTokensPerStep - 1}), nvinfer1::DataType::kINT32); + mTokensPerStepDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); + mTreeIdsDevice + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mMaxTokensPerStep - 1}), nvinfer1::DataType::kINT32); mAcceptedLengthCumSumDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize + 1}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize + 1}), nvinfer1::DataType::kINT32); mPackedPathsDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize * mMaxMedusaHeads}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize * mMaxMedusaHeads}), nvinfer1::DataType::kINT32); } template <typename T> @@ -605,7 +604,7 @@ template <typename T> void DynamicDecodeLayerTest<T>::batchCopy(SizeType32 step) { auto const logitsHost = ITensor::wrap(mTestLogitsInit.data() + step * mVocabSizePadded, - std::is_same_v<T, float> ? tensorrt_llm::DataType::kFLOAT : tensorrt_llm::DataType::kHALF, + std::is_same_v<T, float> ? nvinfer1::DataType::kFLOAT : nvinfer1::DataType::kHALF, ITensor::makeShape({mMaxTokensPerStep, mVocabSizePadded})); for (SizeType32 bi = 0; bi < mBatchSize; ++bi) { diff --git a/cpp/tests/unit_tests/layers/eagleLayerTest.cpp b/cpp/tests/unit_tests/layers/eagleLayerTest.cpp index 47bca93b47b4..bdb53f15618f 100644 --- a/cpp/tests/unit_tests/layers/eagleLayerTest.cpp +++ b/cpp/tests/unit_tests/layers/eagleLayerTest.cpp @@ -22,8 +22,9 @@ #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/runtimeKernels.h" #include "tensorrt_llm/runtime/speculativeDecodingModule.h" +#include "tensorrt_llm/runtime/tllmLogger.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntimeBase.h> #include <algorithm> #include <cstdint> @@ -554,126 +555,126 @@ void EagleDecodingLayerTest<T>::allocateBuffers() // outputs mOutputIds = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxSeqLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mSeqLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mOutputNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingDraftTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mOutputUnpackedNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingDraftTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mAcceptedLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mNextPosIds = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mPrevDraftLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mNextDraftLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mNextGenerationLengths - = mBufferManager->gpu(ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mNextGenerationLengthsHost = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mAcceptedLengthCumSum = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize() + 1}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize() + 1}), nvinfer1::DataType::kINT32); mPathsOffsets = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize() * mSamplingParams.getMaxDraftPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mPackedMasks = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens(), static_cast<SizeType32>(divUp(mSamplingParams.getMaxDecodingTokens(), 32))}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mRandomDataSample = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kFLOAT); mRandomDataValidation = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kFLOAT); + nvinfer1::DataType::kFLOAT); mOutputTemperatures = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kFLOAT); mOutputNextDraftPaths = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mEagleNetCtxRequestTypesHost = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mEagleNetCtxContextLengthsHost = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mEagleNetCtxPastKeyValueLengthsHost = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mEagleNetGenRequestTypesHost = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mEagleNetGenContextLengthsHost = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mEagleNetGenPastKeyValueLengthsHost = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); // inputs - mBatchSlots = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mBatchSlots + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); - mEndIds = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mEndIds + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mInputNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingDraftTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); - mInputNextDraftLens = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mInputNextDraftLens + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mInputNextDraftPaths = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mInputLastDraftTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingDraftTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); - mInputLastDraftLens = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mInputLastDraftLens + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mInputLastDraftPaths = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mInputAcceptedTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); - mInputAcceptedLens = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mInputAcceptedLens + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); - mInputAcceptedPathIds = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mInputAcceptedPathIds + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); - mChunkedContextNextTokens = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mChunkedContextNextTokens + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mDecodingWorkspace = std::make_shared<tensorrt_llm::runtime::DecodingLayerWorkspace>(mBufferManager, decodingDomain, TRTDataType<float>::value, mSamplingParams.getMaxBatchSize() * sizeof(curandState_t)); diff --git a/cpp/tests/unit_tests/layers/explicitDraftTokensLayerTest.cpp b/cpp/tests/unit_tests/layers/explicitDraftTokensLayerTest.cpp index 04d05e0d16a9..e7831b57f77f 100644 --- a/cpp/tests/unit_tests/layers/explicitDraftTokensLayerTest.cpp +++ b/cpp/tests/unit_tests/layers/explicitDraftTokensLayerTest.cpp @@ -22,8 +22,9 @@ #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/runtimeKernels.h" #include "tensorrt_llm/runtime/speculativeDecodingModule.h" +#include "tensorrt_llm/runtime/tllmLogger.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntimeBase.h> #include <algorithm> #include <cstdint> @@ -654,29 +655,29 @@ void ExplicitDraftTokensLayerTest<T>::allocateBuffers() // outputs mOutputIds = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxSeqLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mSeqLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mAcceptedLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mNextDraftLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mPrevDraftLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mAcceptedLengthCumSum = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize() + 1}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize() + 1}), nvinfer1::DataType::kINT32); mOutputNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingDraftTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mOutputPositionIdsBase = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mRandomDataSample = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), dataType); @@ -688,21 +689,21 @@ void ExplicitDraftTokensLayerTest<T>::allocateBuffers() mPackedMasks = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens(), static_cast<SizeType32>(divUp(mSamplingParams.getMaxDecodingTokens(), 32))}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mNextPosIds = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mOutputUnpackedNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mOutputUnpackedNextDraftIndices = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mOutputDraftProbs = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxNumPaths(), @@ -712,68 +713,68 @@ void ExplicitDraftTokensLayerTest<T>::allocateBuffers() mOutputTemperatures = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), dataType); mOutputGenerationLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mOutputGenerationLengthsHost = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); - mMaxGenLengthHost = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + mMaxGenLengthHost = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); // inputs - mBatchSlots = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mBatchSlots + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mTokensPerStep = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mPathsOffsets = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize() * mSamplingParams.getMaxDraftPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mMasks = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kBOOL); + nvinfer1::DataType::kBOOL); mInputNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mLastDraftTokens = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mPackedPosIds = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mBestPathLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mBestPathIndices = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mSpecDecodingGenerationLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mNextFlatTokens = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize() * mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mInputPositionIdsBase = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); mNextDraftIndices = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mLastDraftIndices = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mNextDraftProbs = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getMaxBatchSize(), mSamplingParams.getMaxNumPaths(), @@ -781,20 +782,20 @@ void ExplicitDraftTokensLayerTest<T>::allocateBuffers() dataType); mEndIds = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mSamplingParams.getMaxBatchSize()}), nvinfer1::DataType::kINT32); - mMaxGenLengthDevice = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + mMaxGenLengthDevice = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); // Packed inputs - mMaxGenerationLength = BufferManager::pinnedPool(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); - mCumSumGenerationLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mMaxGenerationLength = BufferManager::pinnedPool(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); + mCumSumGenerationLengths + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); // Packed outputs - mPackedPositionIdsBase = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); - mPackedGenerationLengths = BufferManager::pinnedPool( - ITensor::makeShape({mSamplingParams.getBatchSize()}), tensorrt_llm::DataType::kINT32); + mPackedPositionIdsBase + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); + mPackedGenerationLengths + = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), nvinfer1::DataType::kINT32); mPackedRandomDataSample = BufferManager::pinnedPool(ITensor::makeShape({mSamplingParams.getBatchSize()}), dataType); mPackedRandomDataVerification = BufferManager::pinnedPool( ITensor::makeShape( @@ -803,21 +804,21 @@ void ExplicitDraftTokensLayerTest<T>::allocateBuffers() mPackedNextDraftTokens = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mPackedNextDraftIndices = BufferManager::pinnedPool( ITensor::makeShape( {mSamplingParams.getBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxPathLen()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mPackedPackedMasks = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens(), static_cast<SizeType32>(divUp(mSamplingParams.getMaxDecodingTokens(), 32))}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mPackedPositionOffsets = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mPackedPackedPosIds = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxDecodingTokens()}), - tensorrt_llm::DataType::kINT32); + nvinfer1::DataType::kINT32); mPackedDraftProbs = BufferManager::pinnedPool( ITensor::makeShape({mSamplingParams.getBatchSize(), mSamplingParams.getMaxNumPaths(), mSamplingParams.getMaxDraftPathLen(), mSamplingParams.getVocabSize()}), @@ -1563,6 +1564,7 @@ class FillRandDataTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro- void SetUp() override { + mLogger = std::make_shared<TllmLogger>(); mStream = std::make_shared<tensorrt_llm::runtime::CudaStream>(); mBufferManager = std::make_shared<tensorrt_llm::runtime::BufferManager>(mStream); } @@ -1574,12 +1576,12 @@ class FillRandDataTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro- { SizeType32* batchSlotsPtr{nullptr}; - auto curandState = mBufferManager->gpu(ITensor::makeShape({batchSize, 48}), tensorrt_llm::DataType::kUINT8); + auto curandState = mBufferManager->gpu(ITensor::makeShape({batchSize, 48}), nvinfer1::DataType::kUINT8); auto* curandStatePtr = reinterpret_cast<curandState_t*>(bufferCast<uint8_t>(*curandState)); if (batchInit) { - auto randomSeeds = mBufferManager->gpu(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT64); + auto randomSeeds = mBufferManager->gpu(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT64); trk::invokeFill(*randomSeeds, static_cast<int64_t>(randomSeed), *mStream); auto* randomSeedsPtr = bufferCast<uint64_t>(*randomSeeds); tk::invokeCurandBatchInitialize(curandStatePtr, batchSlotsPtr, batchSize, randomSeedsPtr, mStream->get()); @@ -1623,6 +1625,7 @@ class FillRandDataTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro- } private: + std::shared_ptr<nvinfer1::ILogger> mLogger; std::shared_ptr<tensorrt_llm::runtime::CudaStream> mStream; std::shared_ptr<tensorrt_llm::runtime::BufferManager> mBufferManager; }; diff --git a/cpp/tests/unit_tests/layers/externalDraftTokensLayerTest.cpp b/cpp/tests/unit_tests/layers/externalDraftTokensLayerTest.cpp index cdc913d3b3aa..c216a76bc2b8 100644 --- a/cpp/tests/unit_tests/layers/externalDraftTokensLayerTest.cpp +++ b/cpp/tests/unit_tests/layers/externalDraftTokensLayerTest.cpp @@ -15,7 +15,6 @@ */ #include "tensorrt_llm/common/assert.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/runtime/iBuffer.h" #include "tests/unit_tests/layers/baseSamplingLayerTest.h" @@ -89,7 +88,7 @@ class ExternalDraftTokensLayerTest : public BaseSamplingLayerTest<T> dataType); mDraftTokenIds = this->mBufferManager->gpu( - ITensor::makeShape({this->maxBatchSize(), mMaxDraftLen}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({this->maxBatchSize(), mMaxDraftLen}), nvinfer1::DataType::kINT32); mUseDraftLogits = this->mBufferManager->gpu(ITensor::makeShape({this->maxBatchSize()}), TRTDataType<bool>::value); mUseDraftLogitsHost diff --git a/cpp/tests/unit_tests/layers/lookaheadAlgorithmTest.cpp b/cpp/tests/unit_tests/layers/lookaheadAlgorithmTest.cpp index 7f4791a85ba4..cd1dc4799e6c 100644 --- a/cpp/tests/unit_tests/layers/lookaheadAlgorithmTest.cpp +++ b/cpp/tests/unit_tests/layers/lookaheadAlgorithmTest.cpp @@ -17,7 +17,6 @@ #include <tuple> #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/layers/lookaheadAlgorithm.h" #include "tensorrt_llm/layers/lookaheadDecodingUtils.h" @@ -88,25 +87,24 @@ TEST_P(LookaheadAlgorithmTest, predict) auto shape = ITensor::makeShape({maxTokensPerStep}); auto shape2d = ITensor::makeShape({maxTokensPerStep, maxTokensPerStep}); auto shapeSingle = ITensor::makeShape({1}); - TensorPtr posidMax = BufferManager::cpu(shape, tensorrt_llm::DataType::kINT32); - TensorPtr attentionMaskMax = BufferManager::cpu(shape2d, tensorrt_llm::DataType::kBOOL); - TensorPtr inputLengthPtr = BufferManager::cpu(shapeSingle, tensorrt_llm::DataType::kINT32); + TensorPtr posidMax = BufferManager::cpu(shape, nvinfer1::DataType::kINT32); + TensorPtr attentionMaskMax = BufferManager::cpu(shape2d, nvinfer1::DataType::kBOOL); + TensorPtr inputLengthPtr = BufferManager::cpu(shapeSingle, nvinfer1::DataType::kINT32); auto& inputLength(*BufferRange<SizeType32>(*inputLengthPtr).begin()); - TensorPtr outputMax = BufferManager::cpu(shape, tensorrt_llm::DataType::kINT32); - TensorPtr endIdPtr = BufferManager::cpu(shapeSingle, tensorrt_llm::DataType::kINT32); + TensorPtr outputMax = BufferManager::cpu(shape, nvinfer1::DataType::kINT32); + TensorPtr endIdPtr = BufferManager::cpu(shapeSingle, nvinfer1::DataType::kINT32); auto& endId(*BufferRange<TokenIdType>(*endIdPtr).begin()); endId = ascii->getEndToken(); - TensorPtr acceptedMax = BufferManager::cpu(shape, tensorrt_llm::DataType::kINT32); - TensorPtr acceptedOffsetsMax = BufferManager::cpu(shape, tensorrt_llm::DataType::kINT32); - TensorPtr acceptedLengthPtr = BufferManager::cpu(shapeSingle, tensorrt_llm::DataType::kINT32); + TensorPtr acceptedMax = BufferManager::cpu(shape, nvinfer1::DataType::kINT32); + TensorPtr acceptedOffsetsMax = BufferManager::cpu(shape, nvinfer1::DataType::kINT32); + TensorPtr acceptedLengthPtr = BufferManager::cpu(shapeSingle, nvinfer1::DataType::kINT32); auto& acceptedLength(*BufferRange<SizeType32>(*acceptedLengthPtr).begin()); - TensorPtr sequence - = BufferManager::cpu(ITensor::makeShape({maxSeqLen + maxDraftLen}), tensorrt_llm::DataType::kINT32); + TensorPtr sequence = BufferManager::cpu(ITensor::makeShape({maxSeqLen + maxDraftLen}), nvinfer1::DataType::kINT32); BufferRange<TokenIdType> sequenceRange(*sequence); - TensorPtr sequenceLengthPtr = BufferManager::cpu(shapeSingle, tensorrt_llm::DataType::kINT32); + TensorPtr sequenceLengthPtr = BufferManager::cpu(shapeSingle, nvinfer1::DataType::kINT32); auto& sequenceLength(*bufferCast<SizeType32>(*sequenceLengthPtr)); std::copy(promptRange.begin(), promptRange.end(), sequenceRange.begin()); @@ -226,13 +224,13 @@ TEST(LookaheadAlgorithmTest, treeEncodeTest) auto shape = inputTokens->getShape(); auto shape2d = ITensor::makeShape({shape.d[0], shape.d[0]}); - TensorPtr inputMasks = BufferManager::cpu(shape2d, tensorrt_llm::DataType::kBOOL); + TensorPtr inputMasks = BufferManager::cpu(shape2d, nvinfer1::DataType::kBOOL); LookaheadAlgorithm::posIdsToMask(inputMasks, inputPosIds); - TensorPtr outputTokens = BufferManager::cpu(shape, tensorrt_llm::DataType::kINT32); - TensorPtr outputPosIds = BufferManager::cpu(shape, tensorrt_llm::DataType::kINT32); - TensorPtr encodeMap = BufferManager::cpu(shape, tensorrt_llm::DataType::kINT32); - TensorPtr outputMasks = BufferManager::cpu(shape2d, tensorrt_llm::DataType::kBOOL); + TensorPtr outputTokens = BufferManager::cpu(shape, nvinfer1::DataType::kINT32); + TensorPtr outputPosIds = BufferManager::cpu(shape, nvinfer1::DataType::kINT32); + TensorPtr encodeMap = BufferManager::cpu(shape, nvinfer1::DataType::kINT32); + TensorPtr outputMasks = BufferManager::cpu(shape2d, nvinfer1::DataType::kBOOL); // auto len = LookaheadAlgorithm::treeEncode(outputTokens, outputPosIds, outputMasks, inputTokens, inputPosIds, // inputMasks, '$', 9); diff --git a/cpp/tests/unit_tests/layers/lookaheadDecodingLayerTest.cpp b/cpp/tests/unit_tests/layers/lookaheadDecodingLayerTest.cpp index 917f6dbdca55..414e6f101743 100644 --- a/cpp/tests/unit_tests/layers/lookaheadDecodingLayerTest.cpp +++ b/cpp/tests/unit_tests/layers/lookaheadDecodingLayerTest.cpp @@ -23,7 +23,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/executor.h" #include "tensorrt_llm/layers/decodingParams.h" #include "tensorrt_llm/layers/lookaheadDecodingLayer.h" @@ -321,7 +320,7 @@ void LookaheadDecodingLayerTest::allocateBuffers() mLlm[gbi] = std::make_shared<LookaheadRandomLlm>(mAscii, mOracle[gbi], gbi); mScoreBoard[gbi] = std::ostringstream(); - mHistogram[gbi] = BufferManager::cpu(ITensor::makeShape({mTestParam.n + 1}), tensorrt_llm::DataType::kINT32); + mHistogram[gbi] = BufferManager::cpu(ITensor::makeShape({mTestParam.n + 1}), nvinfer1::DataType::kINT32); } switch (mTestParam.batchType) { @@ -349,50 +348,48 @@ void LookaheadDecodingLayerTest::allocateBuffers() auto maxBatchShape1D = ITensor::makeShape({maxBatchSize}); - mAlgoConfigBatch = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, 3}), tensorrt_llm::DataType::kINT32); + mAlgoConfigBatch = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, 3}), nvinfer1::DataType::kINT32); - mEndIds = BufferManager::pinnedPool(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - mTokensPerStep = BufferManager::pinnedPool(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + mEndIds = BufferManager::pinnedPool(maxBatchShape1D, nvinfer1::DataType::kINT32); + mTokensPerStep = BufferManager::pinnedPool(maxBatchShape1D, nvinfer1::DataType::kINT32); - mOutputIds - = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, maxBeamSize, mMaxSeqLen + mMaxTokensPerStep}), - tensorrt_llm::DataType::kINT32); - mSequenceLengths = BufferManager::pinnedPool(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + mOutputIds = BufferManager::pinnedPool( + ITensor::makeShape({maxBatchSize, maxBeamSize, mMaxSeqLen + mMaxTokensPerStep}), nvinfer1::DataType::kINT32); + mSequenceLengths = BufferManager::pinnedPool(maxBatchShape1D, nvinfer1::DataType::kINT32); mProbs = BufferManager::pinnedPool( - ITensor::makeShape({maxBatchSize, mMaxTokensPerStep, vocabSize}), tensorrt_llm::DataType::kFLOAT); + ITensor::makeShape({maxBatchSize, mMaxTokensPerStep, vocabSize}), nvinfer1::DataType::kFLOAT); mGoldenSampledTokens - = BufferManager::cpu(ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), tensorrt_llm::DataType::kINT32); - mInputTokensBatch = BufferManager::pinnedPool( - ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), tensorrt_llm::DataType::kINT32); - mPositionIdsBatch = BufferManager::pinnedPool( - ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), tensorrt_llm::DataType::kINT32); + = BufferManager::cpu(ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), nvinfer1::DataType::kINT32); + mInputTokensBatch + = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), nvinfer1::DataType::kINT32); + mPositionIdsBatch + = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), nvinfer1::DataType::kINT32); mNewTokens = BufferManager::pinnedPool( - ITensor::makeShape({mMaxTokensPerStep, maxBatchSize, 1}), tensorrt_llm::DataType::kINT32); - mNumNewTokens = BufferManager::pinnedPool(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - mDraftLengths = BufferManager::pinnedPool(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - mPrevDraftLengths = BufferManager::pinnedPool(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mMaxTokensPerStep, maxBatchSize, 1}), nvinfer1::DataType::kINT32); + mNumNewTokens = BufferManager::pinnedPool(maxBatchShape1D, nvinfer1::DataType::kINT32); + mDraftLengths = BufferManager::pinnedPool(maxBatchShape1D, nvinfer1::DataType::kINT32); + mPrevDraftLengths = BufferManager::pinnedPool(maxBatchShape1D, nvinfer1::DataType::kINT32); mDraftTokens - = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, maxDraftLen}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, maxDraftLen}), nvinfer1::DataType::kINT32); auto packedMaskShape = ITensor::makeShape( {maxBatchSize, mMaxTokensPerStep, static_cast<ITensor::DimType64>(common::divUp(mMaxTokensPerStep, 32))}); - mPackedMasks = BufferManager::pinnedPool(packedMaskShape, tensorrt_llm::DataType::kINT32); + mPackedMasks = BufferManager::pinnedPool(packedMaskShape, nvinfer1::DataType::kINT32); mPackedMasksBool = BufferManager::pinnedPool( - ITensor::makeShape({maxBatchSize, mMaxTokensPerStep, mMaxTokensPerStep}), tensorrt_llm::DataType::kBOOL); - mNumNewTokensCumSum - = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize + 1}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize, mMaxTokensPerStep, mMaxTokensPerStep}), nvinfer1::DataType::kBOOL); + mNumNewTokensCumSum = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize + 1}), nvinfer1::DataType::kINT32); mPathsOffsets = BufferManager::pinnedPool( - ITensor::makeShape({maxBatchSize, maxAcceptedDraftLen}), tensorrt_llm::DataType::kINT32); - mGenerationLengths = BufferManager::pinnedPool(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - mPositionOffsets = BufferManager::pinnedPool( - ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), tensorrt_llm::DataType::kINT32); - mPositionIds = BufferManager::pinnedPool( - ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), tensorrt_llm::DataType::kINT32); - mAttentionPackedMask = BufferManager::pinnedPool(packedMaskShape, tensorrt_llm::DataType::kINT32); - - mBatchSlotsMax = BufferManager::pinnedPool(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + ITensor::makeShape({maxBatchSize, maxAcceptedDraftLen}), nvinfer1::DataType::kINT32); + mGenerationLengths = BufferManager::pinnedPool(maxBatchShape1D, nvinfer1::DataType::kINT32); + mPositionOffsets + = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), nvinfer1::DataType::kINT32); + mPositionIds + = BufferManager::pinnedPool(ITensor::makeShape({maxBatchSize, mMaxTokensPerStep}), nvinfer1::DataType::kINT32); + mAttentionPackedMask = BufferManager::pinnedPool(packedMaskShape, nvinfer1::DataType::kINT32); + + mBatchSlotsMax = BufferManager::pinnedPool(maxBatchShape1D, nvinfer1::DataType::kINT32); auto const batchSize = 0; auto batchShape1D = ITensor::makeShape({batchSize}); diff --git a/cpp/tests/unit_tests/layers/lookaheadRandomLlmTest.cpp b/cpp/tests/unit_tests/layers/lookaheadRandomLlmTest.cpp index 4ca7206c3bc8..2cc2523d08cb 100644 --- a/cpp/tests/unit_tests/layers/lookaheadRandomLlmTest.cpp +++ b/cpp/tests/unit_tests/layers/lookaheadRandomLlmTest.cpp @@ -15,7 +15,6 @@ */ #include <gtest/gtest.h> -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/samplingTopKKernels.h" #include "tensorrt_llm/layers/lookaheadAlgorithm.h" #include "tensorrt_llm/layers/lookaheadDecodingUtils.h" @@ -49,7 +48,7 @@ TEST(LookaheadRandomllm, forward) std::string str("hello world!"); TensorPtr logits = BufferManager::cpu(ITensor::makeShape({static_cast<SizeType32>(str.size()), ascii->getVocabSize()}), - tensorrt_llm::DataType::kFLOAT); + nvinfer1::DataType::kFLOAT); ascii->stringToLogits(logits, str); auto result = ascii->logitsToString(logits); EXPECT_EQ(result, str); @@ -68,7 +67,7 @@ TEST(LookaheadRandomllm, forward) std::vector<TokenIdType> positionIdVec({22, 23, 24, 23, 24, 25, 24, 25, 26, 25, 26, 27, 26, 27, 28}); TensorPtr positionIds = ITensor::wrap(positionIdVec, ITensor::makeShape({len})); TensorPtr outputLogits - = BufferManager::cpu(ITensor::makeShape({len, ascii->getVocabSize()}), tensorrt_llm::DataType::kFLOAT); + = BufferManager::cpu(ITensor::makeShape({len, ascii->getVocabSize()}), nvinfer1::DataType::kFLOAT); llm.forward(outputLogits, inputTokens, positionIds); @@ -124,29 +123,29 @@ TEST(LookaheadRandomllm, gpuSampling) SizeType32 workspaceSize = tensorrt_llm::kernels::getTopKWorkspaceSize<float>(maxBatchSize, maxTokensPerStep, mMaxTopK, vocabSizePadded); - TensorPtr workspaceDevice = mBufferManager->pinned( - ITensor::makeShape({static_cast<int32_t>(workspaceSize)}), tensorrt_llm::DataType::kINT8); + TensorPtr workspaceDevice + = mBufferManager->pinned(ITensor::makeShape({static_cast<int32_t>(workspaceSize)}), nvinfer1::DataType::kINT8); auto const dataType = TRTDataType<float>::value; auto const ptrType = TRTDataType<float*>::value; // Allocate GPU data - TensorPtr mSeqLengths = BufferManager::pinned(maxBatchShape1D, tensorrt_llm::DataType::kINT32); + TensorPtr mSeqLengths = BufferManager::pinned(maxBatchShape1D, nvinfer1::DataType::kINT32); TensorPtr mFinished = BufferManager::pinned(maxBatchShape1D, TRTDataType<tk::FinishedState::UnderlyingType>::value); - TensorPtr mEndIds = BufferManager::pinned(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - TensorPtr mTopPs = BufferManager::pinned(maxBatchShape1D, tensorrt_llm::DataType::kFLOAT); - TensorPtr mTopKs = BufferManager::pinned(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - TensorPtr mSkipDecode = BufferManager::pinned(maxBatchShape1D, tensorrt_llm::DataType::kBOOL); - TensorPtr mTokensPerStep = BufferManager::pinned(maxBatchShape1D, tensorrt_llm::DataType::kINT32); - - TensorPtr mCurandStates = BufferManager::pinned( - ITensor::makeShape({maxBatchSize, sizeof(curandState_t)}), tensorrt_llm::DataType::kINT8); + TensorPtr mEndIds = BufferManager::pinned(maxBatchShape1D, nvinfer1::DataType::kINT32); + TensorPtr mTopPs = BufferManager::pinned(maxBatchShape1D, nvinfer1::DataType::kFLOAT); + TensorPtr mTopKs = BufferManager::pinned(maxBatchShape1D, nvinfer1::DataType::kINT32); + TensorPtr mSkipDecode = BufferManager::pinned(maxBatchShape1D, nvinfer1::DataType::kBOOL); + TensorPtr mTokensPerStep = BufferManager::pinned(maxBatchShape1D, nvinfer1::DataType::kINT32); + + TensorPtr mCurandStates + = BufferManager::pinned(ITensor::makeShape({maxBatchSize, sizeof(curandState_t)}), nvinfer1::DataType::kINT8); TensorPtr mOutputIds - = BufferManager::pinned(ITensor::makeShape({maxBatchSize, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + = BufferManager::pinned(ITensor::makeShape({maxBatchSize, mMaxSeqLen}), nvinfer1::DataType::kINT32); TensorPtr mProbs = BufferManager::pinned(maxBatchShape3D, dataType); - TensorPtr mBatchSlots = BufferManager::pinned(batchShape1D, tensorrt_llm::DataType::kINT32); + TensorPtr mBatchSlots = BufferManager::pinned(batchShape1D, nvinfer1::DataType::kINT32); ///////////////////////////////////// std::copy(batchSlotsVec.begin(), batchSlotsVec.end(), BufferRange<SizeType32>(*mBatchSlots).begin()); diff --git a/cpp/tests/unit_tests/layers/medusaDecodeLayerTest.cpp b/cpp/tests/unit_tests/layers/medusaDecodeLayerTest.cpp index a93955d02c9a..37a479eb4214 100644 --- a/cpp/tests/unit_tests/layers/medusaDecodeLayerTest.cpp +++ b/cpp/tests/unit_tests/layers/medusaDecodeLayerTest.cpp @@ -15,7 +15,6 @@ */ #include "medusaDecodeLayerTest.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/decodingCommon.h" #include "tensorrt_llm/runtime/medusaModule.h" #include "tensorrt_llm/runtime/runtimeKernels.h" @@ -139,36 +138,35 @@ void MedusaDecodingLayerTest<T>::allocateBuffers() mFinishedDevice = mBufferManager->gpu( ITensor::makeShape({mMaxBatchSize}), TRTDataType<tk::FinishedState::UnderlyingType>::value); - mOutputIdsDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mMaxSeqLen}), tensorrt_llm::DataType::kINT32); + mOutputIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mMaxSeqLen}), nvinfer1::DataType::kINT32); - mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), tensorrt_llm::DataType::kINT32); + mBatchSlots = BufferManager::pinned(ITensor::makeShape({mBatchSize}), nvinfer1::DataType::kINT32); - mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mEndIdsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mPathsDevice = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, mMaxDecodingTokens, mMaxDraftPathLen + 1}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({mMaxBatchSize, mMaxDecodingTokens, mMaxDraftPathLen + 1}), nvinfer1::DataType::kINT32); - mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mSeqLengthsDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mAcceptedLengths = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mAcceptedLengths = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); - mTreeIdsDevice = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, mMaxDecodingTokens - 1}), tensorrt_llm::DataType::kINT32); + mTreeIdsDevice + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mMaxDecodingTokens - 1}), nvinfer1::DataType::kINT32); mMedusaLogitsDevice = mBufferManager->gpu( ITensor::makeShape({mMaxDraftPathLen, mMaxBatchSize, mMaxDecodingTokens, mVocabSizePadded}), dataType); - mNextDraftTokensDevice = mBufferManager->gpu( - ITensor::makeShape({mMaxBatchSize, mMaxDecodingTokens - 1}), tensorrt_llm::DataType::kINT32); + mNextDraftTokensDevice + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize, mMaxDecodingTokens - 1}), nvinfer1::DataType::kINT32); - mTokensPerStepDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), tensorrt_llm::DataType::kINT32); + mTokensPerStepDevice = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize}), nvinfer1::DataType::kINT32); mAcceptedLengthCumSumDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize + 1}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize + 1}), nvinfer1::DataType::kINT32); mPackedPathsDevice - = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize * mMaxDraftPathLen}), tensorrt_llm::DataType::kINT32); + = mBufferManager->gpu(ITensor::makeShape({mMaxBatchSize * mMaxDraftPathLen}), nvinfer1::DataType::kINT32); for (int32_t bi = 0; bi < mBatchSize; ++bi) { @@ -217,7 +215,7 @@ void MedusaDecodingLayerTest<T>::setup(SamplingParams& params) for (SizeType32 bi = 0; bi < mBatchSize; ++bi) { auto const draftIdsHost = ITensor::wrap(reinterpret_cast<TokenIdType*>(params.draftIds[bi].data()), - tensorrt_llm::DataType::kINT32, ITensor::makeShape({1, mMaxDecodingTokens - 1})); + nvinfer1::DataType::kINT32, ITensor::makeShape({1, mMaxDecodingTokens - 1})); auto draftIdsDeviceSlice = ITensor::slice(mNextDraftTokensDevice, batchSlotsPtr[bi], 1); mBufferManager->copy(*draftIdsHost, *draftIdsDeviceSlice); } @@ -226,7 +224,7 @@ void MedusaDecodingLayerTest<T>::setup(SamplingParams& params) { auto& path = params.paths[bi]; auto const numPaths = static_cast<SizeType32>(params.paths[bi].size() / (mMaxDraftPathLen + 1)); - auto const pathsHost = ITensor::wrap(reinterpret_cast<SizeType32*>(path.data()), tensorrt_llm::DataType::kINT32, + auto const pathsHost = ITensor::wrap(reinterpret_cast<SizeType32*>(path.data()), nvinfer1::DataType::kINT32, ITensor::makeShape({1, numPaths, mMaxDraftPathLen + 1})); TensorPtr pathsDeviceSlice = ITensor::slice(mPathsDevice, batchSlotsPtr[bi], 1); pathsDeviceSlice->squeeze(0); diff --git a/cpp/tests/unit_tests/layers/randomLlm.cpp b/cpp/tests/unit_tests/layers/randomLlm.cpp index 63aa85eaad16..632cf8765c20 100644 --- a/cpp/tests/unit_tests/layers/randomLlm.cpp +++ b/cpp/tests/unit_tests/layers/randomLlm.cpp @@ -16,7 +16,6 @@ #include "tests/unit_tests/layers/randomLlm.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/logger.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/layers/lookaheadDecodingUtils.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/common.h" @@ -35,7 +34,7 @@ TensorPtr initTensor(std::string str, std::optional<ITensor::Shape> shape) { TLLM_CHECK(ITensor::volume(shape1d) == ITensor::volume(shape.value())); } - TensorPtr tensor = BufferManager::cpu(shape.value_or(shape1d), tensorrt_llm::DataType::kINT32); + TensorPtr tensor = BufferManager::cpu(shape.value_or(shape1d), nvinfer1::DataType::kINT32); auto tensorRange = BufferRange<TokenIdType>(*tensor); std::copy(str.begin(), str.end(), tensorRange.begin()); return tensor; @@ -43,7 +42,7 @@ TensorPtr initTensor(std::string str, std::optional<ITensor::Shape> shape) TensorConstPtr RandomTokenLogits::tokenToLogits(TokenIdType token) const { - TensorPtr logits = BufferManager::cpu(mVocabulary->getShape(), tensorrt_llm::DataType::kFLOAT); + TensorPtr logits = BufferManager::cpu(mVocabulary->getShape(), nvinfer1::DataType::kFLOAT); tokenToLogits(logits, token); return logits; } @@ -153,7 +152,7 @@ void RandomTokenLogits::logitsToTensor(TensorPtr const& tokens, TensorConstPtr c TensorConstPtr RandomTokenLogits::logitsToTensor(TensorConstPtr const& logits) const { auto len = logits->getShape().d[0]; - TensorPtr result = BufferManager::cpu(ITensor::makeShape({len}), tensorrt_llm::DataType::kINT32); + TensorPtr result = BufferManager::cpu(ITensor::makeShape({len}), nvinfer1::DataType::kINT32); logitsToTensor(result, logits); return result; } @@ -210,7 +209,7 @@ bool RandomLlm::verify(SizeType32 const offset, TensorConstPtr const& script) co void RandomLlm::forward(TensorPtr const& output, runtime::SizeType32 startId, TensorConstPtr const& input, TensorConstPtr const& offsets, TensorConstPtr const mask) const { - TensorPtr posIds = BufferManager::cpu(input->getShape(), tensorrt_llm::DataType::kINT32); + TensorPtr posIds = BufferManager::cpu(input->getShape(), nvinfer1::DataType::kINT32); BufferRange<SizeType32> idRange(*posIds); BufferRange<SizeType32 const> offsetRange(*offsets); for (auto i = 0; i < idRange.size(); i++) @@ -227,7 +226,7 @@ void RandomLlm::forward(TensorPtr const& output, TensorConstPtr const& input, Te TLLM_CHECK(ITensor::volume(input->getShape()) == ITensor::volume(position->getShape())); TLLM_CHECK(ITensor::volume(output->getShape()) == ITensor::volume(input->getShape()) * mTable->getVocabSize()); - TensorPtr tokens = BufferManager::cpu(input->getShape(), tensorrt_llm::DataType::kINT32); + TensorPtr tokens = BufferManager::cpu(input->getShape(), nvinfer1::DataType::kINT32); foretell(tokens, input, position, mask); // foretellOld(tokens, input, position); mTable->tensorToLogits(output, tokens); @@ -248,7 +247,7 @@ void LookaheadRandomLlm::foretell(TensorPtr const& output, TensorConstPtr const& TLLM_CHECK(mask->getShape().d[1] >= len); } - TensorPtr maskRebuilt = BufferManager::cpu(ITensor::makeShape({len, len}), tensorrt_llm::DataType::kBOOL); + TensorPtr maskRebuilt = BufferManager::cpu(ITensor::makeShape({len, len}), nvinfer1::DataType::kBOOL); posIdsToMask(maskRebuilt, position); auto outputRange = BufferRange<TokenIdType>(*output); diff --git a/cpp/tests/unit_tests/layers/randomLlm.h b/cpp/tests/unit_tests/layers/randomLlm.h index b0f0564944dc..a6e898baedf5 100644 --- a/cpp/tests/unit_tests/layers/randomLlm.h +++ b/cpp/tests/unit_tests/layers/randomLlm.h @@ -18,7 +18,6 @@ #include <gtest/gtest.h> #include <list> -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/layers/lookaheadDecodingUtils.h" #include "tensorrt_llm/runtime/common.h" #include "tensorrt_llm/runtime/runtimeKernels.h" @@ -78,7 +77,7 @@ class AsciiRandomTokenLogits : public RandomTokenLogits : RandomTokenLogits( []() { - auto vocab = BufferManager::cpu(ITensor::makeShape({128}), tensorrt_llm::DataType::kINT32); + auto vocab = BufferManager::cpu(ITensor::makeShape({128}), nvinfer1::DataType::kINT32); auto vocabRange = BufferRange<TokenIdType>(*vocab); TokenIdType token{0}; std::for_each(vocabRange.begin(), vocabRange.end(), [&token](auto& v) { v = token++; }); diff --git a/cpp/tests/unit_tests/multi_gpu/cacheTransceiverTest.cpp b/cpp/tests/unit_tests/multi_gpu/cacheTransceiverTest.cpp index 1f216578f669..49afbf20dfae 100644 --- a/cpp/tests/unit_tests/multi_gpu/cacheTransceiverTest.cpp +++ b/cpp/tests/unit_tests/multi_gpu/cacheTransceiverTest.cpp @@ -36,7 +36,6 @@ #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/envUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/cache_transmission/agent_utils/connection.h" #include "tensorrt_llm/executor/cache_transmission/mpi_utils/connection.h" #include "tensorrt_llm/executor/dataTransceiverState.h" @@ -198,7 +197,7 @@ TEST_F(RequestInfoTest, Basic) } auto state = std::make_unique<texec::DataTransceiverState>(); state->setCommState(texec::kv_cache::CommState{12, "127.0.0.1"}); - state->setCacheState(texec::kv_cache::CacheState{10, 12, 128, 128, 8, 8, 8, {10}, tensorrt_llm::DataType::kFLOAT}); + state->setCacheState(texec::kv_cache::CacheState{10, 12, 128, 128, 8, 8, 8, {10}, nvinfer1::DataType::kFLOAT}); RequestInfo info{1, *state}; auto info2 = serializeDeserialize(info); EXPECT_EQ(info, info2); @@ -228,7 +227,7 @@ TEST_F(CacheConfigTest, EqualTo) constexpr SizeType32 nbRnnLayers{2}; constexpr SizeType32 nbHeads{12}; constexpr SizeType32 hiddenSize{768}; - constexpr tensorrt_llm::DataType dtype{tensorrt_llm::DataType::kFLOAT}; + constexpr nvinfer1::DataType dtype{nvinfer1::DataType::kFLOAT}; constexpr SizeType32 tokensPerBlock{64}; constexpr SizeType32 tensorParallelism{8}; constexpr SizeType32 pipelineParallelism{2}; @@ -286,7 +285,7 @@ class SymmetricalCacheTest : public ::testing::Test // NOLINT(cppcoreguidelines- return mWorldSize; } - void setUpCacheManager(bool enableBlockReuse = false) + void setUpCacheManager() { auto constexpr numLayers = 4; auto constexpr numHeads = 2; @@ -308,7 +307,8 @@ class SymmetricalCacheTest : public ::testing::Test // NOLINT(cppcoreguidelines- auto totalNumBlocks = mMaxNumSequences * numBlocksPerSeq; auto constexpr blocksInSecondaryPool = 0; - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr enableBlockReuse = false; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; using BlocksPerWindow = std::map<SizeType32, std::tuple<SizeType32, SizeType32>>; auto const blocksPerWindow = BlocksPerWindow{{maxAttentionWindow, {totalNumBlocks, blocksInSecondaryPool}}}; @@ -421,21 +421,6 @@ class SymmetricalCacheTest : public ::testing::Test // NOLINT(cppcoreguidelines- return std::make_unique<LlmRequest>(mRequestId++, std::move(request)); } - // Generation-only request whose DataTransceiverState carries the arbitrary-transfer - // provenance marker, as getSerializedDataTransceiverState would produce. - auto makeArbitraryLlmRequest(SizeType32 length, LlmRequest::RequestIdType arbitraryId) - { - constexpr SizeType32 maxNewTokens{1}; - texec::Request request{VecTokens(length, length), maxNewTokens}; - auto state = std::make_unique<texec::DataTransceiverState>(); - state->setCommState(*mContextCommState); - state->setCacheState(*mCacheState); - state->setIsArbitraryTransferState(true); - auto stats = texec::ContextPhaseParams({}, arbitraryId, state.release(), std::nullopt); - request.setContextPhaseParams(std::move(stats)); - return std::make_unique<LlmRequest>(arbitraryId, std::move(request)); - } - void addRequestAndTransportCache(std::shared_ptr<LlmRequest> const& llmRequest) { auto constexpr beamIdx{0}; @@ -533,83 +518,10 @@ TEST_F(SymmetricalCacheTest, SimpleTest) } } -TEST_F(SymmetricalCacheTest, ArbitraryTransferTest) -{ - auto worldSize = setUpCommunicator(); - if (worldSize != 2) - { - GTEST_SKIP() << "mpirun 2 processes is required to run this test."; - } - setUpCacheManager(/*enableBlockReuse=*/true); - setUpCacheTransceiver(); - - // 3 full blocks (tokensPerBlock = 8) so every requested chunk fully matches a stored block. - constexpr SizeType32 promptLen = 24; - constexpr SizeType32 missPromptLen = 40; - constexpr LlmRequest::RequestIdType arbitraryId = 4242; - auto constexpr beamIdx{0}; - auto constexpr beamWidth{1}; - - if (isSender) - { - // Store a patterned request in the reuse tree; no LlmRequest exists on the - // sender for the transfers below. - auto request = makeLlmRequest(promptLen); - mManager->addSequenceBatch( - {{{request->mRequestId, request->getNumTokens(beamIdx), beamWidth}}}, {std::ref(*request)}); - auto blockRange = BlockRange::fromAllBlockIds(*mManager, request->mRequestId); - for (auto const& windowSize : blockRange.getWindowSizes()) - { - auto blockRangeForWindow = blockRange.getBlockRangeForWindow(windowSize); - for (auto it = blockRangeForWindow.begin(); it != blockRangeForWindow.end(); ++it) - { - TLLM_CUDA_CHECK(cudaMemset(it->data(), request->getPromptLen(), it->getSizeInBytes())); - } - } - tensorrt_llm::testing::KvCacheManagerTestUtil::simulatePrefillCompletion(*request); - // A completed context request carries one generated token beyond the prompt; - // without it, storeBlocksForReuse drops the trailing prompt token and the - // final block never enters the reuse tree. - request->addNewToken(0, beamIdx); - mManager->removeSequence(request->mRequestId, request); - } - else - { - // Hit: the requested tokens are stored in the sender's reuse tree. - std::shared_ptr<LlmRequest> request = makeArbitraryLlmRequest(promptLen, arbitraryId); - mManager->addSequenceBatch( - {{{request->mRequestId, request->getNumTokens(beamIdx), beamWidth}}}, {std::ref(*request)}); - auto future = mRequester->receiveAsync(request); - future.get(); - TLLM_CUDA_CHECK(cudaDeviceSynchronize()); - auto blockRange = BlockRange::fromAllBlockIds(*mManager, request->mRequestId); - for (auto const& windowSize : blockRange.getWindowSizes()) - { - auto blockRangeForWindow = blockRange.getBlockRangeForWindow(windowSize); - for (auto it = blockRangeForWindow.begin(); it != blockRangeForWindow.end(); ++it) - { - std::vector<uint8_t> bytes(it->getSizeInBytes()); - TLLM_CUDA_CHECK(cudaMemcpy(bytes.data(), it->data(), it->getSizeInBytes(), cudaMemcpyDeviceToHost)); - EXPECT_TRUE(std::all_of(bytes.begin(), bytes.end(), [](uint8_t i) { return i == (promptLen & 0xff); })); - } - } - - // Miss: tokens never stored on the sender are rejected; the rejection - // surfaces as an exception on the receive future. - std::shared_ptr<LlmRequest> missRequest = makeArbitraryLlmRequest(missPromptLen, arbitraryId + 1); - mManager->addSequenceBatch( - {{{missRequest->mRequestId, missRequest->getNumTokens(beamIdx), beamWidth}}}, {std::ref(*missRequest)}); - auto missFuture = mRequester->receiveAsync(missRequest); - EXPECT_THROW(missFuture.get(), tensorrt_llm::common::TllmException); - } - // The sender must not tear down while the receiver's transfers are in flight. - tensorrt_llm::mpi::MpiComm::world().barrier(); -} - #if ENABLE_MULTI_DEVICE -using AsymmetricTestParam = std::tuple<int, int, int, int, int, int, int, int, int, int, tensorrt_llm::DataType, int, - bool, bool, bool, bool, bool, int, int>; +using AsymmetricTestParam = std::tuple<int, int, int, int, int, int, int, int, int, int, nvinfer1::DataType, int, bool, + bool, bool, bool, bool, int, int>; // CPMetaData struct to hold CP-specific information struct CPMetaData @@ -759,7 +671,7 @@ class AsymmetricalCacheTest : public ::testing::TestWithParam<AsymmetricTestPara } void setUpCacheManager(int numLayers, int numHeads, int sizePerHead, int tokensPerBlock, - tensorrt_llm::DataType dataType, int kvFactor = 2, bool isMLA = false, bool enableDPAttention = false, + nvinfer1::DataType dataType, int kvFactor = 2, bool isMLA = false, bool enableDPAttention = false, bool isWindow = false, bool isIndexerKCache = true, int indexerDimPerHead = 0, int indexerKCacheQuantBlockSize = 128) { @@ -970,18 +882,15 @@ class AsymmetricalCacheTest : public ::testing::TestWithParam<AsymmetricTestPara = [this, bufferManagers]() { return createCacheFormatter(mManager.get(), bufferManagers, mIsMLA); }; TLLM_LOG_DEBUG("setUpCacheTransceiver makeFormatter"); - // Generate a per-instance ID so each ctx/gen instance writes to - // its own CSV files (mirrors CacheTransceiver behaviour). - auto instanceId = "test_" + std::to_string(tensorrt_llm::mpi::MpiComm::world().getRank()); if (mIsContext) { - mSender = std::make_unique<CacheSender>(mConnectionManager.get(), mRankInInstance, - CacheTransferLayer(*mCacheState, makeFormatter()), instanceId); + mSender = std::make_unique<CacheSender>( + mConnectionManager.get(), mRankInInstance, CacheTransferLayer(*mCacheState, makeFormatter())); } else { - mRequester = std::make_unique<CacheReceiver>(mConnectionManager.get(), mRankInInstance, - CacheTransferLayer(*mCacheState, makeFormatter()), instanceId); + mRequester = std::make_unique<CacheReceiver>( + mConnectionManager.get(), mRankInInstance, CacheTransferLayer(*mCacheState, makeFormatter())); } TLLM_LOG_DEBUG("setUpCacheTransceiver mSender"); @@ -1062,18 +971,16 @@ class AsymmetricalCacheTest : public ::testing::TestWithParam<AsymmetricTestPara cpMetaData.emplace(length, tokensPerBlock, mCpRank, mCpSize); seqLen = cpMetaData.value().mSeqLenOnThisCPRank; } - + texec::Request request{VecTokens(seqLen, seqLen), maxNewTokens}; auto state = std::make_unique<texec::DataTransceiverState>(); + TLLM_CHECK(mContextCommState); state->setCommState(texec::kv_cache::CommState{*mContextCommState}); state->setCacheState(*mContextCacheState); auto stats = texec::ContextPhaseParams({}, mRequestId, state.release(), std::nullopt); + request.setContextPhaseParams(std::move(stats)); - tr::SamplingConfig samplingConfig{1}; - auto inputTokens = std::make_shared<VecTokens>(seqLen, seqLen); - auto llmRequestPtr = std::make_shared<LlmRequest>( - mRequestId++, maxNewTokens, inputTokens, samplingConfig, /*isStreaming=*/false); - llmRequestPtr->setContextPhaseParams(std::move(stats)); + auto llmRequestPtr = std::make_shared<LlmRequest>(mRequestId++, std::move(request)); return std::make_unique<WrappedLlmRequest>(std::move(llmRequestPtr), cpMetaData); } @@ -1089,6 +996,7 @@ class AsymmetricalCacheTest : public ::testing::TestWithParam<AsymmetricTestPara cpMetaData.emplace(length, tokensPerBlock, mCpRank, mCpSize); seqLen = cpMetaData.value().mSeqLenOnThisCPRank; } + texec::Request request{VecTokens(seqLen, seqLen), maxNewTokens}; auto state = std::make_unique<texec::DataTransceiverState>(); state->setCommState(texec::kv_cache::CommState{*mContextCommState}); @@ -1103,12 +1011,8 @@ class AsymmetricalCacheTest : public ::testing::TestWithParam<AsymmetricTestPara mContextCacheState->getParallelConfig().mTensorParallelism}; state->setCacheState(cacheState); auto stats = texec::ContextPhaseParams({}, requestId, state.release(), std::nullopt); - - tr::SamplingConfig samplingConfig{1}; - auto inputTokens = std::make_shared<VecTokens>(seqLen, seqLen); - auto llmRequestPtr - = std::make_shared<LlmRequest>(requestId, maxNewTokens, inputTokens, samplingConfig, /*isStreaming=*/false); - llmRequestPtr->setContextPhaseParams(std::move(stats)); + request.setContextPhaseParams(std::move(stats)); + auto llmRequestPtr = std::make_shared<LlmRequest>(requestId, std::move(request)); return std::make_unique<WrappedLlmRequest>(std::move(llmRequestPtr), cpMetaData); } @@ -1431,7 +1335,7 @@ class AsymmetricalCacheTest : public ::testing::TestWithParam<AsymmetricTestPara } std::variant<double, float, int16_t, int8_t, uint8_t> generateExpectedValue(size_t initial, int windowSize, - int tokenId, int layerId, int headId, int hiddenId, bool key, tensorrt_llm::DataType dataType) + int tokenId, int layerId, int headId, int hiddenId, bool key, nvinfer1::DataType dataType) { size_t seed = 0; std::size_t hashValue = std::hash<size_t>{}(initial); @@ -1507,7 +1411,7 @@ TEST_P(AsymmetricalCacheTest, TestCase) int numHeads = std::get<7>(param); int sizePerHead = std::get<8>(param); int tokensPerBlock = std::get<9>(param); - tensorrt_llm::DataType dataType = std::get<10>(param); + nvinfer1::DataType dataType = std::get<10>(param); int kvFactor = std::get<11>(param); bool isMLA = std::get<12>(param); @@ -1622,7 +1526,7 @@ TEST_P(AsymmetricalCacheTestWithDP, TestCase) int numHeads = std::get<7>(param); int sizePerHead = std::get<8>(param); int tokensPerBlock = std::get<9>(param); - tensorrt_llm::DataType dataType = std::get<10>(param); + nvinfer1::DataType dataType = std::get<10>(param); int kvFactor = std::get<11>(param); bool isMLA = std::get<12>(param); @@ -1759,7 +1663,7 @@ TEST_P(UnexpectedTerminationRaceTest, UnexpectedTerminationRaceTest) int numHeads = std::get<7>(param); int sizePerHead = std::get<8>(param); int tokensPerBlock = std::get<9>(param); - tensorrt_llm::DataType dataType = std::get<10>(param); + nvinfer1::DataType dataType = std::get<10>(param); int kvFactor = std::get<11>(param); bool isMLA = std::get<12>(param); @@ -1906,87 +1810,87 @@ TEST_P(UnexpectedTerminationRaceTest, UnexpectedTerminationRaceTest) INSTANTIATE_TEST_CASE_P(UnexpectedTerminationRaceTest, UnexpectedTerminationRaceTest, testing::Combine(testing::Values(2), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(4), testing::Values(4), testing::Values(16), - testing::Values(tensorrt_llm::DataType::kFLOAT), testing::Values(2), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false), - testing::Values(0), testing::Values(128))); + testing::Values(nvinfer1::DataType::kFLOAT), testing::Values(2), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(0), + testing::Values(128))); // Waive off isWindow test for now INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest0, AsymmetricalCacheTest, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(4), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(2), testing::Values(false), testing::Values(false), testing::Values(false), - testing::Values(/*true,*/ false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), + testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(/*true,*/ false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithWindow, AsymmetricalCacheTest, testing::Combine(testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(5), testing::Values(4), testing::Values(4), testing::Values(8), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(2), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(true), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest1, AsymmetricalCacheTest, testing::Combine(testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(8), testing::Values(4), testing::Values(4), testing::Values(8), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(2), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false /*, true*/), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest1EvenLayer, AsymmetricalCacheTest, testing::Combine(testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(10), testing::Values(4), testing::Values(4), testing::Values(8), - testing::Values(tensorrt_llm::DataType::kFLOAT), testing::Values(2), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false), - testing::Values(0), testing::Values(128))); + testing::Values(nvinfer1::DataType::kFLOAT), testing::Values(2), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(0), + testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest2EvenLayer, AsymmetricalCacheTest, testing::Combine(testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(10), testing::Values(4), testing::Values(4), testing::Values(8), - testing::Values(tensorrt_llm::DataType::kFLOAT), testing::Values(2), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false), - testing::Values(0), testing::Values(128))); + testing::Values(nvinfer1::DataType::kFLOAT), testing::Values(2), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(0), + testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest2, AsymmetricalCacheTest, testing::Combine(testing::Values(1), testing::Values(2), testing::Values(1), testing::Values(1), testing::Values(1, 4), testing::Values(1), testing::Values(16), testing::Values(16), testing::Values(4), - testing::Values(8), testing::Values(tensorrt_llm::DataType::kFLOAT), testing::Values(2), testing::Values(false), + testing::Values(8), testing::Values(nvinfer1::DataType::kFLOAT), testing::Values(2), testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest0ForMLA, AsymmetricalCacheTest, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(1), testing::Values(true), testing::Values(false), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), + testing::Values(true), testing::Values(false), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest1ForMLA, AsymmetricalCacheTest, testing::Combine(testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), testing::Values(8), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(1), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), testing::Values(true), testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest1ForMLAEvenLayer, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(10), testing::Values(1), testing::Values(4), testing::Values(8), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(1), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), testing::Values(true), testing::Values(false), testing::Values(false, true), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest2ForMLAEvenLayer, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(10), testing::Values(1), testing::Values(4), testing::Values(8), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(1), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), testing::Values(true), testing::Values(false), testing::Values(false, true), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest0ForMLAWithIndexerKCache, AsymmetricalCacheTest, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(1), testing::Values(true), testing::Values(false), testing::Values(false), - testing::Values(false), testing::Values(true), testing::Values(256), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), + testing::Values(true), testing::Values(false), testing::Values(false), testing::Values(false), + testing::Values(true), testing::Values(256), testing::Values(128))); // Tests cases where there's non-trivial TP and PP on context side but only CP on gen side. INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest0WithCPForMLA, AsymmetricalCacheTest, @@ -2000,7 +1904,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest0WithCPForMLA, AsymmetricalCacheTest, /*numHeads*/ testing::Values(1), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(1), /*isMLA*/ testing::Values(true), /*contextDP*/ testing::Values(false), @@ -2022,7 +1926,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest1WithCPForMLA, AsymmetricalCacheTest, /*numHeads*/ testing::Values(1), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(1), /*isMLA*/ testing::Values(true), /*contextDP*/ testing::Values(false), @@ -2044,7 +1948,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest0WithCPForGQA, AsymmetricalCacheTest, /*numHeads*/ testing::Values(4), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(2), /*isMLA*/ testing::Values(false), /*contextDP*/ testing::Values(false), @@ -2063,7 +1967,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest1WithCPForGQA, AsymmetricalCacheTest, /*numHeads*/ testing::Values(4), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(2), /*isMLA*/ testing::Values(false), /*contextDP*/ testing::Values(false), @@ -2082,7 +1986,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest0WithCPForMLAUnevenLayer, Asymmetrical /*numHeads*/ testing::Values(1), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(1), /*isMLA*/ testing::Values(true), /*contextDP*/ testing::Values(false), @@ -2104,7 +2008,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest1WithCPForMLAUnevenLayer, Asymmetrical /*numHeads*/ testing::Values(1), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(1), /*isMLA*/ testing::Values(true), /*contextDP*/ testing::Values(false), @@ -2126,7 +2030,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTest2WithCPForMLAUnevenLayer, Asymmetrical /*numHeads*/ testing::Values(1), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(1), /*isMLA*/ testing::Values(true), /*contextDP*/ testing::Values(false), @@ -2148,7 +2052,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithCPAndDPForMLA0, AsymmetricalCacheT /*numHeads*/ testing::Values(1), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(1), /*isMLA*/ testing::Values(true), /*contextDP*/ testing::Values(false), @@ -2170,7 +2074,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithCPAndDPForMLA1, AsymmetricalCacheT /*numHeads*/ testing::Values(1), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(1), /*isMLA*/ testing::Values(true), /*contextDP*/ testing::Values(true), @@ -2192,7 +2096,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithCPAndDPForGQA0, AsymmetricalCacheT /*numHeads*/ testing::Values(4), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(2), /*isMLA*/ testing::Values(false), /*contextDP*/ testing::Values(false), @@ -2211,7 +2115,7 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithCPAndDPForGQA1, AsymmetricalCacheT /*numHeads*/ testing::Values(4), /*sizePerHead*/ testing::Values(4), /*tokensPerBlock*/ testing::Values(8), - /*dataType*/ testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), + /*dataType*/ testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), /*kvFactor*/ testing::Values(2), /*isMLA*/ testing::Values(false), /*contextDP*/ testing::Values(true), @@ -2221,97 +2125,97 @@ INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithCPAndDPForGQA1, AsymmetricalCacheT INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForMLA1, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(1), testing::Values(true), testing::Values(true), testing::Values(true), testing::Values(false), + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), + testing::Values(true), testing::Values(true), testing::Values(true), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForMLA2, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(1), testing::Values(true), testing::Values(true), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), + testing::Values(true), testing::Values(true), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForMLA3, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(1), testing::Values(true), testing::Values(false), testing::Values(true), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), + testing::Values(true), testing::Values(false), testing::Values(true), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForMLA4, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(2), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), testing::Values(16), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(1), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), testing::Values(true), testing::Values(false), testing::Values(true), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForMLA5, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(2), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), testing::Values(16), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(1), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(1), testing::Values(true), testing::Values(false), testing::Values(true), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLA, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(4), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(2), testing::Values(false), testing::Values(true), testing::Values(true), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), + testing::Values(false), testing::Values(true), testing::Values(true), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLA1, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(4), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(2), testing::Values(false), testing::Values(true), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), + testing::Values(false), testing::Values(true), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLA2, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(4), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(2), testing::Values(false), testing::Values(false), testing::Values(true), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), + testing::Values(false), testing::Values(false), testing::Values(true), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLADuplicate0, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(2), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(2), testing::Values(false), testing::Values(true, false), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), + testing::Values(false), testing::Values(true, false), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLADuplicate0EvenLayer, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(5), testing::Values(2), testing::Values(4), testing::Values(16), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(2), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), testing::Values(false), testing::Values(true, false), testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLADuplicate1, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(1, 2), testing::Values(1, 2), testing::Values(1), testing::Values(2), testing::Values(2), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(2), testing::Values(false), testing::Values(true, false), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), + testing::Values(false), testing::Values(true, false), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLADuplicate2, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(4, 2), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(2), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(2), testing::Values(false), testing::Values(false), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), + testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLADuplicate3, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(2), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(4), testing::Values(2), testing::Values(4), testing::Values(16), - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), testing::Values(2), + testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), testing::Values(false), testing::Values(false), testing::Values(true), testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); INSTANTIATE_TEST_CASE_P(AsymmetricCaseTestWithDPForNoMLADuplicate4, AsymmetricalCacheTestWithDP, testing::Combine(testing::Values(4), testing::Values(1), testing::Values(1), testing::Values(1, 2), testing::Values(2), testing::Values(1), testing::Values(4), testing::Values(1, 2), testing::Values(4), - testing::Values(16), testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kINT8), - testing::Values(2), testing::Values(false), testing::Values(false), testing::Values(false), - testing::Values(false), testing::Values(false), testing::Values(0), testing::Values(128))); + testing::Values(16), testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kINT8), testing::Values(2), + testing::Values(false), testing::Values(false), testing::Values(false), testing::Values(false), + testing::Values(false), testing::Values(0), testing::Values(128))); #endif @@ -2322,7 +2226,7 @@ TEST(targetTest, CacheStateNODP) int const numHeads = 2; int const sizePerHead = 64; int const tokensPerBlock = 64; - auto const dataType = tensorrt_llm::DataType::kFLOAT; + auto const dataType = nvinfer1::DataType::kFLOAT; bool const isMLA = true; int const kvFactor = 2; @@ -2612,7 +2516,7 @@ TEST(targetTest, CacheStateNODPForGQAWithCP) int const numHeads = 4; int const sizePerHead = 64; int const tokensPerBlock = 64; - auto const dataType = tensorrt_llm::DataType::kFLOAT; + auto const dataType = nvinfer1::DataType::kFLOAT; bool const isMLA = false; int const kvFactor = 2; @@ -2828,7 +2732,7 @@ TEST(targetTest, CacheStateContextDP) int const numHeads = 2; int const sizePerHead = 64; int const tokensPerBlock = 64; - auto const dataType = tensorrt_llm::DataType::kFLOAT; + auto const dataType = nvinfer1::DataType::kFLOAT; bool const isMLA = true; int const kvFactor = 2; diff --git a/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/allReduceFusionTest.cu b/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/allReduceFusionTest.cu index e95c84d7d05a..4b9c7af29a46 100644 --- a/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/allReduceFusionTest.cu +++ b/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/allReduceFusionTest.cu @@ -25,7 +25,6 @@ #include <random> #include <vector> -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/allReduceFusionKernels.h" #include "tensorrt_llm/kernels/communicationKernels/allReduceWorkspace.h" #include "tensorrt_llm/kernels/quantization.h" @@ -236,28 +235,28 @@ template <> struct DTypeTraits<half> { static constexpr ncclDataType_t kNCCLDataType = ncclFloat16; - static constexpr tensorrt_llm::DataType kTRTDataType = tensorrt_llm::DataType::kHALF; + static constexpr nvinfer1::DataType kTRTDataType = nvinfer1::DataType::kHALF; }; template <> struct DTypeTraits<__nv_bfloat16> { static constexpr ncclDataType_t kNCCLDataType = ncclBfloat16; - static constexpr tensorrt_llm::DataType kTRTDataType = tensorrt_llm::DataType::kBF16; + static constexpr nvinfer1::DataType kTRTDataType = nvinfer1::DataType::kBF16; }; template <> struct DTypeTraits<float> { static constexpr ncclDataType_t kNCCLDataType = ncclFloat32; - static constexpr tensorrt_llm::DataType kTRTDataType = tensorrt_llm::DataType::kFLOAT; + static constexpr nvinfer1::DataType kTRTDataType = nvinfer1::DataType::kFLOAT; }; template <typename DType, ar_fusion::AllReduceFusionPattern Pattern> class TestRunner { static constexpr ncclDataType_t kNCCLDataType = DTypeTraits<DType>::kNCCLDataType; - static constexpr tensorrt_llm::DataType kTRTDataType = DTypeTraits<DType>::kTRTDataType; + static constexpr nvinfer1::DataType kTRTDataType = DTypeTraits<DType>::kTRTDataType; static constexpr bool kFP4QuantOutSupport = !std::is_same_v<DType, float>; static_assert(kFP4QuantOutSupport || Pattern != ar_fusion::AllReduceFusionPattern::kARResidualRMSNormFP4Quant, "kARResidualRMSNormFP4Quant is not supported for float dtype"); diff --git a/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/allReduceKernelTest.cu b/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/allReduceKernelTest.cu index 78747373accf..b3d120e7015c 100644 --- a/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/allReduceKernelTest.cu +++ b/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/allReduceKernelTest.cu @@ -32,7 +32,6 @@ #include <type_traits> #include <vector> -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/customAllReduceKernels.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaStream.h" @@ -213,7 +212,7 @@ public: { } - void set_params(AllReduceParams& params, tensorrt_llm::DataType dataType, int token_num, int hidden_size, + void set_params(AllReduceParams& params, nvinfer1::DataType dataType, int token_num, int hidden_size, AllReduceFusionOp op) const { int world_size = world_config.getSize(); @@ -317,7 +316,7 @@ bool test(Workspace const& workspace, int token_num, int hidden_size, bool has_b in.copy_from(input_buffer.data()); AllReduceParams params; - workspace.set_params(params, tensorrt_llm::DataType::kHALF, token_num, hidden_size, fusion_op); + workspace.set_params(params, nvinfer1::DataType::kHALF, token_num, hidden_size, fusion_op); params.ranks_per_node = world_size; params.local_rank = rank; params.local_output_buffer_ptr = out.data(); @@ -335,21 +334,21 @@ bool test(Workspace const& workspace, int token_num, int hidden_size, bool has_b cudaEventCreate(&begin); cudaEventCreate(&end); lamportInitialize( - params.fusion_params.lamport_peer_comm_buffer_ptrs[rank], message_size, tensorrt_llm::DataType::kHALF, s); + params.fusion_params.lamport_peer_comm_buffer_ptrs[rank], message_size, nvinfer1::DataType::kHALF, s); lamportInitialize(params.fusion_params.lamport_peer_comm_buffer_ptrs[rank + MAX_RANKS_PER_NODE], message_size, - tensorrt_llm::DataType::kHALF, s); + nvinfer1::DataType::kHALF, s); lamportInitialize(params.fusion_params.lamport_peer_comm_buffer_ptrs[rank + MAX_RANKS_PER_NODE * 2], message_size, - tensorrt_llm::DataType::kHALF, s); + nvinfer1::DataType::kHALF, s); cudaDeviceSynchronize(); comm.barrier(); for (int i = 0; i < warmup; ++i) { - customAllReduce(params, tensorrt_llm::DataType::kHALF, runtime_strategy, config, fusion_op, s); + customAllReduce(params, nvinfer1::DataType::kHALF, runtime_strategy, config, fusion_op, s); } cudaEventRecord(begin, s); for (int i = 0; i < iter; ++i) { - customAllReduce(params, tensorrt_llm::DataType::kHALF, runtime_strategy, config, fusion_op, s); + customAllReduce(params, nvinfer1::DataType::kHALF, runtime_strategy, config, fusion_op, s); } cudaEventRecord(end, s); cudaEventSynchronize(end); @@ -463,7 +462,7 @@ bool test_prepostnorm(Workspace const& workspace, int token_num, int hidden_size in.copy_from(input_buffer.data()); AllReduceParams params; - workspace.set_params(params, tensorrt_llm::DataType::kHALF, token_num, hidden_size, fusion_op); + workspace.set_params(params, nvinfer1::DataType::kHALF, token_num, hidden_size, fusion_op); params.ranks_per_node = world_size; params.local_rank = rank; params.local_output_buffer_ptr = out.data(); @@ -485,12 +484,12 @@ bool test_prepostnorm(Workspace const& workspace, int token_num, int hidden_size comm.barrier(); for (int i = 0; i < warmup; ++i) { - customAllReduce(params, tensorrt_llm::DataType::kHALF, runtime_strategy, config, fusion_op, s); + customAllReduce(params, nvinfer1::DataType::kHALF, runtime_strategy, config, fusion_op, s); } cudaEventRecord(begin, s); for (int i = 0; i < iter; ++i) { - customAllReduce(params, tensorrt_llm::DataType::kHALF, runtime_strategy, config, fusion_op, s); + customAllReduce(params, nvinfer1::DataType::kHALF, runtime_strategy, config, fusion_op, s); } cudaEventRecord(end, s); cudaEventSynchronize(end); diff --git a/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/gemmAllReduceTest.cu b/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/gemmAllReduceTest.cu index 2ff2c8c37b9c..415a3c32da81 100644 --- a/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/gemmAllReduceTest.cu +++ b/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/gemmAllReduceTest.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2026, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -25,11 +25,12 @@ #else #include "allreduce_gemm_runner.h" #endif +#include "common.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/userbuffers/ub_interface.h" #include "tensorrt_llm/runtime/ipcNvlsMemory.h" #include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include <NvInferRuntime.h> #include "cute/tensor.hpp" #include "cutlass/cutlass.h" @@ -52,59 +53,8 @@ #include "cutlass/util/reference/host/gett.hpp" #include "cutlass/util/reference/host/tensor_fill.h" -namespace tensorrt_llm::testing -{ - -/** - * GPU timer for recording the elapsed time across kernel(s) launched in GPU stream - */ -struct GpuTimer -{ - cudaStream_t _stream_id; - cudaEvent_t _start; - cudaEvent_t _stop; - - /// Constructor - GpuTimer() - : _stream_id(0) - { - TLLM_CUDA_CHECK(cudaEventCreate(&_start)); - TLLM_CUDA_CHECK(cudaEventCreate(&_stop)); - } - - /// Destructor - ~GpuTimer() - { - TLLM_CUDA_CHECK(cudaEventDestroy(_start)); - TLLM_CUDA_CHECK(cudaEventDestroy(_stop)); - } - - /// Start the timer for a given stream (defaults to the default stream) - void start(cudaStream_t stream_id = 0) - { - _stream_id = stream_id; - TLLM_CUDA_CHECK(cudaEventRecord(_start, _stream_id)); - } - - /// Stop the timer - void stop() - { - TLLM_CUDA_CHECK(cudaEventRecord(_stop, _stream_id)); - } - - /// Return the elapsed time (in milliseconds) - float elapsed_millis() - { - float elapsed = 0.0; - TLLM_CUDA_CHECK(cudaEventSynchronize(_stop)); - TLLM_CUDA_CHECK(cudaEventElapsedTime(&elapsed, _start, _stop)); - return elapsed; - } -}; - -} // namespace tensorrt_llm::testing - using namespace cutlass; +using namespace nvinfer1; using namespace tensorrt_llm::mpi; using namespace tensorrt_llm::runtime; using namespace tensorrt_llm::common; @@ -288,7 +238,7 @@ struct ToType template <> struct ToType<cutlass::bfloat16_t> { - tensorrt_llm::DataType trt_value = tensorrt_llm::DataType::kBF16; + nvinfer1::DataType trt_value = nvinfer1::DataType::kBF16; ncclDataType_t nccl_value = ncclBfloat16; char const* str_value = "bf16"; }; @@ -296,7 +246,7 @@ struct ToType<cutlass::bfloat16_t> template <> struct ToType<cutlass::half_t> { - tensorrt_llm::DataType trt_value = tensorrt_llm::DataType::kHALF; + nvinfer1::DataType trt_value = nvinfer1::DataType::kHALF; ncclDataType_t nccl_value = ncclFloat16; char const* str_value = "fp16"; }; @@ -304,7 +254,7 @@ struct ToType<cutlass::half_t> template <> struct ToType<cutlass::float_e4m3_t> { - tensorrt_llm::DataType trt_value = tensorrt_llm::DataType::kFP8; + nvinfer1::DataType trt_value = nvinfer1::DataType::kFP8; ncclDataType_t nccl_value = ncclFloat8e4m3; char const* str_value = "fp8_e4m3"; }; diff --git a/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/moeAllReduceFusionTest.cu b/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/moeAllReduceFusionTest.cu index 4b81a4133fa9..8abccf214b7a 100644 --- a/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/moeAllReduceFusionTest.cu +++ b/cpp/tests/unit_tests/multi_gpu/kernels/allReduce/moeAllReduceFusionTest.cu @@ -24,7 +24,6 @@ #include <random> #include <vector> -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/kernels/communicationKernels/allReduceWorkspace.h" #include "tensorrt_llm/kernels/communicationKernels/moeAllReduceFusionKernels.h" #include "tensorrt_llm/kernels/quantization.h" @@ -393,8 +392,8 @@ class MoEARFuseTestRunner { static_assert(std::is_same_v<DType, half> || std::is_same_v<DType, __nv_bfloat16>); static constexpr ncclDataType_t kNCCLDataType = std::is_same_v<DType, half> ? ncclFloat16 : ncclBfloat16; - static constexpr tensorrt_llm::DataType kTRTDataType - = std::is_same_v<DType, half> ? tensorrt_llm::DataType::kHALF : tensorrt_llm::DataType::kBF16; + static constexpr nvinfer1::DataType kTRTDataType + = std::is_same_v<DType, half> ? nvinfer1::DataType::kHALF : nvinfer1::DataType::kBF16; public: MoEARFuseTestRunner(int max_token_num, int hidden_dim, int max_expert_num) diff --git a/cpp/tests/unit_tests/multi_gpu/mpiUtilsTest.cpp b/cpp/tests/unit_tests/multi_gpu/mpiUtilsTest.cpp index 941d7cb53ed6..b8596d511fab 100644 --- a/cpp/tests/unit_tests/multi_gpu/mpiUtilsTest.cpp +++ b/cpp/tests/unit_tests/multi_gpu/mpiUtilsTest.cpp @@ -20,6 +20,7 @@ #include "tensorrt_llm/runtime/utils/mpiUtils.h" #if ENABLE_MULTI_DEVICE +#include "tensorrt_llm/plugins/common/plugin.h" #include <nccl.h> #endif // ENABLE_MULTI_DEVICE @@ -116,6 +117,11 @@ TEST(MPIUtils, BroadcastNcclId) EXPECT_TRUE(std::any_of( id.internal, id.internal + sizeof(id.internal) / sizeof(id.internal[0]), [](auto x) { return x != 0; })); } + +TEST(MPIUtils, GlobalSessionHandle) +{ + EXPECT_EQ(tensorrt_llm::plugins::getCommSessionHandle(), &COMM_SESSION); +} #endif // ENABLE_MULTI_DEVICE template <typename T> diff --git a/cpp/tests/unit_tests/runtime/CMakeLists.txt b/cpp/tests/unit_tests/runtime/CMakeLists.txt index 3a171ee39877..c022ba31ebca 100644 --- a/cpp/tests/unit_tests/runtime/CMakeLists.txt +++ b/cpp/tests/unit_tests/runtime/CMakeLists.txt @@ -35,6 +35,7 @@ add_gtest(samplingConfigTest samplingConfigTest.cpp) add_gtest(samplingTest samplingTest.cpp) add_gtest(sanitizerTest sanitizerTest.cpp) add_gtest(tllmBuffersTest tllmBuffersTest.cpp) +add_gtest(tllmRuntimeTest tllmRuntimeTest.cpp) add_gtest(transposeKVKernelTest transposeKVKernelTest.cpp) add_gtest(utilsTest utilsTest.cpp) add_gtest(virtualMemoryTest virtualMemoryTest.cpp) diff --git a/cpp/tests/unit_tests/runtime/bufferManagerTest.cpp b/cpp/tests/unit_tests/runtime/bufferManagerTest.cpp index 8bdc6d0352a0..194cf88b6765 100644 --- a/cpp/tests/unit_tests/runtime/bufferManagerTest.cpp +++ b/cpp/tests/unit_tests/runtime/bufferManagerTest.cpp @@ -17,7 +17,6 @@ #include <gtest/gtest.h> #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaMemPool.h" @@ -122,7 +121,7 @@ TEST_F(BufferManagerTest, Pointers) static_assert(std::is_same_v<decltype(trtPointerType), BufferDataType const>); static_assert(trtPointerType.isPointer()); static_assert(trtPointerType.getDataType() == TRTDataType<cppBaseType>::value); - static_assert(static_cast<tensorrt_llm::DataType>(trtPointerType) == BufferDataType::kTrtPointerType); + static_assert(static_cast<nvinfer1::DataType>(trtPointerType) == BufferDataType::kTrtPointerType); static_assert(trtPointerType == BufferDataType::kTrtPointerType); // uses implicit type conversion // The C++ type corresponding to the TensorRT type for storing pointers (int64_t) using cppStorageType = DataTypeTraits<trtPointerType>::type; diff --git a/cpp/tests/unit_tests/runtime/decodingLayerWorkspaceTest.cpp b/cpp/tests/unit_tests/runtime/decodingLayerWorkspaceTest.cpp index 74f8faa37c87..bb6ce6410ad5 100644 --- a/cpp/tests/unit_tests/runtime/decodingLayerWorkspaceTest.cpp +++ b/cpp/tests/unit_tests/runtime/decodingLayerWorkspaceTest.cpp @@ -16,7 +16,6 @@ #include "tensorrt_llm/runtime/decodingLayerWorkspace.h" #include "tensorrt_llm/common/cudaUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/common/workspace.h" #include <gtest/gtest.h> #include <random> @@ -122,9 +121,8 @@ auto const tensorDataTypesTuples = testing::Combine(tensorDataTypes, tensorDataT auto const tensorShapeTuples = testing::Combine(tensorDimensions, tensorDimensions, tensorDimensions); auto const mirrorInWorkspaceParams = testing::Combine(tensorDataTypesTuples, tensorShapeTuples, randomSeeds); -using MirrorInWorkspaceParamType - = std::tuple<std::tuple<tensorrt_llm::DataType, tensorrt_llm::DataType, tensorrt_llm::DataType>, - std::tuple<std::int32_t, std::int32_t, std::int32_t>, std::uint64_t>; +using MirrorInWorkspaceParamType = std::tuple<std::tuple<nvinfer1::DataType, nvinfer1::DataType, nvinfer1::DataType>, + std::tuple<std::int32_t, std::int32_t, std::int32_t>, std::uint64_t>; class MirrorInWorkspaceTest : public testing::TestWithParam<MirrorInWorkspaceParamType> { diff --git a/cpp/tests/unit_tests/runtime/gptDecoderBatchedTest.cpp b/cpp/tests/unit_tests/runtime/gptDecoderBatchedTest.cpp index a979c9a4699f..15476899fce3 100644 --- a/cpp/tests/unit_tests/runtime/gptDecoderBatchedTest.cpp +++ b/cpp/tests/unit_tests/runtime/gptDecoderBatchedTest.cpp @@ -17,11 +17,10 @@ #include "tensorrt_llm/runtime/gptDecoderBatched.h" #include "tensorrt_llm/batch_manager/createNewDecoderRequests.h" #include "tensorrt_llm/batch_manager/decoderBuffers.h" -#include "tensorrt_llm/batch_manager/llmRequest.h" +#include "tensorrt_llm/batch_manager/makeDecodingBatchInputOutput.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/logger.h" #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/common.h" @@ -48,62 +47,6 @@ using TensorPtr = ITensor::SharedPtr; namespace { -// Local copy of the former MakeDecodingBatchInputOutput::createDecoderBatchInputs -// helper, which was removed with the TensorRT-engine execution path. The decoder -// under test is backend-agnostic; this builds its step-batched inputs directly. -void createDecoderBatchInputs(tb::DecoderInputBuffers& inputBuffers, std::vector<SizeType32> const& activeSlots, - decoder::DecoderState const& decoderState) -{ - auto const& numDecodingEngineTokens = decoderState.getNumDecodingEngineTokens(); - auto const& maxDecodingEngineTokens = decoderState.getMaxDecodingEngineTokens(); - auto const& maxDecodingDecoderTokens = decoderState.getMaxDecodingDecoderTokens(); - auto const maxDecoderSteps = tc::ceilDiv(maxDecodingEngineTokens, maxDecodingDecoderTokens); - - auto& batchSlots = inputBuffers.forwardBatchSlots; - auto& decoderLogits = inputBuffers.decoderLogits; - - for (SizeType32 step = 0; step < maxDecoderSteps; ++step) - { - batchSlots.at(step)->resize(activeSlots.size()); - } - - auto constexpr singleRequest = 1; - - std::vector<SizeType32> batchSizes(maxDecoderSteps); - std::vector<std::vector<ITensor::SharedConstPtr>> batchLogits(maxDecoderSteps); - auto maxActiveDecoderSteps = 1; - for (size_t batchIdx = 0; batchIdx < activeSlots.size(); ++batchIdx) - { - auto const slot = activeSlots.at(batchIdx); - auto const& logits = decoderLogits.at(batchIdx); - - auto const numDecoderSteps = tc::ceilDiv(numDecodingEngineTokens.at(slot), maxDecodingDecoderTokens); - maxActiveDecoderSteps = std::max(maxActiveDecoderSteps, numDecoderSteps); - for (SizeType32 step = 0; step < numDecoderSteps; ++step) - { - auto batchSlotsRange = BufferRange<SizeType32>(*batchSlots.at(step)); - batchSlotsRange[batchSizes[step]] = slot; - batchSizes[step]++; - auto logitsSlice = ITensor::slice(logits, step, singleRequest); - batchLogits[step].emplace_back(std::move(logitsSlice)); - } - } - - for (SizeType32 step = 0; step < maxDecoderSteps; ++step) - { - batchSlots.at(step)->resize(batchSizes[step]); - } - batchLogits.resize(maxActiveDecoderSteps); - - inputBuffers.maxDecoderSteps = maxActiveDecoderSteps; - inputBuffers.batchLogits = batchLogits; -} - -} // namespace - -namespace -{ - std::shared_ptr<tb::LlmRequest> createLlmRequest(SizeType32 batchSlot, SizeType32 inputLengths, SizeType32 generatedTokensPerSteps, SizeType32 acceptedTokensPerStep, TokenIdType inputTokenId, TokenIdType expectedTokenId, SizeType32 maxNewTokens, SamplingConfig const& samplingConfig, TokenIdType endId) @@ -150,7 +93,7 @@ std::vector<std::shared_ptr<tb::LlmRequest>> createLlmRequests(std::vector<SizeT } void newRequests(std::vector<std::shared_ptr<tb::LlmRequest>> const& requests, TensorPtr const& batchSlots, - tensorrt_llm::DataType logitsType, ModelConfig const& modelConfig, WorldConfig const& worldConfig, + nvinfer1::DataType logitsType, ModelConfig const& modelConfig, WorldConfig const& worldConfig, tle::DecodingConfig const& decodingConfig, GptDecoderBatched& decoder, CudaStream const& runtimeStream, SizeType32 maxSequenceLength, tb::DecoderInputBuffers& inputBuffers, decoder::DecoderState& decoderState) { @@ -182,7 +125,7 @@ void newRequests(std::vector<std::shared_ptr<tb::LlmRequest>> const& requests, T } void createDecoderInputs(tb::DecoderInputBuffers& inputBuffers, SizeType32 batchSize, SizeType32 vocabSizePadded, - tensorrt_llm::DataType dataType, std::vector<SamplingConfig>& samplingConfigs, + nvinfer1::DataType dataType, std::vector<SamplingConfig>& samplingConfigs, std::vector<SizeType32> const& generatedTokensPerSteps, bool computeLogProbs, BufferManager& manager) { auto& logits = inputBuffers.decoderLogits; @@ -299,8 +242,8 @@ void verifyResults(BufferManager& manager, decoder::DecoderState const& decoderS } } -void testDecoder(tensorrt_llm::DataType const dtype, std::vector<SamplingConfig>& samplingConfigs, - SizeType32 maxBeamWidth, bool computeLogProbs) +void testDecoder(nvinfer1::DataType const dtype, std::vector<SamplingConfig>& samplingConfigs, SizeType32 maxBeamWidth, + bool computeLogProbs) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); SizeType32 constexpr tensorParallelism{1}; @@ -402,7 +345,7 @@ void testDecoder(tensorrt_llm::DataType const dtype, std::vector<SamplingConfig> auto activeSlots = std::vector<SizeType32>(batchSize); std::iota(activeSlots.begin(), activeSlots.end(), 0); - createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); + tb::MakeDecodingBatchInputOutput::createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); decoder.forward(decoderState, inputBuffers); checkSequenceLengths(*decoderState.getSequenceLengths(), expectedLengths, manager); @@ -432,7 +375,7 @@ void testDecoder(tensorrt_llm::DataType const dtype, std::vector<SamplingConfig> EXPECT_FALSE(getFinished(*decoderState.getFinishedSum(), samplingConfigs, manager)[0]); } -void testDecoderWavefront(tensorrt_llm::DataType const dtype, std::vector<SamplingConfig>& samplingConfigs, +void testDecoderWavefront(nvinfer1::DataType const dtype, std::vector<SamplingConfig>& samplingConfigs, SizeType32 maxBeamWidth, bool computeLogProbs) { TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); @@ -531,7 +474,7 @@ void testDecoderWavefront(tensorrt_llm::DataType const dtype, std::vector<Sampli auto activeSlots = std::vector<SizeType32>(batchIdx + 1); std::iota(activeSlots.begin(), activeSlots.end(), 0); - createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); + tb::MakeDecodingBatchInputOutput::createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); decoder.forward(decoderState, inputBuffers); advanceSequenceLengths( @@ -553,7 +496,7 @@ void testDecoderWavefront(tensorrt_llm::DataType const dtype, std::vector<Sampli auto finishedVec = getFinished(*decoderState.getFinishedSum(), samplingConfigs, manager); while (!std::all_of(expectedFinished.begin(), expectedFinished.end(), [](bool finish) { return finish; })) { - createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); + tb::MakeDecodingBatchInputOutput::createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); decoder.forward(decoderState, inputBuffers); finishedVec = getFinished(*decoderState.getFinishedSum(), samplingConfigs, manager); @@ -583,7 +526,7 @@ void testDecoderWavefront(tensorrt_llm::DataType const dtype, std::vector<Sampli maxSeqLength, inputTokenId, expectedTokenId, endId); } -void testDecoderDraft(tensorrt_llm::DataType const dtype, std::vector<SamplingConfig>& samplingConfigs, +void testDecoderDraft(nvinfer1::DataType const dtype, std::vector<SamplingConfig>& samplingConfigs, SizeType32 maxBeamWidth, std::vector<SizeType32> const& generatedTokensPerSteps, std::vector<SizeType32> const& acceptedTokensPerStep, SizeType32 maxGeneratedTokensPerStep) { @@ -688,7 +631,7 @@ void testDecoderDraft(tensorrt_llm::DataType const dtype, std::vector<SamplingCo auto activeSlots = std::vector<SizeType32>(batchSize); std::iota(activeSlots.begin(), activeSlots.end(), 0); - createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); + tb::MakeDecodingBatchInputOutput::createDecoderBatchInputs(inputBuffers, activeSlots, decoderState); decoder.forward(decoderState, inputBuffers); checkSequenceLengths(*decoderState.getSequenceLengths(), expectedLengths, manager); EXPECT_THAT(getFinished(*decoderState.getFinishedSum(), samplingConfigs, manager), ::testing::Each(false)); @@ -705,11 +648,11 @@ struct BeamConfig std::vector<SizeType32> beamWidths; }; -using ParamType = std::tuple<tensorrt_llm::DataType, BeamConfig, bool>; +using ParamType = std::tuple<nvinfer1::DataType, BeamConfig, bool>; std::string generateTestName(testing::TestParamInfo<ParamType> const& info) { - std::string name{std::get<0>(info.param) == tensorrt_llm::DataType::kFLOAT ? "Float" : "Half"}; + std::string name{std::get<0>(info.param) == nvinfer1::DataType::kFLOAT ? "Float" : "Half"}; BeamConfig const beamConfig = std::get<1>(info.param); name.append("MaxBeamWidth" + std::to_string(beamConfig.maxBeamWidth)); for (auto const beamWdith : beamConfig.beamWidths) @@ -730,7 +673,7 @@ class ParamTest : public ::testing::TestWithParam<ParamType> TEST_P(ParamTest, Test) { - tensorrt_llm::DataType const dtype{std::get<0>(GetParam())}; + nvinfer1::DataType const dtype{std::get<0>(GetParam())}; BeamConfig const beamConfig{std::get<1>(GetParam())}; bool const computeLogProbs{std::get<2>(GetParam())}; std::vector<SamplingConfig> samplingConfigs; @@ -743,7 +686,7 @@ TEST_P(ParamTest, Test) } INSTANTIATE_TEST_SUITE_P(DecoderBwTest, ParamTest, - testing::Combine(testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kHALF), + testing::Combine(testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kHALF), testing::Values(BeamConfig{1, {1, 1, 1}}, BeamConfig{3, {3, 3, 3, 3}}, BeamConfig{4, {4, 4, 4}}, BeamConfig{10, {10, 10, 10}}), testing::Values(false, true)), @@ -755,7 +698,7 @@ class ParamWavefrontTest : public ::testing::TestWithParam<ParamType> TEST_P(ParamWavefrontTest, Test) { - tensorrt_llm::DataType const dtype{std::get<0>(GetParam())}; + nvinfer1::DataType const dtype{std::get<0>(GetParam())}; BeamConfig const beamConfig{std::get<1>(GetParam())}; bool const computeLogProbs{std::get<2>(GetParam())}; bool const normalizeLogProbs{true}; @@ -769,7 +712,7 @@ TEST_P(ParamWavefrontTest, Test) } INSTANTIATE_TEST_SUITE_P(DecoderBwTest, ParamWavefrontTest, - testing::Combine(testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kHALF), + testing::Combine(testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kHALF), testing::Values(BeamConfig{1, {1, 1, 1}}, BeamConfig{3, {3, 3, 3, 3}}, BeamConfig{4, {4, 4, 4}}, BeamConfig{10, {10, 10, 10}}), testing::Values(false, true)), @@ -782,7 +725,7 @@ struct DraftConfig std::vector<SizeType32> acceptedTokensPerStep; }; -using DraftTestParamType = std::tuple<tensorrt_llm::DataType, BeamConfig, DraftConfig>; +using DraftTestParamType = std::tuple<nvinfer1::DataType, BeamConfig, DraftConfig>; class ParamDraftTest : public ::testing::TestWithParam<DraftTestParamType> { @@ -790,7 +733,7 @@ class ParamDraftTest : public ::testing::TestWithParam<DraftTestParamType> TEST_P(ParamDraftTest, Test) { - tensorrt_llm::DataType const dtype{std::get<0>(GetParam())}; + nvinfer1::DataType const dtype{std::get<0>(GetParam())}; BeamConfig const beamConfig{std::get<1>(GetParam())}; DraftConfig const draftConfig{std::get<2>(GetParam())}; @@ -808,7 +751,7 @@ TEST_P(ParamDraftTest, Test) } INSTANTIATE_TEST_SUITE_P(DecoderTest, ParamDraftTest, - testing::Combine(testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kHALF), + testing::Combine(testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kHALF), testing::Values(BeamConfig{1, {1, 1, 1}}), testing::Values( // DraftConfig{2, {1, 1, 1}, {0, 0, 0}}, DraftConfig{2, {2, 2, 2}, {1, 1, 1}}, @@ -817,7 +760,7 @@ INSTANTIATE_TEST_SUITE_P(DecoderTest, ParamDraftTest, )), [](testing::TestParamInfo<DraftTestParamType> const& info) { - std::string name{std::get<0>(info.param) == tensorrt_llm::DataType::kFLOAT ? "Float" : "Half"}; + std::string name{std::get<0>(info.param) == nvinfer1::DataType::kFLOAT ? "Float" : "Half"}; BeamConfig const beamConfig = std::get<1>(info.param); DraftConfig const draftConfig = std::get<2>(info.param); name.append("MaxBeamWidth" + std::to_string(beamConfig.maxBeamWidth)); diff --git a/cpp/tests/unit_tests/runtime/gptDecoderTest.cpp b/cpp/tests/unit_tests/runtime/gptDecoderTest.cpp index 5f620aa4d9c4..e1fed49293bd 100644 --- a/cpp/tests/unit_tests/runtime/gptDecoderTest.cpp +++ b/cpp/tests/unit_tests/runtime/gptDecoderTest.cpp @@ -17,7 +17,6 @@ #include <gtest/gtest.h> #include "tensorrt_llm/common/memoryUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/gptDecoder.h" @@ -69,7 +68,7 @@ bool forwardAndSync(std::unique_ptr<IGptDecoder> const& decoder, DecodingOutput& } } -void testDecoder(tensorrt_llm::DataType const dtype, SamplingConfig const& samplingConfig) +void testDecoder(nvinfer1::DataType const dtype, SamplingConfig const& samplingConfig) { SizeType32 constexpr tensorParallelism{1}; SizeType32 constexpr pipelineParallelism{1}; @@ -141,21 +140,21 @@ void testDecoder(tensorrt_llm::DataType const dtype, SamplingConfig const& sampl if (beamWidth > 1) { auto srcCacheIndirection = std::shared_ptr( - manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), tensorrt_llm::DataType::kINT32)); + manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), nvinfer1::DataType::kINT32)); manager.setZero(*srcCacheIndirection); inputs.cacheIndirection = srcCacheIndirection; } // set up outputs auto outputIds = std::shared_ptr( - manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), tensorrt_llm::DataType::kINT32)); + manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), nvinfer1::DataType::kINT32)); manager.setZero(*outputIds); auto gatheredOutputIds = std::shared_ptr( - manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), tensorrt_llm::DataType::kINT32)); + manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), nvinfer1::DataType::kINT32)); manager.setZero(*gatheredOutputIds); DecodingOutput outputs{outputIds, gatheredOutputIds}; auto newTokens - = std::shared_ptr(manager.gpu(ITensor::makeShape({batchSize, beamWidth}), tensorrt_llm::DataType::kINT32)); + = std::shared_ptr(manager.gpu(ITensor::makeShape({batchSize, beamWidth}), nvinfer1::DataType::kINT32)); manager.setZero(*newTokens); outputs.newTokens = newTokens; @@ -166,7 +165,7 @@ void testDecoder(tensorrt_llm::DataType const dtype, SamplingConfig const& sampl TRTDataType<tensorrt_llm::kernels::FinishedState::UnderlyingType>::value); inputs.finishReasons = ITensor::view(outputs.finishReasons); manager.setZero(*outputs.finishReasons); - outputs.finishedSum = BufferManager::pinnedPool(ITensor::makeShape({batchSize}), tensorrt_llm::DataType::kINT32); + outputs.finishedSum = BufferManager::pinnedPool(ITensor::makeShape({batchSize}), nvinfer1::DataType::kINT32); auto finishedSumHost = bufferCast<std::int32_t>(*outputs.finishedSum); for (SizeType32 bi = 0; bi < batchSize; ++bi) { @@ -176,17 +175,17 @@ void testDecoder(tensorrt_llm::DataType const dtype, SamplingConfig const& sampl if (beamWidth > 1) { auto tgtCacheIndirection = std::shared_ptr( - manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), tensorrt_llm::DataType::kINT32)); + manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), nvinfer1::DataType::kINT32)); manager.setZero(*tgtCacheIndirection); outputs.cacheIndirection = tgtCacheIndirection; auto cumLogProbs - = std::shared_ptr(manager.gpu(ITensor::makeShape({batchSize, beamWidth}), tensorrt_llm::DataType::kFLOAT)); + = std::shared_ptr(manager.gpu(ITensor::makeShape({batchSize, beamWidth}), nvinfer1::DataType::kFLOAT)); manager.setZero(*cumLogProbs); outputs.cumLogProbs = cumLogProbs; auto parentIds = std::shared_ptr( - manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), tensorrt_llm::DataType::kINT32)); + manager.gpu(ITensor::makeShape({batchSize, beamWidth, maxSeqLength}), nvinfer1::DataType::kINT32)); manager.setZero(*parentIds); outputs.parentIds = parentIds; } @@ -246,13 +245,13 @@ void testDecoder(tensorrt_llm::DataType const dtype, SamplingConfig const& sampl } // namespace -class ParamTest : public ::testing::TestWithParam<std::tuple<tensorrt_llm::DataType, SizeType32>> +class ParamTest : public ::testing::TestWithParam<std::tuple<nvinfer1::DataType, SizeType32>> { }; TEST_P(ParamTest, Test) { - tensorrt_llm::DataType const dtype{std::get<0>(GetParam())}; + nvinfer1::DataType const dtype{std::get<0>(GetParam())}; SizeType32 const beamWidth{std::get<1>(GetParam())}; SamplingConfig const samplingConfig{beamWidth}; @@ -260,11 +259,10 @@ TEST_P(ParamTest, Test) } INSTANTIATE_TEST_SUITE_P(DecoderTest, ParamTest, - testing::Combine( - testing::Values(tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kHALF), testing::Values(1, 3)), + testing::Combine(testing::Values(nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kHALF), testing::Values(1, 3)), [](testing::TestParamInfo<ParamTest::ParamType> const& info) { - std::string name{std::get<0>(info.param) == tensorrt_llm::DataType::kFLOAT ? "Float" : "Half"}; + std::string name{std::get<0>(info.param) == nvinfer1::DataType::kFLOAT ? "Float" : "Half"}; auto const beamWidth = std::get<1>(info.param); name.append(beamWidth == 1 ? "Sampling" : "BeamWidth" + std::to_string(beamWidth)); return name; diff --git a/cpp/tests/unit_tests/runtime/iTensorTest.cpp b/cpp/tests/unit_tests/runtime/iTensorTest.cpp index 4637474f72c7..54ba8aa3beec 100644 --- a/cpp/tests/unit_tests/runtime/iTensorTest.cpp +++ b/cpp/tests/unit_tests/runtime/iTensorTest.cpp @@ -17,7 +17,6 @@ #include <gmock/gmock.h> #include <gtest/gtest.h> -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -26,7 +25,7 @@ using namespace tensorrt_llm::runtime; TEST(ITensorTest, SqueezeTensor) { auto dims = ITensor::makeShape({16, 1, 4}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor{BufferManager::cpu(dims, dataType)}; auto squeezeDim = 0; @@ -103,7 +102,7 @@ TEST(ITensorTest, UnsqueezeTensor) auto oldShape = ITensor::makeShape({2, 3, 4, 5}); { - auto tensor = BufferManager::cpu(oldShape, tensorrt_llm::DataType::kINT32); + auto tensor = BufferManager::cpu(oldShape, nvinfer1::DataType::kINT32); tensor->unsqueeze(0); auto shape = tensor->getShape(); @@ -115,7 +114,7 @@ TEST(ITensorTest, UnsqueezeTensor) EXPECT_EQ(shape.d[4], 5); } { - auto tensor = BufferManager::cpu(oldShape, tensorrt_llm::DataType::kINT32); + auto tensor = BufferManager::cpu(oldShape, nvinfer1::DataType::kINT32); tensor->unsqueeze(1); auto shape = tensor->getShape(); @@ -128,7 +127,7 @@ TEST(ITensorTest, UnsqueezeTensor) } { - auto tensor = BufferManager::cpu(oldShape, tensorrt_llm::DataType::kINT32); + auto tensor = BufferManager::cpu(oldShape, nvinfer1::DataType::kINT32); tensor->unsqueeze(4); auto shape = tensor->getShape(); @@ -145,7 +144,7 @@ TEST(ITensorTest, UnsqueezeTensor) { try { - auto tensor = BufferManager::cpu(oldShape, tensorrt_llm::DataType::kINT32); + auto tensor = BufferManager::cpu(oldShape, nvinfer1::DataType::kINT32); tensor->unsqueeze(invalidDim); FAIL() << "Expected failure"; } @@ -163,7 +162,7 @@ TEST(ITensorTest, UnsqueezeTensor) TEST(ITensorTest, TensorView) { auto const dims = ITensor::makeShape({16, 1, 4}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor = BufferManager::cpu(dims, dataType); auto const viewDims = ITensor::makeShape({16, 1, 2}); @@ -181,7 +180,7 @@ TEST(ITensorTest, TensorView) TEST(ITensorTest, TensorSlice) { auto dims = ITensor::makeShape({16, 8, 4}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor{BufferManager::cpu(dims, dataType)}; auto offset = dims.d[0] / 4; auto slice = ITensor::slice(tensor, offset); @@ -222,7 +221,7 @@ TEST(ITensorTest, TensorSlice) TEST(ITensorTest, TensorDimsSliceAtManual) { auto shape = ITensor::makeShape({5, 5, 5, 5, 5}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor(BufferManager::cpu(shape, dataType)); auto offsetDims = ITensor::makeShape({4, 3, 3}); auto sizeDim = 2; @@ -283,7 +282,7 @@ TEST(ITensorTest, TensorDimsSliceAtManual) TEST(ITensorTest, TensorDimsSliceAtExtrame) { - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; { auto shape = ITensor::makeShape({5, 5, 5, 5, 5}); ITensor::SharedPtr tensor(BufferManager::cpu(shape, dataType)); @@ -541,7 +540,7 @@ TEST(ShapeRange, test) TEST(ITensorTest, TensorDimsSliceAt) { auto shape = ITensor::makeShape({5, 5, 5, 5}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor(BufferManager::cpu(shape, dataType)); auto verify = [&shape, &tensor, &dataType](ITensor::Shape const& index) @@ -658,7 +657,7 @@ TEST(ITensorTest, TensorDimsSliceAt) TEST(BufferRangeTest, ConstType) { auto shape = ITensor::makeShape({5, 5, 5, 5, 5}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor(BufferManager::cpu(shape, dataType)); ITensor::SharedConstPtr tensorConst = tensor; @@ -695,7 +694,7 @@ TEST(BufferRangeTest, ConstType) TEST(ITensorTest, GetDimension) { auto shape = ITensor::makeShape({10, 11, 12}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor(BufferManager::cpu(shape, dataType)); auto firstDimensionFromStart = tensor->getDimension<0>(); diff --git a/cpp/tests/unit_tests/runtime/loraCacheTest.cpp b/cpp/tests/unit_tests/runtime/loraCacheTest.cpp index 6a91d11df3db..4d4dc86dc824 100644 --- a/cpp/tests/unit_tests/runtime/loraCacheTest.cpp +++ b/cpp/tests/unit_tests/runtime/loraCacheTest.cpp @@ -28,7 +28,7 @@ #include "tensorrt_llm/runtime/utils/numpyUtils.h" #include "tensorrt_llm/runtime/worldConfig.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <gmock/gmock-matchers.h> #include <gmock/gmock.h> @@ -78,7 +78,7 @@ class LoraCacheTest : public ::testing::Test, void SetUp() override { - mModelConfig = std::make_unique<ModelConfig>(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + mModelConfig = std::make_unique<ModelConfig>(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); mModelConfig->setMlpHiddenSize(32); mWorldConfig = std::make_unique<WorldConfig>(2, 1, 1, 0); std::vector<LoraModule> modules{ @@ -101,7 +101,7 @@ class LoraCacheTest : public ::testing::Test, mManager = std::make_unique<BufferManager>(mStream); auto pageConfig = LoraCachePageManagerConfig( - runtime::MemoryType::kCPU, tensorrt_llm::DataType::kFLOAT, 2 * 8, 6, 64, 4 * 16, 1); + runtime::MemoryType::kCPU, nvinfer1::DataType::kFLOAT, 2 * 8, 6, 64, 4 * 16, 1); pageConfig.setInitToZero(true); auto pageConfig2 = pageConfig; pageConfig2.setInitToZero(true); @@ -125,7 +125,7 @@ TEST_F(LoraCacheTest, LoraCachePageManagerTest) auto pageShape = ITensor::makeShape({maxAdapterSize, maxAdapterWeights}); LoraCachePageManagerConfig config( - runtime::MemoryType::kCPU, tensorrt_llm::DataType::kFLOAT, 8, 6, maxAdapterSize, maxAdapterWeights, 1); + runtime::MemoryType::kCPU, nvinfer1::DataType::kFLOAT, 8, 6, maxAdapterSize, maxAdapterWeights, 1); LoraCachePageManager manager(config, *mManager); auto block0 = manager.blockPtr(0); @@ -182,11 +182,11 @@ TEST_F(LoraCacheTest, LoraCachePageManagerTest) TEST_F(LoraCacheTest, determineNumPages) { - ModelConfig modelConfig(0, 2, 2, 0, 1, 4, tensorrt_llm::DataType::kFLOAT); + ModelConfig modelConfig(0, 2, 2, 0, 1, 4, nvinfer1::DataType::kFLOAT); modelConfig.setLoraModules(LoraModule::createLoraModules({"attn_dense", "attn_qkv"}, 4, 4, 1, 1, 2, 2, 0)); WorldConfig worldConfig(1, 1, 1, 0); - LoraCachePageManagerConfig pageConfig(MemoryType::kCPU, tensorrt_llm::DataType::kFLOAT, 12393, 40, 80, 16, 1); + LoraCachePageManagerConfig pageConfig(MemoryType::kCPU, nvinfer1::DataType::kFLOAT, 12393, 40, 80, 16, 1); LoraCache cache(pageConfig, modelConfig, worldConfig, *mManager); @@ -374,7 +374,7 @@ TEST_F(LoraCacheTest, basicPutGet) TEST_F(LoraCacheTest, splitTransposeCpu) { - auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); auto worldConfig = WorldConfig(2, 1, 1, 0); SizeType32 const split{2}; @@ -391,8 +391,8 @@ TEST_F(LoraCacheTest, splitTransposeCpu) auto const outputShape = ITensor::makeShape({batchSize, inputLength / split}); auto inputTensor = mManager->copyFrom(input, inputShape, MemoryType::kCPU); - auto outputTensorRank0 = mManager->cpu(outputShape, tensorrt_llm::DataType::kINT32); - auto outputTensorRank1 = mManager->cpu(outputShape, tensorrt_llm::DataType::kINT32); + auto outputTensorRank0 = mManager->cpu(outputShape, nvinfer1::DataType::kINT32); + auto outputTensorRank1 = mManager->cpu(outputShape, nvinfer1::DataType::kINT32); mManager->setZero(*outputTensorRank0); mManager->setZero(*outputTensorRank1); @@ -416,8 +416,8 @@ TEST_F(LoraCacheTest, splitTransposeCpu) auto const outputShape = ITensor::makeShape({batchSize, inputLength / split}); auto inputTensor = mManager->copyFrom(input, inputShape, MemoryType::kCPU); - auto outputTensorRank0 = mManager->cpu(outputShape, tensorrt_llm::DataType::kINT32); - auto outputTensorRank1 = mManager->cpu(outputShape, tensorrt_llm::DataType::kINT32); + auto outputTensorRank0 = mManager->cpu(outputShape, nvinfer1::DataType::kINT32); + auto outputTensorRank1 = mManager->cpu(outputShape, nvinfer1::DataType::kINT32); mManager->setZero(*outputTensorRank0); mManager->setZero(*outputTensorRank1); @@ -438,7 +438,7 @@ TEST_F(LoraCacheTest, splitTransposeCpu) TEST_P(LoraCacheTest, copyToPages_tp1) { bool const isDora = GetParam(); - auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); modelConfig.setMlpHiddenSize(32); auto worldConfig = WorldConfig(1, 1, 1, 0); std::vector<LoraModule> modules{ @@ -501,7 +501,7 @@ TEST_P(LoraCacheTest, copyToPages_tp1) TEST_P(LoraCacheTest, copyToPages_tp2_rank0) { bool const isDora = GetParam(); - auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); modelConfig.setMlpHiddenSize(32); auto worldConfig = WorldConfig(2, 1, 1, 0); std::vector<LoraModule> modules{ @@ -562,7 +562,7 @@ TEST_P(LoraCacheTest, copyToPages_tp2_rank0) TEST_P(LoraCacheTest, copyToPages_tp2_rank1) { bool const isDora = GetParam(); - auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); modelConfig.setMlpHiddenSize(32); auto worldConfig = WorldConfig(2, 1, 1, 1); std::vector<LoraModule> modules{ diff --git a/cpp/tests/unit_tests/runtime/loraManagerTest.cpp b/cpp/tests/unit_tests/runtime/loraManagerTest.cpp index 11c19d22efb0..6910719da76f 100644 --- a/cpp/tests/unit_tests/runtime/loraManagerTest.cpp +++ b/cpp/tests/unit_tests/runtime/loraManagerTest.cpp @@ -32,7 +32,6 @@ #include "tensorrt_llm/runtime/modelConfig.h" #include "tensorrt_llm/runtime/worldConfig.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/utils/numpyUtils.h" #include <gtest/gtest.h> @@ -67,7 +66,7 @@ class LoraManagerTest { protected: LoraManagerTest() - : mModelConfig(1, 2, 2, 0, 1, 4, tensorrt_llm::DataType::kFLOAT) + : mModelConfig(1, 2, 2, 0, 1, 4, nvinfer1::DataType::kFLOAT) { } @@ -88,7 +87,7 @@ class LoraManagerTest PeftTable getPeftTable(SizeType32 tpRank = 0) { - auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); modelConfig.setMlpHiddenSize(32); auto worldConfig = WorldConfig(2, 2, 1, 3); std::vector<LoraModule> modules{ @@ -103,7 +102,7 @@ class LoraManagerTest }; modelConfig.setLoraModules(modules); auto pageConfig = LoraCachePageManagerConfig( - runtime::MemoryType::kCPU, tensorrt_llm::DataType::kFLOAT, 2 * 8, 6, 64, 4 * 16, 1); + runtime::MemoryType::kCPU, nvinfer1::DataType::kFLOAT, 2 * 8, 6, 64, 4 * 16, 1); pageConfig.setInitToZero(true); LoraCache loraCache(pageConfig, modelConfig, worldConfig, *mManager); @@ -214,7 +213,7 @@ static void checkLoraTensors(LoraManager const& loraManager, std::vector<int64_t auto expectedTensor = expectedTensors.find(fieldName)->second; auto actualTensor = inputTensors.find(fieldName)->second; ITensor::shapeEquals(expectedTensor->getShape(), actualTensor->getShape()); - if (expectedTensor->getDataType() == tensorrt_llm::DataType::kINT64) + if (expectedTensor->getDataType() == nvinfer1::DataType::kINT64) { auto expT = bufferCast<int64_t>(*expectedTensor); auto actT = bufferCast<int64_t>(*actualTensor); @@ -309,7 +308,7 @@ TEST_P(LoraManagerTest, fillInputTensors) bool const isDora = GetParam(); LoraManager loraManager; - auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, tensorrt_llm::DataType::kFLOAT); + auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 16, nvinfer1::DataType::kFLOAT); modelConfig.setMlpHiddenSize(32); auto worldConfig = WorldConfig(1, 1, 1, 0); std::vector<LoraModule> modules{ @@ -333,9 +332,9 @@ TEST_P(LoraManagerTest, fillInputTensors) auto numLayers = static_cast<SizeType32>(modelConfig.getNbAttentionLayers()); SizeType32 numSeqs = 4; TensorPtr weightsPtrs - = mManager->cpu(ITensor::makeShape({numModules, numLayers, numSeqs, 3}), tensorrt_llm::DataType::kINT64); + = mManager->cpu(ITensor::makeShape({numModules, numLayers, numSeqs, 3}), nvinfer1::DataType::kINT64); TensorPtr adapterSizes - = mManager->cpu(ITensor::makeShape({numModules, numLayers, numSeqs}), tensorrt_llm::DataType::kINT32); + = mManager->cpu(ITensor::makeShape({numModules, numLayers, numSeqs}), nvinfer1::DataType::kINT32); mManager->setZero(*weightsPtrs); mManager->setZero(*adapterSizes); diff --git a/cpp/tests/unit_tests/runtime/loraUtilsTest.cpp b/cpp/tests/unit_tests/runtime/loraUtilsTest.cpp index 994a77acf818..a14fa7bb8c47 100644 --- a/cpp/tests/unit_tests/runtime/loraUtilsTest.cpp +++ b/cpp/tests/unit_tests/runtime/loraUtilsTest.cpp @@ -25,7 +25,7 @@ #include "tensorrt_llm/runtime/modelConfig.h" #include "tensorrt_llm/runtime/worldConfig.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <algorithm> #include <optional> @@ -53,7 +53,7 @@ class LoraUtilsTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-typ TEST_F(LoraUtilsTest, null_values) { std::optional<TensorPtr> optReqLoraWeights = std::nullopt; - std::optional<TensorPtr> optReqLoraConfig = mManager->emptyTensor(MemoryType::kCPU, tensorrt_llm::DataType::kHALF); + std::optional<TensorPtr> optReqLoraConfig = mManager->emptyTensor(MemoryType::kCPU, nvinfer1::DataType::kHALF); EXPECT_THAT([&]() { loraValidateRequestTensorDims(optReqLoraWeights, optReqLoraConfig); }, testing::Throws<std::runtime_error>()); @@ -66,35 +66,33 @@ TEST_F(LoraUtilsTest, null_values) TEST_F(LoraUtilsTest, dims_mem_type) { - std::optional<TensorPtr> optReqLoraWeights - = mManager->cpu(ITensor::makeShape({1, 2}), tensorrt_llm::DataType::kHALF); + std::optional<TensorPtr> optReqLoraWeights = mManager->cpu(ITensor::makeShape({1, 2}), nvinfer1::DataType::kHALF); std::optional<TensorPtr> optReqLoraConfig - = mManager->cpu(ITensor::makeShape({1, 2, 3}), tensorrt_llm::DataType::kINT32); + = mManager->cpu(ITensor::makeShape({1, 2, 3}), nvinfer1::DataType::kINT32); EXPECT_THAT([&]() { loraValidateRequestTensorDims(optReqLoraWeights, optReqLoraConfig); }, testing::Throws<std::runtime_error>()); - std::optional<TensorPtr> optGpuWeights - = mManager->gpu(ITensor::makeShape({1, 2, 50}), tensorrt_llm::DataType::kHALF); + std::optional<TensorPtr> optGpuWeights = mManager->gpu(ITensor::makeShape({1, 2, 50}), nvinfer1::DataType::kHALF); EXPECT_THAT([&]() { loraValidateRequestTensorDims(optGpuWeights, optReqLoraConfig); }, testing::Throws<std::runtime_error>()); - optReqLoraWeights = mManager->cpu(ITensor::makeShape({1, 2, 50}), tensorrt_llm::DataType::kHALF); - optReqLoraConfig = mManager->cpu(ITensor::makeShape({1, 2, 3}), tensorrt_llm::DataType::kINT32); + optReqLoraWeights = mManager->cpu(ITensor::makeShape({1, 2, 50}), nvinfer1::DataType::kHALF); + optReqLoraConfig = mManager->cpu(ITensor::makeShape({1, 2, 3}), nvinfer1::DataType::kINT32); loraValidateRequestTensorDims(optReqLoraWeights, optReqLoraConfig); } TEST_F(LoraUtilsTest, loraValidateRequestTensors) { - auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 4, tensorrt_llm::DataType::kFLOAT); + auto modelConfig = ModelConfig(0, 2, 2, 0, 1, 4, nvinfer1::DataType::kFLOAT); auto worldConfig = WorldConfig(); std::optional<TensorPtr> optReqLoraWeights - = mManager->cpu(ITensor::makeShape({1, 2, 32}), tensorrt_llm::DataType::kFLOAT); + = mManager->cpu(ITensor::makeShape({1, 2, 32}), nvinfer1::DataType::kFLOAT); std::optional<TensorPtr> optReqLoraConfig - = mManager->cpu(ITensor::makeShape({1, 2, 3}), tensorrt_llm::DataType::kINT32); + = mManager->cpu(ITensor::makeShape({1, 2, 3}), nvinfer1::DataType::kINT32); std::vector<int32_t> config{1, 0, 4, 1, 1, 4}; diff --git a/cpp/tests/unit_tests/runtime/medusaModuleTest.cpp b/cpp/tests/unit_tests/runtime/medusaModuleTest.cpp index 8e7ff6c75459..3cba1bf2994d 100644 --- a/cpp/tests/unit_tests/runtime/medusaModuleTest.cpp +++ b/cpp/tests/unit_tests/runtime/medusaModuleTest.cpp @@ -20,7 +20,7 @@ #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/utils/speculativeChoicesUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <gmock/gmock-matchers.h> #include <gmock/gmock.h> @@ -58,16 +58,14 @@ class MedusaModuleTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro- auto const tokensPerStep = medusaModule.getMaxDecodingTokens(); // batch size = 1 here. - TensorPtr medusaGenerationLengthsHost - = mManager->pinned(ITensor::makeShape({1}), tensorrt_llm::DataType::kINT32); + TensorPtr medusaGenerationLengthsHost = mManager->pinned(ITensor::makeShape({1}), nvinfer1::DataType::kINT32); TensorPtr medusaPositionOffsetsHost - = mManager->pinned(ITensor::makeShape({tokensPerStep}), tensorrt_llm::DataType::kINT32); - TensorPtr medusaTreeIdsHost - = mManager->pinned(ITensor::makeShape({tokensPerStep}), tensorrt_llm::DataType::kINT32); + = mManager->pinned(ITensor::makeShape({tokensPerStep}), nvinfer1::DataType::kINT32); + TensorPtr medusaTreeIdsHost = mManager->pinned(ITensor::makeShape({tokensPerStep}), nvinfer1::DataType::kINT32); TensorPtr medusaPathsHost - = mManager->pinned(ITensor::makeShape({tokensPerStep, medusaHeads + 1}), tensorrt_llm::DataType::kINT32); + = mManager->pinned(ITensor::makeShape({tokensPerStep, medusaHeads + 1}), nvinfer1::DataType::kINT32); TensorPtr attentionPackedMaskHost - = mManager->pinned(ITensor::makeShape({tokensPerStep, numPackedMasks}), tensorrt_llm::DataType::kINT32); + = mManager->pinned(ITensor::makeShape({tokensPerStep, numPackedMasks}), nvinfer1::DataType::kINT32); std::vector<SizeType32> topKs; utils::initTensorsFromChoices(medusaModule, choices, topKs, medusaGenerationLengthsHost, diff --git a/cpp/tests/unit_tests/runtime/runtimeKernelTest.cpp b/cpp/tests/unit_tests/runtime/runtimeKernelTest.cpp index 58372a6bd479..dd517c5de5a4 100644 --- a/cpp/tests/unit_tests/runtime/runtimeKernelTest.cpp +++ b/cpp/tests/unit_tests/runtime/runtimeKernelTest.cpp @@ -23,7 +23,7 @@ #include "tensorrt_llm/runtime/iTensor.h" #include "tensorrt_llm/runtime/runtimeKernels.h" -#include "tensorrt_llm/common/tllmDataType.h" +#include <NvInferRuntime.h> #include <gtest/gtest.h> #include <algorithm> @@ -85,7 +85,7 @@ TEST_F(RuntimeKernelTest, FillBufferInt8) { for (auto size : {123LLU, 1025LLU, 1LLU << 32}) { - auto buffer = mManager->gpu(size, tensorrt_llm::DataType::kINT8); + auto buffer = mManager->gpu(size, nvinfer1::DataType::kINT8); testFill<std::int8_t>(*buffer, *mManager, *mStream); } } @@ -94,7 +94,7 @@ TEST_F(RuntimeKernelTest, FillTensorInt8) { for (auto size : {123, 1025, std::numeric_limits<int32_t>::max()}) { - auto tensor = mManager->gpu(tr::ITensor::makeShape({size, 2}), tensorrt_llm::DataType::kINT8); + auto tensor = mManager->gpu(tr::ITensor::makeShape({size, 2}), nvinfer1::DataType::kINT8); testFill<std::int8_t>(*tensor, *mManager, *mStream); } } @@ -111,7 +111,7 @@ TEST_F(RuntimeKernelTest, ScatterHalf) auto const outputShape = tr::ITensor::makeShape({batchSize * beamWidth, inputLength}); auto inputTensor = mManager->copyFrom(input, inputShape, tr::MemoryType::kGPU); - auto outputTensor = mManager->gpu(outputShape, tensorrt_llm::DataType::kHALF); + auto outputTensor = mManager->gpu(outputShape, nvinfer1::DataType::kHALF); mManager->setZero(*outputTensor); tr::kernels::scatterTensor(*outputTensor, *inputTensor, beamWidth, *mStream); @@ -174,7 +174,7 @@ TEST_F(RuntimeKernelTest, TileInt32) auto const outputShape = tr::ITensor::makeShape({batchSize * beamWidth, inputLength}); auto inputTensor = mManager->copyFrom(input, inputShape, tr::MemoryType::kGPU); - auto outputTensor = mManager->gpu(outputShape, tensorrt_llm::DataType::kINT32); + auto outputTensor = mManager->gpu(outputShape, nvinfer1::DataType::kINT32); tr::kernels::tileTensor(*outputTensor, *inputTensor, beamWidth, *mStream); @@ -194,7 +194,7 @@ TEST_F(RuntimeKernelTest, TileHalf) auto const outputShape = tr::ITensor::makeShape({batchSize * beamWidth, inputLength}); auto inputTensor = mManager->copyFrom(input, inputShape, tr::MemoryType::kGPU); - auto outputTensor = mManager->gpu(outputShape, tensorrt_llm::DataType::kHALF); + auto outputTensor = mManager->gpu(outputShape, nvinfer1::DataType::kHALF); tr::kernels::tileTensor(*outputTensor, *inputTensor, beamWidth, *mStream); @@ -228,11 +228,11 @@ TEST_F(RuntimeKernelTest, TileInt8Large) // Scope the allocated tensors to ensure they are de-allocated before the test ends. { - auto inputTensor = mManager->gpu(inputShape, tensorrt_llm::DataType::kINT8); + auto inputTensor = mManager->gpu(inputShape, nvinfer1::DataType::kINT8); tr::kernels::invokeFill(*inputTensor, value, *mStream); mStream->synchronize(); - auto outputTensor = mManager->gpu(outputShape, tensorrt_llm::DataType::kINT8); + auto outputTensor = mManager->gpu(outputShape, nvinfer1::DataType::kINT8); tr::kernels::tileTensor(*outputTensor, *inputTensor, beamWidth, *mStream); mStream->synchronize(); @@ -257,11 +257,11 @@ void testCopyBatch(tr::SizeType64 stride, tr::BufferManager& manager, tr::CudaSt auto const bufferShape = tr::ITensor::makeShape({rows, stride}); auto const indicesShape = tr::ITensor::makeShape({numIndices}); - auto srcBufferHost = tr::BufferManager::cpu(bufferShape, tensorrt_llm::DataType::kINT32); - auto dstBufferDevice = manager.gpu(bufferShape, tensorrt_llm::DataType::kINT32); - auto srcOffsets = tr::BufferManager::pinned(indicesShape, tensorrt_llm::DataType::kINT64); - auto dstOffsets = tr::BufferManager::pinned(indicesShape, tensorrt_llm::DataType::kINT64); - auto sizes = tr::BufferManager::pinned(indicesShape, tensorrt_llm::DataType::kINT64); + auto srcBufferHost = tr::BufferManager::cpu(bufferShape, nvinfer1::DataType::kINT32); + auto dstBufferDevice = manager.gpu(bufferShape, nvinfer1::DataType::kINT32); + auto srcOffsets = tr::BufferManager::pinned(indicesShape, nvinfer1::DataType::kINT64); + auto dstOffsets = tr::BufferManager::pinned(indicesShape, nvinfer1::DataType::kINT64); + auto sizes = tr::BufferManager::pinned(indicesShape, nvinfer1::DataType::kINT64); tr::kernels::invokeFill(*dstBufferDevice, 0, stream); auto* srcBufferHostPtr = tr::bufferCast<std::int32_t>(*srcBufferHost); diff --git a/cpp/tests/unit_tests/runtime/samplingTest.cpp b/cpp/tests/unit_tests/runtime/samplingTest.cpp index bb93478cf8ff..dad99323164e 100644 --- a/cpp/tests/unit_tests/runtime/samplingTest.cpp +++ b/cpp/tests/unit_tests/runtime/samplingTest.cpp @@ -14,12 +14,12 @@ * limitations under the License. */ -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/executor/types.h" #include "tensorrt_llm/layers/dynamicDecodeLayer.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaStream.h" #include "tensorrt_llm/runtime/gptDecoder.h" +#include "tensorrt_llm/runtime/tllmLogger.h" #include <gtest/gtest.h> @@ -39,11 +39,14 @@ class SamplingTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type if (mDeviceCount == 0) GTEST_SKIP() << "No GPUs found"; + + mLogger = std::make_shared<TllmLogger>(); } void TearDown() override {} int mDeviceCount; + std::shared_ptr<nvinfer1::ILogger> mLogger; }; std::shared_ptr<tl::BaseDecodingOutputs> dynamicDecodeTest(std::shared_ptr<BufferManager> manager, size_t vocabSize, @@ -64,10 +67,10 @@ std::shared_ptr<tl::BaseDecodingOutputs> dynamicDecodeTest(std::shared_ptr<Buffe tk::FinishedState::UnderlyingType* gpuFinished = nullptr; - ITensor::SharedPtr gpuEndIds = manager->gpu(ITensor::makeShape({signedBatchSize}), tensorrt_llm::DataType::kINT32); + ITensor::SharedPtr gpuEndIds = manager->gpu(ITensor::makeShape({signedBatchSize}), nvinfer1::DataType::kINT32); manager->copy(cpuEndIds.data(), *gpuEndIds, MemoryType::kCPU); ITensor::SharedPtr gpuOutputIds = manager->gpu( - ITensor::makeShape({signedBatchSize, signedBeamWidth, signedMaxSeqLength}), tensorrt_llm::DataType::kINT32); + ITensor::makeShape({signedBatchSize, signedBeamWidth, signedMaxSeqLength}), nvinfer1::DataType::kINT32); manager->copy(cpuOutputIds.data(), *gpuOutputIds, MemoryType::kCPU); auto const decodingMode = beamWidth == 1 ? tle::DecodingMode::TopKTopP() : tle::DecodingMode::BeamSearch(); @@ -89,7 +92,7 @@ std::shared_ptr<tl::BaseDecodingOutputs> dynamicDecodeTest(std::shared_ptr<Buffe auto forwardParams = std::make_shared<tl::SamplingInputs>(gpuEndIds, batchSlots, step, ite, localBatchSize); auto logitsShape = ITensor::makeShape({signedBatchSize, static_cast<int64_t>(beamWidth), static_cast<int64_t>(vocabSizePadded)}); - ITensor::SharedPtr inputLogits = manager->gpu(logitsShape, tensorrt_llm::DataType::kFLOAT); + ITensor::SharedPtr inputLogits = manager->gpu(logitsShape, nvinfer1::DataType::kFLOAT); forwardParams->logits = inputLogits; manager->copy(cpuLogits.data(), *inputLogits, MemoryType::kCPU); @@ -98,10 +101,10 @@ std::shared_ptr<tl::BaseDecodingOutputs> dynamicDecodeTest(std::shared_ptr<Buffe forwardParams->stopCriteriaInputs = std::make_shared<tl::StopCriteriaDecodingInputs>(localBatchSize); auto outputParams = std::make_shared<tl::BaseDecodingOutputs>(gpuOutputIds); - outputParams->sequenceLength = manager->gpu(ITensor::makeShape({signedBatchSize}), tensorrt_llm::DataType::kINT32); + outputParams->sequenceLength = manager->gpu(ITensor::makeShape({signedBatchSize}), nvinfer1::DataType::kINT32); manager->copy(cpuSequenceLengths.data(), *outputParams->sequenceLength.value(), MemoryType::kCPU); outputParams->newTokens - = manager->gpu(ITensor::makeShape({signedBatchSize, signedBeamWidth}), tensorrt_llm::DataType::kINT32); + = manager->gpu(ITensor::makeShape({signedBatchSize, signedBeamWidth}), nvinfer1::DataType::kINT32); outputParams->finished = manager->gpu( ITensor::makeShape({signedBatchSize, signedBeamWidth}), TRTDataType<tk::FinishedState::UnderlyingType>::value); diff --git a/cpp/tests/unit_tests/runtime/tllmBuffersTest.cpp b/cpp/tests/unit_tests/runtime/tllmBuffersTest.cpp index 4080a0f29e9e..c901061695bb 100644 --- a/cpp/tests/unit_tests/runtime/tllmBuffersTest.cpp +++ b/cpp/tests/unit_tests/runtime/tllmBuffersTest.cpp @@ -19,7 +19,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/stringUtils.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/cudaMemPool.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -231,7 +230,7 @@ TEST_F(TllmBuffersTest, DeviceBuffer) { CudaAllocatorAsync allocator{mStream, mMemPool}; { - DeviceBuffer buffer{size, tensorrt_llm::DataType::kFLOAT, allocator}; + DeviceBuffer buffer{size, nvinfer1::DataType::kFLOAT, allocator}; testBuffer(buffer, sizeof(float)); } streamPtr->synchronize(); @@ -243,7 +242,7 @@ TEST_F(TllmBuffersTest, DeviceBuffer) { CudaAllocator allocator{}; { - StaticDeviceBuffer buffer{size, tensorrt_llm::DataType::kFLOAT, allocator}; + StaticDeviceBuffer buffer{size, nvinfer1::DataType::kFLOAT, allocator}; testBuffer(buffer, sizeof(float)); } streamPtr->synchronize(); @@ -264,10 +263,10 @@ TEST_F(TllmBuffersTest, DeviceTensor) GTEST_SKIP() << noPoolSkipReason; } auto streamPtr = std::make_shared<CudaStream>(); - tensorrt_llm::Dims constexpr dims{3, 16, 8, 4}; + nvinfer1::Dims constexpr dims{3, 16, 8, 4}; CudaAllocatorAsync allocator{streamPtr, mMemPool}; { - DeviceTensor tensor{dims, tensorrt_llm::DataType::kFLOAT, allocator}; + DeviceTensor tensor{dims, nvinfer1::DataType::kFLOAT, allocator}; EXPECT_EQ(tensor.getSize(), ITensor::volume(dims)); testBuffer(tensor, sizeof(float)); EXPECT_EQ(tensor.getSize(), ITensor::volume(tensor.getShape())); @@ -282,7 +281,7 @@ TEST_F(TllmBuffersTest, BufferSlice) { auto constexpr size = 1024; HostAllocator allocator{}; - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; auto buffer = std::make_shared<HostBuffer>(size, dataType, allocator); auto offset = size / 8; auto slice = IBuffer::slice(buffer, offset); @@ -320,7 +319,7 @@ TEST_F(TllmBuffersTest, BufferOutput) CudaAllocatorAsync allocator{streamPtr, mMemPool}; for (std::size_t size : {0, 16}) { - DeviceBuffer buffer{size, tensorrt_llm::DataType::kFLOAT, allocator}; + DeviceBuffer buffer{size, nvinfer1::DataType::kFLOAT, allocator}; TLLM_CUDA_CHECK(cudaMemsetAsync(buffer.data(), 0, buffer.getSizeInBytes(), streamPtr->get())); streamPtr->synchronize(); std::stringstream ss; @@ -344,11 +343,11 @@ TEST_F(TllmBuffersTest, TensorOutput) } auto streamPtr = std::make_shared<CudaStream>(); - tensorrt_llm::Dims constexpr dims{3, 16, 8, 4}; + nvinfer1::Dims constexpr dims{3, 16, 8, 4}; CudaAllocatorAsync allocator{streamPtr, mMemPool}; - for (auto dataType : {tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kHALF, tensorrt_llm::DataType::kBOOL, - tensorrt_llm::DataType::kINT8, tensorrt_llm::DataType::kINT32, tensorrt_llm::DataType::kINT64, - tensorrt_llm::DataType::kUINT8}) + for (auto dataType : + {nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kHALF, nvinfer1::DataType::kBOOL, nvinfer1::DataType::kINT8, + nvinfer1::DataType::kINT32, nvinfer1::DataType::kINT64, nvinfer1::DataType::kUINT8}) { DeviceTensor tensor{dims, dataType, allocator}; TLLM_CUDA_CHECK(cudaMemsetAsync(tensor.data(), 0, tensor.getSizeInBytes(), streamPtr->get())); @@ -484,8 +483,8 @@ TEST_F(TllmBuffersTest, PinnedPoolAllocator) EXPECT_EQ(segments.size(), 0); { - auto a = BufferManager::pinnedPool(ITensor::makeShape({512, 4, 4}), tensorrt_llm::DataType::kFLOAT); - auto b = BufferManager::pinnedPool(ITensor::makeShape({512, 10}), tensorrt_llm::DataType::kHALF); + auto a = BufferManager::pinnedPool(ITensor::makeShape({512, 4, 4}), nvinfer1::DataType::kFLOAT); + auto b = BufferManager::pinnedPool(ITensor::makeShape({512, 10}), nvinfer1::DataType::kHALF); pool.logSegments(); auto it = std::begin(segments); EXPECT_NE(it->tag, nullptr); @@ -513,7 +512,7 @@ TEST_F(TllmBuffersTest, PinnedPoolAllocator) std::size_t secondChunkSize; { // Test creating a new chunk - auto c = BufferManager::pinnedPool(ITensor::makeShape({initChunkSize + 1}), tensorrt_llm::DataType::kUINT8); + auto c = BufferManager::pinnedPool(ITensor::makeShape({initChunkSize + 1}), nvinfer1::DataType::kUINT8); pool.logSegments(); auto it = std::begin(segments); EXPECT_EQ(it->tag, nullptr); diff --git a/cpp/tests/unit_tests/runtime/tllmRuntimeTest.cpp b/cpp/tests/unit_tests/runtime/tllmRuntimeTest.cpp new file mode 100644 index 000000000000..b3a6f99146c2 --- /dev/null +++ b/cpp/tests/unit_tests/runtime/tllmRuntimeTest.cpp @@ -0,0 +1,147 @@ +/* + * Copyright (c) 2022-2024, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef TOP_LEVEL_DIR +#error "Define TOP_LEVEL_DIR" +#endif + +#include <NvInfer.h> +#include <NvOnnxParser.h> +#include <gtest/gtest.h> + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/runtime/rawEngine.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include "tensorrt_llm/runtime/tllmRuntime.h" + +#include <algorithm> +#include <array> +#include <filesystem> +#include <memory> +#include <vector> + +namespace fs = std::filesystem; +namespace trt = nvinfer1; + +namespace +{ +auto const TEST_RESOURCE_DIR = fs::path{TOP_LEVEL_DIR} / "cpp/tests/resources"; +auto const MNIST_MODEL_PATH = TEST_RESOURCE_DIR / "models/mnist.onnx"; + +template <typename T> +std::unique_ptr<T> makeUnique(T* ptr) +{ + EXPECT_NE(ptr, nullptr); + return std::unique_ptr<T>(ptr); +} + +std::unique_ptr<trt::IHostMemory> buildMnistEngine(trt::ILogger& logger) +{ + EXPECT_TRUE(fs::exists(MNIST_MODEL_PATH)); + auto builder = makeUnique(trt::createInferBuilder(logger)); + auto const explicitBatch = 1U << static_cast<uint32_t>(trt::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH); + auto network = makeUnique(builder->createNetworkV2(explicitBatch)); + auto parser = makeUnique(nvonnxparser::createParser(*network, logger)); + auto const parsingSuccess = parser->parseFromFile( + MNIST_MODEL_PATH.string().c_str(), static_cast<int32_t>(trt::ILogger::Severity::kWARNING)); + EXPECT_TRUE(parsingSuccess); + auto config = makeUnique(builder->createBuilderConfig()); + return makeUnique(builder->buildSerializedNetwork(*network, *config)); +} +} // namespace + +using namespace tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; + +class TllmRuntimeTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type-member-init) +{ +protected: + void SetUp() override + { + mDeviceCount = tc::getDeviceCount(); + + if (mDeviceCount == 0) + GTEST_SKIP(); + + mLogger.setLevel(trt::ILogger::Severity::kINFO); + mSerializedEngine = buildMnistEngine(mLogger); + ASSERT_NE(mSerializedEngine, nullptr); + } + + void TearDown() override {} + + int mDeviceCount; + TllmLogger mLogger{}; + std::unique_ptr<trt::IHostMemory> mSerializedEngine; +}; + +TEST_F(TllmRuntimeTest, SinglePass) +{ + EXPECT_TRUE(mSerializedEngine); + TllmRuntime rt{RawEngine(mSerializedEngine.get()), &mLogger, false, 1.0F}; + auto& engine = rt.getEngine(); + EXPECT_FALSE(engine.hasImplicitBatchDimension()); + EXPECT_EQ(rt.getNbProfiles(), engine.getNbOptimizationProfiles()); + EXPECT_EQ(rt.getNbContexts(), 0); + auto const nbIoTensors = engine.getNbIOTensors(); + EXPECT_EQ(nbIoTensors, 2); + rt.addContext(0); + EXPECT_EQ(rt.getNbContexts(), 1); + + auto constexpr dataType = trt::DataType::kFLOAT; + + auto const inputName = engine.getIOTensorName(0); + EXPECT_EQ(engine.getTensorIOMode(inputName), trt::TensorIOMode::kINPUT); + auto const inputDims = engine.getTensorShape(inputName); + std::array constexpr inputDimsExpected = {1, 1, 28, 28}; + EXPECT_EQ(inputDims.nbDims, inputDimsExpected.size()); + for (int i = 0; i < inputDims.nbDims; ++i) + { + EXPECT_EQ(inputDims.d[i], inputDimsExpected[i]); + } + EXPECT_EQ(engine.getTensorDataType(inputName), dataType); + + auto const outputName = engine.getIOTensorName(1); + EXPECT_EQ(engine.getTensorIOMode(outputName), trt::TensorIOMode::kOUTPUT); + auto const outputDims = engine.getTensorShape(outputName); + std::array constexpr outputDimsExpected = {1, 10}; + EXPECT_EQ(outputDims.nbDims, outputDimsExpected.size()); + for (int i = 0; i < outputDims.nbDims; ++i) + { + EXPECT_EQ(outputDims.d[i], outputDimsExpected[i]); + } + EXPECT_EQ(engine.getTensorDataType(outputName), dataType); + + auto& allocator = rt.getBufferManager(); + TllmRuntime::TensorMap tensorMap{}; + auto inputBuffer = std::shared_ptr<ITensor>{allocator.gpu(inputDims, dataType)}; + allocator.setZero(*inputBuffer); + tensorMap.insert(std::make_pair(inputName, inputBuffer)); + rt.setInputTensors(0, tensorMap); + rt.setOutputTensors(0, tensorMap); + ASSERT_NE(tensorMap.find(outputName), tensorMap.end()); + auto outputBuffer = tensorMap.at(outputName); + allocator.setZero(*outputBuffer); + rt.executeContext(0); + + std::vector<float> output(outputBuffer->getSize()); + allocator.copy(*outputBuffer, output.data()); + rt.getStream().synchronize(); + auto min = std::min_element(output.begin(), output.end()); + EXPECT_NEAR(*min, -0.126409f, 1e-5f); + auto max = std::max_element(output.begin(), output.end()); + EXPECT_NEAR(*max, 0.140218f, 1e-5f); +} diff --git a/cpp/tests/unit_tests/runtime/torchTest.cpp b/cpp/tests/unit_tests/runtime/torchTest.cpp index 4ca36d875d44..4aa498de8d25 100644 --- a/cpp/tests/unit_tests/runtime/torchTest.cpp +++ b/cpp/tests/unit_tests/runtime/torchTest.cpp @@ -17,7 +17,6 @@ #include <gmock/gmock.h> #include <gtest/gtest.h> -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/torch.h" #include "tensorrt_llm/runtime/torchView.h" @@ -53,7 +52,7 @@ class TorchTest : public ::testing::Test // NOLINT(cppcoreguidelines-pro-type-me namespace { -template <tensorrt_llm::DataType DType> +template <nvinfer1::DataType DType> void checkFilled(IBuffer& buffer, int fillValue) { if (DType == buffer.getDataType()) @@ -80,13 +79,13 @@ TEST_F(TorchTest, Aten) } auto constexpr fillValue = 1; - auto tensorHostBase = manager.allocate(MemoryType::kPINNED, shapeTllm, tensorrt_llm::DataType::kINT64); + auto tensorHostBase = manager.allocate(MemoryType::kPINNED, shapeTllm, nvinfer1::DataType::kINT64); for (auto memoryType : {MemoryType::kCPU, MemoryType::kGPU, MemoryType::kPINNED}) { - for (auto dtype : {tensorrt_llm::DataType::kFLOAT, tensorrt_llm::DataType::kHALF, tensorrt_llm::DataType::kINT8, - tensorrt_llm::DataType::kUINT8, tensorrt_llm::DataType::kINT32, tensorrt_llm::DataType::kINT64, - tensorrt_llm::DataType::kBF16, tensorrt_llm::DataType::kFP8, tensorrt_llm::DataType::kBOOL}) + for (auto dtype : {nvinfer1::DataType::kFLOAT, nvinfer1::DataType::kHALF, nvinfer1::DataType::kINT8, + nvinfer1::DataType::kUINT8, nvinfer1::DataType::kINT32, nvinfer1::DataType::kINT64, + nvinfer1::DataType::kBF16, nvinfer1::DataType::kFP8, nvinfer1::DataType::kBOOL}) { ITensor::SharedPtr tensorTllm{manager.allocate(memoryType, shapeTllm, dtype)}; @@ -99,20 +98,20 @@ TEST_F(TorchTest, Aten) EXPECT_THAT(tensorAten.sizes(), ::testing::ElementsAreArray(shapeAten)); EXPECT_EQ(tensorAten.data_ptr(), tensorTllm->data()); - if (dtype != tensorrt_llm::DataType::kFP8) + if (dtype != nvinfer1::DataType::kFP8) { tensorAten.fill_(c10::Scalar(fillValue)); auto tensorHost = ITensor::wrap(tensorHostBase->data(), dtype, shapeTllm); manager.copy(*tensorTllm, *tensorHost); mStream->synchronize(); - checkFilled<tensorrt_llm::DataType::kFLOAT>(*tensorHost, fillValue); - checkFilled<tensorrt_llm::DataType::kHALF>(*tensorHost, fillValue); - checkFilled<tensorrt_llm::DataType::kINT8>(*tensorHost, fillValue); - checkFilled<tensorrt_llm::DataType::kUINT8>(*tensorHost, fillValue); - checkFilled<tensorrt_llm::DataType::kINT32>(*tensorHost, fillValue); - checkFilled<tensorrt_llm::DataType::kINT64>(*tensorHost, fillValue); - checkFilled<tensorrt_llm::DataType::kBF16>(*tensorHost, fillValue); - checkFilled<tensorrt_llm::DataType::kBOOL>(*tensorHost, fillValue); + checkFilled<nvinfer1::DataType::kFLOAT>(*tensorHost, fillValue); + checkFilled<nvinfer1::DataType::kHALF>(*tensorHost, fillValue); + checkFilled<nvinfer1::DataType::kINT8>(*tensorHost, fillValue); + checkFilled<nvinfer1::DataType::kUINT8>(*tensorHost, fillValue); + checkFilled<nvinfer1::DataType::kINT32>(*tensorHost, fillValue); + checkFilled<nvinfer1::DataType::kINT64>(*tensorHost, fillValue); + checkFilled<nvinfer1::DataType::kBF16>(*tensorHost, fillValue); + checkFilled<nvinfer1::DataType::kBOOL>(*tensorHost, fillValue); } // Conversion back to TRT-LLM tensor diff --git a/cpp/tests/unit_tests/runtime/utilsTest.cpp b/cpp/tests/unit_tests/runtime/utilsTest.cpp index 58882824a7dc..8b69070c03d8 100644 --- a/cpp/tests/unit_tests/runtime/utilsTest.cpp +++ b/cpp/tests/unit_tests/runtime/utilsTest.cpp @@ -18,7 +18,6 @@ #error "Define TOP_LEVEL_DIR" #endif -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/iBuffer.h" #include "tensorrt_llm/runtime/iTensor.h" @@ -72,7 +71,7 @@ TEST_F(UtilsTest, LoadNpy) TEST_F(UtilsTest, LoadStoreNpy) { auto dims = ITensor::makeShape({2, 3, 4}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor{BufferManager::cpu(dims, dataType)}; auto tensorRange = BufferRange<float>(*tensor); std::iota(tensorRange.begin(), tensorRange.end(), 0); @@ -97,7 +96,7 @@ TEST_F(UtilsTest, LoadStoreNpy) TEST_F(UtilsTest, LoadStoreNpyGPU) { auto dims = ITensor::makeShape({2, 3, 4}); - auto constexpr dataType = tensorrt_llm::DataType::kFLOAT; + auto constexpr dataType = nvinfer1::DataType::kFLOAT; ITensor::SharedPtr tensor{BufferManager::cpu(dims, dataType)}; auto tensorRange = BufferRange<float>(*tensor); std::iota(tensorRange.begin(), tensorRange.end(), 0); diff --git a/cpp/tests/unit_tests/runtime/virtualMemoryTest.cpp b/cpp/tests/unit_tests/runtime/virtualMemoryTest.cpp index 159a07770694..f2045e7659d1 100644 --- a/cpp/tests/unit_tests/runtime/virtualMemoryTest.cpp +++ b/cpp/tests/unit_tests/runtime/virtualMemoryTest.cpp @@ -19,7 +19,6 @@ #include "tensorrt_llm/common/cudaUtils.h" #include "tensorrt_llm/common/nvmlWrapper.h" -#include "tensorrt_llm/common/tllmDataType.h" #include "tensorrt_llm/runtime/bufferManager.h" #include "tensorrt_llm/runtime/tllmBuffers.h" #include "tensorrt_llm/runtime/virtualMemory.h" @@ -1503,7 +1502,7 @@ TEST_F(VirtualMemoryManagerTest, TestCudaVirtualMemoryAllocator) // Create a buffer using the virtual address allocator auto buffer = std::make_unique<VirtualAddressDeviceBuffer>( - size, tensorrt_llm::DataType::kINT8, CudaVirtualMemoryAllocator{config}); + size, nvinfer1::DataType::kINT8, CudaVirtualMemoryAllocator{config}); auto memoryAfterAllocation = getCurrentProcessMemoryInfo(); if (memoryInfoAvailable()) @@ -1514,7 +1513,7 @@ TEST_F(VirtualMemoryManagerTest, TestCudaVirtualMemoryAllocator) // Test that we can access the buffer data ASSERT_NE(buffer->data(), nullptr) << "Buffer data should not be null"; ASSERT_EQ(buffer->getSize(), size) << "Buffer size should match requested size"; - ASSERT_EQ(buffer->getDataType(), tensorrt_llm::DataType::kINT8) << "Buffer data type should be INT8"; + ASSERT_EQ(buffer->getDataType(), nvinfer1::DataType::kINT8) << "Buffer data type should be INT8"; ASSERT_EQ(buffer->getMemoryType(), MemoryType::kGPU) << "Buffer memory type should be GPU"; // Test memory access by setting memory to a known pattern @@ -1575,7 +1574,7 @@ TEST_F(VirtualMemoryManagerTest, TestCudaVirtualMemoryAllocatorUnalignedSize) // Create a buffer using the virtual address allocator auto buffer = std::make_unique<VirtualAddressDeviceBuffer>( - size, tensorrt_llm::DataType::kINT8, CudaVirtualMemoryAllocator{config}); + size, nvinfer1::DataType::kINT8, CudaVirtualMemoryAllocator{config}); auto memoryAfterAllocation = getCurrentProcessMemoryInfo(); if (memoryInfoAvailable()) diff --git a/cpp/tests/unit_tests/utils/CMakeLists.txt b/cpp/tests/unit_tests/utils/CMakeLists.txt new file mode 100644 index 000000000000..2d7e9145c817 --- /dev/null +++ b/cpp/tests/unit_tests/utils/CMakeLists.txt @@ -0,0 +1,16 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +add_gtest(testUtilsTest utilsTest.cpp) diff --git a/cpp/tests/unit_tests/utils/utilsTest.cpp b/cpp/tests/unit_tests/utils/utilsTest.cpp new file mode 100644 index 000000000000..38587120ef84 --- /dev/null +++ b/cpp/tests/unit_tests/utils/utilsTest.cpp @@ -0,0 +1,55 @@ + +#include "common.h" +#include "tensorrt_llm/runtime/common.h" + +#include <gtest/gtest.h> + +#include <cstdint> +#include <numeric> + +struct RandomLogitsTestParameters +{ + using TupleT = std::tuple<int32_t, tensorrt_llm::runtime::SizeType32>; + + int32_t randomSeed; + tensorrt_llm::runtime::SizeType32 vocabSize; + + // Constructor that takes a tuple + RandomLogitsTestParameters( // NOLINT: implicit to allow gtest to convert from tuple generated by + // 'combine' + TupleT t) + : randomSeed(std::get<0>(t)) + , vocabSize(std::get<1>(t)) + { + } +}; + +class RandomLogits : public ::testing::Test, public ::testing::WithParamInterface<RandomLogitsTestParameters> +{ +protected: + static constexpr int randomSeed = 2345; +}; + +namespace +{ +constexpr int32_t kRandomSeed1 = 45; +constexpr int32_t kRandomSeed2 = 567; +auto const randomSeeds = ::testing::Values(kRandomSeed1, kRandomSeed2); + +constexpr tensorrt_llm::runtime::SizeType32 kMinVocabSize = 16; +constexpr tensorrt_llm::runtime::SizeType32 kMaxVocabSize = 100000; +auto const vocabSizes = ::testing::Values(kMinVocabSize, kMaxVocabSize); + +auto const paramGenerator + = ::testing::ConvertGenerator<RandomLogitsTestParameters::TupleT>(::testing::Combine(randomSeeds, vocabSizes)); +} // namespace + +TEST_P(RandomLogits, FloatSumToOne) +{ + auto rng = std::mt19937(randomSeed); + auto const randomLogits = tensorrt_llm::testing::randomLogits<std::mt19937, float>(456, &rng); + auto const sum = std::reduce(randomLogits.begin(), randomLogits.end()); + ASSERT_DOUBLE_EQ(sum, 1.0); +} + +INSTANTIATE_TEST_SUITE_P(Float, RandomLogits, paramGenerator); diff --git a/cpp/tests/utils/CMakeLists.txt b/cpp/tests/utils/CMakeLists.txt new file mode 100644 index 000000000000..0123ac1c4600 --- /dev/null +++ b/cpp/tests/utils/CMakeLists.txt @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +add_library(testingUtils common.cpp engines.cpp executorUtils.cpp) +target_link_libraries(testingUtils PUBLIC gtest_main ${SHARED_TARGET}) +target_include_directories(testingUtils PRIVATE ${MPI_C_INCLUDE_DIRS}) +target_compile_definitions(testingUtils PUBLIC TOP_LEVEL_DIR="${TOP_LEVEL_DIR}") diff --git a/cpp/tests/utils/common.cpp b/cpp/tests/utils/common.cpp new file mode 100644 index 000000000000..5640cef7b495 --- /dev/null +++ b/cpp/tests/utils/common.cpp @@ -0,0 +1,684 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "common.h" + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/memoryUtils.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/utils/numpyUtils.h" +#include "tensorrt_llm/testing/modelSpec.h" +#include "tests/utils/common.h" + +#include <gtest/gtest.h> + +#include <algorithm> +#include <vector> + +namespace tensorrt_llm::testing +{ +namespace fs = std::filesystem; +namespace tr = tensorrt_llm::runtime; +namespace tc = tensorrt_llm::common; + +std::string PathUtil::FP16_GPT_ATTENTION_PACKED_DIR() +{ + return ModelSpec::getDefaultModelSpec().setKVCacheType(KVCacheType::kCONTINUOUS).getModelPath(); +} + +std::string PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DIR() +{ + return ModelSpec::getDefaultModelSpec().getModelPath(); +} + +std::string PathUtil::FP16_GPT_LORA_DIR() +{ + return ModelSpec::getDefaultModelSpec().useLoraPlugin().getModelPath(); +} + +std::string PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_DRAFT_TOKENS_DIR() +{ + return ModelSpec::getDefaultModelSpec().useDraftTokensExternalDecoding().getModelPath(); +} + +std::string PathUtil::FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().getModelPath(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_FILE() +{ + return ModelSpec::getDefaultModelSpec().getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_LONG_RESULT_FILE() +{ + return ModelSpec::getDefaultModelSpec().setInputFile("input_tokens_long.npy").getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_RESULT_FILE() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_FILE() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().getGenerationLogitsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_FILE() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().getContextLogitsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_CUM_LOG_PROBS_FILE() +{ + return ModelSpec::getDefaultModelSpec().getCumLogProbsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_CUM_LOG_PROBS_FILE() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().getCumLogProbsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_LOG_PROBS_FILE() +{ + return ModelSpec::getDefaultModelSpec().getLogProbsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_LOG_PROBS_FILE() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().getLogProbsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP1_FILE() +{ + return ModelSpec::getDefaultModelSpec().getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP4_PP1_FILE() +{ + return ModelSpec::getDefaultModelSpec().useTensorParallelism(4).getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP2_FILE() +{ + return ModelSpec::getDefaultModelSpec().useTensorParallelism(2).usePipelineParallelism(2).getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP4_FILE() +{ + return ModelSpec::getDefaultModelSpec().usePipelineParallelism(4).getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP2_FILE() +{ + return ModelSpec::getDefaultModelSpec().usePipelineParallelism(2).getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP1_FILE() +{ + return ModelSpec::getDefaultModelSpec().useTensorParallelism(2).getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_TP4_PP1_FILE() +{ + return ModelSpec::getDefaultModelSpec().useTensorParallelism(4).getContextLogitsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_TP4_PP1_FILE() +{ + return ModelSpec::getDefaultModelSpec().useTensorParallelism(4).getGenerationLogitsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_CUM_LOG_PROBS_TP4_PP1_FILE() +{ + return ModelSpec::getDefaultModelSpec().useTensorParallelism(4).getCumLogProbsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_LOG_PROBS_TP4_PP1_FILE() +{ + return ModelSpec::getDefaultModelSpec().useTensorParallelism(4).getLogProbsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_GATHER_CONTEXTFMHAFP32ACC_RESULT_FILE() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().enableContextFMHAFp32Acc().getResultsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXTFMHAFP32ACC_GENERATION_LOGITS_FILE() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().enableContextFMHAFp32Acc().getGenerationLogitsFile(); +} + +std::string PathUtil::FP16_PLUGIN_PACKED_PAGED_CONTEXTFMHAFP32ACC_CONTEXT_LOGITS_FILE() +{ + return ModelSpec::getDefaultModelSpec().gatherLogits().enableContextFMHAFp32Acc().getContextLogitsFile(); +} + +void TestData::loadLogProbs( + fs::path const& cumLogProbsFile, fs::path const& logProbsFile, tr::BufferManager const& manager) +{ + TLLM_CHECK_WITH_INFO( + cumLogProbsFile != "", "Testing return log probs, but missing the expected cum log probs results file."); + auto expectedCumLogProbsPtr + = std::shared_ptr(tr::utils::loadNpy(manager, cumLogProbsFile.string(), MemoryType::kCPU)); + + TLLM_CHECK_WITH_INFO( + logProbsFile != "", "Testing return log probs, but missing the expected log probs results file."); + auto expectedLogProbsPtr = std::shared_ptr(tr::utils::loadNpy(manager, logProbsFile.string(), MemoryType::kCPU)); + + for (SizeType32 inputIdx = 0; inputIdx < nbGivenInputs; ++inputIdx) + { + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto expectedCumLogProbsBatchSlice = std::shared_ptr(ITensor::slice(expectedCumLogProbsPtr, inputIdx, 1)); + expectedCumLogProbsBatchSlice->squeeze(0); // bs + expectedCumLogProbs[inputIdx] = expectedCumLogProbsBatchSlice; // shape: [beamWidth] + + auto expectedLogProbsBatchSlice = std::shared_ptr(ITensor::slice(expectedLogProbsPtr, inputIdx, 1)); + expectedLogProbsBatchSlice->squeeze(0); // bs + expectedLogProbs[inputIdx] = expectedLogProbsBatchSlice; // shape: [beamWidth, numOutputTokens] + } + } +} + +void TestData::loadContextLogits(fs::path const& contextLogitsFile, std::vector<SizeType32> const& givenInputLengths, + tr::BufferManager const& manager) +{ + TLLM_CHECK_WITH_INFO(contextLogitsFile != "", + "Testing with gather or replace logits, but missing the expected context logits results file."); + auto expectedContextLogitsPtr + = std::shared_ptr(tr::utils::loadNpy(manager, contextLogitsFile.string(), MemoryType::kCPU)); + + int promptOffset = 0; + for (SizeType32 bi = 0; bi < nbGivenInputs; ++bi) + { + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto expectedContextLogitBatchSlice + = std::shared_ptr(ITensor::slice(expectedContextLogitsPtr, promptOffset, givenInputLengths.at(bi))); + expectedContextLogits.at(bi) = expectedContextLogitBatchSlice; // shape: [prompt_length, vocab_size] + } + promptOffset += givenInputLengths.at(bi); + } +} + +void TestData::loadGenerationLogits(fs::path const& genLogitsFile, tr::BufferManager const& manager) +{ + TLLM_CHECK_WITH_INFO(genLogitsFile != "", + "Testing with gather or replace logits, but missing the expected generation logits results file."); + auto expectedGenerationLogitsPtr + = std::shared_ptr(tr::utils::loadNpy(manager, genLogitsFile.string(), MemoryType::kCPU)); + + for (SizeType32 bi = 0; bi < nbGivenInputs; ++bi) + { + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto expectedGenerationLogitBatchSlice + = std::shared_ptr(ITensor::slice(expectedGenerationLogitsPtr, bi, 1)); + expectedGenerationLogitBatchSlice->squeeze(0); // bs + expectedGenerationLogitBatchSlice->squeeze(0); // beam + expectedGenerationLogits.at(bi) = expectedGenerationLogitBatchSlice; // shape: [max_output_len, vocab_size] + } + } +} + +void TestData::makeDraft(SizeType32 maxDraftTokens, bool acceptDraftByLogits, fs::path const& genLogitsFile, + std::vector<SizeType32> const& givenInputLengths, tr::BufferManager const& manager) +{ + TLLM_CHECK(beamWidth == 1); + + ITensor::SharedPtr expectedGenerationLogitsPtr; + if (acceptDraftByLogits) + { + TLLM_CHECK_WITH_INFO( + genLogitsFile != "", "Testing Draft token, but missing the expected generation logits results file."); + expectedGenerationLogitsPtr + = std::shared_ptr(tr::utils::loadNpy(manager, genLogitsFile.string(), MemoryType::kCPU)); + } + + std::vector<SizeType32> draftLengths(givenInputLengths.size()); + // first draft length stays 0 + std::transform(givenInputLengths.begin() + 1, givenInputLengths.end(), draftLengths.begin() + 1, + [this, &maxDraftTokens](auto inputLength) + { return std::rand() % std::min((maxSeqLen - (inputLength + 1)), maxDraftTokens) + 1; }); + + auto* const expectedOutputData = tr::bufferCast<TokenIdType>(*expectedOutputIds); + for (SizeType32 bi = 0; bi < nbGivenInputs; ++bi) + { + SizeType32 constexpr beamIdx{0}; + auto const endId = endIds.at(bi); + auto const draftLen = draftLengths.at(bi); + auto acceptedLen = draftLen > 0 ? std::rand() % draftLen : 0; + + if (acceptDraftByLogits && draftLen > 0) + { + auto expectedLogitBatchSlice = std::shared_ptr(ITensor::slice(expectedGenerationLogitsPtr, bi, 1)); + expectedLogitBatchSlice->squeeze(0); // bs + expectedLogitBatchSlice->squeeze(0); // beam + auto expectedLogitBatchStepSlice = std::shared_ptr(ITensor::slice(expectedLogitBatchSlice, 1, draftLen)); + auto expectedLogitBatchStepView = ITensor::view(expectedLogitBatchStepSlice, + ITensor::makeShape({draftLen, 1, 1, expectedLogitBatchStepSlice->getShape().d[1]})); + draftLogits.at(bi) = manager.copyFrom(*expectedLogitBatchStepView, MemoryType::kCPU); + } + + for (SizeType32 si = 0; si < draftLen; ++si) + { + auto const draftIndex + = tc::flat_index3(bi, beamIdx, givenInputLengths.at(bi) + si + 1, beamWidth, maxSeqLen); + auto draftToken = expectedOutputData[draftIndex]; + if (draftToken == endId) + { + acceptedLen = std::min(acceptedLen, si); + } + if (si >= acceptedLen) + { + draftToken = -1; + if (acceptDraftByLogits) + { + auto vocabSizePadded = expectedGenerationLogitsPtr->getShape().d[3]; + auto* draftLogitsPtr = tr::bufferCast<float>(*draftLogits.at(bi)); + for (SizeType32 vi = 0; vi < vocabSizePadded; ++vi) + { + draftLogitsPtr[si * vocabSizePadded + vi] = 0.f; + } + } + } + draftTokens.at(bi).push_back(draftToken); + } + acceptedDraftTokensLengths.at(bi) = acceptedLen; + + auto const expectedLen = expectedOutputLengths.at(bi * beamWidth + beamIdx); + TLLM_CHECK(expectedLen > 0); + expectedOutputLengths[bi * beamWidth + beamIdx] + = draftLen > 0 ? std::min(expectedLen, (givenInputLengths.at(bi) + 1) + acceptedLen + 1) : expectedLen; + } +} + +template <typename T> +bool invokeCompareLogits(ITensor const& groundTruthLogits, ITensor const& outputLogits, float atol, float rtol) +{ + bool allMatch = true; + T const* const gtLogitsPtr = tr::bufferCast<T>(groundTruthLogits); + T const* const outputLogitsPtr = tr::bufferCast<T>(outputLogits); + + size_t outputSize = outputLogits.getSize(); + int errorNumber = 0; + + for (size_t i = 0; i < outputSize; i++) + { + if (!almostEqual(outputLogitsPtr[i], gtLogitsPtr[i], atol, rtol)) + { + TLLM_LOG_DEBUG("Mismatch value. Position of logits: %d, expected value: %f, output value: %f", i, + gtLogitsPtr[i], outputLogitsPtr[i]); + allMatch = false; + errorNumber++; + if (errorNumber == 10) + { + break; + } + } + } + return allMatch; +} + +bool compareLogits(ITensor const& groundTruthLogits, ITensor const& outputLogits, float atol, float rtol) +{ + EXPECT_EQ(groundTruthLogits.getDataType(), outputLogits.getDataType()); + switch (groundTruthLogits.getDataType()) + { + case nvinfer1::DataType::kFLOAT: return invokeCompareLogits<float>(groundTruthLogits, outputLogits, atol, rtol); + case nvinfer1::DataType::kHALF: return invokeCompareLogits<half>(groundTruthLogits, outputLogits, atol, rtol); + default: TLLM_THROW("Unsupported data type"); + } +} + +std::tuple<SizeType32, SizeType32> getRequestGivenInputIdxLength( + std::uint64_t requestId, SizeType32 nbGivenInputs, std::vector<SizeType32> const& givenInputLengths) +{ + auto const givenInputIdx = requestId % nbGivenInputs; + auto const inputLength = givenInputLengths.at(givenInputIdx); + return {givenInputIdx, inputLength}; +} + +std::tuple<std::vector<SizeType32>, SizeType32, SizeType32> getGivenInputLengths( + ITensor const& givenInput, SizeType32 padId) +{ + auto const& inputShape = givenInput.getShape(); + auto const nbGivenInputs = static_cast<SizeType32>(inputShape.d[0]); + auto const maxInputLength = static_cast<SizeType32>(inputShape.d[1]); + auto const* const givenInputData = tr::bufferCast<TokenIdType const>(givenInput); + + std::vector<SizeType32> givenInputLengths(nbGivenInputs); + for (SizeType32 i = 0; i < nbGivenInputs; ++i) + { + auto const* const seqBegin = givenInputData + i * maxInputLength; + auto const* const it = std::find(seqBegin, seqBegin + maxInputLength, padId); + givenInputLengths[i] = std::distance(seqBegin, it); + } + + return {givenInputLengths, nbGivenInputs, maxInputLength}; +} + +std::vector<executor::TokenIdType> createConsecutiveTokenSequence( + tr::SizeType32 length, tr::SizeType32 vocabSize, tr::TokenIdType firstTokenId) +{ + auto result = std::vector<executor::TokenIdType>(static_cast<size_t>(length), 0); + std::iota(result.begin(), result.end(), firstTokenId); + std::transform(result.begin(), result.end(), result.begin(), [&](auto const i) { return i % vocabSize; }); + return result; +} + +TestData TestData::loadTestData(BeamResult const& beamResults, ITensor const& givenInput, SizeType32 const maxBeamWidth, + tr::BufferManager& manager, executor::OutputConfig const& outConfig, ModelIds const& modelIds) +{ + auto const [givenInputLengths, nbGivenInputs, maxInputLength] = getGivenInputLengths(givenInput, modelIds.padId); + auto const& [beamWidth, resultsFile, contextLogitsFile, genLogitsFile, cumLogProbsFile, logProbsFile] = beamResults; + + TestData testData{nbGivenInputs, beamWidth}; + testData.expectedOutputIds = tr::utils::loadNpy(manager, resultsFile.string(), tr::MemoryType::kCPU); + + auto const& outputShape = testData.expectedOutputIds->getShape(); + EXPECT_EQ(outputShape.nbDims, 2); + EXPECT_EQ(nbGivenInputs * beamWidth, outputShape.d[0]); + testData.maxSeqLen = static_cast<SizeType32>(outputShape.d[1]); + EXPECT_LE(maxInputLength, testData.maxSeqLen); + EXPECT_LE(beamWidth, maxBeamWidth); + + auto const maxNewTokens = testData.maxSeqLen - maxInputLength; + + testData.endIds.insert(testData.endIds.end(), nbGivenInputs, modelIds.endId); + + if (outConfig.returnContextLogits && beamWidth == 1) + { + testData.loadContextLogits(contextLogitsFile, givenInputLengths, manager); + } + if (outConfig.returnGenerationLogits && beamWidth == 1) + { + testData.loadGenerationLogits(genLogitsFile, manager); + } + if (outConfig.returnLogProbs && beamWidth == 1) + { + testData.loadLogProbs(cumLogProbsFile, logProbsFile, manager); + } + + for (SizeType32 inputIdx = 0; inputIdx < nbGivenInputs; ++inputIdx) + { + for (SizeType32 beam = 0; beam < beamWidth; ++beam) + { + SizeType32 expectedLen = givenInputLengths[inputIdx] + maxNewTokens; + testData.expectedOutputLengths[inputIdx * beamWidth + beam] = expectedLen; + } + } + + return testData; +} + +void TestData::verifyOutput(std::unordered_map<SizeType32, std::vector<executor::BeamTokens>> const& resultTokens, + std::vector<SizeType32> const& givenInputLengths, bool streaming, bool excludeInputFromOutput, + FlakyTestInfo flakyTestInfo, bool isSpeculativeDecoding, SizeType32 reqBeamWidth, SizeType32 numReturnSequences, + bool isNonGreedySampling) +{ + for (auto const& [batchId, beamTokens] : resultTokens) + { + for (auto seqIdx = 0; seqIdx < numReturnSequences; seqIdx++) + { + auto const& tokens = beamTokens.at(seqIdx); + auto const inputLength = givenInputLengths.at(batchId); + SizeType32 const numReturnBeams = tokens.size(); + auto const* const expectedOutputData = tr::bufferCast<TokenIdType const>(*this->expectedOutputIds); + auto const expectedOutputLengths = this->expectedOutputLengths; + auto const endId = this->endIds[batchId]; + auto const maxSeqLen = this->maxSeqLen; + + for (SizeType32 beam = 0; beam < numReturnBeams; ++beam) + { + bool isFlaky = flakyTestInfo.batchIdBeams.count(std::make_pair(batchId, beam)); + if (isFlaky) + { + TLLM_LOG_WARNING("Disabling token comparison for batchId %d beam %d, test if flaky", batchId, beam); + } + + auto const expectInputOutputLength + = expectedOutputLengths[batchId * reqBeamWidth + beam]; // Ground truth output length + auto expectedOutputLength + = expectInputOutputLength - inputLength; // Number of new generated output tokens + + bool inputNotIncluded = (streaming || excludeInputFromOutput); + bool anyMismatch = false; + auto predictedTokens = tokens.at(beam); + // Remove the prompt + if (!inputNotIncluded) + { + predictedTokens.erase(predictedTokens.begin(), predictedTokens.begin() + inputLength); + } + + if (!isNonGreedySampling) + { + EXPECT_EQ(predictedTokens.size(), expectedOutputLength) + << "b: " << batchId << " seq: " << seqIdx << " beam: " << beam; + } + + auto numPredTokens = static_cast<SizeType32>(predictedTokens.size()); + + if (isSpeculativeDecoding) + { + // WAR to ensure bulk execution of spec decoding. + // We hope that no request in batch can finish 2x faster than any other request. + // For the cases when BS < 8, some predicted tokens are mismatched to reference data. + numPredTokens /= 2; + } + + for (auto i = 0; i < numPredTokens; ++i) + { + // Use the expected data for that beamWidth + auto const expectIndex = tc::flat_index3(batchId, beam, inputLength + i, reqBeamWidth, maxSeqLen); + auto const expectedToken = expectedOutputData[expectIndex]; + if (expectedToken == endId) + { + // TODO: can not find the error when (expectedToken == endId) && (predictedToken != endId) + break; + } + auto const predictedToken = predictedTokens.at(i); + if (!isFlaky && !isNonGreedySampling) + { + EXPECT_EQ(predictedToken, expectedToken) + << "b: " << batchId << " seq: " << seqIdx << " beam: " << beam << " i: " << i; + } + anyMismatch |= (predictedToken != expectedToken); + } + if (!isFlaky && !isNonGreedySampling) + { + EXPECT_FALSE(anyMismatch) << "b: " << batchId << " seq: " << seqIdx << " beam: " << beam; + } + else if (isNonGreedySampling) + { + EXPECT_TRUE(anyMismatch) << "b: " << batchId << " seq: " << seqIdx << " beam: " << beam; + } + } + } + } +} + +void TestData::verifyLogProbs(bool computeLogProbs, bool streaming, bool excludeInputFromOutput, SizeType32 inputLength, + SizeType32 beamWidth, executor::BeamTokens const& beamTokens, + std::optional<executor::VecLogProbs> const& cumLogProbs, + std::optional<std::vector<executor::VecLogProbs>> const& logProbs, SizeType32 batchId, FlakyTestInfo flakyTestInfo) +{ + auto expectedCumLogProbs = this->expectedCumLogProbs[batchId]; + auto expectedLogProbs = this->expectedLogProbs[batchId]; + auto const expectedOutputLengths = this->expectedOutputLengths; + auto const numReturnBeams = beamTokens.size(); + + if (computeLogProbs) + { + EXPECT_TRUE(cumLogProbs.has_value()) << "bid: " << batchId; + EXPECT_TRUE(logProbs.has_value()) << "bid: " << batchId; + EXPECT_EQ(cumLogProbs.value().size(), numReturnBeams) << "bid: " << batchId; + EXPECT_EQ(logProbs.value().size(), numReturnBeams) << "bid: " << batchId; + + bool removeInput = !excludeInputFromOutput && !streaming; + + for (SizeType32 beam = 0; beam < numReturnBeams; ++beam) + { + bool isFlaky = flakyTestInfo.batchIdBeams.count(std::make_pair(batchId, beam)); + if (isFlaky) + { + TLLM_LOG_WARNING("Disabling token comparison for batchId %d beam %d, test if flaky", batchId, beam); + } + + auto expectedOutputLength = expectedOutputLengths[batchId * beamWidth + beam]; + expectedOutputLength -= inputLength; + + auto numPredTokens = logProbs.value().at(beam).size(); + // Check shape + EXPECT_EQ(numPredTokens, beamTokens.at(beam).size() - (removeInput ? inputLength : 0)) + << "bid: " << batchId << " beam: " << beam; + + // If beamWidth == 1, compare log probs against python runtime + if (beamWidth == 1) + { + auto* const reqExpectedCumLogProbs = tr::bufferCast<float>(*expectedCumLogProbs); + // Only check cumLogProbs for the last generated token + if (numPredTokens == expectedOutputLength && !isFlaky) + { + EXPECT_TRUE(almostEqual(reqExpectedCumLogProbs[beam], cumLogProbs.value().at(beam), 2e-1, 5e-2)) + << "expectedCumLogProbs : " << reqExpectedCumLogProbs[beam] + << " cumlogProbs : " << cumLogProbs.value().at(beam); + } + + auto expectedLogProbsBeam = std::shared_ptr(tr::ITensor::slice(expectedLogProbs, beam, 1)); + expectedLogProbsBeam->squeeze(0); + auto* const reqExpectedLogProbs = tr::bufferCast<float>(*expectedLogProbsBeam); + for (auto i = 0; i < numPredTokens; ++i) + { + if (!isFlaky) + { + EXPECT_TRUE( + almostEqual(reqExpectedLogProbs[inputLength + i], logProbs.value()[beam][i], 5e-2, 5e-2)) + << "expectedLogProbs : " << reqExpectedLogProbs[inputLength + i] + << " logProbs : " << logProbs.value()[beam][i]; + } + } + } + } + } + else + { + EXPECT_FALSE(cumLogProbs.has_value()) << "bid: " << batchId; + EXPECT_FALSE(logProbs.has_value()) << "bid: " << batchId; + } +} + +void TestData::validateContextLogits(bool getContextLogits, SizeType32 inputLength, SizeType32 beamWidth, + std::optional<executor::Tensor> const& contextLogits, SizeType32 vocabSizePadded, SizeType32 batchId, float atol, + float rtol) +{ + if (getContextLogits) + { + EXPECT_TRUE(contextLogits.has_value()) << "bid: " << batchId; + EXPECT_EQ(contextLogits.value().getShape().size(), 2); + EXPECT_EQ(contextLogits.value().getShape()[0], inputLength); + EXPECT_EQ(contextLogits.value().getShape()[1], vocabSizePadded); + auto const expectedContextLogits = this->expectedContextLogits[batchId]; + + if (beamWidth == 1) + { + cudaDeviceSynchronize(); // Make sure the logits copy is complete. + EXPECT_TRUE(compareLogits( + *expectedContextLogits, *(executor::detail::toITensor(contextLogits.value())), atol, rtol)); + } + } + else + { + EXPECT_FALSE(contextLogits.has_value()) << "bid: " << batchId; + } +} + +void TestData::validateGenerationLogits(bool getGenLogits, bool isFinal, bool streaming, bool excludeInputFromOutput, + SizeType32 inputLength, SizeType32 maxOutputLen, SizeType32 beamWidth, executor::BeamTokens const& beamTokens, + std::optional<executor::Tensor> const& genLogits, SizeType32 vocabSizePadded, SizeType32 batchId, + bool const returnAllGeneratedTokens, float atol, float rtol) +{ + auto const numReturnBeams = beamTokens.size(); + + if (getGenLogits) + { + EXPECT_TRUE(genLogits.has_value()) << "bid: " << batchId; + EXPECT_EQ(genLogits.value().getShape().size(), 3); + + // Expected generation logits + auto const& expectedGenerationLogits + = this->expectedGenerationLogits[batchId]; // [maxOutputLen, vocabSizePadded] + // Output generation logits + // 1. non-streaming: [beamWidth, maxOutputLen, vocabSizePadded] + // 2. streaming: [maxOutputLen (or 1), beamWidth, vocabSizePadded] + auto const& outputGenerationLogits = executor::detail::toITensor(genLogits.value()); + + if (streaming) + { + EXPECT_EQ(genLogits.value().getShape()[1], numReturnBeams); + EXPECT_EQ(beamWidth, 1); // Only support streaming && beamWidth == 1 + + SizeType32 const beamIdx = 0; + bool removeInput = !excludeInputFromOutput && !streaming; + // If returnAllGeneratedTokens, will contain duplicate tokens + auto const& numPredTokens = beamTokens.at(beamIdx).size() - (removeInput ? inputLength : 0); + + SizeType32 numGeneratedToken = genLogits.value().getShape()[0]; + if (returnAllGeneratedTokens) + { + EXPECT_EQ(numGeneratedToken, numPredTokens); + } + else + { + EXPECT_EQ(numGeneratedToken, 1); + } + SizeType32 sliceOffset = returnAllGeneratedTokens ? 0 : numPredTokens - 1; + + auto const& expectedGenerationLogitsSlice + = std::shared_ptr(ITensor::slice(expectedGenerationLogits, sliceOffset, + numGeneratedToken)); // [numGeneratedToken, vocabSizePadded] + + cudaDeviceSynchronize(); // Make sure the logits copy is complete. + EXPECT_TRUE(compareLogits(*expectedGenerationLogitsSlice, *outputGenerationLogits, atol, rtol)); + } + else + { + // Non-streaming + EXPECT_EQ(genLogits.value().getShape()[0], numReturnBeams); + EXPECT_EQ(genLogits.value().getShape()[1], maxOutputLen); + + if (isFinal && beamWidth == 1) + { + cudaDeviceSynchronize(); // Make sure the logits copy is complete. + EXPECT_TRUE(compareLogits(*expectedGenerationLogits, *outputGenerationLogits, atol, rtol)); + } + } + EXPECT_EQ(genLogits.value().getShape()[2], vocabSizePadded); + } + else + { + EXPECT_FALSE(genLogits.has_value()) << "bid: " << batchId; + } +} + +} // namespace tensorrt_llm::testing diff --git a/cpp/tests/utils/common.h b/cpp/tests/utils/common.h new file mode 100644 index 000000000000..f7b73a9acea4 --- /dev/null +++ b/cpp/tests/utils/common.h @@ -0,0 +1,352 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#ifndef TOP_LEVEL_DIR +#error "Define TOP_LEVEL_DIR" +#endif + +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/runtime/bufferManager.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" + +#include <cmath> +#include <filesystem> +#include <random> +#include <set> +#include <string> +#include <utility> +#include <vector> + +namespace tensorrt_llm::testing +{ +namespace fs = std::filesystem; +namespace tr = tensorrt_llm::runtime; + +using tr::SizeType32; +using tr::TokenIdType; +using tr::ITensor; +using tr::MemoryType; + +auto const TEST_RESOURCE_PATH = fs::path{TOP_LEVEL_DIR} / "cpp/tests/resources"; + +auto const ENGINE_PATH = TEST_RESOURCE_PATH / "models/rt_engine"; +auto const GPT_MODEL_PATH = ENGINE_PATH / "gpt2"; +auto const LLAMA_MODEL_PATH = ENGINE_PATH / "Llama-3.2-1B"; +auto const MEDUSA_MODEL_PATH = ENGINE_PATH / "vicuna-7b-medusa"; +auto const CHATGLM_MODEL_PATH = ENGINE_PATH / "chatglm-6b"; +auto const CHATGLM2_MODEL_PATH = ENGINE_PATH / "chatglm2-6b"; +auto const CHATGLM3_MODEL_PATH = ENGINE_PATH / "chatglm3-6b"; +auto const GLM_MODEL_PATH = ENGINE_PATH / "glm-10b"; +auto const ENC_DEC_ENGINE_BASE = TEST_RESOURCE_PATH / "models/enc_dec/trt_engines"; + +auto const DATA_PATH = TEST_RESOURCE_PATH / "data"; +auto const GPT_DATA_PATH = DATA_PATH / "gpt2"; +auto const GPT_XGRAMMAR_TOKENIZER_INFO_PATH = GPT_DATA_PATH / "xgrammar_tokenizer_info.json"; +auto const LLAMA_DATA_PATH = DATA_PATH / "Llama-3.2-1B"; +auto const LLAMA_XGRAMMAR_TOKENIZER_INFO_PATH = LLAMA_DATA_PATH / "xgrammar_tokenizer_info.json"; +auto const MEDUSA_DATA_PATH = DATA_PATH / "vicuna-7b-medusa"; +auto const CHATGLM_DATA_PATH = DATA_PATH / "chatglm-6b"; +auto const CHATGLM2_DATA_PATH = DATA_PATH / "chatglm2-6b"; +auto const CHATGLM3_DATA_PATH = DATA_PATH / "chatglm3-6b"; +auto const GLM_DATA_PATH = DATA_PATH / "glm-10b"; +auto const ENC_DEC_DATA_BASE = DATA_PATH / "enc_dec"; + +auto constexpr T5_NAME = "t5-small"; +auto constexpr BART_NAME = "bart-large-cnn"; +auto constexpr LANGUAGE_ADAPTER_NAME = "language_adapter-enc_dec_language_adapter"; + +class PathUtil +{ +public: + static std::string EXECUTOR_WORKER_PATH() + { + return (std::filesystem::path{TOP_LEVEL_DIR} / "cpp/build/tensorrt_llm/executor_worker/executorWorker") + .string(); + } + + // model paths + static std::string FP16_GPT_ATTENTION_PACKED_DIR(); + static std::string FP16_GPT_ATTENTION_PACKED_PAGED_DIR(); + static std::string FP16_GPT_LORA_DIR(); + static std::string FP16_GPT_ATTENTION_PACKED_PAGED_DRAFT_TOKENS_DIR(); + static std::string FP16_GPT_ATTENTION_PACKED_PAGED_GATHER_DIR(); + static std::string FP16_PLUGIN_PACKED_PAGED_RESULT_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_LONG_RESULT_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_GATHER_RESULT_FILE(); + // logits + static std::string FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_CUM_LOG_PROBS_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_GATHER_CUM_LOG_PROBS_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_LOG_PROBS_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_GATHER_LOG_PROBS_FILE(); + // results + static std::string FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP1_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_RESULT_TP4_PP1_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP2_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP4_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_RESULT_TP1_PP2_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_RESULT_TP2_PP1_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_CONTEXT_LOGITS_TP4_PP1_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_GENERATION_LOGITS_TP4_PP1_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_CUM_LOG_PROBS_TP4_PP1_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_LOG_PROBS_TP4_PP1_FILE(); + // GptExecutorTest.GenerationLogitsEarlyStop requires to use context_fmha_fp32_acc flag in runtime for better + // accuracy + static std::string FP16_PLUGIN_PACKED_PAGED_GATHER_CONTEXTFMHAFP32ACC_RESULT_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_CONTEXTFMHAFP32ACC_GENERATION_LOGITS_FILE(); + static std::string FP16_PLUGIN_PACKED_PAGED_CONTEXTFMHAFP32ACC_CONTEXT_LOGITS_FILE(); +}; + +class ModelIds +{ +public: + ModelIds() = default; + + constexpr ModelIds(TokenIdType endId, TokenIdType padId) + : endId{endId} + , padId{padId} + { + } + + TokenIdType endId{}; + TokenIdType padId{}; +}; + +class BeamResult +{ +public: + explicit BeamResult(SizeType32 beamWidth) + : beamWidth{beamWidth} {}; + + BeamResult(SizeType32 beamWidth, fs::path resultsFile, fs::path contextLogitsFile, fs::path genLogitsFile, + fs::path cumLogProbsFile, fs::path logProbsFile) + : beamWidth{beamWidth} + , resultsFile{std::move(resultsFile)} + , contextLogitsFile{std::move(contextLogitsFile)} + , genLogitsFile{std::move(genLogitsFile)} + , cumLogProbsFile{std::move(cumLogProbsFile)} + , logProbsFile{std::move(logProbsFile)} {}; + + SizeType32 beamWidth; + fs::path resultsFile; + + fs::path contextLogitsFile; + fs::path genLogitsFile; + + fs::path cumLogProbsFile; + fs::path logProbsFile; +}; + +using BeamResults = std::vector<BeamResult>; + +struct FlakyTestInfo +{ + // Pair of batch ID + beam which are flaky + std::set<std::pair<SizeType32, SizeType32>> batchIdBeams; +}; + +class TestData +{ +public: + explicit TestData(SizeType32 nbGivenInputs, SizeType32 beamWidth) + : nbGivenInputs{nbGivenInputs} + , beamWidth{beamWidth} + { + expectedOutputLengths.resize(nbGivenInputs * beamWidth); + + draftTokens.resize(nbGivenInputs); + draftLogits.resize(nbGivenInputs); + acceptedDraftTokensLengths.resize(nbGivenInputs); + expectedGenerationLogits.resize(nbGivenInputs); + expectedContextLogits.resize(nbGivenInputs); + expectedCumLogProbs.resize(nbGivenInputs); + expectedLogProbs.resize(nbGivenInputs); + } + + void loadLogProbs(fs::path const& cumLogProbsFile, fs::path const& logProbsFile, tr::BufferManager const& manager); + + void loadContextLogits(fs::path const& contextLogitsFile, std::vector<SizeType32> const& givenInputLengths, + tr::BufferManager const& manager); + void loadGenerationLogits(fs::path const& genLogitsFile, tr::BufferManager const& manager); + + void makeDraft(SizeType32 maxDraftTokens, bool acceptDraftByLogits, fs::path const& genLogitsFile, + std::vector<SizeType32> const& givenInputLengths, tr::BufferManager const& manager); + + static TestData loadTestData(BeamResult const& beamResults, ITensor const& givenInput, SizeType32 maxBeamWidth, + tr::BufferManager& manager, executor::OutputConfig const& outConfig, ModelIds const& modelIds); + + void verifyOutput(std::unordered_map<SizeType32, std::vector<executor::BeamTokens>> const& resultTokens, + std::vector<SizeType32> const& givenInputLengths, bool streaming, bool excludeInputFromOutput, + FlakyTestInfo flakyTestInfo, bool isSpeculativeDecoding, SizeType32 reqBeamWidth, SizeType32 numReturnSequences, + bool isNonGreedySampling); + + void verifyLogProbs(bool computeLogProbs, bool streaming, bool excludeInputFromOutput, SizeType32 inputLength, + SizeType32 beamWidth, executor::BeamTokens const& beamTokens, + std::optional<executor::VecLogProbs> const& cumLogProbs, + std::optional<std::vector<executor::VecLogProbs>> const& logProbs, SizeType32 batchId, + FlakyTestInfo flakyTestInfo); + + void validateContextLogits(bool getContextLogits, SizeType32 inputLength, SizeType32 beamWidth, + std::optional<executor::Tensor> const& contextLogits, SizeType32 vocabSizePadded, SizeType32 batchId, + float atol = 1e-2, float rtol = 1e-3); + + void validateGenerationLogits(bool getGenLogits, bool isFinal, bool streaming, bool excludeInputFromOutput, + SizeType32 inputLength, SizeType32 maxOutputLen, SizeType32 beamWidth, executor::BeamTokens const& beamTokens, + std::optional<executor::Tensor> const& genLogits, SizeType32 vocabSizePadded, SizeType32 batchId, + bool returnAllGeneratedTokens, float atol = 1e-2, float rtol = 1e-3); + + SizeType32 nbGivenInputs{}; + SizeType32 beamWidth{}; + SizeType32 maxSeqLen{}; + ITensor::SharedPtr expectedOutputIds; + std::vector<SizeType32> expectedOutputLengths; + std::vector<TokenIdType> endIds; + std::vector<tensorrt_llm::executor::VecTokens> draftTokens; + std::vector<ITensor::SharedPtr> draftLogits; + std::vector<SizeType32> acceptedDraftTokensLengths; + std::vector<ITensor::SharedPtr> expectedGenerationLogits; + std::vector<ITensor::SharedPtr> expectedContextLogits; + std::vector<ITensor::SharedPtr> expectedCumLogProbs; + std::vector<ITensor::SharedPtr> expectedLogProbs; +}; + +inline bool almostEqual(float a, float b, float atol = 1e-2, float rtol = 1e-3) +{ + // Params: a = value to compare and b = reference + // This function follows implementation of numpy.isclose(), which checks + // abs(a - b) <= (atol + rtol * abs(b)). + // Note that the inequality above is asymmetric where b is considered as + // a reference value. To account into both absolute/relative errors, it + // uses absolute tolerance and relative tolerance at the same time. The + // default values of atol and rtol borrowed from numpy.isclose(). For the + // case of nan value, the result will be true. + if (std::isnan(a) && std::isnan(b)) + { + return true; + } + return fabs(a - b) <= (atol + rtol * fabs(b)); +} + +bool compareLogits(ITensor const& groundTruthLogits, ITensor const& outputLogits, float atol = 1e-2, float rtol = 1e-3); + +std::tuple<SizeType32, SizeType32> getRequestGivenInputIdxLength( + std::uint64_t requestId, SizeType32 nbGivenInputs, std::vector<SizeType32> const& givenInputLengths); + +std::tuple<std::vector<SizeType32>, SizeType32, SizeType32> getGivenInputLengths( + ITensor const& givenInput, SizeType32 padId); + +/// @brief Generates a vector of floating point values summing to 1, that can be used as logits. +/// +/// @tparam TEngine The type of the random engine. +/// @tparam TLogits The type of floating point values. +/// @param vocabSize The vocabulary size, i.e. the size of the vector. +/// @param engine A random engine. +/// @return std::vector<TLogits> A vector of floating point values, summing to 1. +template <typename TEngine, typename TLogits> +std::vector<TLogits> randomLogits(runtime::SizeType32 vocabSize, TEngine* engine) +{ + if constexpr (std::disjunction_v<std::is_floating_point<TLogits>, std::is_same<TLogits, half>>) + { + // This algorithm ensures the resulting values sum to 1 by: + // 1. Sampling in the interval 0..1 + // 2. Sorting the sampled values and adding a last value equal to 1 + // 3. Calculating the adjacent differences of the sorted values + // Since the values are sorted and the last value is 1, we get that all the differences are positive and must + // sum to 1. It can be proven recursively by seeing that the first value sums to itself, and the n-1 first + // values must sum to the value at n, minus the difference between the n-th and n-1-th values. + // It is also helpful to convince yourself of it with a quick drawing. + auto distribution = std::uniform_real_distribution<float>(0, 1); + std::vector<float> samples(vocabSize); + samples.back() = 1.0; + std::transform(samples.begin(), samples.end() - 1, samples.begin(), + [&](auto const /*i*/) { return distribution(*engine); }); + std::sort(samples.begin(), samples.end() - 1); + std::vector<float> result(vocabSize); + std::adjacent_difference(samples.begin(), samples.end(), result.begin()); + if constexpr (std::is_same_v<TLogits, float>) + { + return result; + } + + if constexpr (std::is_same_v<TLogits, half>) + { + std::vector<half> halfResults(vocabSize); + std::transform( + result.begin(), result.end(), halfResults.begin(), [&](auto const f) { return __float2half(f); }); + return halfResults; + } + } + TLLM_THROW("Unsupported logits type."); +} + +std::vector<tensorrt_llm::executor::TokenIdType> createConsecutiveTokenSequence( + tr::SizeType32 length, tr::SizeType32 vocabSize, tr::TokenIdType firstTokenId); + +/** + * GPU timer for recording the elapsed time across kernel(s) launched in GPU stream + */ +struct GpuTimer +{ + cudaStream_t _stream_id; + cudaEvent_t _start; + cudaEvent_t _stop; + + /// Construct`or + GpuTimer() + : _stream_id(0) + { + TLLM_CUDA_CHECK(cudaEventCreate(&_start)); + TLLM_CUDA_CHECK(cudaEventCreate(&_stop)); + } + + /// Destructor + ~GpuTimer() + { + TLLM_CUDA_CHECK(cudaEventDestroy(_start)); + TLLM_CUDA_CHECK(cudaEventDestroy(_stop)); + } + + /// Start the timer for a given stream (defaults to the default stream) + void start(cudaStream_t stream_id = 0) + { + _stream_id = stream_id; + TLLM_CUDA_CHECK(cudaEventRecord(_start, _stream_id)); + } + + /// Stop the timer + void stop() + { + TLLM_CUDA_CHECK(cudaEventRecord(_stop, _stream_id)); + } + + /// Return the elapsed time (in milliseconds) + float elapsed_millis() + { + float elapsed = 0.0; + TLLM_CUDA_CHECK(cudaEventSynchronize(_stop)); + TLLM_CUDA_CHECK(cudaEventElapsedTime(&elapsed, _start, _stop)); + return elapsed; + } +}; + +} // namespace tensorrt_llm::testing diff --git a/cpp/tests/utils/engines.cpp b/cpp/tests/utils/engines.cpp new file mode 100644 index 000000000000..28f0d9b15935 --- /dev/null +++ b/cpp/tests/utils/engines.cpp @@ -0,0 +1,97 @@ +#include "engines.h" + +#include "tensorrt_llm/batch_manager/transformerBuffers.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/tllmLogger.h" + +#include <NvInfer.h> +#include <algorithm> +#include <memory> + +nvinfer1::ITensor& tensorrt_llm::testing::utils::engines::details::addInputIds( + EngineBuildState& buildState, runtime::SizeType32 maxNumTokens) +{ + auto* input_ids = buildState.networkDefinition->addInput(batch_manager::RuntimeBuffers::kInputIdsTensorName, + nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({-1})); + buildState.tensors.push_back(input_ids); + buildState.profile->setDimensions(batch_manager::RuntimeBuffers::kInputIdsTensorName, + nvinfer1::OptProfileSelector::kMAX, runtime::ITensor::makeShape({maxNumTokens})); + buildState.profile->setDimensions(batch_manager::RuntimeBuffers::kInputIdsTensorName, + nvinfer1::OptProfileSelector::kOPT, runtime::ITensor::makeShape({maxNumTokens / 2})); + buildState.profile->setDimensions(batch_manager::RuntimeBuffers::kInputIdsTensorName, + nvinfer1::OptProfileSelector::kMIN, runtime::ITensor::makeShape({1})); + return *input_ids; +} + +nvinfer1::ITensor* tensorrt_llm::testing::utils::engines::details::addLastTokenIds( + EngineBuildState& buildState, runtime::SizeType32 maxBatchSize, runtime::SizeType32 maxBeamWidth) +{ + auto* last_token_ids + = buildState.networkDefinition->addInput(batch_manager::RuntimeBuffers::kLastTokenIdsTensorName, + nvinfer1::DataType::kINT32, runtime::ITensor::makeShape({-1})); + buildState.tensors.push_back(last_token_ids); + buildState.profile->setDimensions(batch_manager::RuntimeBuffers::kLastTokenIdsTensorName, + nvinfer1::OptProfileSelector::kMAX, runtime::ITensor::makeShape({maxBatchSize * maxBeamWidth})); + buildState.profile->setDimensions(batch_manager::RuntimeBuffers::kLastTokenIdsTensorName, + nvinfer1::OptProfileSelector::kOPT, runtime::ITensor::makeShape({maxBatchSize * maxBeamWidth / 2})); + buildState.profile->setDimensions(batch_manager::RuntimeBuffers::kLastTokenIdsTensorName, + nvinfer1::OptProfileSelector::kMIN, runtime::ITensor::makeShape({1})); + return last_token_ids; +} + +nvinfer1::ITensor& tensorrt_llm::testing::utils::engines::details::addKvCacheOffsets(EngineBuildState& buildState, + runtime::SizeType32 numPools, runtime::SizeType32 tokensPerBlock, runtime::SizeType32 maxBatchSize, + runtime::SizeType32 maxNumTokens, runtime::SizeType32 maxBeamWidth) +{ + auto* kvCacheOffsets = buildState.networkDefinition->addInput( + batch_manager::TransformerBuffers::kKvCacheBlockOffsetsTensorName, nvinfer1::DataType::kINT32, + runtime::ITensor::makeShape({numPools, -1, 2, -1})); // [numPools, maxBatch * maxBeamWidth, 2, maxBlocksPerSeq] + buildState.tensors.push_back(kvCacheOffsets); + auto const maxBlocksPerSeq = maxNumTokens / tokensPerBlock; + buildState.profile->setDimensions(batch_manager::TransformerBuffers::kKvCacheBlockOffsetsTensorName, + nvinfer1::OptProfileSelector::kMAX, + runtime::ITensor::makeShape({numPools, maxBatchSize * maxBeamWidth, 2, maxBlocksPerSeq})); + buildState.profile->setDimensions(batch_manager::TransformerBuffers::kKvCacheBlockOffsetsTensorName, + nvinfer1::OptProfileSelector::kOPT, + runtime::ITensor::makeShape({numPools, maxBatchSize * maxBeamWidth / 2, 2, maxBlocksPerSeq / 2})); + buildState.profile->setDimensions(batch_manager::TransformerBuffers::kKvCacheBlockOffsetsTensorName, + nvinfer1::OptProfileSelector::kMIN, runtime::ITensor::makeShape({numPools, 1, 2, 1})); + return *kvCacheOffsets; +} + +tensorrt_llm::testing::utils::engines::details::EngineBuildState +tensorrt_llm::testing::utils::engines::initializeEngineBuild(std::shared_ptr<runtime::TllmLogger> const& logger) +{ + auto* builder = nvinfer1::createInferBuilder(*logger); + auto* profile = builder->createOptimizationProfile(); + auto* network = builder->createNetworkV2( + 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kSTRONGLY_TYPED)); + nvinfer1::IBuilderConfig* config = builder->createBuilderConfig(); + return {builder, network, profile, config}; +} + +nvinfer1::ITensor& tensorrt_llm::testing::utils::engines::details::addSingleOutputLayer( + tensorrt_llm::testing::utils::engines::details::EngineBuildState& buildState, nvinfer1::ILayer* layer) +{ + buildState.layers.push_back(layer); + auto* output = layer->getOutput(0); + buildState.tensors.push_back(output); + TLLM_LOG_INFO("Adding layer %s with output shape %s.", layer->getName(), + tensorrt_llm::runtime::ITensor::toString(output->getDimensions()).c_str()); + + return *output; +} + +tensorrt_llm::common::OptionalRef<nvinfer1::ITensor> tensorrt_llm::testing::utils::engines::details::getTensorByName( + tensorrt_llm::testing::utils::engines::details::EngineBuildState& buildState, std::string_view name) +{ + auto result = std::find_if(buildState.tensors.begin(), buildState.tensors.end(), + [name](auto const tensor) { return tensor->getName() == name; }); + if (result == buildState.tensors.end()) + { + return tensorrt_llm::common::OptionalRef<nvinfer1::ITensor>{}; + } + return **result; +} diff --git a/cpp/tests/utils/engines.h b/cpp/tests/utils/engines.h new file mode 100644 index 000000000000..54b53e5c6097 --- /dev/null +++ b/cpp/tests/utils/engines.h @@ -0,0 +1,356 @@ +#ifndef CA1B91B5_DF64_4CF8_948F_5AFF243A2555 +#define CA1B91B5_DF64_4CF8_948F_5AFF243A2555 + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/optionalRef.h" +#include "tensorrt_llm/runtime/common.h" +#include "tensorrt_llm/runtime/iBuffer.h" +#include "tensorrt_llm/runtime/iTensor.h" +#include "tensorrt_llm/runtime/tllmLogger.h" +#include <NvInfer.h> +#include <NvInferRuntime.h> +#include <algorithm> +#include <cstddef> +#include <memory> +#include <tensorrt_llm/batch_manager/runtimeBuffers.h> +#include <utility> +#include <vector> + +namespace tensorrt_llm::testing::utils::engines +{ + +namespace details +{ + +struct EngineBuildResource +{ + EngineBuildResource() = default; + virtual ~EngineBuildResource() = default; + EngineBuildResource(EngineBuildResource const& vector) = default; + EngineBuildResource& operator=(EngineBuildResource const& vector) = default; + EngineBuildResource(EngineBuildResource&& vector) noexcept = default; + EngineBuildResource& operator=(EngineBuildResource&& vector) noexcept = default; +}; + +template <typename TValue> +struct Vector : public EngineBuildResource +{ + explicit Vector(std::vector<TValue> values) + : values(std::move(values)){}; + Vector(Vector const& vector) = default; + Vector& operator=(Vector const& vector) = default; + Vector(Vector&& vector) noexcept = default; + Vector& operator=(Vector&& vector) noexcept = default; + ~Vector() override = default; + std::vector<TValue> values; +}; + +template <typename TValue, size_t Size> +struct Array : public EngineBuildResource +{ + explicit Array(std::array<TValue, Size> values) + : values(std::move(values)){}; + Array(Array const& vector) = default; + Array& operator=(Array const& vector) = default; + Array(Array&& vector) noexcept = default; + Array& operator=(Array&& vector) noexcept = default; + ~Array() override = default; + std::array<TValue, Size> values; +}; + +struct EngineBuildState +{ + EngineBuildState(nvinfer1::IBuilder* builder, nvinfer1::INetworkDefinition* networkDefinition, + nvinfer1::IOptimizationProfile* profile, nvinfer1::IBuilderConfig* builderConfig) + : builder(builder) + , networkDefinition(networkDefinition) + , profile(profile) + , builderConfig(builderConfig){}; + EngineBuildState(EngineBuildState const& vector) = delete; + EngineBuildState& operator=(EngineBuildState const& vector) = delete; + EngineBuildState(EngineBuildState&& vector) noexcept = default; + EngineBuildState& operator=(EngineBuildState&& vector) noexcept = default; + std::unique_ptr<nvinfer1::IBuilder> builder; + std::unique_ptr<nvinfer1::INetworkDefinition> networkDefinition; + nvinfer1::IOptimizationProfile* profile; + + // While building the engine, one might need some data for weights and such. Turns out, TensorRT does not keep a + // copy of those, so if you create them as temporaries and pass them to the TRT APIs, you will get UB. So we need + // some place where we can keep those things. + std::unique_ptr<nvinfer1::IBuilderConfig> builderConfig; + std::vector<std::unique_ptr<EngineBuildResource>> resources; + std::vector<nvinfer1::ITensor*> tensors; + std::vector<nvinfer1::ILayer*> layers; + + ~EngineBuildState() + { + // Builder needs to be deleteds last. + networkDefinition.reset(); + builderConfig.reset(); + builder.reset(); + } +}; + +common::OptionalRef<nvinfer1::ITensor> getTensorByName(EngineBuildState& buildState, std::string_view name); + +nvinfer1::ITensor& addSingleOutputLayer(EngineBuildState& buildState, nvinfer1::ILayer* layer); + +template <typename TResource> +TResource& addResource(EngineBuildState& buildState, TResource resource) +{ + return *dynamic_cast<TResource*>( + buildState.resources.emplace_back(std::make_unique<TResource>(std::move(resource))).get()); +} + +template <typename TValue> +Vector<TValue>& addSingleConstantVectorResource(EngineBuildState& buildState, TValue value, std::size_t length) +{ + std::vector<TValue> weights(length); + std::fill(weights.begin(), weights.end(), value); + return addResource(buildState, Vector<TValue>{weights}); +} + +template <typename TValue> +Vector<TValue>& addConstantVectorResource(EngineBuildState& buildState, std::vector<TValue> values) +{ + return addResource(buildState, Vector<TValue>{values}); +} + +template <typename TValue> +Array<TValue, 1>& addConstantScalarResource(EngineBuildState& buildState, TValue value) +{ + return addResource(buildState, Array<TValue, 1>{{value}}); +} + +nvinfer1::ITensor& addInputIds(EngineBuildState& buildState, runtime::SizeType32 maxNumTokens); +nvinfer1::ITensor* addLastTokenIds( + EngineBuildState& buildState, runtime::SizeType32 maxBatchSize, runtime::SizeType32 maxBeamWidth); +nvinfer1::ITensor& addKvCacheOffsets(EngineBuildState& buildState, runtime::SizeType32 numPools, + runtime::SizeType32 tokensPerBlock, runtime::SizeType32 maxBatchSize, runtime::SizeType32 maxNumTokens, + runtime::SizeType32 maxBeamWidth); + +template <typename TValue> +nvinfer1::ITensor& addSingleConstantVector(EngineBuildState& buildState, TValue value, runtime::SizeType32 length) +{ + auto& resourceWeights = addSingleConstantVectorResource(buildState, value, length); + auto const trtDatatype = runtime::TRTDataType<TValue>::value; + auto* layer = buildState.networkDefinition->addConstant(runtime::ITensor::makeShape({length}), + {trtDatatype, resourceWeights.values.data(), static_cast<runtime::ITensor::DimType64>(length)}); + return addSingleOutputLayer(buildState, layer); +} + +template <typename TValue> +nvinfer1::ITensor& addSingleConstantTensor(EngineBuildState& buildState, TValue value, runtime::SizeType32 length) +{ + auto& resourceWeights = addSingleConstantVectorResource(buildState, value, length); + auto const trtDatatype = runtime::TRTDataType<TValue>::value; + auto* layer = buildState.networkDefinition->addConstant(runtime::ITensor::makeShape({1, length}), + {trtDatatype, resourceWeights.values.data(), static_cast<runtime::ITensor::DimType64>(length)}); + return addSingleOutputLayer(buildState, layer); +} + +template <typename TValue> +nvinfer1::ITensor& addConstantVector(EngineBuildState& buildState, std::vector<TValue> values) +{ + auto& resourceWeights = addConstantVectorResource(buildState, values); + auto const trtDatatype = runtime::TRTDataType<TValue>::value; + auto const length = static_cast<runtime::ITensor::DimType64>(values.size()); + auto* layer = buildState.networkDefinition->addConstant( + runtime::ITensor::makeShape({length}), {trtDatatype, resourceWeights.values.data(), length}); + return addSingleOutputLayer(buildState, layer); +} + +template <typename TValue> +nvinfer1::ITensor& addConstantTensor( + EngineBuildState& buildState, std::vector<TValue> values, runtime::ITensor::Shape shape) +{ + auto& resourceWeights = addConstantVectorResource(buildState, values); + auto const trtDatatype = runtime::TRTDataType<TValue>::value; + auto const count = runtime::ITensor::volume(shape); + auto* layer = buildState.networkDefinition->addConstant(shape, {trtDatatype, resourceWeights.values.data(), count}); + return addSingleOutputLayer(buildState, layer); +} + +template <typename TValue> +nvinfer1::ITensor& addSingleConstantTensor(EngineBuildState& buildState, TValue value, runtime::ITensor::Shape shape) +{ + auto const count = runtime::ITensor::volume(shape); + auto& resourceWeights = addSingleConstantVectorResource(buildState, value, count); + auto const trtDatatype = runtime::TRTDataType<TValue>::value; + auto* layer = buildState.networkDefinition->addConstant(shape, {trtDatatype, resourceWeights.values.data(), count}); + return addSingleOutputLayer(buildState, layer); +} + +template <typename TValue> +nvinfer1::ITensor& addConstantScalar(EngineBuildState& buildState, TValue value) +{ + auto& resourceWeights = addConstantScalarResource<TValue>(buildState, value); + auto const trtDatatype = runtime::TRTDataType<TValue>::value; + auto* layer = buildState.networkDefinition->addConstant( + runtime::ITensor::makeShape({}), {trtDatatype, resourceWeights.values.data(), 1}); + return addSingleOutputLayer(buildState, layer); +} + +template <typename TValue> +nvinfer1::ITensor& oneHotEncode( + EngineBuildState& buildState, nvinfer1::ITensor& inputIds, runtime::SizeType32 vocabSize) +{ + auto const trtValueType = runtime::TRTDataType<TValue>::value; + auto& oneHotValues = addConstantVector<TValue>(buildState, {0, 1}); + auto& oneHotDepth = addConstantScalar(buildState, vocabSize); + auto* oneHotLayer = buildState.networkDefinition->addOneHot(inputIds, oneHotValues, oneHotDepth, 0); + return addSingleOutputLayer(buildState, oneHotLayer); +} +} // namespace details + +struct TrivialDecoderParameters +{ + TrivialDecoderParameters(runtime::SizeType32 vocabSize, runtime::SizeType32 maxBatchSize, + runtime::SizeType32 maxNumTokens, runtime::SizeType32 tokensPerBlock, runtime::SizeType32 maxBeamWidth, + bool gatherContextLogits) + : vocabSize(vocabSize) + , maxBatchSize(maxBatchSize) + , maxNumTokens(maxNumTokens) + , tokensPerBlock(tokensPerBlock) + , maxBeamWidth(maxBeamWidth) + , gatherContextLogits(gatherContextLogits){}; + runtime::SizeType32 vocabSize; + runtime::SizeType32 maxBatchSize; + runtime::SizeType32 maxNumTokens; + runtime::SizeType32 tokensPerBlock; + runtime::SizeType32 maxBeamWidth; + bool gatherContextLogits; +}; + +details::EngineBuildState initializeEngineBuild(std::shared_ptr<runtime::TllmLogger> const& logger); + +template <typename TLogits> +std::unique_ptr<nvinfer1::IHostMemory> createTrivialDecoder( + TrivialDecoderParameters parameters, std::shared_ptr<runtime::TllmLogger> const& logger) +{ + auto const trtLogitsType = runtime::TRTDataType<TLogits>::value; + auto buildState = initializeEngineBuild(logger); + auto* builder = buildState.builder.get(); + auto* profile = buildState.profile; + auto* network = buildState.networkDefinition.get(); + auto& inputIds = details::addInputIds(buildState, parameters.maxNumTokens); + auto& kvCacheOffsets = details::addKvCacheOffsets(buildState, 1, parameters.tokensPerBlock, parameters.maxBatchSize, + parameters.maxNumTokens, parameters.maxBeamWidth); + + auto& oneHotLayerOutput = details::oneHotEncode<TLogits>(buildState, inputIds, parameters.vocabSize); + oneHotLayerOutput.setName(batch_manager::RuntimeBuffers::kLogitsTensorName); + network->markOutput(oneHotLayerOutput); + + buildState.builderConfig->addOptimizationProfile(profile); + buildState.builderConfig->setProfilingVerbosity(nvinfer1::ProfilingVerbosity::kDETAILED); + auto* engine = builder->buildSerializedNetwork(*network, *buildState.builderConfig); + return std::unique_ptr<nvinfer1::IHostMemory>(engine); +} + +template <typename TLogits> +struct ConstantTrivialDecoderParameters +{ + ConstantTrivialDecoderParameters(TrivialDecoderParameters trivialDecoderParameters, std::vector<TLogits> logits) + : trivialDecoderParameters(trivialDecoderParameters) + , logits(logits) + { + auto const sizeTypeVocabSize = static_cast<std::size_t>(trivialDecoderParameters.vocabSize); + auto const logitsSize = logits.size(); + TLLM_CHECK_WITH_INFO(static_cast<std::size_t>(trivialDecoderParameters.vocabSize) == logits.size(), + "The size of the constant logits (%lu) has to be equal to the vocabulary size (%lu).", logitsSize, + sizeTypeVocabSize); + }; + + TrivialDecoderParameters trivialDecoderParameters; + std::vector<TLogits> logits; +}; + +template <typename TLogits> +details::EngineBuildState createConstantTrivialDecoderBase( + ConstantTrivialDecoderParameters<TLogits> parameters, std::shared_ptr<runtime::TllmLogger> const& logger) +{ + auto const trtLogitsType = runtime::TRTDataType<TLogits>::value; + auto buildState = initializeEngineBuild(logger); + auto* builder = buildState.builder.get(); + auto* profile = buildState.profile; + auto* network = buildState.networkDefinition.get(); + auto& inputIds = details::addInputIds(buildState, parameters.trivialDecoderParameters.maxNumTokens); + nvinfer1::ITensor* lastTokenIds = nullptr; + if (!parameters.trivialDecoderParameters.gatherContextLogits) + { + lastTokenIds = details::addLastTokenIds(buildState, parameters.trivialDecoderParameters.maxBatchSize, + parameters.trivialDecoderParameters.maxBeamWidth); + } + auto& kvCacheOffsets = details::addKvCacheOffsets(buildState, 1, parameters.trivialDecoderParameters.tokensPerBlock, + parameters.trivialDecoderParameters.maxBatchSize, parameters.trivialDecoderParameters.maxNumTokens, + parameters.trivialDecoderParameters.maxBeamWidth); + + auto const vocabSize = static_cast<runtime::ITensor::DimType64>(parameters.logits.size()); + + auto& constantLogitsPerToken = details::addConstantTensor<TLogits>( + buildState, parameters.logits, runtime::ITensor::makeShape({vocabSize, 1})); + auto& oneHotLayerOutput + = details::oneHotEncode<TLogits>(buildState, inputIds, parameters.trivialDecoderParameters.vocabSize); + auto& ones = details::addSingleConstantTensor<TLogits>(buildState, 1, runtime::ITensor::makeShape({1, vocabSize})); + auto* intermediateLayer1 = network->addMatrixMultiply( + ones, nvinfer1::MatrixOperation::kNONE, oneHotLayerOutput, nvinfer1::MatrixOperation::kNONE); + auto* intermediateLayer1Output = intermediateLayer1->getOutput(0); + + nvinfer1::ITensor* gatherLayerOutput = nullptr; + if (!parameters.trivialDecoderParameters.gatherContextLogits) + { + auto& one = details::addSingleConstantTensor<int32_t>(buildState, 1, runtime::ITensor::makeShape({1})); + auto* lastTokenIdsMinus1Layer + = network->addElementWise(*lastTokenIds, one, nvinfer1::ElementWiseOperation::kSUB); + auto* gatherLayer = network->addGather(*intermediateLayer1Output, *lastTokenIdsMinus1Layer->getOutput(0), 1); + gatherLayerOutput = gatherLayer->getOutput(0); + } + else + { + gatherLayerOutput = intermediateLayer1Output; + } + + auto* constLogitsLayer = network->addMatrixMultiply(*gatherLayerOutput, nvinfer1::MatrixOperation::kTRANSPOSE, + constantLogitsPerToken, nvinfer1::MatrixOperation::kTRANSPOSE); + auto* outputLogits = constLogitsLayer->getOutput(0); + network->markOutput(*outputLogits); + outputLogits->setName(batch_manager::RuntimeBuffers::kLogitsTensorName); + buildState.tensors.push_back(outputLogits); + return buildState; +} + +template <typename TLogits> +std::unique_ptr<nvinfer1::IHostMemory> createConstantTrivialDecoder( + ConstantTrivialDecoderParameters<TLogits> parameters, std::shared_ptr<runtime::TllmLogger> const& logger) +{ + auto buildState = createConstantTrivialDecoderBase<TLogits>(parameters, logger); + buildState.builderConfig->addOptimizationProfile(buildState.profile); + buildState.builderConfig->setProfilingVerbosity(nvinfer1::ProfilingVerbosity::kDETAILED); + auto* engine = buildState.builder->buildSerializedNetwork(*buildState.networkDefinition, *buildState.builderConfig); + return std::unique_ptr<nvinfer1::IHostMemory>(engine); +} + +template <typename TLogits> +std::unique_ptr<nvinfer1::IHostMemory> createConstantTrivialDecoderWithTopKLogits( + ConstantTrivialDecoderParameters<TLogits> parameters, runtime::SizeType32 numTopLogits, std::string_view outputName, + std::shared_ptr<runtime::TllmLogger> const& logger) +{ + auto buildState = createConstantTrivialDecoderBase<TLogits>(parameters, logger); + auto logits = details::getTensorByName(buildState, batch_manager::RuntimeBuffers::kLogitsTensorName); + TLLM_CHECK_WITH_INFO(static_cast<bool>(logits), + "You can only add topk logits on top of a network which contains a tensor named %s", + batch_manager::RuntimeBuffers::kLogitsTensorName); + auto* topKLayer = buildState.networkDefinition->addTopK( + logits.value(), nvinfer1::TopKOperation::kMAX, numTopLogits, 1UL << 1UL); + auto* topKLayerOutput = topKLayer->getOutput(0); + topKLayerOutput->setName(outputName.data()); + buildState.networkDefinition->markOutput(*topKLayerOutput); + auto* profile = buildState.profile; + buildState.builderConfig->addOptimizationProfile(profile); + buildState.builderConfig->setProfilingVerbosity(nvinfer1::ProfilingVerbosity::kDETAILED); + auto* engine = buildState.builder->buildSerializedNetwork(*buildState.networkDefinition, *buildState.builderConfig); + return std::unique_ptr<nvinfer1::IHostMemory>(engine); +} +} // namespace tensorrt_llm::testing::utils::engines + +#endif /* CA1B91B5_DF64_4CF8_948F_5AFF243A2555 */ diff --git a/cpp/tests/utils/executorUtils.cpp b/cpp/tests/utils/executorUtils.cpp new file mode 100644 index 000000000000..6dbd6223ef19 --- /dev/null +++ b/cpp/tests/utils/executorUtils.cpp @@ -0,0 +1,55 @@ +#include "executorUtils.h" +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include <exception> +#include <future> + +std::unordered_map<tensorrt_llm::batch_manager::RequestIdType, std::vector<tensorrt_llm::executor::Response>> +tensorrt_llm::testing::runThroughRequests(executor::Executor& executor, std::vector<executor::Request> const& requests, + std::chrono::duration<float, std::milli> timeout) +{ + std::unordered_map<batch_manager::RequestIdType, std::vector<executor::Response>> accumulatedResponses; + auto responseReadFuture = std::async(std::launch::async, + [&]() -> std::optional<std::exception> + { + auto remainingRequests = requests.size(); + try + { + while (remainingRequests > 0) + { + auto const responses = executor.awaitResponses(); + for (auto const& response : responses) + { + auto const requestId = response.getRequestId(); + if (response.hasError()) + { + TLLM_LOG_ERROR("Error response received for request: %lu", requestId); + TLLM_THROW(response.getErrorMsg()); + } + auto const isFinal = response.hasError() || response.getResult().isFinal; + accumulatedResponses[requestId].emplace_back(response); + if (isFinal) + { + TLLM_LOG_DEBUG("Final response received for request: %lu", requestId); + --remainingRequests; + } + } + } + + return std::nullopt; + } + catch (std::exception const& e) + { + TLLM_LOG_EXCEPTION(e); + return e; + } + }); + auto const requestIds = executor.enqueueRequests(requests); + responseReadFuture.wait_for(timeout); + auto const readResult = responseReadFuture.get(); + if (readResult.has_value()) + { + throw std::exception(readResult.value()); + } + return accumulatedResponses; +} diff --git a/cpp/tests/utils/executorUtils.h b/cpp/tests/utils/executorUtils.h new file mode 100644 index 000000000000..97f154ca275d --- /dev/null +++ b/cpp/tests/utils/executorUtils.h @@ -0,0 +1,18 @@ +#ifndef A073F2DA_315E_434B_B811_D420F0A59DF3 +#define A073F2DA_315E_434B_B811_D420F0A59DF3 + +#include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/executor/executor.h" +#include <ratio> +#include <unordered_map> + +namespace tensorrt_llm::testing +{ + +std::unordered_map<batch_manager::RequestIdType, std::vector<executor::Response>> runThroughRequests( + executor::Executor& executor, std::vector<executor::Request> const& requests, + std::chrono::duration<float, std::milli> timeout); + +} // namespace tensorrt_llm::testing + +#endif /* A073F2DA_315E_434B_B811_D420F0A59DF3 */ diff --git a/docker/Dockerfile.multi b/docker/Dockerfile.multi index 6ac69ef9c4fb..c4c91d9fad71 100644 --- a/docker/Dockerfile.multi +++ b/docker/Dockerfile.multi @@ -1,8 +1,8 @@ # Multi-stage Dockerfile ARG BASE_IMAGE=nvcr.io/nvidia/pytorch ARG TRITON_IMAGE=nvcr.io/nvidia/tritonserver -ARG BASE_TAG=26.05-py3 -ARG TRITON_BASE_TAG=26.05-py3 +ARG BASE_TAG=26.04-py3 +ARG TRITON_BASE_TAG=26.04-py3 ARG DEVEL_IMAGE=devel FROM ${BASE_IMAGE}:${BASE_TAG} AS base @@ -31,6 +31,7 @@ FROM base AS devel # NB: PyTorch requires this to be < 1.0 ENV PYTORCH_ALLOC_CONF="garbage_collection_threshold:0.99999" +ARG TRT_VER ARG CUDA_VER ARG CUDNN_VER ARG NCCL_VER @@ -44,11 +45,11 @@ RUN --mount=type=bind,source=docker/common,target=/opt/docker/common \ echo "Using Python version: ${PYTHON_VERSION}" && \ GITHUB_MIRROR=${GITHUB_MIRROR} \ PYTHON_VERSION=${PYTHON_VERSION} \ - CUDA_VER=${CUDA_VER} CUDNN_VER=${CUDNN_VER} \ + TRT_VER=${TRT_VER} CUDA_VER=${CUDA_VER} CUDNN_VER=${CUDNN_VER} \ NCCL_VER=${NCCL_VER} CUBLAS_VER=${CUBLAS_VER} \ TORCH_INSTALL_TYPE=${TORCH_INSTALL_TYPE} \ bash /opt/docker/common/install.sh --base --cmake --ccache --cuda_toolkit \ - --cuda_libs --polygraphy --mpi4py --pytorch + --tensorrt --polygraphy --mpi4py --pytorch --opencv # Install constraints after install.sh so cleanup() doesn't delete the file mid-RUN COPY constraints.txt /tmp/constraints.txt @@ -118,7 +119,7 @@ COPY .gitmodules setup.py requirements.txt requirements-dev.txt constraints.txt ENV CCACHE_DIR=/root/.cache/ccache # Build the TRT-LLM wheel ARG GITHUB_MIRROR="" -ARG BUILD_WHEEL_ARGS="--clean" +ARG BUILD_WHEEL_ARGS="--clean --benchmarks" ARG BUILD_WHEEL_SCRIPT="scripts/build_wheel.py" RUN --mount=type=cache,target=/root/.cache/pip --mount=type=cache,target=${CCACHE_DIR} \ GITHUB_MIRROR=$GITHUB_MIRROR python3 ${BUILD_WHEEL_SCRIPT} ${BUILD_WHEEL_ARGS} @@ -127,9 +128,7 @@ FROM ${DEVEL_IMAGE} AS release WORKDIR /app/tensorrt_llm RUN --mount=type=cache,target=/root/.cache/pip --mount=type=bind,from=wheel,source=/src/tensorrt_llm/build,target=/tmp/wheel \ - TRTLLM_WHEEL=$(find /tmp/wheel -maxdepth 1 -name 'tensorrt_llm*.whl' -print -quit) && \ - test -n "${TRTLLM_WHEEL}" && \ - pip install "${TRTLLM_WHEEL}[mx]" + pip install /tmp/wheel/tensorrt_llm*.whl RUN --mount=type=bind,source=README.md,target=/mnt/ctx/README.md \ --mount=type=bind,source=docs,target=/mnt/ctx/docs \ @@ -137,6 +136,7 @@ RUN --mount=type=bind,source=README.md,target=/mnt/ctx/README.md \ --mount=type=bind,source=examples,target=/mnt/ctx/examples \ --mount=type=bind,from=wheel,source=/src/tensorrt_llm/build,target=/mnt/wheel \ --mount=type=bind,from=wheel,source=/src/tensorrt_llm/benchmarks,target=/mnt/benchmarks \ + --mount=type=bind,from=wheel,source=/src/tensorrt_llm/cpp/build/benchmarks,target=/mnt/cpp_benchmarks \ # Copy build context files cp /mnt/ctx/README.md ./ && \ cp -r /mnt/ctx/docs ./docs && \ @@ -146,12 +146,24 @@ RUN --mount=type=bind,source=README.md,target=/mnt/ctx/README.md \ # Copy wheel stage outputs cp /mnt/wheel/tensorrt_llm*.whl ./ && \ cp -r /mnt/benchmarks ./benchmarks && \ - # Create a symlink to installed package libraries + mkdir -p benchmarks/cpp && \ + cp /mnt/cpp_benchmarks/bertBenchmark \ + /mnt/cpp_benchmarks/gptManagerBenchmark \ + /mnt/cpp_benchmarks/disaggServerBenchmark \ + benchmarks/cpp/ && \ + rm -v \ + benchmarks/cpp/bertBenchmark.cpp \ + benchmarks/cpp/gptManagerBenchmark.cpp \ + benchmarks/cpp/disaggServerBenchmark.cpp \ + benchmarks/cpp/CMakeLists.txt && \ + # Create symlinks to installed package binaries and libraries + ln -sv $(python3 -c 'import site; print(f"{site.getsitepackages()[0]}/tensorrt_llm/bin")') bin && \ + test -f bin/executorWorker && \ ln -sv $(python3 -c 'import site; print(f"{site.getsitepackages()[0]}/tensorrt_llm/libs")') lib && \ - test -f lib/libtensorrt_llm.so && \ + test -f lib/libnvinfer_plugin_tensorrt_llm.so && \ echo "/app/tensorrt_llm/lib" > /etc/ld.so.conf.d/tensorrt_llm.conf && \ ldconfig && \ - ! ( ldd -v lib/libth_common.so | grep tensorrt_llm | grep -q "not found" ) && \ + ! ( ldd -v bin/executorWorker | grep tensorrt_llm | grep -q "not found" ) && \ # Clean up caches and CVE workarounds rm -rf /root/.cache/uv/archive-v0 && \ # WAR against https://github.com/advisories/GHSA-58pv-8j8x-9vj2 @@ -169,9 +181,21 @@ ENV TRT_LLM_GIT_COMMIT=${GIT_COMMIT} \ RUN --mount=type=bind,source=scripts/generate_container_oss_attribution.sh,target=/mnt/gen_attribution.sh \ bash /mnt/gen_attribution.sh "release" "${TRT_LLM_VER}" "${TARGETARCH}" +FROM wheel AS tritonbuild + +WORKDIR /src/tensorrt_llm +RUN pip install /src/tensorrt_llm/build/tensorrt_llm*.whl +COPY ./triton_backend/ ./triton_backend/ +ARG TRITON_BASE_TAG +RUN bash ./triton_backend/inflight_batcher_llm/scripts/build.sh -s "r${TRITON_BASE_TAG%-py3}" + + FROM release AS tritonrelease WORKDIR /app/tensorrt_llm COPY ./triton_backend/all_models ./triton_backend/all_models COPY ./triton_backend/scripts ./triton_backend/scripts COPY ./triton_backend/tools ./triton_backend/tools +COPY ./triton_backend/inflight_batcher_llm/scripts ./triton_backend/inflight_batcher_llm/scripts +COPY ./triton_backend/inflight_batcher_llm/client ./triton_backend/inflight_batcher_llm/client +COPY --from=tritonbuild /opt/tritonserver/backends/tensorrtllm /opt/tritonserver/backends/tensorrtllm diff --git a/docker/Makefile b/docker/Makefile index 7c8568ffe54c..0ca78eef80a0 100644 --- a/docker/Makefile +++ b/docker/Makefile @@ -47,6 +47,7 @@ CUDA_VERSION ?= CUDNN_VERSION ?= NCCL_VERSION ?= CUBLAS_VERSION ?= +TRT_VERSION ?= GIT_COMMIT ?= $(shell git rev-parse HEAD) TRT_LLM_VERSION ?= $(shell grep '^__version__' ../tensorrt_llm/version.py | grep -o '=.*' | tr -d '= "') GITHUB_MIRROR ?= @@ -98,6 +99,7 @@ base_pull: $(if $(CUDNN_VERSION), --build-arg CUDNN_VER="$(CUDNN_VERSION)") \ $(if $(NCCL_VERSION), --build-arg NCCL_VER="$(NCCL_VERSION)") \ $(if $(CUBLAS_VERSION), --build-arg CUBLAS_VER="$(CUBLAS_VERSION)") \ + $(if $(TRT_VERSION), --build-arg TRT_VER="$(TRT_VERSION)") \ $(if $(TRT_LLM_VERSION), --build-arg TRT_LLM_VER="$(TRT_LLM_VERSION)") \ $(if $(DEVEL_IMAGE), --build-arg DEVEL_IMAGE="$(DEVEL_IMAGE)") \ $(if $(GIT_COMMIT), --build-arg GIT_COMMIT="$(GIT_COMMIT)") \ diff --git a/docker/common/install.sh b/docker/common/install.sh index 5f0dc91c8ed8..0de962c9afcb 100755 --- a/docker/common/install.sh +++ b/docker/common/install.sh @@ -11,10 +11,11 @@ base=0 cmake=0 ccache=0 cuda_toolkit=0 -cuda_libs=0 +tensorrt=0 polygraphy=0 mpi4py=0 pytorch=0 +opencv=0 while [[ $# -gt 0 ]]; do case $1 in @@ -34,8 +35,8 @@ while [[ $# -gt 0 ]]; do cuda_toolkit=1 shift 1 ;; - --cuda_libs) - cuda_libs=1 + --tensorrt) + tensorrt=1 shift 1 ;; --polygraphy) @@ -50,15 +51,20 @@ while [[ $# -gt 0 ]]; do pytorch=1 shift 1 ;; + --opencv) + opencv=1 + shift 1 + ;; --all) base=1 cmake=1 ccache=1 cuda_toolkit=1 - cuda_libs=1 + tensorrt=1 polygraphy=1 mpi4py=1 pytorch=1 + opencv=1 shift 1 ;; *) @@ -92,9 +98,10 @@ if [ $cuda_toolkit -eq 1 ]; then bash $SCRIPT_DIR/install_cuda_toolkit.sh fi -if [ $cuda_libs -eq 1 ]; then - echo "Installing CUDA libraries (cuDNN/NCCL/cuBLAS)..." - bash $SCRIPT_DIR/install_cuda_libs.sh \ +if [ $tensorrt -eq 1 ]; then + echo "Installing TensorRT..." + bash $SCRIPT_DIR/install_tensorrt.sh \ + --TRT_VER=${TRT_VER} \ --CUDA_VER=${CUDA_VER} \ --CUDNN_VER=${CUDNN_VER} \ --NCCL_VER=${NCCL_VER} \ @@ -115,3 +122,10 @@ if [ $pytorch -eq 1 ]; then echo "Installing PyTorch..." bash $SCRIPT_DIR/install_pytorch.sh $TORCH_INSTALL_TYPE fi + +if [ $opencv -eq 1 ]; then + echo "Installing OpenCV..." + bash -c "pip3 uninstall -y opencv" + rm -rf /usr/local/lib/python3*/dist-packages/cv2/ + bash -c "pip3 install opencv-python-headless --force-reinstall --no-deps --no-cache-dir" +fi diff --git a/docker/common/install_cuda_libs.sh b/docker/common/install_cuda_libs.sh deleted file mode 100644 index e5a8566c62d3..000000000000 --- a/docker/common/install_cuda_libs.sh +++ /dev/null @@ -1,180 +0,0 @@ -#!/bin/bash - -set -ex - -# Align with the pre-installed cuDNN / cuBLAS / NCCL versions from -# https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-26-05.html#rel-26-05 -CUDA_VER="13.2" # 13.2.1 -# Keep the installation for cuDNN if users want to install PyTorch with source codes. -# PyTorch 2.x can compile with cuDNN v9. -CUDNN_VER="9.22.0.52-1" -NCCL_VER="2.30.4-1+cuda13.2" -CUBLAS_VER="13.4.1.2-1" -# Align with the pre-installed CUDA / NVCC / NVRTC versions from -# https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html -NVRTC_VER="13.2.78-1" -CUDA_RUNTIME="13.2.75-1" -CUDA_DRIVER_VERSION="595.58.03-1.el8" - -for i in "$@"; do - case $i in - --CUDA_VER=?*) CUDA_VER="${i#*=}";; - --CUDNN_VER=?*) CUDNN_VER="${i#*=}";; - --NCCL_VER=?*) NCCL_VER="${i#*=}";; - --CUBLAS_VER=?*) CUBLAS_VER="${i#*=}";; - *) ;; - esac - shift -done - -NVCC_VERSION_OUTPUT=$(nvcc --version) -if [[ $(echo $NVCC_VERSION_OUTPUT | grep -oP "\d+\.\d+" | head -n 1) != ${CUDA_VER} ]]; then - echo "The version of pre-installed CUDA is not equal to ${CUDA_VER}." -fi - -install_ubuntu_requirements() { - apt-get update && apt-get install -y --no-install-recommends gnupg2 curl ca-certificates - ARCH=$(uname -m) - if [ "$ARCH" = "amd64" ];then ARCH="x86_64";fi - if [ "$ARCH" = "aarch64" ];then ARCH="sbsa";fi - - # this file exists in cuda base image, and has conflicts with cuda-keyring with the following error, so we need to remove it first: - # E: Conflicting values set for option Signed-By regarding - # source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/ /: /usr/share/keyrings/cuda-archive-keyring.gpg != - rm -f /etc/apt/sources.list.d/cuda.list - - curl -fsSLO https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/${ARCH}/cuda-keyring_1.1-1_all.deb - dpkg -i cuda-keyring_1.1-1_all.deb - rm cuda-keyring_1.1-1_all.deb - - apt-get update - if [[ $(apt list --installed | grep libcudnn9) ]]; then - apt-get remove --purge -y libcudnn9* - fi - if [[ $(apt list --installed | grep libnccl) ]]; then - apt-get remove --purge -y --allow-change-held-packages libnccl* - fi - if [[ $(apt list --installed | grep libcublas) ]]; then - apt-get remove --purge -y --allow-change-held-packages libcublas* - fi - if [[ $(apt list --installed | grep cuda-nvrtc-dev) ]]; then - apt-get remove --purge -y --allow-change-held-packages cuda-nvrtc-dev* - fi - - CUDA_MAJOR_VER=$(echo $CUDA_VER | cut -d. -f1) - CUBLAS_MAJOR_VER=$(echo $CUBLAS_VER | cut -d. -f1) - NVRTC_CUDA_VERSION=$(echo $CUDA_VER | sed 's/\./-/g') - - apt-get install -y --no-install-recommends \ - libcudnn9-cuda-13=${CUDNN_VER} \ - libcudnn9-dev-cuda-13=${CUDNN_VER} \ - libcudnn9-headers-cuda-13=${CUDNN_VER} \ - libnccl2=${NCCL_VER} \ - libnccl-dev=${NCCL_VER} \ - libcublas${CUBLAS_MAJOR_VER}-cuda-${CUDA_MAJOR_VER}=${CUBLAS_VER} \ - libcublas${CUBLAS_MAJOR_VER}-dev-cuda-${CUDA_MAJOR_VER}=${CUBLAS_VER} \ - cuda-nvrtc-dev-${NVRTC_CUDA_VERSION}=${NVRTC_VER} - - apt-get clean - rm -rf /var/lib/apt/lists/* - - # cublas >= 13.4.1.2 installs headers to /usr/include/libcublas/<major>/ - # instead of /usr/local/cuda/include/. Symlink them back for build compatibility. - CUBLAS_HDR_DIR="/usr/include/libcublas/${CUBLAS_MAJOR_VER}" - if [ -d "${CUBLAS_HDR_DIR}" ]; then - for hdr in "${CUBLAS_HDR_DIR}"/*.h; do - ln -sf "${hdr}" "/usr/local/cuda/include/$(basename ${hdr})" - done - fi - - # cublas >= 13.4.1.2 installs .so files to /usr/lib/<arch>-linux-gnu/libcublas/<major>/ - # instead of /usr/local/cuda/lib64/. Symlink them so LD_LIBRARY_PATH=/usr/local/cuda/lib64 finds them. - ARCH=$(uname -m) - CUBLAS_LIB_DIR="/usr/lib/${ARCH}-linux-gnu/libcublas/${CUBLAS_MAJOR_VER}" - if [ -d "${CUBLAS_LIB_DIR}" ]; then - for lib in "${CUBLAS_LIB_DIR}"/libcublas*.so*; do - [ -e "${lib}" ] || continue - target="/usr/local/cuda/lib64/$(basename ${lib})" - [ -e "${target}" ] || ln -sf "${lib}" "${target}" - done - fi - ldconfig -} - -install_rockylinux_requirements() { - CUBLAS_CUDA_VERSION=$(echo $CUDA_VER | sed 's/\./-/g') - CUDA_MAJOR_VER=$(echo $CUDA_VER | cut -d. -f1) - CUBLAS_MAJOR_VER=$(echo $CUBLAS_VER | cut -d. -f1) - - ARCH=$(uname -m) - if [ "$ARCH" = "x86_64" ];then ARCH1="x86_64" && ARCH2="x64" && ARCH3=$ARCH1;fi - if [ "$ARCH" = "aarch64" ];then ARCH1="aarch64" && ARCH2="aarch64sbsa" && ARCH3="sbsa";fi - - # Download and install packages - for pkg in \ - "libnccl-${NCCL_VER}.${ARCH1}" \ - "libnccl-devel-${NCCL_VER}.${ARCH1}" \ - "cuda-compat-${CUBLAS_CUDA_VERSION}-${CUDA_DRIVER_VERSION}.${ARCH1}" \ - "cuda-toolkit-${CUBLAS_CUDA_VERSION}-config-common-${CUDA_RUNTIME}.noarch" \ - "cuda-toolkit-13-config-common-${CUDA_RUNTIME}.noarch" \ - "cuda-toolkit-config-common-${CUDA_RUNTIME}.noarch" \ - "libcublas${CUBLAS_MAJOR_VER}-cuda-${CUDA_MAJOR_VER}-${CUBLAS_VER}.${ARCH1}" \ - "libcublas${CUBLAS_MAJOR_VER}-devel-cuda-${CUDA_MAJOR_VER}-${CUBLAS_VER}.${ARCH1}"; do - wget --retry-connrefused --timeout=180 --tries=10 --continue "https://developer.download.nvidia.com/compute/cuda/repos/rhel8/${ARCH3}/${pkg}.rpm" - done - - # Remove old packages - dnf remove -y "libnccl*" - - # Install new packages - dnf -y install \ - libnccl-${NCCL_VER}.${ARCH1}.rpm \ - libnccl-devel-${NCCL_VER}.${ARCH1}.rpm \ - cuda-compat-${CUBLAS_CUDA_VERSION}-${CUDA_DRIVER_VERSION}.${ARCH1}.rpm \ - cuda-toolkit-${CUBLAS_CUDA_VERSION}-config-common-${CUDA_RUNTIME}.noarch.rpm \ - cuda-toolkit-13-config-common-${CUDA_RUNTIME}.noarch.rpm \ - cuda-toolkit-config-common-${CUDA_RUNTIME}.noarch.rpm \ - libcublas${CUBLAS_MAJOR_VER}-cuda-${CUDA_MAJOR_VER}-${CUBLAS_VER}.${ARCH1}.rpm \ - libcublas${CUBLAS_MAJOR_VER}-devel-cuda-${CUDA_MAJOR_VER}-${CUBLAS_VER}.${ARCH1}.rpm - - # Clean up - rm -f *.rpm - dnf clean all - nvcc --version - - # cublas >= 13.4.1.2 installs headers to /usr/include/libcublas/<major>/ - # instead of /usr/local/cuda/include/. Symlink them back for build compatibility. - CUBLAS_HDR_DIR="/usr/include/libcublas/${CUBLAS_MAJOR_VER}" - if [ -d "${CUBLAS_HDR_DIR}" ]; then - for hdr in "${CUBLAS_HDR_DIR}"/*.h; do - ln -sf "${hdr}" "/usr/local/cuda/include/$(basename ${hdr})" - done - fi - - # cublas >= 13.4.1.2 installs .so files to /usr/lib64/libcublas/<major>/ - # instead of /usr/local/cuda/lib64/. Symlink them so LD_LIBRARY_PATH=/usr/local/cuda/lib64 finds them. - CUBLAS_LIB_DIR="/usr/lib64/libcublas/${CUBLAS_MAJOR_VER}" - if [ -d "${CUBLAS_LIB_DIR}" ]; then - for lib in "${CUBLAS_LIB_DIR}"/libcublas*.so*; do - [ -e "${lib}" ] || continue - target="/usr/local/cuda/lib64/$(basename ${lib})" - [ -e "${target}" ] || ln -sf "${lib}" "${target}" - done - fi - ldconfig -} - -# Install base packages depending on the base OS -ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"') -case "$ID" in - ubuntu) - install_ubuntu_requirements - ;; - rocky) - install_rockylinux_requirements - ;; - *) - echo "Unable to determine OS..." - exit 1 - ;; -esac diff --git a/docker/common/install_pytorch.sh b/docker/common/install_pytorch.sh index 81dcede0aebf..4ec997fdbeca 100644 --- a/docker/common/install_pytorch.sh +++ b/docker/common/install_pytorch.sh @@ -4,7 +4,7 @@ set -ex # Use latest stable version from https://pypi.org/project/torch/#history # and closest to the version specified in -# https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-26-05.html#rel-26-05 +# https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-26-04.html#rel-26-04 TORCH_VERSION="2.11.0" SYSTEM_ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"') diff --git a/docker/common/install_tensorrt.sh b/docker/common/install_tensorrt.sh new file mode 100644 index 000000000000..66bf24deb2b4 --- /dev/null +++ b/docker/common/install_tensorrt.sh @@ -0,0 +1,169 @@ +#!/bin/bash + +set -ex + +TRT_VER="10.16.1.11" +# Align with the pre-installed cuDNN / cuBLAS / NCCL versions from +# https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel-26-04.html#rel-26-04 +CUDA_VER="13.2" # 13.2.1 +# Keep the installation for cuDNN if users want to install PyTorch with source codes. +# PyTorch 2.x can compile with cuDNN v9. +CUDNN_VER="9.21.0.82-1" +NCCL_VER="2.29.7-1+cuda13.2" +CUBLAS_VER="13.4.0.1-1" +# Align with the pre-installed CUDA / NVCC / NVRTC versions from +# https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html +NVRTC_VER="13.2.78-1" +CUDA_RUNTIME="13.2.75-1" +CUDA_DRIVER_VERSION="595.58.03-1.el8" + +for i in "$@"; do + case $i in + --TRT_VER=?*) TRT_VER="${i#*=}";; + --CUDA_VER=?*) CUDA_VER="${i#*=}";; + --CUDNN_VER=?*) CUDNN_VER="${i#*=}";; + --NCCL_VER=?*) NCCL_VER="${i#*=}";; + --CUBLAS_VER=?*) CUBLAS_VER="${i#*=}";; + *) ;; + esac + shift +done + +NVCC_VERSION_OUTPUT=$(nvcc --version) +if [[ $(echo $NVCC_VERSION_OUTPUT | grep -oP "\d+\.\d+" | head -n 1) != ${CUDA_VER} ]]; then + echo "The version of pre-installed CUDA is not equal to ${CUDA_VER}." +fi + +install_ubuntu_requirements() { + apt-get update && apt-get install -y --no-install-recommends gnupg2 curl ca-certificates + ARCH=$(uname -m) + if [ "$ARCH" = "amd64" ];then ARCH="x86_64";fi + if [ "$ARCH" = "aarch64" ];then ARCH="sbsa";fi + + # this file exists in cuda base image, and has conflicts with cuda-keyring with the following error, so we need to remove it first: + # E: Conflicting values set for option Signed-By regarding + # source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/ /: /usr/share/keyrings/cuda-archive-keyring.gpg != + rm -f /etc/apt/sources.list.d/cuda.list + + curl -fsSLO https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/${ARCH}/cuda-keyring_1.1-1_all.deb + dpkg -i cuda-keyring_1.1-1_all.deb + rm cuda-keyring_1.1-1_all.deb + + apt-get update + if [[ $(apt list --installed | grep libcudnn9) ]]; then + apt-get remove --purge -y libcudnn9* + fi + if [[ $(apt list --installed | grep libnccl) ]]; then + apt-get remove --purge -y --allow-change-held-packages libnccl* + fi + if [[ $(apt list --installed | grep libcublas) ]]; then + apt-get remove --purge -y --allow-change-held-packages libcublas* + fi + if [[ $(apt list --installed | grep cuda-nvrtc-dev) ]]; then + apt-get remove --purge -y --allow-change-held-packages cuda-nvrtc-dev* + fi + + CUBLAS_CUDA_VERSION=$(echo $CUDA_VER | sed 's/\./-/g') + NVRTC_CUDA_VERSION=$(echo $CUDA_VER | sed 's/\./-/g') + + apt-get install -y --no-install-recommends \ + libcudnn9-cuda-13=${CUDNN_VER} \ + libcudnn9-dev-cuda-13=${CUDNN_VER} \ + libcudnn9-headers-cuda-13=${CUDNN_VER} \ + libnccl2=${NCCL_VER} \ + libnccl-dev=${NCCL_VER} \ + libcublas-${CUBLAS_CUDA_VERSION}=${CUBLAS_VER} \ + libcublas-dev-${CUBLAS_CUDA_VERSION}=${CUBLAS_VER} \ + cuda-nvrtc-dev-${NVRTC_CUDA_VERSION}=${NVRTC_VER} + + apt-get clean + rm -rf /var/lib/apt/lists/* +} + +install_rockylinux_requirements() { + CUBLAS_CUDA_VERSION=$(echo $CUDA_VER | sed 's/\./-/g') + + ARCH=$(uname -m) + if [ "$ARCH" = "x86_64" ];then ARCH1="x86_64" && ARCH2="x64" && ARCH3=$ARCH1;fi + if [ "$ARCH" = "aarch64" ];then ARCH1="aarch64" && ARCH2="aarch64sbsa" && ARCH3="sbsa";fi + + # Download and install packages + for pkg in \ + "libnccl-${NCCL_VER}.${ARCH1}" \ + "libnccl-devel-${NCCL_VER}.${ARCH1}" \ + "cuda-compat-${CUBLAS_CUDA_VERSION}-${CUDA_DRIVER_VERSION}.${ARCH1}" \ + "cuda-toolkit-${CUBLAS_CUDA_VERSION}-config-common-${CUDA_RUNTIME}.noarch" \ + "cuda-toolkit-13-config-common-${CUDA_RUNTIME}.noarch" \ + "cuda-toolkit-config-common-${CUDA_RUNTIME}.noarch" \ + "libcublas-${CUBLAS_CUDA_VERSION}-${CUBLAS_VER}.${ARCH1}" \ + "libcublas-devel-${CUBLAS_CUDA_VERSION}-${CUBLAS_VER}.${ARCH1}"; do + wget --retry-connrefused --timeout=180 --tries=10 --continue "https://developer.download.nvidia.com/compute/cuda/repos/rhel8/${ARCH3}/${pkg}.rpm" + done + + # Remove old packages + dnf remove -y "libnccl*" + + # Install new packages + dnf -y install \ + libnccl-${NCCL_VER}.${ARCH1}.rpm \ + libnccl-devel-${NCCL_VER}.${ARCH1}.rpm \ + cuda-compat-${CUBLAS_CUDA_VERSION}-${CUDA_DRIVER_VERSION}.${ARCH1}.rpm \ + cuda-toolkit-${CUBLAS_CUDA_VERSION}-config-common-${CUDA_RUNTIME}.noarch.rpm \ + cuda-toolkit-13-config-common-${CUDA_RUNTIME}.noarch.rpm \ + cuda-toolkit-config-common-${CUDA_RUNTIME}.noarch.rpm \ + libcublas-${CUBLAS_CUDA_VERSION}-${CUBLAS_VER}.${ARCH1}.rpm \ + libcublas-devel-${CUBLAS_CUDA_VERSION}-${CUBLAS_VER}.${ARCH1}.rpm + + # Clean up + rm -f *.rpm + dnf clean all + nvcc --version +} + +install_tensorrt() { + PY_VERSION=$(python3 -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))') + PARSED_PY_VERSION=$(echo "${PY_VERSION//./}") + + TRT_CUDA_VERSION=${CUDA_VER} + TRT_VER_SHORT=$(echo $TRT_VER | cut -d. -f1-3) + + if [ -z "$RELEASE_URL_TRT" ];then + ARCH=${TRT_TARGETARCH} + if [ -z "$ARCH" ];then ARCH=$(uname -m);fi + if [ "$ARCH" = "arm64" ];then ARCH="aarch64";fi + if [ "$ARCH" = "amd64" ];then ARCH="x86_64";fi + RELEASE_URL_TRT="https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/${TRT_VER_SHORT}/tars/TensorRT-${TRT_VER}.Linux.${ARCH}-gnu.cuda-${TRT_CUDA_VERSION}.tar.gz" + fi + + wget --retry-connrefused --timeout=180 --tries=10 --continue ${RELEASE_URL_TRT} -O /tmp/TensorRT.tar + tar -xf /tmp/TensorRT.tar -C /usr/local/ + mv /usr/local/TensorRT-${TRT_VER} /usr/local/tensorrt + pip3 install --no-cache-dir /usr/local/tensorrt/python/tensorrt-*-cp${PARSED_PY_VERSION}-*.whl + rm -rf /tmp/TensorRT.tar + echo 'export LD_LIBRARY_PATH=/usr/local/tensorrt/lib:$LD_LIBRARY_PATH' >> "${ENV}" + + rm -f /usr/local/tensorrt/lib/libnvinfer_vc_plugin_static.a \ + /usr/local/tensorrt/lib/libnvinfer_plugin_static.a \ + /usr/local/tensorrt/lib/libnvinfer_static.a \ + /usr/local/tensorrt/lib/libnvinfer_dispatch_static.a \ + /usr/local/tensorrt/lib/libnvinfer_lean_static.a \ + /usr/local/tensorrt/lib/libnvonnxparser_static.a \ + /usr/local/tensorrt/lib/libnvinfer_builder_resource_win.so.* +} + +# Install base packages depending on the base OS +ID=$(grep -oP '(?<=^ID=).+' /etc/os-release | tr -d '"') +case "$ID" in + ubuntu) + install_ubuntu_requirements + install_tensorrt + ;; + rocky) + install_rockylinux_requirements + install_tensorrt + ;; + *) + echo "Unable to determine OS..." + exit 1 + ;; +esac diff --git a/docs/source/_ext/llmapi_config_telemetry.py b/docs/source/_ext/llmapi_config_telemetry.py index 0b0bf9e63427..1c237bb6889c 100644 --- a/docs/source/_ext/llmapi_config_telemetry.py +++ b/docs/source/_ext/llmapi_config_telemetry.py @@ -41,9 +41,6 @@ and any field may opt out with `telemetry=False`. Every captured field is listed below; the runtime can capture nothing absent from this list. -If the manifest check fails, run `python3 scripts/generate_llm_args_golden_manifest.py`, then commit -`tensorrt_llm/usage/llm_args_golden_manifest.json`; new fields require telemetry/privacy CODEOWNER approval. - ## LLM API Configuration Fields A field can still be absent from a specific payload when its parent config is @@ -76,16 +73,19 @@ def _table(rows: list[dict]) -> str: def generate_telemetry_reference(repo_root: Path | str, output_path: Path | str) -> None: repo_root = Path(repo_root) golden = json.loads((repo_root / _GOLDEN_REL).read_text()) - rows = golden.get("TorchLlmArgs", []) - content = [ - _REFERENCE_PREAMBLE, - "### `TorchLlmArgs`", - "", - f"{len(rows)} captured fields.", - "", - _table(rows), - "", - ] + content = [_REFERENCE_PREAMBLE] + for args_class in ("TorchLlmArgs", "TrtLlmArgs"): + rows = golden.get(args_class, []) + content.extend( + [ + f"### `{args_class}`", + "", + f"{len(rows)} captured fields.", + "", + _table(rows), + "", + ] + ) output = Path(output_path) output.parent.mkdir(parents=True, exist_ok=True) output.write_text("\n".join(content)) diff --git a/docs/source/_static/config_db.json b/docs/source/_static/config_db.json index 8af081380d9c..7671934a6911 100644 --- a/docs/source/_static/config_db.json +++ b/docs/source/_static/config_db.json @@ -179,30 +179,6 @@ "model_display_name": "MiniMax-M3 (MXFP8)", "model_url": "https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8", "scenario": "Max Throughput" - }, - { - "command": "trtllm-serve deepseek-ai/DeepSeek-V4-Pro --config ${TRTLLM_DIR}/examples/configs/curated/deepseek-v4-pro-latency.yaml", - "config_filename": "deepseek-v4-pro-latency.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/curated/deepseek-v4-pro-latency.yaml", - "config_path": "examples/configs/curated/deepseek-v4-pro-latency.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/curated/deepseek-v4-pro-latency.yaml", - "gpu_compatibility": "B200", - "model": "deepseek-ai/DeepSeek-V4-Pro", - "model_display_name": "deepseek-ai/DeepSeek-V4-Pro", - "model_url": "", - "scenario": "Min Latency" - }, - { - "command": "trtllm-serve deepseek-ai/DeepSeek-V4-Pro --config ${TRTLLM_DIR}/examples/configs/curated/deepseek-v4-pro-throughput.yaml", - "config_filename": "deepseek-v4-pro-throughput.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/curated/deepseek-v4-pro-throughput.yaml", - "config_path": "examples/configs/curated/deepseek-v4-pro-throughput.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/curated/deepseek-v4-pro-throughput.yaml", - "gpu_compatibility": "B200", - "model": "deepseek-ai/DeepSeek-V4-Pro", - "model_display_name": "deepseek-ai/DeepSeek-V4-Pro", - "model_url": "", - "scenario": "Max Throughput" } ], "entries": [ @@ -1345,82 +1321,6 @@ "osl": 1024, "performance_profile": "Max Throughput" }, - { - "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc4.yaml", - "concurrency": 4, - "config_filename": "1k1k_tp4_conc4.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc4.yaml", - "config_path": "examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc4.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc4.yaml", - "gpu": "B200_NVL", - "gpu_display": "4xB200_NVL", - "isl": 1024, - "model": "nvidia/DeepSeek-R1-0528-FP4-v2", - "model_display_name": "DeepSeek-R1 (NVFP4)", - "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", - "num_gpus": 4, - "osl": 1024, - "performance_profile": "Min Latency", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" - }, - { - "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc8.yaml", - "concurrency": 8, - "config_filename": "1k1k_tp4_conc8.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc8.yaml", - "config_path": "examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc8.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc8.yaml", - "gpu": "B200_NVL", - "gpu_display": "4xB200_NVL", - "isl": 1024, - "model": "nvidia/DeepSeek-R1-0528-FP4-v2", - "model_display_name": "DeepSeek-R1 (NVFP4)", - "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", - "num_gpus": 4, - "osl": 1024, - "performance_profile": "Balanced", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" - }, - { - "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc16.yaml", - "concurrency": 16, - "config_filename": "1k1k_tp4_conc16.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc16.yaml", - "config_path": "examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc16.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc16.yaml", - "gpu": "B200_NVL", - "gpu_display": "4xB200_NVL", - "isl": 1024, - "model": "nvidia/DeepSeek-R1-0528-FP4-v2", - "model_display_name": "DeepSeek-R1 (NVFP4)", - "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", - "num_gpus": 4, - "osl": 1024, - "performance_profile": "Balanced", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" - }, - { - "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc256.yaml", - "concurrency": 256, - "config_filename": "1k1k_tp4_conc256.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc256.yaml", - "config_path": "examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc256.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc256.yaml", - "gpu": "B200_NVL", - "gpu_display": "4xB200_NVL", - "isl": 1024, - "model": "nvidia/DeepSeek-R1-0528-FP4-v2", - "model_display_name": "DeepSeek-R1 (NVFP4)", - "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", - "num_gpus": 4, - "osl": 1024, - "performance_profile": "Max Throughput", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" - }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k8k_tp4_conc2048.yaml", "concurrency": 2048, @@ -1438,82 +1338,6 @@ "osl": 8192, "performance_profile": "High Throughput" }, - { - "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc4.yaml", - "concurrency": 4, - "config_filename": "8k1k_tp4_conc4.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc4.yaml", - "config_path": "examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc4.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc4.yaml", - "gpu": "B200_NVL", - "gpu_display": "4xB200_NVL", - "isl": 8192, - "model": "nvidia/DeepSeek-R1-0528-FP4-v2", - "model_display_name": "DeepSeek-R1 (NVFP4)", - "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", - "num_gpus": 4, - "osl": 1024, - "performance_profile": "Min Latency", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" - }, - { - "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc8.yaml", - "concurrency": 8, - "config_filename": "8k1k_tp4_conc8.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc8.yaml", - "config_path": "examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc8.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc8.yaml", - "gpu": "B200_NVL", - "gpu_display": "4xB200_NVL", - "isl": 8192, - "model": "nvidia/DeepSeek-R1-0528-FP4-v2", - "model_display_name": "DeepSeek-R1 (NVFP4)", - "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", - "num_gpus": 4, - "osl": 1024, - "performance_profile": "Low Latency", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" - }, - { - "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc16.yaml", - "concurrency": 16, - "config_filename": "8k1k_tp4_conc16.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc16.yaml", - "config_path": "examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc16.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc16.yaml", - "gpu": "B200_NVL", - "gpu_display": "4xB200_NVL", - "isl": 8192, - "model": "nvidia/DeepSeek-R1-0528-FP4-v2", - "model_display_name": "DeepSeek-R1 (NVFP4)", - "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", - "num_gpus": 4, - "osl": 1024, - "performance_profile": "Balanced", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" - }, - { - "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc256.yaml", - "concurrency": 256, - "config_filename": "8k1k_tp4_conc256.yaml", - "config_github_url": "https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc256.yaml", - "config_path": "examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc256.yaml", - "config_raw_url": "https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc256.yaml", - "gpu": "B200_NVL", - "gpu_display": "4xB200_NVL", - "isl": 8192, - "model": "nvidia/DeepSeek-R1-0528-FP4-v2", - "model_display_name": "DeepSeek-R1 (NVFP4)", - "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", - "num_gpus": 4, - "osl": 1024, - "performance_profile": "Balanced", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" - }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc1024.yaml", "concurrency": 1024, @@ -1529,7 +1353,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 4, "osl": 1024, - "performance_profile": "High Throughput" + "performance_profile": "Min Latency" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc2048.yaml", @@ -1597,9 +1421,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "Low Latency", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "Low Latency" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc8.yaml", @@ -1650,9 +1472,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "Balanced", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "Balanced" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc64.yaml", @@ -1669,9 +1489,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "Balanced", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "Balanced" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc128.yaml", @@ -1688,9 +1506,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "High Throughput", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "High Throughput" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc256.yaml", @@ -1707,9 +1523,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "High Throughput", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "High Throughput" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc512.yaml", @@ -1998,9 +1812,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "Low Latency", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "Low Latency" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc8.yaml", @@ -2068,9 +1880,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "High Throughput", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "High Throughput" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc128.yaml", @@ -2087,9 +1897,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "High Throughput", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "High Throughput" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc256.yaml", @@ -2106,9 +1914,7 @@ "model_url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2", "num_gpus": 8, "osl": 1024, - "performance_profile": "High Throughput", - "validated_trtllm_commit": "93cb6518b6d6dbd6095748189e626db731f44545", - "validated_trtllm_version": "1.3.0rc14" + "performance_profile": "High Throughput" }, { "command": "trtllm-serve nvidia/DeepSeek-R1-0528-FP4-v2 --config ${TRTLLM_DIR}/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc512.yaml", @@ -3726,10 +3532,6 @@ "display_name": "DeepSeek-R1", "url": "https://huggingface.co/deepseek-ai/DeepSeek-R1-0528" }, - "deepseek-ai/DeepSeek-V4-Pro": { - "display_name": "deepseek-ai/DeepSeek-V4-Pro", - "url": "" - }, "nvidia/DeepSeek-R1-0528-FP4-v2": { "display_name": "DeepSeek-R1 (NVFP4)", "url": "https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2" diff --git a/docs/source/_static/config_selector.css b/docs/source/_static/config_selector.css index 43c5a4aa7bc6..ca84bb991f2a 100644 --- a/docs/source/_static/config_selector.css +++ b/docs/source/_static/config_selector.css @@ -1,6 +1,3 @@ -/* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. */ -/* SPDX-License-Identifier: Apache-2.0 */ - .trtllm-config-selector { border: 1px solid var(--pst-color-shadow); border-radius: 14px; @@ -32,10 +29,6 @@ align-items: start; } -.trtllm-config-selector__field[hidden] { - display: none; -} - .trtllm-config-selector__label { display: flex; align-items: center; diff --git a/docs/source/_static/config_selector.js b/docs/source/_static/config_selector.js index ca053655923a..e8ef62550a6b 100644 --- a/docs/source/_static/config_selector.js +++ b/docs/source/_static/config_selector.js @@ -1,18 +1,14 @@ -// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - (function () { "use strict"; let dbPromise = null; let widgetId = 0; - const GROUP_ORDER = ["model", "topology", "islOsl", "concurrency", "profile"]; + const GROUP_ORDER = ["model", "topology", "islOsl", "concurrency"]; const GROUP_LABELS = { model: "Model", topology: "GPU(s)", islOsl: "ISL / OSL", concurrency: "Concurrency", - profile: "Profile", }; function $(root, sel) { @@ -62,7 +58,6 @@ state.concurrency != null && state.concurrency !== "" ? String(state.concurrency) : "", - profile: state.profile || "", }; } @@ -143,22 +138,9 @@ function profileLabel(profile) { const text = String(profile || "").trim(); - const labels = { - latency: "Latency", - balanced: "Balanced", - throughput: "Throughput", - }; - if (labels[text]) return labels[text]; return text || "Unknown Profile"; } - function profileOption(profile) { - return { - value: profile, - label: profileLabel(profile), - }; - } - function formatProfileSummary(profiles) { const labels = uniqBy(profiles.map((profile) => profileLabel(profile)), (label) => label); if (!labels.length) { @@ -212,12 +194,7 @@ .map((group) => concurrencyOption(group)) .sort((a, b) => sortNums(a.concurrency, b.concurrency)); - const profile = uniqBy( - entries.filter((entry) => entry.profile).map((entry) => profileOption(entry.profile)), - (option) => option.value - ); - - return { model, topology, islOsl, concurrency, profile }; + return { model, topology, islOsl, concurrency }; } function filterEntriesByState(entries, state) { @@ -237,9 +214,6 @@ if (normalizedState.concurrency) { if (String(entry.concurrency) !== normalizedState.concurrency) return false; } - if (normalizedState.profile && entry.profile !== normalizedState.profile) { - return false; - } return true; }); } @@ -388,8 +362,7 @@ normalizedState.model && normalizedState.topology && normalizedState.islOsl && - normalizedState.concurrency && - (!groups.profile.options.length || normalizedState.profile) + normalizedState.concurrency ) { return "Selection did not resolve to a single configuration."; } @@ -416,12 +389,6 @@ }; } - function validatedCommitUrl(commit) { - const normalized = String(commit || "").trim().toLowerCase(); - if (!/^[0-9a-f]{40}$/.test(normalized)) return ""; - return `https://github.com/NVIDIA/TensorRT-LLM/commit/${normalized}`; - } - function isFileProtocol() { return window.location.protocol === "file:"; } @@ -641,14 +608,14 @@ : allCurated ).map(normalizeEntry); - // curatedIndex lives outside normalizeState's scope and is preserved - // across Object.assign(state, view.state). + // curatedIndex lives outside normalizeState's scope — it is preserved + // across Object.assign(state, view.state) because normalizeState only + // touches the four filter keys. const state = { model: "", topology: "", islOsl: "", concurrency: "", - profile: "", curatedIndex: null, }; @@ -718,7 +685,6 @@ const selTopo = mkOptionGroup("GPU(s)", 2); const selSeq = mkOptionGroup("ISL / OSL", 3); const selConc = mkSelectField("Concurrency", `trtllm-conc-${id}`, 4); - const selProfile = mkSelectField("Profile", `trtllm-profile-${id}`, 5); form.appendChild(selModel.wrap); @@ -741,7 +707,6 @@ form.appendChild(selTopo.wrap); form.appendChild(selSeq.wrap); form.appendChild(selConc.wrap); - form.appendChild(selProfile.wrap); const output = el("div", { class: "trtllm-config-selector__output" }); const cmdPre = el("pre", { class: "trtllm-config-selector__cmd" }, [ @@ -870,10 +835,7 @@ return { type: "model-select" }; } if (activeEl === selConc.select) { - return { type: "select", key: "concurrency" }; - } - if (activeEl === selProfile.select) { - return { type: "select", key: "profile" }; + return { type: "select" }; } if ( activeEl.classList && @@ -895,8 +857,7 @@ return; } if (descriptor.type === "select") { - if (descriptor.key === "profile") selProfile.select.focus(); - else selConc.select.focus(); + selConc.select.focus(); return; } if (descriptor.type !== "button") return; @@ -1036,8 +997,8 @@ selectEl.dataset.status = (selectedOption && selectedOption.status) || "idle"; } - function setSelectOptions(selectEl, group, stateKey, placeholder) { - const previousValue = state[stateKey] || ""; + function setSelectOptions(selectEl, group) { + const previousValue = state.concurrency || ""; const visibleOptions = group.options.filter( (option) => option.status !== "incompatible" ); @@ -1045,7 +1006,7 @@ selectEl.appendChild( el("option", { value: "", - text: visibleOptions.length ? `Select ${placeholder}` : `No ${placeholder} available`, + text: visibleOptions.length ? "Select concurrency" : "No concurrency available", }) ); for (const option of visibleOptions) { @@ -1116,12 +1077,7 @@ setOptionButtons(selTopo.options, "topology", view.groups.topology); setOptionButtons(selSeq.options, "islOsl", view.groups.islOsl); - setSelectOptions(selConc.select, view.groups.concurrency, "concurrency", "concurrency"); - const hasProfileChoices = - Boolean(view.state.concurrency) && - view.groups.profile.options.some((option) => option.status !== "incompatible"); - selProfile.wrap.hidden = !hasProfileChoices; - setSelectOptions(selProfile.select, view.groups.profile, "profile", "profile"); + setSelectOptions(selConc.select, view.groups.concurrency); const code = cmdPre.querySelector("code"); if (curatedSelected) { @@ -1176,26 +1132,6 @@ } else { meta.appendChild(el("span", { text: e.config_path || "" })); } - if (e.validated_trtllm_version) { - meta.appendChild( - el("span", { - text: ` \u00b7 Validated with TensorRT-LLM ${e.validated_trtllm_version}`, - }) - ); - } - const commitUrl = validatedCommitUrl(e.validated_trtllm_commit); - if (commitUrl) { - meta.appendChild(el("span", { text: " \u00b7 Commit: " })); - meta.appendChild( - el("a", { - class: "trtllm-config-selector__configLink", - href: commitUrl, - target: "_blank", - rel: "noopener", - text: e.validated_trtllm_commit.slice(0, 12), - }) - ); - } currentEntry = e; resetYamlPanel(); @@ -1225,12 +1161,6 @@ selConc.select.addEventListener("change", () => { state.concurrency = selConc.select.value; - state.profile = ""; - render(); - }); - - selProfile.select.addEventListener("change", () => { - state.profile = selProfile.select.value; render(); }); @@ -1288,7 +1218,6 @@ formatCuratedCommand: formatCommand, nextStateAfterSelection, normalizeState, - validatedCommitUrl, }; } diff --git a/docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md b/docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md index 91d89d70cacf..22d4688c503d 100644 --- a/docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md +++ b/docs/source/blogs/Best_perf_practice_on_DeepSeek-R1_in_TensorRT-LLM.md @@ -92,7 +92,7 @@ Here we set `LOCAL_USER=1` argument to set up the local user instead of root acc Here we compile the source inside the container: ``` bash -python3 ./scripts/build_wheel.py --cuda_architectures "90-real;100-real" --clean +python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt --benchmarks --cuda_architectures "90-real;100-real" --python_bindings --clean ``` You can set the cuda_architectures to "100-real" if targeting Blackwell only, and "90-real" to target Hopper only to save some build time. diff --git a/docs/source/blogs/media/tech_blog26_agentperf_closed_loop_workflow.svg b/docs/source/blogs/media/tech_blog26_agentperf_closed_loop_workflow.svg deleted file mode 100644 index f9d0003051bd..000000000000 --- a/docs/source/blogs/media/tech_blog26_agentperf_closed_loop_workflow.svg +++ /dev/null @@ -1,156 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- -Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. -SPDX-License-Identifier: Apache-2.0 ---> -<svg xmlns="http://www.w3.org/2000/svg" width="1800" height="815" viewBox="0 0 1800 815" role="img" aria-labelledby="title desc"> - <title id="title">AA-AgentPerf closed-loop request workflow - A simulated agent sends the next recorded turn with its accumulated conversation. The inference deployment routes the request using conversation or cache affinity, reuses matching KV blocks, prefills only the uncached suffix, optionally retrieves the KV cache on the generation side in disaggregated serving, and streams the generated response. The client drains the complete response, simulates tool time when needed, advances the trajectory, and sends the next turn. - - - - - - - - - - - - - - - - - - - - - - - - - - CLIENT / AGENT HARNESS - - - - - INFERENCE DEPLOYMENT - - - - - 1 - Send recorded turn - Accumulated conversation - and the next request - - - - - 8 - Advance trajectory - Next recorded turn or - next assigned trajectory - - - - - 7 - Simulate tool time - Only for turns with a tool call - Client-side delay · no LLM compute - - - - - 6 - Receive complete response - Drain the stream and assemble chunks - Record TTFT and output speed - - - - - - - - - - 2 - Route + prefix lookup - Conversation-aware or cache-aware - worker and rank selection - Reuse matching resident KV blocks - - - - - 3 - Prefill uncached suffix - Reuse removes the cached prefix - from new prefill computation - Context side produces the first token - - - - - 4 - Retrieve KV cache - Returned context metadata locates - the KV cache for the generation side - Disaggregated serving only - - - - - 5 - Generate + stream - Continue decoding the response - on the same worker or a - separate generation worker - - - - - - - - - Disaggregated path - - - - Aggregated serving · same worker - - - - - - - Conversation affinity helps preserve reuse across turns - Later turns prefer the worker and rank that hold the accumulated conversation prefix. - - - - - Inference work - - - Client-simulated tool time - - - Conversation affinity and prefix reuse - - - diff --git a/docs/source/blogs/media/tech_blog26_deepseek_v4_hybrid_attention.png b/docs/source/blogs/media/tech_blog26_deepseek_v4_hybrid_attention.png deleted file mode 100644 index 8238521bcb40..000000000000 --- a/docs/source/blogs/media/tech_blog26_deepseek_v4_hybrid_attention.png +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:0b9b5b49bba503ffcd6b0fb7993cf25e2c6537098ad4ffc232e9a89890fc218f -size 13509614 diff --git a/docs/source/blogs/media/tech_blog26_deepseek_v4_mhc_moe.png b/docs/source/blogs/media/tech_blog26_deepseek_v4_mhc_moe.png deleted file mode 100644 index 32a68bbe5038..000000000000 --- a/docs/source/blogs/media/tech_blog26_deepseek_v4_mhc_moe.png +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:a034c944f943400df7b42b5a1f2321f7d3ee87eb80bd5c509dba5066cec5eee4 -size 1854626 diff --git a/docs/source/blogs/media/tech_blog26_dsv4_performance_evolution_8k1k.png b/docs/source/blogs/media/tech_blog26_dsv4_performance_evolution_8k1k.png deleted file mode 100644 index 3fbaede6ebe4..000000000000 Binary files a/docs/source/blogs/media/tech_blog26_dsv4_performance_evolution_8k1k.png and /dev/null differ diff --git a/docs/source/blogs/media/tech_blog26_dsv4_scratch_swa.png b/docs/source/blogs/media/tech_blog26_dsv4_scratch_swa.png deleted file mode 100644 index 37e806c18c77..000000000000 Binary files a/docs/source/blogs/media/tech_blog26_dsv4_scratch_swa.png and /dev/null differ diff --git a/docs/source/blogs/media/tech_blog26_two_level_routing.svg b/docs/source/blogs/media/tech_blog26_two_level_routing.svg deleted file mode 100644 index f032ffafa1ba..000000000000 --- a/docs/source/blogs/media/tech_blog26_two_level_routing.svg +++ /dev/null @@ -1,167 +0,0 @@ - - - - Two-level routing for context locality in disaggregated serving - An agentic request enters the TensorRT LLM disaggregated front-end. Instance-level routing first selects a context server, then the selected server's attention data-parallel router performs rank-level routing. Each level can use conversation-aware affinity or KV-cache-aware scoring. Full conversation-prefix reuse requires a later turn to return to both the context server and the rank that hold the corresponding KV blocks. The selected context rank transfers KV cache to a generation server, which decodes and streams the response. - - - - - - - - - - - - - - - - - - - - - - - - - - - - AGENT CLIENT - Next turn - - Conversation ID - - Accumulated prompt - - Cache salt (optional) - - - - request - - - - - TRTLLM-SERVE DISAGGREGATED - Front-end orchestrator - - LEVEL 1 · INSTANCE-LEVEL ROUTING - Use either routing policy to select a context server - - - - CONVERSATION-AWARE - Conversation ID → CTX server - Least-loaded first placement, then explicit affinity - - - - KV-CACHE-AWARE - Prompt blocks → cache view - Matched blocks plus active-request load - Content-derived affinity protects later turns - - - Selected destination: CTX server 1 - - - - - CONTEXT (CTX) TIER - - - - CTX server 0 - other rank-local caches - - CTX server 2 - other rank-local caches - - - - - SELECTED - CTX server 1 - - - - Attention data-parallel router - - LEVEL 2 · RANK-LEVEL ROUTING - Again, use either routing policy inside the selected server - - - CONVERSATION-AWARE - Conversation ID → rank - First turn: round-robin - - - KV-CACHE-AWARE - Rank-local prefix probe - Post-reuse work plus load - - - - - - rank-local reuse lost - - - Rank 0 - KV cache - - - Rank 1 - - prefix KV - - - Rank 2 - KV cache - - - Rank 3 - KV cache - - - - same server - - server-local - reuse lost - - - - - GEN SERVER - Generation - - Receive KV cache - - Continue decode - - Stream response - - - CTX → GEN KV transfer - - - - streamed response - diff --git a/docs/source/blogs/tech_blog/blog26_DeepSeek_V4_on_NVIDIA_Blackwell_Model_Specific_and_Agentic_Workload_Optimizations_in_TensorRT-LLM.md b/docs/source/blogs/tech_blog/blog26_DeepSeek_V4_on_NVIDIA_Blackwell_Model_Specific_and_Agentic_Workload_Optimizations_in_TensorRT-LLM.md deleted file mode 100644 index 260b94448f42..000000000000 --- a/docs/source/blogs/tech_blog/blog26_DeepSeek_V4_on_NVIDIA_Blackwell_Model_Specific_and_Agentic_Workload_Optimizations_in_TensorRT-LLM.md +++ /dev/null @@ -1,490 +0,0 @@ - - -# DeepSeek-V4 on NVIDIA Blackwell: Model-Specific and Agentic-Workload Optimizations in TensorRT LLM - -By NVIDIA TensorRT LLM Team - -## Table of Contents - -- [Introduction](#introduction) -- [Part I. DeepSeek-V4 Model Support and Optimizations](#part-i-deepseek-v4-model-support-and-optimizations) - - [Building a Production-Ready DeepSeek-V4 Stack](#building-a-production-ready-deepseek-v4-stack) - - [Model support and parallelism](#model-support-and-parallelism) - - [Hybrid attention: SWA, CSA, and HCA](#hybrid-attention-swa-csa-and-hca) - - [Compressor implementation](#compressor-implementation) - - [Beyond attention: mHC, MoE, and speculative decoding](#beyond-attention-mhc-moe-and-speculative-decoding) - - [Parallelism and deployment notation](#parallelism-and-deployment-notation) - - [Cache management and runtime features](#cache-management-and-runtime-features) - - [Precision strategy and accuracy](#precision-strategy-and-accuracy) - - [DeepSeek-V4 Performance Optimizations](#deepseek-v4-performance-optimizations) - - [Optimize the CSA/HCA hot path](#optimize-the-csahca-hot-path) - - [Compressor and mHC](#compressor-and-mhc) - - [Compressor optimizations](#compressor-optimizations) - - [mHC optimizations](#mhc-optimizations) - - [MoE optimizations](#moe-optimizations) - - [Runtime optimization](#runtime-optimization) - - [DeepSeek-V4 Performance Evolution](#deepseek-v4-performance-evolution) - - [Experimental setup](#experimental-setup) - - [From Baseline to the Latest Measured Curve](#from-baseline-to-the-latest-measured-curve) -- [Part II. Agentic-Workload Optimizations](#part-ii-agentic-workload-optimizations) - - [AgentPerf Workflow](#agentperf-workflow) - - [Lessons from DeepSeek-V3.2 Agentic Workload Optimization](#lessons-from-deepseek-v32-agentic-workload-optimization) - - [End-to-End Optimizations for Agentic Serving](#end-to-end-optimizations-for-agentic-serving) - - [Preserve locality with two-level routing](#preserve-locality-with-two-level-routing) - - [Instance-level routing across context servers](#instance-level-routing-across-context-servers) - - [Rank-level routing within a context server](#rank-level-routing-within-a-context-server) - - [Routing policy selection](#routing-policy-selection) - - [KV cache reuse optimizations](#kv-cache-reuse-optimizations) - - [Optimize scheduling and remove end-to-end overhead](#optimize-scheduling-and-remove-end-to-end-overhead) - - [Optimize host overhead for CTX and GEN](#optimize-host-overhead-for-ctx-and-gen) - - [Optimize scheduling, orchestration, and protocol handling](#optimize-scheduling-orchestration-and-protocol-handling) - - [AgentPerf Results and External Validation](#agentperf-results-and-external-validation) -- [Reproduction and Future Work](#reproduction-and-future-work) - - [How to reproduce](#how-to-reproduce) - - [Future Work](#future-work) -- [Conclusion and Acknowledgments](#conclusion-and-acknowledgments) - -## Introduction - -DeepSeek-V4's hybrid attention, online compression, mHC, and MoE design make production inference a model-and-system co-design problem. This blog follows TensorRT LLM's optimization journey on NVIDIA Blackwell in two parts: first building and optimizing a production-ready DeepSeek-V4 model stack, then extending the optimization boundary to routing, KV reuse, host efficiency, and control-plane efficiency for agentic workloads. On GB300, the latest measured fixed-shape sweep increased peak output throughput from 984 to 1,618 tokens/s/GPU, a 64.5% improvement, while NVIDIA's GB300 AA-AgentPerf configurations, run and verified by Artificial Analysis, reached 57.5 concurrency per GPU (CPG) at SLO20 and 19.2 CPG at SLO60. - -## Part I. DeepSeek-V4 Model Support and Optimizations - -### Building a Production-Ready DeepSeek-V4 Stack - -Supporting DeepSeek-V4 required more than registering a new model class. Relative to DeepSeek-V3.2, the model replaces homogeneous sparse attention with a hybrid scheme that interleaves three attention modes across layers, adds an online sequence Compressor with its own persistent state, wraps every attention and MoE block in manifold-constrained hyper-connections (mHC), and updates the MoE routing and checkpoint layout. All of this must compose with the production features users expect from TensorRT LLM: Multi-Token Prediction (MTP), parallel execution, chunked prefill, KV-cache reuse, CUDA Graphs, the overlap scheduler, and disaggregated serving. - -TensorRT LLM implements DeepSeek-V4 as a dedicated PyTorch-backend model, `DeepseekV4ForCausalLM`, together with a DeepSeek-V4 sparse-attention backend and a specialized `DeepseekV4CacheManager`, targeting NVIDIA Blackwell GPUs (SM100+). This section describes that foundation. The performance optimizations built on top of it are covered in [**DeepSeek-V4 Performance Optimizations**](#deepseek-v4-performance-optimizations). - -#### Model support and parallelism - -DeepSeek-V4 is released in two model scales, each with Base and Instruct variants. TensorRT LLM loads both directly from their checkpoint metadata. - -| Checkpoint family | Total parameters | Activated parameters | Published precision | -| :--- | ---: | ---: | :--- | -| DeepSeek-V4-Flash-Base / -Flash | 284B | 13B | FP8 mixed / FP4 + FP8 mixed | -| DeepSeek-V4-Pro-Base / -Pro | 1.6T | 49B | FP8 mixed / FP4 + FP8 mixed | - -The implementation reads the attention layout from the checkpoint rather than hard-coding serving assumptions: the per-layer compression-ratio list (Flash begins `[0, 0, 4, 128, ...]`, while Pro begins `[128, 128, 4, 128, ...]`, and 0 marks a sliding-window-only layer), the 128-token sliding window, and the Indexer configuration including its Top-K (512 for Flash and 1024 for Pro). - -##### Hybrid attention: SWA, CSA, and HCA - -DeepSeek-V4's largest architectural change is its attention. DeepSeek-V3.2 introduced DeepSeek Sparse Attention (DSA), in which a learned Indexer scores the KV history and a Top-K selector supplies token indices to sparse MLA (see our [DeepSeek-V3.2 optimization blog](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog15_Optimizing_DeepSeek_V32_on_NVIDIA_Blackwell_GPUs.md)). TensorRT LLM implemented DSA on a general [sparse-attention framework](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog17_Sparse_Attention_in_TensorRT-LLM.md) that separates the algorithm-specific selection logic from the attention backend that consumes the selected indices. DeepSeek-V4 builds on the same separation but adds online compression and interleaves three layer types, determined by each layer's compression ratio: - -| Attention mode | Ratio | History visible to each query | Selection and computation | -| :--- | ---: | :--- | :--- | -| Sliding Window Attention (SWA) | 0 | The latest 128 raw tokens | Dense attention inside the window. No Compressor or Indexer. | -| Compressed Sparse Attention (CSA) | 4 | Latest 128 raw tokens + 4× compressed entries from completed compression groups | The Indexer scores the compressed entries. Sparse MLA attends to the window plus the Top-K selected entries. | -| Heavily Compressed Attention (HCA) | 128 | Latest 128 raw tokens + all 128× compressed entries from completed compression groups | Dense attention over all completed compressed entries. No Indexer or Top-K stage. | - -
-
- DeepSeek-V4 hybrid attention architecture. Panel a shows SWA attending over the latest 128 raw tokens. Panel b shows CSA combining the sliding window with 4× compressed KV entries selected by an Indexer and Top-K stage. Panel c shows HCA combining the sliding window with all 128× compressed KV entries without an Indexer. -
-
-

Figure 1. DeepSeek-V4 hybrid attention. (a) SWA attends densely within the 128-token sliding window. (b) CSA augments that window with Top-K entries selected from the 4× compressed history. (c) HCA attends to the window and the complete 128× compressed history without an Indexer.

- -At the kernel boundary, the history is represented as two cache pools, one for the sliding window and one for compressed entries, and a unified dual-pool MLA operator serves all three modes: it processes the sliding window first, then the selected CSA entries or the full HCA compressed stream, and combines both in one online-softmax reduction. An SWA-only layer simply passes an empty second pool. Selected compressed positions are converted from cache-page-local offsets to token-level global addresses before the attention kernel consumes them. Because KV compression makes the attention workload vary by query, TensorRT LLM passes a per-query `topk_lens` vector (`sparse_mla_topk_lens` in the implementation) to the FMHA kernel. It identifies the active SWA-plus-compressed index span within the fixed-width index buffer used for CUDA Graph compatibility. - -Checkpoint-provided attention-sink logits are also remapped and supplied to the backend as an additional softmax sink. This dual-pool design lets the heterogeneous layer types share one attention backend, and extends the sparse-attention framework from DSA's fine-grained token selection to DeepSeek-V4's combination of local attention, compressed memory, and optional sparse selection. - -##### Compressor implementation - -The Compressor produces the compressed KV entries that CSA and HCA attend to. It is a token-level, softmax-gated pooling ([DeepSeek-V4 paper](https://arxiv.org/abs/2606.19348)): each token's projected KV entry carries a learned compression-weight vector, a learnable per-position bias is added, and a softmax over the compression window reduces the group to one entry as a weighted sum. For CSA ($m{=}4$), each output pools projected entries from two adjacent $m$-token groups, giving a $2m$-token receptive field. Consecutive outputs overlap by $m$ raw-token positions through separate projection branches. HCA instead uses disjoint $m'$-token windows ($m'{=}128$). On CSA layers the same compression operation is applied a second time to produce the compressed Indexer keys that the Indexer scores for Top-K selection, so a CSA layer maintains two compressed streams: the attention KV entries and the Indexer keys. - -TensorRT LLM implements the Compressor as a fused `wkv_gate` projection plus dedicated CUDA kernels for prefill reduction, paged decode updates, and cache-write postprocessing. Tokens arrive incrementally: one at a time in decode, and chunk by chunk in chunked prefill, where chunk boundaries rarely align with compression windows. A compression window is often still incomplete when a forward pass ends. TensorRT LLM persists each token's KV and score projection outputs in FP32 Compressor-state buffers for the still-open windows. Once a window completes, the compression kernels finalize its entry from this state, preserving the semantics of compressing the whole sequence in one pass. Compressed entries are written in the dtype their consumer expects: BF16 or per-tensor FP8 for the main attention cache, and blockwise FP8 or packed MXFP4 for the Indexer-K stream. The kernel fusion work on this path is covered in [**Compressor and mHC**](#compressor-and-mhc). - -##### Beyond attention: mHC, MoE, and speculative decoding - -**mHC hyper-connections** ([mHC paper](https://arxiv.org/abs/2512.24880)). mHC widens the residual stream by a factor of 4 and mixes it around every sublayer through three dynamically generated mappings: a **pre-mapping** that reads the sublayer input from the expanded residual, a **residual-mixing** matrix projected onto the manifold of doubly stochastic matrices through 20 Sinkhorn-Knopp iterations, and a **post-mapping** that writes the sublayer output back into the residual streams. TensorRT LLM instantiates two mHC modules per layer (one around attention and one around the MoE block), plus an **HC head** that collapses the expanded residual to a single stream before the language-model head. It fuses adjacent post- and pre-mappings at the attention-to-MoE boundary and, where possible, across consecutive transformer layers. - -**MoE.** Both model scales activate six routed experts and one shared expert per token, selecting from 256 routed experts in Flash and 384 in Pro. The first three layers use hash routing, where a checkpoint-provided token-ID-to-expert table maps each input token directly to its experts. Later layers use a learned gate with Sqrt-Softplus affinity scores and a score-correction bias. TensorRT LLM implements both routing modes through a shared interface that supplies the selected experts and routing weights to its configurable MoE backends. TRTLLM-Gen, DeepGEMM, and the standard CuTe DSL fused MoE backend cover the corresponding DeepSeek-V4 precision paths and preserve the checkpoint-defined `swiglu_limit` activation clamp, including the uniform clamp used by the NVFP4 CuTe DSL path. DeepSeek-V4 is also registered with TensorRT LLM's expert parallel load balancer, allowing supported expert-parallel deployments to rebalance physical expert placement without changing the model's routing semantics. - -
-
- DeepSeek-V4 mHC architecture and MoE routing. Panel a shows the repeated transformer stack with mHC pre- and post-mappings around the attention and MoE blocks, followed by the HC head and language-model head. Panel b shows hash routing through a token-ID embedding table and learned routing through Sqrt-Softplus scores, correction bias, and Top-K expert selection. -
-
-

Figure 2. DeepSeek-V4 components beyond attention. (a) mHC wraps the attention and MoE sublayers with pre- and post-mappings, then uses an HC head to collapse the expanded residual stream before the language-model head. (b) The MoE router supports checkpoint-defined hash routing in the first three layers (left) and learned Sqrt-Softplus scoring with correction bias and Top-6 selection in later layers (right).

- -**Speculative decoding.** DeepSeek-V4 checkpoints with next-token prediction layers run MTP speculative decoding through the Eagle-style one-model path. Each MTP module consumes the expanded residual state from the target model or previous MTP module together with the next-token embedding, then runs the same mHC-wrapped attention-and-MoE structure as a target-model layer, using SWA for its attention. TensorRT LLM remaps the MTP weights, shares the embedding and language-model head with the target model, and extends the attention and cache metadata for the configured draft depth. The speculative-decoding performance results reported in this blog use this MTP path. DeepSeek has released dedicated [DSpark](https://arxiv.org/abs/2607.05147) checkpoints for DeepSeek-V4. DSpark adds a lightweight sequential head to a parallel draft backbone to model dependencies within a draft block, while performance tuning for this path remains in progress. - -##### Parallelism and deployment notation - -DeepSeek-V4 adopts the same parallel strategies as DeepSeek-R1 and DeepSeek-V3.2: attention data parallelism (ADP) combined with expert parallelism (EP) for throughput-oriented serving, and tensor parallelism (TP) for latency-oriented deployments, with pipeline parallelism available for fitting the largest checkpoints. See [Tech Blog 3](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog03_Optimizing_DeepSeek_R1_Throughput_on_NVIDIA_Blackwell_GPUs.md) for the performance rationale and [Tech Blog 4](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog04_Scaling_Expert_Parallelism_in_TensorRT-LLM.md)/[8](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog08_Scaling_Expert_Parallelism_in_TensorRT-LLM_part2.md)/[14](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog14_Scaling_Expert_Parallelism_in_TensorRT-LLM_part3.md) for large-scale expert parallelism. Note that with ADP, `max_batch_size` is a per-rank limit, so benchmark concurrency must be scaled to populate all ranks. - -The rest of this blog uses a compact notation for these combinations. `TEP` shards both attention (TP) and experts (EP) across `N` ranks. `DEP` keeps attention data-parallel (ADP) while distributing experts across `N` ranks. Both compose with disaggregated serving, where context (CTX) and generation (GEN) servers are sized independently. For example, `DEP4 (CTX) + TEP8 (GEN)` runs a four-rank DEP context instance alongside an eight-rank TEP generation instance. - -#### Cache management and runtime features - -DeepSeek-V4's hybrid attention turns the "KV cache" into a collection of states with different shapes and lifetimes. Each attention mode owns a different set of logical cache layers: - -- an **SWA** layer owns only its sliding-window KV. -- a **CSA** layer adds the 4× compressed attention cache, Compressor KV/score state, compressed Indexer-K cache, and Indexer-Compressor KV/score state. -- an **HCA** layer adds the 128× compressed attention cache and Compressor KV/score state, with no Indexer-side cache. - -These tensors differ in dtype, bytes per entry, production rate, and lifetime. Treating them as a uniform KV tensor would either waste memory or give the scheduler an incorrect view of capacity. For the rest of this blog, we group them into two lifecycle categories. The **sliding-window cache** contains SWA's sliding-window KV and the short-lived Compressor KV/score state used by CSA, HCA, and the CSA Indexer. These buffers are needed only for bounded windows over recent raw-token positions, so older pages can be recycled. The **persistent compressed caches** contain the finalized 4× and 128× compressed attention entries and CSA's compressed Indexer-K entries. They represent retained history and remain available for attention and prefix reuse, so their storage grows with sequence length at the corresponding compression rate. These terms describe lifecycle categories rather than a one-to-one mapping to physical pools. `DeepseekV4CacheManager` describes every state as a buffer role with an explicit lifecycle and page-indexing mode, then groups compatible buffers into physical memory pools. Sliding-window cache buffers use per-layer page indices. Persistent compressed caches use shared indices across layers with the same compression layout. - -The Compressor breaks two assumptions made by a conventional paged KV cache: its intermediate state is short-lived, and its persistent output grows more slowly than the raw token sequence. TensorRT LLM maps both cases onto existing KV Cache Manager V2 abstractions. - -**Short-lived Compressor state.** A raw token's intermediate state is needed only while a compression window that contains it remains open. Once the last such window is finalized, that token's intermediate state can be recycled. This lifetime is naturally represented as a sliding window: 128 raw-token positions for HCA, and 8 for CSA because its overlapping Compressor keeps two 4-token groups live. The manager can therefore reuse the existing window-eviction and block-recycling machinery. - -**Slow-growing compressed caches.** A compressed stream produces one entry for every $r$ raw tokens. Rather than introduce a second coordinate system, the manager keeps page allocation in raw-token coordinates. For a page covering `tokens_per_block` raw positions, its compressed buffer stores `tokens_per_block / r` entries. The page still represents the same interval of the original sequence, but its storage cost reflects the 4× or 128× compression ratio. Allocation, block-table construction, prefix reuse, and capacity accounting can consequently share one page model across all streams. - -The physical pool split is workload-dependent. Fresh prefill creates the greatest pressure on the sliding-window cache, whereas persistent compressed caches grow with sequence length and decode concurrency. When `kv_cache_config.pool_ratio` is not specified, KV Cache Manager V2 derives an initial split from a representative mixed batch: one context request and up to `max_batch_size - 1` generation requests. `kv_cache_config.avg_seq_len` controls the representative total sequence length of a typical decode request, while additional constraints reserve enough capacity for a maximum-length decode request and a chunked-prefill step. Advanced deployments can override the initial split with `pool_ratio` (specified in pool-group order and summing to 1.0). An opt-in beta rebalancer, enabled with `enable_kv_pool_rebalance`, can later adjust the split from runtime statistics. It is disabled by default and currently targets a narrower set of aggregated-serving configurations. - -DeepSeek-V4 also enables SWA scratch reuse by default. During prefill, transient sliding-window pages can be recycled across compatible layers instead of being retained as independent long-lived allocations. This reduces the prefill peak that would otherwise dictate the entire pool split. Its performance impact is discussed in [**Runtime optimization**](#runtime-optimization). - -With this state model in place, the same attention implementation composes with the runtime features expected in production: - -- **Chunked prefill.** The Compressor and Indexer carry compressed lengths and open-window state across chunks, preserving the semantics of processing the prompt in one pass. -- **KV-cache reuse.** Prefix reuse restores the attention, Compressor, and Indexer state owned by each layer type, rather than treating the main attention KV as sufficient. -- **CUDA Graphs.** Top-K and Compressor workspaces use graph-safe buffers whose captured shapes remain stable for a given graph batch size. -- **Overlap scheduling.** Cache updates and sparse-attention preparation are designed to avoid host-device synchronizations on the critical path, allowing request preparation to overlap GPU execution. -- **Disaggregated serving.** The V2 transfer path describes cache contents by pool, layer, and buffer role, allowing context workers to transfer every DeepSeek-V4 cache type and generation workers to rebuild their local block tables consistently. - -Production support also extends to the API boundary. For Instruct checkpoints, selecting the `deepseek_v4` tokenizer wrapper applies the reference chat format, including thinking controls, tool definitions, and DSML, DeepSeek's special-token-delimited format for tool calls. Tool results are inserted with `` tags. Matching reasoning and tool parsers turn generated thinking and DSML tool-call blocks back into OpenAI-compatible response fields. Base checkpoints remain completion models and should receive raw prompts without this wrapper. - -#### Precision strategy and accuracy - -DeepSeek-V4 uses three checkpoint-level precision recipes, each mapping low precision to the components where it reduces weight or cache bandwidth while keeping higher precision for stateful reductions and numerically sensitive paths: - -| Component | FP8 (Base) | MXFP4 (Instruct) | NVFP4 (requantized) | -| :--- | :--- | :--- | :--- | -| MoE routed experts GEMM | Blockwise FP8 | MXFP8 activations × MXFP4 weights | NVFP4 | -| MoE shared expert GEMM | Blockwise FP8 | MXFP8 | MXFP8 | -| Attention QKV/O GEMM | Blockwise FP8 | MXFP8 | MXFP8 | -| Indexer Q projection | Blockwise FP8 | MXFP8 | MXFP8 | -| Indexer weights projection | BF16 | BF16 | BF16 | -| Compressor KV/score linears | BF16 | BF16 | BF16 | -| Main attention KV cache | BF16 or per-tensor FP8 | BF16 or per-tensor FP8 | BF16 or per-tensor FP8 | -| Indexer K cache | Blockwise FP8 or MXFP4 | Blockwise FP8 or MXFP4 | Blockwise FP8 or MXFP4 | -| Compressor state cache | FP32 | FP32 | FP32 | - -Apart from the main-transformer routed experts, the NVFP4 checkpoint keeps its tensors identical to the MXFP4 Instruct source. In particular, the MTP subtree retains the Instruct precision recipe. The two FP4 routed-expert formats are distinct recipes served by different kernels. The Instruct checkpoints publish packed MXFP4 routed-expert weights executed with MXFP8 activations and MXFP4 weights, while the main-layer experts in requantized checkpoints such as `nvidia/DeepSeek-V4-Pro-NVFP4` run through the NVFP4 MoE path. On the cache side, FP4 approximately halves the Indexer-K data payload and is TensorRT LLM's default Indexer-K format for DeepSeek-V4 on Blackwell. The attention projections follow the same philosophy of staying in low precision across kernel boundaries. For eligible context-only sparse MLA batches, query quantization is fused into the Q RMSNorm and RoPE kernels, which write the FP8 Q buffer consumed by attention. Generation and mixed batches use the unfused Q path. For eligible DeepSeek-V4 Pro configurations, inverse RoPE and FP8 quantization are further fused into the FMHA epilogue, while other configurations use the optimized standalone inverse-RoPE and quantization kernel before `o_a_proj`. These kernel fusions are discussed further in [**Optimize the CSA/HCA hot path**](#optimize-the-csahca-hot-path). - -Accuracy was validated across Flash and Pro checkpoints, the FP8 (Base), MXFP4 (Instruct), and NVFP4 recipes, MTP on and off, FP8 and BF16 KV cache, and aggregated and disaggregated serving, comparing GPQA-Diamond and LiveCodeBench scores against the paper baselines. In this blog, MTP-1 and MTP-3 denote speculative draft lengths of one and three tokens, respectively. Representative GPQA-Diamond results with MTP-3 and FP8 KV cache unless noted: - -| Configuration | GPQA-Diamond | -| :--- | ---: | -| DeepSeek-V4-Flash | 88.8 | -| DeepSeek-V4-Pro | 89.08 | -| DeepSeek-V4-Pro, disaggregated | 89.39 | -| DeepSeek-V4-Pro-NVFP4 | 89.39 | - -Production readiness also required testing well beyond single-feature correctness: the validation matrix spans CUDA Graph capture, chunked prefill with cached history, MTP, long contexts, high concurrency, disaggregated transfer and shutdown, and memory pressure during loading and autotuning, with model-specific CI covering aggregated and disaggregated serving on Blackwell. This coverage is what turned the initial functional implementation into a stack the subsequent performance work could safely optimize. - -### DeepSeek-V4 Performance Optimizations - -Building on the production-ready execution stack described above, TensorRT LLM optimizes DeepSeek-V4 across four areas: the CSA/HCA attention hot path, Compressor and mHC, MoE, and runtime execution and memory management. The following sections explain the key techniques in each area, while [**DeepSeek-V4 Performance Evolution**](#deepseek-v4-performance-evolution) presents their cumulative end-to-end impact under a consistent model workload. Performance numbers cited here are scoped operator measurements or A/B end-to-end comparisons, not cumulative speedups. - -#### Optimize the CSA/HCA hot path - -CSA and HCA share a dual-pool attention kernel over the 128-token sliding window and compressed history, while CSA additionally runs an Indexer and Top-K selection. TensorRT LLM optimizes this hot path through attention-kernel improvements, low-precision fusion, multi-stream scheduling, and Top-K optimization. - -**Sparse MLA kernel optimization.** The dual-pool kernel processes the 128-token sliding window first, then gathers selected CSA entries or streams all HCA entries from the compressed pool. The optimized TRTLLM-Gen kernel skips redundant dense and sliding-window softmax masking on full tiles and issues sparse V loads with less coordinate setup and register spill/reload pressure. On B200 with an FP8 KV cache, the measured FMHA kernel was up to 1.31× faster for CSA prefill and 1.21× faster for HCA generation in the tested shapes. These are operator-level results, and the model-level gain depends on the model's mix of SWA, CSA, and HCA layers. - -**Kernel fusion and low-precision dataflow.** The functional path initially relied on framework or generic operators around attention. TensorRT LLM adds a DeepSeek-V4-specific Q RMSNorm CUDA kernel and extends the MLA RoPE/assignment and FMHA epilogue kernels so that normalization, rotation, and quantization can be fused into their producers. The resulting optimizations are: - -- **Fused Q RMSNorm, RoPE, and FP8 quantization.** The initial context path used three kernels for Q RMSNorm, RoPE and assignment, and full-Q FP8 quantization. For eligible context-only sparse MLA batches with an FP8 KV cache, TensorRT LLM folds quantization of the 448-dimensional non-RoPE segment into the Q RMSNorm kernel, then folds quantization of the remaining 64-dimensional RoPE segment and BMM-scale generation into the RoPE and assignment kernel. These two producer kernels write directly into the same FP8 Q buffer, eliminating the standalone quantization kernel and the full higher-precision normalized Q tensor. Generation and mixed batches continue to use the unfused path. -- **Standalone inverse-RoPE and FP8 quantization.** DeepSeek-V4 first establishes an FP8-native input path for `o_a_proj` by combining inverse RoPE and 1×128 blockwise FP8 quantization in a standalone operator. This operator consumes the BF16 output from FMHA and writes E4M3 activations and FP32 scales directly in the layout required by the Blackwell `o_a_proj` BMM. Replacing the original Triton implementation with a shape-specialized CUDA kernel made the standalone operator 1.6–2.3× faster in kernel-only measurements across 1K–32K tokens on B200. This path remains the fallback when FMHA epilogue fusion is not applicable. -- **FMHA epilogue fusion.** Building on the same FP8 `o_a_proj` input contract, TensorRT LLM moves inverse RoPE and 1×128 E4M3 quantization into the FMHA correction epilogue for validated DeepSeek-V4 Pro configurations using the FP8 KV cache, sparse MLA, and attention data parallelism. FMHA then writes the E4M3 output and FP32 scales directly in the layout consumed by the `o_a_proj` BMM, eliminating both the standalone operator launch and the intermediate BF16 output write and read. The `o_a_proj` BMM remains a separate kernel. This fusion supports context-only and generation-only batches, while mixed batches and unsupported configurations use the standalone path. Relative to the combined time of standard FMHA and the standalone inverse-RoPE and quantization operator, the fused FMHA kernel was up to 1.62× faster for prefill and 1.34× faster for generation in the tested shapes. -- **Others.** For CSA layers with BF16, bias-free `q_b` projection weights, a shape-autotuned CuTe DSL GEMM replaces the generic linear path on supported SM100f systems and falls back when CuTe DSL is unavailable. The optimized attention path also removes a device-to-device copy after FMHA, and FP8 quantization kernels initialize their own scale-buffer padding instead of requiring a separate host-launched `zero_()` operation. - -**Top-K optimizations.** The CSA Indexer selects hundreds of entries per query, making Top-K particularly visible during long-context decode. TensorRT LLM accelerates it in two complementary ways: - -- **Top-K kernel optimization.** TensorRT LLM makes the standard Top-K path shape-aware and device-aware. Depending on the compressed sequence length and row count, the dispatcher uses insertion selection, a single-CTA radix kernel, or a multi-CTA split-and-merge implementation. The multi-CTA path exposes enough parallel work for small decode batches, while its launch policy uses the row count, compressed sequence length, configured split threshold, and the GPU's actual SM count. On B200 with FP32 logits, 196,608 columns, `next_n=1`, and no previous indices, the device-aware policy reduced the K=512 latency at batch size 148 from 384 µs to 112 µs. -- **GVR Top-K.** GVR, introduced in our [GVR technical blog](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog21_Temporal_Correlation_Meets_Sparse_Attention.md), exploits the overlap between the entries selected by adjacent decode steps. It uses the previous step's indices as candidates, verifies them against current scores, and refines or falls back when the fast-path conditions are not met. DeepSeek-V4 extends GVR to the Indexer's 4× compressed coordinate space and to checkpoint-defined Top-K values of 512 for Flash and 1024 for Pro. In kernel-only B300 tests, GVR was 1.40–2.17× faster than the radix-selection baseline for the tested Flash and Pro shapes. An end-to-end Flash test on eight B300 SXM6 GPUs with a 65K-token input, batch size 4, and MTP-3 reported 6.4% higher request throughput. - -**Multi-stream attention scheduling.** The CSA attention prologue contains the main-attention Compressor, the `q_b` projection, and an Indexer with only a few true dependencies between them. The main-attention Compressor and the query-independent part of the Indexer can start directly from the layer input, while the `q_b` projection and Q RMSNorm wait for the shared projection and `q_a` normalization. TensorRT LLM progressively exposes this parallelism at two levels: - -- **Initial outer overlap.** The first dependency-aware schedule runs the main-attention Compressor concurrently with the `q_b` projection and Q RMSNorm. It then adds a dedicated stream for the complete Indexer branch, allowing all three branches to make progress before sparse MLA consumes their outputs. -- **Nested Indexer overlap.** Inside the Indexer, query projection, RoPE, and Q quantization form one branch, while the query-independent weight projection, Indexer Compressor, and K-cache update form another. Events synchronize them only when Q scales are needed for weight scaling and when the logits and Top-K kernels require the updated cache. -- **Final schedule with earlier launches.** The refined schedule pre-launches the main-attention Compressor and the query-independent half of the Indexer before the shared `kv_a` projection and Q/KV normalizations. Once normalized Q is available, the query-dependent Indexer work runs on its dedicated stream, while the `q_b` projection and Q RMSNorm are queued after the main-attention Compressor on the Compressor stream. This is the final stream assignment, replacing the earlier three-way outer schedule. - -Correctness across streams requires explicit tensor-lifetime tracking. TensorRT LLM calls `record_stream()` on the precomputed Indexer tensors, normalized Q output, and Top-K indices when they move to the consuming stream, preventing the PyTorch caching allocator from reusing their storage before the downstream work completes. - -#### Compressor and mHC - -The Compressor and mHC introduce repeated chains of small, stateful, and reduction-heavy operations around attention and MoE. The Compressor turns projected KV and scores into persistent paged state and compressed cache entries, while mHC transforms the residual streams at each sublayer boundary. TensorRT LLM optimizes both in two layers: first fuse adjacent operations to remove launches and intermediate tensors, then specialize the fused kernels for DeepSeek-V4 shapes and execution regimes. - -##### Compressor optimizations - -The optimized Compressor pipeline uses fusion at three boundaries: - -- **KV and score projection.** A single `wkv_gate` projection produces both the KV vectors and their compression scores. -- **Compression and state update.** The prefill and decode kernels write new KV/score state into paged storage and perform the online-softmax reduction when a compression window completes, avoiding separate state-update and reduction passes. -- **Postprocessing and cache write.** One kernel applies RMSNorm, RoPE, the optional Hadamard transform, output quantization, and scatter into the compressed paged cache. - -**Kernel optimization.** The fused Compressor kernels support the different dtype and cache-layout requirements of both the main-attention Compressor and the Compressor inside the Indexer, including BF16, FP8, and MXFP4 paths. They accept BF16 projection output directly while retaining FP32 state updates and online-softmax accumulation where required, avoiding a full FP32 KV/score intermediate. The reduction kernels are specialized for the 4×-compression decode and MTP regime and the much longer 128×-compression prefill regime, with the latter also reducing newly arrived tokens while writing their paged state to avoid a later readback. - -##### mHC optimizations - -DeepSeek-V4 places mHC around both attention and MoE, making the boundary between consecutive sublayers the primary fusion opportunity. - -**Kernel fusion.** TensorRT LLM fuses the previous sublayer's post-mapping with the next sublayer's pre-mapping into shape-specialized CUDA kernels, with both two-kernel and all-in-one tactics for different token counts. The next sublayer's input RMSNorm is further folded into the fused mHC epilogue, eliminating a separate launch and an HBM round trip. Because mHC owns the residual connection at the attention-to-MoE boundary, the MoE path also skips separate residual handling. - -**Kernel optimization.** No single mHC kernel is optimal across the full range of token counts `M`. TensorRT LLM provides FMA implementations for small-`M` workloads and higher-throughput MMA implementations as `M` grows, with two-kernel and all-in-one variants, multiple tile configurations, and Split-K support. The autotuner evaluates the valid tactics for each workload and selects the best-performing implementation, while pruning choices that are not competitive in the corresponding `M` regime and caching the selected tactic for reuse. - -#### MoE optimizations - -Building on the routing design [described above](#beyond-attention-mhc-moe-and-speculative-decoding), TensorRT LLM optimizes DeepSeek-V4's MoE path through a custom router GEMM, the MXFP8 × MXFP4 DeepGEMM MegaMoE backend, streamlined input preparation, and ongoing NVFP4 MegaMoE integration. These efforts preserve the model's routing metadata, precision recipe, and checkpoint-defined `swiglu_limit` activation clamp. - -**MoE router GEMM optimization.** For the 256-expert Flash shape, the score-based router uses a custom GEMM when it receives one to 16 token rows. The kernel consumes BF16 activations and weights and produces FP32 logits directly, avoiding the full FP32 input and weight casts required by the original linear path. Unsupported shapes, including the 384-expert Pro router, continue to use cuBLAS. In the reported small-shape measurements, router latency decreased from approximately 8 µs to 3 µs. - -**MXFP8 × MXFP4 DeepGEMM MegaMoE.** For the W4A8 recipe with MXFP8 activations and MXFP4 weights, the DeepGEMM MegaMoE backend moves dispatch, the first expert GEMM, SwiGLU, the second expert GEMM, and combine into a fused-communication path that uses symmetric memory and in-kernel synchronization. In a 500-request DeepSeek-V4 Flash test on B200 with 1K-token inputs and outputs, TP4, EP4, and attention data parallelism, switching from the TensorRT LLM MoE backend to DeepGEMM MegaMoE increased throughput by 15.3% and reduced latency by 12.7%. TensorRT LLM also fuses the input preparation for this backend: instead of separately quantizing BF16 activations to MXFP8, converting selected expert indices, and copying activations, scales, indices, and routing weights into symmetric buffers, one CUDA kernel performs the quantization and writes all four outputs directly to their final buffers, reducing six GPU operations to one. No separate end-to-end gain is attributed to the preparation change. - -**NVFP4 MegaMoE (WIP).** For native NVFP4 checkpoints, TensorRT LLM is integrating the ported `Sm100MegaMoEKernel` as a CuTe DSL execution backend on the SM100f path. The integration covers BF16-to-NVFP4 input quantization, dispatch, the two expert GEMMs, SwiGLU, per-route combine writes with native NVFP4 weights, and a separate reduction of the Top-K route contributions. It also includes symmetric-memory exchange for multi-rank expert parallelism and workload-specific tactic selection through the TensorRT LLM autotuner. The NVFP4 and DeepGEMM MegaMoE paths have separate hardware, quantization, and topology requirements. This precision-specific backend remains under integration and validation. - -#### Runtime optimization - -After the major kernels are optimized, runtime overhead shifts to the boundaries between kernels, repeated metadata and data movement, and transient memory growth during long-context prefill. TensorRT LLM addresses these costs through earlier dependent launches, streamlined input preparation, and workload-aware memory control. - -**Programmatic Dependent Launch.** PDL allows a dependent consumer grid to launch before its producer grid fully retires, wait at an explicit dependency point, and continue as soon as the required producer state is ready. TensorRT LLM configures PDL globally for supported DeepGEMM operations and also applies it to the packed 1×128 FP8 quantization, MLA RoPE, sparse-index conversion, sparse FMHA, and fused mHC paths used by DeepSeek-V4. This shortens producer-to-consumer boundaries without changing the model's dependency graph. - -**Metadata preparation.** TensorRT LLM streamlines DeepSeek-V4 attention-metadata preparation by reusing tensors and consolidating the host- and device-side updates required for each step, reducing Python work, kernel launches, and data transfers. The cache manager also builds DeepSeek-V4 block tables on the GPU and uses a dedicated CUDA operator instead of compiling the earlier tensor implementation, reducing host-memory pressure while keeping block-table generation on the device. - -**SWA scratch reuse.** A long prefill step produces KV entries for every token in the input chunk, but sliding-window attention retains only the most recent window after the step. Among the new blocks allocated for the current chunk, blocks that will already be outside the final non-rewindable window are needed only by the currently executing layer. TensorRT LLM maps this scratch-eligible range through per-layer block tables to coalesced scratch subpages. Because these per-layer lifetimes do not overlap, the same physical scratch storage can be reused as execution advances across layers. Pre-existing history from earlier chunks remains in the normal KV cache and is never overwritten by scratch reuse. New blocks in the active window also remain in the normal cache, and the tail that MTP may rewind is excluded from scratch storage. As a result, the cache does not retain the full current prefill chunk independently for every compatible SWA layer, reducing long-context prefill memory consumption and leaving more capacity for larger chunks, longer prompts, or higher concurrency. - -
-
- SWA scratch reuse during long-context prefill. Scratch-eligible KV blocks produced by the current prefill chunk and falling outside the final retained window reuse the same physical storage across SWA layers, while blocks inside the sliding window remain in the normal KV cache. -
-
-

Figure 3. SWA scratch reuse during long-context prefill. Scratch-eligible blocks from the current prefill chunk reuse the same physical storage across SWA layers, while the final sliding-window state remains in the normal KV cache.

- -**Bounded Indexer workspace.** During long-context prefill, the Indexer's MQA-logits transient grows with both the query chunk and compressed history. TensorRT LLM controls this allocation at two levels. An outer heuristic reduces the configured prefill chunk when the compressed KV history crosses long-context thresholds. Within each chunk, query tiling imposes a hard per-call element budget so that only a bounded slice of the logits matrix is materialized at a time. Because logits and Top-K are computed independently for each query row, tiling preserves the exact selection result while enabling very long prompts without reducing the model-wide token budget. - -Together, these runtime optimizations help carry the preceding kernel gains through to the full model. Dependent work starts earlier, recurring host-side metadata work is reduced, and long-context memory follows a bounded working set instead of unchecked temporary growth. The next section, [**DeepSeek-V4 Performance Evolution**](#deepseek-v4-performance-evolution), shows how the kernel and runtime improvements accumulate at the full-model level. - -### DeepSeek-V4 Performance Evolution - -The preceding [**DeepSeek-V4 Performance Optimizations**](#deepseek-v4-performance-optimizations) section described the individual attention, Compressor, mHC, MoE, and runtime optimizations. Their real value, however, is whether they move the end-to-end serving frontier after the full system is re-tuned. We therefore tracked a sequence of complete TensorRT LLM Pareto sweeps rather than multiplying speedups from isolated operator benchmarks. Under the same DeepSeek-V4 Pro 8K/1K workload, the maximum observed output throughput increased from 984 to 1,618 tokens/s/GPU between the first validated functional baseline and the latest fully refreshed May 28 snapshot, a **64.5% improvement**. Additional optimizations have since been integrated into the newer DeepSeek-V4 stack, but have not yet been included in a new full Pareto sweep. - -#### Experimental setup - -The primary evaluation uses the InferenceX fixed-length 8K/1K workload, with exactly 8K input tokens and 1K output tokens per request. This shape stresses both sides of disaggregated serving: the context workers must process a substantial prompt, while the generation workers execute enough decode steps for sparse attention, MTP, and MoE efficiency to materially affect the result. - -| Item | Configuration | -| :--- | :--- | -| Model | DeepSeek-V4 Pro, 1.6T total parameters | -| Hardware | NVIDIA GB300 | -| Precision | Mixed-precision MXFP4 Instruct recipe: MXFP8 activations × MXFP4 weights for routed-expert GEMMs, with component-specific precision elsewhere | -| Workload | InferenceX, 8K input tokens and 1K output tokens per request | -| Serving mode | End-to-end disaggregated serving with independently sized context and generation workers | -| Sweep | TEP4/TEP8 for the latency-oriented end; DEP8/DEP16/DEP32 for the throughput-oriented end; MTP-3 for most latency and mid-frontier points and MTP-1 at the highest-concurrency points | -| Metrics | Output throughput per concurrent request stream (tokens/s/user) and output throughput per GPU (tokens/s/GPU) | - -Each curve in Figure 4 is a newly measured system Pareto sweep. A point can therefore change concurrency, MTP depth, parallel strategy, expert placement, and the context-to-generation resource ratio. The curve movement is the meaningful comparison; the peak values should not be interpreted as a component-by-component A/B test at an identical configuration. - -![Four DeepSeek-V4 Pro 8K/1K disaggregated-serving Pareto curves, showing peak output throughput increasing from 984 to 1,618 tokens/s/GPU](../media/tech_blog26_dsv4_performance_evolution_8k1k.png) - -

Figure 4. DeepSeek-V4 Pro 8K/1K disaggregated serving on GB300, using the mixed-precision MXFP4 Instruct recipe with MXFP8 × MXFP4 routed-expert GEMMs. Each line is a complete TensorRT LLM Pareto sweep from that milestone.

- -#### From Baseline to the Latest Measured Curve - -The four curves capture three distinct phases of optimization. The table reports the highest measured tokens/s/GPU on each sweep; because the maximizing configuration changes as the frontier moves, these values summarize system evolution rather than a strict single-configuration waterfall. - -| Milestone | Peak output throughput (tokens/s/GPU) | Gain from previous | Gain from baseline | Representative changes in this stage | -| :--- | ---: | ---: | ---: | :--- | -| April 28: validated baseline | 984 | — | — | Validated end-to-end disaggregated stack under the common model and workload definition | -| May 8: Top-K and Compressor | 1,167 | +18.6% | +18.6% | Small-batch exact Top-K, BF16 Compressor input and fusion, and disaggregated memory/correctness fixes | -| May 18: MegaMoE and EPLB | 1,505 | +29.0% | +53.0% | MegaMoE, EPLB-enabled DEP placements, fused mHC and RMSNorm, and expanded multi-stream attention | -| May 28: attention and runtime refinements | 1,618 | +7.5% | +64.5% | Deeper attention overlap, the optimized inverse-RoPE and FP8 `o_a_proj` path, PDL, and GPU-side scale-buffer and block-table cleanup | - -**The first measured curve establishes the validated functional baseline.** It represents the complete execution path after functional bring-up: hybrid SWA/CSA/HCA attention, persistent Compressor state, MTP, KV Cache Manager V2, and disaggregated cache transfer. We treat all work required to reach this point as part of the baseline rather than assigning it a performance gain. Every subsequent curve uses the same model, precision recipe, and fixed-shape workload definition. The reported evolution therefore measures optimization of a working system rather than the addition of missing model features. - -**Exact Indexer Top-K and Compressor improvements move the May 8 frontier.** The small-batch multi-CTA path increased GPU utilization for exact Top-K, while BF16 Compressor input and additional Compressor fusion removed conversion, launch, and intermediate-buffer overhead. Runtime fixes were equally important: correcting sliding-window over-allocation improved usable generation-side KV capacity, while the launcher fix ensured that the configured NUMA policy was applied to multi-process disaggregated runs. Together, the full re-tuned sweep raised the peak by 18.6%, from 984 to 1,167 tokens/s/GPU. - -**MegaMoE and EPLB drive the mid-May step.** The largest movement in the system curve came between May 8 and May 18. MegaMoE fused expert dispatch, the two expert GEMMs, activation, and combine into a communication-aware path using symmetric memory and in-kernel synchronization. At the throughput-oriented end of the curve, the May 18 sweep also moved to EPLB-enabled DEP placements to improve expert balance across ranks. In parallel, mHC absorbed the adjacent RMSNorm, and the Indexer and attention paths gained expanded dependency-aware overlap. Together, these changes produced a 29.0% stage-over-stage increase and a 53.0% increase over the initial baseline. Because the maximizing configuration changed across the complete sweeps, this movement is not a single-configuration decomposition of the individual optimizations. - -**Attention overlap and runtime cleanup extend the late-May frontier.** By May 28, deeper Indexer, Compressor, and Q-path overlap reduced exposed attention work. PDL, producer-side FP8 scale-buffer initialization, the optimized inverse-RoPE and FP8 `o_a_proj` input path, and cheaper GPU-side block-table preparation removed additional synchronization, launch, and data-movement overhead. The May 28 curve is roughly 7–8% above May 18 across most throughput-oriented DEP points, lifting the observed peak to 1,618 tokens/s/GPU. This broad movement is more informative than any single microbenchmark: it shows that the optimizations survived full-model scheduling, cache management, and context/generation rate matching. - -May 28 is the latest measured curve, not the current performance ceiling. The newer DeepSeek-V4 stack subsequently integrated sparse-MLA softmax and V-load improvements, fused MegaMoE input preparation, the optimized 128× Compressor prefill reduction, and later attention-epilogue fusion. These changes target attention, MoE preparation, and Compressor overhead, but the complete GB300 8K/1K Pareto sweep has not been rerun with the newer stack. None of their gains is included in the reported 64.5% improvement, and we do not extrapolate a new end-to-end number. We expect a refreshed curve to move further once it is measured under the same methodology. - -The latest measured fixed-shape InferenceX curve establishes the clean model/runtime baseline for [Part II](#part-ii-agentic-workload-optimizations). Agentic workloads change the optimization boundary: dynamic multi-turn prompts, very high KV reuse, conversation locality, routing, and serialization become part of the critical path. The next section, [**AgentPerf Workflow**](#agentperf-workflow), introduces that wider optimization boundary without re-counting the model-specific gains above. - - -## Part II. Agentic-Workload Optimizations - -### AgentPerf Workflow - -[AA-AgentPerf](https://artificialanalysis.ai/methodology/agentperf) measures how many active coding agents an inference deployment can support while meeting a model-specific service level objective (SLO). This [NVIDIA Developer Blog](https://developer.nvidia.com/blog/nvidia-achieves-leading-agentic-coding-performance-on-first-agentic-ai-benchmark/) introduces the benchmark and explains why prerecorded trajectories are needed to represent agentic inference. Unlike the fixed-shape InferenceX workload used in Part I, AA-AgentPerf replays multi-turn coding sessions with interleaved reasoning and tool calls. Each simulated agent advances sequentially and waits for the current response and any client-simulated tool time before sending its next request. - -| Property | Fixed-shape InferenceX | AA-AgentPerf | -| :--- | :--- | :--- | -| Unit of load | Independent request | Active agent following a trajectory | -| Request shape | Fixed input and output lengths | Growing input context and variable output lengths | -| Inter-request dependency | Requests are independent | Each turn waits for the response and any client-simulated tool time | -| Prefix reuse | No accumulated cross-turn history | Accumulated history enables reuse across turns | -| Objective | Throughput and latency at a fixed shape | Highest concurrency satisfying model-specific SLO thresholds | - -The following figure summarizes the closed-loop request path for aggregated and disaggregated serving. - -
-
- AA-AgentPerf closed-loop workflow. A simulated agent sends the next recorded turn with its accumulated conversation. The inference deployment routes the request, reuses matching KV blocks, prefills the uncached suffix, and streams the generated response. In disaggregated serving, the generation side retrieves the KV cache using context metadata. The client drains the complete response, simulates tool time when needed, advances the trajectory, and sends the next turn only after the current cycle completes. -
-
-

Figure 5. AA-AgentPerf closed-loop request workflow. Repeated conversation prefixes can be reused through the KV cache. Aggregated serving performs prefill and generation on the same worker. In disaggregated serving, context metadata allows the generation worker to retrieve the KV cache and continue decoding. Tool time is simulated by the client and consumes no LLM compute.

- -This closed loop makes the complete request cycle part of every agent's progress. Repeated prefixes make KV-cache reuse important because later turns can avoid recomputing accumulated history. Growing contexts and variable output lengths also exercise cache capacity and scheduling differently from fixed-shape requests. Simulated tool delays introduce idle periods between turns, while concurrent agents maintain sustained load on the deployment. - -AA-AgentPerf reports the highest concurrency that satisfies both the P25 of per-request output speed and the P95 of per-request time to first token (TTFT). Because the benchmark is continuously updated, this blog uses the [AA methodology](https://artificialanalysis.ai/methodology/agentperf) current at publication for the exact thresholds. We refer to the current DeepSeek-V4 Pro SLO #1 and SLO #2 tiers as **SLO20** and **SLO60**. SLO20 requires at least 20 tokens/s and P95 TTFT at most 10 seconds. SLO60 requires at least 60 tokens/s and P95 TTFT at most 5 seconds. SLO20 emphasizes serving capacity, while SLO60 preserves a faster interactive experience. - -For engineering comparisons, we use concurrency per GPU (CPG). It is the supported concurrency divided by the total number of GPUs in the deployment, including both context and generation GPUs. - -### Lessons from DeepSeek-V3.2 Agentic Workload Optimization - -Before DeepSeek-V4, we optimized DeepSeek-V3.2 for long, multi-turn agentic workloads in both aggregated and disaggregated serving. The durable value of that work was not a particular benchmark result or deployment configuration. It was a set of system-level rules that guided the DeepSeek-V4 effort: - -1. **High reuse creates a different bottleneck from raw GPU throughput.** High prefix reuse can leave only a small uncached suffix to compute while the accumulated conversation still occupies substantial KV-cache capacity. Scheduling and capacity planning should therefore consider the remaining compute tokens and resident KV footprint rather than raw prompt length alone. -2. **Amortize host overhead without pausing generation.** Two complementary approaches addressed this overhead. First, CUDA Graph capture and kernel fusion reduced recurring host and launch work. **Piecewise CUDA Graph capture** covered graphable regions in prefill and mixed prefill-decode iterations when the full iteration could not be captured as one graph. Generation-only iterations continued to use the regular monolithic CUDA Graph path. Second, context-only **Delay batching** briefly held small context requests so newly arriving context requests could join a larger batch. Generation requests continued to run during this wait. -3. **Treat routing as a trajectory-level balance between locality and load.** Later turns benefit from returning to the workers and ranks that hold their conversation prefix. That affinity must be balanced against load so that preserving reuse does not create persistent rank imbalance. -4. **Measure disaggregated serving end to end.** Front-end preprocessing and KV-cache movement can limit throughput even when context and generation GPUs appear underutilized. Asynchronous preprocessing, parallel data movement, and per-stage observability are therefore part of serving performance. - -DeepSeek-V4 changed the model path, cache layout, and serving control plane, so the earlier configuration could not simply be reused. We carried forward these rules rather than the exact parallel strategy, cache settings, batching parameters, or transfer settings. The following section applies them to DeepSeek-V4 routing, KV lifecycle, host overhead, and scheduling. - -### End-to-End Optimizations for Agentic Serving - -Starting from the final model/runtime stack in [Part I](#part-i-deepseek-v4-model-support-and-optimizations), this section applies the measurement methodology and system-level principles learned from [DeepSeek-V3.2](#lessons-from-deepseek-v32-agentic-workload-optimization) to DeepSeek-V4's AgentPerf-specific request path, without counting the model-specific gains again. In this workload, a routing miss can force a long conversation prefix to be recomputed, erasing gains from faster kernels. Hashing that same prefix token by token in Python can also serialize the orchestrator's request path. These bottlenecks expand the optimization boundary beyond GPU kernels. The remaining work spans three coupled areas: preserving locality through routing, keeping reusable KV blocks resident through lifecycle and capacity management, and removing overhead across the worker host paths, scheduler, orchestrator, and CTX-to-GEN protocol once reuse reduces the GPU work per turn. - -#### Preserve locality with two-level routing - -The DeepSeek-V3.2 study showed the central routing tradeoff: affinity without load control creates imbalance, while load balancing that overrides affinity destroys reuse. DeepSeek-V4 applies that lesson through two consecutive placement decisions. The front-end orchestrator first selects a context (CTX) server instance. The attention data-parallel (ADP) router then selects a rank within that instance. At either level, the router can use an explicit conversation identifier or infer locality from KV-cache contents. Reuse is preserved only when both decisions return a later turn to the server and rank that own its KV blocks. Instance-level affinity cannot compensate for a rank-level miss, and the correct rank is unreachable after the wrong server has been selected. - -
-
- Two-level routing for context locality in disaggregated serving. The front-end orchestrator first selects a context server, then the selected server's attention data-parallel router selects a rank. At each level, the router can use either conversation-aware affinity or KV-cache-aware scoring. Full conversation-prefix reuse requires both decisions to return to the context server and rank that own the corresponding KV blocks. The selected context rank transfers the KV cache to a generation server for decoding. -
-
-

Figure 6. Two-level routing for context locality in disaggregated serving. The front-end orchestrator first selects a CTX server, and that server's ADP router then selects a rank. Conversation-aware routing uses explicit conversation affinity, while KV-cache-aware routing infers placement from reusable prefix blocks and load. Full conversation-prefix reuse requires both decisions to return the request to the CTX server and ADP rank that hold the corresponding KV blocks.

- -##### Instance-level routing across context servers - -The first routing level distributes conversations across CTX server instances. It must preserve locality for later turns while balancing new conversations across servers. - -**Conversation-aware routing.** When the client provides a stable conversation identifier, the orchestrator assigns the first turn to the least-loaded CTX server, using round-robin among equally loaded candidates, and records the binding. For each later turn, the orchestrator uses that binding to send the request back to the same server, preserving reuse of the conversation prefix cached there as long as the server remains available. The affinity decision then becomes a session-table lookup, avoiding full-prompt tokenization and block-hash computation on the front-end event loop. - -**KV-cache-aware routing.** When a reliable conversation identifier is unavailable, the orchestrator infers locality by tokenizing the prompt, deriving its block keys, estimating the reusable prefix on each CTX server, and balancing reuse against current load. We made three improvements to keep this path efficient and stable. First, incremental tokenization reuses the token IDs for the stable prompt prefix and encodes only the appended suffix. The orchestrator passes those token IDs to the worker and applies the same tool definitions and chat-template arguments, avoiding repeated tokenization while ensuring that both sides derive the same block keys. Second, we refined candidate scoring, which selects a CTX server by weighing its reusable prefix against its current load. Instead of normalizing the cache match by the full prompt length, the new score uses the absolute number of matched blocks and a separate active-request penalty. This prevents the long system and tool prefix shared across requests from diluting the score advantage contributed by conversation-specific blocks. Third, we strengthened affinity when temporary cache eviction weakens the cache-match signal. A bounded mapping derived from stable early conversation content remembers the selected CTX server and continues to direct later turns to it even if eviction temporarily lowers the server's cache-match score. - -Cache-aware routing also requires the orchestrator to maintain a compatible and sufficiently current view of each worker's cache. Three design choices make that view both reliable and inexpensive to maintain. First, the orchestrator and workers use the same hash algorithm and cache namespace. An optional cache salt acts as a namespace identifier, allowing KV reuse only between requests that carry the same value. Second, the orchestrator adopts the worker's block size by default. When it tracks cache at a finer supported granularity, completed-request backfill adds the routed block keys to the selected server's cache view after successful execution, which lets the orchestrator track newly cached prefixes at its own granularity without translating between incompatible event boundaries. Third, cache metadata updates are batched, and worker events are refreshed in the background to keep this maintenance off the routing critical path. Together, these changes keep cache tracking useful without turning it into a new routing bottleneck. - -##### Rank-level routing within a context server - -After a CTX server has been selected, the second routing level places the request on one of its ADP ranks. It must preserve the exact rank that owns the reusable prefix while keeping new conversations distributed across the instance. - -**Conversation-aware routing.** The ADP router records a bounded `conversation ID → rank` mapping. It assigns the first turn of each new conversation by round-robin among ranks below a loose fair-share target, then pins later turns to the recorded rank. A returning conversation may exceed that soft target because moving it would discard rank-local reuse. Affinity is relaxed only when the assigned rank reaches the hard active-request limit. An overflow request does not overwrite the original binding, which allows a later turn to return when the rank has capacity again. - -**KV-cache-aware routing.** Without a reliable conversation ID, the ADP router must infer rank locality from the KV blocks that are actually present. For each new request, every rank probes its local radix tree to measure the reusable prefix without reserving any cache blocks. The probe uses the request's cache salt so that it searches the same cache namespace as the KV cache manager. The ranks then exchange their prefix-match lengths and current loads. For each candidate rank, the router estimates the remaining prefill work as the input length minus the matched prefix, then adds a normalized load penalty. A meaningful prefix match therefore favors the rank that can reuse it, while a weak match caused only by the shared system and tool scaffold is ignored in favor of load balancing. A loose fair-share cap provides a second guard against concentrating too many requests on a few warm ranks. When several related requests arrive in the same scheduling batch, the router groups them by their early prefix and places longer requests first. After each assignment, it updates the selected rank's load using only the tokens that still require computation. The next request is therefore routed against the work already placed in that batch rather than a stale load snapshot. - -**Protect rank balance.** A new conversation contains the shared system and tool scaffold but little conversation-specific history. Without explicit cold-start handling, the first few ranks that cache this common prefix can attract more new conversations, while the remaining ranks stay cold. Conversation-aware routing prevents this feedback loop by distributing first turns with round-robin before pinning later turns. KV-cache-aware routing can instead use an optional warmup phase that sends initial requests to ranks not yet selected, allowing every rank to cache the shared prefix before normal scoring begins. - -##### Routing policy selection - -Our evaluation results show that, with completed-request backfill, the cache-aware path achieved a cache hit rate comparable to conversation-aware routing, but required additional preprocessing and cache tracking. The two approaches therefore offer different ways to preserve locality: one uses explicit conversation metadata, while the other infers locality from cache contents. - -#### KV cache reuse optimizations - -Two-level routing keeps successive turns of the same conversation on the CTX server and ADP rank that hold the reusable prefix. Placement alone is not enough. The prefix must also remain resident until the next turn arrives. Analysis of early DeepSeek-V4 AgentPerf runs showed substantial optimization headroom in cache memory management. DeepSeek-V4 divides cache memory among the sliding-window cache and several persistent compressed caches with different storage costs and demand profiles, so reuse depends on both the cache lifecycle and how memory is budgeted across pools. We first added per-pool cache metrics to locate pressure, then used those signals to optimize the sliding-window cache lifecycle and tune cache pool ratios. - -**Add per-pool cache metrics.** An overall cache hit rate shows how much reuse was achieved, but it does not reveal which pool was constrained or whether the cause was sustained pressure or a short-lived capacity spike. We added per-pool telemetry for occupancy, allocation and commit volume, cached KV length, and data movement between HBM and host memory. Reuse counters distinguish fully reused, partially reused, and missed blocks. Peak watermarks and complete iteration histories capture transient pressure that periodic summaries can miss, while memory probes expose the HBM headroom before and after model loading. Host-tier gauges further distinguish evictable blocks from blocks dropped before reuse. Together, these signals identify the constrained pool, distinguish incorrect lifecycle retention from insufficient pool capacity, and provide the evidence for the next two optimizations. - -**Optimize the sliding-window cache lifecycle.** As described in [**Cache management and runtime features**](#cache-management-and-runtime-features), DeepSeek-V4's sliding-window cache includes SWA KV and short-lived Compressor KV/score state. These buffers should remain bounded by their active windows rather than grow with the full prompt. The previous reuse policy retained SWA KV across the full prompt, even though a later agent turn needs only the blocks within the active window. We introduced the `per_request` policy to select the sliding-window state needed by the next turn when the current prefill request completes, allowing older pages to be recycled. The same retention issue appeared at the CTX-to-GEN handoff in disaggregated serving. Before this optimization, GEN allocated sliding-window cache for the full transferred prompt, including blocks outside the active window. We changed GEN initialization to allocate cache only for blocks within the active window. Together, these changes keep the sliding-window cache bounded during prefill and disaggregated handoff. Persistent compressed caches still retain the long-term prefix and continue to grow with sequence length. - -**Tune cache pool ratios.** Correcting the cache lifecycle removes stale allocations, but reusable prefixes can still be evicted when the available HBM is divided poorly across pools. The two cache categories respond differently to workload shape. Sliding-window cache pressure is driven by the active-request count, while persistent compressed caches grow with retained sequence length and the number of concurrent conversations. Each allocation must be satisfied by its own pool group, so free pages in one group cannot cover a shortage in another. A split tuned for a different request-length distribution or serving role can therefore exhaust one pool and evict reusable blocks even while another pool has unused capacity. To address this imbalance, we introduced configurable pool-ratio controls for DeepSeek-V4. On CTX servers, prefix reuse makes pool demand highly dependent on the workload's prefix-reuse pattern, which the runtime cannot infer reliably. Users should therefore tune `pool_ratio` manually using the per-pool metrics described above. On GEN servers, users can set `avg_seq_len` to the workload's average total sequence length, allowing KV Cache Manager V2 to derive the initial pool ratio automatically. These controls direct capacity to the pools that need it and avoid evicting reusable history while memory remains unused elsewhere. An opt-in beta rebalancer can also adjust the split from runtime statistics. It is disabled by default and currently supports a narrower set of aggregated-serving configurations. Extending this mechanism to broader deployments remains ongoing work. - -**Host offloading and incremental transfer.** We also evaluated host KV offloading and incremental CTX-to-GEN transfer. Compared with models that retain full K/V history for every token, DeepSeek-V4 stores long-term history in compressed attention and Indexer caches while keeping raw-token KV and Compressor state within bounded sliding windows. Its KV-cache footprint per input token is therefore relatively small. This compact representation keeps more reusable KV blocks resident, leaving less additional reuse for host offloading to recover. It also reduces the CTX-to-GEN payload, limiting the data movement that incremental transfer can overlap. Neither mechanism therefore produced a meaningful end-to-end improvement for this AgentPerf configuration. The evaluation is still useful for workloads with larger KV-cache footprints or higher transfer volumes. - -#### Optimize scheduling and remove end-to-end overhead - -Routing and cache management make later turns with reusable prefixes much cheaper on the GPU. That saving reaches the user only if the rest of the serving path also avoids processing the full long prompt again. Profiling showed substantial host work during CTX prefill and a persistent gap between GEN inter-token latency and the speed-of-light (SOL) estimate. Long token arrays were still being hashed, copied, tokenized, serialized, and broadcast at several boundaries. We addressed this gap inside both the workers and the serving control plane. First, we removed host work from the CTX and GEN critical paths. We then optimized scheduling, orchestration, and protocol handling. - -##### Optimize host overhead for CTX and GEN - -Effective prefix reuse reduces the GPU work to the uncached suffix, but the surrounding host path can still scale with the full prompt. This is especially costly when the work runs in Python or is repeated on every distributed rank. We therefore moved per-token loops out of Python where possible and skipped host operations that could not produce useful GPU work. - -**CTX host path.** Profiling showed that CTX was paying host work at the wrong granularity. Block-key construction paid a conversion and SHA-256 update for every token, turning a long prompt into many small operations before prefill could begin. Distributed coordination was paid once per iteration, even when no new request was available. Those empty iterations could still enter prefix-gather and request-broadcast collectives. The first cost grew with the full prompt, while the second could create host bubbles without producing any GPU work. We changed each operation to follow the unit of useful work. Block-key construction now packs one block of tokens into a contiguous byte buffer and updates the hash once per block. All ranks perform a lightweight request-count probe before entering the larger collectives. If no new request exists, they take the same fast path and skip both prefix gathering and request broadcast. - -**GEN host path.** On GEN, profiling did not point to one dominant function. Instead, host time accumulated through three prompt-proportional costs as each long-prompt request crossed the CTX-to-GEN handoff, entered GEN, and was distributed across the ranks. First, GEN could repeat tokenization already completed by CTX. Second, request admission and routing repeatedly copied or materialized the full token list. Third, interprocess request transfer serialized token IDs as individual Python objects before rebuilding the full list on the receiving side. The latter two costs occurred on request-ingress or executor-loop paths while holding the Python GIL, so they could delay the next generation iteration. To reduce this accumulated host time, we introduced a set of targeted optimizations, each addressing one of these costs directly. (a) CTX now returns the prompt token IDs that it already computed, allowing GEN to bypass tokenization. (b) During admission, enqueue no longer deep-copies the token list, while routing obtains the token count without materializing the full list. (c) Serialization and C++ request construction use a compact token representation, avoiding per-token Python objects and full-list reconstruction. - -##### Optimize scheduling, orchestration, and protocol handling - -After reducing host work inside CTX and GEN, latency could still accumulate before a request entered a worker and while data moved between workers. We treated this serving control plane as three related problems. Scheduling determines which ready request runs next. The orchestrator performs routing and relays data between CTX and GEN. The protocol must minimize redundant payload while preserving request identity when retries occur. - -**Scheduling.** Under high concurrency, a GEN worker may have to choose between newly admitted requests waiting for their first token and requests that have already begun decoding. When the batch or token budget is tight, placing both groups in the same queue can leave a new request behind many ongoing conversations for several scheduling iterations. We added an opt-in policy for disaggregated GEN that gives first-token requests priority while preserving FIFO order within both groups. This can move a newly admitted request into an earlier feasible batch and reduce the queueing delay before its first decode step without adding GPU work. The policy is designed to improve first-token responsiveness and tail-latency SLO compliance rather than raw throughput. - -**Orchestrator.** Profiling exposed two sources of per-request overhead. First, **native block-key hashing** removed the routing hot spot. The instance-level KV router originally hashed every token ID in a Python loop on its single asyncio thread. This held the Global Interpreter Lock (GIL) and serialized concurrent requests before they could reach CTX. Replacing the loop with a native C++ block-key hasher preserved the cache-key format while moving prompt-length hashing out of the Python event loop. Second, **efficient relay serialization** reduced request-body processing when a request crossed the HTTP boundaries to CTX and GEN. The default path previously materialized a JSON-compatible Python object containing the full prompt-token list, after which the HTTP client traversed that object again to encode the request body. We changed this path to use `pydantic-core` to emit the JSON body directly. For deployments where request-body processing remains a bottleneck, an opt-in `msgspec` MessagePack transport provides an alternative internal orchestrator-to-worker path that removes JSON encoding and parsing from this hop. The external OpenAI API and the default direct-JSON path remain unchanged. - -**CTX-to-GEN protocol.** The CTX-to-GEN protocol had two issues: it could carry more request state than GEN needed, and a retried request could leave CTX and GEN with inconsistent identifiers. In supported text-only, non-Harmony deployments, an opt-in path removes earlier chat history from the GEN request while preserving the prompt token IDs, final message, tool definitions, and generation settings that GEN still needs. This reduces copying, serialization, and transfer without changing the request semantics supported by that path. We also hardened the retry path so that known transient connection failures do not leave CTX and GEN with inconsistent request identifiers. Together, these changes reduce protocol overhead while preserving consistency across the disaggregated handoff. - -Across the host and serving-control-plane paths, these changes reduce prompt-proportional processing, redundant serialization, and avoidable synchronization between GPU iterations. Together with routing and cache management, they allow prefix reuse to translate into end-to-end performance. The next section, [**AgentPerf Results and External Validation**](#agentperf-results-and-external-validation), presents the combined end-to-end results under frozen configurations and evaluates them against the SLO20 and SLO60 objectives. - -### AgentPerf Results and External Validation - -The previous section described how routing, KV-cache policy, host-path cleanup, scheduling, and orchestration removed distinct end-to-end bottlenecks. This section presents the DeepSeek-V4 Pro AgentPerf evaluation setup and the results reported by Artificial Analysis. We keep these agent-workload results separate from the model-specific results in [Part I](#part-i-deepseek-v4-model-support-and-optimizations) and report the best-performing configurations we developed for those AA-AgentPerf measurements. - -The GB300 NVL72 configurations served DeepSeek-V4 Pro at maximum thinking effort using TensorRT LLM with disaggregated serving. Several context (CTX) workers fed one generation (GEN) worker through NIXL KV-cache transfer. Each CTX worker used four GPUs with attention data parallelism and MoE expert parallelism. - -To select the end-to-end topology, we followed the speed-of-light (SOL) rate-matching workflow introduced in the [TensorRT LLM disaggregated serving tech blog](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md#measurement-methodology). We first measured CTX request throughput across candidate parallel mappings and batch settings that could satisfy the TTFT target, and independently measured GEN output throughput and per-agent output speed across mappings, batch sizes, and concurrency levels. We then matched the aggregate service rates of the two tiers to construct an idealized SOL Pareto curve. The final SLO20 configuration used DEP8 with a larger GEN batch to maximize capacity. The final SLO60 configuration used DEP16 with a smaller GEN batch to preserve higher per-agent decode speed. SOL assumes perfect rate matching and excludes practical end-to-end costs such as KV-cache transfer, routing, host work, and workload burstiness, so it served as a configuration-selection reference rather than a reported AgentPerf result. - -The same rate-matching calculation provided the starting `xPyD` ratio, where `xPyD` denotes `x` CTX/prefill instances feeding `y` GEN/decode instances. For each selected GEN operating point, we converted its output-token throughput into a request-equivalent service rate using the workload's output-length distribution, then compared it with the measured per-instance CTX request rate to estimate the ideal number of CTX instances. Because this ratio can be fractional while a deployment requires whole workers, we evaluated the neighboring integer `xPyD` topologies that fit the GPU budget. We selected the topology that provided the best end-to-end balance while satisfying both the output-speed and TTFT requirements, then swept concurrency to find its maximum passing load. - -The configurations used DEP4 CTX workers with `max_batch_size: 128` and the TensorRT LLM MoE backend. The capacity-oriented SLO20 configuration used a DEP8 GEN worker with `max_batch_size: 192`. The higher-speed SLO60 configuration used a DEP16 GEN worker with `max_batch_size: 32` and an offline EPLB map. Both GEN workers used the DeepGEMM MegaMoE backend. Both deployments used the MXFP4 recipe, MTP-3, FP8 KV cache, conversation-affinity routing at the orchestrator and ADP-rank levels, and heuristic GVR Top-K. - -As defined in [**AgentPerf Workflow**](#agentperf-workflow), we use concurrency per GPU (CPG) to compare deployment configurations. The GPU count here includes both the CTX and GEN tiers, and only points that pass the target SLO are reported. The Artificial Analysis article, [First results from AA-AgentPerf: the hardware benchmark for the agent era](https://artificialanalysis.ai/articles/aa-agentperf/), reports the GB300 configurations provided by NVIDIA and run and verified by Artificial Analysis. These results validate the complete workload path rather than an isolated TensorRT LLM microbenchmark. Artificial Analysis replayed the agent trajectories and applied its SLO search and steady-state methodology. The associated [serving-configuration browser](https://artificialanalysis.ai/benchmarks/hardware/configs) discloses the precision, topology, worker YAML, routing mode, MTP setup, cache policy, transfer backend, and launch commands. - -The GB300 results reached 57.5 CPG at SLO20 and 19.2 CPG at SLO60: - -| Target | CTX/GEN topology | Passing concurrency | GPUs used | CPG | Provenance | -| :--- | :--- | ---: | ---: | ---: | :--- | -| SLO20 | 6 × CTX DEP4 + 1 × GEN DEP8 | 1,840 | 32 | 57.5 | NVIDIA configuration, run and verified by AA | -| SLO60 | 6 × CTX DEP4 + 1 × GEN DEP16 | 768 | 40 | 19.2 | NVIDIA configuration, run and verified by AA | - -These optimizations are now also available in [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) for readers looking to deploy in production. - -## Reproduction and Future Work - -### How to reproduce - -TensorRT LLM already provides the common operational steps for launching disaggregated serving. Follow the [disaggregated serving tech blog](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog05_Disaggregated_Serving_in_TensorRT-LLM.md) and the [disaggregated serving guide](https://nvidia.github.io/TensorRT-LLM/features/disagg-serving.html) for environment setup, CTX and GEN worker startup, NIXL KV-cache transfer, and orchestrator launch. Both result sets below build on this foundation but use different workload-specific references. The InferenceX path uses the linked GB300 recipes to reproduce the fixed-shape 8K/1K Pareto sweeps. The AA-AgentPerf path starts from the current TensorRT LLM `main` branch to evaluate the latest optimized stack and uses the linked AA configuration entries and methodology for the submitted topology, settings, and SLO evaluation. - -**InferenceX 8K/1K.** Reproduce the fixed-shape results from [**DeepSeek-V4 Performance Evolution**](#deepseek-v4-performance-evolution) with the [GB300 MXFP4 ISL8K/OSL1K MTP recipes](https://github.com/NVIDIA/srt-slurm/tree/sa-submission-q2-2026/recipes/DeepSeek-V4-Pro/disagg/trtllm_dynamo/gb300_mxfp4/ISL8K_OSL1K/MTP). The directory contains the CTX/GEN topology sweep, MTP settings, benchmark concurrencies, and worker configurations used for the 8K/1K measurements. - -**AA-AgentPerf.** For the results reported above, use the corresponding entries in the [Artificial Analysis configuration browser](https://artificialanalysis.ai/benchmarks/hardware/configs). The 57.5-CPG SLO20 result uses the [GB300 DEP8 configuration](https://artificialanalysis.ai/benchmarks/hardware/configs#config-60987f21-a7db-4a8e-8d3a-810a7527de2f), while the 19.2-CPG SLO60 result uses the [GB300 DEP16 configuration](https://artificialanalysis.ai/benchmarks/hardware/configs#config-8e07c398-605a-457f-96c4-bb002691de79). Each entry provides the CTX and GEN worker YAML, disaggregated server configuration, topology, routing settings, and launch commands needed to reproduce the deployment setup. Use the [AA-AgentPerf methodology](https://artificialanalysis.ai/methodology/agentperf) for the benchmark procedure and SLO evaluation. - -### Future Work - -Our next phase continues to optimize DeepSeek-V4 as a complete serving system: improve the low-precision GPU execution path, reduce host and control-plane overhead, broaden hardware coverage, and expand speculative decoding support. These directions need to advance together, because faster model execution raises end-to-end capacity only when cache management, routing, and orchestration can keep the GPUs supplied efficiently. The following work is already underway. - -**Advance the NVFP4 execution path.** We are moving the performance path to the NVFP4 DeepSeek-V4 checkpoint, continuing to tune the NVFP4 MegaMoE backend, and evaluating a more aggressive quantization recipe that applies NVFP4 to additional linear layers. This work treats the checkpoint, kernel coverage, and quantization recipe as one end-to-end path. - -**Fuse more small kernels.** We will continue combining adjacent small operations to reduce intermediate memory traffic and the number of kernel launches. These fusions target both GPU execution time and the host launch overhead that remains visible in small effective batches. - -**Move the KV Cache Manager V2 hot path to C++.** KV Cache Manager V2 provides the flexible multi-pool model required by DeepSeek-V4, but its performance-critical control path is still primarily implemented in Python. Migrating this path to C++ will reduce GIL contention and recurring host overhead, which become especially visible when high cache reuse leaves only a small amount of GPU work per turn. - -**Optimize and scale the serving control plane.** Conversation-aware and KV-aware routing still have room to reduce TTFT and CPU overhead while preserving cache locality and balanced placement. At larger deployment scales, a multiprocess orchestrator with a coordinator-and-worker architecture is being developed to address the point at which a single orchestrator becomes the bottleneck. We will optimize these paths as one control plane, resolve the current multiprocess TTFT issues, and validate the complete design under high-concurrency, multi-turn workloads. - -**Add DeepSeek-V4 DSpark support.** We are integrating the DSpark speculative-decoding path for DeepSeek-V4 and validating it against the MTP baseline. The remaining work includes measuring acceptance behavior, tuning the runtime configuration, and quantifying the resulting end-to-end gain. - -**Extend DeepSeek-V4 support to Hopper.** The current performance work is centered on Blackwell. We are extending the model and runtime paths required for complete DeepSeek-V4 execution on Hopper, with platform-appropriate kernels and precision support. - -## Conclusion and Acknowledgments - -DeepSeek-V4 optimization in TensorRT LLM is an end-to-end systems effort. Building a production-ready model stack required correctness across hybrid attention, mHC, MTP, MoE, quantization, cache management, and disaggregated execution. Moving the performance frontier then required coordinated work from GPU kernels and execution overlap through KV reuse, host efficiency, routing, orchestration, configuration tuning, and benchmark methodology. The result is a stack that improves both fixed-shape inference and long-running agentic serving, demonstrating that delivered performance comes from optimizing the model and the serving system together. - -This work is the result of close collaboration across many teams and every layer of the inference stack. We sincerely thank everyone who contributed to model bring-up, accuracy validation, benchmark optimization, and infrastructure support. The results shared here came from repeated cycles of analysis, implementation, profiling, debugging, validation, and configuration tuning. We are grateful for the expertise, persistence, and teamwork that turned DeepSeek-V4 into a production-ready, high-performance TensorRT LLM stack. diff --git a/docs/source/commands/trtllm-bench.rst b/docs/source/commands/trtllm-bench.rst index 309422df3fba..fee60a9ab70c 100644 --- a/docs/source/commands/trtllm-bench.rst +++ b/docs/source/commands/trtllm-bench.rst @@ -20,10 +20,10 @@ Syntax Dataset preparation ------------------ -prepare-dataset -^^^^^^^^^^^^^^^ +prepare_dataset.py +^^^^^^^^^^^^^^^^^^ -trtllm-bench ships a ``prepare-dataset`` subcommand which generates benchmark datasets in the required format. It supports: +trtllm-bench is designed to work with the `prepare_dataset.py `_ script, which generates benchmark datasets in the required format. The prepare_dataset script supports: **Dataset Types:** @@ -38,17 +38,17 @@ trtllm-bench ships a ``prepare-dataset`` subcommand which generates benchmark da - Support for LoRA adapters and task IDs - Output in JSON format compatible with trtllm-bench -.. note:: - The tokenizer is taken from the model passed to ``trtllm-bench --model``. Use ``--output`` to write the dataset to a file, or ``--stdout`` to stream it with a JSON dataset entry on each line. +.. important:: + The ``--stdout`` flag is **required** when using prepare_dataset.py with trtllm-bench to ensure proper data streaming format. **Usage:** -prepare-dataset +prepare_dataset """"""""""""""" .. code-block:: bash - trtllm-bench --model prepare-dataset [OPTIONS] + python prepare_dataset.py [OPTIONS] **Options** @@ -60,10 +60,12 @@ prepare-dataset * - Option - Description + * - ``--tokenizer`` + - Tokenizer directory or HuggingFace model name (required) * - ``--output`` - Output JSON filename (default: preprocessed_dataset.json) * - ``--stdout`` - - Print output to stdout with a JSON dataset entry on each line instead of writing a file + - Print output to stdout with JSON dataset entry on each line (**required for trtllm-bench**) * - ``--random-seed`` - Random seed for token generation (default: 420) * - ``--task-id`` @@ -75,14 +77,14 @@ prepare-dataset * - ``--log-level`` - Logging level: info or debug (default: info) -real-dataset -"""""""""""" +dataset +""""""" Process real datasets from various sources. .. code-block:: bash - trtllm-bench --model prepare-dataset real-dataset [OPTIONS] + python prepare_dataset.py dataset [OPTIONS] **Options** @@ -106,14 +108,14 @@ Process real datasets from various sources. - Input format: json, jsonl, csv, or txt (default: auto-detect) -token-norm-dist +token_norm_dist """"""""""""""" Generate synthetic datasets with normal token distribution. .. code-block:: bash - trtllm-bench --model prepare-dataset token-norm-dist [OPTIONS] + python prepare_dataset.py token_norm_dist [OPTIONS] **Options** @@ -137,14 +139,14 @@ Generate synthetic datasets with normal token distribution. - Normal distribution standard deviation for output tokens (required) -token-unif-dist +token_unif_dist """"""""""""""" Generate synthetic datasets with uniform token distribution .. code-block:: bash - trtllm-bench --model prepare-dataset token-unif-dist [OPTIONS] + python prepare_dataset.py token_unif_dist [OPTIONS] **Options** diff --git a/docs/source/commands/trtllm-serve/trtllm-serve.rst b/docs/source/commands/trtllm-serve/trtllm-serve.rst index 5f9297eab256..6c9d9a805e96 100644 --- a/docs/source/commands/trtllm-serve/trtllm-serve.rst +++ b/docs/source/commands/trtllm-serve/trtllm-serve.rst @@ -20,8 +20,6 @@ The server also supports the following endpoints: The ``metrics`` endpoint provides runtime-iteration statistics such as GPU memory use and inflight-batching details. -For encoder-only models (BERT-style classifiers, reward models, text-embedding models), the ``trtllm-serve embeddings`` subcommand starts a server that exposes an OpenAI-compatible ``/v1/embeddings`` endpoint with native dynamic batching. See :doc:`Embeddings <../../features/embeddings>` for details. - Starting a Server ----------------- diff --git a/docs/source/developer-guide/api-change.md b/docs/source/developer-guide/api-change.md index 12e0afde763d..ee447e9f9783 100644 --- a/docs/source/developer-guide/api-change.md +++ b/docs/source/developer-guide/api-change.md @@ -193,8 +193,6 @@ Add the field to the appropriate schema file: # No status field for committed arguments ``` -Before validation, run `python3 scripts/generate_llm_args_golden_manifest.py` and commit the manifest; new fields require telemetry/privacy CODEOWNER approval. - #### 3. Run validation tests ```bash diff --git a/docs/source/developer-guide/ci-overview.md b/docs/source/developer-guide/ci-overview.md index 907816f1e792..30cc613a2e38 100644 --- a/docs/source/developer-guide/ci-overview.md +++ b/docs/source/developer-guide/ci-overview.md @@ -24,14 +24,14 @@ Integration tests are listed under `tests/integration/test_lists/test-db/`. Most - `stage`: either `pre_merge` or `post_merge`. - `backend`: `pytorch`, `tensorrt` or `triton`. -Example from `l0_a30.yml`: +Example from `l0_a100.yml`: ```yaml terms: - stage: pre_merge - backend: pytorch + stage: post_merge + backend: triton tests: - - unittest/_torch/sampler/test_beam_search.py + - triton_server/test_triton.py::test_gpt_ib_ptuning[gpt-ib-ptuning] ``` ## Unit tests @@ -61,8 +61,8 @@ Manually searching YAML and Groovy files can be tedious. The helper script `scripts/test_to_stage_mapping.py` automates the lookup: ```bash -python scripts/test_to_stage_mapping.py --tests "unittest/_torch/sampler/test_beam_search.py" -python scripts/test_to_stage_mapping.py --tests test_beam_search +python scripts/test_to_stage_mapping.py --tests "triton_server/test_triton.py::test_gpt_ib_ptuning[gpt-ib-ptuning]" +python scripts/test_to_stage_mapping.py --tests gpt_ib_ptuning python scripts/test_to_stage_mapping.py --stages A100X-Triton-Post-Merge-1 python scripts/test_to_stage_mapping.py --test-list my_tests.txt python scripts/test_to_stage_mapping.py --test-list my_tests.yml @@ -98,8 +98,8 @@ Each line contains the fully qualified test name followed by an optional specific hardware family. Example: ```text -accuracy/test_disaggregated_serving.py::TestDeepSeekV32Exp::test_auto_dtype[False] SKIP (https://nvbugs/6120535) -full:A100/accuracy/test_llm_api_pytorch_multimodal.py::TestExaone4_5_33B::test_auto_dtype[full_budget] SKIP (https://nvbugs/6422318) +examples/test_openai.py::test_llm_openai_triton_1gpu SKIP (https://nvbugspro.nvidia.com/bug/4963654) +full:GH200/examples/test_qwen2audio.py::test_llm_qwen2audio_single_gpu[qwen2_audio_7b_instruct] SKIP (arm is not supported) ``` Changes to `waives.txt` should include a bug link or brief explanation so other @@ -109,17 +109,8 @@ developers understand why the test is disabled. ### Triggering Post-merge tests -Full `/bot run --post-merge` runs require the `ci: post-merge approved` PR -label because they can consume substantial shared GPU resources. The label is -intended to be applied by an active member of the -`NVIDIA/trt-llm-ci-approvers` GitHub team. A GitHub workflow validates the label -actor and normally removes invalid approvals. This is a best-effort resource -governance guard, not a strict authorization boundary. The label remains in -place when new commits are pushed and can be removed manually when the approval -no longer applies. - -When you only need to verify a handful of post-merge tests, specify exactly -which stages to run: +When you only need to verify a handful of post-merge tests, avoid the heavy +`/bot run --post-merge` command. Instead, specify exactly which stages to run: ```bash /bot run --stage-list "stage-A,stage-B" @@ -132,14 +123,8 @@ default pre-merge set: /bot run --extra-stage "stage-A,stage-B" ``` -Both options accept stage names and wildcard patterns defined in -`jenkins/L0_Test.groovy`. The `"*"`, `"*Post-Merge*"`, and `"*PerfSanity*"` -selectors require the same approval label, including when they appear in a -comma-separated list. Equivalent escaped or repeated-star forms are treated the -same. Other stage selectors, including explicit stage names and other limited -wildcard patterns, retain their existing behavior. - -Being selective keeps CI turnaround fast and conserves hardware resources. +Both options accept any stage name defined in `jenkins/L0_Test.groovy`. Being +selective keeps CI turnaround fast and conserves hardware resources. ### Avoiding unnecessary `--disable-fail-fast` usage diff --git a/docs/source/developer-guide/overview.md b/docs/source/developer-guide/overview.md index d8fe31631612..af7f44f139cf 100644 --- a/docs/source/developer-guide/overview.md +++ b/docs/source/developer-guide/overview.md @@ -101,6 +101,7 @@ Module names longer than 8 characters are abbreviated to fit the fixed-width tag | `deep_ep` | `deep_ep ` | | `deep_gemm` | `deepgemm` | | `executor` | `executor` | +| `executor_worker` | `exec_wkr` | | `flash_mla` | `flashmla` | | `kernels` | `kernels ` | | `layers` | `layers ` | diff --git a/docs/source/developer-guide/perf-benchmarking.md b/docs/source/developer-guide/perf-benchmarking.md index 5903c12e32ea..4eb04eef4d13 100644 --- a/docs/source/developer-guide/perf-benchmarking.md +++ b/docs/source/developer-guide/perf-benchmarking.md @@ -171,7 +171,7 @@ can simply read a line and assume a complete entry. When creating a dataset, be JSON entry is on every line. ``` -In order to prepare a synthetic dataset, you can use the provided script in the `benchmarks` +In order to prepare a synthetic dataset, you can use the provided script in the `benchmarks/cpp` directory. For example, to generate a synthetic dataset of 1000 requests with a uniform ISL/OSL of 128/128 for [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B), run: diff --git a/docs/source/developer-guide/perf-overview.md b/docs/source/developer-guide/perf-overview.md index 58de0107b3d8..223ac5e8e92b 100644 --- a/docs/source/developer-guide/perf-overview.md +++ b/docs/source/developer-guide/perf-overview.md @@ -268,7 +268,7 @@ Testing was performed using the PyTorch backend - this workflow does not require | Stage | Description | Command | | :- | - | - | -| [Dataset](#preparing-a-dataset) | Create a synthetic dataset | `trtllm-bench --model $model_name prepare-dataset --output $dataset_file token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0` | +| [Dataset](#preparing-a-dataset) | Create a synthetic dataset | `python benchmarks/cpp/prepare_dataset.py --tokenizer=$model_name --stdout token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0 > $dataset_file` | | [Run](#running-the-benchmark) | Run a benchmark with a dataset | `trtllm-bench --model $model_name throughput --dataset $dataset_file --backend pytorch --config $llm_options` | ### Variables @@ -281,18 +281,18 @@ Testing was performed using the PyTorch backend - this workflow does not require | `$pp_size` | Pipeline parallel mapping degree to run the benchmark with | | `$ep_size` | Expert parallel mapping degree to run the benchmark with | | `$model_name` | HuggingFace model name eg. meta-llama/Llama-2-7b-hf or use the path to a local weights directory | -| `$dataset_file` | Location of the dataset file generated by `trtllm-bench prepare-dataset` | +| `$dataset_file` | Location of the dataset file generated by `prepare_dataset.py` | | `$num_requests` | The number of requests to generate for dataset generation | | `$seq_len` | A sequence length of ISL + OSL | | `$llm_options` | (optional) A yaml file containing additional options for the LLM API | ### Preparing a Dataset -In order to prepare a dataset, use the `trtllm-bench prepare-dataset` subcommand. +In order to prepare a dataset, you can use the provided [script](source:benchmarks/cpp/prepare_dataset.py). To generate a synthetic dataset, run the following command: ```shell -trtllm-bench --model $model_name prepare-dataset --output $dataset_file token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0 +python benchmarks/cpp/prepare_dataset.py --tokenizer=$model_name --stdout token-norm-dist --num-requests=$num_requests --input-mean=$isl --output-mean=$osl --input-stdev=0 --output-stdev=0 > $dataset_file ``` The command will generate a text file located at the path specified `$dataset_file` where all requests are of the same diff --git a/docs/source/developer-guide/telemetry.md b/docs/source/developer-guide/telemetry.md index 9ca7b7aaa49f..dd4f9cfcb7ed 100644 --- a/docs/source/developer-guide/telemetry.md +++ b/docs/source/developer-guide/telemetry.md @@ -18,9 +18,6 @@ unless a field carries an explicit allowlist (`TelemetryField.categorical(...)`) and any field may opt out with `telemetry=False`. Every captured field is listed below; the runtime can capture nothing absent from this list. -If the manifest check fails, run `python3 scripts/generate_llm_args_golden_manifest.py`, then commit -`tensorrt_llm/usage/llm_args_golden_manifest.json`; new fields require telemetry/privacy CODEOWNER approval. - ## LLM API Configuration Fields A field can still be absent from a specific payload when its parent config is @@ -28,7 +25,7 @@ unset or when the safety sanitizer rejects the runtime value. ### `TorchLlmArgs` -269 captured fields. +234 captured fields. | Captured key | Annotation | Kind | Converter | Allowed values | |--------------|------------|------|-----------|----------------| @@ -36,13 +33,10 @@ unset or when the safety sanitizer rejects the runtime value. | `attention_dp_config.batching_wait_iters` | `` | `value` | | | | `attention_dp_config.enable_balance` | `` | `value` | | | | `attention_dp_config.enable_kv_cache_aware_routing` | `` | `value` | | | -| `attention_dp_config.kv_cache_routing_account_for_in_transfer` | `` | `value` | | | | `attention_dp_config.kv_cache_routing_cold_start_warmup` | `` | `value` | | | -| `attention_dp_config.kv_cache_routing_conversation_affinity` | `` | `value` | | | | `attention_dp_config.kv_cache_routing_fair_share_multiplier` | `` | `value` | | | | `attention_dp_config.kv_cache_routing_load_balance_weight` | `` | `value` | | | | `attention_dp_config.kv_cache_routing_match_rate_threshold` | `` | `value` | | | -| `attention_dp_config.kv_cache_routing_max_sessions` | `` | `value` | | | | `attention_dp_config.timeout_iters` | `` | `value` | | | | `attn_backend` | `` | `categorical` | allowlist | `VANILLA`, `TRTLLM`, `FLASHINFER`, `FLASHINFER_STAR_ATTENTION` | | `backend` | `Literal['pytorch']` | `categorical` | | `pytorch` | @@ -50,12 +44,10 @@ unset or when the safety sanitizer rejects the runtime value. | `batch_wait_timeout_iters` | `` | `value` | | | | `batch_wait_timeout_ms` | `` | `value` | | | | `cache_transceiver_config.backend` | `Optional[Literal['DEFAULT', 'UCX', 'NIXL', 'MOONCAKE', 'MPI']]` | `categorical` | | `DEFAULT`, `UCX`, `NIXL`, `MOONCAKE`, `MPI` | -| `cache_transceiver_config.kv_cache_bounce_size_mb` | `` | `value` | | | -| `cache_transceiver_config.kv_transfer_poll_interval_ms` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | | `cache_transceiver_config.kv_transfer_sender_future_timeout_ms` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | | `cache_transceiver_config.kv_transfer_timeout_ms` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | | `cache_transceiver_config.max_tokens_in_buffer` | `Optional[int]` | `value` | | | -| `cache_transceiver_config.transceiver_runtime` | `Optional[Literal['CPP', 'PYTHON', 'auto']]` | `categorical` | | `CPP`, `PYTHON`, `auto` | +| `cache_transceiver_config.transceiver_runtime` | `Optional[Literal['CPP', 'PYTHON']]` | `categorical` | | `CPP`, `PYTHON` | | `context_parallel_size` | `` | `value` | | | | `cp_config.block_size` | `Optional[int]` | `value` | | | | `cp_config.cp_anchor_size` | `Optional[int]` | `value` | | | @@ -71,7 +63,7 @@ unset or when the safety sanitizer rejects the runtime value. | `cuda_graph_config.mode` | `Literal['decode']` | `categorical` | | `decode`, `encode` | | `cuda_graph_config.num_tokens` | `Optional[List[Annotated[int, Gt(gt=0)]]]` | `value` | | | | `cuda_graph_config.seq_lens` | `Optional[List[Annotated[int, Gt(gt=0)]]]` | `value` | | | -| `disable_mm_encoder` | `` | `value` | | | +| `disable_flashinfer_sampling` | `` | `value` | | | | `disable_overlap_scheduler` | `` | `value` | | | | `dtype` | `` | `categorical` | allowlist | `auto`, `float16`, `bfloat16`, `float32` | | `dwdp_config.contention_opt` | `` | `value` | | | @@ -89,13 +81,10 @@ unset or when the safety sanitizer rejects the runtime value. | `enable_layerwise_nvtx_marker` | `` | `value` | | | | `enable_lm_head_tp_in_adp` | `` | `value` | | | | `enable_lora` | `` | `value` | | | -| `enable_low_latency_host_dispatch` | `` | `value` | | | | `enable_min_latency` | `` | `value` | | | | `enable_resource_governor` | `` | `value` | | | | `enable_speculative_beam_history_d2h` | `` | `value` | | | | `encode_only` | `` | `value` | | | -| `encoder_max_batch_size` | `Optional[int]` | `value` | | | -| `encoder_max_num_tokens` | `Optional[int]` | `value` | | | | `force_dynamic_quantization` | `` | `value` | | | | `garbage_collection_gen0_threshold` | `` | `value` | | | | `gather_generation_logits` | `` | `value` | | | @@ -104,22 +93,15 @@ unset or when the safety sanitizer rejects the runtime value. | `guided_decoding_backend` | `Optional[Literal['xgrammar', 'llguidance']]` | `categorical` | | `xgrammar`, `llguidance` | | `iter_stats_max_iterations` | `Optional[int]` | `value` | | | | `kv_cache_config.attention_dp_events_gather_period_ms` | `` | `value` | | | -| `kv_cache_config.avg_seq_len` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | -| `kv_cache_config.block_reuse_policy` | `Literal['all_reusable', 'per_request', 'per_conversation']` | `categorical` | | `all_reusable`, `per_request`, `per_conversation` | | `kv_cache_config.copy_on_partial_reuse` | `` | `value` | | | | `kv_cache_config.cross_kv_cache_fraction` | `Optional[float]` | `value` | | | -| `kv_cache_config.disk_cache_size` | `Optional[Annotated[int, Ge(ge=0)]]` | `value` | | | -| `kv_cache_config.disk_prefetch_num_reqs` | `` | `value` | | | | `kv_cache_config.dtype` | `` | `categorical` | allowlist | `auto`, `float16`, `bfloat16`, `float32`, `fp8`, `nvfp4` | | `kv_cache_config.enable_block_reuse` | `` | `value` | | | -| `kv_cache_config.enable_kv_pool_rebalance` | `` | `value` | | | | `kv_cache_config.enable_partial_reuse` | `` | `value` | | | -| `kv_cache_config.enable_swa_scratch_reuse` | `` | `value` | | | | `kv_cache_config.event_buffer_max_size` | `` | `value` | | | | `kv_cache_config.free_gpu_memory_fraction` | `Optional[float]` | `value` | | | | `kv_cache_config.host_cache_size` | `Optional[int]` | `value` | | | | `kv_cache_config.iteration_stats_interval` | `` | `value` | | | -| `kv_cache_config.kv_cache_event_hash_algo` | `Literal['auto', 'v1_block_key', 'v2_sha256', 'v2_sha256_64']` | `categorical` | | `auto`, `v1_block_key`, `v2_sha256`, `v2_sha256_64` | | `kv_cache_config.mamba_ssm_cache_dtype` | `Literal['auto', 'float16', 'bfloat16', 'float32']` | `categorical` | | `auto`, `float16`, `bfloat16`, `float32` | | `kv_cache_config.mamba_ssm_philox_rounds` | `` | `value` | | | | `kv_cache_config.mamba_ssm_stochastic_rounding` | `` | `value` | | | @@ -128,11 +110,10 @@ unset or when the safety sanitizer rejects the runtime value. | `kv_cache_config.max_gpu_total_bytes` | `` | `value` | | | | `kv_cache_config.max_tokens` | `Optional[int]` | `value` | | | | `kv_cache_config.max_util_for_resume` | `` | `value` | | | -| `kv_cache_config.pool_ratio` | `Optional[List[float]]` | `value` | | | | `kv_cache_config.secondary_offload_min_priority` | `Optional[int]` | `value` | | | | `kv_cache_config.sink_token_length` | `Optional[int]` | `value` | | | | `kv_cache_config.tokens_per_block` | `` | `value` | | | -| `kv_cache_config.use_kv_cache_manager_v2` | `Union[bool, Literal['auto']]` | `value` | | `auto` | +| `kv_cache_config.use_kv_cache_manager_v2` | `` | `value` | | | | `kv_cache_config.use_uvm` | `` | `value` | | | | `kv_connector_config.connector` | `Optional[str]` | `categorical` | allowlist | `lmcache`, `lmcache-mp`, `kvbm` | | `layer_wise_benchmarks_config.calibration_layer_indices` | `Optional[List[int]]` | `value` | | | @@ -151,20 +132,16 @@ unset or when the safety sanitizer rejects the runtime value. | `max_stats_len` | `` | `value` | | | | `mm_encoder_only` | `` | `value` | | | | `moe_cluster_parallel_size` | `Optional[int]` | `value` | | | -| `moe_config.backend` | `Literal['AUTO', 'CUTLASS', 'CUTEDSL', 'WIDEEP', 'TRTLLM', 'DEEPGEMM', 'DENSEGEMM', 'VANILLA', 'TRITON', 'MARLIN', 'MEGAMOE_DEEPGEMM']` | `categorical` | | `AUTO`, `CUTLASS`, `CUTEDSL`, `WIDEEP`, `TRTLLM`, `DEEPGEMM`, `DENSEGEMM`, `VANILLA`, `TRITON`, `MARLIN`, `MEGAMOE_DEEPGEMM` | +| `moe_config.backend` | `Literal['AUTO', 'CUTLASS', 'CUTEDSL', 'WIDEEP', 'TRTLLM', 'DEEPGEMM', 'DENSEGEMM', 'VANILLA', 'TRITON']` | `categorical` | | `AUTO`, `CUTLASS`, `CUTEDSL`, `WIDEEP`, `TRTLLM`, `DEEPGEMM`, `DENSEGEMM`, `VANILLA`, `TRITON` | | `moe_config.disable_finalize_fusion` | `` | `value` | | | | `moe_config.max_num_tokens` | `Optional[int]` | `value` | | | | `moe_config.use_low_precision_moe_combine` | `` | `value` | | | | `moe_expert_parallel_size` | `Optional[int]` | `value` | | | | `moe_tensor_parallel_size` | `Optional[int]` | `value` | | | -| `multimodal_config.encoder_cache_max_bytes` | `` | `value` | | | -| `multimodal_config.encoder_side_stream_max_ahead` | `` | `value` | | | -| `multimodal_config.video_pruning_rate` | `Optional[float]` | `value` | | | | `mx_config.preshard_strategy` | `` | `categorical` | allowlist | `per_module` | | `mx_config.server_query_timeout_s` | `Optional[Annotated[int, Ge(ge=0)]]` | `value` | | | | `num_postprocess_workers` | `` | `value` | | | -| `num_serve_frontends` | `` | `value` | | | -| `nvfp4_gemm_config.allowed_backends` | `List[Literal['cutlass', 'cublaslt', 'cutedsl', 'cuda_core', 'marlin']]` | `value` | | `cutlass`, `cublaslt`, `cutedsl`, `cuda_core`, `marlin` | +| `nvfp4_gemm_config.allowed_backends` | `List[Literal['cutlass', 'cublaslt', 'cutedsl', 'cuda_core']]` | `value` | | `cutlass`, `cublaslt`, `cutedsl`, `cuda_core` | | `orchestrator_type` | `Optional[Literal['rpc', 'ray']]` | `categorical` | | `rpc`, `ray` | | `peft_cache_config.device_cache_percent` | `` | `value` | | | | `peft_cache_config.host_cache_size` | `` | `value` | | | @@ -208,10 +185,8 @@ unset or when the safety sanitizer rejects the runtime value. | `scheduler_config.use_python_scheduler` | `` | `value` | | | | `scheduler_config.waiting_queue_policy` | `` | `categorical` | | `fcfs`, `priority` | | `skip_tokenizer_init` | `` | `value` | | | -| `sparse_attention_config.algorithm` | `Literal['dsa']` | `categorical` | | `dsa`, `deepseek_v4`, `minimax_m3`, `rocket`, `skip_softmax` | -| `sparse_attention_config.compress_ratios` | `List[int]` | `value` | | | +| `sparse_attention_config.algorithm` | `Literal['dsa']` | `categorical` | | `dsa`, `rocket`, `skip_softmax` | | `sparse_attention_config.enable_heuristic_topk` | `` | `value` | | | -| `sparse_attention_config.implementation` | `Literal['triton', 'msa']` | `categorical` | | `triton`, `msa` | | `sparse_attention_config.index_head_dim` | `Optional[int]` | `value` | | | | `sparse_attention_config.index_n_heads` | `Optional[int]` | `value` | | | | `sparse_attention_config.index_topk` | `Optional[int]` | `value` | | | @@ -220,32 +195,21 @@ unset or when the safety sanitizer rejects the runtime value. | `sparse_attention_config.indexer_rope_interleave` | `` | `value` | | | | `sparse_attention_config.kernel_size` | `Optional[int]` | `value` | | | | `sparse_attention_config.kt_cache_dtype` | `Optional[str]` | `categorical` | allowlist | `bfloat16`, `float8_e5m2` | -| `sparse_attention_config.num_attention_heads` | `Optional[int]` | `value` | | | -| `sparse_attention_config.num_key_value_heads` | `Optional[int]` | `value` | | | | `sparse_attention_config.page_size` | `Optional[int]` | `value` | | | | `sparse_attention_config.prompt_budget` | `Optional[int]` | `value` | | | | `sparse_attention_config.q_split_threshold` | `` | `value` | | | | `sparse_attention_config.seq_len_threshold` | `Optional[int]` | `value` | | | | `sparse_attention_config.skip_indexer_for_short_seqs` | `` | `value` | | | -| `sparse_attention_config.sparse_block_size` | `` | `value` | | | -| `sparse_attention_config.sparse_disable_index_value` | `` | `value` | | | -| `sparse_attention_config.sparse_index_dim` | `` | `value` | | | -| `sparse_attention_config.sparse_init_blocks` | `` | `value` | | | -| `sparse_attention_config.sparse_local_blocks` | `` | `value` | | | -| `sparse_attention_config.sparse_num_index_heads` | `` | `value` | | | -| `sparse_attention_config.sparse_score_type` | `Literal['max']` | `categorical` | | `max` | -| `sparse_attention_config.sparse_topk_blocks` | `` | `value` | | | | `sparse_attention_config.topk` | `Optional[int]` | `value` | | | | `sparse_attention_config.topr` | `Union[int, float, NoneType]` | `value` | | | | `sparse_attention_config.use_cute_dsl_paged_mqa_logits` | `` | `value` | | | | `sparse_attention_config.use_cute_dsl_topk` | `` | `value` | | | -| `sparse_attention_config.window_size` | `` | `value` | | | -| `speculative_config.acceptance_rate_threshold` | `Optional[float]` | `value` | | | -| `speculative_config.acceptance_rate_window_size` | `Optional[Annotated[int, Ge(ge=0)]]` | `value` | | | +| `sparse_attention_config.window_size` | `Optional[int]` | `value` | | | +| `speculative_config.acceptance_length_threshold` | `Optional[Annotated[float, Ge(ge=0)]]` | `value` | | | +| `speculative_config.acceptance_window` | `Optional[Annotated[int, Ge(ge=0)]]` | `value` | | | | `speculative_config.allow_advanced_sampling` | `` | `value` | | | | `speculative_config.begin_thinking_phase_token` | `` | `value` | | | -| `speculative_config.block_size` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | -| `speculative_config.decoding_type` | `Literal['AUTO']` | `categorical` | | `AUTO`, `DFlash`, `DSpark`, `Draft_Target`, `Eagle3`, `Eagle`, `Lookahead`, `MTP`, `Medusa`, `NGram`, `PARD`, `SA`, `SaveState`, `User_Provided` | +| `speculative_config.decoding_type` | `Literal['AUTO']` | `categorical` | | `AUTO`, `DFlash`, `Draft_Target`, `Eagle3`, `Eagle`, `Lookahead`, `MTP`, `Medusa`, `NGram`, `PARD`, `SA`, `SaveState`, `User_Provided` | | `speculative_config.dynamic_tree_max_topK` | `Optional[int]` | `value` | | | | `speculative_config.eagle3_layers_to_capture` | `Optional[Set[int]]` | `value` | | | | `speculative_config.eagle3_model_arch` | `Literal['llama3', 'mistral_large3']` | `categorical` | | `llama3`, `mistral_large3` | @@ -258,8 +222,6 @@ unset or when the safety sanitizer rejects the runtime value. | `speculative_config.is_keep_all` | `` | `value` | | | | `speculative_config.is_public_pool` | `` | `value` | | | | `speculative_config.is_use_oldest` | `` | `value` | | | -| `speculative_config.markov_head_type` | `Optional[Literal['vanilla', 'gated', 'rnn']]` | `categorical` | | `vanilla`, `gated`, `rnn` | -| `speculative_config.markov_rank` | `Optional[int]` | `value` | | | | `speculative_config.mask_token_id` | `Optional[int]` | `value` | | | | `speculative_config.max_concurrency` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | | `speculative_config.max_draft_len` | `Optional[Annotated[int, Ge(ge=0)]]` | `value` | | | @@ -301,3 +263,272 @@ unset or when the safety sanitizer rejects the runtime value. | `use_cute_dsl_bf16_gemm` | `` | `value` | | | | `use_cute_dsl_blockscaling_bmm` | `` | `value` | | | | `use_cute_dsl_blockscaling_mm` | `` | `value` | | | +| `video_pruning_rate` | `Optional[float]` | `value` | | | + +### `TrtLlmArgs` + +260 captured fields. + +| Captured key | Annotation | Kind | Converter | Allowed values | +|--------------|------------|------|-----------|----------------| +| `backend` | `Optional[str]` | `categorical` | allowlist | `pytorch`, `tensorrt`, `_autodeploy` | +| `batching_type` | `Optional[tensorrt_llm.llmapi.llm_args.BatchingType]` | `categorical` | | `STATIC`, `INFLIGHT` | +| `build_config.dry_run` | `` | `value` | | | +| `build_config.enable_debug_output` | `` | `value` | | | +| `build_config.force_num_profiles` | `Optional[int]` | `value` | | | +| `build_config.gather_context_logits` | `` | `value` | | | +| `build_config.gather_generation_logits` | `` | `value` | | | +| `build_config.kv_cache_type` | `Optional[tensorrt_llm.llmapi.kv_cache_type.KVCacheType]` | `categorical` | | `continuous`, `paged`, `disabled` | +| `build_config.lora_config.lora_ckpt_source` | `Literal['hf', 'nemo']` | `categorical` | | `hf`, `nemo` | +| `build_config.lora_config.max_cpu_loras` | `Optional[int]` | `value` | | | +| `build_config.lora_config.max_lora_rank` | `` | `value` | | | +| `build_config.lora_config.max_loras` | `Optional[int]` | `value` | | | +| `build_config.lora_config.swap_gate_up_proj_lora_b_weight` | `` | `value` | | | +| `build_config.max_batch_size` | `` | `value` | | | +| `build_config.max_beam_width` | `` | `value` | | | +| `build_config.max_draft_len` | `` | `value` | | | +| `build_config.max_encoder_input_len` | `` | `value` | | | +| `build_config.max_input_len` | `` | `value` | | | +| `build_config.max_num_tokens` | `` | `value` | | | +| `build_config.max_prompt_embedding_table_size` | `` | `value` | | | +| `build_config.max_seq_len` | `Optional[int]` | `value` | | | +| `build_config.monitor_memory` | `` | `value` | | | +| `build_config.opt_batch_size` | `` | `value` | | | +| `build_config.opt_num_tokens` | `Optional[int]` | `value` | | | +| `build_config.plugin_config.bert_attention_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.bert_context_fmha_fp32_acc` | `` | `value` | | | +| `build_config.plugin_config.context_fmha` | `` | `value` | | | +| `build_config.plugin_config.dora_plugin` | `` | `value` | | | +| `build_config.plugin_config.fp8_rowwise_gemm_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.fuse_fp4_quant` | `` | `value` | | | +| `build_config.plugin_config.gemm_allreduce_plugin` | `Optional[Literal['float16', 'bfloat16', None]]` | `categorical` | | `float16`, `bfloat16`, `None` | +| `build_config.plugin_config.gemm_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', 'fp8', 'nvfp4', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `fp8`, `nvfp4`, `None` | +| `build_config.plugin_config.gemm_swiglu_plugin` | `Optional[Literal['fp8', None]]` | `categorical` | | `fp8`, `None` | +| `build_config.plugin_config.gpt_attention_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.identity_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.layernorm_quantization_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.lora_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.low_latency_gemm_plugin` | `Optional[Literal['fp8', None]]` | `categorical` | | `fp8`, `None` | +| `build_config.plugin_config.low_latency_gemm_swiglu_plugin` | `Optional[Literal['fp8', None]]` | `categorical` | | `fp8`, `None` | +| `build_config.plugin_config.mamba_conv1d_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.manage_weights` | `` | `value` | | | +| `build_config.plugin_config.moe_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.multiple_profiles` | `` | `value` | | | +| `build_config.plugin_config.nccl_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.norm_quant_fusion` | `` | `value` | | | +| `build_config.plugin_config.paged_kv_cache` | `Optional[bool]` | `value` | | | +| `build_config.plugin_config.paged_state` | `` | `value` | | | +| `build_config.plugin_config.pp_reduce_scatter` | `` | `value` | | | +| `build_config.plugin_config.qserve_gemm_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.quantize_per_token_plugin` | `` | `value` | | | +| `build_config.plugin_config.quantize_tensor_plugin` | `` | `value` | | | +| `build_config.plugin_config.reduce_fusion` | `` | `value` | | | +| `build_config.plugin_config.remove_input_padding` | `` | `value` | | | +| `build_config.plugin_config.rmsnorm_quantization_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.smooth_quant_gemm_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.smooth_quant_plugins` | `` | `value` | | | +| `build_config.plugin_config.streamingllm` | `` | `value` | | | +| `build_config.plugin_config.tokens_per_block` | `` | `value` | | | +| `build_config.plugin_config.use_fp8_context_fmha` | `` | `value` | | | +| `build_config.plugin_config.use_fused_mlp` | `` | `value` | | | +| `build_config.plugin_config.use_paged_context_fmha` | `` | `value` | | | +| `build_config.plugin_config.user_buffer` | `` | `value` | | | +| `build_config.plugin_config.weight_only_groupwise_quant_matmul_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.plugin_config.weight_only_quant_matmul_plugin` | `Optional[Literal['auto', 'float16', 'float32', 'bfloat16', 'int32', None]]` | `categorical` | | `auto`, `float16`, `float32`, `bfloat16`, `int32`, `None` | +| `build_config.speculative_decoding_mode` | `` | `categorical` | | `NONE`, `DRAFT_TOKENS_EXTERNAL`, `MEDUSA`, `LOOKAHEAD_DECODING`, `EXPLICIT_DRAFT_TOKENS`, `EAGLE`, `NGRAM`, `USER_PROVIDED`, `SAVE_HIDDEN_STATES`, `AUTO` | +| `build_config.strongly_typed` | `` | `value` | | | +| `build_config.use_mrope` | `` | `value` | | | +| `build_config.use_refit` | `` | `value` | | | +| `build_config.use_strip_plan` | `` | `value` | | | +| `build_config.weight_sparsity` | `` | `value` | | | +| `build_config.weight_streaming` | `` | `value` | | | +| `cache_transceiver_config.backend` | `Optional[Literal['DEFAULT', 'UCX', 'NIXL', 'MOONCAKE', 'MPI']]` | `categorical` | | `DEFAULT`, `UCX`, `NIXL`, `MOONCAKE`, `MPI` | +| `cache_transceiver_config.kv_transfer_sender_future_timeout_ms` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | +| `cache_transceiver_config.kv_transfer_timeout_ms` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | +| `cache_transceiver_config.max_tokens_in_buffer` | `Optional[int]` | `value` | | | +| `cache_transceiver_config.transceiver_runtime` | `Optional[Literal['CPP', 'PYTHON']]` | `categorical` | | `CPP`, `PYTHON` | +| `calib_config.calib_batch_size` | `` | `value` | | | +| `calib_config.calib_batches` | `` | `value` | | | +| `calib_config.calib_max_seq_length` | `` | `value` | | | +| `calib_config.device` | `Literal['cuda', 'cpu']` | `categorical` | | `cuda`, `cpu` | +| `calib_config.random_seed` | `` | `value` | | | +| `calib_config.tokenizer_max_seq_length` | `` | `value` | | | +| `context_parallel_size` | `` | `value` | | | +| `cp_config.block_size` | `Optional[int]` | `value` | | | +| `cp_config.cp_anchor_size` | `Optional[int]` | `value` | | | +| `cp_config.cp_type` | `` | `categorical` | | `ULYSSES`, `STAR`, `RING`, `HELIX` | +| `cp_config.fifo_version` | `Optional[int]` | `value` | | | +| `cp_config.tokens_per_block` | `Optional[int]` | `value` | | | +| `cp_config.use_nccl_for_alltoall` | `Optional[bool]` | `value` | | | +| `dtype` | `` | `categorical` | allowlist | `auto`, `float16`, `bfloat16`, `float32` | +| `embedding_parallel_mode` | `Literal['NONE', 'SHARDING_ALONG_VOCAB', 'SHARDING_ALONG_HIDDEN']` | `categorical` | | `NONE`, `SHARDING_ALONG_VOCAB`, `SHARDING_ALONG_HIDDEN` | +| `enable_attention_dp` | `` | `value` | | | +| `enable_build_cache.max_cache_storage_gb` | `` | `value` | | | +| `enable_build_cache.max_records` | `` | `value` | | | +| `enable_chunked_prefill` | `` | `value` | | | +| `enable_energy_metrics` | `` | `value` | | | +| `enable_lm_head_tp_in_adp` | `` | `value` | | | +| `enable_lora` | `` | `value` | | | +| `enable_prompt_adapter` | `` | `value` | | | +| `enable_tqdm` | `` | `value` | | | +| `extended_runtime_perf_knob_config.cuda_graph_cache_size` | `` | `value` | | | +| `extended_runtime_perf_knob_config.cuda_graph_mode` | `` | `value` | | | +| `extended_runtime_perf_knob_config.enable_context_fmha_fp32_acc` | `` | `value` | | | +| `extended_runtime_perf_knob_config.multi_block_mode` | `` | `value` | | | +| `fail_fast_on_attention_window_too_large` | `` | `value` | | | +| `fast_build` | `` | `value` | | | +| `gather_generation_logits` | `` | `value` | | | +| `gpus_per_node` | `Optional[int]` | `value` | | | +| `guided_decoding_backend` | `Optional[Literal['xgrammar', 'llguidance']]` | `categorical` | | `xgrammar`, `llguidance` | +| `iter_stats_max_iterations` | `Optional[int]` | `value` | | | +| `kv_cache_config.attention_dp_events_gather_period_ms` | `` | `value` | | | +| `kv_cache_config.copy_on_partial_reuse` | `` | `value` | | | +| `kv_cache_config.cross_kv_cache_fraction` | `Optional[float]` | `value` | | | +| `kv_cache_config.dtype` | `` | `categorical` | allowlist | `auto`, `float16`, `bfloat16`, `float32`, `fp8`, `nvfp4` | +| `kv_cache_config.enable_block_reuse` | `` | `value` | | | +| `kv_cache_config.enable_partial_reuse` | `` | `value` | | | +| `kv_cache_config.event_buffer_max_size` | `` | `value` | | | +| `kv_cache_config.free_gpu_memory_fraction` | `Optional[float]` | `value` | | | +| `kv_cache_config.host_cache_size` | `Optional[int]` | `value` | | | +| `kv_cache_config.iteration_stats_interval` | `` | `value` | | | +| `kv_cache_config.mamba_ssm_cache_dtype` | `Literal['auto', 'float16', 'bfloat16', 'float32']` | `categorical` | | `auto`, `float16`, `bfloat16`, `float32` | +| `kv_cache_config.mamba_ssm_philox_rounds` | `` | `value` | | | +| `kv_cache_config.mamba_ssm_stochastic_rounding` | `` | `value` | | | +| `kv_cache_config.mamba_state_cache_interval` | `` | `value` | | | +| `kv_cache_config.max_attention_window` | `Optional[List[int]]` | `value` | | | +| `kv_cache_config.max_gpu_total_bytes` | `` | `value` | | | +| `kv_cache_config.max_tokens` | `Optional[int]` | `value` | | | +| `kv_cache_config.max_util_for_resume` | `` | `value` | | | +| `kv_cache_config.secondary_offload_min_priority` | `Optional[int]` | `value` | | | +| `kv_cache_config.sink_token_length` | `Optional[int]` | `value` | | | +| `kv_cache_config.tokens_per_block` | `` | `value` | | | +| `kv_cache_config.use_kv_cache_manager_v2` | `` | `value` | | | +| `kv_cache_config.use_uvm` | `` | `value` | | | +| `load_format` | `Literal['auto', 'dummy']` | `categorical` | | `auto`, `dummy` | +| `lora_config.lora_ckpt_source` | `Literal['hf', 'nemo']` | `categorical` | | `hf`, `nemo` | +| `lora_config.max_cpu_loras` | `Optional[int]` | `value` | | | +| `lora_config.max_lora_rank` | `` | `value` | | | +| `lora_config.max_loras` | `Optional[int]` | `value` | | | +| `lora_config.swap_gate_up_proj_lora_b_weight` | `` | `value` | | | +| `max_batch_size` | `Optional[int]` | `value` | | | +| `max_beam_width` | `Optional[int]` | `value` | | | +| `max_input_len` | `Optional[int]` | `value` | | | +| `max_num_tokens` | `Optional[int]` | `value` | | | +| `max_prompt_adapter_token` | `` | `value` | | | +| `max_seq_len` | `Optional[int]` | `value` | | | +| `moe_cluster_parallel_size` | `Optional[int]` | `value` | | | +| `moe_expert_parallel_size` | `Optional[int]` | `value` | | | +| `moe_tensor_parallel_size` | `Optional[int]` | `value` | | | +| `normalize_log_probs` | `` | `value` | | | +| `num_postprocess_workers` | `` | `value` | | | +| `orchestrator_type` | `Optional[Literal['rpc', 'ray']]` | `categorical` | | `rpc`, `ray` | +| `peft_cache_config.device_cache_percent` | `` | `value` | | | +| `peft_cache_config.host_cache_size` | `` | `value` | | | +| `peft_cache_config.max_adapter_size` | `` | `value` | | | +| `peft_cache_config.max_pages_per_block_device` | `` | `value` | | | +| `peft_cache_config.max_pages_per_block_host` | `` | `value` | | | +| `peft_cache_config.num_copy_streams` | `` | `value` | | | +| `peft_cache_config.num_device_module_layer` | `` | `value` | | | +| `peft_cache_config.num_ensure_workers` | `` | `value` | | | +| `peft_cache_config.num_host_module_layer` | `` | `value` | | | +| `peft_cache_config.num_put_workers` | `` | `value` | | | +| `peft_cache_config.optimal_adapter_size` | `` | `value` | | | +| `perf_metrics_max_requests` | `` | `value` | | | +| `pipeline_parallel_size` | `` | `value` | | | +| `pp_partition` | `Optional[List[int]]` | `value` | | | +| `prometheus_metrics_config.e2e_request_latency_buckets` | `Optional[List[float]]` | `value` | | | +| `prometheus_metrics_config.request_decode_time_buckets` | `Optional[List[float]]` | `value` | | | +| `prometheus_metrics_config.request_inference_time_buckets` | `Optional[List[float]]` | `value` | | | +| `prometheus_metrics_config.request_prefill_time_buckets` | `Optional[List[float]]` | `value` | | | +| `prometheus_metrics_config.request_queue_time_buckets` | `Optional[List[float]]` | `value` | | | +| `prometheus_metrics_config.time_per_output_token_buckets` | `Optional[List[float]]` | `value` | | | +| `prometheus_metrics_config.time_to_first_token_buckets` | `Optional[List[float]]` | `value` | | | +| `quant_config.clamp_val` | `Optional[List[float]]` | `value` | | | +| `quant_config.group_size` | `Optional[int]` | `value` | | | +| `quant_config.has_zero_point` | `` | `value` | | | +| `quant_config.kv_cache_quant_algo` | `Optional[tensorrt_llm.quantization.mode.QuantAlgo]` | `categorical` | | `W8A16`, `W4A16`, `W4A16_AWQ`, `W4A8_AWQ`, `W8A16_GPTQ`, `W4A16_GPTQ`, `W8A8_SQ_PER_CHANNEL`, `W8A8_SQ_PER_TENSOR_PLUGIN`, `W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN`, `W8A8_SQ_PER_CHANNEL_PER_TENSOR_PLUGIN`, `W8A8_SQ_PER_TENSOR_PER_TOKEN_PLUGIN`, `W4A8_QSERVE_PER_GROUP`, `W4A8_QSERVE_PER_CHANNEL`, `FP8`, `FP8_PER_CHANNEL_PER_TOKEN`, `FP8_BLOCK_SCALES`, `INT8`, `MIXED_PRECISION`, `NVFP4`, `W4A8_NVFP4_FP8`, `W4A8_MXFP4_FP8`, `W4A8_MXFP4_MXFP8`, `W4A16_MXFP4`, `NVFP4_AWQ`, `NVFP4_ARC`, `NO_QUANT` | +| `quant_config.mamba_ssm_philox_rounds` | `` | `value` | | | +| `quant_config.mamba_ssm_stochastic_rounding` | `` | `value` | | | +| `quant_config.pre_quant_scale` | `` | `value` | | | +| `quant_config.quant_algo` | `Optional[tensorrt_llm.quantization.mode.QuantAlgo]` | `categorical` | | `W8A16`, `W4A16`, `W4A16_AWQ`, `W4A8_AWQ`, `W8A16_GPTQ`, `W4A16_GPTQ`, `W8A8_SQ_PER_CHANNEL`, `W8A8_SQ_PER_TENSOR_PLUGIN`, `W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN`, `W8A8_SQ_PER_CHANNEL_PER_TENSOR_PLUGIN`, `W8A8_SQ_PER_TENSOR_PER_TOKEN_PLUGIN`, `W4A8_QSERVE_PER_GROUP`, `W4A8_QSERVE_PER_CHANNEL`, `FP8`, `FP8_PER_CHANNEL_PER_TOKEN`, `FP8_BLOCK_SCALES`, `INT8`, `MIXED_PRECISION`, `NVFP4`, `W4A8_NVFP4_FP8`, `W4A8_MXFP4_FP8`, `W4A8_MXFP4_MXFP8`, `W4A16_MXFP4`, `NVFP4_AWQ`, `NVFP4_ARC`, `NO_QUANT` | +| `quant_config.smoothquant_val` | `` | `value` | | | +| `quant_config.use_meta_recipe` | `` | `value` | | | +| `reasoning_parser` | `Optional[str]` | `categorical` | allowlist | `auto`, `deepseek-r1`, `laguna`, `qwen3`, `qwen3_5`, `minimax_m2`, `minimax_m2_append_think`, `nano-v3`, `gemma4`, `kimi_k2`, `kimi_k25` | +| `request_stats_max_iterations` | `Optional[int]` | `value` | | | +| `return_perf_metrics` | `` | `value` | | | +| `scheduler_config.capacity_scheduler_policy` | `` | `categorical` | | `MAX_UTILIZATION`, `GUARANTEED_NO_EVICT`, `STATIC_BATCH` | +| `scheduler_config.context_chunking_policy` | `Optional[tensorrt_llm.llmapi.llm_args.ContextChunkingPolicy]` | `categorical` | | `FIRST_COME_FIRST_SERVED`, `EQUAL_PROGRESS`, `FORCE_CHUNK` | +| `scheduler_config.dynamic_batch_config.dynamic_batch_moving_average_window` | `` | `value` | | | +| `scheduler_config.dynamic_batch_config.enable_batch_size_tuning` | `` | `value` | | | +| `scheduler_config.dynamic_batch_config.enable_max_num_tokens_tuning` | `` | `value` | | | +| `scheduler_config.enable_prefix_aware_scheduling` | `` | `value` | | | +| `scheduler_config.use_python_scheduler` | `` | `value` | | | +| `scheduler_config.waiting_queue_policy` | `` | `categorical` | | `fcfs`, `priority` | +| `skip_tokenizer_init` | `` | `value` | | | +| `sparse_attention_config.algorithm` | `Literal['dsa']` | `categorical` | | `dsa`, `rocket`, `skip_softmax` | +| `sparse_attention_config.enable_heuristic_topk` | `` | `value` | | | +| `sparse_attention_config.index_head_dim` | `Optional[int]` | `value` | | | +| `sparse_attention_config.index_n_heads` | `Optional[int]` | `value` | | | +| `sparse_attention_config.index_topk` | `Optional[int]` | `value` | | | +| `sparse_attention_config.indexer_k_dtype` | `Literal['fp8', 'fp4']` | `categorical` | | `fp8`, `fp4` | +| `sparse_attention_config.indexer_max_chunk_size` | `Optional[int]` | `value` | | | +| `sparse_attention_config.indexer_rope_interleave` | `` | `value` | | | +| `sparse_attention_config.kernel_size` | `Optional[int]` | `value` | | | +| `sparse_attention_config.kt_cache_dtype` | `Optional[str]` | `categorical` | allowlist | `bfloat16`, `float8_e5m2` | +| `sparse_attention_config.page_size` | `Optional[int]` | `value` | | | +| `sparse_attention_config.prompt_budget` | `Optional[int]` | `value` | | | +| `sparse_attention_config.q_split_threshold` | `` | `value` | | | +| `sparse_attention_config.seq_len_threshold` | `Optional[int]` | `value` | | | +| `sparse_attention_config.skip_indexer_for_short_seqs` | `` | `value` | | | +| `sparse_attention_config.topk` | `Optional[int]` | `value` | | | +| `sparse_attention_config.topr` | `Union[int, float, NoneType]` | `value` | | | +| `sparse_attention_config.use_cute_dsl_paged_mqa_logits` | `` | `value` | | | +| `sparse_attention_config.use_cute_dsl_topk` | `` | `value` | | | +| `sparse_attention_config.window_size` | `Optional[int]` | `value` | | | +| `speculative_config.acceptance_length_threshold` | `Optional[Annotated[float, Ge(ge=0)]]` | `value` | | | +| `speculative_config.acceptance_window` | `Optional[Annotated[int, Ge(ge=0)]]` | `value` | | | +| `speculative_config.allow_advanced_sampling` | `` | `value` | | | +| `speculative_config.begin_thinking_phase_token` | `` | `value` | | | +| `speculative_config.decoding_type` | `Literal['AUTO']` | `categorical` | | `AUTO`, `DFlash`, `Draft_Target`, `Eagle3`, `Eagle`, `Lookahead`, `MTP`, `Medusa`, `NGram`, `PARD`, `SA`, `SaveState`, `User_Provided` | +| `speculative_config.dynamic_tree_max_topK` | `Optional[int]` | `value` | | | +| `speculative_config.eagle3_layers_to_capture` | `Optional[Set[int]]` | `value` | | | +| `speculative_config.eagle3_model_arch` | `Literal['llama3', 'mistral_large3']` | `categorical` | | `llama3`, `mistral_large3` | +| `speculative_config.eagle3_one_model` | `Optional[bool]` | `value` | | | +| `speculative_config.eagle_choices` | `Optional[List[List[int]]]` | `value` | | | +| `speculative_config.enable_global_pool` | `` | `value` | | | +| `speculative_config.end_thinking_phase_token` | `` | `value` | | | +| `speculative_config.global_pool_size` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | +| `speculative_config.greedy_sampling` | `Optional[bool]` | `value` | | | +| `speculative_config.is_keep_all` | `` | `value` | | | +| `speculative_config.is_public_pool` | `` | `value` | | | +| `speculative_config.is_use_oldest` | `` | `value` | | | +| `speculative_config.mask_token_id` | `Optional[int]` | `value` | | | +| `speculative_config.max_concurrency` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | +| `speculative_config.max_draft_len` | `Optional[Annotated[int, Ge(ge=0)]]` | `value` | | | +| `speculative_config.max_matching_ngram_size` | `` | `value` | | | +| `speculative_config.max_ngram_size` | `` | `value` | | | +| `speculative_config.max_non_leaves_per_layer` | `Optional[int]` | `value` | | | +| `speculative_config.max_total_draft_tokens` | `Optional[int]` | `value` | | | +| `speculative_config.max_verification_set_size` | `` | `value` | | | +| `speculative_config.max_window_size` | `` | `value` | | | +| `speculative_config.medusa_choices` | `Optional[List[List[int]]]` | `value` | | | +| `speculative_config.mtp_eagle_one_model` | `` | `value` | | | +| `speculative_config.num_eagle_layers` | `Optional[int]` | `value` | | | +| `speculative_config.num_medusa_heads` | `Optional[int]` | `value` | | | +| `speculative_config.num_nextn_predict_layers` | `Optional[int]` | `value` | | | +| `speculative_config.posterior_threshold` | `Optional[float]` | `value` | | | +| `speculative_config.relaxed_delta` | `` | `value` | | | +| `speculative_config.relaxed_topk` | `` | `value` | | | +| `speculative_config.sa_config.enable_global_pool` | `` | `value` | | | +| `speculative_config.sa_config.threshold` | `` | `value` | | | +| `speculative_config.target_layer_ids` | `Optional[List[int]]` | `value` | | | +| `speculative_config.use_dynamic_tree` | `Optional[bool]` | `value` | | | +| `speculative_config.use_mtp_vanilla` | `` | `value` | | | +| `speculative_config.use_rejection_sampling` | `` | `value` | | | +| `speculative_config.use_relaxed_acceptance_for_thinking` | `` | `value` | | | +| `speculative_config.write_interval` | `` | `value` | | | +| `telemetry_config.disabled` | `` | `value` | | | +| `telemetry_config.usage_context` | `` | `categorical` | | `unknown`, `llm_class`, `cli_serve`, `cli_bench`, `cli_eval` | +| `tensor_parallel_size` | `` | `value` | | | +| `tokenizer_mode` | `Literal['auto', 'slow']` | `categorical` | | `auto`, `slow` | +| `trust_remote_code` | `` | `value` | | | diff --git a/docs/source/examples/customization.md b/docs/source/examples/customization.md index 3a7aa5b3f9e8..4c357554f504 100644 --- a/docs/source/examples/customization.md +++ b/docs/source/examples/customization.md @@ -2,51 +2,70 @@ ## Quantization -TensorRT LLM runs quantized models from pre-quantized checkpoints. Use a checkpoint quantized with [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) (for example, the ready-made FP8/NVFP4 checkpoints published on the [NVIDIA Hugging Face hub](https://huggingface.co/nvidia)), and the quantization configuration is detected automatically when the model loads: +TensorRT LLM can quantize the Hugging Face model automatically. By setting the appropriate flags in the `LLM` instance. For example, to perform an Int4 AWQ quantization, the following code triggers the model quantization. Please refer to complete list of [supported flags](https://nvidia.github.io/TensorRT-LLM/_modules/tensorrt_llm/quantization/mode.html#QuantAlgo) and acceptable values. ``` python -from tensorrt_llm import LLM +from tensorrt_llm.llmapi import QuantConfig, QuantAlgo -llm = LLM("nvidia/Llama-3.1-8B-Instruct-FP8") -``` +quant_config = QuantConfig(quant_algo=QuantAlgo.W4A16_AWQ) -Refer to the [quantization feature documentation](../features/quantization.md) for the supported formats per GPU architecture and instructions on quantizing your own model. +llm = LLM(, quant_config=quant_config) +``` ## Sampling -SamplingParams can customize the sampling strategy to control LLM generated responses, such as beam search, temperature, and many others. +SamplingParams can customize the sampling strategy to control LLM generated responses, such as beam search, temperature, and [others](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/llmapi/utils.py#L55-L76). -As an example, to enable beam search with a beam width of 4, configure the engine limit with `max_beam_width` and request beam search through `SamplingParams`: +As an example, to enable beam search with a beam size of 4, set the `sampling_params` as follows: ```python -from tensorrt_llm import LLM, SamplingParams +from tensorrt_llm.llmapi import LLM, SamplingParams, BuildConfig -llm = LLM(, max_beam_width=4) +build_config = BuildConfig() +build_config.max_beam_width = 4 + +llm = LLM(, build_config=build_config) # Let the LLM object generate text with the default sampling strategy, or # you can create a SamplingParams object as well with several fields set manually -sampling_params = SamplingParams(n=4, use_beam_search=True) +sampling_params = SamplingParams(beam_width=4) # current limitation: beam_width should be equal to max_beam_width for output in llm.generate(, sampling_params=sampling_params): print(output) ``` -Refer to the [class documentation](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html#tensorrt_llm.llmapi.SamplingParams) for the complete list of fields. +`SamplingParams` manages and dispatches fields to C++ classes including: + +* [SamplingConfig](https://nvidia.github.io/TensorRT-LLM/_cpp_gen/runtime.html#_CPPv4N12tensorrt_llm7runtime14SamplingConfigE) +* [OutputConfig](https://nvidia.github.io/TensorRT-LLM/_cpp_gen/executor.html#_CPPv4N12tensorrt_llm8executor12OutputConfigE) + +Refer to the [class documentation](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html#tensorrt_llm.llmapi.SamplingParams) for more details. + +## Build Configuration + +Apart from the arguments mentioned above, you can also customize the build configuration with the `build_config` class and other arguments borrowed from the trtllm-build CLI. These build configuration options provide flexibility in building engines for the target hardware and use cases. Refer to the following example: + +```python +llm = LLM(, + build_config=BuildConfig( + max_num_tokens=4096, + max_batch_size=128, + max_beam_width=4)) +``` +Refer to the [buildconfig documentation](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/builder.py#L470-L501) for more details. ## Runtime Customization -Runtime behavior such as KV cache management and GPU memory allocation can be customized with dedicated configuration classes like `kv_cache_config` and `peft_cache_config` passed to the `LLM` constructor. Refer to the following example: +Similar to `build_config`, you can also customize the runtime configuration with the `runtime_config`, `peft_cache_config` or other [arguments](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/llmapi/llm_utils.py#L186-L223) borrowed from the Executor APIs. These runtime configuration options provide additional flexibility with respect to KV cache management, GPU memory allocation and so on. Refer to the following example: + ```python -from tensorrt_llm import LLM -from tensorrt_llm.llmapi import KvCacheConfig +from tensorrt_llm.llmapi import LLM, KvCacheConfig llm = LLM(, kv_cache_config=KvCacheConfig( free_gpu_memory_fraction=0.8)) ``` -Refer to the [LLM API reference](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) for all available configuration classes. - ## Tokenizer Customization By default, the LLM API uses transformers’ `AutoTokenizer`. You can override it with your own tokenizer by passing it when creating the LLM object. Refer to the following example: @@ -71,8 +90,8 @@ for output in llm.generate([32, 12]): For performance considerations, you can disable the tokenizer by passing `skip_tokenizer_init=True` when creating `LLM`. In this case, `LLM.generate` and `LLM.generate_async` will expect prompt token ids as input. Refer to the following example: ```python -llm = LLM(, skip_tokenizer_init=True) -for output in llm.generate([[32, 12]]): +llm = LLM() +for output in llm.generate([[32, 12]], skip_tokenizer_init=True): print(output) ``` diff --git a/docs/source/features/auto_deploy/advanced/testing_strategy.md b/docs/source/features/auto_deploy/advanced/testing_strategy.md index b60eada7e722..d65841862ee6 100644 --- a/docs/source/features/auto_deploy/advanced/testing_strategy.md +++ b/docs/source/features/auto_deploy/advanced/testing_strategy.md @@ -117,8 +117,8 @@ Format: `path/to/test_file.py::test_function_name[param_id]` Example from `l0_a30.yml`: ```yaml -- accuracy/test_llm_api_pytorch.py::TestLlama3_1_8B::test_auto_dtype -- unittest/_torch/modeling -k "modeling_llama" +- accuracy/test_cli_flow.py::TestLlama3_1_8BInstruct::test_medusa_fp8_prequantized +- examples/test_multimodal.py::test_llm_multimodal_general[Qwen2-VL-7B-Instruct-pp:1-tp:1-float16-bs:1-cpp_e2e:False-nb:4] ``` ### Example: Adding an Accuracy Test diff --git a/docs/source/features/checkpoint-loading.md b/docs/source/features/checkpoint-loading.md index fa617b43dbdf..6caa3a0fcd95 100644 --- a/docs/source/features/checkpoint-loading.md +++ b/docs/source/features/checkpoint-loading.md @@ -83,15 +83,6 @@ Currently, HF checkpoint loader is the primary built-in format, supporting: - **Configuration parser** - Parsing HF stored configuration information to TRTLLM `ModelConfig` object - **Weights Mapping** - Converting HF weights into TRTLLM compatible representation -### ModelExpress (MX) Loading Path - -The PyTorch backend can use ModelExpress (MX) for peer-to-peer weight transfer -from a running TensorRT-LLM source instance before falling back to Hugging Face -checkpoint loading. Selecting MX does not require an MX-specific on-disk -checkpoint or conversion of the Hugging Face checkpoint. For installation, MX -service deployment, and configuration details, see -[ModelExpress (MX) Checkpoint Loading](./model-express.md). - ## Using Checkpoint Loaders ### Basic Usage diff --git a/docs/source/features/disagg-serving.md b/docs/source/features/disagg-serving.md index 9ed621158a26..f38e3c818f57 100644 --- a/docs/source/features/disagg-serving.md +++ b/docs/source/features/disagg-serving.md @@ -94,19 +94,11 @@ The optimizations required for KV cache transmission vary depending on whether i ### Unique Global Request ID -The context and generation phases of one request must share a single request ID: the ctx↔gen KV-cache transfer is keyed by it, so a collision (two in-flight requests with the same ID) corrupts the transfer. This shared ID is carried on `DisaggregatedParams.disagg_request_id`. - -The disaggregated server generates this ID itself as a **snowflake** — a self-contained 64-bit positive integer that is unique without any cross-process coordination. The bit layout is: - -``` -[ 0 (1 bit) | timestamp_ms (39 bits) | node_id (8 bits) | process_id (6 bits) | counter (10 bits) ] -``` - -- `node_id` (0–255) identifies the node (defaults to a hash of the MAC address; overridable via `node_id` in the disaggregated config). -- `process_id` (0–63) identifies the orchestrator process on that node. In a [coordinator + worker fleet](#coordinator-and-worker-fleet) each fleet worker receives a distinct value, so co-located workers never emit the same ID in the same millisecond. It is set from the `TRTLLM_DISAGG_WORKER_PROCESS_ID` environment variable (assigned automatically per worker by the launcher). -- The `(node_id, process_id)` pair therefore makes the ID unique across all orchestrator processes without a shared counter or an extra network round trip — each worker mints its own IDs locally. - -Global disaggregated IDs occupy the range `[1 << 40, 2**63)`; worker-local and warm-up request IDs occupy the disjoint range `[0, 1 << 40)`, so the two never collide. If a client supplies its own positive `disagg_request_id`, that value is used verbatim and must be globally unique; when unset, the server mints a snowflake ID as above. +A disaggregated-serving request can provide a unique global request ID via `DisaggregatedParams.disagg_request_id`. +When this field is a positive integer, the context and generation requests share that value as their internal request ID, which enables end-to-end tracking. +To avoid collisions with worker-local or warm-up requests, it is recommended to use a value larger than `1 << 42 = 4398046511104`. +If the field is unset or non-positive, the context and generation requests instead receive separate local sequence IDs, rotating within the range `(0, 1<<42)`, assigned by the respective workers. When `disagg_request_id` is specified, do not route the context and generation requests to the same worker. +This field is optional at present; however, some forthcoming features will depend on this unique identifier. ## Usage @@ -209,7 +201,7 @@ When routing requests to the context servers, the disaggregated server will mark when routing requests to the generation servers, the disaggregated server will mark the requests as "generation-only" to skip the context phase. Clients can then send requests to the disaggregated server at `localhost:8000`, which is an OpenAI-compatible endpoint. For example, you can send requests to the disaggregated server using curl: -``` +```bash curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ @@ -241,95 +233,6 @@ Example (two-node deployment): - **Client entrypoint** - Send requests or use a load balancer forwarding to `node-a:8000` and `node-b:8000` -### Coordinator and Worker Fleet - -A single disaggregated server process is itself a single-threaded orchestrator and can become a throughput bottleneck (it terminates every client connection, runs routing, and proxies the ctx→gen hop). To scale the orchestrator on one node without standing up multiple independent instances, `trtllm-serve disaggregated` can run a **fleet** of stateless disaggregated-server worker processes behind a shared **coordinator**. - -The two roles split as follows: - -- **Coordinator** — a single process that owns all cluster state: the ctx/gen routers, worker readiness, and (for the KV-cache-aware router) the single ZMQ event-ingest endpoint. It exposes an internal coordination API (`/select`, `/finish`, `/cluster_info`, `/health`). -- **Fleet workers** — `num_workers` stateless disaggregated servers that share the public port via `SO_REUSEPORT` (each worker is its own process binding the same port, so the kernel load-balances incoming connections across them by 4-tuple hash). Each holds a lightweight delegating client: it computes the routing key locally (e.g. block hashes) and delegates the placement decision to the coordinator over HTTP. Workers own no routing state, so routing stays globally consistent no matter which worker terminates a connection. Each worker also gets a distinct `process_id` for the [global request ID](#unique-global-request-id). - -This is controlled by two fields in the disaggregated config: - -- `num_workers` (int, default `1`) — number of disaggregated-server worker processes to run on the public port. -- `disagg_coordinator_url` (str, optional) — URL of an already-running coordinator. When set, this process starts **no** coordinator and its fleet delegates to that external one. - -The three resulting topologies: - -| `num_workers` | `disagg_coordinator_url` | Behavior | -|---------------|--------------------------|----------| -| `1` | unset | Single self-contained server with an in-process coordinator (the default; unchanged from earlier examples). | -| `> 1` | unset | An **implicit** coordinator starts in this process (on `port - 1`) and a fleet of `num_workers` delegating servers runs on the public port. | -| any | set | **No** coordinator starts here; a fleet of `num_workers` delegating servers points at the external `disagg_coordinator_url`. | - -```{note} -The fleet is most useful with a *stateful* router (`kv_cache_aware`, `conversation`) where placement must be globally consistent — that decision is delegated to the coordinator. With a *stateless* router (`round_robin`, `load_balancing`) each worker simply places locally and no coordinator round-trip occurs. -``` - -#### Example: implicit coordinator + 4-worker fleet - -Extend the `disagg_config.yaml` from the [trtllm-serve](#trtllm-serve) example with `num_workers` and a router type: - -```yaml -hostname: localhost -port: 8000 -backend: pytorch -# Run 4 stateless disaggregated-server workers on port 8000, with an implicit -# coordinator started in-process on port 7999 (port - 1). -num_workers: 4 -context_servers: - num_instances: 2 - urls: - - "localhost:8001" - - "localhost:8002" - router: - type: kv_cache_aware -generation_servers: - num_instances: 1 - urls: - - "localhost:8003" - router: - type: kv_cache_aware -``` - -Launch it exactly as before — the coordinator and fleet are started for you: - -```bash -trtllm-serve disaggregated -c disagg_config.yaml -``` - -Clients still send requests to the public endpoint (`localhost:8000`); the fleet transparently delegates routing to the coordinator. - -#### Example: external coordinator - -To point a fleet at a coordinator already running elsewhere (for example, one shared across nodes), set `disagg_coordinator_url` and omit the coordinator from this process: - -```yaml -hostname: localhost -port: 8000 -backend: pytorch -num_workers: 4 -disagg_coordinator_url: "http://coordinator-host:7999" -context_servers: - num_instances: 2 - urls: - - "localhost:8001" - - "localhost:8002" - router: - type: kv_cache_aware -generation_servers: - num_instances: 1 - urls: - - "localhost:8003" - router: - type: kv_cache_aware -``` - -```{note} -A fleet worker fails fast if its coordinator is unreachable: on startup it probes the coordinator's `/cluster_info` with bounded retry (up to `--server_start_timeout` seconds) and exits with an error rather than coming up and returning `Cluster is not ready` for every request. -``` - ## Environment Variables TRT-LLM uses some environment variables to control the behavior of disaggregated service. diff --git a/docs/source/features/embeddings.md b/docs/source/features/embeddings.md deleted file mode 100644 index d34199092f23..000000000000 --- a/docs/source/features/embeddings.md +++ /dev/null @@ -1,165 +0,0 @@ -# Embeddings (Encoder-Only Models) - -`trtllm-serve` can serve **encoder-only models** (BERT-style classifiers, reward -models, text-embedding models) through an OpenAI-compatible **`POST /v1/embeddings`** -endpoint with **native dynamic batching** — coalescing many independent concurrent -requests into a single forward pass for high throughput, the way the NVIDIA Triton -Inference Server dynamic batcher does. - -This replaces the need to run a separate Triton Inference Server in front of an -encoder model: point your existing OpenAI-style embeddings client at `trtllm-serve` -and it works unchanged. - -## Quick start - -Launch an embeddings server with the `embeddings` subcommand: - -```bash -trtllm-serve embeddings \ - --max_batch_size 32 \ - --max_queue_delay 0.005 \ - --max_queue_size 2048 \ - --host 0.0.0.0 --port 8000 -``` - -Send a request with any OpenAI-compatible client or `curl`: - -```bash -curl http://localhost:8000/v1/embeddings \ - -H "Content-Type: application/json" \ - -d '{"model": "", "input": ["hello world", "foo bar"]}' -``` - -```python -from openai import OpenAI - -client = OpenAI(base_url="http://localhost:8000/v1", api_key="tensorrt_llm") -resp = client.embeddings.create(model="", input=["hello world", "foo bar"]) -for item in resp.data: - print(item.index, len(item.embedding)) -``` - -The response is the standard OpenAI embeddings shape: - -```json -{ - "object": "list", - "data": [ - {"object": "embedding", "index": 0, "embedding": [ ... ]}, - {"object": "embedding", "index": 1, "embedding": [ ... ]} - ], - "model": "", - "usage": {"prompt_tokens": 8, "total_tokens": 8} -} -``` - -## Request fields - -The endpoint accepts the standard OpenAI `/v1/embeddings` fields: - -| Field | Type | Notes | -|---|---|---| -| `model` | str | Model name. | -| `input` | str \| list[str] \| list[int] \| list[list[int]] | Text(s) or pre-tokenized token-id list(s). | -| `encoding_format` | `"float"` (default) \| `"base64"` | `base64` packs little-endian float32 values. | -| `dimensions` | int (optional) | Matryoshka output size. Only supported by Matryoshka-trained text-embedding models; rejected with `400` otherwise. None of the served models are Matryoshka-trained (BERT classifiers / reward models emit label/score tensors; Qwen3-Embedding emits a fixed-width pooled vector), so this is currently always rejected. | -| `user` | str (optional) | Ignored; accepted for compatibility. | -| `add_special_tokens` | bool (default `true`) | TRT-LLM extension. Encoder models such as BERT generally need their special tokens (e.g. `[CLS]`/`[SEP]`) added during tokenization. | - -There are **no required TRT-LLM-specific request fields** — existing OpenAI-compatible -embeddings clients work by pointing at the `trtllm-serve` URL. - -## Dynamic batching - -A lightweight in-server batcher coalesces concurrent requests in front of the -encoder forward pass. It exposes three knobs that mirror the Triton dynamic batcher: - -| `trtllm-serve embeddings` flag | Behavior | Triton equivalent | -|---|---|---| -| `--max_batch_size` | Upper bound on the number of requests fused into one forward pass. A batch reaching this size is dispatched immediately. | maximum / `preferred_batch_size` | -| `--max_queue_delay` (seconds) | Hold window: how long an incoming request waits for others to join its batch before dispatch. | `max_queue_delay_microseconds` | -| `--max_queue_size` | Maximum number of in-flight queued requests. Further requests are rejected with HTTP 429 (backpressure). | `default_queue_policy.max_queue_size` | - -A batch is dispatched as soon as **any** of these fires: it reaches `--max_batch_size`, -adding the next request would exceed the engine's `--max_num_tokens` budget, or the -`--max_queue_delay` hold window elapses. - -### Migrating from the Triton Inference Server dynamic batcher - -If you currently serve an encoder model with the Triton `inflight_batcher_llm` backend -and a `config.pbtxt` `dynamic_batching { ... }` block, map the settings directly: - -| Triton `config.pbtxt` | `trtllm-serve embeddings` | -|---|---| -| `dynamic_batching.preferred_batch_size` / model max batch | `--max_batch_size` | -| `dynamic_batching.max_queue_delay_microseconds` | `--max_queue_delay` (in **seconds**, e.g. `100 µs` → `0.0001`) | -| `dynamic_batching.default_queue_policy.max_queue_size` | `--max_queue_size` | - -Adopt the same values you tuned in Triton as a starting point, then adjust for your -latency/throughput budget. - -## Error handling - -| Condition | HTTP status | -|---|---| -| Input longer than `--max_seq_len` | 400 | -| Request queue full (`--max_queue_size` reached) | 429 | -| Invalid request body | 400 | - -Embedding responses are unary (non-streaming). - -## Output semantics and scope - -The endpoint is **model-output-agnostic**: it returns whatever per-request vector the -model emits, serialized into the OpenAI embeddings schema. - -- **Classifier / reward models** (e.g. a BERT sequence classifier): the returned - vector is the model's class-logit / score vector (`[num_labels]`). -- **Text-embedding models** — the **Qwen3-Embedding family** (`Qwen3-Embedding-0.6B`, - `-4B`, `-8B`) is supported. These ship as a `Qwen3ForCausalLM` decoder plus a - sentence-transformers pooling pipeline; the embeddings server detects this and serves - the **L2-normalized last-token hidden state** (a `[hidden_size]` sentence-embedding - vector — 1024 / 2560 / 4096 respectively), with no extra flags. A configurable pooling - method (CLS / mean) for other sentence-transformers backbones remains a follow-up. - -Notes: - -- The embeddings path uses the synchronous `llm.encode()` fast path - (`EncoderExecutor`): a single forward pass per batch, **no KV cache, sampler, or - decode loop**. -- One encoder model per server instance. Generation and embedding modes are not mixed - in one server. -- **Single-GPU per server.** The encode path runs in-process and does not use the - multi-GPU worker proxy, so the `embeddings` command does not expose tensor/pipeline - parallelism flags (if a `--config` file sets them, startup fails with a clear error). - To scale out, see [Scaling out across GPUs](#scaling-out-across-gpus) below. -- A single in-server worker drives the GPU (no `num_workers` knob): the GPU serializes - forwards and the underlying executor is not safe for concurrent calls. Increase - throughput with `--max_batch_size` / `--max_queue_delay`, not more workers. - -## Scaling out across GPUs - -Embedding / encoder-only models are usually small and fit comfortably on a single GPU. -The recommended way to use more GPUs is therefore **data parallelism**: run one -single-GPU `trtllm-serve embeddings` instance per GPU and put a load balancer in front -of them. There is no cross-GPU communication, so throughput scales close to linearly -with the number of replicas. - -```bash -# One replica per GPU (8x B200 example), each on its own port. -for i in $(seq 0 7); do - CUDA_VISIBLE_DEVICES=$i trtllm-serve embeddings --port $((8000 + i)) & -done -# Then point any HTTP load balancer (nginx, k8s Service, etc.) at ports 8000-8007. -``` - -**Tensor / pipeline parallelism** (sharding a single model across GPUs) is only needed -for an embedding model too large to fit on one GPU — uncommon for encoder-only models. -It is **not yet supported** by the `embeddings` command and is planned as a follow-up. - -## Relationship to `llm.encode()` - -The server reuses the existing Python `llm.encode()` API -(`LLM(..., encode_only=True)`) under the hood; the only addition is the async -coalescing layer plus the HTTP surface. The synchronous `llm.encode()` API continues -to work unchanged for direct Python callers. diff --git a/docs/source/features/feature-combination-matrix.md b/docs/source/features/feature-combination-matrix.md index d91322fc8c11..b56c1b219af9 100644 --- a/docs/source/features/feature-combination-matrix.md +++ b/docs/source/features/feature-combination-matrix.md @@ -14,7 +14,7 @@ | Speculative Decoding — Linear | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes | --- | | | | | | | | | | | Speculative Decoding — Dynamic Trees | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes | No | --- | | | | | | | | | | Speculative Decoding — Legacy Path (NGram, user-provided) | Yes | Yes | Yes | No | Yes | No | Yes | Yes | Yes | No | No | --- | | | | | | | | -| Torch Sampler | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes (MTP dynamic tree: greedy only) | Yes | --- | | | | | | | +| Torch Sampler | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | --- | | | | | | | | TLLM C++ Sampler | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No | No | --- | | | | | | | KV Cache Reuse | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | --- | | | | | | Sliding Window Attention | Yes | Yes | Yes | Yes | Yes | Untested | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | --- | | | | diff --git a/docs/source/features/model-express.md b/docs/source/features/model-express.md deleted file mode 100644 index 3007b1402c4a..000000000000 --- a/docs/source/features/model-express.md +++ /dev/null @@ -1,152 +0,0 @@ - - -# ModelExpress (MX) Checkpoint Loading - -The MX checkpoint-loading integration is intended to reduce repeated disk -reads when multiple TensorRT LLM workers load the same model. A worker that -loads from disk can publish its weights as an MX source, and later workers can -receive those weights directly through MX. - -TensorRT LLM can use ModelExpress (MX) as a checkpoint-loading path for -PyTorch backend deployments. `checkpoint_format="MX"` selects this loading -path; it does not identify an MX-specific on-disk checkpoint format, and no -checkpoint conversion is required. TensorRT LLM attempts to fetch compatible -weights from another running TensorRT LLM instance through the MX server. If -no compatible source is available, or if MX transfer fails, loading falls back -to the provided Hugging Face checkpoint. - -## Current Support Scope - -The post-transform MX receive path currently supports only -`LlamaForCausalLM` with transform protocol version 1. TensorRT LLM publishes -post-transform weights together with source-identity and layout metadata. A -receiver whose model family is not allow-listed does not consume those bytes; -it falls back to the standard Hugging Face checkpoint path. - -Loads that require a separately loaded draft model also fall back to the -standard checkpoint path. Target-plus-draft post-transform transfer remains -disabled until layout state is tracked and qualified independently for each -submodel. - -### Adding a Model Family - -Support for another model family requires a focused qualification change: - -1. Audit every post-load hook in the family and its nested modules. Move - structural wiring to `setup_aliases()`, one-time tensor-layout changes to - `transform_weights()`, and process-local derived state to - `cache_derived_state()`. -2. Verify that every one-time transform is guarded by `_weights_transformed` - and that the staged receiver can skip `transform_weights()` without - changing aliases, derived state, tensor layout, or outputs. -3. Add the model class and transform protocol version to the MX staged-receiver - allow-list only after full-load and staged-load equivalence tests pass. -4. Cover compatible transfer, source-identity mismatch, unsupported layout or - protocol, and non-allow-listed fallback. Keep target-plus-draft loading - disabled unless that combination has its own mixed-layout tests. -5. Run a real ModelExpress donor/receiver test with the model configurations - being claimed, including the supported quantization and TP/PP/EP layouts. - Compare deterministic output token IDs with the standard Hugging Face load - path before documenting the family as supported. - -## Installation - -The official TensorRT LLM release container includes the MX Python client. No -additional Python package installation is required in that container. MX -remains opt-in at runtime: TensorRT LLM uses the client only when the MX -checkpoint-loading path and a server URL are configured. Installing the client -does not expand the model support scope described above. - -For pip installations outside the official release container, install the MX -Python client through the optional `mx` extra: - -```bash -pip install "tensorrt-llm[mx]" -``` - -The extra pins the ModelExpress client to version `0.4.1`, matching the client -API qualified by this integration. Deploy a compatible MX server version. -The extra can be added to an existing TensorRT LLM installation. If the MX -loading path is configured but the client cannot be imported, TensorRT LLM -fails with an actionable installation message instead of silently loading from -the Hugging Face checkpoint. Source discovery and transfer failures continue to -use the Hugging Face fallback described above. - -## Deploy the MX Service - -Deploy the MX server and its Redis metadata backend independently of -TensorRT LLM. One MX service can be shared by multiple TensorRT LLM launches, -provided every instance can reach the MX endpoint. TensorRT LLM does not start, -stop, or otherwise manage either service. - -The following commands illustrate a standalone Docker deployment. Production -deployments should manage service lifecycle, persistence, networking, and -security according to their environment. - -```bash -docker network create modelexpress -docker run -d --name modelexpress-redis \ - --network modelexpress \ - redis:8-alpine -docker run -d --name modelexpress-server \ - --network modelexpress \ - -p 8001:8001 \ - -e MODEL_EXPRESS_SERVER_PORT=8001 \ - -e MODEL_EXPRESS_LOG_LEVEL=info \ - -e MX_METADATA_BACKEND=redis \ - -e REDIS_URL=redis://modelexpress-redis:6379 \ - nvcr.io/nvidia/ai-dynamo/modelexpress-server:0.4.1 -``` - -## Configure TensorRT LLM - -Select the MX checkpoint-loading path and provide the MX server URL in a -`trtllm-serve` config. The model argument remains a standard Hugging Face model -ID or checkpoint path: - -```yaml -checkpoint_format: MX -mx_config: - server_url: http://mx-server.example.com:8001 -``` - -```bash -trtllm-serve /path/to/model --config config.yaml -``` - -The `MODEL_EXPRESS_URL` environment variable can also provide the server URL -when `mx_config.server_url` is not set. - -Multiple TensorRT LLM launches can use the same configuration. A worker that -does not find a compatible source loads from Hugging Face storage and publishes -its weights through MX. Later compatible workers can receive those weights by -P2P transfer. - -If neither `mx_config.server_url` nor `MODEL_EXPRESS_URL` is set, MX transfer is -not attempted and checkpoint loading falls back to the standard Hugging Face -path. - -## Configuration - -| Field | Default | Description | -|-------|---------|-------------| -| `mx_config.server_url` | `null` | URL of the separately managed MX server. | -| `mx_config.server_query_timeout_s` | `null` | Timeout for MX source discovery. When unset, TensorRT LLM uses a short fallback cap when no source exists and otherwise lets MX wait for long donor loads. | - -## Notes and Limitations - -- Post-transform MX reception is currently limited to the Llama model family. - Other model families safely fall back to Hugging Face loading until they are - explicitly qualified and added to the staged-receiver allow-list. -- The MX server and Redis lifecycle is external to TensorRT LLM. Every - TensorRT LLM instance must be able to reach the configured MX server URL. -- The MX server coordinates source discovery but does not store model weights. - A source TensorRT LLM process must remain running and network-reachable until - receiver transfers finish. -- The first worker may still load weights from disk if no compatible MX source - is already registered. -- This page describes the MX checkpoint-loading path only. GPU Memory Service - (GMS) integration is configured separately. diff --git a/docs/source/features/multi-modality.md b/docs/source/features/multi-modality.md index 6d6512a38d17..e2be649bf44a 100644 --- a/docs/source/features/multi-modality.md +++ b/docs/source/features/multi-modality.md @@ -26,16 +26,6 @@ Further optimizations are under development and will be updated as they become a Please refer to the latest multimodal [support matrix](../models/supported-models.md#multimodal-feature-support-matrix-pytorch-backend). -## Optional dependencies - -Video input decoding uses OpenCV (`cv2`) and is not installed by default. Install it only for models with video inputs: - -```bash -pip install opencv-python-headless -``` - -Image-only and text-only workflows do not require this package. - ## Examples The following examples demonstrate how to use TensorRT LLM's multimodal support in various scenarios, including quick run examples, serving endpoints, and performance benchmarking. diff --git a/docs/source/features/sampling.md b/docs/source/features/sampling.md index 72ad4a62ad41..fc3cb3ed9324 100644 --- a/docs/source/features/sampling.md +++ b/docs/source/features/sampling.md @@ -75,8 +75,7 @@ llm.generate(["Hello, my name is", * The sampling is controlled via `SamplingParams`. -* By default (`temperature = top_p = top_k = None`), greedy sampling is used - (unless top-p decay is active, see below). +* By default (`temperature = top_p = top_k = None`), greedy sampling is used. * If either `temperature = 0`, `top_p = 0`, and/or `top_k = 1`, is specified, sampling is greedy, irrespective of the values of the remaining parameters. @@ -102,28 +101,17 @@ llm.generate(["Hello, my name is", * The implementation does not guarantee any particular treatment of tied probabilities. -* Top-P decay is supported: if `top_p_decay < 1` is specified, the effective `top_p` is - multiplied by `top_p_decay` after every sampled token, bounded from below by `top_p_min` - (default `1e-6`), and reset to the initial `top_p` whenever the token `top_p_reset_ids` - is sampled (default `-1`, which never matches a token). Out-of-range values - (`top_p_decay` or `top_p_min` outside `(0, 1]`, negative `top_p_reset_ids`) are rejected. - - * An active top-p decay implies top-p sampling even if `top_p` is unspecified or `top_p = 1` - (the initial `top_p` then defaults to 1). However, explicitly requested greedy sampling - (`temperature = 0`, `top_p = 0`, and/or `top_k = 1`) takes precedence over top-p decay. - - * Top-P decay is not supported in combination with beam search or with speculative decoding - modes that route draft tokens through the Torch Sampler; such requests are rejected. - ### Performance The Torch Sampler leverages the optimized sampling kernels provided by -[FlashInfer](https://docs.flashinfer.ai/api/sampling.html), which is a required -dependency for the Torch Sampler. The sampler also uses the -[sorting-free implementations](https://flashinfer.ai/2025/03/10/sampling.html) +[FlashInfer](https://docs.flashinfer.ai/api/sampling.html). The sampler +also uses the [sorting-free implementations](https://flashinfer.ai/2025/03/10/sampling.html) whenever possible. This optimization does not compute the complete set of token sampling probabilities (after top-k / top-p masking etc.), which typically can be omitted unless requested by the user or required for speculative decoding (rejection sampling). +In case of unexpected problems, the use of FlashInfer in Torch Sampler can +be disabled via the `disable_flashinfer_sampling` config option (note that this option is likely +to be removed in a future TensorRT LLM release). Moreover, Torch Sampler internally batches requests with compatible sampling parameters. This can greatly reduce the overall latency of the sampling step when request batches are comprised diff --git a/docs/source/index.rst b/docs/source/index.rst index 15cfe320bd53..69f526d94eb6 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -65,7 +65,6 @@ Welcome to TensorRT LLM's Documentation! features/feature-combination-matrix.md features/attention.md features/disagg-serving.md - features/embeddings.md features/kvcache.md features/long-sequence.md features/lora.md @@ -80,7 +79,6 @@ Welcome to TensorRT LLM's Documentation! features/guided-decoding.md features/speculative-decoding.md features/checkpoint-loading.md - features/model-express.md features/auto_deploy/auto-deploy.md features/auto_deploy/transforms.rst features/ray-orchestrator.md diff --git a/docs/source/installation/installation-guide.md b/docs/source/installation/installation-guide.md index c01ad3c0ebd4..76f65da3871b 100644 --- a/docs/source/installation/installation-guide.md +++ b/docs/source/installation/installation-guide.md @@ -37,15 +37,15 @@ Tested on Ubuntu 24.04. Before the pre-built Python wheel can be installed via `pip`, a few prerequisites must be put into place: -Install CUDA Toolkit 13.2 following the [CUDA Installation Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) +Install CUDA Toolkit 13.1 following the [CUDA Installation Guide for Linux](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) and make sure `CUDA_HOME` environment variable is properly set. -The `cuda-compat-13-2` package may be required depending on your system's NVIDIA GPU +The `cuda-compat-13-1` package may be required depending on your system's NVIDIA GPU driver version. For additional information, refer to the [CUDA Forward Compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html). ```bash # By default, PyTorch CUDA 12.8 package is installed. Install PyTorch CUDA 13.0 package to align with the CUDA version used for building TensorRT LLM wheels. -pip3 install torch==2.11.0 torchvision --index-url https://download.pytorch.org/whl/cu130 +pip3 install torch==2.10.0 torchvision --index-url https://download.pytorch.org/whl/cu130 sudo apt-get -y install libopenmpi-dev @@ -75,7 +75,7 @@ pip3 uninstall nvidia-cutlass-dsl nvidia-cutlass-dsl-libs-base nvidia-cutlass-ds pip3 install --ignore-installed pip setuptools wheel && pip3 install tensorrt_llm ``` -> **Note:** The TensorRT LLM wheel on PyPI is built with the [public PyTorch package](https://pypi.org/project/torch/). This version may be incompatible with the NVIDIA NGC PyTorch container, which uses a different PyTorch build. If you are using the NGC PyTorch container, install the wheel built specifically for that container using the `+ngcpytorch{YYMM}` local version suffix, where `YYMM` is derived from the container tag (e.g., `pytorch:26.02` → `ngcpytorch2602`): +> **Note:** The TensorRT LLM wheel on PyPI is built with PyTorch 2.10.0. This version may be incompatible with the NVIDIA NGC PyTorch container, which uses a more recent PyTorch build. If you are using the NGC PyTorch container, install the wheel built specifically for that container using the `+ngcpytorch{YYMM}` local version suffix, where `YYMM` is derived from the container tag (e.g., `pytorch:26.02` → `ngcpytorch2602`): > > ```bash > # Example: install TensorRT LLM 1.3.0rc16 inside the pytorch:26.02 NGC container diff --git a/docs/source/legacy/performance/perf-benchmarking.md b/docs/source/legacy/performance/perf-benchmarking.md index 8d1567aee020..4fc460596f22 100644 --- a/docs/source/legacy/performance/perf-benchmarking.md +++ b/docs/source/legacy/performance/perf-benchmarking.md @@ -202,7 +202,7 @@ can simply read a line and assume a complete entry. When creating a dataset, be JSON entry is on every line. ``` -In order to prepare a synthetic dataset, you can use the provided script in the `benchmarks` +In order to prepare a synthetic dataset, you can use the provided script in the `benchmarks/cpp` directory. For example, to generate a synthetic dataset of 1000 requests with a uniform ISL/OSL of 128/128 for [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B), run: @@ -481,7 +481,7 @@ The PyTorch workflow supports benchmarking with LoRA (Low-Rank Adaptation) adapt Use `prepare_dataset.py` with LoRA-specific options to generate requests with LoRA metadata: ```shell -python3 benchmarks/prepare_dataset.py \ +python3 benchmarks/cpp/prepare_dataset.py \ --stdout \ --rand-task-id 0 1 \ --tokenizer /path/to/tokenizer \ @@ -555,7 +555,7 @@ To benchmark multi-modal models with PyTorch workflow, you can follow the simila First, prepare the dataset: ``` -python ./benchmarks/prepare_dataset.py \ +python ./benchmarks/cpp/prepare_dataset.py \ --tokenizer Qwen/Qwen2-VL-2B-Instruct \ --stdout \ dataset \ @@ -846,7 +846,7 @@ The following table summarizes the commands needed for running benchmarks: | Scenario | Phase | Command | | - | - | - | -| Dataset | Preparation | `python benchmarks/prepare_dataset.py --stdout --tokenizer $HF_MODEL token-norm-dist --input-mean $ISL --output-mean $OSL --input-stdev 0 --output-stdev 0 --num-requests $NUM_REQUESTS > $DATASET_PATH` | +| Dataset | Preparation | `python benchmarks/cpp/prepare_dataset.py --stdout --tokenizer $HF_MODEL token-norm-dist --input-mean $ISL --output-mean $OSL --input-stdev 0 --output-stdev 0 --num-requests $NUM_REQUESTS > $DATASET_PATH` | | Throughput | Build | `trtllm-bench --model $HF_MODEL build --dataset $DATASET_PATH` | | Throughput | Benchmark | `trtllm-bench --model $HF_MODEL throughput --dataset $DATASET_PATH --engine_dir $ENGINE_DIR` | | Latency | Build | See [section about building low latency engines](#low-latency-tensorrt-llm-engine-for-llama-3-70b) | diff --git a/docs/source/legacy/performance/performance-tuning-guide/benchmarking-default-performance.md b/docs/source/legacy/performance/performance-tuning-guide/benchmarking-default-performance.md index 7277b0afa5b0..17cb9aef45ba 100644 --- a/docs/source/legacy/performance/performance-tuning-guide/benchmarking-default-performance.md +++ b/docs/source/legacy/performance/performance-tuning-guide/benchmarking-default-performance.md @@ -86,7 +86,7 @@ The README in the examples folder for supported models walks through building en `trtllm-bench` expects to be passed in a dataset of requests to run through the model. This guide creates a dummy dataset of 1000 requests with every request having input and output sequence length of 2048. TensorRT-LLM provides the `prepare_dataset.py` script to produce the dataset. To use it clone the TensorRT-LLM Repo and run the following command: -`python benchmarks/prepare_dataset.py --stdout --tokenizer /path/to/hf/Llama-3.3-70B-Instruct/ token-norm-dist --input-mean 2048 --output-mean 2048 --input-stdev 0 --output-stdev 0 --num-requests 1000 > synthetic_2048_2048.txt` +`python benchmarks/cpp/prepare_dataset.py --stdout --tokenizer /path/to/hf/Llama-3.3-70B-Instruct/ token-norm-dist --input-mean 2048 --output-mean 2048 --input-stdev 0 --output-stdev 0 --num-requests 1000 > synthetic_2048_2048.txt` `trtllm-bench` can also take in real data, see [`trtllm-bench` documentation](../perf-benchmarking.md) for more details on the required format. diff --git a/docs/source/models/encoder-decoder.md b/docs/source/models/encoder-decoder.md deleted file mode 100644 index 30fa8d5695b9..000000000000 --- a/docs/source/models/encoder-decoder.md +++ /dev/null @@ -1,580 +0,0 @@ - - -# Use encoder-decoder models with the PyTorch backend - -TensorRT LLM can run supported Hugging Face encoder-decoder checkpoints directly -with the PyTorch backend. You do not need to convert the checkpoint or build a -TensorRT engine. The LLM API treats the supplied prompt as the encoder input and -automatically starts the decoder with the checkpoint's -`decoder_start_token_id` or `bos_token_id`. - -This guide covers text-to-text generation with the following Hugging Face -architectures: - -| Hugging Face architecture | Model families and examples | -| --- | --- | -| `T5ForConditionalGeneration` | T5, Flan-T5, and ByT5, for example `google/flan-t5-small` | -| `BartForConditionalGeneration` | BART checkpoints | -| `MBartForConditionalGeneration` | mBART checkpoints | - -Whisper is not currently supported by the PyTorch encoder-decoder path. Support -is in progress. - -mBART architecture loading is available. When a BART or mBART checkpoint -defines `forced_bos_token_id`, the PyTorch backend seeds that token in the -decoder prefix. Source- and target-language selection remains -checkpoint-specific, so validate the configured language tokens before -deployment. Refer to [Understand BART and mBART decoder tokens](#understand-bart-and-mbart-decoder-tokens) -for BOS, EOS, and output-limit behavior. - -## Feature support - -The following table describes the supported and recommended configurations. - -| Feature | Support | Notes | -| --- | --- | --- | -| KV cache manager V1 | Yes; recommended | This is the default. It supports greedy decoding, beam search, batching, the overlap scheduler, decoder CUDA graphs, and tensor parallelism. | -| KV cache manager V2 | Yes | Set `use_kv_cache_manager_v2=True`. It currently requires `max_beam_width=1`, so use greedy or sampling with a single sequence rather than beam search. | -| Greedy decoding | Yes | Set `temperature=0.0`. | -| Beam search | Yes with V1 | Configure `max_beam_width` when constructing `LLM`, then set `use_beam_search=True` in `SamplingParams`. | -| Attention backend | `TRTLLM` | Use this backend for encoder-decoder models. It is required when `tensor_parallel_size > 1`. | -| Decoder CUDA graphs | Yes | `CudaGraphConfig` captures decoder work. V1 supports greedy and beam search; V2 supports its single-beam path. | -| Encoder CUDA graphs | No | `EncodeCudaGraphConfig` is disabled for encoder-decoder models. The encoder runs eagerly. | -| Overlap scheduler | Yes | Enabled by default. V1 supports greedy decoding and beam search; V2 remains limited to `max_beam_width=1`. | -| Tensor parallelism | Yes | Use `tensor_parallel_size > 1` with `attn_backend="TRTLLM"`. Attention head counts must be divisible by the TP size. | -| Pipeline parallelism | No | Keep `pipeline_parallel_size=1`. | -| Context parallelism | No | Keep `context_parallel_size=1`. | -| Attention data parallelism | No | Keep `enable_attention_dp=False`. | -| Chunked prefill | Not supported for the encoder phase | Set `enable_chunked_prefill=False`. The complete encoder input must fit in the iteration token budget. | -| Piecewise CUDA graph | No | Do not set `torch_compile_config.enable_piecewise_cuda_graph=True`. | - -BF16 is the recommended model dtype. Validate accuracy with your checkpoint and -task before deploying a different precision or quantization configuration. - -## Choose the attention backend - -Use `attn_backend="TRTLLM"` for encoder-decoder models. T5 self-attention needs -this backend to apply relative attention bias, and tensor parallel -encoder-decoder execution explicitly requires it. - -The `TRTLLM` backend can internally select optimized kernels when the hardware -and request are eligible. For example, compatible operations on Blackwell can -use FlashInfer TRTLLM-Gen kernels. This internal selection is different from -setting `attn_backend="FLASHINFER"`. Cases such as T5 relative attention bias or -beam-expanded self-attention can fall back to another kernel within the -`TRTLLM` backend; no customer-side backend change is needed. - -## Run basic generation - -Install TensorRT LLM using the [installation guide](../installation/installation-guide.md) -and make sure the checkpoint is accessible either from the Hugging Face Hub or -from a local directory. - -The following example uses KV cache manager V1, greedy decoding, and the overlap -scheduler: - -```python -from tensorrt_llm.llmapi import LLM, KvCacheConfig, SamplingParams, SchedulerConfig - - -model = "google/flan-t5-small" - -with LLM( - model=model, - backend="pytorch", - max_batch_size=4, - max_input_len=512, - max_num_tokens=2048, - max_seq_len=512, - kv_cache_config=KvCacheConfig( - enable_block_reuse=False, - free_gpu_memory_fraction=0.8, - cross_kv_cache_fraction=0.5, - use_kv_cache_manager_v2=False, - ), - scheduler_config=SchedulerConfig(use_python_scheduler=True), -) as llm: - sampling_params = SamplingParams( - max_tokens=64, - temperature=0.0, - ) - result = llm.generate( - "translate English to German: The house is wonderful.", - sampling_params=sampling_params, - use_tqdm=False, - ) - print(result.outputs[0].text) -``` - -Use the task format expected by the checkpoint. For example, T5 translation -checkpoints commonly expect a task prefix such as `translate English to -German:`, while a summarization checkpoint expects the source document. - -The LLM API performs these encoder-decoder-specific steps automatically: - -1. Tokenizes the supplied string as the encoder input. -2. Runs the encoder once and retains its output for cross-attention. -3. Initializes the decoder from the checkpoint's decoder start token. -4. Generates decoder tokens and returns the detokenized decoder output. - -Do not prepend a decoder start token to the source prompt. If you pass token IDs -instead of text, pass only the encoder-side token IDs: - -```python -from transformers import AutoTokenizer - - -tokenizer = AutoTokenizer.from_pretrained(model) -source_text = "translate English to German: The house is wonderful." -source_token_ids = tokenizer.encode(source_text, add_special_tokens=True) -result = llm.generate(source_token_ids, sampling_params=sampling_params) -``` - -### Configure an mBART tokenizer - -mBART tokenization depends on the source language. Create the Hugging Face -tokenizer with `src_lang` and pass that tokenizer to `LLM` so string prompts -receive the correct source-language token: - -```python -from transformers import AutoTokenizer - -from tensorrt_llm.llmapi import LLM, KvCacheConfig, SamplingParams - - -model = "/path/to/mbart-large-50-many-to-one-mmt" -tokenizer = AutoTokenizer.from_pretrained(model, src_lang="ro_RO") - -with LLM( - model=model, - tokenizer=tokenizer, - backend="pytorch", - attn_backend="TRTLLM", - dtype="bfloat16", - enable_chunked_prefill=False, - kv_cache_config=KvCacheConfig(cross_kv_cache_fraction=0.5), -) as llm: - result = llm.generate( - "Şeful ONU spune că nu există o soluţie militară în Siria.", - sampling_params=SamplingParams(max_tokens=64, temperature=0.0), - use_tqdm=False, - ) - print(result.outputs[0].text) -``` - -For this many-to-one checkpoint, `generation_config.json` selects English with -the `en_XX` forced BOS token. For other mBART checkpoints, confirm that -`decoder_start_token_id`, `forced_bos_token_id`, `eos_token_id`, and the -tokenizer language settings select the source and target languages you intend -to serve. - -### Understand BART and mBART decoder tokens - -When a BART or mBART checkpoint defines `forced_bos_token_id`, the PyTorch -backend initializes the decoder with the following internal prefix: - -```text -[decoder_start_token_id, forced_bos_token_id] -``` - -For example, BART-large-CNN uses `[2, 0]`. Customers provide only the encoder -input; do not prepend either decoder token. By default, the returned token IDs -exclude `decoder_start_token_id` but include `forced_bos_token_id`, so the -BART-large-CNN output begins with token ID 0. - -The forced BOS token counts against `SamplingParams.max_tokens`. Consequently, -a request using this prefix requires `max_tokens` to be at least 2. The runtime -uses the remaining token budget for model-selected tokens. This behavior is the -same for greedy decoding and beam search and does not require a customer logits -processor. - -EOS is a stopping token rather than a forced final token. The runtime uses -`SamplingParams.end_id`, which defaults to the tokenizer's `eos_token_id`. If -the model generates EOS before the output limit, the returned token IDs include -EOS and `finish_reason` is `"stop"`. Set `ignore_eos=True` to continue decoding -past EOS. - -The runtime does not inject `forced_eos_token_id` when a sequence reaches -`max_tokens`. It preserves the model-selected final token and reports -`finish_reason="length"`. - -## Run a batch - -Pass a list of strings to batch inputs. The strings can have different tokenized -lengths: - -```python -sources = [ - "translate English to German: The house is wonderful.", - "translate English to German: The book is on the table.", -] - -results = llm.generate(sources, sampling_params=sampling_params, use_tqdm=False) -for source, result in zip(sources, results): - print(f"source={source!r} output={result.outputs[0].text!r}") -``` - -`max_num_tokens` must cover the encoder tokens admitted in an iteration as well -as decoder work. Increase it for larger batches or longer source sequences. - -## Choose KV cache manager V1 or V2 - -Encoder-decoder execution uses two KV cache pools: - -- The self-attention pool stores decoder-side KV states. -- The cross-attention pool stores encoder-derived K/V states used by every - decoder layer. - -`cross_kv_cache_fraction` is required for every encoder-decoder model. It divides -the configured KV cache memory budget between the two pools. A value of `0.5` -is a reasonable starting point: - -```python -kv_cache_config = KvCacheConfig( - free_gpu_memory_fraction=0.8, - cross_kv_cache_fraction=0.5, - use_kv_cache_manager_v2=False, -) -``` - -Increase `cross_kv_cache_fraction` when long encoder inputs exhaust the cross -pool. Decrease it when long decoder outputs or wide beams exhaust the -self-attention pool. The two fractions are related as follows: - -```text -cross-attention pool = total KV cache budget * cross_kv_cache_fraction -self-attention pool = total KV cache budget * (1 - cross_kv_cache_fraction) -``` - -V1 is the default and should be the first choice for production deployments. -To evaluate V2, change only the manager selection and keep beam width equal to -one: - -```python -kv_cache_config = KvCacheConfig( - free_gpu_memory_fraction=0.8, - cross_kv_cache_fraction=0.5, - use_kv_cache_manager_v2=True, -) - -llm = LLM( - model=model, - backend="pytorch", - attn_backend="TRTLLM", - max_beam_width=1, - kv_cache_config=kv_cache_config, -) -``` - -KV cache manager V2 is a prototype feature and rejects configurations with a -maximum beam width greater than one. - -## Use beam search - -Beam search requires KV cache manager V1. The maximum beam width is a runtime -capacity setting and must be specified when constructing `LLM`: - -```python -beam_width = 4 - -with LLM( - model="/path/to/bart-large-cnn", - backend="pytorch", - attn_backend="TRTLLM", - dtype="bfloat16", - max_beam_width=beam_width, - enable_chunked_prefill=False, - kv_cache_config=KvCacheConfig( - free_gpu_memory_fraction=0.8, - cross_kv_cache_fraction=0.5, - use_kv_cache_manager_v2=False, - ), -) as llm: - beam_params = SamplingParams( - best_of=beam_width, - max_tokens=64, - n=beam_width, - temperature=0.0, - use_beam_search=True, - ) - result = llm.generate( - "The engineering team released a faster inference service on Monday. " - "The update improves batching, lowers latency, and adds detailed " - "monitoring for operators.", - sampling_params=beam_params, - use_tqdm=False, - ) - - for hypothesis in result.outputs: - print(hypothesis.text) -``` - -`best_of` sets the beam width and must not exceed `LLM.max_beam_width`. `n` -sets the number of returned hypotheses and must not exceed `best_of`. Set `n=1` -to return only the best hypothesis. - -Beam search expands decoder-side cache and compute requirements. Include this -expansion when sizing the self-attention KV pool and CUDA graph batch sizes. - -## Enable decoder CUDA graphs - -Pass `CudaGraphConfig` to capture and replay decoder iterations: - -```python -from tensorrt_llm.llmapi import CudaGraphConfig - - -llm = LLM( - model=model, - backend="pytorch", - attn_backend="TRTLLM", - max_batch_size=8, - cuda_graph_config=CudaGraphConfig( - max_batch_size=8, - enable_padding=True, - ), - kv_cache_config=KvCacheConfig( - free_gpu_memory_fraction=0.8, - cross_kv_cache_fraction=0.5, - ), -) -``` - -This configuration captures decoder work only; the encoder continues to run -eagerly. With beam search, graph batch sizes must cover the active decoder -sequences after beam expansion. Padding lets nearby runtime batch sizes reuse a -captured graph. - -Do not use `EncodeCudaGraphConfig` for an encoder-decoder model. The runtime -warns and disables it. Piecewise CUDA graphs through `TorchCompileConfig` are -also unsupported for this model type. - -## Control the overlap scheduler - -The PyTorch backend enables the overlap scheduler by default. The examples set -`disable_overlap_scheduler=False` explicitly to make that choice visible: - -```python -llm = LLM( - model=model, - backend="pytorch", - disable_overlap_scheduler=False, - kv_cache_config=KvCacheConfig(cross_kv_cache_fraction=0.5), -) -``` - -Overlap is not restricted to KV cache manager V1. Both V1 and V2 enter the same -overlap executor loop, and that loop contains V2-specific resource handling. -V2 remains limited to `max_beam_width=1`. Set -`disable_overlap_scheduler=True` when debugging. - -## Use tensor parallelism - -Set `tensor_parallel_size` to the number of GPUs over which to shard the model: - -```python -with LLM( - model=model, - backend="pytorch", - attn_backend="TRTLLM", - tensor_parallel_size=2, - pipeline_parallel_size=1, - context_parallel_size=1, - enable_attention_dp=False, - enable_chunked_prefill=False, - kv_cache_config=KvCacheConfig( - free_gpu_memory_fraction=0.8, - cross_kv_cache_fraction=0.5, - use_kv_cache_manager_v2=False, - ), -) as llm: - result = llm.generate(source_text, sampling_params=sampling_params) -``` - -For single-node execution through the LLM API, do not add an `mpirun` prefix. -TensorRT LLM starts the worker processes. The selected TP size must divide the -encoder and decoder attention head counts. Cross-attention KV head duplication -is not supported, so its KV head count must also be divisible by the TP size. - -Tensor parallelism currently requires `attn_backend="TRTLLM"`. Pipeline -parallelism, context parallelism, and attention DP are rejected for -encoder-decoder models. - -## Serve an encoder-decoder model - -The following configuration starts a greedy Flan-T5 service with the PyTorch -backend. Save it as `enc-dec-config.yaml`: - -```yaml -attn_backend: TRTLLM -dtype: bfloat16 -disable_overlap_scheduler: false -enable_chunked_prefill: false -max_beam_width: 1 -max_input_len: 512 -max_num_tokens: 2048 -max_seq_len: 512 -kv_cache_config: - enable_block_reuse: false - free_gpu_memory_fraction: 0.8 - cross_kv_cache_fraction: 0.5 - use_kv_cache_manager_v2: false -scheduler_config: - use_python_scheduler: true -``` - -Start the server: - -```bash -trtllm-serve google/flan-t5-small \ - --backend pytorch \ - --max_batch_size 4 \ - --config enc-dec-config.yaml -``` - -Send the source text through the completions endpoint: - -```bash -curl http://localhost:8000/v1/completions \ - -H "Content-Type: application/json" \ - -d '{ - "model": "google/flan-t5-small", - "prompt": "translate English to German: The house is wonderful.", - "max_tokens": 64, - "temperature": 0.0 - }' -``` - -To serve beam search, use KV cache manager V1, restart the server with -`max_beam_width` set to the desired maximum, and add the following request -fields: - -```json -{ - "use_beam_search": true, - "best_of": 4, - "n": 1 -} -``` - -For tensor parallel serving, add `--tp_size ` to `trtllm-serve` and keep the -attention backend set to `TRTLLM`. - -## Size the runtime - -Use these guidelines as a starting point: - -- Set `max_input_len` to at least the maximum tokenized encoder input length. -- Set `max_seq_len` to at least the larger of the maximum encoder input length - and maximum decoded sequence length. The current encoder-decoder runtime uses - this value while sizing both phases. -- Set `max_num_tokens` high enough for all encoder tokens admitted together and - for the active decoder tokens. This is especially important for mixed-length - batches. -- Increase `max_batch_size` for more concurrent requests. Beam width multiplies - the number of active decoder sequences but not the number of source requests. -- Tune `free_gpu_memory_fraction` first, then tune - `cross_kv_cache_fraction` based on whether the cross-attention or - self-attention pool is exhausted. - -## Performance - -The following benchmarks compare the PyTorch backend with the legacy TensorRT -encoder-decoder path for large-batch inference. The measurements use BF16 on -one H100 80 GB GPU with greedy decoding, an output limit of 128 tokens, and -mixed encoder input lengths from 260 to 440 tokens. The Flan-T5-XL results are -the average of ten timed runs after three warmup runs. The BART results are the -average of 20 timed runs after five warmup runs. Executed-token throughput -includes the terminal EOS token when a sequence emits it. - -The PyTorch configuration uses the `TRTLLM` attention backend, the overlap -scheduler, the Python scheduler, decoder CUDA graphs with padding, KV cache -manager V1, `max_input_len=512`, `max_seq_len=1024`, and -`max_num_tokens=65536`. Block reuse and chunked prefill are disabled. The KV -cache uses `free_gpu_memory_fraction=0.3` and -`cross_kv_cache_fraction=0.5`. - -The legacy TensorRT configuration uses separate BF16 encoder and decoder -engines built for batch size 128 and beam width 1. The encoder supports 512 -input tokens and 65,536 tokens per iteration; the decoder supports a sequence -length of 129. The benchmark runs these engines through `ModelRunnerCpp` with -greedy `top_k=1` decoding and the same KV cache fractions. For BART, the legacy -TensorRT benchmark starts the decoder with token IDs `[2, 0]` and generates at -most 127 more tokens. The PyTorch LLM API applies the same decoder prefix -internally and counts token ID 0 as the first output token; customers do not -need to provide the decoder prefix. Both paths use token ID 2 as EOS and stop -when the model generates it naturally. If a sequence reaches the output limit, -it retains the model-selected final token and reports a length stop instead of -forcing EOS. This setup also lets beam search begin after the shared decoder -prefix without a per-step Python logits processor. - -### Flan-T5-XL - -For Flan-T5-XL, the PyTorch backend performs on par with the legacy TensorRT -path, with slightly lower latency and higher executed-token throughput across -the tested batch sizes. - -| Batch size | Legacy TensorRT latency | PyTorch latency | PyTorch latency improvement over legacy TensorRT | Legacy TensorRT executed tokens/s | PyTorch executed tokens/s | -| ---: | ---: | ---: | ---: | ---: | ---: | -| 32 | 727.6 ms | 706.1 ms | 3.0% | 3,153 | 3,312 | -| 64 | 1,225.0 ms | 1,136.7 ms | 7.2% | 3,863 | 4,184 | -| 128 | 2,056.8 ms | 1,999.3 ms | 2.8% | 4,601 | 4,768 | - -### BART-large-CNN - -For BART-large-CNN, the PyTorch backend has 21.9% to 36.0% higher latency than -the legacy TensorRT path across the tested batch sizes. - -| Batch size | Legacy TensorRT latency | PyTorch latency | PyTorch latency difference | Legacy TensorRT executed tokens/s | PyTorch executed tokens/s | -| ---: | ---: | ---: | ---: | ---: | ---: | -| 32 | 229.9 ms | 280.2 ms | 21.9% slower | 12,611 | 10,662 | -| 64 | 252.2 ms | 343.0 ms | 36.0% slower | 22,007 | 16,209 | -| 128 | 352.3 ms | 472.7 ms | 34.2% slower | 31,544 | 23,518 | - -Performance depends on the model, request distribution, decoding settings, and -GPU configuration. Benchmark with a representative workload before deployment. - -## Troubleshooting - -### `cross_kv_cache_fraction` is required - -Every encoder-decoder runtime needs a cross-attention KV pool. Add -`KvCacheConfig(cross_kv_cache_fraction=...)`; `0.5` is a reasonable initial -value. Do not set this field for a decoder-only model. - -### `decoder_start_token_id` is required - -The checkpoint must define `decoder_start_token_id` or `bos_token_id` in its -Hugging Face model or generation configuration. Use a checkpoint with a complete -`config.json` and, when applicable, `generation_config.json`. - -### KV cache manager V2 fails with beam search - -V2 currently requires `max_beam_width=1`. Select V1 by setting -`use_kv_cache_manager_v2=False` before enabling beam search. - -### Tensor parallel initialization is rejected - -Check all of the following: - -- `attn_backend` is `TRTLLM`. -- Encoder, decoder, and cross-attention head counts are divisible by the TP - size. -- `pipeline_parallel_size=1` and `context_parallel_size=1`. -- `enable_attention_dp=False`. - -### CUDA graphs do not capture the encoder - -This is expected. `CudaGraphConfig` accelerates decoder iterations only. The -encoder path runs eagerly, and `EncodeCudaGraphConfig` is disabled for -encoder-decoder models. - -### Output quality differs from the Hugging Face example - -Confirm that the source uses the task prefix and language settings expected by -the checkpoint. Also compare the same model dtype, beam width, length penalty, -EOS stopping behavior, and forced BOS configuration. Small numerical -differences can change lower-ranked beam hypotheses when scores are close. diff --git a/docs/source/models/supported-models.md b/docs/source/models/supported-models.md index b03f9961f7cc..3771da0496b8 100644 --- a/docs/source/models/supported-models.md +++ b/docs/source/models/supported-models.md @@ -48,7 +48,6 @@ The following is a table of supported models for the PyTorch backend: | `Qwen2ForProcessRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-PRM-7B` | | `Qwen2ForRewardModel` | Qwen2-based | `Qwen/Qwen2.5-Math-RM-72B` | | `Qwen3ForCausalLM` | Qwen3 | `Qwen/Qwen3-8B` | -| `Qwen3ForTextEmbedding` | Qwen3-Embedding | `Qwen/Qwen3-Embedding-8B` | | `Qwen3MoeForCausalLM` | Qwen3MoE | `Qwen/Qwen3-30B-A3B` | | `Qwen3NextForCausalLM` | Qwen3Next | `Qwen/Qwen3-Next-80B-A3B-Thinking` | | `Qwen3_5MoeForCausalLM` | Qwen3.5-MoE | `Qwen/Qwen3.5-397B-A17B` | @@ -80,8 +79,8 @@ Note: Support for other models may vary. Features marked "N/A" are not applicabl | `Step3p7ForConditionalGeneration`| Yes | Yes | Yes | Untested | Untested | Yes | No | No | No | Yes | Untested | Untested | Yes | Untested | Untested | | `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes | Yes | Yes | Untested | Untested | No | No | No | No | Yes | Untested | No | N/A | Untested | Untested | -[^1]: Chunked Prefill for MLA can only be enabled on SM90/SM100/SM103/SM120. -[^2]: KV cache reuse for MLA can only be enabled on SM90/SM100/SM103/SM120/SM121 and in BF16/FP8 KV cache dtype. +[^1]: Chunked Prefill for MLA can only be enabled on SM100/SM103. +[^2]: KV cache reuse for MLA can only be enabled on SM90/SM100/SM103 and in BF16/FP8 KV cache dtype. [^3]: Qwen3-Next-80B-A3B exhibits relatively low accuracy on the SciCode-AA-v2 benchmark. [^5]: Supported via the [AutoDeploy](../features/auto_deploy/auto-deploy.md) backend. See [AD Configs](../../../examples/auto_deploy/model_registry/configs). [^6]: Also supports text-only inference via the [AutoDeploy](../features/auto_deploy/auto-deploy.md) backend. @@ -115,7 +114,6 @@ Note: Support for other models may vary. Features marked "N/A" are not applicabl | `Step3p7ForConditionalGeneration` | Yes | Yes | Untested | Yes | Untested | Untested | Untested | Untested | L + I | | `MiniMaxM3SparseForConditionalGeneration` [^12] | Yes | Yes | Untested | Yes | Untested | No | Untested | Untested | L + I + V | | `Cosmos3ForConditionalGeneration` [^13] | Yes | Yes | Yes | Yes | Yes | Yes | Untested | Untested | L + I + V | -| `Qwen3_5ForConditionalGeneration` | Yes | Yes | Untested | Yes | Yes | No | Untested | Yes | L + I + V | | `Qwen3_5MoeForConditionalGeneration` | Yes | Yes | Untested | Yes | Yes | No | Untested | Yes | L + I + V | Note: @@ -124,26 +122,6 @@ Note: - V: Video - A: Audio -## Multimodal Encoder Optimizations - -The following optimizations are available to models that implement -`MultimodalModelMixin`. Currently, only `Mistral3ForConditionalGeneration` supports them. - -| Model Architecture | Multimodal Encoder Side Stream | Multimodal Embeddings Cache | -| ------------------ | ------------------------------ | --------------------------- | -| `Mistral3ForConditionalGeneration` | Yes | Yes | - -- **Multimodal encoder side stream** prefetches encoder work for pending requests on a separate - CUDA stream, allowing it to overlap with work on the main stream. Set - `multimodal_config.encoder_side_stream_max_ahead` to a positive value to enable it; the value - limits the number of prefetched requests that can be ahead of admission. This option is mutually - exclusive with `multimodal_config.encoder_cuda_graph` and can increase peak GPU memory use. -- **Multimodal embeddings cache** is a per-model, cross-request LRU cache of encoder embeddings. - Set `multimodal_config.encoder_cache_max_bytes` to its capacity (for example, `"512MiB"`), or - `0` to disable it. Entries are cached per multimodal item, but a request reuses cached embeddings - only when all of its items hit the cache. At present, only single-modality requests are cacheable; - mixed-modality requests bypass the cache. - # Visual Generation Models TensorRT-LLM provides beta support for diffusion-based image and video generation. diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index d1e376b9a13e..23ce2ab941d7 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -28,7 +28,6 @@ TensorRT-LLM **VisualGen** provides a unified inference stack for diffusion mode | `black-forest-labs/FLUX.2-dev` | Text-to-Image | | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` | Text-to-Video | | `Wan-AI/Wan2.1-T2V-14B-Diffusers` | Text-to-Video | -| `FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers` | Text-to-Video (VSA) | | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers` | Image-to-Video | | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers` | Image-to-Video | | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | Text-to-Video | @@ -44,26 +43,23 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models ### Feature Matrix -| Model | FP8 blockwise | NVFP4 | TeaCache | Cache-DiT | CFG Parallelism | Ulysses Parallelism | Parallel VAE | CUDA Graph | torch.compile | trtllm-serve | Attention2D | Ring Attention | Tensor Parallelism | VSA | -|---|---|---|---|---|---|---|---|---|---|---|--|--|--|--| -| **FLUX.1** | Yes | Yes | Yes | Yes | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **FLUX.2** | Yes | Yes | Yes | Yes | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **Wan 2.1** | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | -| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **LTX-2** | Yes | Yes | Yes [^4] | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | No | -| **Qwen-Image** [^5] | Yes | Yes | No | No | No | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No | -| **Cosmos3** | Yes | Yes | No | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | +| Model | FP8 blockwise | NVFP4 | TeaCache | Cache-DiT | CFG Parallelism | Ulysses Parallelism | Parallel VAE | CUDA Graph | torch.compile | trtllm-serve | Attention2D | Ring Attention | Tensor Parallelism | +|---|---|---|---|---|---|---|---|---|---|---|--|--|--| +| **FLUX.1** | Yes | Yes | Yes | Yes | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | +| **FLUX.2** | Yes | Yes | Yes | Yes | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | +| **Wan 2.1** | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | +| **Wan 2.2** | Yes | Yes | Yes [^2] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | +| **LTX-2** | Yes | Yes | Yes [^3] | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | +| **Qwen-Image** [^4] | Yes | Yes | No | No | No | Yes | No | Yes | Yes | Yes | Yes | Yes | No | +| **Cosmos3** | Yes | Yes | No | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | [^1]: FLUX models use embedded guidance and do not have a separate negative prompt path, so CFG parallelism is not applicable. -[^2]: `FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers` — VSA-fine-tuned checkpoint with learned sparse-attention gates. Requires `CUTEDSL` on Blackwell sm_100+ (falls back to dense SDPA on older hardware). Ring and Attention2D not supported (no LSE output); Ulysses supported. +[^2]: Wan 2.2 has two stage transformers; TeaCache requires explicit `teacache.coefficients` (high-noise) and `teacache.coefficients_2` (low-noise). There is no built-in coefficient table for Wan 2.2. -[^3]: Wan 2.2 has two stage transformers; TeaCache requires explicit `teacache.coefficients` (high-noise) and `teacache.coefficients_2` (low-noise). There is no built-in coefficient table for Wan 2.2. +[^3]: LTX-2 has no built-in TeaCache coefficient table in TRT-LLM; set `teacache.coefficients` explicitly when enabling TeaCache. -[^4]: LTX-2 has no built-in TeaCache coefficient table in TRT-LLM; set `teacache.coefficients` explicitly when enabling TeaCache. - -[^5]: Qwen-Image ships a native BF16 implementation with per-module numerical parity against `diffusers.QwenImagePipeline` (cosine similarity >= 0.999 on the full 20B transformer) and supports `trtllm-serve` / `/v1/images/generations`. VisualGen supports FP8 blockwise and NVFP4 dynamic quantization from BF16 checkpoints, as well as direct loading of statically quantized FP8 and NVFP4 ModelOpt checkpoints. +[^4]: Qwen-Image ships a native BF16 implementation with per-module numerical parity against `diffusers.QwenImagePipeline` (cosine similarity >= 0.999 on the full 20B transformer) and supports `trtllm-serve` / `/v1/images/generations`. VisualGen supports FP8 blockwise and NVFP4 dynamic quantization from BF16 checkpoints, as well as direct loading of statically quantized FP8 and NVFP4 ModelOpt checkpoints. ## Quick Start @@ -230,44 +226,6 @@ args = VisualGenArgs( **Wan 2.2 dual-transformer note:** Wan 2.2 uses two expert transformers (high-noise and low-noise stacks). All `CacheDiTConfig` parameters apply to both stacks, except `max_warmup_steps` and `max_cached_steps`: the low-noise stack always uses fixed internal caps (`max_warmup_steps=2`, `max_cached_steps=20`) regardless of user config. -### Video Sparse Attention (VSA) - -VSA reduces the compute cost of self-attention in video diffusion models by selectively attending to only the most relevant spatial-temporal blocks. It uses a two-branch design: a lightweight coarse mean-pool branch computes block-level attention scores to identify the top-K most relevant token blocks, then a fine branch runs a block-sparse CuTe kernel over only those blocks. The two outputs are blended with learned gates. - -**Requirements:** -- VSA-fine-tuned checkpoint: [`FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers`](https://huggingface.co/FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers). Standard Wan checkpoints do not have the learned VSA gates. -- Blackwell GPU (sm_100+) for the CuTe JIT kernel. Falls back to dense SDPA on older hardware with no accuracy loss. -- `CUTEDSL` attention backend. -- Not compatible with Ring attention or Attention2D (VSA does not produce per-split LSE). Ulysses is supported. - -**`vsa_sparsity`** controls the fraction of K/V blocks skipped in the fine branch (0.0 = dense, 0.9 = 90% blocks skipped). Higher sparsity gives more speedup at the cost of some quality. - -Python API: - -```python -from tensorrt_llm import VisualGenArgs -from tensorrt_llm.visual_gen.args import AttentionConfig, VideoSparseAttentionConfig - -args = VisualGenArgs( - model="FastVideo/Wan2.1-VSA-T2V-14B-720P-Diffusers", - attention_config=AttentionConfig( - backend="CUTEDSL", - sparse_attention_config=VideoSparseAttentionConfig(vsa_sparsity=0.9), - ), -) -``` - -YAML (for use with `--visual_gen_args` or `trtllm-serve`): - -```yaml -attention_config: - backend: CUTEDSL - sparse_attention_config: - algorithm: vsa - vsa_sparsity: 0.90 -``` - - ### Multi-GPU Parallelism Configured under `VisualGenArgs.parallel_config`. Modes can be combined: diff --git a/docs/source/torch/adding_custom_kernels.md b/docs/source/torch/adding_custom_kernels.md index 7a3bd12104ec..bed8ee59da6c 100644 --- a/docs/source/torch/adding_custom_kernels.md +++ b/docs/source/torch/adding_custom_kernels.md @@ -258,30 +258,6 @@ The same pattern (`@torch.library.custom_op("trtllm::...")`, `register_fake`, co - **DLPack/CUDA Graphs.** When exporting tensors via DLPack, use the stream override that mimics the `CUDAGraphCompatibleWrapper` in `argmax.py` so the capture replays cleanly. - **JIT compile cache.** Always cache compiled kernels by the keys that affect codegen (dtype, last-dim size, hardware variant). `argmax.py` and the `cute_dsl_custom_ops.py` runners are good references. -### 4.5 Pruning autotuner tactics with nvMatmulHeuristics - -A CuTe DSL runner that exposes many tactics (tile / cluster / swap / prefetch combinations) can be expensive to autotune, because the AutoTuner JIT-compiles and benchmarks every candidate returned by `get_valid_tactics`. A runner can therefore rank and prune its own candidate list with [nvMatmulHeuristics](https://pypi.org/project/nvidia-matmul-heuristics/) *inside* `get_valid_tactics`, before the tactics ever reach the compile+benchmark loop — all shapes and inputs are already visible at that point, so no extra AutoTuner hook is needed. The pruning is a no-op by default and must degrade gracefully to the full list on any failure, so tuning never loses a valid candidate. - -Today this is wired up for the SM100/SM103 (Blackwell B200/B300) NVFP4 dense GEMM: `CuteDSLNVFP4BlackwellRunner.get_valid_tactics` builds the full sweep and then hands it to the private `_rank_prune_tactics` helper. The pruned set is a strict, re-validated subset of the full sweep — it never introduces a tactic the kernel validator rejected — so correctness is unchanged and only autotuner wall-time is affected. On a 4096³ NVFP4 GEMM this cuts a cold autotune from ~120 tactics to ~5 (roughly 7× less wall-time) while keeping the selected kernel within a few percent of the full-sweep best. - -The behavior is controlled entirely by environment variables (all read at tune time; the package must be installed, otherwise the path silently falls back to the full sweep): - -| Environment variable | Default | Meaning | -|---|---|---| -| `TRTLLM_CUTEDSL_NVMMH_ENABLE` | `0` | Master opt-in switch. Set to `1` to enable heuristic tactic pruning. When unset/`0`, or when `nvidia-matmul-heuristics` is not installed, the full tactic sweep runs unchanged. | -| `TRTLLM_CUTEDSL_NVMMH_FIELDS` | `tile,cluster` | Comma-separated list of the knobs nvMatmulHeuristics drives; any knob not listed is swept by the runner. Recognized tokens: `tile`, `cluster`, `swizzle`, `cta_order`. `tile` and `cluster` are a coupled pair (the 2-SM encoding maps the joint `(cta, cluster)`), so naming either drives both. Accepted aliases: `cta_tile` / `mma_tiler` → `tile`. Unknown tokens are warned once and ignored. **Top-K tile/cluster pruning happens only when `tile`/`cluster` is selected; with scheduler-only fields (`swizzle` and/or `cta_order`) every valid tile/cluster is swept and merely annotated with the model's scheduler knobs.** | -| `TRTLLM_CUTEDSL_NVMMH_MAX_TACTICS` | `5` | Top-K distinct `(tile, cluster, swap)` keys to keep from the model's ranking (applies only when `tile`/`cluster` is in `FIELDS`). Both `use_prefetch` variants of a kept key are profiled on top (prefetch is not modeled), so the final tactic count can be up to ~2× this. Larger K = closer to the full-sweep best but slower autotune; smaller K = faster autotune with a larger potential perf gap. | - -Example (enable pruning, keep the top 8 tile/cluster candidates): - -```bash -export TRTLLM_CUTEDSL_NVMMH_ENABLE=1 -export TRTLLM_CUTEDSL_NVMMH_FIELDS=tile,cluster -export TRTLLM_CUTEDSL_NVMMH_MAX_TACTICS=8 -``` - -The adapter and env helpers live in [`tensorrt_llm/_torch/custom_ops/cutedsl_matmul_heuristics.py`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/custom_ops/cutedsl_matmul_heuristics.py); the `get_valid_tactics` / `_rank_prune_tactics` pruning is in [`cute_dsl_custom_ops.py`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/custom_ops/cute_dsl_custom_ops.py). - --- ## 5. Example walkthrough: `indexer_k_cache_scatter_op` diff --git a/examples/apps/chat.py b/examples/apps/chat.py index 661949a3a607..0e3f2d928158 100755 --- a/examples/apps/chat.py +++ b/examples/apps/chat.py @@ -5,8 +5,8 @@ import colorama from transformers import AutoTokenizer, PreTrainedTokenizer -from tensorrt_llm import LLM -from tensorrt_llm.llmapi import KvCacheConfig, SamplingParams +from tensorrt_llm._tensorrt_engine import LLM +from tensorrt_llm.llmapi import BuildConfig, KvCacheConfig, SamplingParams class LlmConsole(code.InteractiveConsole): @@ -72,6 +72,10 @@ def main(model: str, tokenizer: str, tp_size: int): free_gpu_memory_fraction=0.8) kv_cache_config.enable_block_reuse = True + build_config = BuildConfig(max_batch_size=1, + max_input_len=6000, + max_num_tokens=10240) + sampling_params = SamplingParams(max_tokens=100, temperature=0.5, top_p=0.95, @@ -79,9 +83,7 @@ def main(model: str, tokenizer: str, tp_size: int): llm = LLM(model, tokenizer, - max_batch_size=1, - max_input_len=6000, - max_num_tokens=10240, + build_config=build_config, kv_cache_config=kv_cache_config, tensor_parallel_size=tp_size) diff --git a/examples/apps/fastapi_server.py b/examples/apps/fastapi_server.py index e1d90c37934e..510b281a701b 100755 --- a/examples/apps/fastapi_server.py +++ b/examples/apps/fastapi_server.py @@ -18,9 +18,9 @@ from fastapi import FastAPI, Request from fastapi.responses import JSONResponse, Response, StreamingResponse -from tensorrt_llm import LLM +from tensorrt_llm._tensorrt_engine import LLM from tensorrt_llm.executor import CppExecutorError, RequestError -from tensorrt_llm.llmapi import KvCacheConfig, SamplingParams +from tensorrt_llm.llmapi import BuildConfig, KvCacheConfig, SamplingParams TIMEOUT_KEEP_ALIVE = 5 # seconds. @@ -120,6 +120,8 @@ def entrypoint(model_dir: str, port = port or 8000 logging.info(f"Starting server at {host}:{port}") + build_config = BuildConfig(max_batch_size=10, max_beam_width=max_beam_width) + kv_cache_config = KvCacheConfig( free_gpu_memory_fraction=kv_cache_free_gpu_memory_fraction) @@ -127,8 +129,7 @@ def entrypoint(model_dir: str, tokenizer, tensor_parallel_size=tp_size, pipeline_parallel_size=pp_size, - max_batch_size=10, - max_beam_width=max_beam_width, + build_config=build_config, kv_cache_config=kv_cache_config) server = LlmServer(llm=llm) diff --git a/examples/auto_deploy/README.md b/examples/auto_deploy/README.md index 343ad8c87f35..d1485fdd494b 100644 --- a/examples/auto_deploy/README.md +++ b/examples/auto_deploy/README.md @@ -2,7 +2,7 @@ This folder contains runnable examples for **AutoDeploy** as it ships inside [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). For general AutoDeploy documentation, motivation, support matrix, and feature overview, please see the [official docs](https://nvidia.github.io/TensorRT-LLM/features/auto_deploy/auto-deploy.html). -> Looking for the lightweight standalone package (no TRT-LLM required)? See **Paragraf** at [github.com/NVIDIA/llm-compiler](https://github.com/NVIDIA/llm-compiler). That repo is generated from this source tree by [`paragraf/create_standalone_package.py`](./paragraf/create_standalone_package.py). +> Looking for the lightweight standalone package (no TRT-LLM required)? See **LLM Compiler** at [github.com/NVIDIA/llm-compiler](https://github.com/NVIDIA/llm-compiler). That repo is generated from this source tree by [`llmc/create_standalone_package.py`](./llmc/create_standalone_package.py). ______________________________________________________________________ diff --git a/examples/auto_deploy/llmc/CONTRIBUTING.md b/examples/auto_deploy/llmc/CONTRIBUTING.md new file mode 100644 index 000000000000..d4b027ca2e54 --- /dev/null +++ b/examples/auto_deploy/llmc/CONTRIBUTING.md @@ -0,0 +1,56 @@ +# Contributing to llm-compiler + +## Where to send PRs + +The `llmc` standalone package is **generated from the AutoDeploy source tree inside [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)**. The mirror repo on GitHub is **read-only** — it is overwritten on every release by [`create_standalone_package.py`](./create_standalone_package.py). + +**Do not open pull requests against the standalone `llmc` repo.** Any changes pushed there will be lost the next time the package is regenerated. + +The supported workflow is: + +1. **Fork** [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). +1. Edit the AutoDeploy source under `tensorrt_llm/_torch/auto_deploy/` (and its tests under `tests/unittest/_torch/auto_deploy/` and `tests/unittest/auto_deploy/`). +1. **Optional:** validate your change against the standalone repo by regenerating it locally and running its test suite (see below). +1. Open a PR against `main` of `NVIDIA/TensorRT-LLM`. + +You're welcome to fork the standalone `llmc` repo to experiment, iterate, or build on top — just remember the upstream source of truth is TensorRT-LLM. + +## Validating a change with the standalone build + +You can run the full standalone test suite locally before sending a PR. From the TensorRT-LLM checkout: + +```bash +# 1. Run the import-discipline lint (also enforced as a pre-commit hook) +python scripts/check_auto_deploy_imports.py + +# 2. Generate, install, and test the standalone package end-to-end +pytest tests/unittest/auto_deploy/standalone/ -q +``` + +`tests/unittest/auto_deploy/standalone/test_standalone_package.py` regenerates the `llmc` tree, installs it in an isolated venv, and runs the copied unit tests against the standalone install. This is the same job that gates merges. + +To regenerate the standalone tree without running tests: + +```bash +python examples/auto_deploy/llmc/create_standalone_package.py \ + --output-dir /path/to/llmc_pkg +``` + +## PR conventions + +PRs against `NVIDIA/TensorRT-LLM` follow the project-wide rules: + +- **Title format:** `[JIRA/NVBUG/None][type] description` (e.g. `[None][feat] llmc: add foo transform`). +- **DCO sign-off** is required: commit with `git commit -s`. Don't add AI tools or co-authors to the sign-off line. +- The pre-commit hooks (formatting, lint, the `auto-deploy-import-discipline` hook) run on commit. If they modify files, re-stage and commit again. + +For the rest (full coding guidelines, branching policy, CI commands), see [the TensorRT-LLM `CONTRIBUTING.md`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/CONTRIBUTING.md) and the [project-level `AGENTS.md`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/AGENTS.md). + +## Import discipline + +`llmc` source is the same files as `tensorrt_llm/_torch/auto_deploy/` — copied verbatim, never rewritten. For that to work cleanly: + +- Imports **inside** the auto_deploy package must be **relative** (`from ..foo import bar`). +- Imports **to the rest of TensorRT-LLM** must be **absolute** (`from tensorrt_llm.X import Y`) and gated at runtime by `_compat.TRTLLM_AVAILABLE` so they fail gracefully in standalone mode. + +Both rules are enforced by `scripts/check_auto_deploy_imports.py`, which runs as a pre-commit hook on every change to `tensorrt_llm/_torch/auto_deploy/`. diff --git a/examples/auto_deploy/llmc/README.md b/examples/auto_deploy/llmc/README.md new file mode 100644 index 000000000000..de7126b95e29 --- /dev/null +++ b/examples/auto_deploy/llmc/README.md @@ -0,0 +1,151 @@ +# 🔥🚀⚡ LLM Compiler + +**LLM Compiler** (Python package: `llmc`) is the standalone, lightweight distribution of the AutoDeploy graph-transformation pipeline. It exposes the same `ModelFactory` and `InferenceOptimizer` building blocks as TensorRT-LLM's AutoDeploy backend, but with a minimal dependency footprint (PyTorch + Triton + FlashInfer), so you can prototype optimization pipelines or run inference without a full TRT-LLM install. + +The `llmc` package is **generated** from the AutoDeploy source tree inside the [TensorRT-LLM repo](https://github.com/NVIDIA/TensorRT-LLM). The standalone repo is **read-only** — see [`CONTRIBUTING.md`](./CONTRIBUTING.md) for how to land changes. + +For general AutoDeploy documentation (motivation, support matrix, feature overview), see the [official docs](https://nvidia.github.io/TensorRT-LLM/features/auto_deploy/auto-deploy.html). + +______________________________________________________________________ + +## Install + +Install with [uv](https://docs.astral.sh/uv/) directly from the latest commit: + +```bash +# https +uv pip install "git+https://github.com/NVIDIA/llm-compiler.git" + +# ssh +uv pip install "git+ssh://git@github.com/NVIDIA/llm-compiler.git" +``` + +For development, clone the repo and do an editable install: + +```bash +# https +git clone https://github.com/NVIDIA/llm-compiler.git +# or ssh +git clone git@github.com:NVIDIA/llm-compiler.git + +cd llm-compiler +uv venv .venv --python 3.12 +source .venv/bin/activate +uv pip install -e ".[dev]" +``` + +### Sanity check + +```python +from llmc._compat import TRTLLM_AVAILABLE +print(f"TRT-LLM available: {TRTLLM_AVAILABLE}") # False in standalone mode +``` + +In standalone mode the package uses the PyTorch, Triton, and FlashInfer kernel paths. TRT-LLM-only kernels (custom CUDA, optimized all-reduce, MoE fused kernels, the `pyexecutor` runtime) are skipped at registration time. + +### Run the bundled tests + +```bash +pytest tests/ +``` + +______________________________________________________________________ + +## Building a custom inference pipeline + +The two core abstractions in `llmc` are: + +- **`ModelFactory`** — wraps a HuggingFace checkpoint (or any other source) and produces an initialized `nn.Module` plus dynamic-shape metadata for export. +- **`InferenceOptimizer`** — runs a configured pipeline of graph transforms (export → fuse → quantize → shard → compile → cache) over a model produced by a factory, against a `CachedSequenceInterface` that defines the input contract. + +The code below builds a custom optimization pipeline end-to-end, without going through the higher-level `LLM` API. It's the same machinery `LLM(...)` uses internally — direct access is useful when you want to insert your own transforms, target ONNX/MLIR export, or drive a non-standard runtime. + +```python +import torch + +from llmc.models.factory import ModelFactoryRegistry +from llmc.shim.interface import CachedSequenceInterface +from llmc.transform.optimizer import InferenceOptimizer +from llmc.utils.dist_config import DistConfig + +# 1. Build a ModelFactory. +# +# The registry is populated at import time. Built-in factories include +# "AutoModelForCausalLM" (text generation) and +# "AutoModelForImageTextToText" (multimodal). Factories know how to +# resolve a HF model id, instantiate the architecture, optionally load +# weights, and report dynamic shapes for export. +factory_cls = ModelFactoryRegistry.get("AutoModelForCausalLM") +factory = factory_cls( + model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", + model_kwargs={"torch_dtype": "float16"}, + skip_loading_weights=False, # set True to inspect graph without weights + max_seq_len=2048, +) + +# 2. Define the runtime input contract via a CachedSequenceInterface. +# +# This object owns the dummy inputs (input_ids, position_ids, ...) used +# during export and tracing, and — when running with a TRT-LLM-style +# KV cache — manages cache tensors. In standalone mode the cache +# manager is a no-op; CachedSequenceInterface still drives the +# transform pipeline. +cache_seq = CachedSequenceInterface( + max_seq_len=2048, + max_batch_size=4, + device="cuda", + kv_cache_config=None, # or llmc._compat.KvCacheConfig(...) + max_num_tokens=4 * 2048, + vocab_size_padded=factory.vocab_size_padded, +) + +# 3. Describe the transform pipeline. +# +# Each entry maps a registered transform name to its config. +# `stage` controls ordering across the pipeline; transforms are +# re-sorted by stage at construction time. The names below are +# illustrative — see `llmc.transform.library` for the full set. +transforms_config = { + "export_to_gm": {"stage": "export"}, + "fuse_gemms": {"stage": "post_export"}, + "fuse_rmsnorm": {"stage": "post_export"}, + "shard_attention": {"stage": "sharding", "world_size": 1}, + "compile_model": {"stage": "compile", "backend": "torch-simple"}, + "initialize_cache": {"stage": "cache_init"}, +} + +# 4. Build the optimizer and run it. +# +# `dist_config` is required when sharding across devices; for a single +# GPU the default (world_size=1) is fine. +optimizer = InferenceOptimizer( + factory=factory, + config=transforms_config, + dist_config=DistConfig(world_size=1, rank=0, tp_size=1), +) +optimized_model = optimizer(cache_seq) + +# 5. Run the optimized model. +# +# The exact call signature depends on the runtime your pipeline +# targets — for prefill-style cached attention, the inputs are +# (input_ids, position_ids) plus the cache state owned by `cache_seq`. +input_ids = torch.tensor([[1, 2, 3, 4]], device="cuda") +position_ids = torch.arange(input_ids.size(1), device="cuda").unsqueeze(0) +with torch.inference_mode(): + logits = optimized_model(input_ids, position_ids) +print(logits.shape) +``` + +### Where to look next + +| Topic | Path | +|-------|------| +| Available transforms | `llmc/transform/library/` | +| Available factories | `llmc/models/`, `llmc/models/custom/` | +| Custom ops & backends | `llmc/custom_ops/` | +| Compile backends | `llmc/compile/backends/` | +| Type compatibility shims | `llmc/_compat.py` | +| Configuration data classes | `llmc/transform/interface.py`, `llmc/llm_args.py` | + +For higher-level usage, the `llmc.LLM` class (available when running inside TRT-LLM) wraps all of the above behind a familiar generate API. In pure-standalone mode use `InferenceOptimizer` directly as shown. diff --git a/examples/auto_deploy/paragraf/_license_data.py b/examples/auto_deploy/llmc/_license_data.py similarity index 99% rename from examples/auto_deploy/paragraf/_license_data.py rename to examples/auto_deploy/llmc/_license_data.py index f0322edf4abb..1b986d5a81aa 100644 --- a/examples/auto_deploy/paragraf/_license_data.py +++ b/examples/auto_deploy/llmc/_license_data.py @@ -13,7 +13,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""License and attribution data and generators for the standalone paragraf package. +"""License and attribution data and generators for the standalone llmc package. Separated from create_standalone_package.py to keep data and logic apart. """ diff --git a/examples/auto_deploy/llmc/create_standalone_package.py b/examples/auto_deploy/llmc/create_standalone_package.py index c0e53741a4c4..52232937cb87 100644 --- a/examples/auto_deploy/llmc/create_standalone_package.py +++ b/examples/auto_deploy/llmc/create_standalone_package.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); @@ -14,17 +14,667 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Forward the legacy standalone generator entry point to Paragraf.""" +"""Create a standalone llmc package from the TensorRT-LLM source tree. -import runpy +This script copies the ``tensorrt_llm/_torch/auto_deploy`` source tree and +tests into a standalone pip-installable package. The output directory can +be pushed directly to the read-only standalone repository. + +Usage: + python create_standalone_package.py [--output-dir /path/to/output] + +The generated package uses ``llmc`` as the top-level Python package name and +``nvidia-llmc`` as the distribution name: + + from llmc._compat import TRTLLM_AVAILABLE + from llmc.custom_ops.attention_interface import SequenceInfo + +The ``auto_deploy`` source tree itself is copied verbatim — internal imports +must already be relative (enforced by the +``auto-deploy-import-discipline`` pre-commit hook), so no source rewriting +is required. Test files use absolute ``tensorrt_llm._torch.auto_deploy`` +imports by design and ARE rewritten to ``llmc`` on copy. +""" + +import argparse +import os +import re +import shutil import sys -from pathlib import Path +import textwrap + +from _license_data import VENDORED_PROJECTS, generate_attributions, generate_license + +# --------------------------------------------------------------------------- +# Path constants +# --------------------------------------------------------------------------- +SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) +REPO_ROOT = os.path.abspath(os.path.join(SCRIPT_DIR, "..", "..", "..")) +AUTO_DEPLOY_SRC = os.path.join(REPO_ROOT, "tensorrt_llm", "_torch", "auto_deploy") +TRTLLM_REQUIREMENTS = os.path.join(REPO_ROOT, "requirements.txt") +TRTLLM_DEV_REQUIREMENTS = os.path.join(REPO_ROOT, "requirements-dev.txt") +TRTLLM_LICENSE = os.path.join(REPO_ROOT, "LICENSE") +TRTLLM_GITIGNORE = os.path.join(REPO_ROOT, ".gitignore") +TRTLLM_EDITORCONFIG = os.path.join(REPO_ROOT, ".editorconfig") +TRTLLM_CODE_OF_CONDUCT = os.path.join(REPO_ROOT, "CODE_OF_CONDUCT.md") +TRTLLM_SECURITY = os.path.join(REPO_ROOT, "SECURITY.md") +TRTLLM_ATTRIBUTIONS_PYTHON = os.path.join(REPO_ROOT, "ATTRIBUTIONS-Python.md") +LLMC_README = os.path.join(SCRIPT_DIR, "README.md") +LLMC_CONTRIBUTING = os.path.join(SCRIPT_DIR, "CONTRIBUTING.md") + +# Test source directories +AD_TESTS_DIR = os.path.join(REPO_ROOT, "tests", "unittest", "auto_deploy") +AD_UTILS_TEST_DIR = os.path.join(AD_TESTS_DIR, "_utils_test") +AD_TORCH_TESTS_DIR = os.path.join(REPO_ROOT, "tests", "unittest", "_torch", "auto_deploy") + +# Example/e2e harness sources (Tier-1 e2e: build_and_run_ad.py + model registry). +# These ship with the package so the standalone install can run e2e models via +# the same entrypoint as TRT-LLM. Python is rewritten auto_deploy -> llmc; the +# model_registry YAML is data and copied verbatim. +AD_EXAMPLES_SRC = os.path.join(REPO_ROOT, "examples", "auto_deploy") +# Source filename -> destination filename in the standalone package. The e2e +# entrypoint is renamed to make its scope explicit in the llmc distribution. +EXAMPLE_FILES = {"build_and_run_ad.py": "build_and_run_llmc_trtllm.py"} +EXAMPLE_DIRS = ["model_registry"] + +# --------------------------------------------------------------------------- +# Configuration +# --------------------------------------------------------------------------- +COPY_EXTENSIONS = {".py", ".yaml", ".yml", ".json", ".txt", ".md"} +EXCLUDE_PATTERNS = {"__pycache__", ".pyc", ".pyo"} + +# Standalone runtime dependencies (version pins pulled from TRT-LLM requirements.txt) +STANDALONE_DEPS = [ + "torch", + "transformers", + "pydantic", + "pydantic-settings", + "triton", + "flashinfer-python", + "safetensors", + "accelerate", + "huggingface-hub", + "omegaconf", + "pyyaml", + "numpy", + "pillow", + "einops", + # Transitive deps that may not be pulled in by all installers/platforms + "six", + "importlib-metadata", + "werkzeug", + "StrEnum", + "graphviz", +] + +# Dev/test dependencies (version pins pulled from TRT-LLM requirements-dev.txt) +DEV_DEPS = [ + "pytest", + "pytest-timeout", + "pytest-xdist", + "pytest-cov", + "pytest-mock", + "parameterized", +] + +# Test directories to exclude from the standalone package (require TRT-LLM runtime) +EXCLUDE_TEST_DIRS = {"smoke", "shim", "standalone"} + +# Individual test files to exclude (require TRT-LLM runtime/kernels or external scripts) +EXCLUDE_TEST_FILES = { + # TRT-LLM kernel tests + "test_trtllm_moe.py", + "test_trtllm_attention_op.py", + "test_fla_cached_gated_delta_rule.py", + "test_flashinfer_trtllm_mla_op.py", + "test_trtllm_mla_op.py", + "test_fuse_trtllm_attention_quant_fp8.py", + "test_fuse_relu2_quant_nvfp4.py", + "test_moe_fusion.py", + "test_trtllm_gen_diag.py", + # Standalone flashinfer ROPE path has a known BF16 strided-interleaved mismatch. + "test_rope_op_variants.py", + # QKV fusion → trtllm cache insertion (TRT-LLM attention backend only) + "test_gemm_fusion_trtllm.py", + # Require TRT-LLM LlmArgs / runtime + "test_eagle.py", + "test_modeling_nemotron_h.py", + "test_example_configs.py", + "test_hybrid_patches.py", + "test_captured_graph.py", + # Require external scripts (build_and_run_ad.py) + "test_llama4_vlm_patch.py", + "test_mistral3_patches.py", + # Require TRT-LLM test utils not in standalone + "test_ad_moe_op.py", + "test_triton_moe.py", + # Require onnx (optional dep) + "test_export_fp8_linear_to_onnx.py", + # Uses hardcoded TRT-LLM repo path + "test_mrope_delta_cache.py", + # Depend on TRT-LLM mamba/fla kernels (relative imports beyond auto_deploy) + "test_mamba_rms_norm.py", + "test_triton_rms_norm.py", + "test_fuse_rmsnorm.py", + "test_fused_add_rms_norm.py", + "test_fuse_l2norm.py", + # Require TRT-LLM KVCacheManager or runtime + "test_kv_cache.py", + "test_torch_gated_delta_rule_cache.py", + "test_gated_delta_rule_cache.py", + "test_kv_cache_transformers.py", + # trtllm attention backend (insert_cached_attention backend=trtllm) not available standalone + "test_kv_cache_trtllm_multipool.py", + # Require TRT-LLM CUDA causal conv / mamba kernels (ops not registered standalone) + "test_cuda_causal_conv_cached_op.py", + "test_triton_causal_conv_cached_op.py", + "test_triton_mamba_cached_op.py", + "test_flashinfer_mamba_cached_op.py", + # Require TRT-LLM custom ops (dsv3_router_gemm_op, noaux_tc_op, etc.) + "test_deepseek_custom.py", + "test_glm4_moe_modeling.py", + "test_glm4_moe_lite_modeling.py", + "test_glm_moe_dsa_modeling.py", + # Full-model tests hit standalone-incompatible HF cache behavior. + "test_granite_moe_hybrid_modeling.py", + # Imports triton_kernels, which is not a standalone dependency. + "test_mxfp4_moe_layout.py", + # Require TRT-LLM distributed ops (trtllm_dist_all_gather) + "test_gather_logits_before_lm_head.py", + # Multimodal processors depend on TensorRT-LLM multimodal request types. + "test_gemma4_modeling.py", + "test_qwen3_5_moe.py", + # Hardware-specific (requires H100+ shared memory) + "test_triton_mla_op.py", + # Require TRT-LLM ops (noaux_tc_op) — split from test_export.py + "test_export_glm4_moe_lite.py", + # fuse_fp8_linear / fuse_nvfp4_linear / fuse_finegrained_fp8_linear transforms + # live in fuse_quant.py which imports tensorrt_llm.quantization.utils.fp8_utils; + # the module is silently skipped in standalone so the transforms aren't registered. + "test_quant_fusion.py", + # Imports utils.util.skip_pre_blackwell (not shipped in standalone) and exercises + # fuse_finegrained_fp8_swiglu which depends on TRT-LLM runtime. + "test_finegrained_fp8_swiglu.py", + # Exercise trtllm-gen MXFP4 MoE kernels (Blackwell-only) and import the + # prepare_trtllm_gen_moe_mxfp4_weights / utils.util helpers not in standalone. + "test_fuse_mxfp4_moe.py", + "test_trtllm_quant_mxfp4_trtllm_gen_moe.py", +} + +# Import path rewrite: old -> new (applied to test files only). +_IMPORT_REWRITE = "tensorrt_llm._torch.auto_deploy" +_IMPORT_TARGET = "llmc" + +# Paths that the script owns and regenerates on every run. +# Everything else in the output directory (e.g., .git/, .github/) is preserved +# and owned by the standalone repo itself. +_MANAGED_PATHS = [ + "llmc", + "tests", + "runners", + "pyproject.toml", + "README.md", + "LICENSE", + "ATTRIBUTIONS-Python.md", + "CONTRIBUTING.md", + ".gitignore", + ".editorconfig", + "CODE_OF_CONDUCT.md", + "SECURITY.md", + "ATTRIBUTIONS-Python.md", +] + + +# --------------------------------------------------------------------------- +# Helper functions +# --------------------------------------------------------------------------- +def _should_copy(filepath: str) -> bool: + for pattern in EXCLUDE_PATTERNS: + if pattern in filepath: + return False + _, ext = os.path.splitext(filepath) + return ext in COPY_EXTENSIONS + + +def _copy_tree(src_dir: str, dst_dir: str) -> int: + """Copy files from src_dir to dst_dir, preserving directory structure.""" + count = 0 + for root, dirs, files in os.walk(src_dir): + dirs[:] = [d for d in dirs if d != "__pycache__"] + for filename in files: + src_path = os.path.join(root, filename) + if not _should_copy(src_path): + continue + rel_path = os.path.relpath(src_path, src_dir) + dst_path = os.path.join(dst_dir, rel_path) + os.makedirs(os.path.dirname(dst_path), exist_ok=True) + shutil.copy2(src_path, dst_path) + count += 1 + return count + + +def _rewrite_imports_in_file(filepath: str) -> int: + """Rewrite imports in a copied test file for standalone mode. + + Source files inside ``tensorrt_llm/_torch/auto_deploy`` already use + relative imports (enforced by the ``auto-deploy-import-discipline`` + pre-commit hook), so no rewriting is needed for them. Tests, however, + are written against the canonical absolute path + ``tensorrt_llm._torch.auto_deploy`` and need to be rewritten to + ``llmc``. Cross-package types (e.g. ``KvCacheConfig``, + ``ActivationType``) are sourced via ``..._torch.auto_deploy._compat``, + so the primary rewrite handles them too. + + Returns the number of line-level changes made. + """ + with open(filepath) as f: + content = f.read() + + original = content + content = content.replace(_IMPORT_REWRITE, _IMPORT_TARGET) + + replacements = sum(1 for a, b in zip(original, content) if a != b) # rough count + if content != original: + with open(filepath, "w") as f: + f.write(content) + # Count actual line-level changes + replacements = sum(1 for a, b in zip(original.splitlines(), content.splitlines()) if a != b) + + return replacements + + +def _rewrite_imports_in_dir(directory: str) -> int: + """Rewrite imports in all .py files in a directory tree.""" + total = 0 + for root, _, files in os.walk(directory): + for filename in files: + if filename.endswith(".py"): + total += _rewrite_imports_in_file(os.path.join(root, filename)) + return total + + +def _read_pinned_versions(req_file: str) -> dict: + """Read requirements.txt and extract package->version-spec mapping.""" + versions = {} + if not os.path.exists(req_file): + return versions + with open(req_file) as f: + for line in f: + line = line.strip() + if not line or line.startswith("#") or line.startswith("-"): + continue + line = line.split("#")[0].strip() + # Handle semicolons (environment markers like ; python_version >= "3.10") + line = line.split(";")[0].strip() + match = re.match(r"^([a-zA-Z0-9_-]+(?:\[[^\]]+\])?)(.*)", line) + if match: + pkg_name = match.group(1).split("[")[0].lower() + version_spec = match.group(2).strip() + if version_spec: + versions[pkg_name] = version_spec + return versions + + +def _resolve_dependencies(dep_names: list, pinned: dict) -> list: + """Resolve dependency list by adding version pins from requirements.""" + resolved = [] + for name in dep_names: + extras = "" + if "[" in name: + base, extras_part = name.split("[", 1) + extras = f"[{extras_part}" + else: + base = name + version = pinned.get(base.lower(), "") + resolved.append(f"{base}{extras}{version}") + return resolved + + +# --------------------------------------------------------------------------- +# Test copying +# --------------------------------------------------------------------------- +def _should_exclude_test(filepath: str) -> bool: + """Check if a test file should be excluded from the standalone package.""" + basename = os.path.basename(filepath) + if basename in EXCLUDE_TEST_FILES: + return True + parts = filepath.replace("\\", "/").split("/") + return any(d in EXCLUDE_TEST_DIRS for d in parts) + + +def _copy_tests(output_dir: str) -> int: + """Copy auto_deploy test files to the standalone package tests/ directory.""" + tests_dst = os.path.join(output_dir, "tests") + count = 0 + + # Copy singlegpu tests (excluding TRT-LLM-only dirs/files) + singlegpu_src = os.path.join(AD_TESTS_DIR, "singlegpu") + if os.path.isdir(singlegpu_src): + for root, dirs, files in os.walk(singlegpu_src): + # Skip excluded directories + dirs[:] = [d for d in dirs if d not in EXCLUDE_TEST_DIRS and d != "__pycache__"] + + for filename in files: + src_path = os.path.join(root, filename) + if not _should_copy(src_path) or _should_exclude_test(src_path): + continue + rel_path = os.path.relpath(src_path, AD_TESTS_DIR) + dst_path = os.path.join(tests_dst, rel_path) + os.makedirs(os.path.dirname(dst_path), exist_ok=True) + shutil.copy2(src_path, dst_path) + count += 1 + + # Copy test utilities + if os.path.isdir(AD_UTILS_TEST_DIR): + utils_dst = os.path.join(tests_dst, "_utils_test") + for filename in os.listdir(AD_UTILS_TEST_DIR): + src_path = os.path.join(AD_UTILS_TEST_DIR, filename) + if os.path.isfile(src_path) and _should_copy(src_path): + dst_path = os.path.join(utils_dst, filename) + os.makedirs(utils_dst, exist_ok=True) + shutil.copy2(src_path, dst_path) + count += 1 + + # Create conftest.py for test discovery and imports + _create_test_conftest(tests_dst) + + # Create a stub for test_common.llm_data (used by some model tests) + _create_test_common_stub(tests_dst) + + return count + + +def _create_test_conftest(tests_dir: str) -> None: + """Create a conftest.py that configures the test environment for standalone mode.""" + content = textwrap.dedent("""\ + # SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + # SPDX-License-Identifier: Apache-2.0 + # + # Licensed under the Apache License, Version 2.0 (the "License"); + # you may not use this file except in compliance with the License. + # You may obtain a copy of the License at + # + # http://www.apache.org/licenses/LICENSE-2.0 + # + # Unless required by applicable law or agreed to in writing, software + # distributed under the License is distributed on an "AS IS" BASIS, + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + # See the License for the specific language governing permissions and + # limitations under the License. + + \"\"\"Conftest for standalone auto_deploy tests.\"\"\" + import importlib.util + import os + import sys + + _trtllm_spec = importlib.util.find_spec("tensorrt_llm") + if _trtllm_spec is not None: + raise RuntimeError( + "Standalone llmc tests must not be able to import tensorrt_llm; " + f"found {getattr(_trtllm_spec, 'origin', None)!r}" + ) + + # Add _utils_test to the Python path so test files can import from it + sys.path.insert(0, os.path.join(os.path.dirname(__file__), "_utils_test")) + + # Add the tests directory itself to the path for cross-test imports + sys.path.insert(0, os.path.dirname(__file__)) + """) + with open(os.path.join(tests_dir, "conftest.py"), "w") as f: + f.write(content) + + +def _create_test_common_stub(tests_dir: str) -> None: + """Create a stub for test_common.llm_data (provides HF model path resolution). + + In standalone mode, tests that need local model weights will be skipped + unless LLM_MODELS_ROOT is set. + """ + stub_dir = os.path.join(tests_dir, "test_common") + os.makedirs(stub_dir, exist_ok=True) + + with open(os.path.join(stub_dir, "__init__.py"), "w") as f: + f.write( + "# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION" + " & AFFILIATES. All rights reserved.\n" + "# SPDX-License-Identifier: Apache-2.0\n" + ) + + content = textwrap.dedent("""\ + # SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + # SPDX-License-Identifier: Apache-2.0 + # + # Licensed under the Apache License, Version 2.0 (the "License"); + # you may not use this file except in compliance with the License. + # You may obtain a copy of the License at + # + # http://www.apache.org/licenses/LICENSE-2.0 + # + # Unless required by applicable law or agreed to in writing, software + # distributed under the License is distributed on an "AS IS" BASIS, + # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + # See the License for the specific language governing permissions and + # limitations under the License. + + \"\"\"Stub for test_common.llm_data in standalone mode.\"\"\" + import os + from pathlib import Path + from unittest.mock import patch + + LLM_MODELS_ROOT = os.environ.get("LLM_MODELS_ROOT") + + + def llm_models_root(): + return Path(LLM_MODELS_ROOT) if LLM_MODELS_ROOT else None + + + def hf_id_to_local_model_dir(hf_id: str): + root = llm_models_root() + if root is None: + return hf_id # Fall back to HF hub download + # Try direct match + candidate = root / hf_id.split("/")[-1] + if candidate.exists(): + return str(candidate) + return hf_id + + + def with_mocked_hf_download_for_single_gpu(func): + return func # No-op in standalone mode + """) + with open(os.path.join(stub_dir, "llm_data.py"), "w") as f: + f.write(content) + + +# --------------------------------------------------------------------------- +# Example / e2e harness copying +# --------------------------------------------------------------------------- +def _copy_runners(output_dir: str) -> int: + """Copy the Tier-1 e2e harness into the standalone package under ``runners/trtllm/``. + + ``examples/auto_deploy/build_and_run_ad.py`` is copied to + ``runners/trtllm/build_and_run_llmc_trtllm.py`` (see ``EXAMPLE_FILES``) together + with its sibling ``model_registry/``. Because the script resolves the registry + relative to its own location (``Path(__file__).parent / "model_registry"``), + the rename + relocation are safe and ``--use-registry`` keeps working. + ``.py`` files get the usual ``auto_deploy -> llmc`` import rewrite (applied by + the caller); the ``model_registry`` YAML is data, copied verbatim. + """ + runners_dst = os.path.join(output_dir, "runners", "trtllm") + count = 0 + for src_name, dst_name in EXAMPLE_FILES.items(): + src = os.path.join(AD_EXAMPLES_SRC, src_name) + if os.path.isfile(src): + os.makedirs(runners_dst, exist_ok=True) + shutil.copy2(src, os.path.join(runners_dst, dst_name)) + count += 1 + for dname in EXAMPLE_DIRS: + src = os.path.join(AD_EXAMPLES_SRC, dname) + if os.path.isdir(src): + count += _copy_tree(src, os.path.join(runners_dst, dname)) + return count + + +# --------------------------------------------------------------------------- +# Package generation +# --------------------------------------------------------------------------- +def _create_pyproject_toml(output_dir: str, dependencies: list, dev_dependencies: list) -> None: + """Create a pyproject.toml for the standalone package.""" + deps_lines = "\n".join(f' "{dep}",' for dep in dependencies) + dev_deps_lines = "\n".join(f' "{dep}",' for dep in dev_dependencies) + + content = ( + "[build-system]\n" + 'requires = ["setuptools>=64", "wheel"]\n' + 'build-backend = "setuptools.build_meta"\n' + "\n" + "[project]\n" + 'name = "nvidia-llmc"\n' + 'version = "0.1.0"\n' + 'description = "llmc: standalone LLM compiler — ' + 'automatic model optimization and deployment for LLM inference"\n' + 'readme = "README.md"\n' + 'license = {text = "Apache-2.0"}\n' + 'requires-python = ">=3.10"\n' + "dependencies = [\n" + f"{deps_lines}\n" + "]\n" + "\n" + "[project.optional-dependencies]\n" + "dev = [\n" + f"{dev_deps_lines}\n" + "]\n" + "\n" + "[tool.setuptools.packages.find]\n" + 'include = ["llmc*"]\n' + "\n" + "[tool.pytest.ini_options]\n" + 'testpaths = ["tests"]\n' + ) + + with open(os.path.join(output_dir, "pyproject.toml"), "w") as f: + f.write(content) + + +def create_standalone_package(output_dir: str) -> None: + """Create the standalone llmc package at the given output directory. + + Safe to run against an existing git repository: only the managed paths + (source, tests, and packaging files) are deleted and regenerated. The .git + directory and any repo-specific files (e.g., .github/) are preserved. + After running, ``git add -A && git commit`` captures all changes. + """ + if not os.path.isdir(AUTO_DEPLOY_SRC): + print(f"ERROR: auto_deploy source not found at {AUTO_DEPLOY_SRC}", file=sys.stderr) + sys.exit(1) + + os.makedirs(output_dir, exist_ok=True) + + # Clean only the paths this script manages, preserving .git and other repo files + for name in _MANAGED_PATHS: + target = os.path.join(output_dir, name) + if os.path.isdir(target): + shutil.rmtree(target) + elif os.path.isfile(target): + os.remove(target) + + print(f"Creating standalone package at: {output_dir}") + + # 1. Copy auto_deploy source as top-level `llmc/` package. No import + # rewriting is needed: in-package imports are relative (enforced by + # the auto-deploy-import-discipline pre-commit hook). + ad_dst = os.path.join(output_dir, "llmc") + count = _copy_tree(AUTO_DEPLOY_SRC, ad_dst) + print(f" Copied {count} source files to llmc/") + + # 2. Copy and rewrite tests (tests use absolute self-imports by design). + test_count = _copy_tests(output_dir) + rewrite_count = _rewrite_imports_in_dir(os.path.join(output_dir, "tests")) + print(f" Copied {test_count} test files to tests/ ({rewrite_count} import rewrites)") + + # 2b. Copy the Tier-1 e2e harness into runners/ (build_and_run_llmc_trtllm.py + # + model_registry) and rewrite its imports auto_deploy -> llmc. YAML is + # left untouched. + runner_count = _copy_runners(output_dir) + runner_rewrites = _rewrite_imports_in_dir(os.path.join(output_dir, "runners")) + print( + f" Copied {runner_count} runner files to runners/trtllm/ ({runner_rewrites} import rewrites)" + ) + + # 3. Resolve dependencies and create pyproject.toml + pinned = _read_pinned_versions(TRTLLM_REQUIREMENTS) + dev_pinned = _read_pinned_versions(TRTLLM_DEV_REQUIREMENTS) + # Merge: dev_pinned has the same packages as pinned plus test-only packages + all_pinned = {**pinned, **dev_pinned} + dependencies = _resolve_dependencies(STANDALONE_DEPS, pinned) + dev_dependencies = _resolve_dependencies(DEV_DEPS, all_pinned) + _create_pyproject_toml(output_dir, dependencies, dev_dependencies) + print(f" Created pyproject.toml ({len(dependencies)} deps + {len(dev_dependencies)} dev deps)") + + # 4. Generate standalone LICENSE (only vendored projects in auto_deploy) + generate_license(output_dir) + print(f" Generated LICENSE ({len(VENDORED_PROJECTS)} vendored projects)") + + # 5. Generate ATTRIBUTIONS-Python.md (direct dependency licenses) + generate_attributions(output_dir, dependencies) + print(f" Generated ATTRIBUTIONS-Python.md ({len(dependencies)} direct deps)") + + # 6. Copy README + if os.path.exists(LLMC_README): + shutil.copy2(LLMC_README, os.path.join(output_dir, "README.md")) + print(" Copied README.md") + + # 7. Copy CONTRIBUTING.md + if os.path.exists(LLMC_CONTRIBUTING): + shutil.copy2(LLMC_CONTRIBUTING, os.path.join(output_dir, "CONTRIBUTING.md")) + print(" Copied CONTRIBUTING.md") + + # 8. Copy .gitignore + if os.path.exists(TRTLLM_GITIGNORE): + shutil.copy2(TRTLLM_GITIGNORE, os.path.join(output_dir, ".gitignore")) + print(" Copied .gitignore") + + # 9. Copy .editorconfig + if os.path.exists(TRTLLM_EDITORCONFIG): + shutil.copy2(TRTLLM_EDITORCONFIG, os.path.join(output_dir, ".editorconfig")) + print(" Copied .editorconfig") + + # 10. Copy OSS compliance files (CODE_OF_CONDUCT, SECURITY) + for src, name in ( + (TRTLLM_CODE_OF_CONDUCT, "CODE_OF_CONDUCT.md"), + (TRTLLM_SECURITY, "SECURITY.md"), + ): + if os.path.exists(src): + shutil.copy2(src, os.path.join(output_dir, name)) + print(f" Copied {name}") + + print(f"\nStandalone package created at: {output_dir}") + print("\nTo install:") + print(f" cd {output_dir}") + print(" uv venv .venv --python 3.12") + print(" source .venv/bin/activate") + print(" uv pip install -e '.[dev]'") + print("\nTo run tests: pytest tests/") + print( + 'To verify: python -c "from llmc._compat import TRTLLM_AVAILABLE; print(TRTLLM_AVAILABLE)"' + ) + print( + "To run e2e: python runners/trtllm/build_and_run_llmc_trtllm.py " + "--model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --use-registry (needs tensorrt-llm installed)" + ) -def main() -> None: - target = Path(__file__).resolve().parents[1] / "paragraf" / "create_standalone_package.py" - sys.path.insert(0, str(target.parent)) - runpy.run_path(str(target), run_name="__main__") +def main(): + parser = argparse.ArgumentParser( + description="Create a standalone llmc package from TensorRT-LLM source.", + ) + parser.add_argument( + "--output-dir", + default=os.path.join(REPO_ROOT, "build", "llmc_standalone"), + help="Output directory for the standalone package (default: build/llmc_standalone)", + ) + args = parser.parse_args() + create_standalone_package(os.path.abspath(args.output_dir)) if __name__ == "__main__": diff --git a/examples/auto_deploy/model_registry/configs/qwen3.5_moe_400b.yaml b/examples/auto_deploy/model_registry/configs/qwen3.5_moe_400b.yaml index 687fb88c8181..e8fc2f04f468 100644 --- a/examples/auto_deploy/model_registry/configs/qwen3.5_moe_400b.yaml +++ b/examples/auto_deploy/model_registry/configs/qwen3.5_moe_400b.yaml @@ -36,6 +36,8 @@ transforms: apply_sharding_hints: enabled: true allreduce_strategy: SYMM_MEM + # Shared expert is excluded from sharding for performance purpose + shard_layers: ["moe", "delta", "mha"] simple_shard_filter: "lm_head" multi_stream_moe: stage: compile diff --git a/examples/auto_deploy/paragraf/CONTRIBUTING.md b/examples/auto_deploy/paragraf/CONTRIBUTING.md deleted file mode 100644 index 0df6dc35400c..000000000000 --- a/examples/auto_deploy/paragraf/CONTRIBUTING.md +++ /dev/null @@ -1,56 +0,0 @@ -# Contributing to paragraf - -## Where to send PRs - -The `paragraf` standalone package is **generated from the AutoDeploy source tree inside [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)**. The mirror repo on GitHub is **read-only** — it is overwritten on every release by [`create_standalone_package.py`](./create_standalone_package.py). - -**Do not open pull requests against the standalone `paragraf` repo.** Any changes pushed there will be lost the next time the package is regenerated. - -The supported workflow is: - -1. **Fork** [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM). -1. Edit the AutoDeploy source under `tensorrt_llm/_torch/auto_deploy/` (and its tests under `tests/unittest/_torch/auto_deploy/` and `tests/unittest/auto_deploy/`). -1. **Optional:** validate your change against the standalone repo by regenerating it locally and running its test suite (see below). -1. Open a PR against `main` of `NVIDIA/TensorRT-LLM`. - -You're welcome to fork the standalone `paragraf` repo to experiment, iterate, or build on top — just remember the upstream source of truth is TensorRT-LLM. - -## Validating a change with the standalone build - -You can run the full standalone test suite locally before sending a PR. From the TensorRT-LLM checkout: - -```bash -# 1. Run the import-discipline lint (also enforced as a pre-commit hook) -python scripts/check_auto_deploy_imports.py - -# 2. Generate, install, and test the standalone package end-to-end -pytest tests/unittest/auto_deploy/standalone/ -q -``` - -`tests/unittest/auto_deploy/standalone/test_standalone_package.py` regenerates the `paragraf` tree, installs it in an isolated venv, and runs the copied unit tests against the standalone install. This is the same job that gates merges. - -To regenerate the standalone tree without running tests: - -```bash -python examples/auto_deploy/paragraf/create_standalone_package.py \ - --output-dir /path/to/paragraf_pkg -``` - -## PR conventions - -PRs against `NVIDIA/TensorRT-LLM` follow the project-wide rules: - -- **Title format:** `[JIRA/NVBUG/None][type] description` (e.g. `[None][feat] paragraf: add foo transform`). -- **DCO sign-off** is required: commit with `git commit -s`. Don't add AI tools or co-authors to the sign-off line. -- The pre-commit hooks (formatting, lint, the `auto-deploy-import-discipline` hook) run on commit. If they modify files, re-stage and commit again. - -For the rest (full coding guidelines, branching policy, CI commands), see [the TensorRT-LLM `CONTRIBUTING.md`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/CONTRIBUTING.md) and the [project-level `AGENTS.md`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/AGENTS.md). - -## Import discipline - -`paragraf` source is the same files as `tensorrt_llm/_torch/auto_deploy/` — copied verbatim, never rewritten. For that to work cleanly: - -- Imports **inside** the auto_deploy package must be **relative** (`from ..foo import bar`). -- Imports **to the rest of TensorRT-LLM** must be **absolute** (`from tensorrt_llm.X import Y`) and gated at runtime by `_compat.TRTLLM_AVAILABLE` so they fail gracefully in standalone mode. - -Both rules are enforced by `scripts/check_auto_deploy_imports.py`, which runs as a pre-commit hook on every change to `tensorrt_llm/_torch/auto_deploy/`. diff --git a/examples/auto_deploy/paragraf/README.md b/examples/auto_deploy/paragraf/README.md deleted file mode 100644 index 07d96ae6fc10..000000000000 --- a/examples/auto_deploy/paragraf/README.md +++ /dev/null @@ -1,182 +0,0 @@ -# 🔥🚀⚡ Paragraf - -**Paragraf** (Python package: `paragraf`) is the standalone, lightweight distribution of the AutoDeploy graph-transformation pipeline. It exposes the same `ModelFactory` and `InferenceOptimizer` building blocks as TensorRT-LLM's AutoDeploy backend, but with a minimal dependency footprint (PyTorch + Triton + FlashInfer), so you can prototype optimization pipelines or run inference without a full TRT-LLM install. - -The `paragraf` package is **generated** from the AutoDeploy source tree inside the [TensorRT-LLM repo](https://github.com/NVIDIA/TensorRT-LLM). The standalone repo is **read-only** — see [`CONTRIBUTING.md`](./CONTRIBUTING.md) for how to land changes. - -During the rename transition, `llmc` remains a deprecated import alias for `paragraf`, -the distribution metadata remains `nvidia-llmc`, and the legacy redirect environment -variable and runner entry point remain supported. New code should use the Paragraf -names shown below. - -For general AutoDeploy documentation (motivation, support matrix, feature overview), see the [official docs](https://nvidia.github.io/TensorRT-LLM/features/auto_deploy/auto-deploy.html). - -______________________________________________________________________ - -## Install - -Install with [uv](https://docs.astral.sh/uv/) directly from the latest commit: - -```bash -# https -uv pip install "git+https://github.com/NVIDIA/llm-compiler.git" - -# ssh -uv pip install "git+ssh://git@github.com/NVIDIA/llm-compiler.git" -``` - -For development, clone the repo and do an editable install: - -```bash -# https -git clone https://github.com/NVIDIA/llm-compiler.git -# or ssh -git clone git@github.com:NVIDIA/llm-compiler.git - -cd llm-compiler -uv venv .venv --python 3.12 -source .venv/bin/activate -uv pip install -e ".[dev]" -``` - -### Sanity check - -```python -from paragraf._compat import TRTLLM_AVAILABLE -print(f"TRT-LLM available: {TRTLLM_AVAILABLE}") # False in standalone mode -``` - -### Use the TensorRT-LLM runtime - -TensorRT-LLM releases that still import their bundled AutoDeploy package can be redirected to use -Paragraf as the compiler implementation: - -```bash -TRTLLM_REDIRECT_AD_TO_PARAGRAF=true \ -python runners/trtllm/build_and_run_paragraf_trtllm.py ... -``` - -The redirect is disabled by default and must be enabled before importing either Paragraf or bundled -AutoDeploy. It maps `tensorrt_llm._torch.auto_deploy.*` imports to the corresponding canonical -`paragraf.*` modules in the parent process and TensorRT-LLM MPI workers. - -In standalone mode the package uses the PyTorch, Triton, and FlashInfer kernel paths. TRT-LLM-only kernels (custom CUDA, optimized all-reduce, MoE fused kernels, the `pyexecutor` runtime) are skipped at registration time. - -### Run the bundled tests - -The generated repository contains every AutoDeploy test selected by the -TensorRT-LLM test-name classifier. Tests that need TensorRT-LLM skip cleanly -when its optional wheel is absent. - -```bash -pytest tests/ -``` - -To include tests that exercise TensorRT-LLM kernels, runtime, or test helpers, -install the optional wheel and explicitly enable the Paragraf redirect: - -```bash -uv pip install -e ".[dev,trtllm]" -TRTLLM_REDIRECT_AD_TO_PARAGRAF=true pytest tests/ -``` - -______________________________________________________________________ - -## Building a custom inference pipeline - -The two core abstractions in `paragraf` are: - -- **`ModelFactory`** — wraps a HuggingFace checkpoint (or any other source) and produces an initialized `nn.Module` plus dynamic-shape metadata for export. -- **`InferenceOptimizer`** — runs a configured pipeline of graph transforms (export → fuse → quantize → shard → compile → cache) over a model produced by a factory, against a `CachedSequenceInterface` that defines the input contract. - -The code below builds a custom optimization pipeline end-to-end, without going through the higher-level `LLM` API. It's the same machinery `LLM(...)` uses internally — direct access is useful when you want to insert your own transforms, target ONNX/MLIR export, or drive a non-standard runtime. - -```python -import torch - -from paragraf.models.factory import ModelFactoryRegistry -from paragraf.shim.interface import CachedSequenceInterface -from paragraf.transform.optimizer import InferenceOptimizer -from paragraf.utils.dist_config import DistConfig - -# 1. Build a ModelFactory. -# -# The registry is populated at import time. Built-in factories include -# "AutoModelForCausalLM" (text generation) and -# "AutoModelForImageTextToText" (multimodal). Factories know how to -# resolve a HF model id, instantiate the architecture, optionally load -# weights, and report dynamic shapes for export. -factory_cls = ModelFactoryRegistry.get("AutoModelForCausalLM") -factory = factory_cls( - model="TinyLlama/TinyLlama-1.1B-Chat-v1.0", - model_kwargs={"torch_dtype": "float16"}, - skip_loading_weights=False, # set True to inspect graph without weights - max_seq_len=2048, -) - -# 2. Define the runtime input contract via a CachedSequenceInterface. -# -# This object owns the dummy inputs (input_ids, position_ids, ...) used -# during export and tracing, and — when running with a TRT-LLM-style -# KV cache — manages cache tensors. In standalone mode the cache -# manager is a no-op; CachedSequenceInterface still drives the -# transform pipeline. -cache_seq = CachedSequenceInterface( - max_seq_len=2048, - max_batch_size=4, - device="cuda", - kv_cache_config=None, # or paragraf._compat.KvCacheConfig(...) - max_num_tokens=4 * 2048, - vocab_size_padded=factory.vocab_size_padded, -) - -# 3. Describe the transform pipeline. -# -# Each entry maps a registered transform name to its config. -# `stage` controls ordering across the pipeline; transforms are -# re-sorted by stage at construction time. The names below are -# illustrative — see `paragraf.transform.library` for the full set. -transforms_config = { - "export_to_gm": {"stage": "export"}, - "fuse_gemms": {"stage": "post_export"}, - "fuse_rmsnorm": {"stage": "post_export"}, - "shard_attention": {"stage": "sharding", "world_size": 1}, - "compile_model": {"stage": "compile", "backend": "torch-simple"}, - "initialize_cache": {"stage": "cache_init"}, -} - -# 4. Build the optimizer and run it. -# -# `dist_config` is required when sharding across devices; for a single -# GPU the default (world_size=1) is fine. -optimizer = InferenceOptimizer( - factory=factory, - config=transforms_config, - dist_config=DistConfig(world_size=1, rank=0, tp_size=1), -) -optimized_model = optimizer(cache_seq) - -# 5. Run the optimized model. -# -# The exact call signature depends on the runtime your pipeline -# targets — for prefill-style cached attention, the inputs are -# (input_ids, position_ids) plus the cache state owned by `cache_seq`. -input_ids = torch.tensor([[1, 2, 3, 4]], device="cuda") -position_ids = torch.arange(input_ids.size(1), device="cuda").unsqueeze(0) -with torch.inference_mode(): - logits = optimized_model(input_ids, position_ids) -print(logits.shape) -``` - -### Where to look next - -| Topic | Path | -|-------|------| -| Available transforms | `paragraf/transform/library/` | -| Available factories | `paragraf/models/`, `paragraf/models/custom/` | -| Custom ops & backends | `paragraf/custom_ops/` | -| Compile backends | `paragraf/compile/backends/` | -| Type compatibility shims | `paragraf/_compat.py` | -| Configuration data classes | `paragraf/transform/interface.py`, `paragraf/llm_args.py` | - -For higher-level usage, the `paragraf.LLM` class (available when running inside TRT-LLM) wraps all of the above behind a familiar generate API. In pure-standalone mode use `InferenceOptimizer` directly as shown. diff --git a/examples/auto_deploy/paragraf/create_standalone_package.py b/examples/auto_deploy/paragraf/create_standalone_package.py deleted file mode 100644 index d2e75e5cbf39..000000000000 --- a/examples/auto_deploy/paragraf/create_standalone_package.py +++ /dev/null @@ -1,1250 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Create a standalone paragraf package from the TensorRT-LLM source tree. - -This script copies the ``tensorrt_llm/_torch/auto_deploy`` source tree and -tests into a standalone pip-installable package. The output directory can -be pushed directly to the read-only standalone repository. - -Usage: - python create_standalone_package.py [--output-dir /path/to/output] - -The generated package uses ``paragraf`` as the canonical top-level Python package -name. During the compatibility window, ``llmc`` remains an import alias and -``nvidia-llmc`` remains the distribution name: - - from paragraf._compat import TRTLLM_AVAILABLE - from paragraf.custom_ops.attention_interface import SequenceInfo - -The ``auto_deploy`` source tree itself is copied verbatim — internal imports -must already be relative (enforced by the -``auto-deploy-import-discipline`` pre-commit hook), so no source rewriting -is required. Test files use absolute ``tensorrt_llm._torch.auto_deploy`` -imports by design and ARE rewritten to ``paragraf`` on copy. -""" - -import argparse -import os -import re -import shutil -import subprocess -import sys -import textwrap - -from _license_data import VENDORED_PROJECTS, generate_attributions, generate_license - -# --------------------------------------------------------------------------- -# Path constants -# --------------------------------------------------------------------------- -SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) -REPO_ROOT = os.path.abspath(os.path.join(SCRIPT_DIR, "..", "..", "..")) -AUTO_DEPLOY_SRC = os.path.join(REPO_ROOT, "tensorrt_llm", "_torch", "auto_deploy") -TRTLLM_REQUIREMENTS = os.path.join(REPO_ROOT, "requirements.txt") -TRTLLM_DEV_REQUIREMENTS = os.path.join(REPO_ROOT, "requirements-dev.txt") -TRTLLM_LICENSE = os.path.join(REPO_ROOT, "LICENSE") -TRTLLM_GITIGNORE = os.path.join(REPO_ROOT, ".gitignore") -TRTLLM_EDITORCONFIG = os.path.join(REPO_ROOT, ".editorconfig") -TRTLLM_CODE_OF_CONDUCT = os.path.join(REPO_ROOT, "CODE_OF_CONDUCT.md") -TRTLLM_SECURITY = os.path.join(REPO_ROOT, "SECURITY.md") -TRTLLM_ATTRIBUTIONS_PYTHON = os.path.join(REPO_ROOT, "ATTRIBUTIONS-Python.md") -PARAGRAF_README = os.path.join(SCRIPT_DIR, "README.md") -PARAGRAF_CONTRIBUTING = os.path.join(SCRIPT_DIR, "CONTRIBUTING.md") - -# Test source directories -AD_TESTS_DIR = os.path.join(REPO_ROOT, "tests", "unittest", "auto_deploy") -AD_TORCH_TESTS_DIR = os.path.join(REPO_ROOT, "tests", "unittest", "_torch", "auto_deploy") -AD_INTEGRATION_TESTS_DIR = os.path.join(REPO_ROOT, "tests", "integration", "defs") - -# Example/e2e harness sources (Tier-1 e2e: build_and_run_ad.py + model registry). -# These ship with the package so the standalone install can run e2e models via -# the same entrypoint as TRT-LLM. Python is rewritten auto_deploy -> paragraf; the -# model_registry YAML is data and copied verbatim. -AD_EXAMPLES_SRC = os.path.join(REPO_ROOT, "examples", "auto_deploy") -# Source filename -> destination filename in the standalone package. The e2e -# entrypoint is renamed to make its scope explicit in the paragraf distribution. -EXAMPLE_FILES = {"build_and_run_ad.py": "build_and_run_paragraf_trtllm.py"} -EXAMPLE_DIRS = ["model_registry"] -LEGACY_RUNNER_NAME = "build_and_run_llmc_trtllm.py" -LEGACY_RUNNER_WRAPPER = textwrap.dedent("""\ - #!/usr/bin/env python3 - # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # SPDX-License-Identifier: Apache-2.0 - # - # Licensed under the Apache License, Version 2.0 (the "License"); - # you may not use this file except in compliance with the License. - # You may obtain a copy of the License at - # - # http://www.apache.org/licenses/LICENSE-2.0 - # - # Unless required by applicable law or agreed to in writing, software - # distributed under the License is distributed on an "AS IS" BASIS, - # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - # See the License for the specific language governing permissions and - # limitations under the License. - - if __package__: - from .build_and_run_paragraf_trtllm import * - else: - from build_and_run_paragraf_trtllm import * - - if __name__ == "__main__": - main() - """) - -# --------------------------------------------------------------------------- -# Configuration -# --------------------------------------------------------------------------- -COPY_EXTENSIONS = {".py", ".yaml", ".yml", ".json", ".txt", ".md"} -EXCLUDE_PATTERNS = {"__pycache__", ".pyc", ".pyo"} - -# Standalone runtime dependencies (version pins pulled from TRT-LLM requirements.txt) -STANDALONE_DEPS = [ - "torch", - "transformers", - "pydantic", - "pydantic-settings", - "triton", - "flashinfer-python", - "safetensors", - "accelerate", - "huggingface-hub", - "omegaconf", - "pyyaml", - "numpy", - "pillow", - "einops", - # Transitive deps that may not be pulled in by all installers/platforms - "six", - "importlib-metadata", - "werkzeug", - "StrEnum", - "graphviz", -] - -# Dev/test dependencies (version pins pulled from TRT-LLM requirements-dev.txt) -DEV_DEPS = [ - "pytest", - "pytest-timeout", - "pytest-xdist", - "pytest-cov", - "pytest-mock", - "pytest-asyncio", - "parameterized", - "cloudpickle", - "mpi4py", - "openai", - "requests", - "scipy", -] - -# These tests validate the generator and its output. They belong to the source -# repository rather than the generated package. -SOURCE_ONLY_TEST_DIRS = {"standalone"} - -# Source test names retain their AutoDeploy identity inside TensorRT-LLM. Rename -# them only in the generated repository, where they exercise Paragraf through -# the optional TensorRT-LLM integration. -PARAGRAF_TRTLLM_TEST_RENAMES = { - "test_llm_api_autodeploy.py": "test_llm_api_paragraf_trtllm.py", - "test_ad_disagg.py": "test_paragraf_trtllm_disagg.py", - "test_ad_disagg_trtllm_serve.py": "test_paragraf_trtllm_disagg_serve.py", - "test_ad_guided_decoding.py": "test_paragraf_trtllm_guided_decoding.py", - "test_ad_speculative_decoding.py": "test_paragraf_trtllm_speculative_decoding.py", - "test_ad_dist_strategies.py": "test_paragraf_trtllm_dist_strategies.py", - "test_ad_allreduce_strategies.py": "test_paragraf_trtllm_allreduce_strategies.py", - "test_ad_build_small_multi.py": "test_paragraf_trtllm_build_small_multi.py", - "test_ad_moe_op.py": "test_paragraf_trtllm_moe_op.py", - "test_ad_executor_swa_eviction.py": "test_paragraf_trtllm_executor_swa_eviction.py", - "test_create_ad_executor.py": "test_create_paragraf_trtllm_executor.py", - "test_ad_build_small_single.py": "test_paragraf_trtllm_build_small_single.py", - "test_ad_guided_decoding_regex.py": "test_paragraf_trtllm_guided_decoding_regex.py", - "test_ad_trtllm_bench.py": "test_paragraf_trtllm_bench.py", - "test_ad_trtllm_sampler.py": "test_paragraf_trtllm_sampler.py", - "test_ad_trtllm_serve.py": "test_paragraf_trtllm_serve.py", -} -SOURCE_TEST_NAMES_BY_GENERATED_NAME = { - generated_name: source_name - for source_name, generated_name in PARAGRAF_TRTLLM_TEST_RENAMES.items() -} - -# Tests in this set are copied, but only collected when the optional TensorRT-LLM -# wheel is enabled through TRTLLM_REDIRECT_AD_TO_PARAGRAF. Keeping this explicit -# also covers indirect dependencies where a rewritten ``paragraf`` import loads -# a module that depends on TensorRT-LLM. -OPTIONAL_TRTLLM_TEST_FILES = { - # TRT-LLM kernel tests - "test_trtllm_moe.py", - "test_trtllm_attention_op.py", - "test_fla_cached_gated_delta_rule.py", - "test_flashinfer_trtllm_mla_op.py", - "test_trtllm_mla_op.py", - "test_fuse_trtllm_attention_quant_fp8.py", - "test_fuse_relu2_quant_nvfp4.py", - "test_moe_fusion.py", - "test_trtllm_gen_diag.py", - # Standalone flashinfer ROPE path has a known BF16 strided-interleaved mismatch. - "test_rope_op_variants.py", - # QKV fusion → trtllm cache insertion (TRT-LLM attention backend only) - "test_gemm_fusion_trtllm.py", - # Require TRT-LLM LlmArgs / runtime - "test_eagle.py", - "test_modeling_nemotron_h.py", - "test_example_configs.py", - "test_hybrid_patches.py", - "test_captured_graph.py", - # Require external scripts (build_and_run_ad.py) - "test_llama4_vlm_patch.py", - "test_mistral3_patches.py", - # Require TRT-LLM test utils not in standalone - "test_ad_moe_op.py", - "test_triton_moe.py", - # Require onnx (optional dep) - "test_export_fp8_linear_to_onnx.py", - # Depend on TRT-LLM mamba/fla kernels (relative imports beyond auto_deploy) - "test_mamba_rms_norm.py", - "test_triton_rms_norm.py", - "test_fuse_rmsnorm.py", - "test_fused_add_rms_norm.py", - "test_fuse_l2norm.py", - # Require TRT-LLM KVCacheManager or runtime - "test_kv_cache.py", - "test_torch_gated_delta_rule_cache.py", - "test_gated_delta_rule_cache.py", - "test_kv_cache_transformers.py", - # trtllm attention backend (insert_cached_attention backend=trtllm) not available standalone - "test_kv_cache_trtllm_multipool.py", - # Require TRT-LLM CUDA causal conv / mamba kernels (ops not registered standalone) - "test_cuda_causal_conv_cached_op.py", - "test_triton_causal_conv_cached_op.py", - "test_triton_mamba_cached_op.py", - "test_flashinfer_mamba_cached_op.py", - # Require TRT-LLM custom ops (dsv3_router_gemm_op, noaux_tc_op, etc.) - "test_deepseek_custom.py", - "test_glm4_moe_modeling.py", - "test_glm4_moe_lite_modeling.py", - "test_glm_moe_dsa_modeling.py", - # Full-model tests hit standalone-incompatible HF cache behavior. - "test_granite_moe_hybrid_modeling.py", - # Imports triton_kernels, which is not a standalone dependency. - "test_mxfp4_moe_layout.py", - # Require TRT-LLM distributed ops (trtllm_dist_all_gather) - "test_gather_logits_before_lm_head.py", - # Multimodal processors depend on TensorRT-LLM multimodal request types. - "test_gemma4_modeling.py", - "test_qwen3_5_moe.py", - # Hardware-specific (requires H100+ shared memory) - "test_triton_mla_op.py", - # Require TRT-LLM ops (noaux_tc_op) — split from test_export.py - "test_export_glm4_moe_lite.py", - # fuse_fp8_linear / fuse_nvfp4_linear / fuse_finegrained_fp8_linear transforms - # live in fuse_quant.py which imports tensorrt_llm.quantization.utils.fp8_utils; - # the module is silently skipped in standalone so the transforms aren't registered. - "test_quant_fusion.py", - # Imports utils.util.skip_pre_blackwell (not shipped in standalone) and exercises - # fuse_finegrained_fp8_swiglu which depends on TRT-LLM runtime. - "test_finegrained_fp8_swiglu.py", - # Exercise trtllm-gen MXFP4 MoE kernels (Blackwell-only) and import the - # prepare_trtllm_gen_moe_mxfp4_weights / utils.util helpers not in standalone. - "test_fuse_mxfp4_moe.py", - "test_trtllm_quant_mxfp4_trtllm_gen_moe.py", -} - -# Multi-GPU tests known to run without the optional TensorRT-LLM wheel. Other -# AutoDeploy multi-GPU tests are still copied, but receive the optional-wheel -# collection guard. -PURE_STANDALONE_MULTIGPU_TEST_FILES = { - "custom_ops/test_dist.py", - "custom_ops/test_sharded_rmsnorm.py", - "smoke/test_ad_build_small_multi.py", - "transformations/library/test_apply_sharding_hints.py", - "transformations/library/test_bmm_sharding.py", - "transformations/library/test_ep_sharding.py", - "transformations/library/test_rmsnorm_sharding.py", - "transformations/library/test_sharding_num_correctness.py", - "transformations/library/test_step3p7_sharding_ir.py", - "transformations/library/test_tp_sharding.py", -} - -# AutoDeploy integration files selected by the same classifier used by CI. -AUTODEPLOY_TEST_RE = re.compile(r"auto_?deploy|_ad_", re.IGNORECASE) -TEST_FILE_RE = re.compile(r"(?:^test_.*|.*_test)\.py$") - -# Support files needed by the selected integration tests. The source integration -# conftest is intentionally not copied because it pulls in the complete TRT-LLM -# CI harness; a focused replacement is generated below. -INTEGRATION_SUPPORT_FILES = ( - "__init__.py", - "common.py", - "trt_test_alternative.py", - "accuracy/__init__.py", - "accuracy/accuracy_core.py", - "accuracy/video_mme.py", - "disaggregated/disagg_test_utils.py", -) -INTEGRATION_SUPPORT_DIRS = ("accuracy/references",) - -# Additional source-tree test helpers imported by copied AutoDeploy tests. -TORCH_TEST_SUPPORT_FILES = ( - "tests/unittest/_torch/__init__.py", - "tests/unittest/_torch/helpers.py", -) - -# Import path rewrite: old -> new (applied to test files only). -_IMPORT_REWRITE = "tensorrt_llm._torch.auto_deploy" -_IMPORT_TARGET = "paragraf" -_BUILD_AND_RUN_AD_IMPORT = "from build_and_run_ad import ExperimentConfig, main" -_TRTLLM_IMPORT_RE = re.compile( - r"(?m)^(?:from|import) " - r"(?:tensorrt_llm(?:\.|\b)|paragraf\.models\.custom\.modeling_gpt_oss(?:\.|\b))" -) -_PARAGRAF_OPTIONAL_TRTLLM_GUARD = """ -_trtllm_environ = __import__("os").environ -_trtllm_redirect_value = _trtllm_environ.get("TRTLLM_REDIRECT_AD_TO_PARAGRAF") -if _trtllm_redirect_value is None: - _trtllm_redirect_value = _trtllm_environ.get("TRTLLM_REDIRECT_AD_TO_LLMC", "") -_trtllm_redirect_value = _trtllm_redirect_value.lower() -if _trtllm_redirect_value not in {"1", "true", "yes", "on"}: - pytest.skip( - "Paragraf optional TRT-LLM tests require TRTLLM_REDIRECT_AD_TO_PARAGRAF=true", - allow_module_level=True, - ) -pytest.importorskip("tensorrt_llm")""" -_PARAGRAF_TRTLLM_RUNNER_IMPORT = ( - "from runners.trtllm.build_and_run_paragraf_trtllm import ExperimentConfig, main" -) - -# Paths that the script owns and regenerates on every run. -# Everything else in the output directory (e.g., .git/, .github/) is preserved -# and owned by the standalone repo itself. -_MANAGED_PATHS = [ - "paragraf", - # Remove the package directory produced before the Paragraf rename. - "llmc", - "tests", - # Remove the source-shaped test-data path produced by older generators. - "examples/auto_deploy", - "runners", - "pyproject.toml", - "README.md", - "LICENSE", - "ATTRIBUTIONS-Python.md", - "CONTRIBUTING.md", - ".gitignore", - ".editorconfig", - "CODE_OF_CONDUCT.md", - "SECURITY.md", - "ATTRIBUTIONS-Python.md", -] - - -# --------------------------------------------------------------------------- -# Helper functions -# --------------------------------------------------------------------------- -def _should_copy(filepath: str) -> bool: - for pattern in EXCLUDE_PATTERNS: - if pattern in filepath: - return False - basename = os.path.basename(filepath) - if ".bak." in basename or basename.endswith("~"): - return False - _, ext = os.path.splitext(filepath) - return ext in COPY_EXTENSIONS - - -def _tracked_files_under(directory: str) -> list[str]: - """Return tracked files below a source directory, with an archive fallback.""" - relative_directory = os.path.relpath(directory, REPO_ROOT) - try: - result = subprocess.run( - ["git", "-C", REPO_ROOT, "ls-files", "--", relative_directory], - check=True, - capture_output=True, - text=True, - ) - except (FileNotFoundError, subprocess.CalledProcessError): - discovered_files = [] - for root, dirs, files in os.walk(directory): - dirs[:] = [directory_name for directory_name in dirs if directory_name != "__pycache__"] - discovered_files.extend(os.path.join(root, filename) for filename in files) - return discovered_files - - return [os.path.join(REPO_ROOT, path) for path in result.stdout.splitlines()] - - -def _copy_file(src_path: str, dst_path: str) -> int: - if not os.path.isfile(src_path) or not _should_copy(src_path): - return 0 - os.makedirs(os.path.dirname(dst_path), exist_ok=True) - shutil.copy2(src_path, dst_path) - return 1 - - -def _copy_tracked_tree(src_dir: str, dst_dir: str) -> int: - """Copy tracked files from a tree, or all files in a source archive.""" - count = 0 - for src_path in _tracked_files_under(src_dir): - rel_path = os.path.relpath(src_path, src_dir) - count += _copy_file(src_path, os.path.join(dst_dir, rel_path)) - return count - - -def _copy_tree(src_dir: str, dst_dir: str) -> int: - """Copy files from src_dir to dst_dir, preserving directory structure.""" - count = 0 - for root, dirs, files in os.walk(src_dir): - dirs[:] = [d for d in dirs if d != "__pycache__"] - for filename in files: - src_path = os.path.join(root, filename) - if not _should_copy(src_path): - continue - rel_path = os.path.relpath(src_path, src_dir) - dst_path = os.path.join(dst_dir, rel_path) - os.makedirs(os.path.dirname(dst_path), exist_ok=True) - shutil.copy2(src_path, dst_path) - count += 1 - return count - - -def _rewrite_generated_test_layout(filepath: str, content: str) -> str: - """Rewrite TensorRT-LLM source-tree paths in selected generated tests.""" - - def replace_required(old: str, new: str) -> None: - nonlocal content - if old not in content: - raise ValueError(f"Expected standalone layout pattern not found in {filepath}: {old}") - content = content.replace(old, new) - - def substitute_required(pattern: str, replacement: str) -> None: - nonlocal content - content, count = re.subn(pattern, replacement, content) - if count == 0: - raise ValueError( - f"Expected standalone layout pattern not found in {filepath}: {pattern}" - ) - - filename = os.path.basename(filepath) - model_registry_pattern = ( - r"""["']examples["']\s*/\s*["']auto_deploy["']\s*/\s*""" - r"""["']model_registry["']""" - ) - runner_model_registry = '"runners" / "trtllm" / "model_registry"' - - if filename == "test_llm_api_paragraf_trtllm.py": - substitute_required(model_registry_pattern, runner_model_registry) - for config_name in ("nano_v3.yaml", "super_v3.yaml"): - substitute_required( - rf"""["']examples["']\s*/\s*["']auto_deploy["']\s*/\s*["']{config_name}["']""", - f'{runner_model_registry} / "configs" / "{config_name}"', - ) - elif filename == "test_mrope_delta_cache.py": - replace_required( - "return Path(__file__).resolve().parents[6]", - "return Path(__file__).resolve().parents[4]", - ) - replace_required( - '_repo_root() / "tensorrt_llm" / "_torch" / "auto_deploy" / "config"', - '_repo_root() / "paragraf" / "config"', - ) - substitute_required(model_registry_pattern, runner_model_registry) - elif filename == "test_example_configs.py": - replace_required( - '_AD_EXAMPLES_DIR = _REPO_ROOT / "examples" / "auto_deploy"', - '_AD_EXAMPLES_DIR = _REPO_ROOT / "runners" / "trtllm" / "model_registry"', - ) - - return content - - -def _rewrite_imports_in_file( - filepath: str, - *, - optional_trtllm_guards: bool = True, - force_optional_trtllm_guard: bool = False, -) -> int: - """Rewrite imports in a copied test file for standalone mode. - - Source files inside ``tensorrt_llm/_torch/auto_deploy`` already use - relative imports (enforced by the ``auto-deploy-import-discipline`` - pre-commit hook), so no rewriting is needed for them. Tests, however, - are written against the canonical absolute path - ``tensorrt_llm._torch.auto_deploy`` and need to be rewritten to - ``paragraf``. Cross-package types (e.g. ``KvCacheConfig``, - ``ActivationType``) are sourced via ``..._torch.auto_deploy._compat``, - so the primary rewrite handles them too. - - Returns the number of line-level changes made. - """ - with open(filepath) as f: - content = f.read() - - original = content - content = content.replace(_IMPORT_REWRITE, _IMPORT_TARGET) - content = content.replace("from auto_deploy.", "from ") - content = re.sub( - r"(?m)^from (test_[A-Za-z0-9_]+|disagg_test_utils) import ", - r"from .\1 import ", - content, - ) - content = content.replace( - 'pytest_plugins = ["disagg_test_utils"]', - 'pytest_plugins = ["integration.defs.disaggregated.disagg_test_utils"]', - ) - content = content.replace( - "_REPO_ROOT = pathlib.Path(__file__).resolve().parents[5]", - "_REPO_ROOT = pathlib.Path(__file__).resolve().parents[3]", - ) - content = _rewrite_generated_test_layout(filepath, content) - - def ensure_imports(before_pos: int, *imports: str) -> None: - nonlocal content - prefix = content[:before_pos] - missing_imports = [ - import_name for import_name in imports if f"import {import_name}\n" not in prefix - ] - if not missing_imports: - return - first_import = re.search(r"(?m)^(?:import|from) ", content) - if first_import is None: - raise ValueError(f"No import block found in {filepath}") - content = ( - content[: first_import.start()] - + "\n".join(f"import {import_name}" for import_name in missing_imports) - + "\n" - + content[first_import.start() :] - ) - - def insert_optional_trtllm_guard() -> None: - nonlocal content - if _PARAGRAF_OPTIONAL_TRTLLM_GUARD in content: - return - pytest_import = re.search(r"(?m)^import pytest\n", content) - if pytest_import is None: - raise ValueError(f"No pytest import found in {filepath}") - content = ( - content[: pytest_import.end()] - + _PARAGRAF_OPTIONAL_TRTLLM_GUARD - + "\n" - + content[pytest_import.end() :] - ) - - if optional_trtllm_guards and _BUILD_AND_RUN_AD_IMPORT in content: - build_import_pos = content.index(_BUILD_AND_RUN_AD_IMPORT) - ensure_imports(build_import_pos, "os", "pytest") - insert_optional_trtllm_guard() - content = content.replace(_BUILD_AND_RUN_AD_IMPORT, _PARAGRAF_TRTLLM_RUNNER_IMPORT) - elif optional_trtllm_guards and force_optional_trtllm_guard: - ensure_imports(len(content), "os", "pytest") - insert_optional_trtllm_guard() - elif optional_trtllm_guards: - trtllm_import = _TRTLLM_IMPORT_RE.search(content) - if trtllm_import is not None: - ensure_imports(trtllm_import.start(), "os", "pytest") - insert_optional_trtllm_guard() - - replacements = sum(1 for a, b in zip(original, content) if a != b) # rough count - if content != original: - with open(filepath, "w") as f: - f.write(content) - # Count actual line-level changes - replacements = sum(1 for a, b in zip(original.splitlines(), content.splitlines()) if a != b) - - return replacements - - -def _requires_optional_trtllm_guard(filepath: str, tests_dir: str) -> bool: - relative_path = os.path.relpath(filepath, tests_dir).replace("\\", "/") - if not TEST_FILE_RE.fullmatch(os.path.basename(filepath)): - return False - generated_basename = os.path.basename(filepath) - source_basename = SOURCE_TEST_NAMES_BY_GENERATED_NAME.get( - generated_basename, generated_basename - ) - path_parts = relative_path.split("/") - if relative_path.startswith("integration/"): - return True - if "shim" in path_parts: - return True - if source_basename in OPTIONAL_TRTLLM_TEST_FILES: - return True - if relative_path.startswith("multigpu/"): - multigpu_parts = relative_path.removeprefix("multigpu/").split("/") - multigpu_parts[-1] = source_basename - multigpu_path = "/".join(multigpu_parts) - return multigpu_path not in PURE_STANDALONE_MULTIGPU_TEST_FILES - return False - - -def _rewrite_imports_in_dir(directory: str, *, optional_trtllm_guards: bool = True) -> int: - """Rewrite imports in all .py files in a directory tree.""" - total = 0 - for root, _, files in os.walk(directory): - for filename in files: - if filename.endswith(".py"): - total += _rewrite_imports_in_file( - os.path.join(root, filename), - optional_trtllm_guards=optional_trtllm_guards, - force_optional_trtllm_guard=( - optional_trtllm_guards - and _requires_optional_trtllm_guard(os.path.join(root, filename), directory) - ), - ) - return total - - -def _read_pinned_versions(req_file: str) -> dict: - """Read requirements.txt and extract package->version-spec mapping.""" - versions = {} - if not os.path.exists(req_file): - return versions - with open(req_file) as f: - for line in f: - line = line.strip() - if not line or line.startswith("#") or line.startswith("-"): - continue - line = line.split("#")[0].strip() - # Handle semicolons (environment markers like ; python_version >= "3.10") - line = line.split(";")[0].strip() - match = re.match(r"^([a-zA-Z0-9_-]+(?:\[[^\]]+\])?)(.*)", line) - if match: - pkg_name = match.group(1).split("[")[0].lower() - version_spec = match.group(2).strip() - if version_spec: - versions[pkg_name] = version_spec - return versions - - -def _resolve_dependencies(dep_names: list, pinned: dict) -> list: - """Resolve dependency list by adding version pins from requirements.""" - resolved = [] - for name in dep_names: - extras = "" - if "[" in name: - base, extras_part = name.split("[", 1) - extras = f"[{extras_part}" - else: - base = name - version = pinned.get(base.lower(), "") - resolved.append(f"{base}{extras}{version}") - return resolved - - -# --------------------------------------------------------------------------- -# Test copying -# --------------------------------------------------------------------------- -def _copy_tests(output_dir: str) -> int: - """Copy auto_deploy test files to the standalone package tests/ directory.""" - tests_dst = os.path.join(output_dir, "tests") - count = 0 - - # Copy every tracked legacy AutoDeploy unit-test file except the tests of - # this generator itself. Optional TensorRT-LLM dependencies are handled by - # collection guards after the files are copied. - for src_path in _tracked_files_under(AD_TESTS_DIR): - rel_path = os.path.relpath(src_path, AD_TESTS_DIR) - if any(part in SOURCE_ONLY_TEST_DIRS for part in rel_path.split(os.sep)): - continue - generated_name = PARAGRAF_TRTLLM_TEST_RENAMES.get( - os.path.basename(rel_path), os.path.basename(rel_path) - ) - generated_rel_path = os.path.join(os.path.dirname(rel_path), generated_name) - count += _copy_file(src_path, os.path.join(tests_dst, generated_rel_path)) - - # Copy every tracked test from the newer unit-test tree. This avoids an - # allowlist that silently misses tests added alongside new AutoDeploy code. - for src_path in _tracked_files_under(AD_TORCH_TESTS_DIR): - rel_path = os.path.relpath(src_path, AD_TORCH_TESTS_DIR) - path_parts = rel_path.split(os.sep) - if path_parts[0] == "unit": - path_parts = path_parts[1:] - generated_rel_path = os.path.join(*path_parts) - generated_path = os.path.join(tests_dst, generated_rel_path) - if os.path.exists(generated_path): - raise FileExistsError(f"Generated test path collision for {src_path}: {generated_path}") - count += _copy_file( - src_path, - generated_path, - ) - - # The CI classifier also finds a small number of AutoDeploy integration - # tests outside the unit-test roots. Copy those tests and their focused - # support modules without bringing in the complete TensorRT-LLM CI suite. - integration_tests_root = os.path.join(REPO_ROOT, "tests", "integration") - for src_path in _tracked_files_under(AD_INTEGRATION_TESTS_DIR): - rel_from_tests = os.path.relpath(src_path, integration_tests_root) - if not TEST_FILE_RE.fullmatch(os.path.basename(src_path)): - continue - if not AUTODEPLOY_TEST_RE.search(rel_from_tests.replace("\\", "/")): - continue - generated_name = PARAGRAF_TRTLLM_TEST_RENAMES.get( - os.path.basename(rel_from_tests), os.path.basename(rel_from_tests) - ) - generated_rel_path = os.path.join(os.path.dirname(rel_from_tests), generated_name) - count += _copy_file( - src_path, - os.path.join(tests_dst, "integration", generated_rel_path), - ) - - for rel_path in INTEGRATION_SUPPORT_FILES: - count += _copy_file( - os.path.join(AD_INTEGRATION_TESTS_DIR, rel_path), - os.path.join(tests_dst, "integration", "defs", rel_path), - ) - for rel_path in INTEGRATION_SUPPORT_DIRS: - count += _copy_tracked_tree( - os.path.join(AD_INTEGRATION_TESTS_DIR, rel_path), - os.path.join(tests_dst, "integration", "defs", rel_path), - ) - - for rel_path in TORCH_TEST_SUPPORT_FILES: - count += _copy_file( - os.path.join(REPO_ROOT, rel_path), - os.path.join(tests_dst, os.path.relpath(rel_path, "tests/unittest")), - ) - - # Create conftest.py for test discovery and imports - _create_test_package_init_files(tests_dst) - _create_test_conftest(tests_dst) - _create_integration_conftest(tests_dst) - - # Create a stub for test_common.llm_data (used by some model tests) - _create_test_common_stub(tests_dst) - _create_test_utils_stub(tests_dst) - - return count - - -def _create_test_package_init_files(tests_dir: str) -> None: - """Give copied test directories stable package-qualified module names.""" - content = ( - "# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION" - " & AFFILIATES. All rights reserved.\n" - "# SPDX-License-Identifier: Apache-2.0\n" - ) - for root, dirs, _ in os.walk(tests_dir): - dirs[:] = [directory for directory in dirs if directory != "__pycache__"] - if root == tests_dir: - continue - init_path = os.path.join(root, "__init__.py") - if not os.path.exists(init_path): - with open(init_path, "w") as f: - f.write(content) - - -def _create_test_conftest(tests_dir: str) -> None: - """Create a conftest.py that configures the test environment for standalone mode.""" - content = textwrap.dedent("""\ - # SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # SPDX-License-Identifier: Apache-2.0 - # - # Licensed under the Apache License, Version 2.0 (the "License"); - # you may not use this file except in compliance with the License. - # You may obtain a copy of the License at - # - # http://www.apache.org/licenses/LICENSE-2.0 - # - # Unless required by applicable law or agreed to in writing, software - # distributed under the License is distributed on an "AS IS" BASIS, - # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - # See the License for the specific language governing permissions and - # limitations under the License. - - \"\"\"Conftest for standalone auto_deploy tests.\"\"\" - import importlib.util - import os - import sys - from pathlib import Path - - import pytest - - _trtllm_redirect_value = os.environ.get("TRTLLM_REDIRECT_AD_TO_PARAGRAF") - if _trtllm_redirect_value is None: - _trtllm_redirect_value = os.environ.get("TRTLLM_REDIRECT_AD_TO_LLMC", "") - _allow_trtllm_redirect = _trtllm_redirect_value.lower() in { - "1", "true", "yes", "on" - } - _trtllm_spec = importlib.util.find_spec("tensorrt_llm") - if _trtllm_spec is not None and not _allow_trtllm_redirect: - raise RuntimeError( - "Standalone paragraf tests must not be able to import tensorrt_llm; " - "set TRTLLM_REDIRECT_AD_TO_PARAGRAF=true only for optional TRT-LLM tests; " - f"found {getattr(_trtllm_spec, 'origin', None)!r}" - ) - - _tests_dir = os.path.dirname(__file__) - _package_root = os.path.dirname(_tests_dir) - _integration_tests_dir = os.path.join(_tests_dir, "integration") - - # Add generated package/test roots to the Python path so tests can import - # local paragraf, runners, integration helpers, and _utils_test even - # under safe-path settings. - sys.path.insert(0, _package_root) - sys.path.insert(0, _tests_dir) - sys.path.insert(0, _integration_tests_dir) - sys.path.insert(0, os.path.join(_tests_dir, "_utils_test")) - - - @pytest.fixture(scope="module") - def llm_root(): - env_root = os.environ.get("LLM_ROOT") - if env_root: - return Path(env_root) - return Path(_package_root) - """) - with open(os.path.join(tests_dir, "conftest.py"), "w") as f: - f.write(content) - - -def _create_integration_conftest(tests_dir: str) -> None: - """Create the focused helpers needed by copied AutoDeploy integration tests.""" - defs_dir = os.path.join(tests_dir, "integration", "defs") - os.makedirs(defs_dir, exist_ok=True) - content = textwrap.dedent("""\ - # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # SPDX-License-Identifier: Apache-2.0 - - import os - from pathlib import Path - - import pytest - import torch - - - def get_llm_root(): - return str(Path(__file__).resolve().parents[3]) - - - def llm_models_root(): - models_root = os.environ.get("LLM_MODELS_ROOT") - if not models_root: - pytest.skip("LLM_MODELS_ROOT is required for AutoDeploy integration tests") - return models_root - - - def get_sm_version(): - if not torch.cuda.is_available(): - return 0 - major, minor = torch.cuda.get_device_capability(0) - return major * 10 + minor - - - def get_device_count(): - return torch.cuda.device_count() - - - def get_device_memory(): - if not torch.cuda.is_available(): - return 0 - return torch.cuda.get_device_properties(0).total_memory // (1024 * 1024) - - - def check_device_contain(keyword_list): - if not torch.cuda.is_available(): - return False - device_name = torch.cuda.get_device_name(0) - return any(keyword in device_name for keyword in keyword_list) - - - skip_pre_ada = pytest.mark.skipif( - get_sm_version() < 89, - reason="This test is not supported in pre-Ada architecture", - ) - skip_pre_hopper = pytest.mark.skipif( - get_sm_version() < 90, - reason="This test is not supported in pre-Hopper architecture", - ) - skip_pre_blackwell = pytest.mark.skipif( - get_sm_version() < 100, - reason="This test is not supported in pre-Blackwell architecture", - ) - - - @pytest.fixture(autouse=True) - def _apply_resource_markers(request): - device_marker = request.node.get_closest_marker("skip_less_device") - if device_marker and get_device_count() < device_marker.args[0]: - pytest.skip(f"Test requires {device_marker.args[0]} GPUs") - - for memory_marker in request.node.iter_markers("skip_less_device_memory"): - if get_device_memory() < memory_marker.args[0]: - pytest.skip(f"Test requires {memory_marker.args[0]} MiB of GPU memory") - """) - with open(os.path.join(defs_dir, "conftest.py"), "w") as f: - f.write(content) - - -def _create_test_common_stub(tests_dir: str) -> None: - """Create a stub for test_common.llm_data (provides HF model path resolution). - - In standalone mode, tests that need local model weights will be skipped - unless LLM_MODELS_ROOT is set. - """ - stub_dir = os.path.join(tests_dir, "test_common") - os.makedirs(stub_dir, exist_ok=True) - - with open(os.path.join(stub_dir, "__init__.py"), "w") as f: - f.write( - "# SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION" - " & AFFILIATES. All rights reserved.\n" - "# SPDX-License-Identifier: Apache-2.0\n" - ) - - content = textwrap.dedent("""\ - # SPDX-FileCopyrightText: Copyright (c) 2024-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # SPDX-License-Identifier: Apache-2.0 - # - # Licensed under the Apache License, Version 2.0 (the "License"); - # you may not use this file except in compliance with the License. - # You may obtain a copy of the License at - # - # http://www.apache.org/licenses/LICENSE-2.0 - # - # Unless required by applicable law or agreed to in writing, software - # distributed under the License is distributed on an "AS IS" BASIS, - # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - # See the License for the specific language governing permissions and - # limitations under the License. - - \"\"\"Stub for test_common.llm_data in standalone mode.\"\"\" - import os - from pathlib import Path - from unittest.mock import patch - - LLM_MODELS_ROOT = os.environ.get("LLM_MODELS_ROOT") - - - def llm_models_root(): - return Path(LLM_MODELS_ROOT) if LLM_MODELS_ROOT else None - - - def hf_id_to_local_model_dir(hf_id: str): - root = llm_models_root() - if root is None: - return hf_id # Fall back to HF hub download - # Try direct match - candidate = root / hf_id.split("/")[-1] - if candidate.exists(): - return str(candidate) - return hf_id - - - def with_mocked_hf_download_for_single_gpu(func): - return func # No-op in standalone mode - """) - with open(os.path.join(stub_dir, "llm_data.py"), "w") as f: - f.write(content) - - -def _create_test_utils_stub(tests_dir: str) -> None: - """Create minimal TensorRT-LLM unittest utility shims used by copied tests.""" - utils_dir = os.path.join(tests_dir, "utils") - os.makedirs(utils_dir, exist_ok=True) - - with open(os.path.join(utils_dir, "__init__.py"), "w") as f: - f.write( - "# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION" - " & AFFILIATES. All rights reserved.\n" - "# SPDX-License-Identifier: Apache-2.0\n" - ) - - content = textwrap.dedent("""\ - # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # SPDX-License-Identifier: Apache-2.0 - # - # Licensed under the Apache License, Version 2.0 (the "License"); - # you may not use this file except in compliance with the License. - # You may obtain a copy of the License at - # - # http://www.apache.org/licenses/LICENSE-2.0 - # - # Unless required by applicable law or agreed to in writing, software - # distributed under the License is distributed on an "AS IS" BASIS, - # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - # See the License for the specific language governing permissions and - # limitations under the License. - - \"\"\"Minimal unittest utility shims for standalone Paragraf tests.\"\"\" - - import pytest - import torch - - - def _sm_version() -> int: - if not torch.cuda.is_available(): - return 0 - major, minor = torch.cuda.get_device_capability(0) - return major * 10 + minor - - - skip_pre_hopper = pytest.mark.skipif( - _sm_version() < 90, - reason="This test is not supported in pre-Hopper architecture", - ) - skip_no_hopper = pytest.mark.skipif( - _sm_version() != 90, - reason="This test is only supported in Hopper architecture", - ) - skip_pre_blackwell = pytest.mark.skipif( - _sm_version() < 100, - reason="This test is not supported in pre-Blackwell architecture", - ) - """) - with open(os.path.join(utils_dir, "util.py"), "w") as f: - f.write(content) - - cpp_paths_content = textwrap.dedent("""\ - # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # SPDX-License-Identifier: Apache-2.0 - - from pathlib import Path - - import pytest - - - @pytest.fixture(scope="module") - def llm_root(): - return Path(__file__).resolve().parents[2] - """) - with open(os.path.join(utils_dir, "cpp_paths.py"), "w") as f: - f.write(cpp_paths_content) - - llm_data_content = textwrap.dedent("""\ - # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. - # SPDX-License-Identifier: Apache-2.0 - - from test_common.llm_data import llm_models_root - - __all__ = ["llm_models_root"] - """) - with open(os.path.join(utils_dir, "llm_data.py"), "w") as f: - f.write(llm_data_content) - - -# --------------------------------------------------------------------------- -# Example / e2e harness copying -# --------------------------------------------------------------------------- -def _copy_runners(output_dir: str) -> int: - """Copy the Tier-1 e2e harness into the standalone package under ``runners/trtllm/``. - - ``examples/auto_deploy/build_and_run_ad.py`` is copied to - ``runners/trtllm/build_and_run_paragraf_trtllm.py`` (see ``EXAMPLE_FILES``) together - with its sibling ``model_registry/``. Because the script resolves the registry - relative to its own location (``Path(__file__).parent / "model_registry"``), - the rename + relocation are safe and ``--use-registry`` keeps working. - ``.py`` files get the usual ``auto_deploy -> paragraf`` import rewrite (applied by - the caller); the ``model_registry`` YAML is data, copied verbatim. - """ - runners_dst = os.path.join(output_dir, "runners", "trtllm") - count = 0 - for src_name, dst_name in EXAMPLE_FILES.items(): - src = os.path.join(AD_EXAMPLES_SRC, src_name) - if os.path.isfile(src): - os.makedirs(runners_dst, exist_ok=True) - shutil.copy2(src, os.path.join(runners_dst, dst_name)) - count += 1 - for dname in EXAMPLE_DIRS: - src = os.path.join(AD_EXAMPLES_SRC, dname) - if os.path.isdir(src): - count += _copy_tree(src, os.path.join(runners_dst, dname)) - if os.path.isdir(runners_dst): - with open(os.path.join(runners_dst, LEGACY_RUNNER_NAME), "w") as f: - f.write(LEGACY_RUNNER_WRAPPER) - count += 1 - return count - - -# --------------------------------------------------------------------------- -# Package generation -# --------------------------------------------------------------------------- -def _create_pyproject_toml(output_dir: str, dependencies: list, dev_dependencies: list) -> None: - """Create a pyproject.toml for the standalone package.""" - deps_lines = "\n".join(f' "{dep}",' for dep in dependencies) - dev_deps_lines = "\n".join(f' "{dep}",' for dep in dev_dependencies) - - content = ( - "[build-system]\n" - 'requires = ["setuptools>=64", "wheel"]\n' - 'build-backend = "setuptools.build_meta"\n' - "\n" - "[project]\n" - 'name = "nvidia-llmc"\n' - 'version = "0.1.0"\n' - 'description = "paragraf: standalone LLM compiler — ' - 'automatic model optimization and deployment for LLM inference"\n' - 'readme = "README.md"\n' - 'license = {text = "Apache-2.0"}\n' - 'requires-python = ">=3.10"\n' - "dependencies = [\n" - f"{deps_lines}\n" - "]\n" - "\n" - "[project.optional-dependencies]\n" - 'trtllm = ["tensorrt-llm"]\n' - "dev = [\n" - f"{dev_deps_lines}\n" - "]\n" - "\n" - "[tool.setuptools.packages.find]\n" - 'include = ["paragraf*", "llmc"]\n' - "\n" - "[tool.pytest.ini_options]\n" - 'testpaths = ["tests"]\n' - "markers = [\n" - ' "threadleak(enabled): configure thread-leak checks (inert in standalone tests)",\n' - ' "skip_less_device(count): require at least count GPUs",\n' - ' "skip_less_device_memory(mib): require at least mib MiB on one GPU",\n' - "]\n" - ) - - with open(os.path.join(output_dir, "pyproject.toml"), "w") as f: - f.write(content) - - -def create_standalone_package(output_dir: str) -> None: - """Create the standalone paragraf package at the given output directory. - - Safe to run against an existing git repository: only the managed paths - (source, tests, and packaging files) are deleted and regenerated. The .git - directory and any repo-specific files (e.g., .github/) are preserved. - After running, ``git add -A && git commit`` captures all changes. - """ - if not os.path.isdir(AUTO_DEPLOY_SRC): - print(f"ERROR: auto_deploy source not found at {AUTO_DEPLOY_SRC}", file=sys.stderr) - sys.exit(1) - - os.makedirs(output_dir, exist_ok=True) - - # Clean only the paths this script manages, preserving .git and other repo files - for name in _MANAGED_PATHS: - target = os.path.join(output_dir, name) - if os.path.islink(target): - os.remove(target) - elif os.path.isdir(target): - shutil.rmtree(target) - elif os.path.isfile(target): - os.remove(target) - - print(f"Creating standalone package at: {output_dir}") - - # 1. Copy auto_deploy source as top-level `paragraf/` package. No import - # rewriting is needed: in-package imports are relative (enforced by - # the auto-deploy-import-discipline pre-commit hook). - ad_dst = os.path.join(output_dir, "paragraf") - count = _copy_tree(AUTO_DEPLOY_SRC, ad_dst) - print(f" Copied {count} source files to paragraf/") - - legacy_dst = os.path.join(output_dir, "llmc") - os.symlink("paragraf", legacy_dst, target_is_directory=True) - print(" Created legacy llmc -> paragraf package alias") - - # 2. Copy and rewrite tests (tests use absolute self-imports by design). - test_count = _copy_tests(output_dir) - rewrite_count = _rewrite_imports_in_dir(os.path.join(output_dir, "tests")) - print(f" Copied {test_count} test/support files ({rewrite_count} import rewrites)") - - # 2b. Copy the Tier-1 e2e harness into runners/ (build_and_run_paragraf_trtllm.py - # + model_registry) and rewrite its imports auto_deploy -> paragraf. YAML is - # left untouched. - runner_count = _copy_runners(output_dir) - runner_rewrites = _rewrite_imports_in_dir( - os.path.join(output_dir, "runners"), - optional_trtllm_guards=False, - ) - print( - f" Copied {runner_count} runner files to runners/trtllm/ ({runner_rewrites} import rewrites)" - ) - - # 3. Resolve dependencies and create pyproject.toml - pinned = _read_pinned_versions(TRTLLM_REQUIREMENTS) - dev_pinned = _read_pinned_versions(TRTLLM_DEV_REQUIREMENTS) - # Merge: dev_pinned has the same packages as pinned plus test-only packages - all_pinned = {**pinned, **dev_pinned} - dependencies = _resolve_dependencies(STANDALONE_DEPS, pinned) - dev_dependencies = _resolve_dependencies(DEV_DEPS, all_pinned) - _create_pyproject_toml(output_dir, dependencies, dev_dependencies) - print(f" Created pyproject.toml ({len(dependencies)} deps + {len(dev_dependencies)} dev deps)") - - # 4. Generate standalone LICENSE (only vendored projects in auto_deploy) - generate_license(output_dir) - print(f" Generated LICENSE ({len(VENDORED_PROJECTS)} vendored projects)") - - # 5. Generate ATTRIBUTIONS-Python.md (direct dependency licenses) - generate_attributions(output_dir, dependencies) - print(f" Generated ATTRIBUTIONS-Python.md ({len(dependencies)} direct deps)") - - # 6. Copy README - if os.path.exists(PARAGRAF_README): - shutil.copy2(PARAGRAF_README, os.path.join(output_dir, "README.md")) - print(" Copied README.md") - - # 7. Copy CONTRIBUTING.md - if os.path.exists(PARAGRAF_CONTRIBUTING): - shutil.copy2(PARAGRAF_CONTRIBUTING, os.path.join(output_dir, "CONTRIBUTING.md")) - print(" Copied CONTRIBUTING.md") - - # 8. Copy .gitignore - if os.path.exists(TRTLLM_GITIGNORE): - shutil.copy2(TRTLLM_GITIGNORE, os.path.join(output_dir, ".gitignore")) - print(" Copied .gitignore") - - # 9. Copy .editorconfig - if os.path.exists(TRTLLM_EDITORCONFIG): - shutil.copy2(TRTLLM_EDITORCONFIG, os.path.join(output_dir, ".editorconfig")) - print(" Copied .editorconfig") - - # 10. Copy OSS compliance files (CODE_OF_CONDUCT, SECURITY) - for src, name in ( - (TRTLLM_CODE_OF_CONDUCT, "CODE_OF_CONDUCT.md"), - (TRTLLM_SECURITY, "SECURITY.md"), - ): - if os.path.exists(src): - shutil.copy2(src, os.path.join(output_dir, name)) - print(f" Copied {name}") - - print(f"\nStandalone package created at: {output_dir}") - print("\nTo install:") - print(f" cd {output_dir}") - print(" uv venv .venv --python 3.12") - print(" source .venv/bin/activate") - print(" uv pip install -e '.[dev]'") - print("\nTo run tests: pytest tests/") - print("To run optional TensorRT-LLM tests:") - print(" uv pip install -e '.[dev,trtllm]'") - print(" TRTLLM_REDIRECT_AD_TO_PARAGRAF=true pytest tests/") - print( - 'To verify: python -c "from paragraf._compat import TRTLLM_AVAILABLE; print(TRTLLM_AVAILABLE)"' - ) - print( - "To run e2e: python runners/trtllm/build_and_run_paragraf_trtllm.py " - "--model TinyLlama/TinyLlama-1.1B-Chat-v1.0 --use-registry (needs tensorrt-llm installed)" - ) - - -def main(): - parser = argparse.ArgumentParser( - description="Create a standalone paragraf package from TensorRT-LLM source.", - ) - parser.add_argument( - "--output-dir", - default=os.path.join(REPO_ROOT, "build", "paragraf_standalone"), - help="Output directory for the standalone package (default: build/paragraf_standalone)", - ) - args = parser.parse_args() - create_standalone_package(os.path.abspath(args.output_dir)) - - -if __name__ == "__main__": - main() diff --git a/examples/bindings/executor/README.md b/examples/bindings/executor/README.md new file mode 100644 index 000000000000..df44568fd4f1 --- /dev/null +++ b/examples/bindings/executor/README.md @@ -0,0 +1,76 @@ +# Python Bindings Example + +This example shows how to use the python bindings interface to generate tokens +using a TensorRT engine. + +## Setup + +Build a TensorRT engine for one of the supported TensorRT LLM model following +instructions in the corresponding `examples` folder. + +## Usage + +### Basic example + +Run `example_basic.py`, passing in the directory where the TensorRT engine was generated. For example: + +``` +cd examples/bindings +python3 example_basic.py --model_path=../llama/tmp/7B/trt_engines/fp16/1-gpu/ +``` + +### Debug example + +This example shows how you can define which engine IO tensors should be kept or dumped to numpy files. +Run `example_debug.py`, passing in the directory where the TensorRT engine was generated. For example: + +``` +cd examples/bindings +python3 example_debug.py --model_path=../llama/tmp/7B/trt_engines/fp16/1-gpu/ +``` + +### Advanced example + +This example shows how you can use the python bindings to generate tokens for a larger number of requests concurrently and demonstrate how tokens can be returned in a streaming fashion. + +The full list of supported input parameters can be obtained with: +``` +pytho3 example_advanced.py -h +``` + +For example, assuming a CSV file named `input_tokens.csv` exist which contains the following input tokens: +``` +1, 2, 3, 4, 5, 6 +1, 2, 3, 4 +1, 2, 3, 4, 5, 6, 7, 8, 9, 10 +``` +one can generate output tokens for those 3 prompts with: +``` +python3 example_advanced.py --model_path --input_tokens_csv_file input_tokens.csv +``` +Upon successful completion, the output tokens will be written to file `output_tokens.csv`. + +### Multi-GPU Example + +To run the two examples for models requiring more than one gpu, you can run the example with MPI. + +For example, the basic example can be run as follows: +``` +mpirun -n 4 --allow-run-as-root python3 example_basic.py --model_path=../llama/tmp/7B/trt_engines/fp16/4gpu_tp4_pp1/ +``` + +The advanced example can also be run using the ORCHESTRATOR mode, where the additional processes needed for multi-GPU runs will automatically be spawned. +This can be done by running: +``` +python3 example_advanced.py --model_path=../llama/tmp/7B/trt_engines/fp16/4gpu_tp4_pp1/ --use_orchestrator_mode +``` + +### Logits post processor example + +This example shows how to generate JSON structured output using LogitsPostProcessor API. + +``` +python3 example_logits_processor.py -t -e --batch_size 8 +``` + +LogitsPostProcessorBatched, which fuses logits processing for all samples in a batch into a single callback, is enabled by `--lpp_batched` diff --git a/examples/bindings/executor/example_advanced.py b/examples/bindings/executor/example_advanced.py new file mode 100644 index 000000000000..25f063a855e2 --- /dev/null +++ b/examples/bindings/executor/example_advanced.py @@ -0,0 +1,166 @@ +import argparse +import csv +import datetime +from pathlib import Path + +import tensorrt_llm + +trtllm_package_dir = Path(tensorrt_llm.__file__).parent +executor_worker_path = trtllm_package_dir / 'bin' / 'executorWorker' + +import tensorrt_llm.bindings.executor as trtllm + + +# Read input tokens from csv file +def read_input_tokens(input_tokens_csv_file: str) -> list[int]: + + input_tokens = [] + with open(input_tokens_csv_file, mode='r') as file: + csvFile = csv.reader(file) + for lines in csvFile: + input_tokens.append([int(item) for item in lines]) + return input_tokens + + +# Prepare and enqueue the requests +def enqueue_requests(args: argparse.Namespace, + executor: trtllm.Executor) -> None: + + output_config = trtllm.OutputConfig() + output_config.exclude_input_from_output = args.exclude_input_from_output + sampling_config = trtllm.SamplingConfig(args.beam_width) + input_tokens = read_input_tokens(args.input_tokens_csv_file) + + request_ids = [] + for tokens in input_tokens: + req = trtllm.Request(input_token_ids=tokens, + max_tokens=args.max_tokens, + streaming=args.streaming, + sampling_config=sampling_config, + output_config=output_config) + req_id = executor.enqueue_request(req) + request_ids.append(req_id) + + return request_ids + + +# Wait for responses and store output tokens +def wait_for_responses(args: argparse.Namespace, request_ids: list[int], + executor: trtllm.Executor) -> dict[dict[list[int]]]: + + output_tokens = { + req_id: { + beam: [] + for beam in range(args.beam_width) + } + for req_id in request_ids + } + num_finished = 0 + iter = 0 + while (num_finished < len(request_ids) and iter < args.timeout_ms): + responses = executor.await_responses( + datetime.timedelta(milliseconds=args.timeout_ms)) + for response in responses: + req_id = response.request_id + if not response.has_error(): + result = response.result + num_finished += 1 if result.is_final else 0 + for beam, outTokens in enumerate(result.output_token_ids): + output_tokens[req_id][beam].extend(outTokens) + else: + raise RuntimeError( + str(req_id) + " encountered error:" + response.error_msg) + + return output_tokens + + +# Write the output tokens to file +def write_output_tokens(output_tokens_csv_file: str, request_ids: list[int], + output_tokens: dict[dict[list[int]]], + beam_width: int) -> None: + + with open(output_tokens_csv_file, 'w') as csvfile: + + writer = csv.writer(csvfile) + for req_id in request_ids: + out_tokens = output_tokens[req_id] + for beam in range(args.beam_width): + beam_tokens = out_tokens[beam] + writer.writerow(beam_tokens) + + print("Output tokens written to:", output_tokens_csv_file) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Executor Bindings Example") + parser.add_argument("--model_path", + type=str, + required=True, + help="Directory containing model engine") + parser.add_argument("--input_tokens_csv_file", + type=str, + required=True, + help="CSV file containing the input tokens") + parser.add_argument("--output_tokens_csv_file", + type=str, + required=False, + default="output_tokens.csv", + help="CSV file where to write output tokens") + parser.add_argument("--beam_width", + type=int, + required=False, + default=1, + help="The beam width") + parser.add_argument("--streaming", + default=False, + action="store_true", + help="Operate in streaming mode") + + parser.add_argument("--use_orchestrator_mode", + default=False, + action="store_true", + help="Operate in orchestrator mode for multi-GPU runs") + + parser.add_argument( + "--exclude_input_from_output", + default=False, + action="store_true", + help= + "Exclude input token when writing output tokens. Only has effect for streaming=False since in streaming mode, input tokens are never included in output." + ) + parser.add_argument("--max_tokens", + type=int, + required=False, + default=10, + help="The max number of tokens to be generated") + parser.add_argument( + "--timeout_ms", + type=int, + required=False, + default=10000, + help="The maximum time to wait for all responses, in milliseconds") + + args = parser.parse_args() + executor_config = trtllm.ExecutorConfig(args.beam_width) + + if args.use_orchestrator_mode: + orchestrator_config = trtllm.OrchestratorConfig( + True, str(executor_worker_path)) + executor_config.parallel_config = trtllm.ParallelConfig( + trtllm.CommunicationType.MPI, trtllm.CommunicationMode.ORCHESTRATOR, + None, None, orchestrator_config) + + # Create the executor. + executor = trtllm.Executor(args.model_path, trtllm.ModelType.DECODER_ONLY, + executor_config) + + if executor.can_enqueue_requests(): + # Enqueue the requests + request_ids = enqueue_requests(args, executor) + + # Wait for the responses + output_tokens = wait_for_responses(args, request_ids, executor) + + # Write the output tokens + write_output_tokens(args.output_tokens_csv_file, request_ids, + output_tokens, args.beam_width) diff --git a/examples/bindings/executor/example_basic.py b/examples/bindings/executor/example_basic.py new file mode 100644 index 000000000000..3c71bde594e1 --- /dev/null +++ b/examples/bindings/executor/example_basic.py @@ -0,0 +1,34 @@ +import argparse + +import tensorrt_llm.bindings.executor as trtllm + +# This example hows to use the python bindings to create an executor, enqueue a +# request, and get the generated tokens. + +# First, follow the steps in README.md to generate the engines. + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Executor Bindings Example") + parser.add_argument("--model_path", + type=str, + required=True, + help="Directory containing model engine") + args = parser.parse_args() + + # Create the executor. + executor = trtllm.Executor(args.model_path, trtllm.ModelType.DECODER_ONLY, + trtllm.ExecutorConfig(1)) + + if executor.can_enqueue_requests(): + # Create the request. + request = trtllm.Request(input_token_ids=[1, 2, 3, 4], max_tokens=10) + + # Enqueue the request. + request_id = executor.enqueue_request(request) + + # Wait for the new tokens. + responses = executor.await_responses(request_id) + output_tokens = responses[0].result.output_token_ids + + # Print tokens. + print(output_tokens) diff --git a/examples/bindings/executor/example_debug.py b/examples/bindings/executor/example_debug.py new file mode 100644 index 000000000000..f7c0669b1254 --- /dev/null +++ b/examples/bindings/executor/example_debug.py @@ -0,0 +1,67 @@ +import argparse +import pathlib as pl + +import numpy as np + +import tensorrt_llm.bindings.executor as trtllm + +# This example hows to use the python bindings to create an executor, enqueue a +# request, and get the generated tokens. + +# First, follow the steps in README.md to generate the engines. + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Executor Bindings Example") + parser.add_argument("--model_path", + type=str, + required=True, + help="Directory containing model engine") + parser.add_argument("--dump_tensors", + action="store_true", + help="Dump debug tensors to files") + args = parser.parse_args() + + max_tokens = 2 + + # Select which tensors should be kept or dumped + debug_config = trtllm.DebugConfig( + debug_tensor_names=["sequence_length"], + debug_tensors_max_iterations=0 if args.dump_tensors else max_tokens) + + # Create the executor. + executor = trtllm.Executor( + args.model_path, trtllm.ModelType.DECODER_ONLY, + trtllm.ExecutorConfig(1, debug_config=debug_config)) + + if executor.can_enqueue_requests(): + # Create the request. + request = trtllm.Request(input_token_ids=[1, 2, 3, 4], + max_tokens=max_tokens) + + # Enqueue the request. + request_id = executor.enqueue_request(request) + + # Wait for the new tokens. + responses = executor.await_responses(request_id) + output_tokens = responses[0].result.output_token_ids + + # Print tokens. + print(output_tokens) + + if args.dump_tensors: + print("debug tensors from files:") + debug_dir = pl.Path("/tmp/tllm_debug/PP_1/TP_1") + if debug_dir.is_dir(): + for iter_dir in [x for x in debug_dir.iterdir() if x.is_dir()]: + print(iter_dir.name) + for file in [x for x in iter_dir.iterdir() if x.is_file()]: + print(file.name, np.load(file)) + else: + print("debug dir not found") + else: + print("debug tensors from queue:") + debug_tensors = executor.get_latest_debug_tensors() + for debug_iter in debug_tensors: + print(f"iteration {debug_iter.iter}") + for [name, tensor] in debug_iter.debug_tensors.items(): + print(name, tensor) diff --git a/examples/bindings/executor/example_logits_processor.py b/examples/bindings/executor/example_logits_processor.py new file mode 100644 index 000000000000..6cb1a751a6da --- /dev/null +++ b/examples/bindings/executor/example_logits_processor.py @@ -0,0 +1,212 @@ +import argparse +import datetime +import typing as _tp + +import torch as _tor +from lmformatenforcer import (JsonSchemaParser, TokenEnforcer, + TokenEnforcerTokenizerData) +from pydantic import BaseModel +from transformers import AutoTokenizer + +import tensorrt_llm.bindings.executor as trtllm + + +def _build_regular_tokens_list( + tokenizer) -> _tp.List[_tp.Tuple[int, str, bool]]: + token_0 = [tokenizer.encode("0")[-1]] + regular_tokens = [] + vocab_size = tokenizer.vocab_size + for token_idx in range(vocab_size): + if token_idx in tokenizer.all_special_ids: + continue + # We prepend token 0 and skip the first letter of the result to get a space if the token is a start word. + tensor_after_0 = _tor.tensor(token_0 + [token_idx], dtype=_tor.long) + decoded_after_0 = tokenizer.decode(tensor_after_0)[1:] + decoded_regular = tokenizer.decode(token_0) + is_word_start_token = len(decoded_after_0) > len(decoded_regular) + regular_tokens.append((token_idx, decoded_after_0, is_word_start_token)) + return regular_tokens + + +def build_token_enforcer(tokenizer, character_level_parser): + """ + Build logits processor for feeding it into generate function (use_py_session should be True) + """ + regular_tokens = _build_regular_tokens_list(tokenizer) + + def _decode(tokens: _tp.List[int]) -> str: + tensor = _tor.tensor(tokens, dtype=_tor.long) + return tokenizer.decode(tensor) + + tokenizer_data = TokenEnforcerTokenizerData(regular_tokens, _decode, + tokenizer.eos_token_id) + return TokenEnforcer(tokenizer_data, character_level_parser) + + +# Prepare and enqueue the requests +def enqueue_requests(args: argparse.Namespace, + executor: trtllm.Executor) -> None: + + sampling_config = trtllm.SamplingConfig(args.beam_width) + + request_ids = [] + for iter_id in range(args.batch_size): + # Create the request. + request = trtllm.Request(input_token_ids=prompt, + max_tokens=25, + end_id=tokenizer.eos_token_id, + sampling_config=sampling_config, + client_id=iter_id % 2) + request.logits_post_processor_name = request.BATCHED_POST_PROCESSOR_NAME if args.lpp_batched else "my_logits_pp" + + # Enqueue the request. + req_id = executor.enqueue_request(request) + request_ids.append(req_id) + + return request_ids + + +# Wait for responses and store output tokens +def wait_for_responses(args: argparse.Namespace, request_ids: list[int], + executor: trtllm.Executor) -> dict[dict[list[int]]]: + + output_tokens = { + req_id: { + beam: [] + for beam in range(args.beam_width) + } + for req_id in request_ids + } + num_finished = 0 + iter = 0 + while (num_finished < len(request_ids) and iter < args.timeout_ms): + responses = executor.await_responses( + datetime.timedelta(milliseconds=args.timeout_ms)) + for response in responses: + req_id = response.request_id + if not response.has_error(): + result = response.result + num_finished += 1 if result.is_final else 0 + for beam, outTokens in enumerate(result.output_token_ids): + output_tokens[req_id][beam].extend(outTokens) + else: + raise RuntimeError( + str(req_id) + " encountered error:" + response.error_msg) + + return output_tokens + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description="Executor Bindings Example") + parser.add_argument("--tokenizer_path", + "-t", + type=str, + required=True, + help="Directory containing model tokenizer") + parser.add_argument("--engine_path", + "-e", + type=str, + required=True, + help="Directory containing model engine") + parser.add_argument("--beam_width", + type=int, + required=False, + default=1, + help="The beam width") + parser.add_argument("--batch_size", + type=int, + required=False, + default=1, + help="The batch size") + parser.add_argument( + "--timeout_ms", + type=int, + required=False, + default=10000, + help="The maximum time to wait for all responses, in milliseconds") + parser.add_argument("--lpp_batched", + action="store_true", + default=False, + help="Enable batched logits post processor") + + args = parser.parse_args() + + tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path) + + class AnswerFormat(BaseModel): + last_name: str + year_of_birth: int + + parser = JsonSchemaParser(AnswerFormat.model_json_schema()) + token_enforcer = build_token_enforcer(tokenizer, parser) + + def get_allowed_tokens(ids, client_id): + if client_id is None or client_id == 0: return [42] + + def _trim(ids): + return [x for x in ids if x != tokenizer.eos_token_id] + + allowed = token_enforcer.get_allowed_tokens(_trim(ids[0])) + return allowed + + def logits_post_processor(req_id: int, logits: _tor.Tensor, + ids: _tp.List[_tp.List[int]], stream_ptr: int, + client_id: _tp.Optional[int]): + mask = _tor.full_like(logits, fill_value=float("-inf"), device="cpu") + allowed = get_allowed_tokens(ids, client_id) + mask[:, :, allowed] = 0 + + with _tor.cuda.stream(_tor.cuda.ExternalStream(stream_ptr)): + mask = mask.to(logits.device, non_blocking=True) + logits += mask + + def logits_post_processor_batched( + req_ids_batch: _tp.List[int], logits_batch: _tp.List[_tor.Tensor], + ids_batch: _tp.List[_tp.List[_tp.List[int]]], stream_ptr, + client_ids_batch: _tp.List[_tp.Optional[int]]): + masks = [] + for req_id, logits, ids, client_id in zip(req_ids_batch, logits_batch, + ids_batch, client_ids_batch): + del req_id + mask = _tor.full_like(logits, + fill_value=float("-inf"), + device="cpu") + allowed = get_allowed_tokens(ids, client_id) + mask[:, :, allowed] = 0 + masks.append(mask) + + with _tor.cuda.stream(_tor.cuda.ExternalStream(stream_ptr)): + for logits, mask in zip(logits_batch, masks): + logits += mask.to(logits.device, non_blocking=True) + + # Create the executor. + executor_config = trtllm.ExecutorConfig(args.beam_width) + logits_proc_config = trtllm.LogitsPostProcessorConfig() + if not args.lpp_batched: + logits_proc_config.processor_map = { + "my_logits_pp": logits_post_processor + } + else: + logits_proc_config.processor_batched = logits_post_processor_batched + executor_config.logits_post_processor_config = logits_proc_config + executor = trtllm.Executor(args.engine_path, trtllm.ModelType.DECODER_ONLY, + executor_config) + + input = "Please give me information about Michael Jordan. You MUST answer using the following json schema: " + prompt = tokenizer.encode(input) + print(f"Input text: {input}\n") + + if executor.can_enqueue_requests(): + request_ids = enqueue_requests(args, executor) + output_tokens = wait_for_responses(args, request_ids, executor) + + # Print output + for req_id in request_ids: + for beam_id in range(args.beam_width): + result = tokenizer.decode( + output_tokens[req_id][beam_id][len(prompt):]) + generated_tokens = len( + output_tokens[req_id][beam_id]) - len(prompt) + print( + f"Request {req_id} Beam {beam_id} ({generated_tokens} tokens): {result}" + ) diff --git a/examples/configs/curated/deepseek-v4-pro-latency.yaml b/examples/configs/curated/deepseek-v4-pro-latency.yaml deleted file mode 100644 index fa477ffa41aa..000000000000 --- a/examples/configs/curated/deepseek-v4-pro-latency.yaml +++ /dev/null @@ -1,45 +0,0 @@ -cuda_graph_config: - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 24 - - 32 - - 40 - - 48 - - 56 - - 64 - - 72 - - 80 - - 88 - - 96 - - 104 - - 112 - - 120 - - 128 - enable_padding: true -disable_overlap_scheduler: false -enable_attention_dp: false -enable_lm_head_tp_in_adp: false -kv_cache_config: - dtype: fp8 - enable_block_reuse: false - free_gpu_memory_fraction: 0.9 - tokens_per_block: 128 -max_batch_size: 128 -max_num_tokens: 8448 -max_seq_len: 9256 -moe_config: - backend: TRTLLM - use_low_precision_moe_combine: true -moe_expert_parallel_size: 8 -num_postprocess_workers: 4 -pipeline_parallel_size: 1 -print_iter_log: true -speculative_config: - decoding_type: MTP - max_draft_len: 3 -stream_interval: 100 -tensor_parallel_size: 8 diff --git a/examples/configs/curated/deepseek-v4-pro-throughput.yaml b/examples/configs/curated/deepseek-v4-pro-throughput.yaml deleted file mode 100644 index c4fb8453466d..000000000000 --- a/examples/configs/curated/deepseek-v4-pro-throughput.yaml +++ /dev/null @@ -1,35 +0,0 @@ -attention_dp_config: - enable_balance: true -cuda_graph_config: - batch_sizes: - - 1 - - 2 - - 4 - - 8 - - 16 - - 24 - - 32 - enable_padding: true -disable_overlap_scheduler: false -enable_attention_dp: true -enable_lm_head_tp_in_adp: true -kv_cache_config: - dtype: fp8 - enable_block_reuse: false - free_gpu_memory_fraction: 0.6 - tokens_per_block: 128 -max_batch_size: 32 -max_num_tokens: 8448 -max_seq_len: 9256 -moe_config: - backend: TRTLLM - use_low_precision_moe_combine: true -moe_expert_parallel_size: 8 -num_postprocess_workers: 4 -pipeline_parallel_size: 1 -print_iter_log: true -speculative_config: - decoding_type: MTP - max_draft_len: 1 -stream_interval: 100 -tensor_parallel_size: 8 diff --git a/examples/configs/curated/lookup.yaml b/examples/configs/curated/lookup.yaml index e813d6f8f0fe..e03f118f916c 100644 --- a/examples/configs/curated/lookup.yaml +++ b/examples/configs/curated/lookup.yaml @@ -86,13 +86,3 @@ config_path: examples/configs/curated/minimax-m3-throughput.yaml scenario: Max Throughput gpu_compatibility: "GB200" -- model: deepseek-ai/DeepSeek-V4-Pro - arch: DeepseekV4ForCausalLM - config_path: examples/configs/curated/deepseek-v4-pro-latency.yaml - scenario: Min Latency - gpu_compatibility: "B200" -- model: deepseek-ai/DeepSeek-V4-Pro - arch: DeepseekV4ForCausalLM - config_path: examples/configs/curated/deepseek-v4-pro-throughput.yaml - scenario: Max Throughput - gpu_compatibility: "B200" diff --git a/examples/configs/database/database.py b/examples/configs/database/database.py index af33a4dab41b..3dcdd13c0300 100644 --- a/examples/configs/database/database.py +++ b/examples/configs/database/database.py @@ -13,22 +13,13 @@ # See the License for the specific language governing permissions and # limitations under the License. + import logging -import re -from collections import defaultdict from pathlib import Path -from typing import Any, Iterator, Literal +from typing import Any, Dict, Iterator, List, Tuple import yaml -from pydantic import ( - BaseModel, - ConfigDict, - Field, - PositiveInt, - RootModel, - field_validator, - model_validator, -) +from pydantic import BaseModel, Field, RootModel, field_validator logger = logging.getLogger(__name__) @@ -39,22 +30,11 @@ LOW_LATENCY_CONCURRENCY_THRESHOLD = 8 HIGH_THROUGHPUT_CONCURRENCY_THRESHOLD = 32 KEY_PROFILES = {"Min Latency", "Balanced", "Max Throughput"} -PROFILE_DISPLAY_NAMES = { - "latency": "Min Latency", - "balanced": "Balanced", - "throughput": "Max Throughput", -} -PROFILE_ORDER = {profile: idx for idx, profile in enumerate(PROFILE_DISPLAY_NAMES)} -VALIDATED_COMMIT_PATTERN = re.compile(r"^[0-9a-f]{40}$") -VALIDATED_VERSION_PATTERN = re.compile(r"^[0-9A-Za-z][0-9A-Za-z._+-]*$") -Profile = Literal["latency", "balanced", "throughput"] class CuratedRecipe(BaseModel): """A curated (hand-tuned) recipe entry.""" - model_config = ConfigDict(extra="forbid") - model: str = Field(description="HuggingFace model ID") arch: str = Field(description="Model architecture class name") config_path: str = Field(description="Relative path to YAML config") @@ -71,7 +51,7 @@ def _validate_config_path(cls, v: str) -> str: return v -class CuratedRecipeList(RootModel[list[CuratedRecipe]]): +class CuratedRecipeList(RootModel[List[CuratedRecipe]]): """Validated list of curated recipe entries.""" @classmethod @@ -95,58 +75,15 @@ def __len__(self) -> int: class Recipe(BaseModel): """Recipe record for scenario list.""" - model_config = ConfigDict(extra="forbid") - - model: str = Field(min_length=1, description="Model name") - arch: str = Field(min_length=1, description="Model architecture class name") - gpu: str = Field(min_length=1, description="GPU name") - isl: PositiveInt = Field(description="Input sequence length") - osl: PositiveInt = Field(description="Output sequence length") - concurrency: PositiveInt = Field(description="Concurrency") - config_path: str = Field(min_length=1, description="Configuration path") - num_gpus: PositiveInt = Field(description="Number of GPUs") - profile: Profile | None = Field( - default=None, - description="Profile discriminator used only when the workload key has multiple configs", - ) - validated_trtllm_commit: str | None = Field( - default=None, - description="Full TensorRT-LLM commit SHA against which the recipe was validated", - ) - validated_trtllm_version: str | None = Field( - default=None, - description="TensorRT-LLM release version reported by the validation source", - ) - - @field_validator("validated_trtllm_commit") - @classmethod - def _validate_commit(cls, value: str | None) -> str | None: - if value is None: - return None - normalized = value.strip().lower() - if not VALIDATED_COMMIT_PATTERN.fullmatch(normalized): - raise ValueError("validated_trtllm_commit must be a full 40-character Git SHA") - return normalized - - @field_validator("validated_trtllm_version") - @classmethod - def _validate_version(cls, value: str | None) -> str | None: - if value is None: - return None - normalized = value.strip() - if not VALIDATED_VERSION_PATTERN.fullmatch(normalized): - raise ValueError("validated_trtllm_version must be a release-tag-safe version") - return normalized - - @model_validator(mode="after") - def _validate_provenance_pair(self) -> "Recipe": - if bool(self.validated_trtllm_commit) != bool(self.validated_trtllm_version): - raise ValueError( - "validated_trtllm_commit and validated_trtllm_version must be provided together" - ) - return self - - def load_config(self) -> dict[str, Any]: + model: str = Field(description="Model name") + gpu: str = Field(description="GPU name") + isl: int = Field(description="Input sequence length") + osl: int = Field(description="Output sequence length") + concurrency: int = Field(description="Concurrency") + config_path: str = Field(description="Configuration path") + num_gpus: int = Field(description="Number of GPUs") + + def load_config(self) -> Dict[str, Any]: """Load and return the YAML config at config_path.""" config_relative_path = Path(self.config_path) # Ensure config path is within the repo root @@ -159,41 +96,7 @@ def load_config(self) -> dict[str, Any]: return yaml.safe_load(f) -class RecipeList(RootModel[list[Recipe]]): - @model_validator(mode="after") - def _validate_conflict_profiles(self) -> "RecipeList": - groups = defaultdict(list) - for recipe in self.root: - key = ( - recipe.model, - recipe.gpu, - recipe.num_gpus, - recipe.isl, - recipe.osl, - recipe.concurrency, - ) - groups[key].append(recipe) - - required_profiles = {"latency", "throughput"} - for key, recipes in groups.items(): - profiles = [recipe.profile for recipe in recipes] - if len(recipes) == 1: - if profiles[0] is not None: - raise ValueError(f"profile is only allowed for conflicting workload key {key}") - continue - profile_set = set(profiles) - if ( - None in profile_set - or len(profile_set) != len(profiles) - or not required_profiles.issubset(profile_set) - ): - raise ValueError( - "conflicting workload key " - f"{key} must have exactly one latency and throughput profile, " - "with an optional balanced profile" - ) - return self - +class RecipeList(RootModel[List[Recipe]]): @classmethod def from_yaml(cls, yaml_path: Path) -> "RecipeList": """Load and validate recipe list from YAML file.""" @@ -229,15 +132,12 @@ def assign_profile(num_recipes: int, idx: int, concurrency: int) -> str: return "High Throughput" -def select_key_recipes(recipes: list[Recipe]) -> list[tuple[Recipe, str]]: +def select_key_recipes(recipes: List[Recipe]) -> List[Tuple[Recipe, str]]: """Select key recipes (min latency, balanced, max throughput) from a list of recipes.""" if not recipes: return [] - sorted_recipes = sorted( - recipes, - key=lambda r: (r.concurrency, PROFILE_ORDER.get(r.profile, -1)), - ) + sorted_recipes = sorted(recipes, key=lambda r: r.concurrency) n = len(sorted_recipes) result = [] diff --git a/examples/configs/database/lookup.yaml b/examples/configs/database/lookup.yaml index d773fa361596..523cda14c412 100644 --- a/examples/configs/database/lookup.yaml +++ b/examples/configs/database/lookup.yaml @@ -558,8 +558,6 @@ concurrency: 4 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc4.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -584,8 +582,6 @@ concurrency: 32 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc32.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -594,8 +590,6 @@ concurrency: 64 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc64.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -604,8 +598,6 @@ concurrency: 128 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc128.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -614,8 +606,6 @@ concurrency: 256 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc256.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -760,8 +750,6 @@ concurrency: 4 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc4.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -794,8 +782,6 @@ concurrency: 64 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc64.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -804,8 +790,6 @@ concurrency: 128 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc128.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -814,8 +798,6 @@ concurrency: 256 config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc256.yaml num_gpus: 8 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 - model: nvidia/DeepSeek-R1-0528-FP4-v2 arch: DeepseekV3ForCausalLM gpu: B200_NVL @@ -1584,83 +1566,3 @@ concurrency: 1536 config_path: examples/configs/database/openai/gpt-oss-120b/H200/8k1k_tp8_conc1536.yaml num_gpus: 8 -- model: nvidia/DeepSeek-R1-0528-FP4-v2 - arch: DeepseekV3ForCausalLM - gpu: B200_NVL - isl: 1024 - osl: 1024 - concurrency: 4 - config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc4.yaml - num_gpus: 4 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 -- model: nvidia/DeepSeek-R1-0528-FP4-v2 - arch: DeepseekV3ForCausalLM - gpu: B200_NVL - isl: 1024 - osl: 1024 - concurrency: 8 - config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc8.yaml - num_gpus: 4 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 -- model: nvidia/DeepSeek-R1-0528-FP4-v2 - arch: DeepseekV3ForCausalLM - gpu: B200_NVL - isl: 1024 - osl: 1024 - concurrency: 16 - config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc16.yaml - num_gpus: 4 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 -- model: nvidia/DeepSeek-R1-0528-FP4-v2 - arch: DeepseekV3ForCausalLM - gpu: B200_NVL - isl: 1024 - osl: 1024 - concurrency: 256 - config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc256.yaml - num_gpus: 4 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 -- model: nvidia/DeepSeek-R1-0528-FP4-v2 - arch: DeepseekV3ForCausalLM - gpu: B200_NVL - isl: 8192 - osl: 1024 - concurrency: 4 - config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc4.yaml - num_gpus: 4 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 -- model: nvidia/DeepSeek-R1-0528-FP4-v2 - arch: DeepseekV3ForCausalLM - gpu: B200_NVL - isl: 8192 - osl: 1024 - concurrency: 8 - config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc8.yaml - num_gpus: 4 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 -- model: nvidia/DeepSeek-R1-0528-FP4-v2 - arch: DeepseekV3ForCausalLM - gpu: B200_NVL - isl: 8192 - osl: 1024 - concurrency: 16 - config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc16.yaml - num_gpus: 4 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 -- model: nvidia/DeepSeek-R1-0528-FP4-v2 - arch: DeepseekV3ForCausalLM - gpu: B200_NVL - isl: 8192 - osl: 1024 - concurrency: 256 - config_path: examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc256.yaml - num_gpus: 4 - validated_trtllm_commit: 93cb6518b6d6dbd6095748189e626db731f44545 - validated_trtllm_version: 1.3.0rc14 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc16.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc16.yaml deleted file mode 100644 index c9986a111b5b..000000000000 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc16.yaml +++ /dev/null @@ -1,14 +0,0 @@ -cuda_graph_config: - enable_padding: true - max_batch_size: 16 -print_iter_log: true -kv_cache_config: - dtype: fp8 - free_gpu_memory_fraction: 0.8 -stream_interval: 10 -moe_config: - backend: TRTLLM -tensor_parallel_size: 4 -moe_expert_parallel_size: 1 -trust_remote_code: true -max_seq_len: 8192 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc256.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc256.yaml deleted file mode 100644 index bb43a3699209..000000000000 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc256.yaml +++ /dev/null @@ -1,24 +0,0 @@ -max_batch_size: 64 -cuda_graph_config: - enable_padding: true - max_batch_size: 64 -enable_attention_dp: true -print_iter_log: true -kv_cache_config: - dtype: fp8 - free_gpu_memory_fraction: 0.8 -stream_interval: 10 -moe_config: - backend: CUTLASS -attention_dp_config: - batching_wait_iters: 0 - enable_balance: true - timeout_iters: 60 -speculative_config: - decoding_type: MTP - max_draft_len: 1 -tensor_parallel_size: 4 -moe_expert_parallel_size: 4 -trust_remote_code: true -max_num_tokens: 1216 -max_seq_len: 2304 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc4.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc4.yaml deleted file mode 100644 index ade4e5b1de21..000000000000 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc4.yaml +++ /dev/null @@ -1,19 +0,0 @@ -max_batch_size: 4 -cuda_graph_config: - enable_padding: true - max_batch_size: 4 -print_iter_log: true -kv_cache_config: - dtype: fp8 - free_gpu_memory_fraction: 0.8 -stream_interval: 10 -moe_config: - backend: TRTLLM -speculative_config: - decoding_type: MTP - max_draft_len: 3 -tensor_parallel_size: 4 -moe_expert_parallel_size: 1 -trust_remote_code: true -max_num_tokens: 1152 -max_seq_len: 2304 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc8.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc8.yaml deleted file mode 100644 index 7f49059c11b1..000000000000 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp4_conc8.yaml +++ /dev/null @@ -1,19 +0,0 @@ -max_batch_size: 8 -cuda_graph_config: - enable_padding: true - max_batch_size: 8 -print_iter_log: true -kv_cache_config: - dtype: fp8 - free_gpu_memory_fraction: 0.8 -stream_interval: 10 -moe_config: - backend: TRTLLM -speculative_config: - decoding_type: MTP - max_draft_len: 3 -tensor_parallel_size: 4 -moe_expert_parallel_size: 1 -trust_remote_code: true -max_num_tokens: 1152 -max_seq_len: 2304 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc128.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc128.yaml index 3db82c4881df..ea4b03e17984 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc128.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc128.yaml @@ -1,19 +1,24 @@ -max_batch_size: 128 +max_batch_size: 512 cuda_graph_config: enable_padding: true - max_batch_size: 128 + max_batch_size: 32 +enable_attention_dp: true print_iter_log: true kv_cache_config: dtype: fp8 free_gpu_memory_fraction: 0.8 stream_interval: 10 moe_config: - backend: TRTLLM + backend: CUTLASS +attention_dp_config: + batching_wait_iters: 0 + enable_balance: true + timeout_iters: 60 speculative_config: decoding_type: MTP - max_draft_len: 3 + max_draft_len: 1 tensor_parallel_size: 8 -moe_expert_parallel_size: 1 +moe_expert_parallel_size: 8 trust_remote_code: true -max_num_tokens: 1600 -max_seq_len: 2304 +max_num_tokens: 3072 +max_seq_len: 2068 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc256.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc256.yaml index 9bc761a79868..9031655d28fd 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc256.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc256.yaml @@ -1,6 +1,7 @@ +max_batch_size: 512 cuda_graph_config: enable_padding: true - max_batch_size: 64 + max_batch_size: 256 enable_attention_dp: true print_iter_log: true kv_cache_config: @@ -13,7 +14,11 @@ attention_dp_config: batching_wait_iters: 0 enable_balance: true timeout_iters: 60 +speculative_config: + decoding_type: MTP + max_draft_len: 1 tensor_parallel_size: 8 moe_expert_parallel_size: 8 trust_remote_code: true -max_seq_len: 8192 +max_num_tokens: 2112 +max_seq_len: 2068 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc32.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc32.yaml index 03c9e6b27901..5d7f74990a3a 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc32.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc32.yaml @@ -1,4 +1,4 @@ -max_batch_size: 32 +max_batch_size: 512 cuda_graph_config: enable_padding: true max_batch_size: 32 @@ -15,5 +15,5 @@ speculative_config: tensor_parallel_size: 8 moe_expert_parallel_size: 8 trust_remote_code: true -max_num_tokens: 1216 -max_seq_len: 2304 +max_num_tokens: 3136 +max_seq_len: 2068 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc4.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc4.yaml index cf0bb6a44862..68eab71800d9 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc4.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc4.yaml @@ -1,4 +1,4 @@ -max_batch_size: 4 +max_batch_size: 512 cuda_graph_config: enable_padding: true max_batch_size: 4 @@ -13,7 +13,7 @@ speculative_config: decoding_type: MTP max_draft_len: 3 tensor_parallel_size: 8 -moe_expert_parallel_size: 1 +moe_expert_parallel_size: 8 trust_remote_code: true -max_num_tokens: 1152 -max_seq_len: 2304 +max_num_tokens: 3136 +max_seq_len: 2068 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc64.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc64.yaml index 3348357986a6..c239238e8f0d 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc64.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/1k1k_tp8_conc64.yaml @@ -1,7 +1,7 @@ -max_batch_size: 64 cuda_graph_config: enable_padding: true - max_batch_size: 64 + max_batch_size: 16 +enable_attention_dp: true print_iter_log: true kv_cache_config: dtype: fp8 @@ -9,11 +9,14 @@ kv_cache_config: stream_interval: 10 moe_config: backend: TRTLLM +attention_dp_config: + batching_wait_iters: 0 + enable_balance: true + timeout_iters: 60 speculative_config: decoding_type: MTP - max_draft_len: 3 + max_draft_len: 1 tensor_parallel_size: 8 moe_expert_parallel_size: 8 trust_remote_code: true -max_num_tokens: 1344 -max_seq_len: 2304 +max_seq_len: 2068 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc16.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc16.yaml deleted file mode 100644 index 283c8ccd18a3..000000000000 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc16.yaml +++ /dev/null @@ -1,19 +0,0 @@ -max_batch_size: 16 -cuda_graph_config: - enable_padding: true - max_batch_size: 16 -print_iter_log: true -kv_cache_config: - dtype: fp8 - free_gpu_memory_fraction: 0.8 -stream_interval: 10 -moe_config: - backend: TRTLLM -speculative_config: - decoding_type: MTP - max_draft_len: 3 -tensor_parallel_size: 4 -moe_expert_parallel_size: 1 -trust_remote_code: true -max_num_tokens: 8320 -max_seq_len: 9472 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc256.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc256.yaml deleted file mode 100644 index aa75a4c8d4c5..000000000000 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc256.yaml +++ /dev/null @@ -1,24 +0,0 @@ -max_batch_size: 64 -cuda_graph_config: - enable_padding: true - max_batch_size: 64 -enable_attention_dp: true -print_iter_log: true -kv_cache_config: - dtype: fp8 - free_gpu_memory_fraction: 0.8 -stream_interval: 10 -moe_config: - backend: CUTLASS -attention_dp_config: - batching_wait_iters: 0 - enable_balance: true - timeout_iters: 60 -speculative_config: - decoding_type: MTP - max_draft_len: 1 -tensor_parallel_size: 4 -moe_expert_parallel_size: 4 -trust_remote_code: true -max_num_tokens: 8384 -max_seq_len: 9472 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc4.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc4.yaml deleted file mode 100644 index 594aae2eaba4..000000000000 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc4.yaml +++ /dev/null @@ -1,19 +0,0 @@ -max_batch_size: 4 -cuda_graph_config: - enable_padding: true - max_batch_size: 4 -print_iter_log: true -kv_cache_config: - dtype: fp8 - free_gpu_memory_fraction: 0.8 -stream_interval: 10 -moe_config: - backend: TRTLLM -speculative_config: - decoding_type: MTP - max_draft_len: 3 -tensor_parallel_size: 4 -moe_expert_parallel_size: 1 -trust_remote_code: true -max_num_tokens: 8320 -max_seq_len: 9472 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc8.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc8.yaml deleted file mode 100644 index 2dbc0435be33..000000000000 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp4_conc8.yaml +++ /dev/null @@ -1,19 +0,0 @@ -max_batch_size: 8 -cuda_graph_config: - enable_padding: true - max_batch_size: 8 -print_iter_log: true -kv_cache_config: - dtype: fp8 - free_gpu_memory_fraction: 0.8 -stream_interval: 10 -moe_config: - backend: TRTLLM -speculative_config: - decoding_type: MTP - max_draft_len: 3 -tensor_parallel_size: 4 -moe_expert_parallel_size: 1 -trust_remote_code: true -max_num_tokens: 8320 -max_seq_len: 9472 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc128.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc128.yaml index 143fe4bba30b..baaa644ca539 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc128.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc128.yaml @@ -1,4 +1,4 @@ -max_batch_size: 32 +max_batch_size: 256 cuda_graph_config: enable_padding: true max_batch_size: 32 @@ -20,5 +20,5 @@ speculative_config: tensor_parallel_size: 8 moe_expert_parallel_size: 8 trust_remote_code: true -max_num_tokens: 8320 -max_seq_len: 9472 +max_num_tokens: 8768 +max_seq_len: 9416 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc256.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc256.yaml index 251ed48facac..93df61d6fa56 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc256.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc256.yaml @@ -1,4 +1,4 @@ -max_batch_size: 64 +max_batch_size: 256 cuda_graph_config: enable_padding: true max_batch_size: 64 @@ -20,5 +20,5 @@ speculative_config: tensor_parallel_size: 8 moe_expert_parallel_size: 8 trust_remote_code: true -max_num_tokens: 8384 -max_seq_len: 9472 +max_num_tokens: 8768 +max_seq_len: 9416 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc4.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc4.yaml index 97ce51217d82..6a6b9fb25c63 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc4.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc4.yaml @@ -1,4 +1,4 @@ -max_batch_size: 4 +max_batch_size: 512 cuda_graph_config: enable_padding: true max_batch_size: 4 @@ -13,7 +13,7 @@ speculative_config: decoding_type: MTP max_draft_len: 3 tensor_parallel_size: 8 -moe_expert_parallel_size: 1 +moe_expert_parallel_size: 8 trust_remote_code: true -max_num_tokens: 8320 -max_seq_len: 9472 +max_num_tokens: 10304 +max_seq_len: 9416 diff --git a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc64.yaml b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc64.yaml index 6729a6fa5c6d..b39f07478407 100644 --- a/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc64.yaml +++ b/examples/configs/database/nvidia/DeepSeek-R1-0528-FP4-v2/B200/8k1k_tp8_conc64.yaml @@ -1,4 +1,4 @@ -max_batch_size: 16 +max_batch_size: 256 cuda_graph_config: enable_padding: true max_batch_size: 16 @@ -20,5 +20,5 @@ speculative_config: tensor_parallel_size: 8 moe_expert_parallel_size: 8 trust_remote_code: true -max_num_tokens: 8320 -max_seq_len: 9472 +max_num_tokens: 8768 +max_seq_len: 9416 diff --git a/examples/constraints.txt b/examples/constraints.txt index 01ee827c13ea..4141980256e5 100644 --- a/examples/constraints.txt +++ b/examples/constraints.txt @@ -1,3 +1,3 @@ -tensorrt_llm==1.3.0rc23 +tensorrt_llm==1.3.0rc21 evaluate~=0.4.1 rouge_score~=0.1.2 diff --git a/examples/cpp/executor/CMakeLists.txt b/examples/cpp/executor/CMakeLists.txt new file mode 100644 index 000000000000..b448667e8d75 --- /dev/null +++ b/examples/cpp/executor/CMakeLists.txt @@ -0,0 +1,161 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. cmake needs this line + +cmake_minimum_required(VERSION 3.27) + +set(TRTLLM_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../../..") +list(APPEND CMAKE_MODULE_PATH "${TRTLLM_DIR}/cpp/cmake/modules") + +if(NOT TRTLLM_BUILD_DIR) + set(TRTLLM_BUILD_DIR "${TRTLLM_DIR}/cpp/build") +endif() +set(TRTLLM_LIB_PATH "${TRTLLM_BUILD_DIR}/tensorrt_llm/libtensorrt_llm.so") +if(NOT EXISTS ${TRTLLM_LIB_PATH}) + message(FATAL_ERROR "Cannot find ${TRTLLM_LIB_PATH}") +endif() + +set(TRTLLM_PLUGIN_PATH + "${TRTLLM_BUILD_DIR}/tensorrt_llm/plugins/libnvinfer_plugin_tensorrt_llm.so" +) +set(TRTLLM_INCLUDE_DIR "${TRTLLM_DIR}/cpp/include") + +option( + ENABLE_MULTI_DEVICE + "Enable multi device/instance examples building (requires MPI headers/libs)" + ON) + +# Determine CXX11 ABI compatibility +execute_process( + COMMAND bash -c "nm -f posix -D ${TRTLLM_LIB_PATH} | grep __cxx11" + RESULT_VARIABLE GLIB_CXX11_FOUND + OUTPUT_QUIET) +if(GLIB_CXX11_FOUND EQUAL 0) + set(USE_CXX11_ABI 1) +else() + set(USE_CXX11_ABI 0) +endif() +message(STATUS "Use CXX11 ABI: ${USE_CXX11_ABI}") +add_compile_options("-D_GLIBCXX_USE_CXX11_ABI=${USE_CXX11_ABI}") + +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED TRUE) +set(CMAKE_VERBOSE_MAKEFILE 1) + +# Define project name +project(executorExamples) + +# Compile options $ ? +if(ENABLE_MULTI_DEVICE) + set(EMD 1) +else() + set(EMD 0) +endif() +set(CMAKE_CXX_FLAGS "-Wall -pthread -lstdc++ -DENABLE_MULTI_DEVICE=${EMD} ") +set(CMAKE_CXX_FLAGS_RELEASE "-O3") +set(CMAKE_BUILD_TYPE release) + +find_package(CUDAToolkit REQUIRED COMPONENTS cuda_driver cudart_static nvml) +message(STATUS "CUDA library status:") +message(STATUS " version: ${CUDAToolkit_VERSION}") +message(STATUS " libraries: ${CUDAToolkit_LIBRARY_DIR}") +message(STATUS " include path: ${CUDAToolkit_INCLUDE_DIRS}") + +# TRT dependencies +find_package(TensorRT 10 REQUIRED) + +if(${CUDAToolkit_VERSION} VERSION_GREATER_EQUAL "11") + add_definitions("-DENABLE_BF16") + message( + STATUS + "CUDA_VERSION ${CUDA_VERSION} is greater or equal than 11.0, enable -DENABLE_BF16 flag" + ) +endif() + +if(${CUDAToolkit_VERSION} VERSION_GREATER_EQUAL "11.8") + add_definitions("-DENABLE_FP8") + message( + STATUS + "CUDA_VERSION ${CUDA_VERSION} is greater or equal than 11.8, enable -DENABLE_FP8 flag" + ) +endif() + +add_subdirectory(${TRTLLM_DIR}/3rdparty 3rdparty) +FetchContent_MakeAvailable(cxxopts) + +# tensorrt_llm shared lib +add_library(tensorrt_llm SHARED IMPORTED) +set_property(TARGET tensorrt_llm PROPERTY IMPORTED_LOCATION ${TRTLLM_LIB_PATH}) +set_property( + TARGET tensorrt_llm PROPERTY IMPORTED_LINK_INTERFACE_LIBRARIES + CUDA::cuda_driver CUDA::cudart_static CUDA::nvml) + +# nvinfer_plugin_tensorrt_llm shared lib +add_library(nvinfer_plugin_tensorrt_llm SHARED IMPORTED) +set_property(TARGET nvinfer_plugin_tensorrt_llm PROPERTY IMPORTED_LOCATION + ${TRTLLM_PLUGIN_PATH}) +set_property(TARGET nvinfer_plugin_tensorrt_llm + PROPERTY IMPORTED_LINK_INTERFACE_LIBRARIES tensorrt_llm) + +include_directories(${TRTLLM_INCLUDE_DIR} ${CUDAToolkit_INCLUDE_DIRS}) + +# Basic +add_executable(executorExampleBasic executorExampleBasic.cpp) +target_link_libraries(executorExampleBasic nvinfer_plugin_tensorrt_llm) + +add_executable(executorExampleDebug executorExampleDebug.cpp) +target_link_libraries(executorExampleDebug nvinfer_plugin_tensorrt_llm) + +add_executable(executorExampleKvEvents executorExampleKvEvents.cpp) +target_link_libraries(executorExampleKvEvents nvinfer_plugin_tensorrt_llm + cxxopts::cxxopts) + +add_executable(executorExampleLogitsProcessor + executorExampleLogitsProcessor.cpp) +target_link_libraries(executorExampleLogitsProcessor + nvinfer_plugin_tensorrt_llm) + +# Advanced +if(NOT TARGET cxxopts::cxxopts) + add_subdirectory(${CMAKE_BINARY_DIR}/_deps/cxxopts-src + ${CMAKE_CURRENT_BINARY_DIR}/cxxopts) +endif() + +add_executable(executorExampleAdvanced executorExampleAdvanced.cpp) +target_link_libraries(executorExampleAdvanced nvinfer_plugin_tensorrt_llm + cxxopts::cxxopts) + +# MultiInstance +if(ENABLE_MULTI_DEVICE) + find_package(MPI REQUIRED) + message(STATUS "Using MPI_C_INCLUDE_DIRS: ${MPI_C_INCLUDE_DIRS}") + message(STATUS "Using MPI_C_LIBRARIES: ${MPI_C_LIBRARIES}") + include_directories(${MPI_C_INCLUDE_DIRS}) + + add_executable(executorExampleAdvancedMultiInstances + executorExampleAdvancedMultiInstances.cpp) + target_link_libraries( + executorExampleAdvancedMultiInstances nvinfer_plugin_tensorrt_llm + cxxopts::cxxopts ${MPI_C_LIBRARIES}) + + # FastLogits + add_executable(executorExampleFastLogits executorExampleFastLogits.cpp) + target_link_libraries(executorExampleFastLogits nvinfer_plugin_tensorrt_llm + cxxopts::cxxopts ${MPI_C_LIBRARIES}) + + add_executable(executorExampleDisaggregated executorExampleDisaggregated.cpp) + target_link_libraries( + executorExampleDisaggregated nvinfer_plugin_tensorrt_llm cxxopts::cxxopts + ${MPI_C_LIBRARIES}) +endif() diff --git a/examples/cpp/executor/README.md b/examples/cpp/executor/README.md new file mode 100644 index 000000000000..597a38effe01 --- /dev/null +++ b/examples/cpp/executor/README.md @@ -0,0 +1,135 @@ +# Executor API examples + +This directory contains several examples that demonstrate how to use the `Executor` API: +- The example defined in `executorExampleBasic.cpp` shows how you can generate output tokens for a single prompt in only a few lines of code. +- The example defined in `executorExampleAdvanced.cpp` supports more options such as providing an arbitrary number of input requests with arbitrary tokens per request and running in streaming mode. +- The example defined in `executorExampleLogitsProcessor.cpp` shows how to use `LogitsPostProcessor` to control output tokens. +- The example defined in `executorExampleFastLogits.cpp` shows how to use `ExternalDraftTokensConfig` for speculative decoding and optionally use the fast logits feature. +- The example defined in `executorExampleKvEvents.cpp` shows how to use the KV cache event API. +- The example defined in `executorExampleDisaggregated.cpp` shows how to use the disaggregated executor API. + +## Building the examples + +To build the examples, you first need to build the TensorRT LLM C++ shared libraries (`libtensorrt_llm.so` and `libnvinfer_plugin_tensorrt_llm.so`) using the [`build_wheel.py`](source:scripts/build_wheel.py) script. Alternatively, if you have already build the TensorRT LLM libraries, you can modify the provided `CMakeLists.txt` such that the `libtensorrt_llm.so` and `libnvinfer_plugin_tensorrt_llm.so` are imported properly. + +Once the TensorRT LLM libraries are built, you can run + +``` +mkdir build +cd build +cmake .. +make -j +``` +from the `./examples/cpp/executor/` folder to build the basic and advanced examples. + +## Preparing the TensorRT LLM engine(s) + +Before you run the examples, please make sure that you have already built engine(s) using the TensorRT LLM API. + +Use `trtllm-build` to build the TRT-LLM engine. + +## Running the examples + +### executorExampleBasic + +From the `examples/cpp/executor/build` folder, you can get run the `executorExampleBasic` example with: + +``` +./executorExampleBasic +``` +where `` is the path to the directly containing the TensorRT engine files. + +### executorExampleDebug + +This example shows how you can define which engine IO tensors should be dumped to numpy files. +From the `examples/cpp/executor/build` folder, you can get run the `executorExampleDebug` example with: + +``` +./executorExampleDebug +``` +where `` is the path to the directly containing the TensorRT engine files. + +### executorExampleAdvanced + +From the `examples/cpp/executor/build` folder, you can also run the `executorExampleAdvanced` example. To get the full list of supported input arguments, type + +``` +./executorExampleAdvanced -h +``` + +For example, you can run: + +``` +./executorExampleAdvanced --engine_dir --input_tokens_csv_file ../inputTokens.csv +``` + +to run with the provided dummy input tokens from `inputTokens.csv`. Upon successful completion, you should see the following in the logs: +``` +[TensorRT-LLM][INFO] Creating request with 6 input tokens +[TensorRT-LLM][INFO] Creating request with 4 input tokens +[TensorRT-LLM][INFO] Creating request with 10 input tokens +[TensorRT-LLM][INFO] Got 20 tokens for beam 0 for requestId 3 +[TensorRT-LLM][INFO] Request id 3 is completed. +[TensorRT-LLM][INFO] Got 14 tokens for beam 0 for requestId 2 +[TensorRT-LLM][INFO] Request id 2 is completed. +[TensorRT-LLM][INFO] Got 16 tokens for beam 0 for requestId 1 +[TensorRT-LLM][INFO] Request id 1 is completed. +[TensorRT-LLM][INFO] Writing output tokens to outputTokens.csv +[TensorRT-LLM][INFO] Exiting. +``` + +#### Multi-GPU run + +To run the `executorExampleAdvanced` on models that require multiple GPUs, you can run the example using MPI as follows: + +``` +mpirun -n --allow-run-as-root ./executorExampleAdvanced --engine_dir --input_tokens_csv_file ../inputTokens.csv +``` +where `` must equal to `tp*pp` for the TensorRT engine. By default GPU device IDs `[0...(num_ranks-1)]` will be used. + +Alternatively, it's also possible to run multi-GPU model by using the so-called `Orchestrator` communication mode, where the `Executor` instance will automatically spawn additional processes to run the model on multiple GPUs. To use the `Orchestrator` communication mode, you can run the example with: + +``` +./executorExampleAdvanced --engine_dir --input_tokens_csv_file ../inputTokens.csv --use_orchestrator_mode --worker_executable_path +``` +where `` is the absolute path to the stand-alone executor worker executable, located at`cpp/build/tensorrt_llm/executor_worker/executorWorker` by default. + + +### executorExampleFastLogits + +To run the `executorExampleFastLogits`, you need two GPUs (one for the draft model and one for the target model). You can run it as follows: + +``` +mpirun -n 3 --allow-run-as-root ./executorExampleFastLogits --engine_dir --draft_engine_dir --num_draft_tokens=3 +``` + +The examples uses 3 MPI ranks (one for the orchestrator, one for the draft model and one for the target model). + +Use `--fast_logits=false` to disable the fast logits feature. + +### executorExampleKvEvents + +From the `examples/cpp/executor/build` folder, you can get run the `executorExampleKvEvents` example with: + +``` +./executorExampleKvEvents --engine_dir +``` +where `` is the path to the directly containing the TensorRT engine files. + +This example shows how the KV Cache Event API can be used to reconstruct the state of TRT-LLM's internal radix tree. This can be used in applications such as smart routing to route requests between multiple executor instances to maximize KV Cache reuse. Events are emitted when blocks are stored, removed, or updated in the radix tree. + +### executorExampleDisaggregated + +From the `examples/cpp/executor/build` folder, you can also run the `executorExampleDisaggregated` example. To get the full list of supported input arguments, type +``` +./executorExampleDisaggregated -h +``` +Note setting `TRTLLM_USE_UCX_KVCACHE=1` is required to run disaggregated executor. +For example, you can run : +``` +export TRTLLM_USE_UCX_KVCACHE=1 + +mpirun -n --allow-run-as-root --oversubscribe ./executorExampleDisaggregated --context_engine_dir --context_rank_size --generation_engine_dir --generation_rank_size --input_tokens_csv_file ../inputTokens.csv + +``` +where `` must equal to `tp*pp` for the context engine, and `` must equal to `tp*pp` for the generation engine,the context engine and generation engine can be heterogeneous in parallelism. `` must equal to `++1`, the additional rank is used as orchestrator process. diff --git a/examples/cpp/executor/executorExampleAdvanced.cpp b/examples/cpp/executor/executorExampleAdvanced.cpp new file mode 100644 index 000000000000..e2fdf489f63d --- /dev/null +++ b/examples/cpp/executor/executorExampleAdvanced.cpp @@ -0,0 +1,368 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include +#include + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include + +namespace tle = tensorrt_llm::executor; + +namespace fs = std::filesystem; + +struct RuntimeOptions +{ + std::string trtEnginePath; + std::string inputTokensCsvFile; + std::string outputTokensCsvFile; + + bool streaming; + bool excludeInputFromOutput; + tle::SizeType32 maxNewTokens; + tle::SizeType32 beamWidth; + std::optional numReturnSequences; + tle::SizeType32 timeoutMs; + + bool useOrchestratorMode; + std::string workerExecutablePath; +}; + +// Utility function to parse input arguments +RuntimeOptions parseArgs(int argc, char* argv[]); + +// Function that enqueues requests +std::vector enqueueRequests(RuntimeOptions const& runtimeOpts, tle::Executor& executor); + +// Function that waits for responses and stores output tokens +std::unordered_map waitForResponses( + RuntimeOptions const& runtimeOpts, std::vector const& requestIds, tle::Executor& executor); + +// Utility function to read input tokens from csv file +std::vector readInputTokens(std::string const& path); + +// Utility function to write output tokens from csv file +void writeOutputTokens(std::string const& path, std::vector& requestIds, + std::unordered_map const& outputTokens, tle::SizeType32 beamWidth); + +tle::SizeType32 getNumSequencesPerRequest(RuntimeOptions const& runtimeOpts); + +// Main +int main(int argc, char* argv[]) +{ + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + auto runtimeOpts = parseArgs(argc, argv); + + // Create the executor for this engine + auto executorConfig = tle::ExecutorConfig(runtimeOpts.beamWidth); + + if (runtimeOpts.useOrchestratorMode) + { + auto orchestratorConfig = tle::OrchestratorConfig(true, runtimeOpts.workerExecutablePath); + auto parallelConfig = tle::ParallelConfig(tle::CommunicationType::kMPI, tle::CommunicationMode::kORCHESTRATOR, + std::nullopt, std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + } + + auto executor = tle::Executor(runtimeOpts.trtEnginePath, tle::ModelType::kDECODER_ONLY, executorConfig); + + if (executor.canEnqueueRequests()) + { + // Create the requests + auto requestIds = enqueueRequests(runtimeOpts, executor); + + // Wait for responses and store output tokens + auto outputTokens = waitForResponses(runtimeOpts, requestIds, executor); + + // Write output tokens csv file + TLLM_LOG_INFO("Writing output tokens to %s", runtimeOpts.outputTokensCsvFile.c_str()); + auto numSequences = getNumSequencesPerRequest(runtimeOpts); + writeOutputTokens(runtimeOpts.outputTokensCsvFile, requestIds, outputTokens, numSequences); + } + TLLM_LOG_INFO("Exiting."); + return 0; +} + +RuntimeOptions parseArgs(int argc, char* argv[]) +{ + RuntimeOptions runtimeOpts; + + cxxopts::Options options(argv[0], "Example that demonstrates how to use the Executor API"); + options.add_options()("h,help", "Print usage"); + options.add_options()("engine_dir", "Directory that store the engines.", cxxopts::value()); + options.add_options()("beam_width", "The beam width", cxxopts::value()->default_value("1")); + options.add_options()( + "num_return_sequences", "The number of return sequences per request.", cxxopts::value>()); + options.add_options()("streaming", "Operate in streaming mode", cxxopts::value()->default_value("false")); + options.add_options()("exclude_input_from_output", + "Exclude input tokens when writing output tokens. Only has effect for streaming = false. For streaming = true, " + "output tokens are not included.", + cxxopts::value()->default_value("false")); + options.add_options()( + "max_new_tokens", "The maximum number of tokens to generate", cxxopts::value()->default_value("10")); + options.add_options()( + "input_tokens_csv_file", "Path to a csv file that contains input tokens", cxxopts::value()); + options.add_options()("output_tokens_csv_file", "Path to a csv file that will contain the output tokens", + cxxopts::value()->default_value("outputTokens.csv")); + options.add_options()("timeout_ms", "The maximum time to wait for all responses, in milliseconds.", + cxxopts::value()->default_value("10000")); + options.add_options()("use_orchestrator_mode", "Use orchestrator communication mode.", + cxxopts::value()->default_value("false")); + options.add_options()("worker_executable_path", "The location of the worker executable.", + cxxopts::value()->default_value("")); + + auto parsedOptions = options.parse(argc, argv); + + // Argument: help + if (parsedOptions.count("help")) + { + TLLM_LOG_ERROR(options.help()); + exit(0); + } + + // Argument: Engine directory + if (!parsedOptions.count("engine_dir")) + { + TLLM_LOG_ERROR(options.help()); + TLLM_LOG_ERROR("Please specify engine directory."); + exit(1); + } + runtimeOpts.trtEnginePath = parsedOptions["engine_dir"].as(); + if (!fs::exists(runtimeOpts.trtEnginePath) || !fs::is_directory(runtimeOpts.trtEnginePath)) + { + TLLM_LOG_ERROR("Engine directory doesn't exist."); + exit(1); + } + + // Argument: Input tokens csv file + if (!parsedOptions.count("input_tokens_csv_file")) + { + TLLM_LOG_ERROR(options.help()); + TLLM_LOG_ERROR("Please specify input_tokens_csv_file"); + exit(1); + } + runtimeOpts.inputTokensCsvFile = parsedOptions["input_tokens_csv_file"].as(); + runtimeOpts.streaming = parsedOptions["streaming"].as(); + runtimeOpts.excludeInputFromOutput = parsedOptions["exclude_input_from_output"].as(); + runtimeOpts.maxNewTokens = parsedOptions["max_new_tokens"].as(); + runtimeOpts.beamWidth = parsedOptions["beam_width"].as(); + if (parsedOptions.count("num_return_sequences") > 0) + { + runtimeOpts.numReturnSequences = parsedOptions["num_return_sequences"].as>(); + } + runtimeOpts.timeoutMs = parsedOptions["timeout_ms"].as(); + runtimeOpts.outputTokensCsvFile = parsedOptions["output_tokens_csv_file"].as(); + + runtimeOpts.useOrchestratorMode = parsedOptions["use_orchestrator_mode"].as(); + runtimeOpts.workerExecutablePath = parsedOptions["worker_executable_path"].as(); + + return runtimeOpts; +} + +std::vector enqueueRequests(RuntimeOptions const& runtimeOpts, tle::Executor& executor) +{ + tle::OutputConfig outputConfig; + outputConfig.excludeInputFromOutput = runtimeOpts.excludeInputFromOutput; + tle::SamplingConfig samplingConfig(runtimeOpts.beamWidth); + if (runtimeOpts.numReturnSequences && runtimeOpts.beamWidth == 1) + { + samplingConfig.setTopP(0.9); + } + samplingConfig.setNumReturnSequences(runtimeOpts.numReturnSequences); + + TLLM_LOG_INFO("Reading input tokens from %s", runtimeOpts.inputTokensCsvFile.c_str()); + auto inputTokens = readInputTokens(runtimeOpts.inputTokensCsvFile); + TLLM_LOG_INFO("Number of requests: %d", inputTokens.size()); + + std::vector requests; + for (auto& tokens : inputTokens) + { + TLLM_LOG_INFO("Creating request with %d input tokens", tokens.size()); + requests.emplace_back( + std::move(tokens), runtimeOpts.maxNewTokens, runtimeOpts.streaming, samplingConfig, outputConfig); + } + + // Enqueue the requests + auto requestIds = executor.enqueueRequests(std::move(requests)); + + return requestIds; +} + +std::unordered_map waitForResponses( + RuntimeOptions const& runtimeOpts, std::vector const& requestIds, tle::Executor& executor) +{ + // Map that will be used to store output tokens for requests + std::unordered_map outputTokens; + auto numSequences = getNumSequencesPerRequest(runtimeOpts); + for (auto requestId : requestIds) + { + outputTokens[requestId] = tle::BeamTokens(numSequences); + } + + tle::SizeType32 numFinished{0}; + tle::SizeType32 iter{0}; + + // Get the new tokens for each request + while (numFinished < static_cast(requestIds.size()) && iter < runtimeOpts.timeoutMs) + { + std::chrono::milliseconds waitTime(1); + // Wait for any response + auto responses = executor.awaitResponses(waitTime); + + auto insertResponseTokens + = [&outputTokens](tle::IdType requestId, tle::SizeType32 seqIdx, tle::VecTokens const& respTokens) + { + TLLM_LOG_INFO("Got %d tokens for seqIdx %d for requestId %d", respTokens.size(), seqIdx, requestId); + + // Store the output tokens for that request id + auto& outTokens = outputTokens.at(requestId).at(seqIdx); + outTokens.insert(outTokens.end(), std::make_move_iterator(respTokens.begin()), + std::make_move_iterator(respTokens.end())); + }; + + // Loop over the responses + for (auto const& response : responses) + { + auto requestId = response.getRequestId(); + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + if (runtimeOpts.beamWidth > 1) + { + for (tle::SizeType32 beam = 0; beam < numSequences; ++beam) + { + insertResponseTokens(requestId, beam, result.outputTokenIds.at(beam)); + } + } + else + { + insertResponseTokens(requestId, result.sequenceIndex, result.outputTokenIds.at(0)); + } + if (result.isFinal) + { + TLLM_LOG_INFO("Request id %lu is completed.", requestId); + } + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + if (response.getErrorMsg() != err) + { + TLLM_THROW("Request id %lu encountered error: %s", requestId, response.getErrorMsg().c_str()); + } + } + } + ++iter; + } + if (iter == runtimeOpts.timeoutMs) + { + TLLM_THROW("Timeout exceeded."); + } + + return outputTokens; +} + +std::vector readInputTokens(std::string const& path) +{ + std::vector data; + std::ifstream file(path); + + if (!file.is_open()) + { + auto const err = std::string{"Failed to open file: "} + path; + TLLM_LOG_ERROR(err); + TLLM_THROW(err); + } + + std::string line; + while (std::getline(file, line)) + { + std::vector row; + std::stringstream ss(line); + std::string token; + + while (std::getline(ss, token, ',')) + { + try + { + row.push_back(std::stoi(token)); + } + catch (std::invalid_argument const& e) + { + TLLM_LOG_ERROR("Invalid argument: %s", e.what()); + } + catch (std::out_of_range const& e) + { + TLLM_LOG_ERROR("Out of range: %s", e.what()); + } + } + + data.push_back(row); + } + + file.close(); + return data; +} + +void writeOutputTokens(std::string const& path, std::vector& requestIds, + std::unordered_map const& outputTokens, tle::SizeType32 beamWidth) +{ + std::ofstream file(path); + + if (!file.is_open()) + { + TLLM_LOG_ERROR("Failed to open file %s", path.c_str()); + return; + } + + for (auto requestId : requestIds) + { + auto const& outTokens = outputTokens.at(requestId); + for (tle::SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto const& beamTokens = outTokens.at(beam); + for (size_t i = 0; i < beamTokens.size(); ++i) + { + file << beamTokens[i]; + if (i < beamTokens.size() - 1) + { + file << ", "; + } + } + file << "\n"; + } + } + + file.close(); +} + +tle::SizeType32 getNumSequencesPerRequest(RuntimeOptions const& runtimeOpts) +{ + auto numReturnSequences = runtimeOpts.numReturnSequences.value_or(runtimeOpts.beamWidth); + return runtimeOpts.beamWidth > 1 ? std::min(numReturnSequences, runtimeOpts.beamWidth) : numReturnSequences; +} diff --git a/examples/cpp/executor/executorExampleAdvancedMultiInstances.cpp b/examples/cpp/executor/executorExampleAdvancedMultiInstances.cpp new file mode 100644 index 000000000000..f967661ccd27 --- /dev/null +++ b/examples/cpp/executor/executorExampleAdvancedMultiInstances.cpp @@ -0,0 +1,381 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include +#include + +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include + +namespace tle = tensorrt_llm::executor; + +namespace fs = std::filesystem; + +struct RuntimeOptions +{ + std::string trtEnginePath; + std::string inputTokensCsvFile; + std::string outputTokensCsvFile; + + bool streaming; + bool excludeInputFromOutput; + tle::SizeType32 maxNewTokens; + tle::SizeType32 beamWidth; + tle::SizeType32 timeoutMs; + + bool useOrchestratorMode; + std::string workerExecutablePath; + bool spawnProcesses; +}; + +// Utility function to parse input arguments +RuntimeOptions parseArgs(int argc, char* argv[]); + +// Function that enqueues requests +std::vector> enqueueRequests( + RuntimeOptions const& runtimeOpts, std::deque& executors); + +// Function that waits for responses and stores output tokens +std::map, tle::BeamTokens> waitForResponses(RuntimeOptions const& runtimeOpts, + std::vector> const& instanceRequestIds, std::deque& executors); + +// Utility function to read input tokens from csv file +std::vector readInputTokens(std::string const& path); + +// Utility function to write output tokens from csv file +void writeOutputTokens(std::string const& path, std::vector>& requestIds, + std::map, tle::BeamTokens> const& outputTokens, tle::SizeType32 beamWidth); + +// Main +int main(int argc, char* argv[]) +{ + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + auto runtimeOpts = parseArgs(argc, argv); + + // Create the executor for this engine + auto executorConfig = tle::ExecutorConfig(runtimeOpts.beamWidth); + + tle::KvCacheConfig kvCacheConfig{false, 10000}; + executorConfig.setKvCacheConfig(kvCacheConfig); + + bool isOrchestrator = true; + if (!runtimeOpts.spawnProcesses) + { + tensorrt_llm::mpi::initialize(tensorrt_llm::mpi::MpiThreadSupport::THREAD_MULTIPLE); + int myRank = tensorrt_llm::mpi::MpiComm::world().getRank(); + isOrchestrator = (myRank == 0); + } + + auto orchestratorConfig = tle::OrchestratorConfig( + isOrchestrator, runtimeOpts.workerExecutablePath, nullptr, runtimeOpts.spawnProcesses); + auto parallelConfig = tle::ParallelConfig(tle::CommunicationType::kMPI, tle::CommunicationMode::kORCHESTRATOR, + std::nullopt, std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + int numInstances = 3; + if (!runtimeOpts.spawnProcesses) + { + // Keep one rank for orchestrator + numInstances = tensorrt_llm::mpi::MpiComm::world().getSize() - 1; + } + std::deque executors; + for (int instanceId = 0; instanceId < numInstances; ++instanceId) + { + auto executorConfigTmp = executorConfig; + // Set the rank id participating in each model instance + if (!runtimeOpts.spawnProcesses) + { + parallelConfig.setParticipantIds({instanceId + 1}); + } + executorConfigTmp.setParallelConfig(parallelConfig); + executors.emplace_back(runtimeOpts.trtEnginePath, tle::ModelType::kDECODER_ONLY, executorConfigTmp); + } + + // Only orchestrator rank (rank 0) will enter + if (isOrchestrator) + { + // Create the requests + auto instanceRequestIds = enqueueRequests(runtimeOpts, executors); + + // Wait for responses and store output tokens + auto outputTokens = waitForResponses(runtimeOpts, instanceRequestIds, executors); + + // Write output tokens csv file + TLLM_LOG_INFO("Writing output tokens to %s", runtimeOpts.outputTokensCsvFile.c_str()); + writeOutputTokens(runtimeOpts.outputTokensCsvFile, instanceRequestIds, outputTokens, runtimeOpts.beamWidth); + } + TLLM_LOG_INFO("Exiting."); + return 0; +} + +RuntimeOptions parseArgs(int argc, char* argv[]) +{ + RuntimeOptions runtimeOpts; + + cxxopts::Options options(argv[0], "Example that demonstrates how to use the Executor API"); + options.add_options()("h,help", "Print usage"); + options.add_options()("engine_dir", "Directory that store the engines.", cxxopts::value()); + options.add_options()("beam_width", "The beam width", cxxopts::value()->default_value("1")); + options.add_options()("streaming", "Operate in streaming mode", cxxopts::value()->default_value("false")); + options.add_options()("exclude_input_from_output", + "Exclude input tokens when writing output tokens. Only has effect for streaming = false. For streaming = true, " + "output tokens are not included.", + cxxopts::value()->default_value("false")); + options.add_options()( + "max_new_tokens", "The maximum number of tokens to generate", cxxopts::value()->default_value("10")); + options.add_options()( + "input_tokens_csv_file", "Path to a csv file that contains input tokens", cxxopts::value()); + options.add_options()("output_tokens_csv_file", "Path to a csv file that will contain the output tokens", + cxxopts::value()->default_value("outputTokens.csv")); + options.add_options()("timeout_ms", "The maximum time to wait for all responses, in milliseconds.", + cxxopts::value()->default_value("10000")); + options.add_options()("worker_executable_path", "The location of the worker executable.", + cxxopts::value()->default_value("")); + options.add_options()("spawn_processes", + "Flag that controls if MPI_Comm_spawn should be used to spawn worker processes, or if they have been launched " + "with mpi already.", + cxxopts::value()->default_value("true")); + + auto parsedOptions = options.parse(argc, argv); + + // Argument: help + if (parsedOptions.count("help")) + { + TLLM_LOG_ERROR(options.help()); + exit(0); + } + + // Argument: Engine directory + if (!parsedOptions.count("engine_dir")) + { + TLLM_LOG_ERROR(options.help()); + TLLM_LOG_ERROR("Please specify engine directory."); + exit(1); + } + runtimeOpts.trtEnginePath = parsedOptions["engine_dir"].as(); + if (!fs::exists(runtimeOpts.trtEnginePath) || !fs::is_directory(runtimeOpts.trtEnginePath)) + { + TLLM_LOG_ERROR("Engine directory doesn't exist."); + exit(1); + } + + // Argument: Input tokens csv file + if (!parsedOptions.count("input_tokens_csv_file")) + { + TLLM_LOG_ERROR(options.help()); + TLLM_LOG_ERROR("Please specify input_tokens_csv_file"); + exit(1); + } + runtimeOpts.inputTokensCsvFile = parsedOptions["input_tokens_csv_file"].as(); + runtimeOpts.streaming = parsedOptions["streaming"].as(); + runtimeOpts.excludeInputFromOutput = parsedOptions["exclude_input_from_output"].as(); + runtimeOpts.maxNewTokens = parsedOptions["max_new_tokens"].as(); + runtimeOpts.beamWidth = parsedOptions["beam_width"].as(); + runtimeOpts.timeoutMs = parsedOptions["timeout_ms"].as(); + runtimeOpts.outputTokensCsvFile = parsedOptions["output_tokens_csv_file"].as(); + + runtimeOpts.workerExecutablePath = parsedOptions["worker_executable_path"].as(); + runtimeOpts.spawnProcesses = parsedOptions["spawn_processes"].as(); + + return runtimeOpts; +} + +std::vector> enqueueRequests( + RuntimeOptions const& runtimeOpts, std::deque& executors) +{ + tle::OutputConfig outputConfig; + outputConfig.excludeInputFromOutput = runtimeOpts.excludeInputFromOutput; + tle::SamplingConfig samplingConfig(runtimeOpts.beamWidth); + + TLLM_LOG_INFO("Reading input tokens from %s", runtimeOpts.inputTokensCsvFile.c_str()); + auto inputTokens = readInputTokens(runtimeOpts.inputTokensCsvFile); + TLLM_LOG_INFO("Number of requests: %d", inputTokens.size()); + + std::vector requests; + for (auto& tokens : inputTokens) + { + TLLM_LOG_INFO("Creating request with %d input tokens", tokens.size()); + requests.emplace_back( + std::move(tokens), runtimeOpts.maxNewTokens, runtimeOpts.streaming, samplingConfig, outputConfig); + } + + // Enqueue the requests + // Round robin over instances + std::vector> instanceRequestIds; + for (size_t req = 0; req < requests.size(); ++req) + { + auto instanceId = req % executors.size(); + TLLM_LOG_INFO("Enqueuing request %d for instance %d", req, instanceId); + auto requestId = executors.at(instanceId).enqueueRequest(requests[req]); + instanceRequestIds.emplace_back(instanceId, requestId); + } + TLLM_LOG_INFO("Enqueued %d requests", instanceRequestIds.size()); + return instanceRequestIds; +} + +std::map, tle::BeamTokens> waitForResponses(RuntimeOptions const& runtimeOpts, + std::vector> const& instanceRequestIds, std::deque& executors) +{ + // Map that will be used to store output tokens for requests + int numRequests = 0; + std::map, tle::BeamTokens> outputTokens; + for (auto instanceRequestId : instanceRequestIds) + { + outputTokens[instanceRequestId] = tle::BeamTokens(runtimeOpts.beamWidth); + numRequests++; + } + + tle::SizeType32 numFinished{0}; + tle::SizeType32 iter{0}; + + // Get the new tokens for each request + while (numFinished < numRequests && iter < runtimeOpts.timeoutMs) + { + std::chrono::milliseconds waitTime(1); + for (size_t instanceId = 0; instanceId < executors.size(); ++instanceId) + { + // Wait for any response for given instance + auto responses = executors.at(instanceId).awaitResponses(waitTime); + // Loop over the responses + for (auto const& response : responses) + { + auto requestId = response.getRequestId(); + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + TLLM_LOG_INFO("Number of finished requests: %d", numFinished); + + for (tle::SizeType32 beam = 0; beam < runtimeOpts.beamWidth; ++beam) + { + auto& respTokens = result.outputTokenIds.at(beam); + + TLLM_LOG_INFO("Got %d tokens for beam %d for requestId %d", respTokens.size(), beam, requestId); + + // Store the output tokens for that request id + auto& outTokens = outputTokens.at(std::make_pair(instanceId, requestId)).at(beam); + outTokens.insert(outTokens.end(), std::make_move_iterator(respTokens.begin()), + std::make_move_iterator(respTokens.end())); + } + if (result.isFinal) + { + TLLM_LOG_INFO("Request id %lu is completed.", requestId); + } + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "ReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + if (response.getErrorMsg() != err) + { + TLLM_THROW("Request id %lu encountered error: %s", requestId, response.getErrorMsg().c_str()); + } + } + } + } + ++iter; + } + if (iter == runtimeOpts.timeoutMs) + { + TLLM_THROW("Timeout exceeded."); + } + + return outputTokens; +} + +std::vector readInputTokens(std::string const& path) +{ + std::vector data; + std::ifstream file(path); + + if (!file.is_open()) + { + auto const err = std::string{"Failed to open file: "} + path; + TLLM_LOG_ERROR(err); + TLLM_THROW(err); + } + + std::string line; + while (std::getline(file, line)) + { + std::vector row; + std::stringstream ss(line); + std::string token; + + while (std::getline(ss, token, ',')) + { + try + { + row.push_back(std::stoi(token)); + } + catch (std::invalid_argument const& e) + { + TLLM_LOG_ERROR("Invalid argument: %s", e.what()); + } + catch (std::out_of_range const& e) + { + TLLM_LOG_ERROR("Out of range: %s", e.what()); + } + } + + data.push_back(row); + } + + file.close(); + return data; +} + +void writeOutputTokens(std::string const& path, std::vector>& instanceRequestIds, + std::map, tle::BeamTokens> const& outputTokens, tle::SizeType32 beamWidth) +{ + std::ofstream file(path); + + if (!file.is_open()) + { + TLLM_LOG_ERROR("Failed to open file %s", path.c_str()); + return; + } + + for (auto instanceRequestId : instanceRequestIds) + { + auto const& outTokens = outputTokens.at(instanceRequestId); + for (tle::SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto const& beamTokens = outTokens.at(beam); + for (size_t i = 0; i < beamTokens.size(); ++i) + { + file << beamTokens[i]; + if (i < beamTokens.size() - 1) + { + file << ", "; + } + } + file << "\n"; + } + } + + file.close(); +} diff --git a/examples/cpp/executor/executorExampleBasic.cpp b/examples/cpp/executor/executorExampleBasic.cpp new file mode 100644 index 000000000000..b3ae3328392c --- /dev/null +++ b/examples/cpp/executor/executorExampleBasic.cpp @@ -0,0 +1,60 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include + +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" + +namespace tlc = tensorrt_llm::common; +namespace tle = tensorrt_llm::executor; + +int main(int argc, char* argv[]) +{ + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + if (argc != 2) + { + TLLM_LOG_ERROR("Usage: %s ", argv[0]); + return 1; + } + + // Create the executor for this engine + tle::SizeType32 beamWidth = 1; + auto executorConfig = tle::ExecutorConfig(beamWidth); + auto trtEnginePath = argv[1]; + auto executor = tle::Executor(trtEnginePath, tle::ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + tle::SizeType32 maxNewTokens = 5; + tle::VecTokens inputTokens{1, 2, 3, 4}; + auto request = tle::Request(inputTokens, maxNewTokens); + + // Enqueue the request + auto requestId = executor.enqueueRequest(request); + + // Wait for the response + auto responses = executor.awaitResponses(requestId); + + // Get outputTokens + auto outputTokens = responses.at(0).getResult().outputTokenIds.at(beamWidth - 1); + + TLLM_LOG_INFO("Output tokens: %s", tlc::vec2str(outputTokens).c_str()); + + return 0; +} diff --git a/examples/cpp/executor/executorExampleDebug.cpp b/examples/cpp/executor/executorExampleDebug.cpp new file mode 100644 index 000000000000..d0af1a8140b5 --- /dev/null +++ b/examples/cpp/executor/executorExampleDebug.cpp @@ -0,0 +1,65 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include + +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" + +namespace tlc = tensorrt_llm::common; +namespace tle = tensorrt_llm::executor; + +int main(int argc, char* argv[]) +{ + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + if (argc != 2) + { + TLLM_LOG_ERROR("Usage: %s ", argv[0]); + return 1; + } + + // Create the executor for this engine + tle::SizeType32 beamWidth = 1; + auto executorConfig = tle::ExecutorConfig(beamWidth); + // Select which tensors should be dumped + auto debugConfig = tle::DebugConfig(); + debugConfig.setDebugTensorNames({"host_request_types"}); + executorConfig.setDebugConfig(debugConfig); + + auto trtEnginePath = argv[1]; + auto executor = tle::Executor(trtEnginePath, tle::ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + tle::SizeType32 maxNewTokens = 2; + tle::VecTokens inputTokens{1, 2, 3, 4}; + auto request = tle::Request(inputTokens, maxNewTokens); + + // Enqueue the request + auto requestId = executor.enqueueRequest(request); + + // Wait for the response + auto responses = executor.awaitResponses(requestId); + + // Get outputTokens + auto outputTokens = responses.at(0).getResult().outputTokenIds.at(beamWidth - 1); + + TLLM_LOG_INFO("Output tokens: %s", tlc::vec2str(outputTokens).c_str()); + + return 0; +} diff --git a/examples/cpp/executor/executorExampleDisaggregated.cpp b/examples/cpp/executor/executorExampleDisaggregated.cpp new file mode 100644 index 000000000000..ef9ff85c5e3b --- /dev/null +++ b/examples/cpp/executor/executorExampleDisaggregated.cpp @@ -0,0 +1,441 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include +#include +#include + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/executor/types.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" + +#include + +namespace tle = tensorrt_llm::executor; + +namespace fs = std::filesystem; + +struct RuntimeOptions +{ + std::string trtContextEnginePath; + std::string trtGenerationEnginePath; + std::string inputTokensCsvFile; + std::string outputTokensCsvFile; + + bool streaming; + bool excludeInputFromOutput; + int contextRankSize; + int generationRankSize; + tle::SizeType32 maxNewTokens; + tle::SizeType32 beamWidth; + std::optional numReturnSequences; + tle::SizeType32 timeoutMs; +}; + +RuntimeOptions parseArgs(int argc, char* argv[]); + +// Function that enqueues requests into context executor and generation executor +std::unordered_map enqueueRequests( + RuntimeOptions const& runtimeOpts, tle::Executor& contextExecutor, tle::Executor& generationExecutor); + +// Function that waits for gen responses and stores output tokens +std::unordered_map waitForGenResponses(RuntimeOptions const& runtimeOpts, + std::unordered_map const& genRequestIdToContextRequestId, + tle::Executor& generationExecutor); + +// Utility function to read input tokens from csv file +std::vector readInputTokens(std::string const& path); + +// Utility function to write output tokens from csv file +void writeOutputTokens(std::string const& path, + std::unordered_map& genRequestIdToContextRequestId, + std::unordered_map const& outputTokens, tle::SizeType32 beamWidth); + +int main(int argc, char* argv[]) +{ + + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + auto runtimeOpts = parseArgs(argc, argv); + TLLM_CHECK_WITH_INFO(runtimeOpts.beamWidth == 1, "Only support beamWidth =1"); + TLLM_CHECK_WITH_INFO( + runtimeOpts.numReturnSequences.has_value() == false || runtimeOpts.numReturnSequences.value() == 1, + "Only support numReturnSequences =1"); + // Create the executor for this engine + auto contextExecutorConfig = tle::ExecutorConfig(runtimeOpts.beamWidth); + auto generationExecutorConfig = tle::ExecutorConfig(runtimeOpts.beamWidth); + bool isOrchestrator = (tensorrt_llm::mpi::MpiComm::world().getRank() == 0); + auto orchestratorConfig = tle::OrchestratorConfig(isOrchestrator, "", nullptr, false); + int contextRankSize = runtimeOpts.contextRankSize; + int generationRankSize = runtimeOpts.generationRankSize; + TLLM_CHECK_WITH_INFO(tensorrt_llm::mpi::MpiComm::world().getSize() >= contextRankSize + generationRankSize + 1, + " MPI should launch at least [contextRankSize+generationRankSize+1]: %d processes", + contextRankSize + generationRankSize + 1); + int deviceCount = -1; + TLLM_CHECK(cudaGetDeviceCount(&deviceCount) == cudaSuccess); + + std::vector contextRankIds(contextRankSize); + std::vector contextDeviceIds(contextRankSize); + std::vector generationRankIds(generationRankSize); + std::vector generationDeviceIds(generationRankSize); + for (int i = 0; i < contextRankSize; i++) + { + contextRankIds[i] = i + 1; + contextDeviceIds[i] = i % deviceCount; + TLLM_LOG_INFO("context Rank %d on device %d", contextRankIds[i], contextDeviceIds[i]); + } + tle::ParallelConfig contextParallelConfig{tensorrt_llm::executor::CommunicationType::kMPI, + tensorrt_llm::executor::CommunicationMode::kORCHESTRATOR, contextDeviceIds, contextRankIds, orchestratorConfig}; + + for (int i = 0; i < generationRankSize; i++) + { + generationRankIds[i] = i + 1 + contextRankSize; + generationDeviceIds[i] = (i + contextRankSize) % deviceCount; + TLLM_LOG_INFO("generation Rank %d on device %d", generationRankIds[i], generationDeviceIds[i]); + } + tle::ParallelConfig generationParallelConfig{tensorrt_llm::executor::CommunicationType::kMPI, + tensorrt_llm::executor::CommunicationMode::kORCHESTRATOR, generationDeviceIds, generationRankIds, + orchestratorConfig}; + + contextExecutorConfig.setParallelConfig(contextParallelConfig); + generationExecutorConfig.setParallelConfig(generationParallelConfig); + + auto contextExecutor + = tle::Executor(runtimeOpts.trtContextEnginePath, tle::ModelType::kDECODER_ONLY, contextExecutorConfig); + auto generationExecutor + = tle::Executor(runtimeOpts.trtGenerationEnginePath, tle::ModelType::kDECODER_ONLY, generationExecutorConfig); + tensorrt_llm::mpi::MpiComm::world().barrier(); + + if (tensorrt_llm::mpi::MpiComm::world().getRank() == 0) + { + + TLLM_CHECK_WITH_INFO(contextExecutor.canEnqueueRequests(), "contextExecutor can't enqueue requests"); + TLLM_CHECK_WITH_INFO(generationExecutor.canEnqueueRequests(), "generationExecutor can't enqueue requests"); + auto genRequestIdsToContextRequestIds = enqueueRequests(runtimeOpts, contextExecutor, generationExecutor); + auto outputTokens = waitForGenResponses(runtimeOpts, genRequestIdsToContextRequestIds, generationExecutor); + TLLM_LOG_INFO("Writing output tokens to %s", runtimeOpts.outputTokensCsvFile.c_str()); + writeOutputTokens( + runtimeOpts.outputTokensCsvFile, genRequestIdsToContextRequestIds, outputTokens, runtimeOpts.beamWidth); + } + tensorrt_llm::mpi::MpiComm::world().barrier(); + + TLLM_LOG_INFO("Exiting."); + return 0; +} + +RuntimeOptions parseArgs(int argc, char* argv[]) +{ + + RuntimeOptions runtimeOpts; + + cxxopts::Options options(argv[0], "Example that demonstrates how to use the Executor Disaggregated API"); + options.add_options()("h,help", "Print usage"); + options.add_options()( + "context_engine_dir", "Directory that store the context engine.", cxxopts::value()); + options.add_options()( + "generation_engine_dir", "Directory that store the generation engine.", cxxopts::value()); + options.add_options()( + "context_rank_size", "The number of ranks for the context engine", cxxopts::value()->default_value("1")); + options.add_options()("generation_rank_size", "The number of ranks for the generation engine", + cxxopts::value()->default_value("1")); + options.add_options()("beam_width", "The beam width", cxxopts::value()->default_value("1")); + options.add_options()( + "num_return_sequences", "The number of return sequences per request.", cxxopts::value>()); + options.add_options()("streaming", "Operate in streaming mode", cxxopts::value()->default_value("false")); + options.add_options()("exclude_input_from_output", + "Exclude input tokens when writing output tokens. Only has effect for streaming = false. For streaming = true, " + "output tokens are not included.", + cxxopts::value()->default_value("false")); + options.add_options()( + "max_new_tokens", "The maximum number of tokens to generate", cxxopts::value()->default_value("10")); + options.add_options()( + "input_tokens_csv_file", "Path to a csv file that contains input tokens", cxxopts::value()); + options.add_options()("output_tokens_csv_file", "Path to a csv file that will contain the output tokens", + cxxopts::value()->default_value("outputTokens.csv")); + options.add_options()("timeout_ms", "The maximum time to wait for all responses, in milliseconds.", + cxxopts::value()->default_value("10000")); + + auto parsedOptions = options.parse(argc, argv); + + // Argument: help + if (parsedOptions.count("help")) + { + TLLM_LOG_ERROR(options.help()); + exit(0); + } + + runtimeOpts.trtContextEnginePath = parsedOptions["context_engine_dir"].as(); + if (!fs::exists(runtimeOpts.trtContextEnginePath) || !fs::is_directory(runtimeOpts.trtContextEnginePath)) + { + TLLM_LOG_ERROR("Context engine directory doesn't exist."); + exit(1); + } + + runtimeOpts.trtGenerationEnginePath = parsedOptions["generation_engine_dir"].as(); + if (!fs::exists(runtimeOpts.trtGenerationEnginePath) || !fs::is_directory(runtimeOpts.trtGenerationEnginePath)) + { + TLLM_LOG_ERROR("Generation engine directory doesn't exist."); + exit(1); + } + // Argument: Input tokens csv file + if (!parsedOptions.count("input_tokens_csv_file")) + { + TLLM_LOG_ERROR(options.help()); + TLLM_LOG_ERROR("Please specify input_tokens_csv_file"); + exit(1); + } + + runtimeOpts.inputTokensCsvFile = parsedOptions["input_tokens_csv_file"].as(); + runtimeOpts.streaming = parsedOptions["streaming"].as(); + runtimeOpts.excludeInputFromOutput = parsedOptions["exclude_input_from_output"].as(); + runtimeOpts.maxNewTokens = parsedOptions["max_new_tokens"].as(); + runtimeOpts.beamWidth = parsedOptions["beam_width"].as(); + runtimeOpts.contextRankSize = parsedOptions["context_rank_size"].as(); + runtimeOpts.generationRankSize = parsedOptions["generation_rank_size"].as(); + if (parsedOptions.count("num_return_sequences") > 0) + { + runtimeOpts.numReturnSequences = parsedOptions["num_return_sequences"].as>(); + } + runtimeOpts.timeoutMs = parsedOptions["timeout_ms"].as(); + runtimeOpts.outputTokensCsvFile = parsedOptions["output_tokens_csv_file"].as(); + + return runtimeOpts; +} + +std::unordered_map enqueueRequests( + RuntimeOptions const& runtimeOpts, tle::Executor& contextExecutor, tle::Executor& generationExecutor) +{ + + tle::OutputConfig outputConfig; + outputConfig.excludeInputFromOutput = runtimeOpts.excludeInputFromOutput; + tle::SamplingConfig samplingConfig(runtimeOpts.beamWidth); + std::unordered_map genRequestIdToContextRequestId; + if (runtimeOpts.numReturnSequences && runtimeOpts.beamWidth == 1) + { + samplingConfig.setTopP(0.9); + } + samplingConfig.setNumReturnSequences(runtimeOpts.numReturnSequences); + + TLLM_LOG_INFO("Reading input tokens from %s", runtimeOpts.inputTokensCsvFile.c_str()); + auto inputTokens = readInputTokens(runtimeOpts.inputTokensCsvFile); + TLLM_LOG_INFO("Number of requests: %d", inputTokens.size()); + + std::vector requests; + for (auto& tokens : inputTokens) + { + TLLM_LOG_INFO("Creating request with %d input tokens", tokens.size()); + requests.emplace_back( + std::move(tokens), runtimeOpts.maxNewTokens, runtimeOpts.streaming, samplingConfig, outputConfig); + requests.back().setRequestType(tensorrt_llm::executor::RequestType::REQUEST_TYPE_CONTEXT_ONLY); + } + + auto contextRequestIds = contextExecutor.enqueueRequests(requests); + + for (size_t i = 0; i < requests.size(); i++) + { + + TLLM_LOG_INFO("waiting response for Context request id: %lu,", contextRequestIds[i]); + auto response = contextExecutor.awaitResponses(contextRequestIds[i]); + TLLM_LOG_INFO("response received for Context request id: %lu", contextRequestIds[i]); + TLLM_CHECK(response.size() == 1); + TLLM_CHECK(response.back().getResult().contextPhaseParams.has_value()); + requests.at(i).setContextPhaseParams(response.back().getResult().contextPhaseParams.value()); + requests.at(i).setRequestType(tensorrt_llm::executor::RequestType::REQUEST_TYPE_GENERATION_ONLY); + auto genRequestId = generationExecutor.enqueueRequest(requests.at(i)); + genRequestIdToContextRequestId[genRequestId] = contextRequestIds[i]; + + TLLM_LOG_INFO("enqueuing generation request for Context request id: %lu, generation request id: %lu", + contextRequestIds[i], genRequestId); + } + + return genRequestIdToContextRequestId; +} + +std::unordered_map waitForGenResponses(RuntimeOptions const& runtimeOpts, + std::unordered_map const& genRequestIdToContextRequestId, + tle::Executor& generationExecutor) +{ + + // Map that will be used to store output tokens for requests + std::unordered_map outputTokens; + std::vector contextRequestIds{}; + std::vector genRequestIds{}; + for (auto const& [key, value] : genRequestIdToContextRequestId) + { + genRequestIds.push_back(key); + contextRequestIds.push_back(value); + } + for (auto contextRequestId : contextRequestIds) + { + outputTokens[contextRequestId] = tle::BeamTokens(runtimeOpts.beamWidth); + } + + tle::SizeType32 numFinished{0}; + tle::SizeType32 iter{0}; + + // Get the new tokens for each request + while (numFinished < static_cast(genRequestIds.size()) && iter < runtimeOpts.timeoutMs) + { + std::chrono::milliseconds waitTime(1); + // Wait for any response + auto responses = generationExecutor.awaitResponses(waitTime); + + auto insertResponseTokens = [&outputTokens, &genRequestIdToContextRequestId](tle::IdType genRequestId, + tle::SizeType32 seqIdx, tle::VecTokens const& respTokens) + { + TLLM_LOG_INFO("Got %d tokens for seqIdx %d for genRequestId %d,contextRequestId %d", respTokens.size(), + seqIdx, genRequestId, genRequestIdToContextRequestId.at(genRequestId)); + + // Store the output tokens for that request id + auto& outTokens = outputTokens.at(genRequestIdToContextRequestId.at(genRequestId)).at(seqIdx); + outTokens.insert(outTokens.end(), std::make_move_iterator(respTokens.begin()), + std::make_move_iterator(respTokens.end())); + }; + + // Loop over the responses + for (auto const& response : responses) + { + auto genRequestId = response.getRequestId(); + if (!response.hasError()) + { + auto result = response.getResult(); + numFinished += result.isFinal; + if (runtimeOpts.beamWidth > 1) + { + for (tle::SizeType32 beam = 0; beam < runtimeOpts.beamWidth; ++beam) + { + insertResponseTokens(genRequestId, beam, result.outputTokenIds.at(beam)); + } + } + else + { + insertResponseTokens(genRequestId, result.sequenceIndex, result.outputTokenIds.at(0)); + } + if (result.isFinal) + { + TLLM_LOG_INFO("genRequest id %lu ,contextRequestId %lu is completed.", genRequestId, + genRequestIdToContextRequestId.at(genRequestId)); + } + } + else + { + // Allow response with error only if awaitResponse processed a terminated request id + std::string err = "genReqId " + std::to_string(response.getRequestId()) + + " has already been processed and was terminated."; + if (response.getErrorMsg() != err) + { + TLLM_THROW("GenRequest id %lu encountered error: %s", genRequestId, response.getErrorMsg().c_str()); + } + } + } + ++iter; + } + if (iter == runtimeOpts.timeoutMs) + { + TLLM_THROW("Timeout exceeded."); + } + + return outputTokens; +} + +std::vector readInputTokens(std::string const& path) +{ + std::vector data; + std::ifstream file(path); + + if (!file.is_open()) + { + auto const err = std::string{"Failed to open file: "} + path; + TLLM_LOG_ERROR(err); + TLLM_THROW(err); + } + + std::string line; + while (std::getline(file, line)) + { + std::vector row; + std::stringstream ss(line); + std::string token; + + while (std::getline(ss, token, ',')) + { + try + { + row.push_back(std::stoi(token)); + } + catch (std::invalid_argument const& e) + { + TLLM_LOG_ERROR("Invalid argument: %s", e.what()); + } + catch (std::out_of_range const& e) + { + TLLM_LOG_ERROR("Out of range: %s", e.what()); + } + } + + data.push_back(row); + } + + file.close(); + return data; +} + +void writeOutputTokens(std::string const& path, + std::unordered_map& genRequestIdToContextRequestId, + std::unordered_map const& outputTokens, tle::SizeType32 beamWidth) +{ + std::ofstream file(path); + + if (!file.is_open()) + { + TLLM_LOG_ERROR("Failed to open file %s", path.c_str()); + return; + } + std::vector requestIds; + for (auto const& [key, value] : genRequestIdToContextRequestId) + { + requestIds.push_back(value); + } + std::sort(requestIds.begin(), requestIds.end()); + + for (auto requestId : requestIds) + { + auto const& outTokens = outputTokens.at(requestId); + for (tle::SizeType32 beam = 0; beam < beamWidth; ++beam) + { + auto const& beamTokens = outTokens.at(beam); + for (size_t i = 0; i < beamTokens.size(); ++i) + { + file << beamTokens[i]; + if (i < beamTokens.size() - 1) + { + file << ", "; + } + } + file << "\n"; + } + } + + file.close(); +} diff --git a/examples/cpp/executor/executorExampleFastLogits.cpp b/examples/cpp/executor/executorExampleFastLogits.cpp new file mode 100644 index 000000000000..3611a1bb73d7 --- /dev/null +++ b/examples/cpp/executor/executorExampleFastLogits.cpp @@ -0,0 +1,264 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include +#include + +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include "tensorrt_llm/runtime/utils/mpiUtils.h" +#include + +namespace tlc = tensorrt_llm::common; +namespace tle = tensorrt_llm::executor; + +namespace fs = std::filesystem; + +struct RuntimeOptions +{ + std::string trtDraftEnginePath; + std::string trtEnginePath; + bool fastLogits; + tle::SizeType32 numDraftTokens; +}; + +// Utility function to parse input arguments +RuntimeOptions parseArgs(int argc, char* argv[]); + +// Runs a draft request +tle::Result executeDraftRequest(tle::Executor& executor, RuntimeOptions const& runtimeOpts); + +// Runs a target request +tle::Result executeTargetRequest( + tle::Executor& executor, tle::Result const& draftResult, RuntimeOptions const& runtimeOpts); + +// Main +int main(int argc, char* argv[]) +{ + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + auto runtimeOpts = parseArgs(argc, argv); + + // Create the executor for this engine + auto executorConfig = tle::ExecutorConfig(); + + tensorrt_llm::mpi::initialize(tensorrt_llm::mpi::MpiThreadSupport::THREAD_MULTIPLE); + int const myRank = tensorrt_llm::mpi::MpiComm::world().getRank(); + bool const isOrchestrator = (myRank == 0); + + auto kvCacheConfig = tle::KvCacheConfig(true /* enableBlockReuse */); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto orchestratorConfig + = tle::OrchestratorConfig(isOrchestrator, "" /* workerExecutablePath */, nullptr, false /* spawnPrcesses */); + auto parallelConfig = tle::ParallelConfig(tle::CommunicationType::kMPI, tle::CommunicationMode::kORCHESTRATOR, + std::nullopt, std::nullopt, orchestratorConfig); + executorConfig.setParallelConfig(parallelConfig); + + auto specDecConfig = tle::SpeculativeDecodingConfig(runtimeOpts.fastLogits); + executorConfig.setSpecDecConfig(specDecConfig); + + std::unique_ptr draftExecutor; + std::unique_ptr targetExecutor; + + if (isOrchestrator) + { + auto executorConfigDraft = executorConfig; + parallelConfig.setParticipantIds({1}); + executorConfigDraft.setParallelConfig(parallelConfig); + + draftExecutor = std::make_unique( + runtimeOpts.trtDraftEnginePath, tle::ModelType::kDECODER_ONLY, executorConfigDraft); + + parallelConfig.setParticipantIds({2}); + executorConfig.setParallelConfig(parallelConfig); + + targetExecutor + = std::make_unique(runtimeOpts.trtEnginePath, tle::ModelType::kDECODER_ONLY, executorConfig); + } + else if (myRank == 1) // draft model process + { + parallelConfig.setParticipantIds({1}); + parallelConfig.setDeviceIds({0}); + executorConfig.setParallelConfig(parallelConfig); + draftExecutor = std::make_unique( + runtimeOpts.trtDraftEnginePath, tle::ModelType::kDECODER_ONLY, executorConfig); + } + else if (myRank == 2) // target model process + { + parallelConfig.setParticipantIds({2}); + parallelConfig.setDeviceIds({1}); + executorConfig.setParallelConfig(parallelConfig); + targetExecutor + = std::make_unique(runtimeOpts.trtEnginePath, tle::ModelType::kDECODER_ONLY, executorConfig); + ; + } + + // Only orchestrator rank (rank 0) will enter + if (isOrchestrator) + { + auto draftResult = executeDraftRequest(*draftExecutor, runtimeOpts); + + executeTargetRequest(*targetExecutor, draftResult, runtimeOpts); + } + TLLM_LOG_INFO("Exiting."); + return 0; +} + +RuntimeOptions parseArgs(int argc, char* argv[]) +{ + RuntimeOptions runtimeOpts; + + cxxopts::Options options(argv[0], "Example that demonstrates how to use the Executor API"); + options.add_options()("h,help", "Print usage"); + options.add_options()("engine_dir", "Directory that store the engine.", cxxopts::value()); + options.add_options()("draft_engine_dir", "Directory that store the draft engine.", cxxopts::value()); + options.add_options()( + "fast_logits", "Use speculative decoding fast logits feature", cxxopts::value()->default_value("true")); + options.add_options()( + "num_draft_tokens", "Number of draft tokens to use", cxxopts::value()->default_value("5")); + + auto parsedOptions = options.parse(argc, argv); + + // Argument: help + if (parsedOptions.count("help")) + { + TLLM_LOG_ERROR(options.help()); + exit(0); + } + + // Argument: Engine directory + if (!parsedOptions.count("engine_dir")) + { + TLLM_LOG_ERROR(options.help()); + TLLM_LOG_ERROR("Please specify engine directory."); + exit(1); + } + runtimeOpts.trtEnginePath = parsedOptions["engine_dir"].as(); + if (!fs::exists(runtimeOpts.trtEnginePath) || !fs::is_directory(runtimeOpts.trtEnginePath)) + { + TLLM_LOG_ERROR("Engine directory doesn't exist."); + exit(1); + } + + // Argument: Draft engine directory + if (!parsedOptions.count("draft_engine_dir")) + { + TLLM_LOG_ERROR(options.help()); + TLLM_LOG_ERROR("Please specify draft engine directory."); + exit(1); + } + runtimeOpts.trtDraftEnginePath = parsedOptions["draft_engine_dir"].as(); + if (!fs::exists(runtimeOpts.trtDraftEnginePath) || !fs::is_directory(runtimeOpts.trtDraftEnginePath)) + { + TLLM_LOG_ERROR("Draft engine directory doesn't exist."); + exit(1); + } + + runtimeOpts.fastLogits = parsedOptions["fast_logits"].as(); + runtimeOpts.numDraftTokens = parsedOptions["num_draft_tokens"].as(); + + return runtimeOpts; +} + +tle::Result executeDraftRequest(tle::Executor& executor, RuntimeOptions const& runtimeOpts) +{ + tle::OutputConfig outputConfig; + outputConfig.returnGenerationLogits = true; + + // Create the request + tle::SizeType32 maxNewTokens = runtimeOpts.numDraftTokens; + tle::VecTokens inputTokens{1, 2, 3, 4}; + + tle::Request request{std::move(inputTokens), maxNewTokens}; + request.setOutputConfig(outputConfig); + + // Enqueue the request + auto requestId = executor.enqueueRequest(std::move(request)); + + // Wait for the response + auto responses = executor.awaitResponses(requestId); + + if (responses.at(0).hasError()) + { + TLLM_LOG_ERROR(responses.at(0).getErrorMsg()); + exit(1); + } + + auto outputTokens = responses.at(0).getResult().outputTokenIds.at(0); + + TLLM_LOG_INFO("[DRAFT] Output tokens: %s", tlc::vec2str(outputTokens).c_str()); + + return responses.at(0).getResult(); +} + +tle::Result executeTargetRequest( + tle::Executor& executor, tle::Result const& draftResult, RuntimeOptions const& runtimeOpts) +{ + // Create the request + tle::SizeType32 maxNewTokens = runtimeOpts.numDraftTokens + 1; + tle::VecTokens inputTokens{1, 2, 3, 4}; + + tle::Request request{std::move(inputTokens), maxNewTokens}; + + tle::VecTokens const& outputTokenIds = draftResult.outputTokenIds.at(0); + tle::VecTokens draftTokens(outputTokenIds.end() - runtimeOpts.numDraftTokens, outputTokenIds.end()); + TLLM_LOG_INFO("[DRAFT] Draft tokens: %s", tlc::vec2str(draftTokens).c_str()); + + tle::Tensor logitsTensor; + + if (runtimeOpts.fastLogits) + { + auto const& logitsInfo = draftResult.specDecFastLogitsInfo.value(); + logitsTensor = logitsInfo.toTensor(); + } + else + { + auto generationLogits = draftResult.generationLogits.value(); + auto logitsShape = generationLogits.getShape(); + TLLM_CHECK(logitsShape[0] == 1); + logitsTensor = tle::Tensor::cpu(generationLogits.getDataType(), {logitsShape[1], logitsShape[2]}); + std::memcpy(logitsTensor.getData(), generationLogits.getData(), generationLogits.getSizeInBytes()); + } + + tle::ExternalDraftTokensConfig draftTokensConfig( + std::move(draftTokens), logitsTensor, std::nullopt /* acceptance threshold */, runtimeOpts.fastLogits); + request.setExternalDraftTokensConfig(draftTokensConfig); + + // Enqueue the request + auto requestId = executor.enqueueRequest(std::move(request)); + + // Wait for the response + auto responses = executor.awaitResponses(requestId); + + if (responses.at(0).hasError()) + { + TLLM_LOG_ERROR(responses.at(0).getErrorMsg()); + exit(1); + } + + auto outputTokens = responses.at(0).getResult().outputTokenIds.at(0); + + TLLM_LOG_INFO("[TARGET] Output tokens: %s", tlc::vec2str(outputTokens).c_str()); + + return responses.at(0).getResult(); +} diff --git a/examples/cpp/executor/executorExampleKvEvents.cpp b/examples/cpp/executor/executorExampleKvEvents.cpp new file mode 100644 index 000000000000..ea1923294382 --- /dev/null +++ b/examples/cpp/executor/executorExampleKvEvents.cpp @@ -0,0 +1,341 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include +#include +#include +#include +#include +#include + +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" +#include + +namespace tlc = tensorrt_llm::common; +namespace tle = tensorrt_llm::executor; +namespace fs = std::filesystem; + +struct RuntimeOptions +{ + std::string trtEnginePath; + tle::SizeType32 numSysPrompts; + + tle::SizeType32 sysPromptTokens; + tle::SizeType32 contextTokens; + + tle::SizeType32 maxTokensMean; + tle::SizeType32 maxTokensStddev; + + tle::SizeType32 numRequests; + + size_t hostCacheSize; + size_t maxTokensInPagedKvCache; +}; + +struct KVCacheBlock +{ + KVCacheBlock(size_t hash, int cacheLevel, int priority, std::optional loraId = std::nullopt, + std::shared_ptr prevBlock = nullptr, std::optional cacheSalt = std::nullopt); + + size_t hash; + int cacheLevel; + int priority; + + std::optional loraId; + std::optional cacheSalt; + + std::shared_ptr prevBlock; + std::unordered_map> nextBlocks; +}; + +class RadixTree +{ +public: + explicit RadixTree(tle::Executor& executor); + // Check the executor for new events. + void pollEvents(); + +private: + std::shared_ptr mCacheEventManager; + // The root block of the radix tree + std::shared_ptr root; + // A table mapping block hashes to their pointers + std::unordered_map> blockTable; + // Event counter + size_t eventCounter; +}; + +// Utility function to parse input arguments +RuntimeOptions parseArgs(int argc, char* argv[]); + +// Create a tle::Request +tle::Request makeRequest(int sysPromptTokens, int contextTokens, std::uniform_int_distribution sysPromptSelector, + std::normal_distribution maxNumTokensSelector); + +std::default_random_engine gen; + +int main(int argc, char* argv[]) +{ + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + auto runtimeOpts = parseArgs(argc, argv); + + // Create the executor for this engine + auto executorConfig = tle::ExecutorConfig(1); // Beam width 1 is required for cache block reuse + auto kvCacheConfig = tle::KvCacheConfig(true, + runtimeOpts.maxTokensInPagedKvCache ? std::optional(runtimeOpts.maxTokensInPagedKvCache) + : std::nullopt); // Enable cache block reuse + kvCacheConfig.setHostCacheSize(runtimeOpts.hostCacheSize); + kvCacheConfig.setEventBufferMaxSize(32768); + executorConfig.setKvCacheConfig(kvCacheConfig); + + auto executor = tle::Executor(runtimeOpts.trtEnginePath, tle::ModelType::kDECODER_ONLY, executorConfig); + + auto radixTree = RadixTree(executor); + + auto activeRequests = runtimeOpts.numRequests; + + std::uniform_int_distribution sysPromptSelector( + 1, runtimeOpts.numSysPrompts); // Select a system prompt between 1 and `runtimeOpts.numSysPrompts` + std::normal_distribution maxNumTokensSelector(runtimeOpts.maxTokensMean, runtimeOpts.maxTokensStddev); + + // Create and enqueue the requests + for (int i = 0; i < runtimeOpts.numRequests; i++) + { + std::ignore = executor.enqueueRequest(makeRequest( + runtimeOpts.sysPromptTokens, runtimeOpts.contextTokens, sysPromptSelector, maxNumTokensSelector)); + } + + while (activeRequests > 0) + { + auto responses = executor.awaitResponses(std::chrono::milliseconds(20)); + for (auto const& response : responses) + { + if (response.getResult().isFinal) + activeRequests--; + } + // Only call pollEvents once every 20ms. Events are only added to the queue once per iteration, so no need to + // poll faster than this. + radixTree.pollEvents(); + } + + return 0; +} + +RuntimeOptions parseArgs(int argc, char* argv[]) +{ + RuntimeOptions runtimeOpts; + + cxxopts::Options options(argv[0], "Example that demonstrates how to use the ExecutorKVCacheManager API"); + options.add_options()("h,help", "Print usage"); + options.add_options()("engine_dir", "Directory that store the engines.", cxxopts::value()); + options.add_options()("num_sys_prompts", "Amount of unique simulated system prompts to use", + cxxopts::value()->default_value("10")); + options.add_options()( + "sys_prompt_tokens", "Size of the simulated system prompts", cxxopts::value()->default_value("256")); + options.add_options()("context_tokens", "Amount of varying context tokens coming after the system prompts", + cxxopts::value()->default_value("128")); + options.add_options()( + "max_tokens_mean", "Mean number of max output tokens", cxxopts::value()->default_value("128")); + options.add_options()( + "max_tokens_stddev", "Standard deviation of max output tokens", cxxopts::value()->default_value("32")); + options.add_options()( + "num_requests", "Amount of requests to send to the engine", cxxopts::value()->default_value("100")); + options.add_options()("host_cache_size", "Size of the KV Cache in host memory in bytes", + cxxopts::value()->default_value("0")); + options.add_options()("max_tokens_in_paged_kv_cache", "Amount of tokens in the kv cache", + cxxopts::value()->default_value("0")); + + auto parsedOptions = options.parse(argc, argv); + + // Argument: help + if (parsedOptions.count("help")) + { + TLLM_LOG_ERROR(options.help()); + exit(0); + } + + // Argument: Engine directory + if (!parsedOptions.count("engine_dir")) + { + TLLM_LOG_ERROR(options.help()); + TLLM_LOG_ERROR("Please specify engine directory."); + exit(1); + } + runtimeOpts.trtEnginePath = parsedOptions["engine_dir"].as(); + if (!fs::exists(runtimeOpts.trtEnginePath) || !fs::is_directory(runtimeOpts.trtEnginePath)) + { + TLLM_LOG_ERROR("Engine directory doesn't exist."); + exit(1); + } + + runtimeOpts.numSysPrompts = parsedOptions["num_sys_prompts"].as(); + runtimeOpts.sysPromptTokens = parsedOptions["sys_prompt_tokens"].as(); + runtimeOpts.contextTokens = parsedOptions["context_tokens"].as(); + runtimeOpts.maxTokensMean = parsedOptions["max_tokens_mean"].as(); + runtimeOpts.maxTokensStddev = parsedOptions["max_tokens_stddev"].as(); + runtimeOpts.numRequests = parsedOptions["num_requests"].as(); + runtimeOpts.hostCacheSize = parsedOptions["host_cache_size"].as(); + runtimeOpts.maxTokensInPagedKvCache = parsedOptions["max_tokens_in_paged_kv_cache"].as(); + + return runtimeOpts; +} + +KVCacheBlock::KVCacheBlock(size_t hash, int cacheLevel, int priority, std::optional loraId, + std::shared_ptr prevBlock, std::optional cacheSalt) + : hash{hash} + , cacheLevel{cacheLevel} + , priority{priority} + , loraId{loraId} + , cacheSalt{std::move(cacheSalt)} + , prevBlock{prevBlock} + , nextBlocks{} +{ +} + +RadixTree::RadixTree(tle::Executor& executor) + : mCacheEventManager(*executor.getKVCacheEventManager()) + , eventCounter{1} +{ + // Use id=-1 for the root block. Doesn't matter what exact id is used, just that it is unique. + root = std::make_shared(-1, -1, -1); + blockTable[-1] = root; + + // Wait for the `CREATED` event to be emitted. + while (true) + { + auto events = mCacheEventManager->getLatestEvents(); + if (events.size() == 1) + { + auto const& eventData = std::get(events.front().data); + TLLM_LOG_INFO("Event ID %d: KV Cache Manager initialized with blocks per level of: %s", + events.front().eventId, tlc::vec2str(eventData.numBlocksPerCacheLevel).c_str()); + break; + } + } +}; + +void RadixTree::pollEvents() +{ + auto events = mCacheEventManager->getLatestEvents(std::chrono::milliseconds(20)); + for (tle::KVCacheEvent const& event : events) + { + TLLM_CHECK(event.eventId == eventCounter++); + if (std::holds_alternative(event.data)) + { + // Blocks have been stored into the radix tree + auto const& eventData = std::get(event.data); + auto prevBlock = blockTable[eventData.parentHash.value_or(-1)]; + + // This block should be in the tree + TLLM_CHECK(blockTable.find(prevBlock->hash) != blockTable.end()); + + for (auto& block : eventData.blocks) + { + + TLLM_LOG_INFO("Event ID %d: Block %04x was inserted into the radix tree with parent %04x.", + event.eventId, block.blockHash, prevBlock->hash); + + // This block shouldn't already exist in the tree, and should have tokens associated with it + TLLM_CHECK(blockTable.find(block.blockHash) == blockTable.end()); + TLLM_CHECK(block.tokens.size() > 0); + + auto thisBlock = std::make_shared( + block.blockHash, block.cacheLevel, block.priority, block.loraId, prevBlock, block.cacheSalt); + + blockTable[block.blockHash] = thisBlock; + // Link the parent to the new block + prevBlock->nextBlocks[block.blockHash] = thisBlock; + + prevBlock = thisBlock; + } + } + else if (std::holds_alternative(event.data)) + { + auto const& eventData = std::get(event.data); + + for (auto const& hash : eventData.blockHashes) + { + + TLLM_LOG_INFO("Event ID %d: Block %04x was removed from the radix tree.", event.eventId, hash); + + // This block should exist in the tree + TLLM_CHECK(blockTable.find(hash) != blockTable.end()); + + auto& block = blockTable[hash]; + + // Check that the block has no children, and that the parent has the block listed as a child + TLLM_CHECK(block->nextBlocks.size() == 0); + TLLM_CHECK(block->prevBlock->nextBlocks.find(block->hash) != block->prevBlock->nextBlocks.end()); + + // Remove the block from it's parent, and remove the entry in the block table + block->prevBlock->nextBlocks.erase(block->hash); + blockTable.erase(hash); + } + } + else if (std::holds_alternative(event.data)) + { + auto const& eventData = std::get(event.data); + + if (eventData.priority.has_value()) + { + // The block priority was updated + TLLM_LOG_INFO("Event ID %d: Block %04x priority was changed from %d to %d", event.eventId, + eventData.blockHash, eventData.priority->oldValue, eventData.priority->newValue); + + TLLM_CHECK(blockTable[eventData.blockHash]->priority == eventData.priority->oldValue); + blockTable[eventData.blockHash]->priority = eventData.priority->newValue; + } + + if (eventData.cacheLevel.has_value()) + { + // The block cache level was updated + TLLM_LOG_INFO("Event ID %d: Block %04x cache level was changed from %d to %d", event.eventId, + eventData.blockHash, eventData.cacheLevel->oldValue, eventData.cacheLevel->newValue); + + TLLM_CHECK(blockTable[eventData.blockHash]->cacheLevel == eventData.cacheLevel->oldValue); + blockTable[eventData.blockHash]->cacheLevel = eventData.cacheLevel->newValue; + } + } + else + { + TLLM_LOG_ERROR("Unsupported event type. This shouldn't happen!"); + } + } +} + +tle::Request makeRequest(int sysPromptTokens, int contextTokens, std::uniform_int_distribution sysPromptSelector, + std::normal_distribution maxNumTokensSelector) +{ + int sysPromptVersion = sysPromptSelector(gen); + tle::VecTokens inputTokens; + + // Add `sysPromptTokens` tokens. Add the version to the token ids to create a unique system prompt + for (int i = 0; i < sysPromptTokens; i++) + { + inputTokens.emplace_back(sysPromptVersion + i); + } + // Add random context tokens + for (int i = 0; i < contextTokens; i++) + { + inputTokens.emplace_back(rand() % 1000); + } + + return tle::Request(inputTokens, maxNumTokensSelector(gen)); +} diff --git a/examples/cpp/executor/executorExampleLogitsProcessor.cpp b/examples/cpp/executor/executorExampleLogitsProcessor.cpp new file mode 100644 index 000000000000..0913b77b1775 --- /dev/null +++ b/examples/cpp/executor/executorExampleLogitsProcessor.cpp @@ -0,0 +1,91 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include + +#include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" +#include "tensorrt_llm/executor/executor.h" +#include "tensorrt_llm/plugins/api/tllmPlugin.h" + +namespace tlc = tensorrt_llm::common; +namespace tle = tensorrt_llm::executor; + +int main(int argc, char* argv[]) +{ + // Register the TRT-LLM plugins + initTrtLlmPlugins(); + + if (argc != 2) + { + TLLM_LOG_ERROR("Usage: %s ", argv[0]); + return 1; + } + + int constexpr sentinels[] = {42, 29}; + int step = 0; + + auto logitsPostProcessorFn + = [&step, &sentinels](tle::IdType reqId, tle::Tensor& logits, tle::BeamTokens const& tokens, + tle::StreamPtr const& streamPtr, std::optional clientId) + { + auto logitsDataType = logits.getDataType(); + auto logitsCpu = tensorrt_llm::executor::Tensor::cpu(logitsDataType, logits.getShape()); + auto* dataPtr = logitsCpu.getData(); + auto* dataPtrFloat = static_cast(dataPtr); + for (size_t i = 0; i < logitsCpu.getSize(); ++i) + { + dataPtrFloat[i] = -1.0e20; + } + dataPtrFloat[sentinels[step]] = 0.0f; + + logits.setFrom(logitsCpu, streamPtr); + step = (1 - step); + }; + + std::string logitsPostProcessorName = "MyLogitsPP"; + + // Create the executor for this engine + tle::SizeType32 beamWidth = 1; + auto executorConfig = tle::ExecutorConfig(beamWidth); + + auto logitsProcConfig = tle::LogitsPostProcessorConfig(); + logitsProcConfig.setProcessorMap(std::unordered_map{ + {logitsPostProcessorName, logitsPostProcessorFn}}); + executorConfig.setLogitsPostProcessorConfig(logitsProcConfig); + + auto trtEnginePath = argv[1]; + auto executor = tle::Executor(trtEnginePath, tle::ModelType::kDECODER_ONLY, executorConfig); + + // Create the request + tle::SizeType32 maxNewTokens = 5; + tle::VecTokens inputTokens{1, 2, 3, 4}; + auto request = tle::Request(inputTokens, maxNewTokens); + request.setLogitsPostProcessorName(logitsPostProcessorName); + + // Enqueue the request + auto requestId = executor.enqueueRequest(std::move(request)); + + // Wait for the response + auto responses = executor.awaitResponses(requestId); + + // Get outputTokens + auto outputTokens = responses.at(0).getResult().outputTokenIds.at(beamWidth - 1); + + TLLM_LOG_INFO("Output tokens: %s", tlc::vec2str(outputTokens).c_str()); + + return 0; +} diff --git a/examples/cpp/executor/inputTokens.csv b/examples/cpp/executor/inputTokens.csv new file mode 100644 index 000000000000..4cb3974a91b5 --- /dev/null +++ b/examples/cpp/executor/inputTokens.csv @@ -0,0 +1,3 @@ +1, 2, 3, 4, 5, 6 +1, 2, 3, 4 +1, 2, 3, 4, 5, 6, 7, 8, 9, 10 diff --git a/examples/cpp_library/CMakeLists.txt b/examples/cpp_library/CMakeLists.txt new file mode 100644 index 000000000000..c60ff48d4cde --- /dev/null +++ b/examples/cpp_library/CMakeLists.txt @@ -0,0 +1,55 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. cmake needs this line +cmake_minimum_required(VERSION 3.1) +# cmake_minimum_required(VERSION 2.8) + +# Enable C++11 +set(CMAKE_CXX_STANDARD 14) +set(CMAKE_CXX_STANDARD_REQUIRED TRUE) + +# Define project name +set(TARGET_NAME trt_llm_plugins_cpp_load_example) +project(${TARGET_NAME}) + +set(CMAKE_VERBOSE_MAKEFILE 1) + +# Compile options +set(CMAKE_C_FLAGS "-Wall -pthread ") +set(CMAKE_C_FLAGS_DEBUG "-g -O0") +set(CMAKE_C_FLAGS_RELEASE "-O2") +set(CMAKE_CXX_FLAGS "${CMAKE_C_FLAGS} -lstdc++") +set(CMAKE_CXX_FLAGS_DEBUG ${CMAKE_C_FLAGS_DEBUG}) +set(CMAKE_CXX_FLAGS_RELEASE ${CMAKE_C_FLAGS_RELEASE}) + +set(CMAKE_BUILD_TYPE release) +# set(CMAKE_BUILD_TYPE debug) + +find_package(CUDA REQUIRED) +message(STATUS "CUDA library status:") +message(STATUS " config: ${CUDA_DIR}") +message(STATUS " version: ${CUDA_VERSION}") +message(STATUS " libraries: ${CUDA_LIBRARIES}") +message(STATUS " include path: ${CUDA_INCLUDE_DIRS}") + +# Declare the executable target built from your sources +add_executable(${TARGET_NAME} main.cpp) + +# Link your application with CUDA libraries +target_link_libraries(${TARGET_NAME} LINK_PRIVATE ${CUDA_LIBRARIES}) +target_link_libraries(${TARGET_NAME} LINK_PRIVATE cudnn) +target_link_libraries(${TARGET_NAME} LINK_PRIVATE nvinfer) +target_link_libraries(${TARGET_NAME} LINK_PRIVATE nvinfer_plugin_tensorrt_llm) + +target_include_directories(${TARGET_NAME} PUBLIC /usr/local/cuda/include) diff --git a/examples/cpp_library/build.sh b/examples/cpp_library/build.sh new file mode 100755 index 000000000000..a384e5cad4a0 --- /dev/null +++ b/examples/cpp_library/build.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash + +BUILD_DIR="build" +SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) + +rm -rf ${BUILD_DIR} && mkdir -p ${BUILD_DIR} + +pushd ${BUILD_DIR} + +cmake \ + -DCMAKE_BUILD_TYPE=Release \ + .. + +make -j"$(grep -c ^processor /proc/cpuinfo)" + +export LD_LIBRARY_PATH="${SCRIPT_DIR}:${LD_LIBRARY_PATH}" + +# Test Lib +echo +echo "--------------------------------------------------------------------" +./trt_llm_plugins_cpp_load_example +echo "--------------------------------------------------------------------" +echo + +popd diff --git a/examples/cpp_library/main.cpp b/examples/cpp_library/main.cpp new file mode 100644 index 000000000000..7613a75a140c --- /dev/null +++ b/examples/cpp_library/main.cpp @@ -0,0 +1,79 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include +#include +#include + +#include "tensorrt_llm_libutils.h" + +int main(int argc, char* argv[]) +{ + class TRTLogger : public nvinfer1::ILogger + { + public: + void log(nvinfer1::ILogger::Severity severity, char const* msg) noexcept override + { + if (severity <= nvinfer1::ILogger::Severity::kERROR) + std::cerr << "[TensorRT LLM ERR]: " << msg << std::endl; + else if (severity == nvinfer1::ILogger::Severity::kWARNING) + std::cerr << "[TensorRT LLM WARNING]: " << msg << std::endl; + else + std::cout << "[TensorRT LLM LOG]: " << msg << std::endl; + } + }; + + TRTLogger* trtLogger = new TRTLogger(); + + std::string libname = "libtensorrt_llm_plugin.so"; + + /* =============== initLibNvInferPlugins =============== */ + + typedef bool (*initLibNvInferPlugins_sig)(void*, void const*); + + auto initLibNvInferPlugins = getTrtLLMFunction( + /*libFileSoName=*/libname, + /*symbol=*/"initLibNvInferPlugins"); + + std::cout << std::endl; + + std::string libNamespace = "tensorrt_llm"; + char const* libNamespace_cstr = libNamespace.data(); + + bool status1 = initLibNvInferPlugins(trtLogger, libNamespace_cstr); + std::cout << "Success Status: " << status1 << std::endl << std::endl; + + bool status2 = initLibNvInferPlugins(trtLogger, libNamespace_cstr); + std::cout << "Success Status: " << status2 << std::endl; + + /* =============== getInferLibVersion =============== */ + + std::cout << std::endl; + std::cout << "--------------------------------------------------------------------" << std::endl; + + typedef int32_t (*getInferLibVersion_sig)(); + + auto getInferLibVersion = getTrtLLMFunction( + /*libFileSoName=*/libname, + /*symbol=*/"getInferLibVersion"); + + std::cout << std::endl; + + int32_t version = getInferLibVersion(); + std::cout << "Version: " << version << std::endl; + + return 0; +} diff --git a/examples/cpp_library/tensorrt_llm_libutils.h b/examples/cpp_library/tensorrt_llm_libutils.h new file mode 100644 index 000000000000..aa60444eefa3 --- /dev/null +++ b/examples/cpp_library/tensorrt_llm_libutils.h @@ -0,0 +1,62 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#if !defined(_WIN32) +#include +#endif // !defined(_WIN32) +#include +#include +#include + +#include "NvInfer.h" + +template +tSymbolSignature getTrtLLMFunction(std::string libFileSoName, std::string symbol) +{ +#if !defined(_WIN32) + std::cout << "Trying to load " << libFileSoName << " ..." << std::endl; + + // 1. Defining a handle to the library + void* handle = dlopen(libFileSoName.c_str(), RTLD_LAZY | RTLD_GLOBAL); + + // 2. Check for errors + char const* dl_error1 = dlerror(); + if (!handle) + { + throw std::runtime_error("Cannot open library: " + std::string(dl_error1)); + } + + // 3. Load actual queried `symbol` + std::cout << "Loading symbol `" << symbol << "` ..." << std::endl; + + tSymbolSignature symbolFctn = nullptr; + *(void**) (&symbolFctn) = dlsym(handle, symbol.c_str()); + + // 4. Check for errors + char const* dl_error2 = dlerror(); + if (dl_error2) + { + dlclose(handle); + throw std::runtime_error("Cannot load symbol '" + symbol + "': " + std::string(dl_error2)); + } + + return symbolFctn; +#else // on windows + throw std::runtime_error( + "`tSymbolSignature getTrtLLMFunction(std::string, std::string)` is not implemented on Windows."); + return nullptr; +#endif // !defined(_WIN32) +} diff --git a/examples/disaggregated/slurm/benchmark/disaggr_torch.slurm b/examples/disaggregated/slurm/benchmark/disaggr_torch.slurm index d73430ba5820..275016c255dd 100644 --- a/examples/disaggregated/slurm/benchmark/disaggr_torch.slurm +++ b/examples/disaggregated/slurm/benchmark/disaggr_torch.slurm @@ -101,7 +101,7 @@ elif [ -d "${trtllm_repo}" ]; then if [ "${build_wheel}" = "true" ]; then echo "Building TensorRT-LLM wheel on one node..." - build_command="python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt --use_ccache --clean" + build_command="python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt --benchmarks --use_ccache --clean" if [ -n "${cuda_architectures:-}" ]; then build_command="${build_command} --cuda_architectures \"${cuda_architectures}\"" fi diff --git a/examples/disaggregated/slurm/benchmark/disaggr_torch_dwdp.slurm b/examples/disaggregated/slurm/benchmark/disaggr_torch_dwdp.slurm index 07015e6473a0..8835399d6c50 100644 --- a/examples/disaggregated/slurm/benchmark/disaggr_torch_dwdp.slurm +++ b/examples/disaggregated/slurm/benchmark/disaggr_torch_dwdp.slurm @@ -101,7 +101,7 @@ elif [ -d "${trtllm_repo}" ]; then if [ "${build_wheel}" = "true" ]; then echo "Building TensorRT-LLM wheel on one node..." - build_command="python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt --use_ccache --clean" + build_command="python3 ./scripts/build_wheel.py --trt_root /usr/local/tensorrt --benchmarks --use_ccache --clean" if [ -n "${cuda_architectures:-}" ]; then build_command="${build_command} --cuda_architectures \"${cuda_architectures}\"" fi diff --git a/examples/disaggregated/slurm/cache_transceiver_test/README.md b/examples/disaggregated/slurm/cache_transceiver_test/README.md index e99c57dc4fe3..14619e4d387c 100644 --- a/examples/disaggregated/slurm/cache_transceiver_test/README.md +++ b/examples/disaggregated/slurm/cache_transceiver_test/README.md @@ -64,7 +64,7 @@ resolved_config.json # the validated config the job ran with logs/ctt-.log # batch-level log (stdout+stderr merged) logs/install.log # TensorRT-LLM install log logs/sweep__rank*.log # per-rank logs: transfer START/DONE+verify, UCX_PROTO_INFO -csv//ctx|gen/_*_{send,recv}.csv # C++ transceiver timing (Bandwidth(Gbps)); renamed to _*_{send,recv}__c.csv per combination +csv//ctx|gen/rank_*_{send,recv}.csv # C++ transceiver timing (Bandwidth(Gbps)) csv//ctx/py_*_*.csv # Python transceiver perf log (throughput_mbs) status/sweep_.jsonl # PASS / MISMATCH / TRANSFER_ERROR / TIMEOUT results.json # full results, grouped per combination (longest req_len) @@ -96,7 +96,7 @@ deliverable for tuning your cluster. | Transceiver | Env enabling timing | File | Column (native) | |---|---|---|---| -| C++ (UCX/NIXL) | `TRTLLM_KVCACHE_TIME_OUTPUT_PATH` (set by the driver) | `_*_recv.csv` (renamed `_*_recv__c.csv` per combination) | `Bandwidth(Gbps)` | +| C++ (UCX/NIXL) | `TRTLLM_KVCACHE_TIME_OUTPUT_PATH` (set by the driver) | `rank_*_recv.csv` | `Bandwidth(Gbps)` | | Python (NIXL) | `TLLM_ENABLE_CACHE_TRANSFER_PERF_INFO=1`, `TLLM_KV_TRANSFER_PERF_LOG_FILE` (set by the driver) | `py_*_*.csv` | `throughput_mbs` (MiB/s) | `report.py` normalizes both to **per-GPU GB/s** (bytes, ÷1e9): C++ diff --git a/examples/disaggregated/slurm/cache_transceiver_test/report.py b/examples/disaggregated/slurm/cache_transceiver_test/report.py index 00ecab6b4e49..b03a9a0d07bc 100644 --- a/examples/disaggregated/slurm/cache_transceiver_test/report.py +++ b/examples/disaggregated/slurm/cache_transceiver_test/report.py @@ -184,7 +184,7 @@ def _mean(values): def _parse_cpp_recv_csvs(csv_dir): - """Return {rid: [per_rank_GBps, ...]} from C++ *_recv.csv files. + """Return {rid: [per_rank_GBps, ...]} from C++ rank_*_recv.csv files. Each rank writes one row per request with a repeating Bandwidth(Gbps) column per transmission; we take the mean transmission bandwidth as that rank's @@ -194,10 +194,9 @@ def _parse_cpp_recv_csvs(csv_dir): per-rank rates with unequal durations would overstate the real throughput. """ per_rid = {} # rid -> list[per-rank mean bw] - # C++ writes "__recv.csv" (instanceId is a runtime UUID); - # the driver renames each combination's *_recv.csv to *_recv__cl
  • .csv + # The driver renames each combination's rank_*_recv.csv to rank_*_recv__cl
  • .csv # (and the un-renamed name may exist for the last iteration). Match both. - for path in glob.glob(os.path.join(csv_dir, "*_recv*.csv")): + for path in glob.glob(os.path.join(csv_dir, "rank_*_recv*.csv")): with open(path) as f: reader = csv.reader(f) header = next(reader, None) @@ -254,14 +253,10 @@ def _parse_python_csvs(csv_dir): return per_rid # {rid: [per-rank GB/s]} -# Protocol selection in the last column of a UCX_PROTO_INFO=y table row, e.g. +# Transport in the last column of a UCX_PROTO_INFO=y table row, e.g. # "... | rendezvous zero-copy read from remote | cuda_ipc/cuda |" -# Multi-lane configs may contain weighted entries such as -# `50% on rc_mlx5/mlx5_0:1 and 50% on rc_mlx5/mlx5_1:1`. -_PROTO_LAST_COL = re.compile(r"\|\s*([^|]+?)\s*\|\s*$") -_TRANSPORT_TOKEN = re.compile( - rf"(? _TransportAcc saw_marker = False @@ -404,7 +387,7 @@ def _parse_proto_info_by_case(log_glob, kv_only=False): accs[cur].feed(line) if not saw_marker: return {} - return {ci: acc.ranked(kv_only=kv_only) for ci, acc in accs.items()} + return {ci: acc.ranked() for ci, acc in accs.items()} # Driver per-request log timestamp, e.g. "[06/03/2026-06:59:17] ... rid=12 ... @@ -483,13 +466,13 @@ def _ci_for(ts): if not in_kv: continue mt = _UCX_TS.search(line) - transports = _proto_row_transports(line) - if not mt or not transports: + mr = _PROTO_LAST_COL.search(line) + if not mt or not mr or mr.group(1) not in TRANSPORT_TOKENS: continue ci = _ci_for(float(mt.group(1))) if ci is None: continue - kv.setdefault(ci, set()).update(transports) + kv.setdefault(ci, set()).add(mr.group(1)) if "software emulation" in line: sw[ci] = True out = {} @@ -528,7 +511,7 @@ def _read_status(work_dir, sweep_idx): return merged -def aggregate(cfg, out_path, require_kv_transport=False): +def aggregate(cfg, out_path): work_dir = cfg["environment"]["work_dir"] cases = build_cases(cfg) req_lens = cfg["test_matrix"]["request_lengths"] @@ -554,8 +537,8 @@ def aggregate(cfg, out_path, require_kv_transport=False): # then sweep-level, for logs lacking per-request timestamps. per_case_transport = _parse_proto_info_by_case_ts(log_glob, _case_start_times(log_glob)) if not per_case_transport: - per_case_transport = _parse_proto_info_by_case(log_glob, kv_only=require_kv_transport) - sweep_transport = _parse_proto_info(log_glob, kv_only=require_kv_transport) + per_case_transport = _parse_proto_info_by_case(log_glob) + sweep_transport = _parse_proto_info(log_glob) status_map = _read_status(work_dir, sweep_idx) cpp_bw = _parse_cpp_recv_csvs(gen_csv_dir) @@ -719,11 +702,6 @@ def main(): g.add_argument("--aggregate", action="store_true") ap.add_argument("--sweep", type=int, default=0) ap.add_argument("--out", type=str, default=None) - ap.add_argument( - "--require-kv-transport", - action="store_true", - help="accept transport only from CUDA KV-data protocol tables", - ) args = ap.parse_args() cfg = load_config(args.config) @@ -733,7 +711,7 @@ def main(): emit_ucx_env(cfg, args.sweep) elif args.aggregate: out = args.out or os.path.join(cfg["environment"]["work_dir"], "results.json") - aggregate(cfg, out, require_kv_transport=args.require_kv_transport) + aggregate(cfg, out) print(f"\nWrote {out}", file=sys.stderr) diff --git a/examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py b/examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py index 138a43c96063..6a6775d5c08e 100644 --- a/examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py +++ b/examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py @@ -24,7 +24,7 @@ deterministic, rank-specific pattern, sends it, and the gen side verifies the received blocks regenerate to the same pattern. Bandwidth is emitted by the transceivers themselves into per-rank CSVs (parsed later by report.py): - C++ -> TRTLLM_KVCACHE_TIME_OUTPUT_PATH (_*_send.csv / _*_recv.csv) + C++ -> TRTLLM_KVCACHE_TIME_OUTPUT_PATH (rank_*_send.csv / rank_*_recv.csv) Py -> TLLM_KV_TRANSFER_PERF_LOG_FILE (py_*_*.csv, throughput_mbs) This driver mirrors the single-process test (tests/unittest/others/ @@ -437,19 +437,15 @@ def _preserve_cpp_csvs(csv_dir, ci, rank): request lengths of a combination and appends a row per request, so we move the whole combination's output aside (rid encodes req_len). - C++ names files "__.csv" (instanceId is a runtime - UUID). Each rank touches ONLY files carrying its own "__.csv" - suffix: all ranks share `csv_dir`, so matching a broader pattern would race - -- multiple ranks renaming the same file, leaving some with - FileNotFoundError, crashing those ranks and deadlocking the rest on the next - case's collective KVCacheManager allreduce. + Each rank touches ONLY its own files: all ranks share `csv_dir`, so a glob + over `rank_*` would race -- multiple ranks renaming the same file, leaving + some with FileNotFoundError, crashing those ranks and deadlocking the rest on + the next case's collective KVCacheManager allreduce. """ for tag in ("send", "recv"): - suffix = f"_{rank}_{tag}.csv" - for name in os.listdir(csv_dir): - if name.endswith(suffix) and "__c" not in name: - base = name[: -len(".csv")] - os.replace(os.path.join(csv_dir, name), os.path.join(csv_dir, f"{base}__c{ci}.csv")) + path = os.path.join(csv_dir, f"rank_{rank}_{tag}.csv") + if os.path.exists(path): + os.replace(path, os.path.join(csv_dir, f"rank_{rank}_{tag}__c{ci}.csv")) def main(): diff --git a/examples/dora/README.md b/examples/dora/README.md new file mode 100644 index 000000000000..21f70768a257 --- /dev/null +++ b/examples/dora/README.md @@ -0,0 +1,65 @@ +# DoRA + +This document shows how to run a model using DoRA adapters. +DoRA is a PEFT strategy extending LoRA. It is fully supported in the Huggingface `peft` library. For a more detailed description please refer to the DoRA [paper](https://arxiv.org/abs/2402.09353) or official [repo](https://github.com/NVlabs/DoRA). + +## Support Matrix + * FP16/BF16 (over arbitrary precision of the base model). + * Supports adapters from Huggingface `peft` or from the official NVlabs [checkpoints](https://huggingface.co/sliuau/DoRA-weights). + * Multiple adapters (+ mixed LoRA/DoRA setups). + * inflight loading of new adapters to a preloaded base model. + * C++ and python runtime. + * Tensor parallelism and Pipeline parallelism. + +## Usage +Using DoRA is almost exactly the same as using LoRA in TRTLLM, with an additional preprocessing step. +While the official DoRA paper describes the magnitude normalization as part of the execution flow, it can be performed once beforehand to boost inference performance. + +Start by obtaining a local copy of your desired DoRA adapter **and** your base model. We'll use the official NVlabs checkpoint for LLaMA3-8B as an example: + +``` bash +git clone https://huggingface.co/sliuau/DoRA-weights +git clone https://huggingface.co/meta-llama/Meta-Llama-3-8B +``` + +Next, use the [normalize_weights.py](./normalize_weights.py) script to normalize the DoRA magnitude vectors in the adapter checkpoint. +The script requires access to both the local adapter weights and the local base model weights: + +``` bash +export NORMALIZED_DORA_ADAPTER=path/to/normalized/adapter/ckpt + +python ./normalize_weights.py -i DoRA-weights/llama_dora_commonsense_checkpoints/LLama3-8B/dora_r32 -b Meta-Llama-3-8B -o $NORMALIZED_DORA_ADAPTER +``` + +The script will create a new adapter checkpoint, with normalized DoRA vectors, in the provided path. + +Now we may convert our Llama checkpoint and build our TRT engine as described in the Llama [examples](../models/core/llama/README.md). When doing so, ensure you pass `--dora_plugin=enable` to the `trtllm-build` command, as well as enabling the lora plugin: + +``` bash +export CHECKPOINT_DIR=path/to/trtllm/ckpt +export ENGIRE_DIR=path/to/trtllm/engine + +python ../models/core/llama/convert_checkpoint.py --model_dir Meta-Llama-3-8B \ + --output_dir $CHECKPOINT_DIR \ + --dtype float16 + +trtllm-build --checkpoint_dir $CHECKPOINT_DIR \ + --output_dir $ENGINE_DIR \ + --gemm_plugin=auto \ + --lora_plugin=auto \ + --dora_plugin=enable \ + --lora_dir $NORMALIZED_DORA_ADAPTER +``` + +If you wish, you may provide additional LoRA / DoRA adapters to `trtllm-build`. + +**NOTE**: if you omit `--dora_plugin=enable`, you will not receive any warning even if you provide a DoRA adapter to `--lora_dir`. In such a case the DoRA magnitudes will simply be ignored during inference and you may receive wrong output. + +Proceed to execute the engine as you would a normal LoRA engine: + +``` bash +python ../run.py --engine_dir $ENGINE_DIR --tokenizer_dir Meta-Llama-3-8B --lora_task_uids 0 --max_output_len 32 --input_text ... +``` + +## Usage with Triton Server +Using DoRA over Triton is the same as using LoRA, but before using [hf_lora_convert.py](../hf_lora_convert.py), make sure you call [normalize_weights.py](./normalize_weights.py) and use the resulting normalized adapter. diff --git a/examples/dora/normalize_weights.py b/examples/dora/normalize_weights.py new file mode 100755 index 000000000000..40d5ae172431 --- /dev/null +++ b/examples/dora/normalize_weights.py @@ -0,0 +1,289 @@ +#! /usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +This script applies preprocessing to the DoRA magnitude vector to speed up inference. +DoRA applies columnwise normalization and scaling to the LoRA output. +By applying the normalization to the scaling vector, we can skip calculating the normalization vector at inference time. +""" +import abc +import enum +import json +from pathlib import Path + +import numpy as np +import safetensors.torch as st +import torch + +StateDict = dict[str, torch.Tensor] + +PEFT_MODULE_PREFIX = "base_model.model." +PEFT_MODULE_SUFFIXES = [".lora_A.weight", ".lora_B.weight"] +DORA_VECTOR_SUFFIXES = [ + ".lora_magnitude_vector", # HF peft + ".weight_m_wdecomp.weight" # NVLabs +] + + +class HFWeightsReader(abc.ABC): + + @abc.abstractmethod + def __init__(self, model_dir: str) -> None: + ... + + @abc.abstractmethod + def get_weight(self, weight_name: str) -> torch.Tensor: + ... + + @abc.abstractmethod + def get_all(self) -> StateDict: + ... + + +class HFSafeTensorsReader(HFWeightsReader): + + def __init__(self, model_dir: str) -> None: + self.model_dir = Path(model_dir) + + self._fds = [ + st.safe_open(f, framework="torch") + for f in self.model_dir.glob("*.safetensors") + ] + + self._weight_to_fd = {} + for f in self._fds: + for k in f.keys(): + self._weight_to_fd[k] = f + + def get_weight(self, weight_name: str) -> torch.Tensor: + return self._weight_to_fd[weight_name].get_tensor(weight_name) + + def get_all(self) -> StateDict: + return {k: self.get_weight(k) for k in self._weight_to_fd.keys()} + + +class HFBinWeightsReader(HFWeightsReader): + + def __init__(self, model_dir: str) -> None: + self.model_dir = Path(model_dir) + + self._weights = {} + + for f in self.model_dir.glob("*.bin"): + self._weights.update( + torch.load(f, weights_only=True, mmap=True, map_location="cpu")) + + def get_weight(self, weight_name: str) -> torch.Tensor: + weight_name = f"{weight_name}.weight" + return self._weights[weight_name] + + def get_all(self) -> StateDict: + return self._weights + + +class WeightsFormat(enum.Enum): + BINARY = enum.auto() + SAFETENSORS = enum.auto() + UNKNOWN = enum.auto() + + def __str__(self) -> str: + if self == self.BINARY: + return "bin" + elif self == self.SAFETENSORS: + return "safetensors" + return "unknown" + + +def deduce_weights_format(model_dir: str) -> WeightsFormat: + model_dir_p = Path(model_dir) + + if any(model_dir_p.glob("*.safetensors")): + return WeightsFormat.SAFETENSORS + elif any(model_dir_p.glob("*.bin")): + return WeightsFormat.BINARY + return WeightsFormat.UNKNOWN + + +def get_weights_reader(model_dir: str) -> HFWeightsReader: + model_dir_p = Path(model_dir) + + if not model_dir_p.is_dir(): + raise ValueError( + f"{model_dir} is not a valid model directory: not found") + + weights_format = deduce_weights_format(model_dir) + + if weights_format == WeightsFormat.SAFETENSORS: + return HFSafeTensorsReader(model_dir) + elif weights_format == WeightsFormat.BINARY: + return HFBinWeightsReader(model_dir) + else: + raise ValueError( + f"{model_dir} does not contain .safetensors or .bin weights") + + +def normalize_hf_peft_module_name(module_name: str) -> str: + """ + Remove parts of the module name in the peft adapter to derive the module name of the base model. + """ + + if module_name.startswith(PEFT_MODULE_PREFIX): + module_name = module_name[len(PEFT_MODULE_PREFIX):] + + for suffix in PEFT_MODULE_SUFFIXES + DORA_VECTOR_SUFFIXES: + if module_name.endswith(suffix): + module_name = module_name[:-len(suffix)] + + return module_name + + +def get_peft_module_names(base_module_name: str) -> tuple[str, ...]: + """ + Convert the name of a base module to the names of its LoRA A and LoRA B weights. + """ + return tuple([ + f"{PEFT_MODULE_PREFIX}{base_module_name}{suffix}" + for suffix in PEFT_MODULE_SUFFIXES + ]) + + +def get_dora_magnitude_names(base_module_name: str) -> tuple[str, ...]: + """ + Convert the name of a base module to the potential names of its DoRA magnitude vectors. + """ + return tuple([ + f"{PEFT_MODULE_PREFIX}{base_module_name}{suffix}" + for suffix in DORA_VECTOR_SUFFIXES + ]) + + +def normalize_dora_vector(W: torch.Tensor, A: torch.Tensor, B: torch.Tensor, + mag: torch.Tensor, scale: float) -> torch.Tensor: + return mag / torch.linalg.norm(W + scale * B @ A, dim=1).to(W.dtype) + + +def normalize_dora_scales(lora_sd: StateDict, + weights_reader: HFWeightsReader, + alpha: float, + use_rslora: bool, + strip: bool = False) -> StateDict: + out_sd = {} + + while lora_sd: + # take some lora weight name + module_name = next(iter(lora_sd.keys())) + base_module_name = normalize_hf_peft_module_name(module_name) + A_name, B_name = get_peft_module_names(base_module_name) + magnitude_names = get_dora_magnitude_names(base_module_name) + + if module_name not in [A_name, B_name] + list(magnitude_names): + raise ValueError(f"Encountered unknown weight: {module_name}") + + # get lora weights + A = lora_sd.pop(A_name) + B = lora_sd.pop(B_name) + for name in magnitude_names: + if name in lora_sd: + mag_name = name + mag = lora_sd.pop(mag_name).view(-1) + break + else: + mag_name = "" + mag = None + + out_sd[A_name] = A.contiguous() + out_sd[B_name] = B.contiguous() + + if mag is not None and not strip: + # get base weight and normalize + W = weights_reader.get_weight(base_module_name + ".weight") + + adapter_size = A.size(0) + + if use_rslora: + scale = alpha / np.sqrt(adapter_size) + else: + scale = alpha / adapter_size + + mag = normalize_dora_vector(W, A, B, mag, scale) + out_sd[mag_name] = mag.contiguous() + + return out_sd + + +def save_state_dict(out_file: str, sd: StateDict) -> None: + out_path = Path(out_file) + + if out_path.suffix == ".safetensors": + st.save_file(sd, out_path) + elif out_path.suffix == ".bin": + torch.save(sd, out_path) + else: + raise ValueError(f"Unregornized weights format: {out_path.suffix}") + + +def normalize_peft_ckpt(model_dir: str, + base_model_dir: str, + out_dir: str, + strip: bool = False) -> None: + out_path = Path(out_dir) + out_path.mkdir(parents=True, exist_ok=True) + + weights_reader = get_weights_reader(base_model_dir) + lora_sd = get_weights_reader(model_dir).get_all() + + adapter_config_path = Path(f"{model_dir}/adapter_config.json") + with adapter_config_path.open() as f: + adapter_config = json.load(f) + + alpha = adapter_config["lora_alpha"] + use_rslora = adapter_config.get("use_rslora", False) + + new_sd = normalize_dora_scales(lora_sd, weights_reader, alpha, use_rslora, + strip) + + with (out_path / "adapter_config.json").open("w") as f: + json.dump(adapter_config, f) + + weights_format = deduce_weights_format(model_dir) + save_state_dict( + (out_path / f"adapter_model.{str(weights_format)}").as_posix(), new_sd) + + +if __name__ == "__main__": + import argparse + parser = argparse.ArgumentParser() + parser.add_argument( + '--out-dir', + '-o', + type=Path, + help='path to output adapter weights with normalized DoRA vectors', + required=True) + parser.add_argument('--in-dir', + '-i', + type=Path, + help='path to input lora checkpoint file', + required=True) + parser.add_argument("--base-model", + "-b", + help="Path to base model", + required=True) + parser.add_argument("--strip", + action="store_true", + help="remove DoRA vectors entirely") + + args = parser.parse_args() + + normalize_peft_ckpt(args.in_dir, args.base_model, args.out_dir, args.strip) diff --git a/examples/eval_long_context.py b/examples/eval_long_context.py new file mode 100644 index 000000000000..90b7ef2dd270 --- /dev/null +++ b/examples/eval_long_context.py @@ -0,0 +1,331 @@ +# MIT License + +# Copyright (c) 2023 OpenBMB + +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# reference: https://github.com/OpenBMB/InfiniteBench/blob/main/src/eval_yarn_mistral.py + +import argparse +import ast +import json +from pathlib import Path + +import torch +from infinitebench.compute_scores import compute_scores +from infinitebench.eval_utils import (DATA_NAME_TO_MAX_NEW_TOKENS, + create_prompt, dump_jsonl, get_answer, + load_data) +from utils import (DEFAULT_HF_MODEL_DIRS, DEFAULT_PROMPT_TEMPLATES, + add_common_args, load_tokenizer, read_model_name) + +import tensorrt_llm +import tensorrt_llm.profiler as profiler +from tensorrt_llm.logger import logger +from tensorrt_llm.runtime import PYTHON_BINDINGS, ModelRunner + +if PYTHON_BINDINGS: + from tensorrt_llm.runtime import ModelRunnerCpp + +MAX_POSITION_ID = 128 * 1024 # Determined by the model +TRUNCATE_LEN = 128 * 1024 + + +def parse_arguments(args=None): + parser = argparse.ArgumentParser() + parser.add_argument('--batch_size', type=int, default=1) + parser.add_argument('--max_input_length', type=int, default=923) + parser.add_argument('--output_log_probs_npy', + type=str, + help='Numpy file where the log_probs are stored', + default=None) + + parser.add_argument('--output_cum_log_probs_npy', + type=str, + help='Numpy file where the cum_log_probs are stored', + default=None) + + parser.add_argument( + "--task", + type=str, + choices=['passkey', 'kv_retrieval'], + required=True, + help= + "Which task to use. Note that \"all\" can only be used in `compute_scores.py`.", # noqa + ) + parser.add_argument('--data_dir', + type=str, + default='./', + help="The directory of data.") + parser.add_argument("--output_dir", + type=str, + default=None, + help="Where to dump the prediction results.") # noqa + parser.add_argument( + "--start_idx", + type=int, + default=0, + help= + "The index of the first example to infer on. This is used if you want to evaluate on a (contiguous) subset of the data." + ) # noqa + parser.add_argument( + "--stop_idx", + type=int, + help= + "The index of the last example to infer on. This is used if you want to evaluate on a (contiguous) subset of the data. Defaults to the length of dataset." + ) # noqa + parser.add_argument('--tensorrt_llm_accuracy_threshold', + type=float, + default=99) + parser = add_common_args(parser) + + return parser.parse_args(args=args) + + +def parse_input(tokenizer, + input_text=None, + prompt_template=None, + add_special_tokens=True, + max_input_length=923, + pad_id=None, + num_prepend_vtokens=[], + model_name=None, + model_version=None): + if pad_id is None: + pad_id = tokenizer.pad_token_id + + batch_input_ids = [] + for curr_text in input_text: + if prompt_template is not None: + curr_text = prompt_template.format(input_text=curr_text) + input_ids = tokenizer.encode(curr_text, + add_special_tokens=add_special_tokens, + truncation=True, + max_length=max_input_length) + batch_input_ids.append(input_ids) + + if num_prepend_vtokens: + assert len(num_prepend_vtokens) == len(batch_input_ids) + base_vocab_size = tokenizer.vocab_size - len( + tokenizer.special_tokens_map.get('additional_special_tokens', [])) + for i, length in enumerate(num_prepend_vtokens): + batch_input_ids[i] = list( + range(base_vocab_size, + base_vocab_size + length)) + batch_input_ids[i] + + if 'GLM' in model_name and model_version == 'glm': + for ids in batch_input_ids: + ids.append(tokenizer.sop_token_id) + + batch_input_ids = [ + torch.tensor(x, dtype=torch.int32) for x in batch_input_ids + ] + return batch_input_ids + + +def main(args): + # model_name = "yarn-mistral" + runtime_rank = tensorrt_llm.mpi_rank() + logger.set_level(args.log_level) + + print(json.dumps(vars(args), indent=4)) + data_name = args.task + + # Model + max_tokens = DATA_NAME_TO_MAX_NEW_TOKENS[data_name] + + model_name, model_version = read_model_name(args.engine_dir) + if args.tokenizer_dir is None: + logger.warning( + "tokenizer_dir is not specified. Try to infer from model_name, but this may be incorrect." + ) + args.tokenizer_dir = DEFAULT_HF_MODEL_DIRS[model_name] + + tokenizer, pad_id, end_id = load_tokenizer( + tokenizer_dir=args.tokenizer_dir, + vocab_file=args.vocab_file, + model_name=model_name, + model_version=model_version, + tokenizer_type=args.tokenizer_type, + ) + + if not PYTHON_BINDINGS and not args.use_py_session: + logger.warning( + "Python bindings of C++ session is unavailable, fallback to Python session." + ) + args.use_py_session = True + if args.debug_mode and not args.use_py_session: + logger.warning( + "Debug mode is not supported in C++ session for now, fallback to Python session." + ) + args.use_py_session = True + runner_cls = ModelRunner if args.use_py_session else ModelRunnerCpp + runner_kwargs = dict( + engine_dir=args.engine_dir, + lora_dir=args.lora_dir, + rank=runtime_rank, + debug_mode=args.debug_mode, + lora_ckpt_source=args.lora_ckpt_source, + gpu_weights_percent=args.gpu_weights_percent, + ) + if args.medusa_choices is not None: + args.medusa_choices = ast.literal_eval(args.medusa_choices) + assert args.temperature == 1.0, "Medusa should use temperature == 1.0" + assert args.num_beams == 1, "Medusa should use num_beams == 1" + runner_kwargs.update(medusa_choices=args.medusa_choices) + if not args.use_py_session: + runner_kwargs.update( + max_batch_size=args.batch_size, + max_input_len=args.max_input_length, + max_output_len=max_tokens, + max_beam_width=args.num_beams, + max_attention_window_size=args.max_attention_window_size, + sink_token_length=args.sink_token_length, + max_tokens_in_paged_kv_cache=args.max_tokens_in_paged_kv_cache, + kv_cache_enable_block_reuse=args.kv_cache_enable_block_reuse, + kv_cache_free_gpu_memory_fraction=args. + kv_cache_free_gpu_memory_fraction, + enable_chunked_context=args.enable_chunked_context, + ) + runner = runner_cls.from_dir(**runner_kwargs) + + # Data + examples = load_data(data_name, data_dir=args.data_dir) + if args.stop_idx is None: + args.stop_idx = len(examples) + + output_path = None + if runtime_rank == 0: + if args.output_dir is not None: + result_dir = Path(args.output_dir, model_name) + result_dir.mkdir(exist_ok=True, parents=True) + + if args.stop_idx is None: + output_path = (result_dir / f"preds_{data_name}.jsonl") + else: + output_path = ( + result_dir / + f"preds_{data_name}_{args.start_idx}-{args.stop_idx}.jsonl" # noqa + ) + + prompt_template = None + if args.use_prompt_template and model_name in DEFAULT_PROMPT_TEMPLATES: + prompt_template = DEFAULT_PROMPT_TEMPLATES[model_name] + + if runtime_rank == 0: + preds = [] + logger.info("==== Evaluation ====") + logger.info(f"# examples: {len(examples)}") + logger.info(f"Start index: {args.start_idx}") + logger.info(f"Stop index: {args.stop_idx}") + logger.info(f"Max tokens: {max_tokens}") + assert args.batch_size == 1 + profiler.start('Evaluation') + for i in range(args.start_idx, args.stop_idx): + eg = examples[i] + input_text = [create_prompt(eg, data_name, args.data_dir)] + batch_input_ids = parse_input( + tokenizer=tokenizer, + input_text=input_text, + prompt_template=prompt_template, + add_special_tokens=args.add_special_tokens, + max_input_length=args.max_input_length, + pad_id=pad_id, + num_prepend_vtokens=args.num_prepend_vtokens, + model_name=model_name, + model_version=model_version) + input_lengths = [x.size(0) for x in batch_input_ids] + + if runtime_rank == 0: + logger.debug(f"====== Example {i} ======") + logger.debug(f"input_lengths: {input_lengths}") + logger.debug(f"input_text: {input_text}") + logger.debug(f"answer: {get_answer(eg, data_name)}") + outputs = runner.generate( + batch_input_ids, + max_new_tokens=max_tokens, + max_attention_window_size=args.max_attention_window_size, + sink_token_length=args.sink_token_length, + end_id=end_id, + pad_id=pad_id, + temperature=args.temperature, + top_k=args.top_k, + top_p=args.top_p, + num_beams=args.num_beams, + length_penalty=args.length_penalty, + early_stopping=args.early_stopping, + beam_width_array=args.beam_width_array, + repetition_penalty=args.repetition_penalty, + presence_penalty=args.presence_penalty, + frequency_penalty=args.frequency_penalty, + prompt_ignore_length=args.prompt_ignore_length, + # stop_words_list=stop_words_list, + # bad_words_list=bad_words_list, + output_cum_log_probs=(args.output_cum_log_probs_npy != None), + output_log_probs=(args.output_log_probs_npy != None), + lora_uids=args.lora_task_uids, + prompt_table=args.prompt_table_path, + prompt_tasks=args.prompt_tasks, + streaming=args.streaming, + output_sequence_lengths=True, + return_dict=True, + medusa_choices=args.medusa_choices) + torch.cuda.synchronize() + if runtime_rank == 0: + output_ids = outputs['output_ids'] + output_beams_list = [ + tokenizer.batch_decode(output_ids[batch_idx, :, + input_lengths[batch_idx]:], + skip_special_tokens=True) + for batch_idx in range(args.batch_size) + ] + + logger.debug(f"preds: {output_beams_list[0]}") + preds.append({ + "id": i, + "prediction": output_beams_list[0][0], + "ground_truth": get_answer(eg, data_name), + "input_lengths": input_lengths, + }) + if output_path is not None: + dump_jsonl(preds, output_path) + profiler.stop('Evaluation') + + if runtime_rank == 0: + logger.info( + f'Evaluation takes: {profiler.elapsed_time_in_sec("Evaluation")} sec.' + ) + logger.info("Compute the score") + acc = compute_scores(preds, args.task) * 100 + logger.info(f"{args.task} accuracy: {acc:.2f} ({len(preds)})") + + if args.tensorrt_llm_accuracy_threshold is not None: + assert acc >= args.tensorrt_llm_accuracy_threshold, f"acc ({acc}) < tensorrt_llm_accuracy_threshold ({args.tensorrt_llm_accuracy_threshold})" + + +if __name__ == "__main__": + args = parse_arguments() + main(args) diff --git a/examples/generate_checkpoint_config.py b/examples/generate_checkpoint_config.py new file mode 100644 index 000000000000..a11104eeba68 --- /dev/null +++ b/examples/generate_checkpoint_config.py @@ -0,0 +1,149 @@ +import argparse +import json +import os + +from tensorrt_llm.quantization import KV_CACHE_QUANT_ALGO_LIST, QUANT_ALGO_LIST + + +def parse_arguments(): + parser = argparse.ArgumentParser() + + parser.add_argument( + '--output_path', + type=str, + default='config.json', + help='The path to save the TensorRT LLM checkpoint config.json file') + parser.add_argument('--architecture', type=str, default='GPTForCausalLM') + parser.add_argument('--dtype', + type=str, + default='float16', + choices=['float32', 'bfloat16', 'float16']) + parser.add_argument('--vocab_size', type=int, default=32000) + parser.add_argument('--max_position_embeddings', type=int, default=1024) + parser.add_argument('--hidden_size', type=int, default=768) + parser.add_argument('--intermediate_size', type=int, default=None) + parser.add_argument('--num_hidden_layers', type=int, default=12) + parser.add_argument('--num_attention_heads', type=int, default=12) + parser.add_argument('--num_key_value_heads', type=int, default=None) + parser.add_argument('--hidden_act', type=str, default='gelu') + parser.add_argument('--norm_epsilon', type=float, default=1e-5) + parser.add_argument('--position_embedding_type', + type=str, + default='learned_absolute') + parser.add_argument( + '--use_parallel_embedding', + action='store_true', + default=False, + help= + 'By default embedding parallelism is disabled. By setting this flag, embedding parallelism is enabled' + ) + parser.add_argument( + '--embedding_sharding_dim', + type=int, + default=0, + choices=[0, 1], + help= + 'By default the embedding lookup table is sharded along vocab dimension (embedding_sharding_dim=0). ' + 'To shard it along hidden dimension, set embedding_sharding_dim=1' + 'Note: embedding sharing is only enabled when embedding_sharding_dim = 0' + ) + + parser.add_argument('--tp_size', + type=int, + default=1, + help='N-way tensor parallelism size') + parser.add_argument('--pp_size', + type=int, + default=1, + help='N-way pipeline parallelism size') + + parser.add_argument('--quant_algo', + type=str, + default=None, + choices=[None] + QUANT_ALGO_LIST) + parser.add_argument('--kv_cache_quant_algo', + type=str, + default=None, + choices=[None] + KV_CACHE_QUANT_ALGO_LIST) + parser.add_argument('--group_size', type=int, default=64) + parser.add_argument('--smoothquant_val', type=float, default=None) + parser.add_argument('--has_zero_point', default=False, action='store_true') + parser.add_argument('--pre_quant_scale', default=False, action='store_true') + parser.add_argument('--exclude_modules', nargs='+', default=None) + + parser.add_argument('--bias', default=False, action='store_true') + parser.add_argument('--apply_query_key_layer_scaling', + default=False, + action='store_true') + parser.add_argument('--rotary_pct', type=float, default=1.0) + parser.add_argument('--rotary_base', type=float, default=10000.0) + parser.add_argument('--rotary_scaling', nargs=2, type=str, default=None) + + args = parser.parse_args() + return args + + +if __name__ == '__main__': + args = parse_arguments() + world_size = args.tp_size * args.pp_size + + assert args.output_path.endswith('.json') + output_dir = os.path.dirname(args.output_path) + if output_dir and not os.path.exists(output_dir): + os.makedirs(output_dir) + + config = { + 'architecture': args.architecture, + 'dtype': args.dtype, + 'vocab_size': args.vocab_size, + 'max_position_embeddings': args.max_position_embeddings, + 'hidden_size': args.hidden_size, + 'intermediate_size': args.intermediate_size, + 'num_hidden_layers': args.num_hidden_layers, + 'num_attention_heads': args.num_attention_heads, + 'num_key_value_heads': args.num_key_value_heads, + 'hidden_act': args.hidden_act, + 'norm_epsilon': args.norm_epsilon, + 'position_embedding_type': args.position_embedding_type, + 'use_parallel_embedding': args.use_parallel_embedding, + 'embedding_sharding_dim': args.embedding_sharding_dim, + 'quantization': { + 'quant_algo': args.quant_algo, + 'kv_cache_quant_algo': args.kv_cache_quant_algo, + 'exclude_modules': args.exclude_modules, + }, + 'mapping': { + 'world_size': world_size, + 'tp_size': args.tp_size, + 'pp_size': args.pp_size, + }, + 'bias': args.bias, + 'apply_query_key_layer_scaling': args.apply_query_key_layer_scaling, + 'rotary_pct': args.rotary_pct, + 'rotary_base': args.rotary_base, + 'rotary_scaling': args.rotary_scaling, + } + + if args.intermediate_size is None: + config['intermediate_size'] = args.hidden_size * 4 + if args.num_key_value_heads is None: + config['num_key_value_heads'] = args.num_attention_heads + + if args.quant_algo is not None: + if 'AWQ' in args.quant_algo or 'GPTQ' in args.quant_algo: + config['quantization'].update({ + 'group_size': + args.group_size, + 'has_zero_point': + args.has_zero_point, + 'pre_quant_scale': + args.pre_quant_scale, + }) + if 'SQ' in args.quant_algo: + config['quantization'].update({ + 'smoothquant_val': + args.smoothquant_val, + }) + + with open(args.output_path, 'w') as f: + json.dump(config, f, indent=4) diff --git a/examples/generate_xgrammar_tokenizer_info.py b/examples/generate_xgrammar_tokenizer_info.py new file mode 100644 index 000000000000..67e05eb9c968 --- /dev/null +++ b/examples/generate_xgrammar_tokenizer_info.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import json +import os +from pathlib import Path + +from transformers import AutoTokenizer + +from tensorrt_llm.llmapi.tokenizer import _xgrammar_tokenizer_info + + +def generate_xgrammar_tokenizer_info(args): + + tokenizer = AutoTokenizer.from_pretrained(str(args.model_dir)) + tokenizer_info = _xgrammar_tokenizer_info(tokenizer) + + os.makedirs(args.output_dir, exist_ok=True) + with open(str(args.output_dir / "xgrammar_tokenizer_info.json"), 'w') as f: + json.dump(tokenizer_info, f) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument('--model_dir', + type=Path, + default=None, + required=True, + help="HF model directory") + parser.add_argument( + '--output_dir', + type=Path, + default=None, + required=True, + help="File path to save xgrammar's info. in json format") + args = parser.parse_args() + generate_xgrammar_tokenizer_info(args) diff --git a/examples/hf_lora_convert.py b/examples/hf_lora_convert.py new file mode 100755 index 000000000000..019d78a48563 --- /dev/null +++ b/examples/hf_lora_convert.py @@ -0,0 +1,266 @@ +#! /usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import argparse +import datetime +import json +import logging +import re +from collections import defaultdict +from pathlib import Path + +import numpy as np +import torch + +from tensorrt_llm._utils import str_dtype_to_torch, torch_to_numpy +from tensorrt_llm.lora_manager import LoraManager +from tensorrt_llm.models.convert_utils import get_model_path, load_state_dict + +log_format = "%(asctime)s %(name)s [%(levelname)s] %(message)s" +logging.basicConfig(format=log_format) +LOGGER = logging.getLogger(__name__) + + +def save_val(val, dir, key, tp_num=None, write_npy=False): + ext = "npy" if write_npy else "bin" + suffix = ext if tp_num is None else f"{tp_num}.{ext}" + if write_npy: + np.save(dir / f"model.{key}.{suffix}", val) + else: + val.tofile(dir / f"model.{key}.{suffix}") + + +def get_all_lora_weights(lora_weights): + all_weights = defaultdict(lambda: defaultdict(dict)) + pattern = re.compile( + r'(.*\.layers\.([0-9]+)\.(self_attn|mlp)\.([a-z_]+))\.(?:lora_(?:(A|B)\.weight|(magnitude)_vector)|weight_(m_wdecomp).weight).*' + ) + moe_pattern = re.compile( + r'(.*\.layers\.([0-9]+)\.(block_sparse_moe)\.((experts)\.([0-9]+)\.|)([a-zA-Z0-9_]+))\.(?:lora_(?:(A|B)\.weight|(magnitude)_vector)|weight_(m_wdecomp).weight).*' + ) + for key, weights in lora_weights.items(): + m = pattern.match(key) + m_moe = moe_pattern.match(key) + if m: + layer_idx = int(m.group(2)) + hf_module = m.group(4) + inout = m.group(5) + dora_magnitude = m.group(6) or m.group(7) + + if inout: + inout = "in" if inout == "A" else "out" + all_weights[layer_idx][hf_module][inout] = weights + elif dora_magnitude: + LOGGER.warning( + "Detected DoRA magnitude vector, make sure it was preprocessed and normalized using the proper base model weights" + ) + all_weights[layer_idx][hf_module]["magnitude"] = weights.view( + -1) + + elif m_moe: + layer_idx = int(m_moe.group(2)) + hf_module = m_moe.group(7) + inout = m_moe.group(8) + dora_magnitude = m_moe.group(9) or m.group(10) + + if inout: + inout = "in" if inout == "A" else "out" + all_weights[layer_idx][hf_module][inout] = weights + elif dora_magnitude: + LOGGER.warning( + "Detected DoRA magnitude vector, make sure it was preprocessed and normalized using the proper base model weights" + ) + all_weights[layer_idx][hf_module]["magnitude"] = weights.view( + -1) + else: + print(f"no match {key}") + continue + return all_weights + + +def preprocess_lora_weights(lora_model): + # Swap weights of gate_up_proj + for key, value in lora_model.items(): + if "gate_up_proj.lora_B.weight" in key: + print("Swap {}".format(key)) + original_weights = value.contiguous().clone() + half_split = original_weights.shape[0] // 2 + first_half = original_weights[:half_split, :] + second_half = original_weights[half_split:, :] + value = torch.cat((second_half, first_half), dim=0) + lora_model[key] = value + return lora_model + + +hf_modules_to_trtllm_modules = { + "q_proj": "attn_q", + "v_proj": "attn_v", + "k_proj": "attn_k", + "qkv_proj": "attn_qkv", + "query_key_value": "attn_qkv", + "o_proj": "attn_dense", + "dense": "attn_dense", + "gate_proj": "mlp_h_to_4h", + "down_proj": "mlp_4h_to_h", + "up_proj": "mlp_gate", + "gate_up_proj": "mlp_h_to_4h", + "c_fc": "mlp_h_to_4h", + "c_proj": "mlp_4h_to_h", + "w1": "moe_h_to_4h", + "w2": "moe_4h_to_h", + "w3": "moe_gate", + "gate": "moe_router", +} # lora modules on llama +hf_modules_to_module_id = { + k: LoraManager.LORA_MODULE_IDS[v] + for k, v in hf_modules_to_trtllm_modules.items() +} + + +def convert_hf_model(model_dir, dtype, out_dir): + saved_dir = Path(out_dir) + saved_dir.mkdir(parents=True, exist_ok=True) + with open(f"{model_dir}/adapter_config.json", "r") as f: + config = json.load(f) + + alpha = config.get("lora_alpha") + use_rslora = config.get("use_rslora", False) + + lora_model = load_state_dict(get_model_path(model_dir, "adapter_model")) + lora_model = preprocess_lora_weights(lora_model) + all_weights = get_all_lora_weights(lora_model) + converted_weights = [] + converted_config = [] + + def derive_adapter_size(inout_weight: torch.Tensor) -> int: + assert len(inout_weight.shape) == 2 + dim0, dim1 = inout_weight.shape + # assume the hidden dim is the larger of the 2 + adapter_size = min(dim0, dim1) + return adapter_size + + def derive_weights_scale(adapter_size: int, alpha: float, + use_rslora: bool) -> float: + if use_rslora: + return alpha / np.sqrt(adapter_size) + return alpha / adapter_size + + for layer_idx, layer_weights in all_weights.items(): + for hf_module, module_weights in layer_weights.items(): + in_weights = module_weights['in'] + out_weights = module_weights['out'] + magnitude = module_weights.get("magnitude", None) + is_dora = magnitude is not None + + processed_weights = [] + + assert len(in_weights.shape) == 2 + assert len(out_weights.shape) == 2 + assert not is_dora or len(magnitude.shape) == 1 + + adapter_size = derive_adapter_size(in_weights) + assert adapter_size == derive_adapter_size( + out_weights), "adapter size of A mismatches adapter size of B" + scale = derive_weights_scale(adapter_size, alpha, use_rslora) + + for w, inout in ((in_weights, "in"), (out_weights, "out")): + dim0 = w.shape[0] + dim1 = w.shape[1] + # in_weights should have shape [adaper_size, hidden] + if dim1 < dim0 and inout == "in": + w = w.transpose(1, 0) + # out_weights should have shape [hidden, adapter_size] + elif dim0 < dim1 and inout == "out": + w = w.transpose(1, 0) + if inout == "out": + w = w * scale + w = w.contiguous().flatten().to(dtype=str_dtype_to_torch(dtype)) + processed_weights.append(w) + + if is_dora: + processed_weights.append(magnitude.contiguous().flatten().to( + dtype=str_dtype_to_torch(dtype))) + + processed_weights = torch.concatenate(processed_weights).flatten() + converted_weights.append(processed_weights) + converted_config.append([ + hf_modules_to_module_id[hf_module], layer_idx, adapter_size, + 1 if is_dora else 0 + ]) + max_row_size = 0 + for t in converted_weights: + max_row_size = max(max_row_size, t.shape[0]) + for i in range(len(converted_weights)): + converted_weights[i] = torch.nn.functional.pad( + converted_weights[i], + (0, max_row_size - converted_weights[i].shape[0])).unsqueeze(0) + converted_weights = torch_to_numpy( + torch.concatenate( + converted_weights, + dim=0).unsqueeze(0).to(dtype=str_dtype_to_torch(dtype)).cpu()) + converted_config = torch.tensor(converted_config, + dtype=torch.int32, + device='cpu').unsqueeze(0).numpy() + + save_val(converted_weights, + saved_dir, + "lora_weights", + tp_num=None, + write_npy=True) + save_val(converted_config, + saved_dir, + "lora_config", + tp_num=None, + write_npy=True) + + +def main(args): + start_time = datetime.datetime.now() + convert_hf_model(args.in_file, args.storage_type, args.out_dir) + + LOGGER.info("Spent %s (h:m:s) to convert the prompt model", + datetime.datetime.now() - start_time) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument( + '--out-dir', + '-o', + type=Path, + help='path to output embedding table file in the .npy format', + required=True) + parser.add_argument('--in-file', + '-i', + type=Path, + help='path to input lora checkpoint file', + required=True) + parser.add_argument("--verbose", + action="store_true", + help="Provide verbose messages") + parser.add_argument("--storage-type", + type=str, + default="float16", + choices=["float32", "float16", "bfloat16"]) + args = parser.parse_args() + + LOGGER.setLevel(logging.DEBUG if args.verbose else logging.INFO) + + print("\n=============== Argument ===============") + for key in vars(args): + print(f"{key}: {vars(args)[key]}") + print("========================================") + + main(args) diff --git a/examples/language_adapter/README.md b/examples/language_adapter/README.md new file mode 100755 index 000000000000..8487c8ab42a0 --- /dev/null +++ b/examples/language_adapter/README.md @@ -0,0 +1,99 @@ +# Language-Adapter + +This document shows how to build and run a model with Language-Adapter plugin in TensorRT LLM on NVIDIA GPUs. + +## Overview +The concept of Language Adapter during inference time was introduced in [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer +](https://arxiv.org/pdf/2005.00052): +> we can simply replace a language-specific adapter trained for English with a language-specific adapter trained for Quechua at inference time. + +The implementation is done with MOE plugin with static expert selection passed during runtime as a parameter in request. + +For instance, encoder-decoder model may leverage language adapter for language-specific translation tasks when each of the language-adapter is trained for a specific language, this language adapter plugin achieves the language switching within one session only by passing in the `language_task_uid` to the plugin. + +The model checkpoint here is not publicly available. Please leverage `layers/language_adapter.py` in your own model. + +### Engine Preparation (convert and build) +``` +MODEL_DIR="dummy_model" # model not publicly available +INFERENCE_PRECISION="float16" +TP_SIZE=1 +PP_SIZE=1 +WORLD_SIZE=1 +MODEL_TYPE=language_adapter +MODEL_NAME=$MODEL_TYPE +CKPT_DIR=/scratch/tmp/trt_models/${MODEL_NAME}/${WORLD_SIZE}-gpu/${INFERENCE_PRECISION} +ENGINE_DIR=/scratch/tmp/trt_engines/${MODEL_NAME}/${WORLD_SIZE}-gpu/${INFERENCE_PRECISION} + +max_beam=5 +max_batch=32 +max_input_len=1024 +max_output_len=1024 + +python ../enc_dec/convert_checkpoint.py --model_type ${MODEL_TYPE} \ + --model_dir ${MODEL_DIR} \ + --output_dir $CKPT_DIR \ + --tp_size ${TP_SIZE} \ + --pp_size ${PP_SIZE} \ + --dtype ${INFERENCE_PRECISION} \ + --workers 1 + +trtllm-build --checkpoint_dir $CKPT_DIR/encoder \ + --output_dir $ENGINE_DIR/encoder \ + --paged_kv_cache disable \ + --moe_plugin auto \ + --bert_attention_plugin ${INFERENCE_PRECISION} \ + --gpt_attention_plugin ${INFERENCE_PRECISION} \ + --gemm_plugin ${INFERENCE_PRECISION} \ + --remove_input_padding enable \ + --max_input_len ${max_input_len} \ + --max_beam_width ${max_beam} \ + --max_batch_size ${max_batch} + +trtllm-build --checkpoint_dir $CKPT_DIR/decoder \ + --output_dir $ENGINE_DIR/decoder \ + --paged_kv_cache enable \ + --moe_plugin auto \ + --bert_attention_plugin ${INFERENCE_PRECISION} \ + --gpt_attention_plugin ${INFERENCE_PRECISION} \ + --gemm_plugin ${INFERENCE_PRECISION} \ + --remove_input_padding enable \ + --max_input_len 1 \ + --max_beam_width ${max_beam} \ + --max_batch_size ${max_batch} \ + --max_seq_len ${max_output_len} +``` + +### CPP runtime +A list `language_task_uids` that includes the language_task_uid for each input prompt is required: +``` +# translate 2 sentence, 1 to France (language_task_uid=3) 1 to Spanish (language_task_uid=2). +# language_task_uids = [3, 2] + +TEXT="Where is the nearest restaurant? Wikipedia is a free online encyclopedia written and maintained by a community of volunteers (called Wikis) through open collaboration and the use of MediaWiki, a wiki-based editing system." + +python3 ../run.py --engine_dir $ENGINE_DIR --tokenizer_type "language_adapter" --max_input_length 512 --max_output_len 512 --num_beams 1 --input_file input_ids.npy --tokenizer_dir $MODEL_DIR --language_task_uids 3 2 + +# Input [Text 0]: "" +# Output [Text 0 Beam 0]: "Où se trouve le restaurant le plus proche ? Wikipédia est une encyclopédie en ligne gratuite écrite et maintenue par une communauté de bénévoles (appelés Wikis) grâce à une collaboration ouverte et à l'utilisation de MediaWiki, un système d'édition basé sur wiki." +# Input [Text 1]: "" +# Output [Text 1 Beam 0]: "¿Dónde está el restaurante más cercano? Wikipedia es una enciclopedia en línea gratuita escrita y mantenida por una comunidad de voluntarios (llamada Wikis) a través de la colaboración abierta y el uso de MediaWiki, un sistema de edición basado en wiki." + +``` + +### Python runtime +Currently Python runtime does not support beam_width > 1. + +For Python runtime, full routing information of length [num_tokens, 1] is required for both encoder and decoder, which stacks routing information for each token in a batch of requests. +``` +# language_adapter_routing = get_language_adapter_routings(language_task_uid, input_ids) + +TEXT="Where is the nearest restaurant? Wikipedia is a free online encyclopedia written and maintained by a community of volunteers (called Wikis) through open collaboration and the use of MediaWiki, a wiki-based editing system." + +python3 ../enc_dec/run.py --engine_dir $ENGINE_DIR --engine_name ${MODEL_NAME} --model_name $MODEL_DIR --max_new_token=64 --num_beams=1 + +# in the run.py, 2 input prompts and 2 language task uids are provided. The two task uid represent the language of the input prompts to be translated to. + +# TRT-LLM output text: ['¿Dónde está el restaurante más cercano? Wikipedia es una enciclopedia en línea gratuita escrita y mantenida por una comunidad de voluntarios (llamada Wikis) a través de la colaboración abierta y el uso de MediaWiki, un sistema de edición basado en wiki.', "Où se trouve le restaurant le plus proche ? Wikipédia est une encyclopédie en ligne gratuite é +crite et maintenue par une communauté de bénévoles (appelés Wikis) grâce à une collaboration ouverte et à l'utilisation de MediaWiki, un système d'édition basé sur wiki."] +``` diff --git a/examples/layer_wise_benchmarks/README.md b/examples/layer_wise_benchmarks/README.md index 2b8070f955c1..91662c6b1269 100644 --- a/examples/layer_wise_benchmarks/README.md +++ b/examples/layer_wise_benchmarks/README.md @@ -255,22 +255,21 @@ Two E2E traces are required because the two pieces of information cannot be capt calibration_file_path: profiles/calibration_data.json ``` - 2. Set `TLLM_PROFILE_START_STOP` to a range that captures some iterations (typically tens of iterations) of the GEN phase. Ensure that every iteration has the same batch size. Capture 5 extra iterations at the beginning, because the first 5 iterations are treated as warm-ups and will be dropped by the parser by default. Calibration data is collected for the iterations `[start, stop)`, i.e. the stop iteration itself is not collected. + 2. Set `TLLM_PROFILE_START_STOP` to a range that captures some iterations (typically tens of iterations) of the GEN phase. Ensure that every iteration has the same batch size. Capture 5 extra iterations at the beginning, because the first 5 iterations are treated as warm-ups and will be dropped by the parser by default. 3. Capture per-rank nsys profiles; each rank should produce a separate file. Place `nsys profile` after `mpirun` or `srun`. To minimize profiling overhead and file size, there is no need to capture samples or GPU metrics. - Trace the whole process instead of gating the collection with `-c cudaProfilerApi`: recent nsys versions export kernels launched by CUDA graphs that were instantiated before the capture range opened without runtime correlation, which breaks `parse_e2e.py`, and stopping the collection mid-run can hang the executor (the engine captures its CUDA graphs during warmup, before any capture range can open). `parse_e2e.py` selects the analyzed iterations via `--start-iter`/`--stop-iter` instead. - If you use `trtllm-serve` or `trtllm-bench`, use the following command order. If you use `examples/disaggregated/slurm/benchmark/submit.py`, setting `gen_profile_range` is sufficient. ```bash NP=$NP ./mpi_launch.sh middleware/mpi_env_from_ompi \ nsys profile \ - -t cuda,nvtx -s none \ + -t cuda,nvtx \ --cpuctxsw none --cuda-event-trace false \ --cuda-graph-trace node \ + -c cudaProfilerApi --capture-range-end stop \ -o profiles/report_e2e_collect_rank%q{RANK}.nsys-rep \ --force-overwrite true \ trtllm-llmapi-launch \ @@ -307,7 +306,7 @@ Two E2E traces are required because the two pieces of information cannot be capt --balance-method NotModified \ --replay-file-path profiles/calibration_data.json \ --replay-start-iter 47 \ - --replay-stop-iter 66 + --replay-stop-iter 67 ``` Argument explanations: @@ -321,7 +320,7 @@ Two E2E traces are required because the two pieces of information cannot be capt | `--seq-len-q 1` | Should match (1 + MTP) of the end-to-end run. | | `--seq-len-kv-cache 2090` | An estimate of the average context length for the captured iterations. The first 5 iterations should be excluded from this estimate because they will be dropped by the parser. | | `--replay-file-path` | The calibration file obtained from Step 1. | - | `--replay-start-iter` and `--replay-stop-iter` | Both inclusive. The calibration file contains the end-to-end `TLLM_PROFILE_START_STOP` iterations `[start, stop)`. Do not replay the first 5 iterations because they will be dropped by the parser. | + | `--replay-start-iter` and `--replay-stop-iter` | Should match the end-to-end `TLLM_PROFILE_START_STOP`. Do not replay the first 5 iterations because they will be dropped by the parser. | 4. Parse end-to-end profiles with `parse_e2e.py`, and parse layer-wise benchmarks profiles with `parse.py`. @@ -331,8 +330,6 @@ Two E2E traces are required because the two pieces of information cannot be capt --graph-trace profiles/report_e2e_collect_rank%.nsys-rep \ --layer-indices 5,6,7 \ --warmup-times 5 \ - --start-iter 42 \ - --stop-iter 66 \ -o profiles/report_e2e_collect_rank%.json seq 0 $((NP - 1)) | xargs -I% python3 parse.py \ --world-size $NP \ diff --git a/examples/layer_wise_benchmarks/parse.py b/examples/layer_wise_benchmarks/parse.py index 33b8002ef3b8..b52f656b769f 100644 --- a/examples/layer_wise_benchmarks/parse.py +++ b/examples/layer_wise_benchmarks/parse.py @@ -13,7 +13,6 @@ from parser_utils import ( kernel_short_name, lazy_convert_sqlite, - require_cuda_kernel_events, shortest_common_supersequence, warned_names, ) @@ -102,7 +101,6 @@ class CategoryTime(NamedTuple): lazy_convert_sqlite(nsys_rep_file_path, sqlite_file_path) conn = sqlite3.connect(f"file:{sqlite_file_path}?mode=ro", uri=True) -require_cuda_kernel_events(conn, sqlite_file_path) query = "SELECT * FROM ENUM_NSYS_EVENT_TYPE" df = pd.read_sql_query(query, conn) diff --git a/examples/layer_wise_benchmarks/parse_e2e.py b/examples/layer_wise_benchmarks/parse_e2e.py index a867eecf2c57..b4df91052a77 100644 --- a/examples/layer_wise_benchmarks/parse_e2e.py +++ b/examples/layer_wise_benchmarks/parse_e2e.py @@ -11,7 +11,6 @@ from parser_utils import ( kernel_short_name, lazy_convert_sqlite, - require_cuda_kernel_events, shortest_common_supersequence, warned_names, ) @@ -57,8 +56,6 @@ def comma_separated_ints(s: str) -> list[int]: parser.add_argument("--target-gen-reqs", type=int) parser.add_argument("--layer-indices", type=comma_separated_ints, required=True) parser.add_argument("--warmup-times", type=int, default=5) -parser.add_argument("--start-iter", type=int, help="Only consider iterations with ID >= this value") -parser.add_argument("--stop-iter", type=int, help="Only consider iterations with ID <= this value") group = parser.add_mutually_exclusive_group() group.add_argument("--error-on-unknown-kernel", action="store_true", dest="error_on_unknown_kernel") group.add_argument( @@ -78,21 +75,6 @@ def is_gemm(name: str) -> bool: return "nvjet" in name or "gemm" in name.lower() -# Groups: (1) iteration id, (2) ctx reqs, (3) gen reqs. The "ctx tokens" -# segment is optional so traces recorded before it was added still parse. -FORWARD_STEP_RE = re.compile( - r"^\[Executor\] _forward_step (\d+): (\d+) ctx reqs(?:, \d+ ctx tokens)?, (\d+) gen reqs" -) - - -def iter_in_window(iter_id: int) -> bool: - if args.start_iter is not None and iter_id < args.start_iter: - return False - if args.stop_iter is not None and iter_id > args.stop_iter: - return False - return True - - eager_nsys_rep_file_path = Path(args.eager_trace) # For CTX phase which does not use CUDA Graphs, analyze the eager trace instead. # Here we do not change the identifier name "graph_*" for convenience. @@ -107,8 +89,6 @@ def iter_in_window(iter_id: int) -> bool: lazy_convert_sqlite(graph_nsys_rep_file_path, graph_sqlite_file_path) eager_conn = sqlite3.connect(f"file:{eager_sqlite_file_path}?mode=ro", uri=True) graph_conn = sqlite3.connect(f"file:{graph_sqlite_file_path}?mode=ro", uri=True) -require_cuda_kernel_events(eager_conn, eager_sqlite_file_path) -require_cuda_kernel_events(graph_conn, graph_sqlite_file_path) query = "SELECT * FROM ENUM_NSYS_EVENT_TYPE" df = pd.read_sql_query(query, eager_conn) @@ -126,10 +106,12 @@ def iter_in_window(iter_id: int) -> bool: if target_gen_reqs is None: if target_ctx_reqs == 0: for _, _, text in df.itertuples(index=False): - if m := FORWARD_STEP_RE.match(text): + if m := re.match( + r"^\[Executor\] _forward_step (\d+): (\d+) ctx reqs, (\d+) gen reqs", text + ): ctx_reqs = int(m.group(2)) gen_reqs = int(m.group(3)) - if ctx_reqs == target_ctx_reqs and iter_in_window(int(m.group(1))): + if ctx_reqs == target_ctx_reqs: target_gen_reqs = gen_reqs break else: @@ -139,11 +121,11 @@ def iter_in_window(iter_id: int) -> bool: print(f"{target_ctx_reqs=} {target_gen_reqs=}") eager_iters: list[IterInfo] = [] for start, end, text in df.itertuples(index=False): - if m := FORWARD_STEP_RE.match(text): + if m := re.match(r"^\[Executor\] _forward_step (\d+): (\d+) ctx reqs, (\d+) gen reqs", text): iter_id = int(m.group(1)) ctx_reqs = int(m.group(2)) gen_reqs = int(m.group(3)) - if ctx_reqs == target_ctx_reqs and gen_reqs == target_gen_reqs and iter_in_window(iter_id): + if ctx_reqs == target_ctx_reqs and gen_reqs == target_gen_reqs: eager_iters.append(IterInfo(start, end, iter_id)) eager_iters = sorted(eager_iters)[args.warmup_times :] iter_id_list = [it.iter_id for it in eager_iters] @@ -164,11 +146,11 @@ def iter_in_window(iter_id: int) -> bool: df = pd.read_sql_query(query, graph_conn, params=(graph_event_id_NvtxPushPopRange,)) graph_iters: list[IterInfo] = [] for start, end, text in df.itertuples(index=False): - if m := FORWARD_STEP_RE.match(text): + if m := re.match(r"^\[Executor\] _forward_step (\d+): (\d+) ctx reqs, (\d+) gen reqs", text): iter_id = int(m.group(1)) ctx_reqs = int(m.group(2)) gen_reqs = int(m.group(3)) - if ctx_reqs == target_ctx_reqs and gen_reqs == target_gen_reqs and iter_in_window(iter_id): + if ctx_reqs == target_ctx_reqs and gen_reqs == target_gen_reqs: graph_iters.append(IterInfo(start, end, iter_id)) graph_iters = sorted(graph_iters)[args.warmup_times :] graph_iter_id_list = [it.iter_id for it in graph_iters] diff --git a/examples/layer_wise_benchmarks/parser_utils.py b/examples/layer_wise_benchmarks/parser_utils.py index 500f8cf90667..95491cc2f7f7 100644 --- a/examples/layer_wise_benchmarks/parser_utils.py +++ b/examples/layer_wise_benchmarks/parser_utils.py @@ -24,29 +24,6 @@ def lazy_convert_sqlite(nsys_rep_file_path, sqlite_file_path): ) -def require_cuda_kernel_events(conn, trace_path): - """Fail fast with an actionable error when a trace has no CUDA kernel events. - - A trace with NVTX ranges but no CUPTI_ACTIVITY_KIND_KERNEL table means the - workload ran while nsys was not recording CUDA activity. This happens on - nodes where the host driver cannot natively serve the container's CUDA - stack (CUDA forward compatibility mode), under which profiling tools are - unsupported. Without this check the parsers fail later with an opaque - "no such table" database error. - """ - num_kernel_tables = conn.execute( - "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table'" - " AND name = 'CUPTI_ACTIVITY_KIND_KERNEL'" - ).fetchone()[0] - if num_kernel_tables == 0: - raise RuntimeError( - f"{trace_path} contains no CUDA kernel events. The workload ran, but nsys" - " recorded no CUDA activity; this typically means CUDA tracing was not" - " functional on this node, e.g. because the host driver cannot natively" - " serve the container's CUDA stack (CUDA forward compatibility mode)." - ) - - parser_keywords = [ ("cuBLASGemm", "nvjet"), ("cutlassGroupGemm", "cutlass::device_kernel bool | Literal["auto"]: - if value == "auto": - return "auto" - if value == "true": - return True - if value == "false": - return False - raise argparse.ArgumentTypeError("expected one of: auto, true, false") - - def add_llm_args(parser): parser.add_argument('--model_dir', type=str, @@ -157,11 +131,9 @@ def add_llm_args(parser): action='store_true') parser.add_argument( '--use_kv_cache_manager_v2', - default='auto', - type=_parse_kv_cache_manager_v2, - metavar='{auto,true,false}', - help= - 'Whether to use KVCacheManagerV2 for KV cache management (PyTorch backend). Defaults to model-specific auto selection.', + default=False, + action='store_true', + help='Use KVCacheManagerV2 for KV cache management (PyTorch backend).', ) # Runtime @@ -310,9 +282,6 @@ def setup_llm(args, **kwargs): relaxed_topk=args.relaxed_topk, relaxed_delta=args.relaxed_delta, mtp_eagle_one_model=args.use_one_model, - use_dynamic_tree=args.use_dynamic_tree, - dynamic_tree_max_topK=args.dynamic_tree_max_topK, - max_total_draft_tokens=args.max_total_draft_tokens, speculative_model=args.model_dir) elif spec_decode_algo == "EAGLE3": spec_config = Eagle3DecodingConfig( diff --git a/examples/llm-api/quickstart_multimodal.py b/examples/llm-api/quickstart_multimodal.py index 471986294304..e0c7aca5f17d 100644 --- a/examples/llm-api/quickstart_multimodal.py +++ b/examples/llm-api/quickstart_multimodal.py @@ -7,7 +7,6 @@ from tensorrt_llm.inputs import default_multimodal_input_loader from tensorrt_llm.inputs.registry import MULTIMODAL_PLACEHOLDER_REGISTRY -from tensorrt_llm.llmapi import MultimodalConfig from tensorrt_llm.tools.importlib_utils import import_custom_module_from_dir example_medias_and_prompts = { @@ -187,14 +186,9 @@ def main(): lora_config.max_loras = 2 lora_config.max_cpu_loras = 2 - multimodal_config = None - if args.video_pruning_rate is not None: - multimodal_config = MultimodalConfig( - video_pruning_rate=args.video_pruning_rate) - llm, sampling_params = setup_llm(args, lora_config=lora_config, - multimodal_config=multimodal_config) + video_pruning_rate=args.video_pruning_rate) image_format = args.image_format if args.model_type is not None: diff --git a/examples/llm-eval/lm-eval-harness/README.md b/examples/llm-eval/lm-eval-harness/README.md new file mode 100644 index 000000000000..c3854654dd0d --- /dev/null +++ b/examples/llm-eval/lm-eval-harness/README.md @@ -0,0 +1,67 @@ +# Evaluation scripts for LLM tasks + +This folder includes code to use the [LM-Eval-Harness](https://github.com/EleutherAI/lm-evaluation-harness), a unified framework to test generative language models on a large number of different evaluation tasks. The supported eval tasks are [here](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks). + +The following instructions show how to evaluate TRT-LLM engines with the benchmark. + +## Instructions + +### TRT-LLM API + +Build the TRT-LLM engine using `trtllm-build`. + +Install the `lm_eval` package in the `requirements.txt` file in this folder. + +Run the evaluation script with the following command: + +```sh +python lm_eval_tensorrt_llm.py --model trt-llm \ + --model_args tokenizer=,model=,chunk_size= \ + --tasks +``` + +In the LM-Eval-Harness, model args are submitted as a comma-separated list of the form `arg=value`. The `trt-llm` model supports the following `model_args`: + +| Name | Description | Default Value | +|--------------------------|-------------------------------------------------------------------|----------------| +| tokenizer | directory containing the HF tokenizer. | | +| model | directory containing the TRTLLM engine or torch model. | | +| max_gen_toks | max number of tokens to generate (if not specified in gen_kwargs) | 256 | +| chunk_size | number of async requests to send at once to the engine | 200 | +| max_tokens_kv_cache | max tokens in paged KV cache | None | +| free_gpu_memory_fraction | KV cache free GPU memory fraction | 0.9 | +| trust_remote_code | trust remote code; use if necessary to set up the tokenizer | False | +| tp | tensor parallel size (for torch backend) | no. of workers | +| use_cuda_graph | enable CUDA graph | True | +| max_context_length | maximum context length for evaluation | None | +| moe_expert_parallel_size | expert parallel size for MoE models | None | +| moe_backend | backend for MoE models (e.g., "TRTLLM") | "TRTLLM" | + +### Torch backend + +Install the `lm_eval` package in the `requirements.txt` file in this folder. + +Run the evaluation script with the same command as above, but include `backend=torch` in the `model_args`. For example: + +```sh +python lm_eval_tensorrt_llm.py --model trt-llm \ + --model_args model=,backend=torch,chunk_size= \ + --tasks +``` + +### trtllm-serve + +Build the TRT-LLM engine using `trtllm-build` and deploy with `trtllm-serve`. + +Install the `lm_eval` package in the `requirements.txt` file in this folder. + +Run the evaluation script with the following command: + +```sh +python lm_eval_tensorrt_llm.py --model local-completions \ + --model_args base_url=http://${HOST_NAME}:8001/v1/completions,model=,tokenizer= \ + --tasks \ + --batch_size <#> +``` + +Because `trtllm-serve` is OpenAI API compatible, we can use the `local-completions` model built in to `lm_eval`, which supports [these model_args](https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.7/lm_eval/models/openai_completions.py#L12). diff --git a/examples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.py b/examples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.py new file mode 100644 index 000000000000..1738242267d9 --- /dev/null +++ b/examples/llm-eval/lm-eval-harness/lm_eval_tensorrt_llm.py @@ -0,0 +1,317 @@ +# SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import gc +import json +import logging +import os +import signal +import threading +import time +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +import torch +import torch.nn.functional as F +import transformers +from lm_eval.__main__ import cli_evaluate +from lm_eval.api.model import TemplateLM +from lm_eval.api.registry import register_model +from packaging.version import parse +from tqdm import tqdm + +import tensorrt_llm +from tensorrt_llm import LLM as TORCH_LLM +from tensorrt_llm._tensorrt_engine import LLM as TRT_LLM +from tensorrt_llm.bindings.executor import DecodingConfig +from tensorrt_llm.llmapi import KvCacheConfig as TRT_KvCacheConfig +from tensorrt_llm.llmapi import RequestOutput, SamplingParams +from tensorrt_llm.llmapi.llm_args import MoeConfig + +logger = logging.getLogger(__name__) + + +@register_model("trt-llm") +class TRTLLMEvalBase(TemplateLM): + + def __init__( + self, + model: str, + tokenizer: Optional[str] = None, + tp: int = 0, # tensor_parallel_size + max_gen_toks: int = 256, + chunk_size: int = 200, + max_tokens_kv_cache: Optional[int] = None, + free_gpu_memory_fraction: float = 0.9, + trust_remote_code: bool = False, + use_cuda_graph: bool = True, + backend: str = 'trt', + max_context_length: Optional[int] = None, + moe_expert_parallel_size: Optional[int] = None, + moe_backend: Optional[str] = "TRTLLM", + enable_chunked_prefill: bool = False, + max_num_tokens: Optional[int] = None, + **kwargs, + ): + # initialize TemplateLM, copied from TemplateAPI + super().__init__() + assert isinstance(model, str) + assert parse(tensorrt_llm.__version__) >= parse("0.15.0") + + self.max_gen_toks = max_gen_toks + self.chunk_size = chunk_size + self.backend = backend + self.max_context_length = max_context_length + self.moe_expert_parallel_size = moe_expert_parallel_size + self.moe_backend = moe_backend + trt_kv_cache_config = TRT_KvCacheConfig(enable_block_reuse=False) + trt_kv_cache_config.free_gpu_memory_fraction = free_gpu_memory_fraction + if max_tokens_kv_cache is not None: + trt_kv_cache_config.max_tokens = max_tokens_kv_cache + + if tokenizer is None: + # Assume the tokenizer is stored in the model_dir if not specified. + tokenizer = model + logger.info(f"Tokenizer: {tokenizer}") + self.tokenizer = transformers.AutoTokenizer.from_pretrained( + tokenizer, trust_remote_code=trust_remote_code) + + if self.tokenizer.pad_token_id is None: + self.tokenizer.pad_token_id = self.tokenizer.eos_token_id + + if self.backend == 'torch': + kwargs.pop('batch_size') + if tp < 1: + tp = torch.cuda.device_count() + + pytorch_config_params = { + 'cuda_graph_config': {} if use_cuda_graph else None, + "print_iter_log": False, + 'moe_config': MoeConfig(backend=self.moe_backend) + } + + # stop words not currently supported by torch backend + self.use_stop_words = False + + self.llm = TORCH_LLM( + model=model, + tensor_parallel_size=tp, + trust_remote_code=trust_remote_code, + enable_chunked_prefill=enable_chunked_prefill, + max_num_tokens=max_num_tokens, + **pytorch_config_params, + tokenizer=self.tokenizer, + kv_cache_config=trt_kv_cache_config, + moe_expert_parallel_size=self.moe_expert_parallel_size, + **kwargs) + logger.info("Loaded TRT-LLM Torch engine") + else: + with open(Path(model) / "config.json", "r") as engine_config_file: + engine_config = json.load(engine_config_file) + build_config = engine_config["build_config"] + world_size = (engine_config.get("pretrained_config", {}).get( + "mapping", {}).get("world_size", 1)) + if max_tokens_kv_cache is None: + max_tokens_kv_cache = build_config[ + "max_seq_len"] * build_config["max_batch_size"] + self.gather_context_logits = build_config.get( + "gather_context_logits", False) + + medusa_choices = kwargs[ + 'medusa_choices'] if 'medusa_choices' in kwargs else None + kwargs = {} + if medusa_choices is not None: + decoding_config = DecodingConfig() + decoding_config.medusa_choices = medusa_choices + kwargs["decoding_config"] = decoding_config + assert world_size == 1, "decoding_config does not support multi TP in HLAPI." + + self.llm = TRT_LLM(model=model, + tokenizer=self.tokenizer, + kv_cache_config=trt_kv_cache_config, + **kwargs) + self.max_length = build_config['max_seq_len'] - 1 + logger.info("Loaded TRT-LLM engine") + + @property + def eot_token_id(self) -> int: + return self.llm.tokenizer.eos_token_id + + def tok_encode(self, string, add_special_tokens=False, **kwargs): + return self.llm.tokenizer.encode(string, + add_special_tokens=add_special_tokens, + **kwargs) + + def _loglikelihood_tokens( + self, + requests: List[Any], + disable_tqdm: bool = False) -> List[Tuple[float, bool]]: + """Compute the log likelihood of the continuation given the context.""" + if self.backend == 'torch': + raise NotImplementedError( + 'Torch backend does not return context logits yet') + + num_r = len(requests) + desc = "Processing loglikelihood requests" + sampling_params = SamplingParams(max_tokens=1, + return_context_logits=True) + + # process requests + futures: Dict[int, RequestOutput] = {} + results = [] + for i, request in tqdm(enumerate(requests), + desc=desc, + total=num_r, + disable=disable_tqdm): + # asynchronously submit a chunk of requests ahead of time... + if i % self.chunk_size == 0: + for j in range(i, min(i + self.chunk_size, num_r)): + prompt_ids = requests[j][1] + requests[j][2] + futures[j] = self.llm.generate_async( + prompt_ids, sampling_params) + + # process the output of the request i + r_out: RequestOutput = futures.pop(i).result() + + # check continuation portion of the prompt + # NOTE: context_logits are offset by 1 since they predict future token + ctxlen = len(request[1]) + token_ids_cont = request[2] + logits_cont = r_out.context_logits[ctxlen - 1:-1] # [sl, vocab] + logprobs_cont = F.log_softmax(logits_cont, dim=-1) # [sl, vocab] + top_tokens_cont = logprobs_cont.argmax(dim=-1).tolist() # [sl] + + # compute logprob and check for greedy + logprob_sum = sum(logprobs_cont[list(range(len(logprobs_cont))), + token_ids_cont]).item() + is_greedy = top_tokens_cont == token_ids_cont + + results.append((logprob_sum, is_greedy)) + + # clear response + del r_out + + return results + + def loglikelihood_rolling(self, requests, disable_tqdm: bool = False): + raise NotImplementedError + + def generate_until(self, + requests: List[Any], + disable_tqdm: bool = False) -> List[str]: + # some book-keeping and parameters... + num_r = len(requests) + desc = "Processing generate requests" + + if self.max_context_length is not None: + """ + Create updated_requests to contain qualified requests with the context length <= max_context_length. + Unqualified requests cannot simply be dropped as lm-eval library requires the number of requests to be the same. + + Note: The final score will drop if disqualified requests exist. + """ + request_idx_to_replace = [] + qualified_requests = [] + updated_requests = [] + for i, request in enumerate(requests): + context, gen_kwargs = request.args + if len(self.tok_encode(context)) > self.max_context_length: + request_idx_to_replace.append(i) + else: + qualified_requests.append(request) + + assert len( + qualified_requests + ) > 1, "No requests with context length <= max_context_length. Cannot run the evaluation." + if len(request_idx_to_replace) > 0: + print( + f"Warning: {len(request_idx_to_replace)} requests with context length > max_context_length will be replaced. The final score will drop." + ) + + for i, request in enumerate(requests): + if i in request_idx_to_replace: + # Replace the requests with context length > max_context_length with the qualified requests + updated_requests.append( + qualified_requests[i % len(qualified_requests)]) + else: + updated_requests.append(request) + assert len( + updated_requests + ) == num_r, "Number of updated requests does not match the number of requests." + requests = updated_requests + + def _get_sp(gen_kwargs): + k_mapping = { + "temperature": "temperature", + "top_p": "top_p", + "max_gen_toks": "max_tokens", + "until": "stop", + } + kwargs_mapped = { + k_sp: gen_kwargs[k_gen] + for k_gen, k_sp in k_mapping.items() if k_gen in gen_kwargs + } + if "max_tokens" not in kwargs_mapped: + kwargs_mapped["max_tokens"] = self.max_gen_toks + return SamplingParams(**kwargs_mapped) + + # process requests + futures: Dict[int, RequestOutput] = {} + future_stop_words: Dict[int, RequestOutput] = {} + results = [] + for i, _ in tqdm(enumerate(requests), + desc=desc, + total=num_r, + disable=disable_tqdm): + # asynchronously submit a chunk of requests ahead of time... + if i % self.chunk_size == 0: + for j in range(i, min(i + self.chunk_size, num_r)): + context, gen_kwargs = requests[j].args + prompt_ids = self.tok_encode(context) + if self.max_context_length is not None: + assert len( + prompt_ids + ) <= self.max_context_length, f"Prompt length > {self.max_context_length}, {len(prompt_ids)}, should be filtered out." + kwargs_mapped = _get_sp(gen_kwargs) + futures[j] = self.llm.generate_async( + prompt_ids, kwargs_mapped) + del kwargs_mapped + future_stop_words[j] = gen_kwargs["until"] + + # process the output of the request i + r_out: RequestOutput = futures.pop(i).result() + stop_words = future_stop_words.pop(i) + txt = r_out.outputs[0].text + if stop_words: + for word in stop_words: + word_index = txt.find(word) + if word_index >= 0: + txt = txt[:word_index] + results.append(txt) + + return results + + +if __name__ == "__main__": + cli_evaluate() + # Force clean up the LLM instance and void hanging. + gc.collect() + + # Force terminate in case gc.collect() is not enough. + def _terminate(): + time.sleep(10) + os.kill(os.getpid(), signal.SIGTERM) + + termination_thread = threading.Thread(target=_terminate, daemon=True) + termination_thread.start() diff --git a/examples/llm-eval/lm-eval-harness/requirements.txt b/examples/llm-eval/lm-eval-harness/requirements.txt new file mode 100644 index 000000000000..44a3383f612d --- /dev/null +++ b/examples/llm-eval/lm-eval-harness/requirements.txt @@ -0,0 +1 @@ +lm_eval[api]==0.4.7 diff --git a/examples/mmlu.py b/examples/mmlu.py new file mode 100644 index 000000000000..82bb9a7ec9ad --- /dev/null +++ b/examples/mmlu.py @@ -0,0 +1,479 @@ +# SPDX-FileCopyrightText: Copyright (c) 2020 Dan Hendrycks +# SPDX-FileCopyrightText: Copyright (c) 2023 Deep Cognition and Language Research (DeCLaRe) Lab +# SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 and MIT +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Adapted from https://github.com/declare-lab/instruct-eval +Helper script to compare TRTLLM and HF models on the MMLU dataset. +Example usage: + mkdir data; wget https://people.eecs.berkeley.edu/~hendrycks/data.tar -O data/mmlu.tar + tar -xf data/mmlu.tar -C data && mv data/data data/mmlu + + python mmlu.py --hf_model_dir --engine_dir --test_trt_llm + python mmlu.py --hf_model_dir --engine_dir --test_hf +""" + +import argparse +import os +import random + +import numpy as np +import pandas as pd +import torch +import torch.nn as nn +from tqdm import tqdm +from transformers import (AutoConfig, AutoModel, AutoModelForCausalLM, + AutoModelForSeq2SeqLM, AutoTokenizer, + GenerationConfig) +from utils import (add_common_args, load_tokenizer, prepare_enc_dec_inputs, + read_is_enc_dec, read_model_name) + +import tensorrt_llm +from tensorrt_llm.runtime import PYTHON_BINDINGS, ModelRunner + +if PYTHON_BINDINGS: + from tensorrt_llm.runtime import ModelRunnerCpp + +os.environ["TOKENIZERS_PARALLELISM"] = "false" + +DTYPE_STR_MAPPING = { + "fp32": torch.float32, + "fp16": torch.float16, + "bf16": torch.bfloat16, + "float32": torch.float32, + "float16": torch.float16, + "bfloat16": torch.bfloat16, +} +RAND_SEED = 1234 + + +def get_choices(): + return ["A", "B", "C", "D"] + + +def get_subcategories(): + return { + "abstract_algebra": ["math"], + "anatomy": ["health"], + "astronomy": ["physics"], + "business_ethics": ["business"], + "clinical_knowledge": ["health"], + "college_biology": ["biology"], + "college_chemistry": ["chemistry"], + "college_computer_science": ["computer science"], + "college_mathematics": ["math"], + "college_medicine": ["health"], + "college_physics": ["physics"], + "computer_security": ["computer science"], + "conceptual_physics": ["physics"], + "econometrics": ["economics"], + "electrical_engineering": ["engineering"], + "elementary_mathematics": ["math"], + "formal_logic": ["philosophy"], + "global_facts": ["other"], + "high_school_biology": ["biology"], + "high_school_chemistry": ["chemistry"], + "high_school_computer_science": ["computer science"], + "high_school_european_history": ["history"], + "high_school_geography": ["geography"], + "high_school_government_and_politics": ["politics"], + "high_school_macroeconomics": ["economics"], + "high_school_mathematics": ["math"], + "high_school_microeconomics": ["economics"], + "high_school_physics": ["physics"], + "high_school_psychology": ["psychology"], + "high_school_statistics": ["math"], + "high_school_us_history": ["history"], + "high_school_world_history": ["history"], + "human_aging": ["health"], + "human_sexuality": ["culture"], + "international_law": ["law"], + "jurisprudence": ["law"], + "logical_fallacies": ["philosophy"], + "machine_learning": ["computer science"], + "management": ["business"], + "marketing": ["business"], + "medical_genetics": ["health"], + "miscellaneous": ["other"], + "moral_disputes": ["philosophy"], + "moral_scenarios": ["philosophy"], + "nutrition": ["health"], + "philosophy": ["philosophy"], + "prehistory": ["history"], + "professional_accounting": ["other"], + "professional_law": ["law"], + "professional_medicine": ["health"], + "professional_psychology": ["psychology"], + "public_relations": ["politics"], + "security_studies": ["politics"], + "sociology": ["culture"], + "us_foreign_policy": ["politics"], + "virology": ["health"], + "world_religions": ["philosophy"], + } + + +def get_categories(): + return { + "STEM": [ + "physics", + "chemistry", + "biology", + "computer science", + "math", + "engineering", + ], + "humanities": ["history", "philosophy", "law"], + "social sciences": [ + "politics", + "culture", + "economics", + "geography", + "psychology", + ], + "other (business, health, misc.)": ["other", "business", "health"], + } + + +def format_subject(subject): + line = subject.split("_") + s = "" + for entry in line: + s += " " + entry + return s + + +def format_example(df, idx, include_answer=True): + prompt = df.iloc[idx, 0] + k = df.shape[1] - 2 + for j in range(k): + prompt += "\n{}. {}".format(get_choices()[j], df.iloc[idx, j + 1]) + prompt += "\nAnswer:" + if include_answer: + prompt += " {}\n\n".format(df.iloc[idx, k + 1]) + return prompt + + +def gen_prompt(train_df, subject, k=-1): + prompt = "The following are multiple choice questions (with answers) about {}.\n\n".format( + format_subject(subject)) + if k == -1: + k = train_df.shape[0] + for i in range(k): + prompt += format_example(train_df, i) + return prompt + + +def evaluate(args, subject, pipeline, dev_df, test_df): + rank = tensorrt_llm.mpi_rank() + cors = [] + all_probs = [] + for i in range(test_df.shape[0]): + if i >= args.max_ite: + break + # get prompt and make sure it fits + k = args.ntrain + prompt_end = format_example(test_df, i, include_answer=False) + train_prompt = gen_prompt(dev_df, subject, k) + prompt = train_prompt + prompt_end + + while not pipeline.check_valid_length(prompt) and k > 0: + k -= 1 + train_prompt = gen_prompt(dev_df, subject, k) + prompt = train_prompt + prompt_end + + label = test_df.iloc[i, test_df.shape[1] - 1] + pred = pipeline(prompt) + + if rank == 0: + probs = [0 for _ in get_choices()] + cor = pred.strip().startswith(label) + cors.append(cor) + all_probs.append(probs) + + if rank == 0: + acc = np.mean(cors) + cors = np.array(cors) + + all_probs = np.array(all_probs) + print("Average accuracy {:.3f} - {}".format(acc, subject)) + + return cors, acc, all_probs + else: + return None, 0, None + + +def get_tokenizer(ckpt_path, max_seq_len): + print(f"Initializing tokenizer from {ckpt_path}") + tokenizer = AutoTokenizer.from_pretrained( + ckpt_path, + model_max_length=max_seq_len, + padding_side="left", + trust_remote_code=True, + ) + tokenizer.pad_token = tokenizer.eos_token + + return tokenizer + + +class Pipeline: + + def __init__(self, tokenizer, model, model_name, pad_id, end_id, + max_attention_window_size, is_enc_dec, hf_model_dir, + engine_dir): + self.tokenizer = tokenizer + self.model = model + self.model_name = model_name + self.pad_id = pad_id + self.end_id = end_id + self.max_attention_window_size = max_attention_window_size + self.output_len = 2 + self.is_enc_dec = is_enc_dec + self.decoder_start_token_id = None + self.engine_dir = engine_dir + if self.is_enc_dec: + self.decoder_start_token_id = AutoConfig.from_pretrained( + hf_model_dir).decoder_start_token_id + + def __call__(self, prompt): + rank = tensorrt_llm.mpi_rank() + # Run the model in batch size 1 and beam size 1 + inputs = self.tokenizer.encode(prompt, return_tensors="pt").squeeze(0) + batch_input_ids = [inputs] + + # For multi-choice tasks like MMLU, we don't need to adjust following parameters + output_len = self.output_len + top_k = 1 + top_p = 0.0 + + input_lengths = [x.size(0) for x in batch_input_ids] + + with torch.no_grad(): + if isinstance(self.model, nn.Module): + # Left padding for HF + max_length = max(input_lengths) + paddings = [ + torch.ones(max_length - l, dtype=torch.int32) * self.pad_id + for l in input_lengths + ] + batch_input_ids = [ + torch.cat([pad, x]) + for x, pad in zip(batch_input_ids, paddings) + ] + batch_input_ids = torch.stack(batch_input_ids) + batch_input_ids = batch_input_ids.cuda() + if self.is_enc_dec: + batch_decoder_input_ids = torch.IntTensor( + [[self.decoder_start_token_id]]).to('cuda') + batch_decoder_input_ids = batch_decoder_input_ids.repeat( + (batch_input_ids.shape[0], 1)) + + with torch.no_grad(): + # Use default temperature and top_k + outputs = self.model.generate( + batch_input_ids, + max_new_tokens=output_len, + top_k=top_k, + decoder_input_ids=batch_decoder_input_ids + if self.is_enc_dec else None) + if not self.is_enc_dec: + output_ids = outputs[0, input_lengths[0]:] + else: + output_ids = outputs[0] + + elif isinstance(self.model, ModelRunnerCpp) or isinstance( + self.model, ModelRunner): + if self.is_enc_dec: + encoder_input_ids, encoder_input_features, encoder_output_lengths, decoder_input_ids = prepare_enc_dec_inputs( + batch_input_ids, self.model_name, self.engine_dir, None) + + outputs = self.model.generate( + batch_input_ids=decoder_input_ids + if self.is_enc_dec else batch_input_ids, + encoder_input_ids=encoder_input_ids + if self.is_enc_dec else None, + encoder_input_features=encoder_input_features + if self.is_enc_dec else None, + encoder_output_lengths=encoder_output_lengths + if self.is_enc_dec else None, + max_new_tokens=output_len, + max_attention_window_size=self.max_attention_window_size, + end_id=self.end_id, + pad_id=self.pad_id, + top_k=top_k, + top_p=top_p, + ) + torch.cuda.synchronize() + if rank == 0: + if not self.is_enc_dec: + output_ids = outputs[0, 0, input_lengths[0]:] + else: + output_ids = outputs[0, 0] + if rank == 0: + return self.tokenizer.decode(output_ids, skip_special_tokens=True) + else: + return None + + def check_valid_length(self, prompt): + if isinstance(self.model, nn.Module): + return True + input_len = len(self.tokenizer.encode(prompt)) + return input_len <= self.model.max_input_len and input_len + self.output_len <= self.model.max_seq_len + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--data_dir", + type=str, + default="data/mmlu", + help=("Path to the data directory. If not available, " + "download https://people.eecs.berkeley.edu/~hendrycks/data.tar"), + ) + parser.add_argument("--ntrain", type=int, default=5) + parser.add_argument("--max_input_length", type=int, default=2048) + parser.add_argument("--test_trt_llm", action="store_true") + parser.add_argument("--test_hf", action="store_true") + parser.add_argument('--check_accuracy', action='store_true') + parser.add_argument('--accuracy_threshold', type=float, default=30) + parser.add_argument('--max_ite', type=int, default=10000000) + parser = add_common_args(parser) + + args = parser.parse_args() + + return args + + +def main(): + args = parse_args() + if args.tokenizer_dir is None: + args.tokenizer_dir = args.hf_model_dir + random.seed(RAND_SEED) + np.random.seed(RAND_SEED) + runtime_rank = tensorrt_llm.mpi_rank() + + os.path.dirname(os.path.abspath(__file__)) + data_fullpath = os.path.join(args.data_dir, "test") + + subjects = sorted([ + f.split("_test.csv")[0] for f in os.listdir(data_fullpath) + if "_test.csv" in f + ]) + + all_cors = [] + subcat_cors = { + subcat: [] + for subcat_lists in get_subcategories().values() + for subcat in subcat_lists + } + cat_cors = {cat: [] for cat in get_categories()} + + # different handling if encoder-decoder models + is_enc_dec = read_is_enc_dec( + args.engine_dir if not args.test_hf else args.hf_model_dir, + args.test_hf) + + model_name, model_version = read_model_name( + (args.engine_dir if not is_enc_dec else os.path.join( + args.engine_dir, 'encoder')) + if not args.test_hf else args.hf_model_dir, args.test_hf) + + tokenizer, pad_id, end_id = load_tokenizer( + tokenizer_dir=args.tokenizer_dir, + vocab_file=args.vocab_file, + model_name=model_name, + model_version=model_version, + ) + + if args.test_trt_llm: + assert not args.test_hf, "Cannot test both TRT-LLM and HF" + runner_cls = ModelRunner if not PYTHON_BINDINGS else ModelRunnerCpp + runner_kwargs = {} + if PYTHON_BINDINGS: + runner_kwargs.update(max_beam_width=1) + runner_kwargs.update( + is_enc_dec=is_enc_dec, + max_tokens_in_paged_kv_cache=args.max_tokens_in_paged_kv_cache, + kv_cache_enable_block_reuse=args.kv_cache_enable_block_reuse, + kv_cache_free_gpu_memory_fraction=args. + kv_cache_free_gpu_memory_fraction, + cross_kv_cache_fraction=args.cross_kv_cache_fraction + if is_enc_dec else None, + enable_chunked_context=args.enable_chunked_context, + multi_block_mode=args.multi_block_mode) + model = runner_cls.from_dir(engine_dir=args.engine_dir, + rank=runtime_rank, + **runner_kwargs) + else: + assert args.test_hf, "Must test either TRT-LLM or HF" + if 'GLM' in model_name and model_version == 'glm': + auto_model_cls = AutoModelForSeq2SeqLM + elif 'GLM' in model_name and model_version == 'chatglm': + auto_model_cls = AutoModel + elif is_enc_dec: + auto_model_cls = AutoModelForSeq2SeqLM + else: + auto_model_cls = AutoModelForCausalLM + model = auto_model_cls.from_pretrained( + args.hf_model_dir, + trust_remote_code=True, + dtype=DTYPE_STR_MAPPING[args.hf_data_type], + device_map="auto" if args.hf_device_map_auto else None, + ) + if not args.hf_device_map_auto: + model.cuda() + if model_name == "qwen": + model.generation_config = GenerationConfig.from_pretrained( + args.hf_model_dir, trust_remote_code=True) + + pipeline = Pipeline(tokenizer, model, model_name, pad_id, end_id, + args.max_attention_window_size, is_enc_dec, + args.hf_model_dir, args.engine_dir) + + for subject in tqdm(subjects): + dev_df = pd.read_csv(os.path.join(args.data_dir, "dev", + subject + "_dev.csv"), + header=None)[:args.ntrain] + test_df = pd.read_csv(os.path.join(args.data_dir, "test", + subject + "_test.csv"), + header=None) + + cors, acc, probs = evaluate(args, subject, pipeline, dev_df, test_df) + subcats = get_subcategories()[subject] + for subcat in subcats: + subcat_cors[subcat].append(cors) + for key in get_categories().keys(): + if subcat in get_categories()[key]: + cat_cors[key].append(cors) + all_cors.append(cors) + + if runtime_rank == 0: + for subcat in subcat_cors: + acc = np.mean(np.concatenate(subcat_cors[subcat])) * 100 + print(f"Average accuracy {acc:.2f} - {subcat}") + + for cat in cat_cors: + acc = np.mean(np.concatenate(cat_cors[cat])) * 100 + print(f"Average accuracy {acc:.2f} - {cat}") + + weighted_acc = np.mean(np.concatenate(all_cors)) * 100 + print(f"MMLU weighted average accuracy: {weighted_acc:.2f}") + + if args.check_accuracy: + assert weighted_acc >= args.accuracy_threshold, f"Expected accuracy >= {args.accuracy_threshold} while got {weighted_acc}" + return weighted_acc + + +if __name__ == "__main__": + main() diff --git a/examples/models/contrib/arctic/README.md b/examples/models/contrib/arctic/README.md new file mode 100644 index 000000000000..d346a3644517 --- /dev/null +++ b/examples/models/contrib/arctic/README.md @@ -0,0 +1,94 @@ +# Arctic + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document shows how to build and run a [Arctic](https://huggingface.co/Snowflake/snowflake-arctic-instruct) model in TensorRT-LLM. + +The TensorRT LLM Arctic implementation is based on the LLaMA model, with Mixture of Experts (MoE) enabled. The implementation can +be found in [llama/model.py](../../../../tensorrt_llm/models/llama/model.py). +See the LLaMA example [`examples/models/core/llama`](../../core/llama) for details. + +- [Arctic](#arctic) + - [Download model checkpoints](#download-model-checkpoints) + - [TensorRT LLM workflow](#tensorrt-llm-workflow) + - [Apply FP8 PTQ](#apply-fp8-ptq) + - [Build TensorRT engine](#build-tensorrt-engine) + - [Run Engine](#run-engine) + - [OOTB](#ootb) + +## Download model checkpoints + +First, download the HuggingFace BF16 checkpoints of Arctic model. + +**CAVEAT: this model is a pretty large Mixture-of-Experts (MoE) model, which has nearly 500B parameters and requires around 900GB disk space for storage. Please make sure you have enough space before proceeding.** + +```bash +HF_MODEL="arctic" +git clone https://huggingface.co/Snowflake/snowflake-arctic-instruct tmp/hf_checkpoints/${HF_MODEL} + +``` + +## TensorRT LLM workflow +Next, we use the general quantization script `quantize.py` to convert the checkpoints in FP8, and build the model with `trtllm-build` on multi-GPUs. In the example below, we use Tensor Parallelism (TP) across 8 GPUs. + +**Note: for such large model, it is deemed necessary to apply Post-Training Quantization (PTQ) methods on the model weights to deploy it on a cluster node, e.g., 8xH100 GPUs. In this example, we demonstrate the FP8 quantization workflow, which is supported on Hopper-and-next GPU architectures. For instructions of other PTQ methods other than FP8, please refer to the LLaMA or Mixtral examples.** + + +Set environment variables and necessary directory: + +```bash +PREC_RAW="bfloat16" +PREC_QUANT="fp8" +TP=8 +ENGINE="${HF_MODEL}_${PREC_QUANT}_tp${TP}" + +mkdir -p tmp/trt_engines +``` + +### Apply FP8 PTQ + +Notes: +- currently quantize.py does not support for Expert Parallelism (EP) mode yet. User should use `../../core/llama/convert_checkpoint.py` and specify `--moe_ep_size 1` instead, if needed. +- TensorRT LLM uses static quantization methods, which is expected to be faster at runtime as compared to dynamic quantization methods. This comes at a cost of an offline calibration step during quantization. `batch_size` and `calib_size` can be adjusted to shorten the calibration time. Please refer to ../quantization/README.md for explanation. +- **due to the large model size and the calibration step (which has to load the HuggingFace model and run forward passes), it is likely that you will need more number of GPUs during quantization step than the number of GPUs for engine building and final deployment. For example, using 16xH100 or 8xH200 for quantization & 8xH100 for deployment.** + +```bash +python ../../../quantization/quantize.py --model_dir tmp/hf_checkpoints/${HF_MODEL} \ + --dtype ${PREC_RAW} \ + --qformat ${PREC_QUANT} \ + --kv_cache_dtype ${PREC_QUANT} \ + --output_dir tmp/tllm_checkpoints/${ENGINE} \ + --batch_size 1 \ + --calib_size 128 \ + --tp_size ${TP} |& tee tmp/trt_engines/${ENGINE}_quantize.log + +``` + +### Build TensorRT engine +```bash +# Enable fp8 context fmha to get further acceleration by setting `--use_fp8_context_fmha enable` +# Use --workers to enable parallel build +trtllm-build --checkpoint_dir ./tmp/tllm_checkpoints/${ENGINE} \ + --output_dir ./tmp/trt_engines/${ENGINE} \ + --gpt_attention_plugin ${PREC_RAW} \ + --gemm_plugin ${PREC_RAW} \ + --workers ${TP} |& tee tmp/trt_engines/${ENGINE}_build.log +``` + +### Run Engine +Test your engine with the [run.py](../../../run.py) script: + +```bash +mpirun -n ${TP} --allow-run-as-root python ../../../run.py --engine_dir ./tmp/trt_engines/${ENGINE} --tokenizer_dir tmp/hf_checkpoints/${HF_MODEL} --max_output_len 20 --input_text "The future of AI is" |& tee tmp/trt_engines/${ENGINE}_run.log +``` + +For more examples see [`examples/models/core/llama/README.md`](../../core/llama/README.md) + + +### OOTB + +Arctic supports OOTB operation without the plugin, however this comes at a significant performance cost. Users should prefer using the plugin path whenever possible. diff --git a/examples/models/contrib/blip2/README.md b/examples/models/contrib/blip2/README.md new file mode 100644 index 000000000000..0e80ed4fd81b --- /dev/null +++ b/examples/models/contrib/blip2/README.md @@ -0,0 +1,4 @@ +This example has been moved to [`../multimodal`](../../../multimodal) and merged with other multimodal examples. +Please follow [`../multimodal/README.md`](../../../multimodal/README.md) for new instructions. + +**NOTICE:** This folder will be removed in v1.0 release. diff --git a/examples/models/contrib/chatglm-6b/README.md b/examples/models/contrib/chatglm-6b/README.md new file mode 100644 index 000000000000..00a8c2424665 --- /dev/null +++ b/examples/models/contrib/chatglm-6b/README.md @@ -0,0 +1,112 @@ +# ChatGLM + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document explains how to build the [ChatGLM-6B](https://huggingface.co/THUDM/chatglm-6b) models using TensorRT LLM and run on a single GPU, a single node with multiple GPUs or multiple nodes with multiple GPUs. + +- [ChatGLM](#chatglm) + - [Overview](#overview) + - [Support Matrix](#support-matrix) + - [Model comparison](#model-comparison) + - [Tokenizer and special tokens comparison](#tokenizer-and-special-tokens-comparison) + - [Usage](#usage) + - [1. Download repo and weights from HuggingFace Transformers](#1-download-repo-and-weights-from-huggingface-transformers) + - [2. Convert weights from HF Transformers to TensorRT LLM format](#2-convert-weights-from-hf-transformers-to-tensorrt-llm-format) + - [3. Build TensorRT engine(s)](#3-build-tensorrt-engines) + - [Enable plugins](#enable-plugins) + - [In-flight batching](#in-flight-batching) + - [4. Run inference](#4-run-inference) + - [Single node, single GPU](#single-node-single-gpu) + - [Single node, multi GPU](#single-node-multi-gpu) + - [5. Run summarization task](#5-run-summarization-task) + - [Weight Only quantization](#weight-only-quantization) + - [Smooth Quantization (SQ)](#smooth-quantization-sq) + - [Activation-aware Weight Quantization (AWQ)](#activation-aware-weight-quantization-awq) + - [FP8 Quantization](#fp8-quantization) + - [Benchmark](#benchmark) + + +## Overview + +The TensorRT LLM ChatGLM implementation can be found in [`tensorrt_llm/models/chatglm/model.py`](../../tensorrt_llm/models/chatglm/model.py). +The TensorRT LLM ChatGLM example code is located in [`examples/models/contrib/chatglm-6b`](./). There is one main file: + +* [`examples/models/core/glm-4-9b/convert_checkpoint.py`](../../../glm-4-9b/convert_checkpoint.py) to convert a checkpoint from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. + +In addition, there are two shared files in the parent folder [`examples`](../../../) for inference and evaluation: + +* [`../../../run.py`](../../../run.py) to run the inference on an input text; +* [`../../../summarize.py`](../../../summarize.py) to summarize the articles in the [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +## Support Matrix + +| Model Name | FP16 | FMHA | WO | SQ | AWQ | FP8 | TP | PP | ST | C++ | benchmark | IFB | +| :--------------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :-------: | :---: | +| chatglm_6b | Y | | Y | | | | Y | | Y | Y | Y | Y | +| glm_10b | Y | Y | Y | | Y | Y | Y | | Y | Y | Y | Y | + +* Model Name: the name of the model, the same as the name on HuggingFace +* FMHA: Fused MultiHead Attention (see introduction below) +* WO: Weight Only Quantization (int8 / int4) +* SQ: Smooth Quantization (int8) +* AWQ: Activation Aware Weight Quantization (int4) +* FP8: FP8 Quantization +* TP: Tensor Parallel +* PP: Pipeline Parallel +* ST: Strongly Typed +* C++: C++ Runtime +* benchmark: benchmark by python / C++ Runtime +* IFB: In-flight Batching (see introduction below) + +## Model comparison + +| Name | nL | nAH | nKH | nHW | nH | nF | nMSL | nV | bP2D | bBQKV | bBDense | Comments | +| :--------------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :----: | :---: | :---: | :-----: | :----------------------------------------------------------------- | +| chatglm_6b | 28 | 32 | 32 | 128 | 4096 | 16384 | 2048 | 130528 | Y | Y | Y | | +| glm_10b | 48 | 64 | 32 | 64 | 4096 | 16384 | 1024 | 50304 | Y | Y | Y | | + +* nL: number of layers +* nAH: number of attention heads +* nKH: number of kv heads (less than nAH if multi_query_attention is used) +* nHW: head width +* nH: hidden size +* nF: FFN hidden size +* nMSL: max sequence length (input + output) +* nV: vocabulary size +* bP2D: use position_encoding_2d (Y: Yes, N: No) +* bBQKV: use bias for QKV multiplication in self-attention +* bBDense: use bias for Dense multiplication in self-attention + +## Tokenizer and special tokens comparison + +| Name | Tokenizer | bos | eos | pad | cls | startofpiece | endofpiece | mask | smask | gmask | +| :--------------: | :--------------: | :----: | :----: | :---: | :---: | :----------: | :--------: | :----: | :---: | :----: | +| chatglm_6b | ChatGLMTokenizer | 130004 | 130005 | 3 | | 130004 | 130005 | 130000 | | 130001 | +| glm_10b | GLMGPT2Tokenizer | 50257 | 50256 | 50256 | 50259 | 50257 | 50258 | 50260 | 50264 | 50263 | + +## Usage + +The next section describe how to build the engine and run the inference demo. + +### 1. Download repo and weights from HuggingFace Transformers + +```bash +pip install -r requirements.txt +apt-get update +apt-get install git-lfs +rm -rf chatglm* + +# clone one or more models we want to build +git clone https://huggingface.co/THUDM/chatglm-6b chatglm_6b +git clone https://huggingface.co/THUDM/glm-10b glm_10b + +# replace tokenization file if using transformers-4.36.1 for model ChatGLM-6B (this might be needless in the future) +cp chatglm_6b/tokenization_chatglm.py chatglm_6b/tokenization_chatglm.py-backup +cp tokenization_chatglm.py chatglm_6b +``` + +For more example codes, please refer to the [examples/models/core/glm-4-9b/README.md](../../../glm-4-9b/README.md). diff --git a/examples/models/contrib/chatglm-6b/requirements.txt b/examples/models/contrib/chatglm-6b/requirements.txt new file mode 100644 index 000000000000..cdc65bf2bb38 --- /dev/null +++ b/examples/models/contrib/chatglm-6b/requirements.txt @@ -0,0 +1,8 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +datasets==3.1.0 +evaluate +protobuf +rouge_score +sentencepiece +tiktoken diff --git a/examples/models/contrib/chatglm-6b/tokenization_chatglm.py b/examples/models/contrib/chatglm-6b/tokenization_chatglm.py new file mode 100755 index 000000000000..8ae124909e98 --- /dev/null +++ b/examples/models/contrib/chatglm-6b/tokenization_chatglm.py @@ -0,0 +1,467 @@ +"""Tokenization classes for ChatGLM.""" +import os +from typing import Dict, List, Optional, Union + +import numpy as np +import sentencepiece as spm +from transformers.tokenization_utils import PreTrainedTokenizer +from transformers.tokenization_utils_base import BatchEncoding, EncodedInput +from transformers.utils import PaddingStrategy, logging + +logger = logging.get_logger(__name__) + +PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = { + "THUDM/chatglm-6b": 2048, +} + + +class TextTokenizer: + + def __init__(self, model_path): + self.sp = spm.SentencePieceProcessor() + self.sp.Load(model_path) + self.num_tokens = self.sp.vocab_size() + + def encode(self, text): + return self.sp.EncodeAsIds(text) + + def decode(self, ids: List[int]): + return self.sp.DecodeIds(ids) + + def tokenize(self, text): + return self.sp.EncodeAsPieces(text) + + def convert_tokens_to_string(self, tokens): + return self.sp.DecodePieces(tokens) + + def convert_tokens_to_ids(self, tokens): + return [self.sp.PieceToId(token) for token in tokens] + + def convert_token_to_id(self, token): + return self.sp.PieceToId(token) + + def convert_id_to_token(self, idx): + return self.sp.IdToPiece(idx) + + def __len__(self): + return self.num_tokens + + +class SPTokenizer: + + def __init__( + self, + vocab_file, + num_image_tokens=20000, + max_blank_length=80, + byte_fallback=True, + ): + assert vocab_file is not None + self.vocab_file = vocab_file + self.num_image_tokens = num_image_tokens + self.special_tokens = [ + "[MASK]", "[gMASK]", "[sMASK]", "", "", "", + "", "" + ] + self.max_blank_length = max_blank_length + self.byte_fallback = byte_fallback + self.text_tokenizer = TextTokenizer(vocab_file) + + def _get_text_tokenizer(self): + return self.text_tokenizer + + @staticmethod + def get_blank_token(length: int): + assert length >= 2 + return f"<|blank_{length}|>" + + @staticmethod + def get_tab_token(): + return f"<|tab|>" + + @property + def num_text_tokens(self): + return self.text_tokenizer.num_tokens + + @property + def num_tokens(self): + return self.num_image_tokens + self.num_text_tokens + + @staticmethod + def _encode_whitespaces(text: str, max_len: int = 80): + text = text.replace("\t", SPTokenizer.get_tab_token()) + for i in range(max_len, 1, -1): + text = text.replace(" " * i, SPTokenizer.get_blank_token(i)) + return text + + def _preprocess(self, text: str, linebreak=True, whitespaces=True): + if linebreak: + text = text.replace("\n", "") + if whitespaces: + text = self._encode_whitespaces(text, max_len=self.max_blank_length) + return text + + def encode(self, + text: str, + linebreak=True, + whitespaces=True, + add_dummy_prefix=True) -> List[int]: + """ + @param text: Text to encode. + @param linebreak: Whether to encode newline (\n) in text. + @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding. + @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text. + @param add_dummy_prefix: Whether to add dummy blank space in the beginning. + """ + text = self._preprocess(text, linebreak, whitespaces) + if not add_dummy_prefix: + text = "" + text + tmp = self._get_text_tokenizer().encode(text) + tokens = [x + self.num_image_tokens for x in tmp] + return tokens if add_dummy_prefix else tokens[2:] + + def postprocess(self, text): + text = text.replace("", "\n") + text = text.replace(SPTokenizer.get_tab_token(), "\t") + for i in range(2, self.max_blank_length + 1): + text = text.replace(self.get_blank_token(i), " " * i) + return text + + def decode(self, text_ids: List[int]) -> str: + ids = [int(_id) - self.num_image_tokens for _id in text_ids] + ids = [_id for _id in ids if _id >= 0] + text = self._get_text_tokenizer().decode(ids) + text = self.postprocess(text) + return text + + def decode_tokens(self, tokens: List[str]) -> str: + text = self._get_text_tokenizer().convert_tokens_to_string(tokens) + text = self.postprocess(text) + return text + + def tokenize(self, + text: str, + linebreak=True, + whitespaces=True, + add_dummy_prefix=True) -> List[str]: + """ + @param text: Text to encode. + @param linebreak: Whether to encode newline (\n) in text. + @param whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding. + @param special_tokens: Whether to encode special token ([MASK], [gMASK], etc.) in text. + @param add_dummy_prefix: Whether to add dummy blank space in the beginning. + """ + text = self._preprocess(text, linebreak, whitespaces) + if not add_dummy_prefix: + text = "" + text + tokens = self._get_text_tokenizer().tokenize(text) + return tokens if add_dummy_prefix else tokens[2:] + + def __getitem__(self, x: Union[int, str]): + if isinstance(x, int): + if x < self.num_image_tokens: + return "".format(x) + else: + return self.text_tokenizer.convert_id_to_token( + x - self.num_image_tokens) + elif isinstance(x, str): + if x.startswith("") and x[7:-1].isdigit(): + return int(x[7:-1]) + else: + return self.text_tokenizer.convert_token_to_id( + x) + self.num_image_tokens + else: + raise ValueError("The key should be str or int.") + + +class ChatGLMTokenizer(PreTrainedTokenizer): + """ + Construct a ChatGLM tokenizer. Based on byte-level Byte-Pair-Encoding. + + Args: + vocab_file (`str`): + Path to the vocabulary file. + """ + + vocab_files_names = {"vocab_file": "ice_text.model"} + max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES + model_input_names = ["input_ids", "attention_mask", "position_ids"] + + def __init__(self, + vocab_file, + do_lower_case=False, + remove_space=False, + bos_token='', + eos_token='', + end_token='', + mask_token='[MASK]', + gmask_token='[gMASK]', + padding_side="left", + pad_token="", + unk_token="", + num_image_tokens=20000, + **kwargs) -> None: # Fix for new transformers + + self.do_lower_case = do_lower_case + self.remove_space = remove_space + self.vocab_file = vocab_file + + self.bos_token = bos_token + self.eos_token = eos_token + self.end_token = end_token + self.mask_token = mask_token + self.gmask_token = gmask_token + + self.sp_tokenizer = SPTokenizer(vocab_file, + num_image_tokens=num_image_tokens) + + super().__init__( + do_lower_case=do_lower_case, # Fix for new transformers + remove_space=remove_space, + padding_side=padding_side, + bos_token=bos_token, + eos_token=eos_token, + end_token=end_token, + mask_token=mask_token, + gmask_token=gmask_token, + pad_token=pad_token, + unk_token=unk_token, + num_image_tokens=num_image_tokens, + **kwargs) + """ Initialization """ + + @property + def gmask_token_id(self) -> Optional[int]: + if self.gmask_token is None: + return None + return self.convert_tokens_to_ids(self.gmask_token) + + @property + def end_token_id(self) -> Optional[int]: + """ + `Optional[int]`: Id of the end of context token in the vocabulary. Returns `None` if the token has not been + set. + """ + if self.end_token is None: + return None + return self.convert_tokens_to_ids(self.end_token) + + @property + def vocab_size(self): + """ Returns vocab size """ + return self.sp_tokenizer.num_tokens + + def get_vocab(self): + """ Returns vocab as a dict """ + vocab = { + self._convert_id_to_token(i): i + for i in range(self.vocab_size) + } + vocab.update(self.added_tokens_encoder) + return vocab + + def preprocess_text(self, inputs): + if self.remove_space: + outputs = " ".join(inputs.strip().split()) + else: + outputs = inputs + + if self.do_lower_case: + outputs = outputs.lower() + + return outputs + + def _tokenize(self, text, **kwargs): + """ Returns a tokenized string. """ + text = self.preprocess_text(text) + + seq = self.sp_tokenizer.tokenize(text) + + return seq + + def convert_tokens_to_string(self, tokens: List[str]) -> str: + return self.sp_tokenizer.decode_tokens(tokens) + + def _decode(self, token_ids: Union[int, List[int]], **kwargs) -> str: + if isinstance(token_ids, int): + token_ids = [token_ids] + if len(token_ids) == 0: + return "" + if self.pad_token_id in token_ids: # remove pad + token_ids = list(filter((self.pad_token_id).__ne__, token_ids)) + return super()._decode(token_ids, **kwargs) + + def _convert_token_to_id(self, token): + """ Converts a token (str) in an id using the vocab. """ + return self.sp_tokenizer[token] + + def _convert_id_to_token(self, index): + """Converts an index (integer) in a token (str) using the vocab.""" + return self.sp_tokenizer[index] + + def save_vocabulary(self, save_directory, filename_prefix=None): + """ + Save the vocabulary and special tokens file to a directory. + + Args: + save_directory (`str`): + The directory in which to save the vocabulary. + filename_prefix (`str`, *optional*): + An optional prefix to add to the named of the saved files. + + Returns: + `Tuple(str)`: Paths to the files saved. + """ + if os.path.isdir(save_directory): + vocab_file = os.path.join(save_directory, + self.vocab_files_names["vocab_file"]) + else: + vocab_file = save_directory + + with open(self.vocab_file, 'rb') as fin: + proto_str = fin.read() + + with open(vocab_file, "wb") as writer: + writer.write(proto_str) + + return (vocab_file, ) + + def build_inputs_with_special_tokens( + self, + token_ids_0: List[int], + token_ids_1: Optional[List[int]] = None) -> List[int]: + """ + Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and + adding special tokens. A BERT sequence has the following format: + + - single sequence: `[CLS] X [SEP]` + - pair of sequences: `[CLS] A [SEP] B [SEP]` + + Args: + token_ids_0 (`List[int]`): + List of IDs to which the special tokens will be added. + token_ids_1 (`List[int]`, *optional*): + Optional second list of IDs for sequence pairs. + + Returns: + `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. + """ + gmask_id = self.sp_tokenizer[self.gmask_token] + eos_id = self.sp_tokenizer[self.eos_token] + token_ids_0 = token_ids_0 + [ + gmask_id, self.sp_tokenizer[self.bos_token] + ] + if token_ids_1 is not None: + token_ids_0 = token_ids_0 + token_ids_1 + [eos_id] + return token_ids_0 + + def _pad( + self, + encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding], + max_length: Optional[int] = None, + padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, + pad_to_multiple_of: Optional[int] = None, + return_attention_mask: Optional[bool] = None, + padding_side: str = "left", # Fix for new transformers + ) -> dict: + """ + Pad encoded inputs (on left/right and up to predefined length or max length in the batch) + + Args: + encoded_inputs: + Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). + max_length: maximum length of the returned list and optionally padding length (see below). + Will truncate by taking into account the special tokens. + padding_strategy: PaddingStrategy to use for padding. + + - PaddingStrategy.LONGEST Pad to the longest sequence in the batch + - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) + - PaddingStrategy.DO_NOT_PAD: Do not pad + The tokenizer padding sides are defined in self.padding_side: + + - 'left': pads on the left of the sequences + - 'right': pads on the right of the sequences + pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. + This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability + `>= 7.5` (Volta). + return_attention_mask: + (optional) Set to False to avoid returning attention mask (default: set to model specifics) + """ + # Load from model defaults + bos_token_id = self.sp_tokenizer[self.bos_token] + mask_token_id = self.sp_tokenizer[self.mask_token] + gmask_token_id = self.sp_tokenizer[self.gmask_token] + assert self.padding_side == "left" + + required_input = encoded_inputs[self.model_input_names[0]] + seq_length = len(required_input) + + if padding_strategy == PaddingStrategy.LONGEST: + max_length = len(required_input) + + if max_length is not None and pad_to_multiple_of is not None and ( + max_length % pad_to_multiple_of != 0): + max_length = ( + (max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of + + needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len( + required_input) != max_length + + # Initialize attention mask if not present. + if max_length is not None: + if "attention_mask" not in encoded_inputs: + if bos_token_id in required_input: + context_length = required_input.index(bos_token_id) + else: + context_length = seq_length + attention_mask = np.ones((1, seq_length, seq_length)) + attention_mask = np.tril(attention_mask) + attention_mask[:, :, :context_length] = 1 + attention_mask = np.bool_(attention_mask < 0.5) + encoded_inputs["attention_mask"] = attention_mask + + if "position_ids" not in encoded_inputs: + if bos_token_id in required_input: + context_length = required_input.index(bos_token_id) + else: + context_length = seq_length + position_ids = np.arange(seq_length, dtype=np.int64) + mask_token = mask_token_id if mask_token_id in required_input else gmask_token_id + if mask_token in required_input: + mask_position = required_input.index(mask_token) + position_ids[context_length:] = mask_position + block_position_ids = np.concatenate([ + np.zeros(context_length, dtype=np.int64), + np.arange(1, + seq_length - context_length + 1, + dtype=np.int64) + ]) + encoded_inputs["position_ids"] = np.stack( + [position_ids, block_position_ids], axis=0) + + if needs_to_be_padded: + difference = max_length - len(required_input) + + if "attention_mask" in encoded_inputs: + encoded_inputs["attention_mask"] = np.pad( + encoded_inputs["attention_mask"], + pad_width=[(0, 0), (difference, 0), (difference, 0)], + mode='constant', + constant_values=True) + if "token_type_ids" in encoded_inputs: + encoded_inputs["token_type_ids"] = [ + self.pad_token_type_id + ] * difference + encoded_inputs["token_type_ids"] + if "special_tokens_mask" in encoded_inputs: + encoded_inputs["special_tokens_mask"] = [ + 1 + ] * difference + encoded_inputs["special_tokens_mask"] + if "position_ids" in encoded_inputs: + encoded_inputs["position_ids"] = np.pad( + encoded_inputs["position_ids"], + pad_width=[(0, 0), (difference, 0)]) + encoded_inputs[self.model_input_names[ + 0]] = [self.pad_token_id] * difference + required_input + + return encoded_inputs diff --git a/examples/models/contrib/chatglm2-6b/README.md b/examples/models/contrib/chatglm2-6b/README.md new file mode 100644 index 000000000000..c25818e3f95c --- /dev/null +++ b/examples/models/contrib/chatglm2-6b/README.md @@ -0,0 +1,108 @@ +# ChatGLM + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document explains how to build the [ChatGLM2-6B](https://huggingface.co/THUDM/chatglm2-6b), [ChatGLM2-6B-32k](https://huggingface.co/THUDM/chatglm2-6b-32k) models using TensorRT LLM and run on a single GPU, a single node with multiple GPUs or multiple nodes with multiple GPUs. + +- [ChatGLM](#chatglm) + - [Overview](#overview) + - [Support Matrix](#support-matrix) + - [Model comparison](#model-comparison) + - [Tokenizer and special tokens comparison](#tokenizer-and-special-tokens-comparison) + - [Usage](#usage) + - [1. Download repo and weights from HuggingFace Transformers](#1-download-repo-and-weights-from-huggingface-transformers) + - [2. Convert weights from HF Transformers to TensorRT LLM format](#2-convert-weights-from-hf-transformers-to-tensorrt-llm-format) + - [3. Build TensorRT engine(s)](#3-build-tensorrt-engines) + - [Enable plugins](#enable-plugins) + - [In-flight batching](#in-flight-batching) + - [4. Run inference](#4-run-inference) + - [Single node, single GPU](#single-node-single-gpu) + - [Single node, multi GPU](#single-node-multi-gpu) + - [5. Run summarization task](#5-run-summarization-task) + - [Weight Only quantization](#weight-only-quantization) + - [Smooth Quantization (SQ)](#smooth-quantization-sq) + - [Activation-aware Weight Quantization (AWQ)](#activation-aware-weight-quantization-awq) + - [FP8 Quantization](#fp8-quantization) + - [Benchmark](#benchmark) + + +## Overview + +The TensorRT LLM ChatGLM implementation can be found in [`tensorrt_llm/models/chatglm/model.py`](../../tensorrt_llm/models/chatglm/model.py). +The TensorRT LLM ChatGLM example code is located in [`examples/models/contrib/chatglm2-6b`](./). There is one main file: + +* [`examples/models/core/glm-4-9b/convert_checkpoint.py`](../../../glm-4-9b/convert_checkpoint.py) to convert a checkpoint from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. + +In addition, there are two shared files in the parent folder [`examples`](../../../) for inference and evaluation: + +* [`../../../run.py`](../../../run.py) to run the inference on an input text; +* [`../../../summarize.py`](../../../summarize.py) to summarize the articles in the [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +## Support Matrix + +| Model Name | FP16 | FMHA | WO | SQ | AWQ | FP8 | TP | PP | ST | C++ | benchmark | IFB | +| :--------------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :-------: | :---: | +| chatglm2_6b | Y | Y | Y | Y | Y | Y | Y | | Y | Y | Y | Y | +| chatglm2_6b_32k | Y | Y | Y | | Y | Y | Y | | Y | Y | Y | Y | + +* Model Name: the name of the model, the same as the name on HuggingFace +* FMHA: Fused MultiHead Attention (see introduction below) +* WO: Weight Only Quantization (int8 / int4) +* SQ: Smooth Quantization (int8) +* AWQ: Activation Aware Weight Quantization (int4) +* FP8: FP8 Quantization +* TP: Tensor Parallel +* PP: Pipeline Parallel +* ST: Strongly Typed +* C++: C++ Runtime +* benchmark: benchmark by python / C++ Runtime +* IFB: In-flight Batching (see introduction below) + +## Model comparison + +| Name | nL | nAH | nKH | nHW | nH | nF | nMSL | nV | bP2D | bBQKV | bBDense | Comments | +| :--------------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :----: | :---: | :---: | :-----: | :----------------------------------------------------------------- | +| chatglm2_6b | 28 | 32 | 2 | 128 | 4096 | 13696 | 32768 | 65024 | N | Y | N | Multi_query_attention, RMSNorm rather than LayerNorm in chatglm_6b | +| chatglm2_6b_32k | 28 | 32 | 2 | 128 | 4096 | 13696 | 32768 | 65024 | N | Y | N | RoPE base=160000 rather than 10000 in chatglm2_6b | + +* nL: number of layers +* nAH: number of attention heads +* nKH: number of kv heads (less than nAH if multi_query_attention is used) +* nHW: head width +* nH: hidden size +* nF: FFN hidden size +* nMSL: max sequence length (input + output) +* nV: vocabulary size +* bP2D: use position_encoding_2d (Y: Yes, N: No) +* bBQKV: use bias for QKV multiplication in self-attention +* bBDense: use bias for Dense multiplication in self-attention + +## Tokenizer and special tokens comparison + +| Name | Tokenizer | bos | eos | pad | cls | startofpiece | endofpiece | mask | smask | gmask | +| :--------------: | :--------------: | :----: | :----: | :---: | :---: | :----------: | :--------: | :----: | :---: | :----: | +| chatglm2_6b | ChatGLMTokenizer | 1 | 2 | 0 | | | | | | | +| chatglm2_6b_32k | ChatGLMTokenizer | 1 | 2 | 0 | | | | | | | + +## Usage + +The next section describe how to build the engine and run the inference demo. + +### 1. Download repo and weights from HuggingFace Transformers + +```bash +pip install -r requirements.txt +apt-get update +apt-get install git-lfs +rm -rf chatglm* + +# clone one or more models we want to build +git clone https://huggingface.co/THUDM/chatglm2-6b chatglm2_6b +git clone https://huggingface.co/THUDM/chatglm2-6b-32k chatglm2_6b_32k +``` + +For more example codes, please refer to the [examples/models/core/glm-4-9b/README.md](../../../glm-4-9b/README.md). diff --git a/examples/models/contrib/chatglm2-6b/requirements.txt b/examples/models/contrib/chatglm2-6b/requirements.txt new file mode 100644 index 000000000000..cdc65bf2bb38 --- /dev/null +++ b/examples/models/contrib/chatglm2-6b/requirements.txt @@ -0,0 +1,8 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +datasets==3.1.0 +evaluate +protobuf +rouge_score +sentencepiece +tiktoken diff --git a/examples/models/contrib/chatglm2-6b/tokenization_chatglm.py b/examples/models/contrib/chatglm2-6b/tokenization_chatglm.py new file mode 100644 index 000000000000..dea5fdc7c7dc --- /dev/null +++ b/examples/models/contrib/chatglm2-6b/tokenization_chatglm.py @@ -0,0 +1,282 @@ +import os +from typing import Dict, List, Optional, Union + +from sentencepiece import SentencePieceProcessor +from transformers import PreTrainedTokenizer +from transformers.tokenization_utils_base import BatchEncoding, EncodedInput +from transformers.utils import PaddingStrategy + + +class SPTokenizer: + + def __init__(self, model_path: str): + # reload tokenizer + assert os.path.isfile(model_path), model_path + self.sp_model = SentencePieceProcessor(model_file=model_path) + + # BOS / EOS token IDs + self.n_words: int = self.sp_model.vocab_size() + self.bos_id: int = self.sp_model.bos_id() + self.eos_id: int = self.sp_model.eos_id() + self.pad_id: int = self.sp_model.unk_id() + assert self.sp_model.vocab_size() == self.sp_model.get_piece_size() + + special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"] + self.special_tokens = {} + self.index_special_tokens = {} + for token in special_tokens: + self.special_tokens[token] = self.n_words + self.index_special_tokens[self.n_words] = token + self.n_words += 1 + + def tokenize(self, s: str): + return self.sp_model.EncodeAsPieces(s) + + def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]: + assert type(s) is str + t = self.sp_model.encode(s) + if bos: + t = [self.bos_id] + t + if eos: + t = t + [self.eos_id] + return t + + def decode(self, t: List[int]) -> str: + return self.sp_model.decode(t) + + def decode_tokens(self, tokens: List[str]) -> str: + text = self.sp_model.DecodePieces(tokens) + return text + + def convert_token_to_id(self, token): + """ Converts a token (str) in an id using the vocab. """ + if token in self.special_tokens: + return self.special_tokens[token] + return self.sp_model.PieceToId(token) + + def convert_id_to_token(self, index): + """Converts an index (integer) in a token (str) using the vocab.""" + if index in self.index_special_tokens or index in [ + self.eos_id, self.bos_id, self.pad_id + ] or index < 0: + return "" + return self.sp_model.IdToPiece(index) + + +class ChatGLMTokenizer(PreTrainedTokenizer): + vocab_files_names = {"vocab_file": "tokenizer.model"} + + model_input_names = ["input_ids", "attention_mask", "position_ids"] + + def __init__(self, + vocab_file, + padding_side="left", + clean_up_tokenization_spaces=False, + **kwargs): + self.name = "GLMTokenizer" + + self.vocab_file = vocab_file + self.tokenizer = SPTokenizer(vocab_file) + self.special_tokens = { + "": self.tokenizer.bos_id, + "": self.tokenizer.eos_id, + "": self.tokenizer.pad_id + } + super().__init__( + padding_side=padding_side, + clean_up_tokenization_spaces=clean_up_tokenization_spaces, + **kwargs) + + def get_command(self, token): + if token in self.special_tokens: + return self.special_tokens[token] + assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}" + return self.tokenizer.special_tokens[token] + + @property + def unk_token(self) -> str: + return "" + + @property + def pad_token(self) -> str: + return "" + + @property + def pad_token_id(self): + return self.get_command("") + + @property + def eos_token(self) -> str: + return "" + + @property + def eos_token_id(self): + return self.get_command("") + + @property + def vocab_size(self): + return self.tokenizer.n_words + + def get_vocab(self): + """ Returns vocab as a dict """ + vocab = { + self._convert_id_to_token(i): i + for i in range(self.vocab_size) + } + vocab.update(self.added_tokens_encoder) + return vocab + + def _tokenize(self, text, **kwargs): + return self.tokenizer.tokenize(text) + + def _convert_token_to_id(self, token): + """ Converts a token (str) in an id using the vocab. """ + return self.tokenizer.convert_token_to_id(token) + + def _convert_id_to_token(self, index): + """Converts an index (integer) in a token (str) using the vocab.""" + return self.tokenizer.convert_id_to_token(index) + + def convert_tokens_to_string(self, tokens: List[str]) -> str: + return self.tokenizer.decode_tokens(tokens) + + def save_vocabulary(self, save_directory, filename_prefix=None): + """ + Save the vocabulary and special tokens file to a directory. + + Args: + save_directory (`str`): + The directory in which to save the vocabulary. + filename_prefix (`str`, *optional*): + An optional prefix to add to the named of the saved files. + + Returns: + `Tuple(str)`: Paths to the files saved. + """ + if os.path.isdir(save_directory): + vocab_file = os.path.join(save_directory, + self.vocab_files_names["vocab_file"]) + else: + vocab_file = save_directory + + with open(self.vocab_file, 'rb') as fin: + proto_str = fin.read() + + with open(vocab_file, "wb") as writer: + writer.write(proto_str) + + return (vocab_file, ) + + def get_prefix_tokens(self): + prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")] + return prefix_tokens + + def build_prompt(self, query, history=None): + if history is None: + history = [] + prompt = "" + for i, (old_query, response) in enumerate(history): + prompt += "[Round {}]\n\n问：{}\n\n答：{}\n\n".format( + i + 1, old_query, response) + prompt += "[Round {}]\n\n问：{}\n\n答：".format(len(history) + 1, query) + return prompt + + def build_inputs_with_special_tokens( + self, + token_ids_0: List[int], + token_ids_1: Optional[List[int]] = None) -> List[int]: + """ + Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and + adding special tokens. A BERT sequence has the following format: + + - single sequence: `[CLS] X [SEP]` + - pair of sequences: `[CLS] A [SEP] B [SEP]` + + Args: + token_ids_0 (`List[int]`): + List of IDs to which the special tokens will be added. + token_ids_1 (`List[int]`, *optional*): + Optional second list of IDs for sequence pairs. + + Returns: + `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. + """ + prefix_tokens = self.get_prefix_tokens() + token_ids_0 = prefix_tokens + token_ids_0 + if token_ids_1 is not None: + token_ids_0 = token_ids_0 + token_ids_1 + [ + self.get_command("") + ] + return token_ids_0 + + def _pad( + self, + encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding], + max_length: Optional[int] = None, + padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, + pad_to_multiple_of: Optional[int] = None, + return_attention_mask: Optional[bool] = None, + padding_side: str = "left", # Fix for new transformers + ) -> dict: + """ + Pad encoded inputs (on left/right and up to predefined length or max length in the batch) + + Args: + encoded_inputs: + Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). + max_length: maximum length of the returned list and optionally padding length (see below). + Will truncate by taking into account the special tokens. + padding_strategy: PaddingStrategy to use for padding. + + - PaddingStrategy.LONGEST Pad to the longest sequence in the batch + - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) + - PaddingStrategy.DO_NOT_PAD: Do not pad + The tokenizer padding sides are defined in self.padding_side: + + - 'left': pads on the left of the sequences + - 'right': pads on the right of the sequences + pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. + This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability + `>= 7.5` (Volta). + return_attention_mask: + (optional) Set to False to avoid returning attention mask (default: set to model specifics) + """ + # Load from model defaults + assert self.padding_side == "left" + + required_input = encoded_inputs[self.model_input_names[0]] + seq_length = len(required_input) + + if padding_strategy == PaddingStrategy.LONGEST: + max_length = len(required_input) + + if max_length is not None and pad_to_multiple_of is not None and ( + max_length % pad_to_multiple_of != 0): + max_length = ( + (max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of + + needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len( + required_input) != max_length + + # Initialize attention mask if not present. + if "attention_mask" not in encoded_inputs: + encoded_inputs["attention_mask"] = [1] * seq_length + + if "position_ids" not in encoded_inputs: + encoded_inputs["position_ids"] = list(range(seq_length)) + + if needs_to_be_padded: + difference = max_length - len(required_input) + + if "attention_mask" in encoded_inputs: + encoded_inputs["attention_mask"] = [ + 0 + ] * difference + encoded_inputs["attention_mask"] + if "position_ids" in encoded_inputs: + encoded_inputs["position_ids"] = [ + 0 + ] * difference + encoded_inputs["position_ids"] + encoded_inputs[self.model_input_names[ + 0]] = [self.pad_token_id] * difference + required_input + + return encoded_inputs diff --git a/examples/models/contrib/chatglm3-6b-32k/README.md b/examples/models/contrib/chatglm3-6b-32k/README.md new file mode 100644 index 000000000000..0636cba34f9c --- /dev/null +++ b/examples/models/contrib/chatglm3-6b-32k/README.md @@ -0,0 +1,112 @@ +# ChatGLM + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document explains how to build the [ChatGLM3-6B](https://huggingface.co/THUDM/chatglm3-6b), [ChatGLM3-6B-Base](https://huggingface.co/THUDM/chatglm3-6b-base), [ChatGLM3-6B-32k](https://huggingface.co/THUDM/chatglm3-6b-32k) models using TensorRT LLM and run on a single GPU, a single node with multiple GPUs or multiple nodes with multiple GPUs. + +- [ChatGLM](#chatglm) + - [Overview](#overview) + - [Support Matrix](#support-matrix) + - [Model comparison](#model-comparison) + - [Tokenizer and special tokens comparison](#tokenizer-and-special-tokens-comparison) + - [Usage](#usage) + - [1. Download repo and weights from HuggingFace Transformers](#1-download-repo-and-weights-from-huggingface-transformers) + - [2. Convert weights from HF Transformers to TensorRT LLM format](#2-convert-weights-from-hf-transformers-to-tensorrt-llm-format) + - [3. Build TensorRT engine(s)](#3-build-tensorrt-engines) + - [Enable plugins](#enable-plugins) + - [In-flight batching](#in-flight-batching) + - [4. Run inference](#4-run-inference) + - [Single node, single GPU](#single-node-single-gpu) + - [Single node, multi GPU](#single-node-multi-gpu) + - [5. Run summarization task](#5-run-summarization-task) + - [Weight Only quantization](#weight-only-quantization) + - [Smooth Quantization (SQ)](#smooth-quantization-sq) + - [Activation-aware Weight Quantization (AWQ)](#activation-aware-weight-quantization-awq) + - [FP8 Quantization](#fp8-quantization) + - [Benchmark](#benchmark) + + +## Overview + +The TensorRT LLM ChatGLM implementation can be found in [`tensorrt_llm/models/chatglm/model.py`](../../tensorrt_llm/models/chatglm/model.py). +The TensorRT LLM ChatGLM example code is located in [`examples/models/contrib/chatglm3-6b-32k`](./). There is one main file: + +* [`examples/models/core/glm-4-9b/convert_checkpoint.py`](../../../glm-4-9b/convert_checkpoint.py) to convert a checkpoint from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format. + +In addition, there are two shared files in the parent folder [`examples`](../../../) for inference and evaluation: + +* [`../../../run.py`](../../../run.py) to run the inference on an input text; +* [`../../../summarize.py`](../../../summarize.py) to summarize the articles in the [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +## Support Matrix + +| Model Name | FP16 | FMHA | WO | SQ | AWQ | FP8 | TP | PP | ST | C++ | benchmark | IFB | +| :--------------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :-------: | :---: | +| chatglm3_6b | Y | Y | Y | Y | Y | Y | Y | | Y | Y | Y | Y | +| chatglm3_6b_base | Y | Y | Y | Y | Y | Y | Y | | Y | Y | Y | Y | +| chatglm3_6b_32k | Y | Y | Y | Y | Y | Y | Y | | Y | Y | Y | Y | + +* Model Name: the name of the model, the same as the name on HuggingFace +* FMHA: Fused MultiHead Attention (see introduction below) +* WO: Weight Only Quantization (int8 / int4) +* SQ: Smooth Quantization (int8) +* AWQ: Activation Aware Weight Quantization (int4) +* FP8: FP8 Quantization +* TP: Tensor Parallel +* PP: Pipeline Parallel +* ST: Strongly Typed +* C++: C++ Runtime +* benchmark: benchmark by python / C++ Runtime +* IFB: In-flight Batching (see introduction below) + +## Model comparison + +| Name | nL | nAH | nKH | nHW | nH | nF | nMSL | nV | bP2D | bBQKV | bBDense | Comments | +| :--------------: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :----: | :---: | :---: | :-----: | :----------------------------------------------------------------- | +| chatglm3_6b | 28 | 32 | 2 | 128 | 4096 | 13696 | 8192 | 65024 | N | Y | N | Different in preprocess and postprocess than chatglm2_6b | +| chatglm3_6b_base | 28 | 32 | 2 | 128 | 4096 | 13696 | 32768 | 65024 | N | Y | N | | +| chatglm3_6b_32k | 28 | 32 | 2 | 128 | 4096 | 13696 | 32768 | 65024 | N | Y | N | RoPE base=500000 rather than 10000 in chatglm3_6b | + +* nL: number of layers +* nAH: number of attention heads +* nKH: number of kv heads (less than nAH if multi_query_attention is used) +* nHW: head width +* nH: hidden size +* nF: FFN hidden size +* nMSL: max sequence length (input + output) +* nV: vocabulary size +* bP2D: use position_encoding_2d (Y: Yes, N: No) +* bBQKV: use bias for QKV multiplication in self-attention +* bBDense: use bias for Dense multiplication in self-attention + +## Tokenizer and special tokens comparison + +| Name | Tokenizer | bos | eos | pad | cls | startofpiece | endofpiece | mask | smask | gmask | +| :--------------: | :--------------: | :----: | :----: | :---: | :---: | :----------: | :--------: | :----: | :---: | :----: | +| chatglm3_6b | ChatGLMTokenizer | 1 | 2 | 0 | | | | 130000 | | | +| chatglm3_6b_base | ChatGLMTokenizer | 1 | 2 | 0 | | | | 130000 | | | +| chatglm3_6b_32k | ChatGLMTokenizer | 1 | 2 | 0 | | | | 130000 | | | + +## Usage + +The next section describe how to build the engine and run the inference demo. + +### 1. Download repo and weights from HuggingFace Transformers + +```bash +pip install -r requirements.txt +apt-get update +apt-get install git-lfs +rm -rf chatglm* + +# clone one or more models we want to build +git clone https://huggingface.co/THUDM/chatglm3-6b chatglm3_6b +git clone https://huggingface.co/THUDM/chatglm3-6b-base chatglm3_6b_base +git clone https://huggingface.co/THUDM/chatglm3-6b-32k chatglm3_6b_32k +``` + +For more example codes, please refer to the [examples/models/core/glm-4-9b/README.md](../../../glm-4-9b/README.md). diff --git a/examples/models/contrib/chatglm3-6b-32k/requirements.txt b/examples/models/contrib/chatglm3-6b-32k/requirements.txt new file mode 100644 index 000000000000..cdc65bf2bb38 --- /dev/null +++ b/examples/models/contrib/chatglm3-6b-32k/requirements.txt @@ -0,0 +1,8 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +datasets==3.1.0 +evaluate +protobuf +rouge_score +sentencepiece +tiktoken diff --git a/examples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.py b/examples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.py new file mode 100644 index 000000000000..67a3d52442a6 --- /dev/null +++ b/examples/models/contrib/chatglm3-6b-32k/tokenization_chatglm.py @@ -0,0 +1,313 @@ +import json +import os +from typing import Dict, List, Optional, Union + +from sentencepiece import SentencePieceProcessor +from transformers import PreTrainedTokenizer +from transformers.tokenization_utils_base import BatchEncoding, EncodedInput +from transformers.utils import PaddingStrategy + + +class SPTokenizer: + + def __init__(self, model_path: str): + # reload tokenizer + assert os.path.isfile(model_path), model_path + self.sp_model = SentencePieceProcessor(model_file=model_path) + + # BOS / EOS token IDs + self.n_words: int = self.sp_model.vocab_size() + self.bos_id: int = self.sp_model.bos_id() + self.eos_id: int = self.sp_model.eos_id() + self.pad_id: int = self.sp_model.unk_id() + assert self.sp_model.vocab_size() == self.sp_model.get_piece_size() + + special_tokens = [ + "[MASK]", "[gMASK]", "[sMASK]", "sop", "eop", "<|system|>", + "<|user|>", "<|assistant|>", "<|observation|>" + ] + self.special_tokens = {} + self.index_special_tokens = {} + for token in special_tokens: + self.special_tokens[token] = self.n_words + self.index_special_tokens[self.n_words] = token + self.n_words += 1 + + def tokenize(self, s: str): + return self.sp_model.EncodeAsPieces(s) + + def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]: + assert type(s) is str + t = self.sp_model.encode(s) + if bos: + t = [self.bos_id] + t + if eos: + t = t + [self.eos_id] + return t + + def decode(self, t: List[int]) -> str: + text, buffer = "", [] + for token in t: + if token in self.index_special_tokens: + if buffer: + text += self.sp_model.decode(buffer) + buffer = [] + text += self.index_special_tokens[token] + else: + buffer.append(token) + if buffer: + text += self.sp_model.decode(buffer) + return text + + def decode_tokens(self, tokens: List[str]) -> str: + text = self.sp_model.DecodePieces(tokens) + return text + + def convert_token_to_id(self, token): + """ Converts a token (str) in an id using the vocab. """ + if token in self.special_tokens: + return self.special_tokens[token] + return self.sp_model.PieceToId(token) + + def convert_id_to_token(self, index): + """Converts an index (integer) in a token (str) using the vocab.""" + if index in self.index_special_tokens: + return self.index_special_tokens[index] + if index in [self.eos_id, self.bos_id, self.pad_id] or index < 0: + return "" + return self.sp_model.IdToPiece(index) + + +class ChatGLMTokenizer(PreTrainedTokenizer): + vocab_files_names = {"vocab_file": "tokenizer.model"} + + model_input_names = ["input_ids", "attention_mask", "position_ids"] + + def __init__(self, + vocab_file, + padding_side="left", + clean_up_tokenization_spaces=False, + **kwargs): + self.name = "GLMTokenizer" + + self.vocab_file = vocab_file + self.tokenizer = SPTokenizer(vocab_file) + self.special_tokens = { + "": self.tokenizer.bos_id, + "": self.tokenizer.eos_id, + "": self.tokenizer.pad_id + } + super().__init__( + padding_side=padding_side, + clean_up_tokenization_spaces=clean_up_tokenization_spaces, + **kwargs) + + def get_command(self, token): + if token in self.special_tokens: + return self.special_tokens[token] + assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}" + return self.tokenizer.special_tokens[token] + + @property + def unk_token(self) -> str: + return "" + + @property + def pad_token(self) -> str: + return "" + + @property + def pad_token_id(self): + return self.get_command("") + + @property + def eos_token(self) -> str: + return "" + + @property + def eos_token_id(self): + return self.get_command("") + + @property + def vocab_size(self): + return self.tokenizer.n_words + + def get_vocab(self): + """ Returns vocab as a dict """ + vocab = { + self._convert_id_to_token(i): i + for i in range(self.vocab_size) + } + vocab.update(self.added_tokens_encoder) + return vocab + + def _tokenize(self, text, **kwargs): + return self.tokenizer.tokenize(text) + + def _convert_token_to_id(self, token): + """ Converts a token (str) in an id using the vocab. """ + return self.tokenizer.convert_token_to_id(token) + + def _convert_id_to_token(self, index): + """Converts an index (integer) in a token (str) using the vocab.""" + return self.tokenizer.convert_id_to_token(index) + + def convert_tokens_to_string(self, tokens: List[str]) -> str: + return self.tokenizer.decode_tokens(tokens) + + def save_vocabulary(self, save_directory, filename_prefix=None): + """ + Save the vocabulary and special tokens file to a directory. + + Args: + save_directory (`str`): + The directory in which to save the vocabulary. + filename_prefix (`str`, *optional*): + An optional prefix to add to the named of the saved files. + + Returns: + `Tuple(str)`: Paths to the files saved. + """ + if os.path.isdir(save_directory): + vocab_file = os.path.join(save_directory, + self.vocab_files_names["vocab_file"]) + else: + vocab_file = save_directory + + with open(self.vocab_file, 'rb') as fin: + proto_bytes = fin.read() + + with open(vocab_file, "wb") as writer: + writer.write(proto_bytes) + + return (vocab_file, ) + + def get_prefix_tokens(self): + prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")] + return prefix_tokens + + def build_single_message(self, role, metadata, message): + assert role in ["system", "user", "assistant", "observation"], role + role_tokens = [self.get_command(f"<|{role}|>") + ] + self.tokenizer.encode(f"{metadata}\n") + message_tokens = self.tokenizer.encode(message) + tokens = role_tokens + message_tokens + return tokens + + def build_chat_input(self, query, history=None, role="user"): + if history is None: + history = [] + input_ids = [] + for item in history: + content = item["content"] + if item["role"] == "system" and "tools" in item: + content = content + "\n" + json.dumps( + item["tools"], indent=4, ensure_ascii=False) + input_ids.extend( + self.build_single_message(item["role"], + item.get("metadata", ""), content)) + input_ids.extend(self.build_single_message(role, "", query)) + input_ids.extend([self.get_command("<|assistant|>")]) + return self.batch_encode_plus([input_ids], + return_tensors="pt", + is_split_into_words=True) + + def build_inputs_with_special_tokens( + self, + token_ids_0: List[int], + token_ids_1: Optional[List[int]] = None) -> List[int]: + """ + Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and + adding special tokens. A BERT sequence has the following format: + + - single sequence: `[CLS] X [SEP]` + - pair of sequences: `[CLS] A [SEP] B [SEP]` + + Args: + token_ids_0 (`List[int]`): + List of IDs to which the special tokens will be added. + token_ids_1 (`List[int]`, *optional*): + Optional second list of IDs for sequence pairs. + + Returns: + `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. + """ + prefix_tokens = self.get_prefix_tokens() + token_ids_0 = prefix_tokens + token_ids_0 + if token_ids_1 is not None: + token_ids_0 = token_ids_0 + token_ids_1 + [ + self.get_command("") + ] + return token_ids_0 + + def _pad( + self, + encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding], + max_length: Optional[int] = None, + padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, + pad_to_multiple_of: Optional[int] = None, + return_attention_mask: Optional[bool] = None, + padding_side: str = "left", # Fix for new transformers + ) -> dict: + """ + Pad encoded inputs (on left/right and up to predefined length or max length in the batch) + + Args: + encoded_inputs: + Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). + max_length: maximum length of the returned list and optionally padding length (see below). + Will truncate by taking into account the special tokens. + padding_strategy: PaddingStrategy to use for padding. + + - PaddingStrategy.LONGEST Pad to the longest sequence in the batch + - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) + - PaddingStrategy.DO_NOT_PAD: Do not pad + The tokenizer padding sides are defined in self.padding_side: + + - 'left': pads on the left of the sequences + - 'right': pads on the right of the sequences + pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. + This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability + `>= 7.5` (Volta). + return_attention_mask: + (optional) Set to False to avoid returning attention mask (default: set to model specifics) + """ + # Load from model defaults + assert self.padding_side == "left" + + required_input = encoded_inputs[self.model_input_names[0]] + seq_length = len(required_input) + + if padding_strategy == PaddingStrategy.LONGEST: + max_length = len(required_input) + + if max_length is not None and pad_to_multiple_of is not None and ( + max_length % pad_to_multiple_of != 0): + max_length = ( + (max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of + + needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len( + required_input) != max_length + + # Initialize attention mask if not present. + if "attention_mask" not in encoded_inputs: + encoded_inputs["attention_mask"] = [1] * seq_length + + if "position_ids" not in encoded_inputs: + encoded_inputs["position_ids"] = list(range(seq_length)) + + if needs_to_be_padded: + difference = max_length - len(required_input) + + if "attention_mask" in encoded_inputs: + encoded_inputs["attention_mask"] = [ + 0 + ] * difference + encoded_inputs["attention_mask"] + if "position_ids" in encoded_inputs: + encoded_inputs["position_ids"] = [ + 0 + ] * difference + encoded_inputs["position_ids"] + encoded_inputs[self.model_input_names[ + 0]] = [self.pad_token_id] * difference + required_input + + return encoded_inputs diff --git a/examples/models/contrib/internlm/.gitignore b/examples/models/contrib/internlm/.gitignore new file mode 100644 index 000000000000..7ce339719a3e --- /dev/null +++ b/examples/models/contrib/internlm/.gitignore @@ -0,0 +1,2 @@ +internlm* +tokenizer.model diff --git a/examples/models/contrib/internlm/README.md b/examples/models/contrib/internlm/README.md new file mode 100644 index 000000000000..f44180f01aad --- /dev/null +++ b/examples/models/contrib/internlm/README.md @@ -0,0 +1,320 @@ +# InternLM + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document shows how to build and run InternLM 7B / 20B models in TensorRT LLM on both single GPU, single node multi-GPU and multi-node multi-GPU. + +- [InternLM](#internlm) + - [Overview](#overview) + - [Support Matrix](#support-matrix) + - [Usage](#usage) + - [Build TensorRT engine(s)](#build-tensorrt-engines) + - [INT8 weight only + INT8 KV cache](#int8-weight-only--int8-kv-cache) + - [SmoothQuant](#smoothquant) + - [Run](#run) + - [Summarization using the InternLM model](#summarization-using-the-internlm-model) + +## Overview + +The TensorRT LLM InternLM implementation is based on the LLaMA model. The implementation can +be found in [tensorrt_llm/models/llama/model.py](../../../../tensorrt_llm/models/llama/model.py). +The TensorRT LLM InternLM example code lies in [`examples/models/contrib/internlm`](./): + +* [`convert_checkpoint.py`](../../core/llama/convert_checkpoint.py) converts the Huggingface Model of InternLM into TensorRT LLM checkpoint. +* [`convert_checkpoint.py`] to to convert a checkpoint from the [HuggingFace (HF) Transformers](https://github.com/huggingface/transformers) format to the TensorRT LLM format + +In addition, there are two shared files in the parent folder [`examples`](../../../) for inference and evaluation: + +* [`../../../run.py`](../../../run.py) to run the inference on an input text; +* [`../../../summarize.py`](../../../summarize.py) to summarize the articles in the [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +## Support Matrix + * FP16 / BF16 + * INT8 & INT4 Weight-Only + * Smooth Quant + * INT8 KV Cache + * Tensor Parallel & Pipeline Parallel + +## Usage + +The TensorRT LLM InternLM example code locates at [examples/models/contrib/internlm](./). It takes HF weights as input, and builds the corresponding TensorRT engines. The number of TensorRT engines depends on the number of GPUs used to run inference. + +### Build TensorRT engine(s) + +Please install required packages first: + +```bash +pip install -r requirements.txt +``` + +TensorRT LLM InternLM builds TensorRT engine(s) from HF checkpoint. If no checkpoint directory is specified, TensorRT LLM will build engine(s) with dummy weights. + +InternLM has released several checkpoints of different size or capabilities under https://huggingface.co/internlm. Users can pick any one repository and follow instructions to prepare the checkpoint. + +Below examples use [internlm-chat-7b](https://huggingface.co/internlm/internlm-chat-7b) and [internlm-chat-20b](https://huggingface.co/internlm/internlm-chat-20b) and assume these repositories are cloned or linked under this directory, for example `./internlm-chat-7b/`. + +Normally `trtllm-build` only requires single GPU, but if you've already got all the GPUs needed for inference, you could enable parallel building to make the engine building process faster by adding `--workers` argument. Please note that currently `--workers` feature only supports single node. + +Here're some examples: + +```bash +# Build a single-GPU float16 engine from HF weights. +# gpt_attention_plugin is necessary in InternLM. +# Try use_gemm_plugin to prevent accuracy issue. +cd examples/models/core/llama + +# Convert the InternLM 7B model using a single GPU and FP16. +python convert_checkpoint.py --model_dir ./internlm-chat-7b/ \ + --dtype float16 \ + --output_dir ./internlm-chat-7b/trt_engines/fp16/1-gpu/ +# Note: setting `--dtype bfloat16` to use bfloat16 precision. + +# BUild the InternLM 7B model using a single GPU +trtllm-build --checkpoint_dir ./internlm-chat-7b/trt_engines/fp16/1-gpu/ \ + --output_dir ./engine_outputs \ + --gemm_plugin float16 + +# Convert the InternLM 7B model using a single GPU and apply INT8 weight-only quantization.. +python convert_checkpoint.py --model_dir ./internlm-chat-7b/ \ + --dtype float16 \ + --output_dir ./internlm-chat-7b/trt_engines/int8/1-gpu/ \ + --use_weight_only \ + --weight_only_precision int8 + +trtllm-build --checkpoint_dir ./internlm-chat-7b/trt_engines/int8/1-gpu/ \ + --output_dir ./engine_outputs \ + --gemm_plugin float16 + +# Note: setting `--weight_only_precision int4` to use INT4 weight-only quantization + +# Build InternLM 7B using 2-way tensor parallelism. +python convert_checkpoint.py --model_dir ./internlm-chat-7b/ \ + --dtype float16 \ + --output_dir ./internlm-chat-7b/trt_engines/fp16/2-gpu/ \ + --tp_size 2 + +trtllm-build --checkpoint_dir ./internlm-chat-7b/trt_engines/fp16/2-gpu/ \ + --output_dir ./engine_outputs \ + --gemm_plugin float16 + +# Build InternLM 20B using 2-way tensor parallelism. +python convert_checkpoint.py --model_dir ./internlm-chat-20b/ \ + --dtype bfloat16 \ + --output_dir ./internlm-chat-20b/trt_engines/bf16/2-gpu/ \ + --tp_size 2 --workers 2 + +trtllm-build --checkpoint_dir ./internlm-chat-7b/trt_engines/bf16/2-gpu/ \ + --output_dir ./engine_outputs \ + --gpt_attention_plugin bfloat16 \ + --gemm_plugin bfloat16 +``` + +#### INT8 weight only + INT8 KV cache + +For INT8 KV cache, [`convert_checkpoint.py`](./convert_checkpoint.py) features a +`--int8_kv_cache` option. Setting `--int8_kv_cache` will calibrate the model, +and then export the scaling factors needed for INT8 KV cache inference. + + +Example: + +```bash +cd examples/models/core/llama + +# For 7B models +python convert_checkpoint.py --model_dir ./internlm-chat-7b \ + --output_dir ./internlm-chat-7b/smooth_internlm/int8_kv_cache/ \ + --dtype float16 \ + --use_weight_only \ + --weight_only_precision int8 \ + --int8_kv_cache + +# Build 7B model with both INT8 weight-only and INT8 KV cache enabled +trtllm-build --checkpoint_dir ./internlm-chat-7b/smooth_internlm/int8_kv_cache/ \ + --output_dir ./engine_outputs \ + --gemm_plugin float16 \ +``` + + +```bash +cd examples/models/core/llama + +# For 20B models +python convert_checkpoint.py --model_dir ./internlm-chat-20b \ + --output_dir ./internlm-chat-20b/smooth_internlm/int8_kv_cache/ \ + --dtype float16 \ + --use_weight_only \ + --weight_only_precision int8 \ + --int8_kv_cache + +# Build 20B model with both INT8 weight-only and INT8 KV cache enabled +trtllm-build --checkpoint_dir ./internlm-chat-20b/smooth_internlm/int8_kv_cache/ \ + --output_dir ./engine_outputs \ + --gemm_plugin float16 \ +``` + + +Test with `../../../run.py` or `../../../summarize.py`: + +```bash +python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-7b/ \ + --engine_dir ./internlm-chat-7b/trt_engines/int8_kv_cache_weight_only/1-gpu + +python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-20b/ \ + --engine_dir ./internlm-chat-20b/trt_engines/int8_kv_cache_weight_only/1-gpu + +python ../../../summarize.py --test_trt_llm --test_hf \ + --hf_model_dir ./internlm-chat-7b \ + --data_type fp16 \ + --engine_dir ./internlm-chat-7b/trt_engines/int8_kv_cache_weight_only/1-gpu + +python ../../../summarize.py --test_trt_llm --test_hf \ + --hf_model_dir ./internlm-chat-20b \ + --data_type fp16 \ + --engine_dir ./internlm-chat-20b/trt_engines/int8_kv_cache_weight_only/1-gpu +``` + +#### SmoothQuant + +Unlike the FP16 build where the HF weights are processed and loaded into the TensorRT LLM directly, the SmoothQuant needs to load INT8 weights which should be pre-processed before building an engine. + +Example: +```bash +cd examples/models/core/llama + +# For 7B models +python convert_checkpoint.py --model_dir ./internlm-chat-7b --output_dir ./internlm-chat-7b/smooth_internlm/sq0.5/ --dtype float16 --smoothquant 0.5 +# Build the engine +trtllm-build --checkpoint_dir ./internlm-chat-7b/smooth_internlm/sq0.5/ \ + --output_dir ./engine_outputs \ + --gemm_plugin float16 + +# For 20B models +cd examples/models/core/llama + +python convert_checkpoint.py --model_dir ./internlm-chat-20b --output_dir ./internlm-chat-20b/smooth_internlm/sq0.5/ --dtype float16 --smoothquant 0.5 +trtllm-build --checkpoint_dir ./internlm-chat-20b/smooth_internlm/sq0.5/ \ + --output_dir ./engine_outputs \ + --gemm_plugin float16 +``` + +[`convert_checkpoint.py`](./convert_checkpoint.py) add new options for the support of INT8 inference of SmoothQuant models. + +`--smoothquant` is the starting point of INT8 inference. By default, it +will run the model in the _per-tensor_ mode. + +Then, you can add any combination of `--per-token` and `--per-channel` to get the corresponding behaviors. + +Examples of build invocations: + +```bash +# Build model for SmoothQuant in the _per_token_ + _per_channel_ mode +cd examples/models/core/llama + +# 7B model +python convert_checkpoint.py --model_dir ./internlm-chat-7b --output_dir ./internlm-chat-7b/smooth_internlm/sq0.5/ --dtype float16 --smoothquant 0.5 --per_channel --per_token + +# 20B model +python convert_checkpoint.py --model_dir ./internlm-chat-20b --output_dir ./internlm-chat-20b/smooth_internlm/sq0.5/ --dtype float16 --smoothquant 0.5 --per_channel --per_token +``` + + +Test with `../../../run.py` or `../../../summarize.py`: + +```bash +python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-7b/ \ + --engine_dir ./internlm-chat-7b/smooth_internlm/sq0.5/ + +python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-20b/ \ + --engine_dir ./internlm-chat-20b/smooth_internlm/sq0.5/ + +python ../../../summarize.py --test_trt_llm --test_hf \ + --hf_model_dir ./internlm-chat-7b \ + --data_type fp16 \ + --engine_dir ./internlm-chat-7b/smooth_internlm/sq0.5/ + +python ../../../summarize.py --test_trt_llm --test_hf \ + --hf_model_dir ./internlm-chat-20b \ + --data_type fp16 \ + --engine_dir ./internlm-chat-20b/smooth_internlm/sq0.5/ +``` + +### Run + +To run a TensorRT LLM InternLM model using the engines generated by `trtllm-build` + +```bash +# InternLM 7B with fp16 +python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-7b/ \ + --engine_dir=./internlm-chat-7b/trt_engines/fp16/1-gpu/ + +# InternLM 7B with bf16 +python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-7b/ \ + --engine_dir=./internlm-chat-7b/trt_engines/bf16/1-gpu/ + +# InternLM 7B with int8 weight only quantization +python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-7b/ \ + --engine_dir=./internlm-chat-7b/trt_engines/weight_only/1-gpu/ + +# InternLM 7B with fp16 and tensor parallelism +mpirun -n 2 --allow-run-as-root \ + python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-7b/ \ + --engine_dir=./internlm-chat-7b/trt_engines/fp16/2-gpu/ + +# InternLM 20B with fp16 and tensor parallelism and pipeline parallelism +mpirun -n 4 --allow-run-as-root \ + python ../../../run.py --max_output_len=120 \ + --input_text 'Tell me about yourself.' \ + --tokenizer_dir ./internlm-chat-7b/ \ + --engine_dir=./internlm-chat-7b/trt_engines/bf16/4-gpu/ +``` + +### Summarization using the InternLM model + +```bash +# Run summarization using the InternLM 7B model in FP16. +python ../../../summarize.py --test_trt_llm --test_hf \ + --hf_model_dir ./internlm-chat-7b/ \ + --data_type fp16 \ + --engine_dir ./engine_outputs + +# Run summarization using the InternLM 7B model quantized to INT8. +python ../../../summarize.py --test_trt_llm --test_hf \ + --hf_model_dir ./internlm-chat-7b/ \ + --data_type fp16 \ + --engine_dir ./engine_outputs + +# Run summarization using the InternLM 7B model in FP16 using two GPUs. +mpirun -n 2 --allow-run-as-root \ + python ../../../summarize.py --test_trt_llm --test_hf \ + --hf_model_dir ./internlm-chat-7b/ \ + --data_type fp16 \ + --engine_dir ./internlm-chat-7b/trt_engines/fp16/2-gpu/ + +# Run summarization using the InternLM 20B model in BF16 using 4 GPUs. +mpirun -n 4 --allow-run-as-root \ + python ../../../summarize.py --test_trt_llm --test_hf \ + --hf_model_dir ./internlm-chat-20b/ \ + --data_type bf16 \ + --engine_dir ./internlm-chat-20b/trt_engines/bf16/4-gpu/ +``` diff --git a/examples/models/contrib/internlm/requirements.txt b/examples/models/contrib/internlm/requirements.txt new file mode 100644 index 000000000000..d9354a133c65 --- /dev/null +++ b/examples/models/contrib/internlm/requirements.txt @@ -0,0 +1,6 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +datasets==3.1.0 +rouge_score +sentencepiece>=0.1.99 +evaluate diff --git a/examples/models/contrib/jais/README.md b/examples/models/contrib/jais/README.md new file mode 100644 index 000000000000..d15cfd116962 --- /dev/null +++ b/examples/models/contrib/jais/README.md @@ -0,0 +1,125 @@ +# Jais + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document elaborates how to build Jais model to runnable engines on multi-GPU node and perform a summarization task using these engines. + +Currently it has been tested on +- [Jais-13b-chat](https://huggingface.co/core42/jais-13b-chat) +- [Jais-30b-chat-v3](https://huggingface.co/core42/jais-30b-chat-v3) + + +- [Jais](#jais) + - [Overview](#overview) + - [Support Matrix](#support-matrix) + - [Usage](#usage) + - [Build TensorRT engine(s)](#build-tensorrt-engines) + - [Run inference](#run) + +## Overview + +The TensorRT LLM support for Jais is based on the GPT model, the implementation can be found in [tensorrt_llm/models/gpt/model.py](../../../../tensorrt_llm/models/gpt/model.py). Jais model resembles GPT very much except it uses alibi embedding, embedding scale, swiglu, and logits scale, we therefore reuse the [GPT example code](../../../gpt) for Jais, + +* [`convert_checkpoint.py`](../../../gpt/convert_checkpoint.py) to convert the Jais model into TensorRT LLM checkpoint format. + +In addition, there are two shared files in the parent folder [`examples`](../) for inference and evaluation: + +* [`../../../run.py`](../../../run.py) to run the inference on an input text; +* [`../../../summarize.py`](../../../summarize.py) to summarize the articles in the [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +## Support Matrix +The tested configurations are: + * FP16 + * FP8 + * Inflight Batching + * Tensor Parallel + +## Usage + +This section gives a whole process where we convert HF models, build TensorRT LLM engines and ultimately perform summarization. + +### Build TensorRT engine(s) + +Run the following commands and TRT-LLM will first transforms a HF model into its own checkpoint format, then builds a TRT engine based on the checkpoint + +```bash +# single gpu, dtype float16 for jais-13b-chat +python3 ../../../gpt/convert_checkpoint.py --model_dir core42/jais-13b-chat \ + --dtype float16 \ + --output_dir jais-13b-chat/trt_ckpt/fp16/1-gpu + +# 2-way tensor parallelism for jais-30b-chat-v3 +python3 ../../../gpt/convert_checkpoint.py --model_dir core42/jais-30b-chat-v3 \ + --dtype float16 \ + --tp_size 2 \ + --output_dir jais-30b-chat-v3/trt_ckpt/fp16/2-gpu +``` + +```bash +# Build a single-GPU float16 engine from TensorRT LLM checkpoint for jais-13b-chat +# Enable the special TensorRT LLM GPT Attention plugin (--gpt_attention_plugin) to increase runtime performance. +# It is recommend to use --remove_input_padding along with --gpt_attention_plugin for better performance +trtllm-build --checkpoint_dir jais-13b-chat/trt_ckpt/fp16/1-gpu \ + --gpt_attention_plugin float16 \ + --remove_input_padding enable \ + --output_dir jais-13b-chat/trt_engines/fp16/1-gpu + +# Build 2-way tensor parallelism engines from TensorRT LLM checkpoint for jais-30b-chat-v3 +trtllm-build --checkpoint_dir jais-30b-chat-v3/trt_ckpt/fp16/2-gpu \ + --gpt_attention_plugin float16 \ + --remove_input_padding enable \ + --output_dir jais-30b-chat-v3/trt_engines/fp16/2-gpu +``` + + +### Run + +The [`../../../run.py`](../../../run.py) script can be used to run inference with the built engine(s). + +```bash +python3 ../../../run.py --engine_dir jais-13b-chat/trt_engines/fp16/1-gpu \ + --tokenizer_dir core42/jais-13b-chat \ + --max_output_len 10 +``` + +If the engines are run successfully, you will see output like: +``` +...... +Input [Text 0]: "Born in north-east France, Soyer trained as a" +Output [Text 0 Beam 0]: " chef in Paris before moving to England in 1816" +``` + +```bash +python3 ../../../run.py --engine_dir jais-13b-chat/trt_engines/fp16/1-gpu \ + --tokenizer_dir core42/jais-13b-chat \ + --max_output_len 8 \ + --input_text "ولد في 1304 ميلادياً ابن بطوطه, لقد ذهب" +``` + +If the engines are run successfully, you will see output like: +``` +..... +Input [Text 0]: "ولد في 1304 ميلادياً ابن بطوطه, لقد ذهب" +Output [Text 0 Beam 0]: " في جميع أنحاء العالم المعروف في ذلك الوقت" +``` + + +To run a 2 TP model you can do the following +```bash +mpirun -np 2 \ + python3 ../../../run.py --engine_dir jais-30b-chat-v3/trt_engines/fp16/2-gpu \ + --tokenizer_dir core42/jais-30b-chat-v3 \ + --max_output_len 30 +``` + +If the engines are run successfully, you will see output like: +``` +Input [Text 0]: "Born in north-east France, Soyer trained as a" +Output [Text 0 Beam 0]: " chef, working in a series of high-end establishments. + +Soyer's career took him to work in a number of establishments across Europe," +``` diff --git a/examples/models/contrib/jais/requirements.txt b/examples/models/contrib/jais/requirements.txt new file mode 100644 index 000000000000..592e01e5ba6d --- /dev/null +++ b/examples/models/contrib/jais/requirements.txt @@ -0,0 +1,6 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +datasets==3.1.0 +evaluate +rouge_score +SentencePiece>=0.1.99 diff --git a/examples/models/contrib/sdxl/README.md b/examples/models/contrib/sdxl/README.md new file mode 100644 index 000000000000..05d874f5f27a --- /dev/null +++ b/examples/models/contrib/sdxl/README.md @@ -0,0 +1,42 @@ +# Stable Diffusion XL + +This document showcases how to build and run the [Stable Diffusion XL (SDXL)](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) model on multiple GPUs using TensorRT-LLM. The community-contributed SDXL example in TRT-LLM is intended solely to showcase distributed inference for high-resolution use cases. For an optimized single-GPU setup in Stable Diffusion inference, please refer to the [TensorRT DemoDiffusion example](https://github.com/NVIDIA/TensorRT/tree/main/demo/Diffusion). + +The design of distributed parallel inference comes from the CVPR 2024 paper [DistriFusion](https://github.com/mit-han-lab/distrifuser) from [MIT HAN Lab](https://hanlab.mit.edu/). To simplify the implementation, all communications in this example are handled synchronously. + +## Usage + +### 1. Build TensorRT Engine + +```bash +# 1 gpu +python build_sdxl_unet.py --size 1024 + +# 2 gpus +mpirun -n 2 --allow-run-as-root python build_sdxl_unet.py --size 1024 +``` + +### 2. Generate images using the engine + + +```bash +# 1 gpu +python run_sdxl.py --size 1024 --prompt "flowers, rabbit" + +# 2 gpus +mpirun -n 2 --allow-run-as-root python run_sdxl.py --size 1024 --prompt "flowers, rabbit" +``` + +## Latency Benchmark +This benchmark is provided as reference points and should not be considered as the peak inference speed that can be delivered by TensorRT-LLM. + +| Framework | Resolution | n_gpu | A100 latency (s) | A100 speedup | H100 latency (s) | H100 speedup | +|:---------:|:----------:|:-----:|:---------------:|:-------------:|:---------------:|:-------------:| +| Torch | 1024x1024 | 1 | 6.280 | 1 | 5.820 | 1 | +| TRT-LLM | 1024x1024 | 2 | 2.803 | **2.24x** | 1.719 | **3.39x** | +| TRT-LLM | 1024x1024 | 4 | 2.962 | **2.12x** | 2.592 | **2.25x** | +| Torch | 2048x2048 | 1 | 27.865 | 1 | 18.330 | 1 | +| TRT-LLM | 2048x2048 | 2 | 13.152 | **2.12x** | 7.943 | **2.31x** | +| TRT-LLM | 2048x2048 | 4 | 9.781 | **2.85x** | 7.596 | **2.41x** | + +torch v2.5.0. TRT-LLM v0.15.0.dev2024102900, `--num-warmup-runs=5; --avg-runs=20`. All communications are synchronous. diff --git a/examples/models/contrib/sdxl/build_sdxl_unet.py b/examples/models/contrib/sdxl/build_sdxl_unet.py new file mode 100755 index 000000000000..e2893859f731 --- /dev/null +++ b/examples/models/contrib/sdxl/build_sdxl_unet.py @@ -0,0 +1,148 @@ +import argparse +import os + +import tensorrt as trt +import torch +from diffusers import DiffusionPipeline + +import tensorrt_llm +from tensorrt_llm.builder import Builder +from tensorrt_llm.mapping import Mapping +from tensorrt_llm.models.unet.pp.unet_pp import DistriUNetPP +from tensorrt_llm.models.unet.unet_2d_condition import UNet2DConditionModel +from tensorrt_llm.models.unet.weights import load_from_hf_unet +from tensorrt_llm.network import net_guard + +parser = argparse.ArgumentParser(description='build the UNet TensorRT engine.') +parser.add_argument('--model_dir', + type=str, + default='stabilityai/stable-diffusion-xl-base-1.0') +parser.add_argument('--size', type=int, default=1024, help='image size') +parser.add_argument('--output_dir', + type=str, + default=None, + help='output directory') + +args = parser.parse_args() + +model_dir = args.model_dir +size = args.size +sample_size = size // 8 + +world_size = tensorrt_llm.mpi_world_size() +rank = tensorrt_llm.mpi_rank() +output_dir = f'sdxl_s{size}_w{world_size}' if args.output_dir is None else args.output_dir +if rank == 0 and not os.path.exists(output_dir): + os.makedirs(output_dir) + +device_per_batch = world_size // 2 if world_size > 1 else 1 +batch_group = 2 if world_size > 1 else 1 + +# Use tp_size to indicate the size of patch parallelism +# Use pp_size to indicate the size of batch parallelism +mapping = Mapping(world_size=world_size, + rank=rank, + tp_size=device_per_batch, + pp_size=batch_group) + +torch.cuda.set_device(tensorrt_llm.mpi_rank()) + +tensorrt_llm.logger.set_level('verbose') +builder = Builder() +builder_config = builder.create_builder_config( + name='UNet2DConditionModel', + precision='float16', + timing_cache='model.cache', + profiling_verbosity='detailed', + tensor_parallel=world_size, + precision_constraints= + None, # do not use obey or the precision error will be too large +) + +pipeline = DiffusionPipeline.from_pretrained(model_dir, + torch_dtype=torch.float16) +model = UNet2DConditionModel( + sample_size=sample_size, + in_channels=4, + out_channels=4, + center_input_sample=False, + flip_sin_to_cos=True, + freq_shift=0, + down_block_types=("DownBlock2D", "CrossAttnDownBlock2D", + "CrossAttnDownBlock2D"), + up_block_types=("CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "UpBlock2D"), + block_out_channels=(320, 640, 1280), + layers_per_block=2, + downsample_padding=1, + mid_block_scale_factor=1.0, + act_fn="silu", + norm_num_groups=32, + norm_eps=1e-5, + cross_attention_dim=2048, + attention_head_dim=[5, 10, 20], + addition_embed_type="text_time", + addition_time_embed_dim=256, + projection_class_embeddings_input_dim=2816, + transformer_layers_per_block=[1, 2, 10], + use_linear_projection=True, + dtype=trt.float16, +) + +load_from_hf_unet(pipeline.unet, model) +model = DistriUNetPP(model, mapping) + +# Module -> Network +network = builder.create_network() +network.plugin_config.to_legacy_setting() +if mapping.world_size > 1: + network.plugin_config.set_nccl_plugin('float16') + +with net_guard(network): + # Prepare + network.set_named_parameters(model.named_parameters()) + + # Forward + sample = tensorrt_llm.Tensor( + name='sample', + dtype=trt.float16, + shape=[2, 4, sample_size, sample_size], + ) + timesteps = tensorrt_llm.Tensor( + name='timesteps', + dtype=trt.float16, + shape=[ + 1, + ], + ) + encoder_hidden_states = tensorrt_llm.Tensor( + name='encoder_hidden_states', + dtype=trt.float16, + shape=[2, 77, 2048], + ) + text_embeds = tensorrt_llm.Tensor( + name='text_embeds', + dtype=trt.float16, + shape=[2, 1280], + ) + time_ids = tensorrt_llm.Tensor( + name='time_ids', + dtype=trt.float16, + shape=[2, 6], + ) + + output = model(sample, timesteps, encoder_hidden_states, text_embeds, + time_ids) + + # Mark outputs + output_dtype = trt.float16 + output.mark_output('pred', output_dtype) + +# Network -> Engine +engine = builder.build_engine(network, builder_config) +assert engine is not None, 'Failed to build engine.' + +engine_name = f'sdxl_unet_s{size}_w{world_size}_r{rank}.engine' +engine_path = os.path.join(output_dir, engine_name) +with open(engine_path, 'wb') as f: + f.write(engine) +builder.save_config(builder_config, os.path.join(output_dir, 'config.json')) diff --git a/examples/models/contrib/sdxl/pipeline_stable_diffusion_xl.py b/examples/models/contrib/sdxl/pipeline_stable_diffusion_xl.py new file mode 100755 index 000000000000..018f3fad783f --- /dev/null +++ b/examples/models/contrib/sdxl/pipeline_stable_diffusion_xl.py @@ -0,0 +1,1365 @@ +# Copyright 2023 The HuggingFace Team. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import inspect +import json +import os +from typing import Any, Callable, Dict, List, Optional, Tuple, Union + +import tensorrt as trt +import torch +from diffusers.image_processor import PipelineImageInput, VaeImageProcessor +from diffusers.loaders import (FromSingleFileMixin, IPAdapterMixin, + StableDiffusionXLLoraLoaderMixin, + TextualInversionLoaderMixin) +from diffusers.models import AutoencoderKL, UNet2DConditionModel +from diffusers.models.attention_processor import (AttnProcessor2_0, + LoRAAttnProcessor2_0, + LoRAXFormersAttnProcessor, + XFormersAttnProcessor) +from diffusers.models.lora import adjust_lora_scale_text_encoder +from diffusers.pipelines.pipeline_utils import DiffusionPipeline +from diffusers.pipelines.stable_diffusion_xl.pipeline_output import \ + StableDiffusionXLPipelineOutput +from diffusers.schedulers import KarrasDiffusionSchedulers +from diffusers.utils import (USE_PEFT_BACKEND, deprecate, + is_invisible_watermark_available, + is_torch_xla_available, logging, + replace_example_docstring, scale_lora_layers, + unscale_lora_layers) +from diffusers.utils.torch_utils import randn_tensor +from transformers import (CLIPImageProcessor, CLIPTextModel, + CLIPTextModelWithProjection, CLIPTokenizer, + CLIPVisionModelWithProjection) + +import tensorrt_llm +from tensorrt_llm.runtime import Session, TensorInfo + +if is_invisible_watermark_available(): + from diffusers.pipelines.stable_diffusion_xl.watermark import \ + StableDiffusionXLWatermarker + +if is_torch_xla_available(): + import torch_xla.core.xla_model as xm + + XLA_AVAILABLE = True +else: + XLA_AVAILABLE = False + +logger = logging.get_logger(__name__) # pylint: disable=invalid-name + +EXAMPLE_DOC_STRING = """ + Examples: + ```py + >>> import torch + >>> from diffusers import StableDiffusionXLPipeline + + >>> pipe = StableDiffusionXLPipeline.from_pretrained( + ... "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16 + ... ) + >>> pipe = pipe.to("cuda") + + >>> prompt = "a photo of an astronaut riding a horse on mars" + >>> image = pipe(prompt).images[0] + ``` +""" + + +# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.rescale_noise_cfg +def rescale_noise_cfg(noise_cfg, noise_pred_text, guidance_rescale=0.0): + """ + Rescale `noise_cfg` according to `guidance_rescale`. Based on findings of [Common Diffusion Noise Schedules and + Sample Steps are Flawed](https://arxiv.org/pdf/2305.08891.pdf). See Section 3.4 + """ + std_text = noise_pred_text.std(dim=list(range(1, noise_pred_text.ndim)), + keepdim=True) + std_cfg = noise_cfg.std(dim=list(range(1, noise_cfg.ndim)), keepdim=True) + # rescale the results from guidance (fixes overexposure) + noise_pred_rescaled = noise_cfg * (std_text / std_cfg) + # mix with the original results from guidance by factor guidance_rescale to avoid "plain looking" images + noise_cfg = guidance_rescale * noise_pred_rescaled + \ + (1 - guidance_rescale) * noise_cfg + return noise_cfg + + +# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.retrieve_timesteps +def retrieve_timesteps( + scheduler, + num_inference_steps: Optional[int] = None, + device: Optional[Union[str, torch.device]] = None, + timesteps: Optional[List[int]] = None, + **kwargs, +): + """ + Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles + custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. + + Args: + scheduler (`SchedulerMixin`): + The scheduler to get timesteps from. + num_inference_steps (`int`): + The number of diffusion steps used when generating samples with a pre-trained model. If used, + `timesteps` must be `None`. + device (`str` or `torch.device`, *optional*): + The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. + timesteps (`List[int]`, *optional*): + Custom timesteps used to support arbitrary spacing between timesteps. If `None`, then the default + timestep spacing strategy of the scheduler is used. If `timesteps` is passed, `num_inference_steps` + must be `None`. + + Returns: + `Tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the + second element is the number of inference steps. + """ + if timesteps is not None: + accepts_timesteps = "timesteps" in set( + inspect.signature(scheduler.set_timesteps).parameters.keys()) + if not accepts_timesteps: + raise ValueError( + f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom" + f" timestep schedules. Please check whether you are using the correct scheduler." + ) + scheduler.set_timesteps(timesteps=timesteps, device=device, **kwargs) + timesteps = scheduler.timesteps + num_inference_steps = len(timesteps) + else: + scheduler.set_timesteps(num_inference_steps, device=device, **kwargs) + timesteps = scheduler.timesteps + return timesteps, num_inference_steps + + +class StableDiffusionXLPipeline( + DiffusionPipeline, + FromSingleFileMixin, + StableDiffusionXLLoraLoaderMixin, + TextualInversionLoaderMixin, + IPAdapterMixin, +): + r""" + Pipeline for text-to-image generation using Stable Diffusion XL. + + This model inherits from [`DiffusionPipeline`]. Check the superclass documentation for the generic methods the + library implements for all the pipelines (such as downloading or saving, running on a particular device, etc.) + + In addition the pipeline inherits the following loading methods: + - *LoRA*: [`loaders.StableDiffusionXLLoraLoaderMixin.load_lora_weights`] + - *Ckpt*: [`loaders.FromSingleFileMixin.from_single_file`] + + as well as the following saving methods: + - *LoRA*: [`loaders.StableDiffusionXLLoraLoaderMixin.save_lora_weights`] + + Args: + vae ([`AutoencoderKL`]): + Variational Auto-Encoder (VAE) Model to encode and decode images to and from latent representations. + text_encoder ([`CLIPTextModel`]): + Frozen text-encoder. Stable Diffusion XL uses the text portion of + [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModel), specifically + the [clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) variant. + text_encoder_2 ([` CLIPTextModelWithProjection`]): + Second frozen text-encoder. Stable Diffusion XL uses the text and pool portion of + [CLIP](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPTextModelWithProjection), + specifically the + [laion/CLIP-ViT-bigG-14-laion2B-39B-b160k](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k) + variant. + tokenizer (`CLIPTokenizer`): + Tokenizer of class + [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.21.0/en/model_doc/clip#transformers.CLIPTokenizer). + tokenizer_2 (`CLIPTokenizer`): + Second Tokenizer of class + [CLIPTokenizer](https://huggingface.co/docs/transformers/v4.21.0/en/model_doc/clip#transformers.CLIPTokenizer). + unet ([`UNet2DConditionModel`]): Conditional U-Net architecture to denoise the encoded image latents. + scheduler ([`SchedulerMixin`]): + A scheduler to be used in combination with `unet` to denoise the encoded image latents. Can be one of + [`DDIMScheduler`], [`LMSDiscreteScheduler`], or [`PNDMScheduler`]. + force_zeros_for_empty_prompt (`bool`, *optional*, defaults to `"True"`): + Whether the negative prompt embeddings shall be forced to always be set to 0. Also see the config of + `stabilityai/stable-diffusion-xl-base-1-0`. + add_watermarker (`bool`, *optional*): + Whether to use the [invisible_watermark library](https://github.com/ShieldMnt/invisible-watermark/) to + watermark output images. If not defined, it will default to True if the package is installed, otherwise no + watermarker will be used. + """ + + model_cpu_offload_seq = "text_encoder->text_encoder_2->unet->vae" + _optional_components = [ + "tokenizer", + "tokenizer_2", + "text_encoder", + "text_encoder_2", + "image_encoder", + "feature_extractor", + ] + _callback_tensor_inputs = [ + "latents", + "prompt_embeds", + "negative_prompt_embeds", + "add_text_embeds", + "add_time_ids", + "negative_pooled_prompt_embeds", + "negative_add_time_ids", + ] + + def __init__( + self, + vae: AutoencoderKL, + text_encoder: CLIPTextModel, + text_encoder_2: CLIPTextModelWithProjection, + tokenizer: CLIPTokenizer, + tokenizer_2: CLIPTokenizer, + unet: UNet2DConditionModel, + scheduler: KarrasDiffusionSchedulers, + image_encoder: CLIPVisionModelWithProjection = None, + feature_extractor: CLIPImageProcessor = None, + force_zeros_for_empty_prompt: bool = True, + add_watermarker: Optional[bool] = None, + ): + super().__init__() + + self.register_modules( + vae=vae, + text_encoder=text_encoder, + text_encoder_2=text_encoder_2, + tokenizer=tokenizer, + tokenizer_2=tokenizer_2, + unet=unet, + scheduler=scheduler, + image_encoder=image_encoder, + feature_extractor=feature_extractor, + ) + self.register_to_config( + force_zeros_for_empty_prompt=force_zeros_for_empty_prompt) + self.vae_scale_factor = 2**(len(self.vae.config.block_out_channels) - 1) + self.image_processor = VaeImageProcessor( + vae_scale_factor=self.vae_scale_factor) + + self.default_sample_size = self.unet.config.sample_size + + add_watermarker = add_watermarker if add_watermarker is not None else is_invisible_watermark_available( + ) + + if add_watermarker: + self.watermark = StableDiffusionXLWatermarker() + else: + self.watermark = None + + self.execution_device = torch.device('cpu') + self.engine = {} + + def to( + self, + torch_device: Optional[Union[str, torch.device]] = None, + torch_dtype: Optional[torch.dtype] = None, + silence_dtype_warnings: bool = False, + ): + super().to(torch_device) + if isinstance(torch_device, str): + torch_device = torch.device(torch_device) + self.execution_device = torch_device + return self + + def prepare(self, path, size): + self.unet.cpu() + torch.cuda.empty_cache() + + def trt_dtype_to_torch(dtype): + if dtype == trt.float16: + return torch.float16 + elif dtype == trt.float32: + return torch.float32 + elif dtype == trt.int32: + return torch.int32 + else: + raise TypeError("%s is not supported" % dtype) + + config_path = os.path.join(path, 'config.json') + with open(config_path, 'r') as f: + config = json.load(f) + config['builder_config']['precision'] + world_size = config['builder_config']['tensor_parallel'] + + runtime_world_size = tensorrt_llm.mpi_world_size() + assert world_size == runtime_world_size, f'Engine world size ({world_size}) != Runtime world size ({runtime_world_size})' + runtime_rank = tensorrt_llm.mpi_rank() if world_size > 1 else 0 + torch.cuda.set_device(runtime_rank) + + serialize_file = f'sdxl_unet_s{size}_w{world_size}_r{runtime_rank}.engine' + serialize_path = os.path.join(path, serialize_file) + self.stream = torch.cuda.current_stream().cuda_stream + print(f'Loading engine from {serialize_path}') + with open(serialize_path, 'rb') as f: + engine_buffer = f.read() + print(f'Creating session from engine') + self.session = Session.from_serialized_engine(engine_buffer) + + output_info = self.session.infer_shapes([ + TensorInfo('sample', trt.DataType.HALF, + [2, 4, size // 8, size // 8]), + TensorInfo('timesteps', trt.DataType.HALF, [ + 1, + ]), + TensorInfo('encoder_hidden_states', trt.DataType.HALF, + [2, 77, 2048]), + TensorInfo('text_embeds', trt.DataType.HALF, [2, 1280]), + TensorInfo('time_ids', trt.DataType.HALF, [2, 6]), + ]) + self.outputs = { + t.name: + torch.empty(tuple(t.shape), + dtype=trt_dtype_to_torch(t.dtype), + device='cuda') + for t in output_info + } + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.enable_vae_slicing + def enable_vae_slicing(self): + r""" + Enable sliced VAE decoding. When this option is enabled, the VAE will split the input tensor in slices to + compute decoding in several steps. This is useful to save some memory and allow larger batch sizes. + """ + self.vae.enable_slicing() + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.disable_vae_slicing + def disable_vae_slicing(self): + r""" + Disable sliced VAE decoding. If `enable_vae_slicing` was previously enabled, this method will go back to + computing decoding in one step. + """ + self.vae.disable_slicing() + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.enable_vae_tiling + def enable_vae_tiling(self): + r""" + Enable tiled VAE decoding. When this option is enabled, the VAE will split the input tensor into tiles to + compute decoding and encoding in several steps. This is useful for saving a large amount of memory and to allow + processing larger images. + """ + self.vae.enable_tiling() + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.disable_vae_tiling + def disable_vae_tiling(self): + r""" + Disable tiled VAE decoding. If `enable_vae_tiling` was previously enabled, this method will go back to + computing decoding in one step. + """ + self.vae.disable_tiling() + + def encode_prompt( + self, + prompt: str, + prompt_2: Optional[str] = None, + device: Optional[torch.device] = None, + num_images_per_prompt: int = 1, + do_classifier_free_guidance: bool = True, + negative_prompt: Optional[str] = None, + negative_prompt_2: Optional[str] = None, + prompt_embeds: Optional[torch.FloatTensor] = None, + negative_prompt_embeds: Optional[torch.FloatTensor] = None, + pooled_prompt_embeds: Optional[torch.FloatTensor] = None, + negative_pooled_prompt_embeds: Optional[torch.FloatTensor] = None, + lora_scale: Optional[float] = None, + clip_skip: Optional[int] = None, + ): + r""" + Encodes the prompt into text encoder hidden states. + + Args: + prompt (`str` or `List[str]`, *optional*): + prompt to be encoded + prompt_2 (`str` or `List[str]`, *optional*): + The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is + used in both text-encoders + device: (`torch.device`): + torch device + num_images_per_prompt (`int`): + number of images that should be generated per prompt + do_classifier_free_guidance (`bool`): + whether to use classifier free guidance or not + negative_prompt (`str` or `List[str]`, *optional*): + The prompt or prompts not to guide the image generation. If not defined, one has to pass + `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is + less than `1`). + negative_prompt_2 (`str` or `List[str]`, *optional*): + The prompt or prompts not to guide the image generation to be sent to `tokenizer_2` and + `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders + prompt_embeds (`torch.FloatTensor`, *optional*): + Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not + provided, text embeddings will be generated from `prompt` input argument. + negative_prompt_embeds (`torch.FloatTensor`, *optional*): + Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt + weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input + argument. + pooled_prompt_embeds (`torch.FloatTensor`, *optional*): + Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. + If not provided, pooled text embeddings will be generated from `prompt` input argument. + negative_pooled_prompt_embeds (`torch.FloatTensor`, *optional*): + Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt + weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt` + input argument. + lora_scale (`float`, *optional*): + A lora scale that will be applied to all LoRA layers of the text encoder if LoRA layers are loaded. + clip_skip (`int`, *optional*): + Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that + the output of the pre-final layer will be used for computing the prompt embeddings. + """ + device = self.execution_device + + # set lora scale so that monkey patched LoRA + # function of text encoder can correctly access it + if lora_scale is not None and isinstance( + self, StableDiffusionXLLoraLoaderMixin): + self._lora_scale = lora_scale + + # dynamically adjust the LoRA scale + if self.text_encoder is not None: + if not USE_PEFT_BACKEND: + adjust_lora_scale_text_encoder(self.text_encoder, + lora_scale) + else: + scale_lora_layers(self.text_encoder, lora_scale) + + if self.text_encoder_2 is not None: + if not USE_PEFT_BACKEND: + adjust_lora_scale_text_encoder(self.text_encoder_2, + lora_scale) + else: + scale_lora_layers(self.text_encoder_2, lora_scale) + + prompt = [prompt] if isinstance(prompt, str) else prompt + + if prompt is not None: + batch_size = len(prompt) + else: + batch_size = prompt_embeds.shape[0] + + # Define tokenizers and text encoders + tokenizers = [self.tokenizer, self.tokenizer_2 + ] if self.tokenizer is not None else [self.tokenizer_2] + text_encoders = ([ + self.text_encoder, self.text_encoder_2 + ] if self.text_encoder is not None else [self.text_encoder_2]) + + if prompt_embeds is None: + prompt_2 = prompt_2 or prompt + prompt_2 = [prompt_2] if isinstance(prompt_2, str) else prompt_2 + + # textual inversion: procecss multi-vector tokens if necessary + prompt_embeds_list = [] + prompts = [prompt, prompt_2] + for prompt, tokenizer, text_encoder in zip(prompts, tokenizers, + text_encoders): + if isinstance(self, TextualInversionLoaderMixin): + prompt = self.maybe_convert_prompt(prompt, tokenizer) + + text_inputs = tokenizer( + prompt, + padding="max_length", + max_length=tokenizer.model_max_length, + truncation=True, + return_tensors="pt", + ) + + text_input_ids = text_inputs.input_ids + untruncated_ids = tokenizer(prompt, + padding="longest", + return_tensors="pt").input_ids + + if untruncated_ids.shape[ + -1] >= text_input_ids.shape[-1] and not torch.equal( + text_input_ids, untruncated_ids): + removed_text = tokenizer.batch_decode( + untruncated_ids[:, tokenizer.model_max_length - 1:-1]) + logger.warning( + "The following part of your input was truncated because CLIP can only handle sequences up to" + f" {tokenizer.model_max_length} tokens: {removed_text}") + + prompt_embeds = text_encoder(text_input_ids.to(device), + output_hidden_states=True) + + # We are only ALWAYS interested in the pooled output of the final text encoder + pooled_prompt_embeds = prompt_embeds[0] + if clip_skip is None: + prompt_embeds = prompt_embeds.hidden_states[-2] + else: + # "2" because SDXL always indexes from the penultimate layer. + prompt_embeds = prompt_embeds.hidden_states[-(clip_skip + + 2)] + + prompt_embeds_list.append(prompt_embeds) + + prompt_embeds = torch.concat(prompt_embeds_list, dim=-1) + + # get unconditional embeddings for classifier free guidance + zero_out_negative_prompt = negative_prompt is None and self.config.force_zeros_for_empty_prompt + if do_classifier_free_guidance and negative_prompt_embeds is None and zero_out_negative_prompt: + negative_prompt_embeds = torch.zeros_like(prompt_embeds) + negative_pooled_prompt_embeds = torch.zeros_like( + pooled_prompt_embeds) + elif do_classifier_free_guidance and negative_prompt_embeds is None: + negative_prompt = negative_prompt or "" + negative_prompt_2 = negative_prompt_2 or negative_prompt + + # normalize str to list + negative_prompt = batch_size * \ + [negative_prompt] if isinstance( + negative_prompt, str) else negative_prompt + negative_prompt_2 = (batch_size * [negative_prompt_2] if isinstance( + negative_prompt_2, str) else negative_prompt_2) + + uncond_tokens: List[str] + if prompt is not None and type(prompt) is not type(negative_prompt): + raise TypeError( + f"`negative_prompt` should be the same type to `prompt`, but got {type(negative_prompt)} !=" + f" {type(prompt)}.") + elif batch_size != len(negative_prompt): + raise ValueError( + f"`negative_prompt`: {negative_prompt} has batch size {len(negative_prompt)}, but `prompt`:" + f" {prompt} has batch size {batch_size}. Please make sure that passed `negative_prompt` matches" + " the batch size of `prompt`.") + else: + uncond_tokens = [negative_prompt, negative_prompt_2] + + negative_prompt_embeds_list = [] + for negative_prompt, tokenizer, text_encoder in zip( + uncond_tokens, tokenizers, text_encoders): + if isinstance(self, TextualInversionLoaderMixin): + negative_prompt = self.maybe_convert_prompt( + negative_prompt, tokenizer) + + max_length = prompt_embeds.shape[1] + uncond_input = tokenizer( + negative_prompt, + padding="max_length", + max_length=max_length, + truncation=True, + return_tensors="pt", + ) + + negative_prompt_embeds = text_encoder( + uncond_input.input_ids.to(device), + output_hidden_states=True, + ) + # We are only ALWAYS interested in the pooled output of the final text encoder + negative_pooled_prompt_embeds = negative_prompt_embeds[0] + negative_prompt_embeds = negative_prompt_embeds.hidden_states[ + -2] + + negative_prompt_embeds_list.append(negative_prompt_embeds) + + negative_prompt_embeds = torch.concat(negative_prompt_embeds_list, + dim=-1) + + if self.text_encoder_2 is not None: + prompt_embeds = prompt_embeds.to(dtype=self.text_encoder_2.dtype, + device=device) + else: + prompt_embeds = prompt_embeds.to(dtype=self.unet.dtype, + device=device) + + bs_embed, seq_len, _ = prompt_embeds.shape + # duplicate text embeddings for each generation per prompt, using mps friendly method + prompt_embeds = prompt_embeds.repeat(1, num_images_per_prompt, 1) + prompt_embeds = prompt_embeds.view(bs_embed * num_images_per_prompt, + seq_len, -1) + + if do_classifier_free_guidance: + # duplicate unconditional embeddings for each generation per prompt, using mps friendly method + seq_len = negative_prompt_embeds.shape[1] + + if self.text_encoder_2 is not None: + negative_prompt_embeds = negative_prompt_embeds.to( + dtype=self.text_encoder_2.dtype, device=device) + else: + negative_prompt_embeds = negative_prompt_embeds.to( + dtype=self.unet.dtype, device=device) + + negative_prompt_embeds = negative_prompt_embeds.repeat( + 1, num_images_per_prompt, 1) + negative_prompt_embeds = negative_prompt_embeds.view( + batch_size * num_images_per_prompt, seq_len, -1) + + pooled_prompt_embeds = pooled_prompt_embeds.repeat( + 1, num_images_per_prompt).view(bs_embed * num_images_per_prompt, -1) + if do_classifier_free_guidance: + negative_pooled_prompt_embeds = negative_pooled_prompt_embeds.repeat( + 1, num_images_per_prompt).view(bs_embed * num_images_per_prompt, + -1) + + if self.text_encoder is not None: + if isinstance( + self, + StableDiffusionXLLoraLoaderMixin) and USE_PEFT_BACKEND: + # Retrieve the original scale by scaling back the LoRA layers + unscale_lora_layers(self.text_encoder, lora_scale) + + if self.text_encoder_2 is not None: + if isinstance( + self, + StableDiffusionXLLoraLoaderMixin) and USE_PEFT_BACKEND: + # Retrieve the original scale by scaling back the LoRA layers + unscale_lora_layers(self.text_encoder_2, lora_scale) + + return prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.encode_image + def encode_image(self, image, device, num_images_per_prompt): + dtype = next(self.image_encoder.parameters()).dtype + + if not isinstance(image, torch.Tensor): + image = self.feature_extractor(image, + return_tensors="pt").pixel_values + + image = image.to(device=device, dtype=dtype) + image_embeds = self.image_encoder(image).image_embeds + image_embeds = image_embeds.repeat_interleave(num_images_per_prompt, + dim=0) + + uncond_image_embeds = torch.zeros_like(image_embeds) + return image_embeds, uncond_image_embeds + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.prepare_extra_step_kwargs + def prepare_extra_step_kwargs(self, generator, eta): + # prepare extra kwargs for the scheduler step, since not all schedulers have the same signature + # eta (η) is only used with the DDIMScheduler, it will be ignored for other schedulers. + # eta corresponds to η in DDIM paper: https://arxiv.org/abs/2010.02502 + # and should be between [0, 1] + + accepts_eta = "eta" in set( + inspect.signature(self.scheduler.step).parameters.keys()) + extra_step_kwargs = {} + if accepts_eta: + extra_step_kwargs["eta"] = eta + + # check if the scheduler accepts generator + accepts_generator = "generator" in set( + inspect.signature(self.scheduler.step).parameters.keys()) + if accepts_generator: + extra_step_kwargs["generator"] = generator + return extra_step_kwargs + + def check_inputs( + self, + prompt, + prompt_2, + height, + width, + callback_steps, + negative_prompt=None, + negative_prompt_2=None, + prompt_embeds=None, + negative_prompt_embeds=None, + pooled_prompt_embeds=None, + negative_pooled_prompt_embeds=None, + callback_on_step_end_tensor_inputs=None, + ): + if height % 8 != 0 or width % 8 != 0: + raise ValueError( + f"`height` and `width` have to be divisible by 8 but are {height} and {width}." + ) + + if callback_steps is not None and (not isinstance(callback_steps, int) + or callback_steps <= 0): + raise ValueError( + f"`callback_steps` has to be a positive integer but is {callback_steps} of type" + f" {type(callback_steps)}.") + + if callback_on_step_end_tensor_inputs is not None and not all( + k in self._callback_tensor_inputs + for k in callback_on_step_end_tensor_inputs): + raise ValueError( + f"`callback_on_step_end_tensor_inputs` has to be in {self._callback_tensor_inputs}, but found {[k for k in callback_on_step_end_tensor_inputs if k not in self._callback_tensor_inputs]}" + ) + + if prompt is not None and prompt_embeds is not None: + raise ValueError( + f"Cannot forward both `prompt`: {prompt} and `prompt_embeds`: {prompt_embeds}. Please make sure to" + " only forward one of the two.") + elif prompt_2 is not None and prompt_embeds is not None: + raise ValueError( + f"Cannot forward both `prompt_2`: {prompt_2} and `prompt_embeds`: {prompt_embeds}. Please make sure to" + " only forward one of the two.") + elif prompt is None and prompt_embeds is None: + raise ValueError( + "Provide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined." + ) + elif prompt is not None and (not isinstance(prompt, str) + and not isinstance(prompt, list)): + raise ValueError( + f"`prompt` has to be of type `str` or `list` but is {type(prompt)}" + ) + elif prompt_2 is not None and (not isinstance(prompt_2, str) + and not isinstance(prompt_2, list)): + raise ValueError( + f"`prompt_2` has to be of type `str` or `list` but is {type(prompt_2)}" + ) + + if negative_prompt is not None and negative_prompt_embeds is not None: + raise ValueError( + f"Cannot forward both `negative_prompt`: {negative_prompt} and `negative_prompt_embeds`:" + f" {negative_prompt_embeds}. Please make sure to only forward one of the two." + ) + elif negative_prompt_2 is not None and negative_prompt_embeds is not None: + raise ValueError( + f"Cannot forward both `negative_prompt_2`: {negative_prompt_2} and `negative_prompt_embeds`:" + f" {negative_prompt_embeds}. Please make sure to only forward one of the two." + ) + + if prompt_embeds is not None and negative_prompt_embeds is not None: + if prompt_embeds.shape != negative_prompt_embeds.shape: + raise ValueError( + "`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but" + f" got: `prompt_embeds` {prompt_embeds.shape} != `negative_prompt_embeds`" + f" {negative_prompt_embeds.shape}.") + + if prompt_embeds is not None and pooled_prompt_embeds is None: + raise ValueError( + "If `prompt_embeds` are provided, `pooled_prompt_embeds` also have to be passed. Make sure to generate `pooled_prompt_embeds` from the same text encoder that was used to generate `prompt_embeds`." + ) + + if negative_prompt_embeds is not None and negative_pooled_prompt_embeds is None: + raise ValueError( + "If `negative_prompt_embeds` are provided, `negative_pooled_prompt_embeds` also have to be passed. Make sure to generate `negative_pooled_prompt_embeds` from the same text encoder that was used to generate `negative_prompt_embeds`." + ) + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.prepare_latents + def prepare_latents(self, + batch_size, + num_channels_latents, + height, + width, + dtype, + device, + generator, + latents=None): + shape = (batch_size, num_channels_latents, + height // self.vae_scale_factor, + width // self.vae_scale_factor) + if isinstance(generator, list) and len(generator) != batch_size: + raise ValueError( + f"You have passed a list of generators of length {len(generator)}, but requested an effective batch" + f" size of {batch_size}. Make sure the batch size matches the length of the generators." + ) + + if latents is None: + latents = randn_tensor(shape, + generator=generator, + device=device, + dtype=dtype) + else: + latents = latents.to(device) + + # scale the initial noise by the standard deviation required by the scheduler + latents = latents * self.scheduler.init_noise_sigma + return latents + + def _get_add_time_ids(self, + original_size, + crops_coords_top_left, + target_size, + dtype, + text_encoder_projection_dim=None): + add_time_ids = list(original_size + crops_coords_top_left + target_size) + + passed_add_embed_dim = ( + self.unet.config.addition_time_embed_dim * len(add_time_ids) + + text_encoder_projection_dim) + expected_add_embed_dim = self.unet.add_embedding.linear_1.in_features + + if expected_add_embed_dim != passed_add_embed_dim: + raise ValueError( + f"Model expects an added time embedding vector of length {expected_add_embed_dim}, but a vector of {passed_add_embed_dim} was created. The model has an incorrect config. Please check `unet.config.time_embedding_type` and `text_encoder_2.config.projection_dim`." + ) + + add_time_ids = torch.tensor([add_time_ids], dtype=dtype) + return add_time_ids + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion_upscale.StableDiffusionUpscalePipeline.upcast_vae + def upcast_vae(self): + dtype = self.vae.dtype + self.vae.to(dtype=torch.float32) + use_torch_2_0_or_xformers = isinstance( + self.vae.decoder.mid_block.attentions[0].processor, + ( + AttnProcessor2_0, + XFormersAttnProcessor, + LoRAXFormersAttnProcessor, + LoRAAttnProcessor2_0, + ), + ) + # if xformers or torch_2_0 is used attention block does not need + # to be in float32 which can save lots of memory + if use_torch_2_0_or_xformers: + self.vae.post_quant_conv.to(dtype) + self.vae.decoder.conv_in.to(dtype) + self.vae.decoder.mid_block.to(dtype) + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.enable_freeu + def enable_freeu(self, s1: float, s2: float, b1: float, b2: float): + r"""Enables the FreeU mechanism as in https://arxiv.org/abs/2309.11497. + + The suffixes after the scaling factors represent the stages where they are being applied. + + Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of the values + that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. + + Args: + s1 (`float`): + Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to + mitigate "oversmoothing effect" in the enhanced denoising process. + s2 (`float`): + Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to + mitigate "oversmoothing effect" in the enhanced denoising process. + b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. + b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. + """ + if not hasattr(self, "unet"): + raise ValueError("The pipeline must have `unet` for using FreeU.") + self.unet.enable_freeu(s1=s1, s2=s2, b1=b1, b2=b2) + + # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.disable_freeu + def disable_freeu(self): + """Disables the FreeU mechanism if enabled.""" + self.unet.disable_freeu() + + # Copied from diffusers.pipelines.latent_consistency_models.pipeline_latent_consistency_text2img.LatentConsistencyModelPipeline.get_guidance_scale_embedding + def get_guidance_scale_embedding(self, + w, + embedding_dim=512, + dtype=torch.float32): + """ + See https://github.com/google-research/vdm/blob/dc27b98a554f65cdc654b800da5aa1846545d41b/model_vdm.py#L298 + + Args: + timesteps (`torch.Tensor`): + generate embedding vectors at these timesteps + embedding_dim (`int`, *optional*, defaults to 512): + dimension of the embeddings to generate + dtype: + data type of the generated embeddings + + Returns: + `torch.FloatTensor`: Embedding vectors with shape `(len(timesteps), embedding_dim)` + """ + assert len(w.shape) == 1 + w = w * 1000.0 + + half_dim = embedding_dim // 2 + emb = torch.log(torch.tensor(10000.0)) / (half_dim - 1) + emb = torch.exp(torch.arange(half_dim, dtype=dtype) * -emb) + emb = w.to(dtype)[:, None] * emb[None, :] + emb = torch.cat([torch.sin(emb), torch.cos(emb)], dim=1) + if embedding_dim % 2 == 1: # zero pad + emb = torch.nn.functional.pad(emb, (0, 1)) + assert emb.shape == (w.shape[0], embedding_dim) + return emb + + @property + def guidance_scale(self): + return self._guidance_scale + + @property + def guidance_rescale(self): + return self._guidance_rescale + + @property + def clip_skip(self): + return self._clip_skip + + # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2) + # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1` + # corresponds to doing no classifier free guidance. + @property + def do_classifier_free_guidance(self): + return self._guidance_scale > 1 and self.unet.config.time_cond_proj_dim is None + + @property + def cross_attention_kwargs(self): + return self._cross_attention_kwargs + + @property + def denoising_end(self): + return self._denoising_end + + @property + def num_timesteps(self): + return self._num_timesteps + + @torch.no_grad() + @replace_example_docstring(EXAMPLE_DOC_STRING) + def __call__( + self, + prompt: Union[str, List[str]] = None, + prompt_2: Optional[Union[str, List[str]]] = None, + height: Optional[int] = None, + width: Optional[int] = None, + num_inference_steps: int = 50, + timesteps: List[int] = None, + denoising_end: Optional[float] = None, + guidance_scale: float = 5.0, + negative_prompt: Optional[Union[str, List[str]]] = None, + negative_prompt_2: Optional[Union[str, List[str]]] = None, + num_images_per_prompt: Optional[int] = 1, + eta: float = 0.0, + generator: Optional[Union[torch.Generator, + List[torch.Generator]]] = None, + latents: Optional[torch.FloatTensor] = None, + prompt_embeds: Optional[torch.FloatTensor] = None, + negative_prompt_embeds: Optional[torch.FloatTensor] = None, + pooled_prompt_embeds: Optional[torch.FloatTensor] = None, + negative_pooled_prompt_embeds: Optional[torch.FloatTensor] = None, + ip_adapter_image: Optional[PipelineImageInput] = None, + output_type: Optional[str] = "pil", + return_dict: bool = True, + cross_attention_kwargs: Optional[Dict[str, Any]] = None, + guidance_rescale: float = 0.0, + original_size: Optional[Tuple[int, int]] = None, + crops_coords_top_left: Tuple[int, int] = (0, 0), + target_size: Optional[Tuple[int, int]] = None, + negative_original_size: Optional[Tuple[int, int]] = None, + negative_crops_coords_top_left: Tuple[int, int] = (0, 0), + negative_target_size: Optional[Tuple[int, int]] = None, + clip_skip: Optional[int] = None, + callback_on_step_end: Optional[Callable[[int, int, Dict], None]] = None, + callback_on_step_end_tensor_inputs: List[str] = ["latents"], + **kwargs, + ): + r""" + Function invoked when calling the pipeline for generation. + + Args: + prompt (`str` or `List[str]`, *optional*): + The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. + instead. + prompt_2 (`str` or `List[str]`, *optional*): + The prompt or prompts to be sent to the `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is + used in both text-encoders + height (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor): + The height in pixels of the generated image. This is set to 1024 by default for the best results. + Anything below 512 pixels won't work well for + [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) + and checkpoints that are not specifically fine-tuned on low resolutions. + width (`int`, *optional*, defaults to self.unet.config.sample_size * self.vae_scale_factor): + The width in pixels of the generated image. This is set to 1024 by default for the best results. + Anything below 512 pixels won't work well for + [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) + and checkpoints that are not specifically fine-tuned on low resolutions. + num_inference_steps (`int`, *optional*, defaults to 50): + The number of denoising steps. More denoising steps usually lead to a higher quality image at the + expense of slower inference. + timesteps (`List[int]`, *optional*): + Custom timesteps to use for the denoising process with schedulers which support a `timesteps` argument + in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is + passed will be used. Must be in descending order. + denoising_end (`float`, *optional*): + When specified, determines the fraction (between 0.0 and 1.0) of the total denoising process to be + completed before it is intentionally prematurely terminated. As a result, the returned sample will + still retain a substantial amount of noise as determined by the discrete timesteps selected by the + scheduler. The denoising_end parameter should ideally be utilized when this pipeline forms a part of a + "Mixture of Denoisers" multi-pipeline setup, as elaborated in [**Refining the Image + Output**](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl#refining-the-image-output) + guidance_scale (`float`, *optional*, defaults to 5.0): + Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://arxiv.org/abs/2207.12598). + `guidance_scale` is defined as `w` of equation 2. of [Imagen + Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale > + 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, + usually at the expense of lower image quality. + negative_prompt (`str` or `List[str]`, *optional*): + The prompt or prompts not to guide the image generation. If not defined, one has to pass + `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is + less than `1`). + negative_prompt_2 (`str` or `List[str]`, *optional*): + The prompt or prompts not to guide the image generation to be sent to `tokenizer_2` and + `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders + num_images_per_prompt (`int`, *optional*, defaults to 1): + The number of images to generate per prompt. + eta (`float`, *optional*, defaults to 0.0): + Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to + [`schedulers.DDIMScheduler`], will be ignored for others. + generator (`torch.Generator` or `List[torch.Generator]`, *optional*): + One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) + to make generation deterministic. + latents (`torch.FloatTensor`, *optional*): + Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for image + generation. Can be used to tweak the same generation with different prompts. If not provided, a latents + tensor will ge generated by sampling using the supplied random `generator`. + prompt_embeds (`torch.FloatTensor`, *optional*): + Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not + provided, text embeddings will be generated from `prompt` input argument. + negative_prompt_embeds (`torch.FloatTensor`, *optional*): + Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt + weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input + argument. + pooled_prompt_embeds (`torch.FloatTensor`, *optional*): + Pre-generated pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. + If not provided, pooled text embeddings will be generated from `prompt` input argument. + negative_pooled_prompt_embeds (`torch.FloatTensor`, *optional*): + Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt + weighting. If not provided, pooled negative_prompt_embeds will be generated from `negative_prompt` + input argument. + ip_adapter_image: (`PipelineImageInput`, *optional*): Optional image input to work with IP Adapters. + output_type (`str`, *optional*, defaults to `"pil"`): + The output format of the generate image. Choose between + [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. + return_dict (`bool`, *optional*, defaults to `True`): + Whether or not to return a [`~pipelines.stable_diffusion_xl.StableDiffusionXLPipelineOutput`] instead + of a plain tuple. + cross_attention_kwargs (`dict`, *optional*): + A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under + `self.processor` in + [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). + guidance_rescale (`float`, *optional*, defaults to 0.0): + Guidance rescale factor proposed by [Common Diffusion Noise Schedules and Sample Steps are + Flawed](https://arxiv.org/pdf/2305.08891.pdf) `guidance_scale` is defined as `φ` in equation 16. of + [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://arxiv.org/pdf/2305.08891.pdf). + Guidance rescale factor should fix overexposure when using zero terminal SNR. + original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): + If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled. + `original_size` defaults to `(height, width)` if not specified. Part of SDXL's micro-conditioning as + explained in section 2.2 of + [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). + crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)): + `crops_coords_top_left` can be used to generate an image that appears to be "cropped" from the position + `crops_coords_top_left` downwards. Favorable, well-centered images are usually achieved by setting + `crops_coords_top_left` to (0, 0). Part of SDXL's micro-conditioning as explained in section 2.2 of + [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). + target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): + For most cases, `target_size` should be set to the desired height and width of the generated image. If + not specified it will default to `(height, width)`. Part of SDXL's micro-conditioning as explained in + section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). + negative_original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): + To negatively condition the generation process based on a specific image resolution. Part of SDXL's + micro-conditioning as explained in section 2.2 of + [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more + information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. + negative_crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)): + To negatively condition the generation process based on a specific crop coordinates. Part of SDXL's + micro-conditioning as explained in section 2.2 of + [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more + information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. + negative_target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)): + To negatively condition the generation process based on a target image resolution. It should be as same + as the `target_size` for most cases. Part of SDXL's micro-conditioning as explained in section 2.2 of + [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more + information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208. + callback_on_step_end (`Callable`, *optional*): + A function that calls at the end of each denoising steps during the inference. The function is called + with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, + callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by + `callback_on_step_end_tensor_inputs`. + callback_on_step_end_tensor_inputs (`List`, *optional*): + The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list + will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the + `._callback_tensor_inputs` attribute of your pipeline class. + + Examples: + + Returns: + [`~pipelines.stable_diffusion_xl.StableDiffusionXLPipelineOutput`] or `tuple`: + [`~pipelines.stable_diffusion_xl.StableDiffusionXLPipelineOutput`] if `return_dict` is True, otherwise a + `tuple`. When returning a tuple, the first element is a list with the generated images. + """ + + callback = kwargs.pop("callback", None) + callback_steps = kwargs.pop("callback_steps", None) + + if callback is not None: + deprecate( + "callback", + "1.0.0", + "Passing `callback` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`", + ) + if callback_steps is not None: + deprecate( + "callback_steps", + "1.0.0", + "Passing `callback_steps` as an input argument to `__call__` is deprecated, consider use `callback_on_step_end`", + ) + + # 0. Default height and width to unet + height = height or self.default_sample_size * self.vae_scale_factor + width = width or self.default_sample_size * self.vae_scale_factor + + original_size = original_size or (height, width) + target_size = target_size or (height, width) + + # 1. Check inputs. Raise error if not correct + self.check_inputs( + prompt, + prompt_2, + height, + width, + callback_steps, + negative_prompt, + negative_prompt_2, + prompt_embeds, + negative_prompt_embeds, + pooled_prompt_embeds, + negative_pooled_prompt_embeds, + callback_on_step_end_tensor_inputs, + ) + + self._guidance_scale = guidance_scale + self._guidance_rescale = guidance_rescale + self._clip_skip = clip_skip + self._cross_attention_kwargs = cross_attention_kwargs + self._denoising_end = denoising_end + + # 2. Define call parameters + if prompt is not None and isinstance(prompt, str): + batch_size = 1 + elif prompt is not None and isinstance(prompt, list): + batch_size = len(prompt) + else: + batch_size = prompt_embeds.shape[0] + + device = self.execution_device + + # 3. Encode input prompt + lora_scale = (self.cross_attention_kwargs.get("scale", None) + if self.cross_attention_kwargs is not None else None) + + ( + prompt_embeds, + negative_prompt_embeds, + pooled_prompt_embeds, + negative_pooled_prompt_embeds, + ) = self.encode_prompt( + prompt=prompt, + prompt_2=prompt_2, + device=device, + num_images_per_prompt=num_images_per_prompt, + do_classifier_free_guidance=self.do_classifier_free_guidance, + negative_prompt=negative_prompt, + negative_prompt_2=negative_prompt_2, + prompt_embeds=prompt_embeds, + negative_prompt_embeds=negative_prompt_embeds, + pooled_prompt_embeds=pooled_prompt_embeds, + negative_pooled_prompt_embeds=negative_pooled_prompt_embeds, + lora_scale=lora_scale, + clip_skip=self.clip_skip, + ) + + # 4. Prepare timesteps + timesteps, num_inference_steps = retrieve_timesteps( + self.scheduler, num_inference_steps, device, timesteps) + + # 5. Prepare latent variables + num_channels_latents = self.unet.config.in_channels + latents = self.prepare_latents( + batch_size * num_images_per_prompt, + num_channels_latents, + height, + width, + prompt_embeds.dtype, + device, + generator, + latents, + ) + + # 6. Prepare extra step kwargs. TODO: Logic should ideally just be moved out of the pipeline + extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta) + + # 7. Prepare added time ids & embeddings + add_text_embeds = pooled_prompt_embeds + if self.text_encoder_2 is None: + text_encoder_projection_dim = int(pooled_prompt_embeds.shape[-1]) + else: + text_encoder_projection_dim = self.text_encoder_2.config.projection_dim + + add_time_ids = self._get_add_time_ids( + original_size, + crops_coords_top_left, + target_size, + dtype=prompt_embeds.dtype, + text_encoder_projection_dim=text_encoder_projection_dim, + ) + if negative_original_size is not None and negative_target_size is not None: + negative_add_time_ids = self._get_add_time_ids( + negative_original_size, + negative_crops_coords_top_left, + negative_target_size, + dtype=prompt_embeds.dtype, + text_encoder_projection_dim=text_encoder_projection_dim, + ) + else: + negative_add_time_ids = add_time_ids + + if self.do_classifier_free_guidance: + prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds], + dim=0) + add_text_embeds = torch.cat( + [negative_pooled_prompt_embeds, add_text_embeds], dim=0) + add_time_ids = torch.cat([negative_add_time_ids, add_time_ids], + dim=0) + + prompt_embeds = prompt_embeds.to(device) + add_text_embeds = add_text_embeds.to(device) + add_time_ids = add_time_ids.to(device).repeat( + batch_size * num_images_per_prompt, 1) + + if ip_adapter_image is not None: + image_embeds, negative_image_embeds = self.encode_image( + ip_adapter_image, device, num_images_per_prompt) + if self.do_classifier_free_guidance: + image_embeds = torch.cat([negative_image_embeds, image_embeds]) + image_embeds = image_embeds.to(device) + + # 8. Denoising loop + num_warmup_steps = max( + len(timesteps) - num_inference_steps * self.scheduler.order, 0) + + # 8.1 Apply denoising_end + if (self.denoising_end is not None + and isinstance(self.denoising_end, float) + and self.denoising_end > 0 and self.denoising_end < 1): + discrete_timestep_cutoff = int( + round(self.scheduler.config.num_train_timesteps - + (self.denoising_end * + self.scheduler.config.num_train_timesteps))) + num_inference_steps = len( + list( + filter(lambda ts: ts >= discrete_timestep_cutoff, + timesteps))) + timesteps = timesteps[:num_inference_steps] + + # 9. Optionally get Guidance Scale Embedding + timestep_cond = None + if self.unet.config.time_cond_proj_dim is not None: + guidance_scale_tensor = torch.tensor(self.guidance_scale - + 1).repeat( + batch_size * + num_images_per_prompt) + timestep_cond = self.get_guidance_scale_embedding( + guidance_scale_tensor, + embedding_dim=self.unet.config.time_cond_proj_dim).to( + device=device, dtype=latents.dtype) + + self._num_timesteps = len(timesteps) + with self.progress_bar(total=num_inference_steps) as progress_bar: + for i, t in enumerate(timesteps): + # expand the latents if we are doing classifier free guidance + latent_model_input = torch.cat( + [latents] * + 2) if self.do_classifier_free_guidance else latents + + latent_model_input = self.scheduler.scale_model_input( + latent_model_input, t) + + # predict the noise residual + added_cond_kwargs = { + "text_embeds": add_text_embeds, + "time_ids": add_time_ids + } + if ip_adapter_image is not None: + added_cond_kwargs["image_embeds"] = image_embeds + + t = t.to(latent_model_input.dtype) + feed_dict = { + 'sample': latent_model_input, + 'timesteps': t.unsqueeze(0), + 'encoder_hidden_states': prompt_embeds, + 'text_embeds': add_text_embeds, + 'time_ids': add_time_ids, + } + ok = self.session.run(feed_dict, self.outputs, self.stream) + assert ok, "Runtime execution failed" + noise_pred = self.outputs['pred'] + torch.cuda.synchronize() + + # perform guidance + if self.do_classifier_free_guidance: + noise_pred_uncond, noise_pred_text = noise_pred.chunk(2) + noise_pred = noise_pred_uncond + self.guidance_scale * \ + (noise_pred_text - noise_pred_uncond) + + if self.do_classifier_free_guidance and self.guidance_rescale > 0.0: + # Based on 3.4. in https://arxiv.org/pdf/2305.08891.pdf + noise_pred = rescale_noise_cfg( + noise_pred, + noise_pred_text, + guidance_rescale=self.guidance_rescale) + + # compute the previous noisy sample x_t -> x_t-1 + latents = self.scheduler.step(noise_pred, + t, + latents, + **extra_step_kwargs, + return_dict=False)[0] + + if callback_on_step_end is not None: + callback_kwargs = {} + for k in callback_on_step_end_tensor_inputs: + callback_kwargs[k] = locals()[k] + callback_outputs = callback_on_step_end( + self, i, t, callback_kwargs) + + latents = callback_outputs.pop("latents", latents) + prompt_embeds = callback_outputs.pop( + "prompt_embeds", prompt_embeds) + negative_prompt_embeds = callback_outputs.pop( + "negative_prompt_embeds", negative_prompt_embeds) + add_text_embeds = callback_outputs.pop( + "add_text_embeds", add_text_embeds) + negative_pooled_prompt_embeds = callback_outputs.pop( + "negative_pooled_prompt_embeds", + negative_pooled_prompt_embeds) + add_time_ids = callback_outputs.pop("add_time_ids", + add_time_ids) + negative_add_time_ids = callback_outputs.pop( + "negative_add_time_ids", negative_add_time_ids) + + # call the callback, if provided + if i == len(timesteps) - 1 or ( + (i + 1) > num_warmup_steps and + (i + 1) % self.scheduler.order == 0): + progress_bar.update() + if callback is not None and i % callback_steps == 0: + step_idx = i // getattr(self.scheduler, "order", 1) + callback(step_idx, t, latents) + + if XLA_AVAILABLE: + xm.mark_step() + + if not output_type == "latent": + # make sure the VAE is in float32 mode, as it overflows in float16 + needs_upcasting = self.vae.dtype == torch.float16 and self.vae.config.force_upcast + + if needs_upcasting: + self.upcast_vae() + latents = latents.to( + next(iter(self.vae.post_quant_conv.parameters())).dtype) + + image = self.vae.decode(latents / self.vae.config.scaling_factor, + return_dict=False)[0] + + # cast back to fp16 if needed + if needs_upcasting: + self.vae.to(dtype=torch.float16) + else: + image = latents + + if not output_type == "latent": + # apply watermark if available + if self.watermark is not None: + image = self.watermark.apply_watermark(image) + + image = self.image_processor.postprocess(image, + output_type=output_type) + + # Offload all models + self.maybe_free_model_hooks() + + if not return_dict: + return (image, ) + + return StableDiffusionXLPipelineOutput(images=image) diff --git a/examples/models/contrib/sdxl/run_sdxl.py b/examples/models/contrib/sdxl/run_sdxl.py new file mode 100755 index 000000000000..077b93f9ab7e --- /dev/null +++ b/examples/models/contrib/sdxl/run_sdxl.py @@ -0,0 +1,96 @@ +import argparse +import time + +import numpy as np +import torch +from pipeline_stable_diffusion_xl import StableDiffusionXLPipeline + +import tensorrt_llm + +world_size = tensorrt_llm.mpi_world_size() +rank = tensorrt_llm.mpi_rank() + + +def parseArgs(): + parser = argparse.ArgumentParser( + description='run SDXL with the UNet TensorRT engine.') + parser.add_argument('--model_dir', + type=str, + default='stabilityai/stable-diffusion-xl-base-1.0') + parser.add_argument('--size', type=int, default=1024) + parser.add_argument('--seed', type=int, default=233) + parser.add_argument('--num_inference_steps', type=int, default=50) + parser.add_argument( + '--prompt', + type=str, + default= + "masterpiece, gouache painting, 1girl, distant view, lone boat, willow trees" + ) + parser.add_argument('--engine_dir', + type=str, + default=None, + help='engine directory') + parser.add_argument('--num-warmup-runs', type=int, default=3) + parser.add_argument('--avg-runs', type=int, default=10) + parser.add_argument("--ignore_ratio", + type=float, + default=0.2, + help="Ignored ratio of the slowest and fastest steps") + parser.add_argument("--output", + type=str, + default="output.png", + help="Output file name") + return parser.parse_args() + + +if __name__ == "__main__": + args = parseArgs() + model_dir = args.model_dir + size = args.size + seed = args.seed + prompt = args.prompt + num_inference_steps = args.num_inference_steps + engine_dir = f'sdxl_s{size}_w{world_size}' if args.engine_dir is None else args.engine_dir + num_warmup_runs = args.num_warmup_runs + avg_runs = args.avg_runs + output_file = args.output + + pipeline = StableDiffusionXLPipeline.from_pretrained( + model_dir, + torch_dtype=torch.float16, + use_safetensors=True, + ) + pipeline.set_progress_bar_config(disable=rank != 0) + pipeline.prepare(engine_dir, size) + pipeline.to('cuda') + + # warm up + for i in range(num_warmup_runs): + image = pipeline( + num_inference_steps=num_inference_steps, + prompt=prompt, + generator=torch.Generator(device="cuda").manual_seed(seed), + height=size, + width=size).images[0] + + latency_list = [] + for i in range(avg_runs): + st = time.time() + image = pipeline( + num_inference_steps=num_inference_steps, + prompt=prompt, + generator=torch.Generator(device="cuda").manual_seed(seed), + height=size, + width=size, + ).images[0] + ed = time.time() + latency_list.append(ed - st) + + latency_list = sorted(latency_list) + ignored_count = int(args.ignore_ratio * len(latency_list) / 2) + if ignored_count > 0: + latency_list = latency_list[ignored_count:-ignored_count] + + if rank == 0: + print(f"Avg latency: {np.sum(latency_list) / len(latency_list):.5f} s") + image.save(output_file) diff --git a/examples/models/contrib/skywork/README.md b/examples/models/contrib/skywork/README.md new file mode 100644 index 000000000000..59b22a861987 --- /dev/null +++ b/examples/models/contrib/skywork/README.md @@ -0,0 +1,113 @@ +# Skywork + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document elaborates how to build the [Skywork](https://huggingface.co/Skywork/) model to runnable engines on single GPU node and perform a summarization task using these engines. + +## Overview +The TensorRT LLM Skywork implementation is based on the LLaMA model. The implementation can +be found in [tensorrt_llm/models/llama/model.py](../../../../tensorrt_llm/models/llama/model.py). +The TensorRT LLM Skywork example code lies in [`examples/models/contrib/skywork`](./): + +* [`convert_checkpoint.py`](../../core/llama/convert_checkpoint.py) converts the Huggingface Model of Skywork into TensorRT LLM checkpoint. + +In addition, there are two shared files in the parent folder [`examples`](../../../) for inference and evaluation: + +* [`../../../run.py`](../../../run.py) to run the inference on an input text; +* [`../../../summarize.py`](../../../summarize.py) to summarize the articles in the [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +## Support Matrix + * FP16 & BF16 + +## Usage + +This section gives a whole process where we convert HF models, build TensorRT LLM engines and ultimately perform summarization. + +### 1. Clone Code and Weights from Huggingface + +To download checkpoints from HF, you need to have `git-lfs` installed in your machine: + +```bash +pip install -r requirements.txt && sudo apt-get install git-lfs +``` + +Then clone the HF repository with: + +```bash +# Skywork 13B Base Model +git clone https://huggingface.co/Skywork/Skywork-13B-base +``` + +### 2. Convert HF Model to TRT Checkpoint + +```bash +cd examples/models/core/llama + +# fp16 model +python3 convert_checkpoint.py --model_dir ./Skywork-13B-base \ + --dtype float16 \ + --output_dir ./skywork-13b-base/trt_ckpt/fp16 + +# bf16 model +python3 convert_checkpoint.py --model_dir ./Skywork-13B-base \ + --dtype bfloat16 \ + --output_dir ./skywork-13b-base/trt_ckpt/bf16 +``` + +### 3. Build TensorRT Engine(s) + +```bash +# fp16 +trtllm-build --checkpoint_dir ./skywork-13b-base/trt_ckpt/fp16 \ + --gemm_plugin float16 \ + --gpt_attention_plugin float16 \ + --context_fmha enable \ + --max_batch_size 32 \ + --max_input_len 512 \ + --max_seq_len 1024 \ + --output_dir ./skywork-13b-base/trt_engine/fp16 + +# bf16 +trtllm-build --checkpoint_dir ./skywork-13b-base/trt_ckpt/bf16 \ + --gemm_plugin bfloat16 \ + --gpt_attention_plugin bfloat16 \ + --context_fmha enable \ + --max_batch_size 32 \ + --max_input_len 512 \ + --max_seq_len 1024 \ + --output_dir ./skywork-13b-base/trt_engine/bf16 +``` + +### 4. Summarization using the Engines + +After building TRT engines, we can use them to perform various tasks. TensorRT LLM provides handy code to run summarization on [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset and get [ROUGE](https://en.wikipedia.org/wiki/ROUGE_(metric)) scores. The `ROUGE-1` score can be used to validate model implementations. + +```bash +# fp16 +python ../../../summarize.py --hf_model_dir ./Skywork-13B-base \ + --test_hf \ + --batch_size 32 \ + --max_input_length 512 \ + --output_len 512 \ + --test_trt_llm \ + --engine_dir ./skywork-13b-base/trt_engine/fp16 \ + --data_type fp16 \ + --check_accuracy \ + --tensorrt_llm_rouge1_threshold=14 + +# bf16 +python ../../../summarize.py --hf_model_dir ./Skywork-13B-base \ + --test_hf \ + --batch_size 32 \ + --max_input_length 512 \ + --output_len 512 \ + --test_trt_llm \ + --engine_dir ./skywork-13b-base/trt_engine/bf16 \ + --data_type bf16 \ + --check_accuracy \ + --tensorrt_llm_rouge1_threshold=14 +``` diff --git a/examples/models/contrib/skywork/requirements.txt b/examples/models/contrib/skywork/requirements.txt new file mode 100644 index 000000000000..88232baef811 --- /dev/null +++ b/examples/models/contrib/skywork/requirements.txt @@ -0,0 +1,6 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +datasets==3.1.0 +evaluate +rouge_score +sentencepiece>=0.1.99 diff --git a/examples/models/contrib/smaug/README.md b/examples/models/contrib/smaug/README.md new file mode 100644 index 000000000000..2d96da5854e1 --- /dev/null +++ b/examples/models/contrib/smaug/README.md @@ -0,0 +1,62 @@ +# Smaug + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document elaborates how to build the [Smaug-72B-v0.1](https://huggingface.co/abacusai/Smaug-72B-v0.1) model to runnable engines on multi-GPU node and perform a summarization task using these engines. + +## Overview + +The TensorRT LLM support for Smaug-72B-v0.1 is based on the LLaMA model, the implementation can be found in [tensorrt_llm/models/llama/model.py](../../../../tensorrt_llm/models/llama/model.py). Smaug model resembles LLaMA very much except it uses bias term in its attention module, we therefore reuse the [LLaMA example code](../../core/llama) for Smaug, + +* [`convert_checkpoint.py`](./convert_checkpoint.py) to convert the LLaMA model into TensorRT LLM checkpoint format. + +In addition, there are two shared files in the parent folder [`examples`](../../../) for inference and evaluation: + +* [`../../../run.py`](../../../run.py) to run the inference on an input text; +* [`../../../summarize.py`](../../../summarize.py) to summarize the articles in the [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +## Support Matrix + +* FP16 + +## Usage + +This section gives a whole process where we convert HF models, build TensorRT LLM engines and ultimately perform summarization. + +### Build TensorRT engine(s) + +Run the following commands and TRT-LLM will first transforms a HF model into its own checkpoint format, then builds a TRT engine based on the checkpoint + +```bash +python ../../../llama/convert_checkpoint.py \ + --model_dir ./Smaug-72B-v0.1 \ + --output_dir ./tllm_checkpoint_8gpu_tp8 \ + --dtype float16 \ + --tp_size 8 + +trtllm-build --checkpoint_dir ./tllm_checkpoint_8gpu_tp8 \ + --output_dir ./Smaug_72B_tp8 \ + --gemm_plugin float16 \ + --gpt_attention_plugin float16 \ + --context_fmha=enable \ + --max_batch_size 64 \ + --remove_input_padding=enable +``` + +### Run Summarization + +After building TRT engine, we can use it to perform various tasks. TensorRT LLM provides handy code to run summarization on [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset and get [ROUGE](https://en.wikipedia.org/wiki/ROUGE_(metric)) scores. The `ROUGE-1` score can be used to validate model implementations. + +```bash +mpirun -n 8 -allow-run-as-root python ../../../summarize.py \ + --hf_model_dir ../Smaug-72B-v0.1 \ + --engine_dir ./Smaug_72B_tp8 \ + --data_type fp16 \ + --test_hf \ + --hf_device_map_auto \ + --test_trt_llm +``` diff --git a/examples/models/contrib/smaug/requirements.txt b/examples/models/contrib/smaug/requirements.txt new file mode 100644 index 000000000000..88232baef811 --- /dev/null +++ b/examples/models/contrib/smaug/requirements.txt @@ -0,0 +1,6 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +datasets==3.1.0 +evaluate +rouge_score +sentencepiece>=0.1.99 diff --git a/examples/models/core/deepseek_v3/README.md b/examples/models/core/deepseek_v3/README.md index 6f3af476866a..6c9774b83624 100644 --- a/examples/models/core/deepseek_v3/README.md +++ b/examples/models/core/deepseek_v3/README.md @@ -863,10 +863,10 @@ echo "All processes completed!" The converted checkpoint could be used as `` and consumed by other commands. ### KV Cache Reuse -KV cache reuse is supported for MLA on SM90, SM100, SM103, SM120 and SM121. It is enabled by default. Due to extra operations like memcpy and GEMMs, GPU memory consumption may be higher and the E2E performance may have regression in some cases. Users could pass `KvCacheConfig(enable_block_reuse=False)` to LLM API to disable it. +KV cache reuse is supported for MLA on SM90, SM100 and SM120. It is enabled by default. Due to extra operations like memcpy and GEMMs, GPU memory consumption may be higher and the E2E performance may have regression in some cases. Users could pass `KvCacheConfig(enable_block_reuse=False)` to LLM API to disable it. ### Chunked Prefill -Chunked Prefill is supported for MLA on SM90, SM100, SM103 and SM120. You should add `--enable_chunked_prefill` to enable it. The GPU memory consumption is highly correlated with `max_num_tokens` and `max_batch_size`. If encountering out-of-memory errors, you may make these values smaller. (`max_num_tokens` must be divisible by kv cache's `tokens_per_block`) +Chunked Prefill is supported for MLA only on SM90 and SM100 currently. You should add `--enable_chunked_prefill` to enable it. The GPU memory consumption is highly correlated with `max_num_tokens` and `max_batch_size`. If encountering out-of-memory errors, you may make these values smaller. (`max_num_tokens` must be divisible by kv cache's `tokens_per_block`) More specifically, we can imitate what we did in the [Quick Start](#quick-start): diff --git a/examples/models/core/granite/README.md b/examples/models/core/granite/README.md new file mode 100644 index 000000000000..b6cf34c50c5f --- /dev/null +++ b/examples/models/core/granite/README.md @@ -0,0 +1,89 @@ +# Granite + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document shows how to build and run a [Granite 3.0](https://huggingface.co/collections/ibm-granite/granite-30-language-models-66fdb59bbb54785c3512114f) model in TensorRT-LLM. + +The TensorRT LLM Granite implementation is based on the LLaMA model, with Mixture of Experts (MoE) enabled. The implementation can be found in [`llama/model.py`](../../../../tensorrt_llm/models/llama/model.py). See the LLaMA example [`examples/models/core/llama`](../llama) for details. + +- [Granite 3.0](#Granite) + - [Download model checkpoints](#download-model-checkpoints) + - [Convert weights from HF Transformers to TensorRT LLM format](#Convert-weights-from-HF-Transformers-to-TensorRT-LLM-format) + - [Build TensorRT engine](#build-tensorrt-engine) + - [Run Engine](#run-engine) + +## Download model checkpoints + +First, download the HuggingFace BF16 checkpoints of Granite 3.0 model. + +```bash +HF_MODEL="granite-3.0-8b-instruct" # or granite-3.0-3b-a800m-instruct +# clone the model we want to build +git clone https://huggingface.co/ibm-granite/${HF_MODEL} tmp/hf_checkpoints/${HF_MODEL} +``` + +## Convert weights from HF Transformers to TensorRT LLM format +Set environment variables and necessary directory: + +```bash +PREC_RAW="bfloat16" +TP=1 +mkdir -p tmp/trt_engines +``` + +### BF16 +Convert the weights using the `convert_checkpoint.py` script: + +```bash +ENGINE="${HF_MODEL}_${PREC_RAW}_tp${TP}" +export TRTLLM_DISABLE_UNIFIED_CONVERTER=1 # The current checkpoint conversion code requires legacy path +python3 ../llama/convert_checkpoint.py --model_dir tmp/hf_checkpoints/${HF_MODEL} \ + --output_dir tmp/tllm_checkpoints/${ENGINE} \ + --dtype ${PREC_RAW} \ + --tp_size ${TP} \ + --use_embedding_sharing + + +``` +### FP8 PTQ +Notes: +- Currently quantize.py does not support Expert Parallelism (EP) mode yet. User should use `../llama/convert_checkpoint.py` and specify `--moe_ep_size 1` instead, if needed. +- TensorRT LLM uses static quantization methods, which is expected to be faster at runtime as compared to dynamic quantization methods. This comes at a cost of an offline calibration step during quantization. `batch_size` and `calib_size` can be adjusted to shorten the calibration time. Please refer to `../../../quantization/README.md` for explanation. + +```bash +PREC_QUANT="fp8" +ENGINE="${HF_MODEL}_${PREC_QUANT}_tp${TP}" +python ../../../quantization/quantize.py --model_dir tmp/hf_checkpoints/${HF_MODEL} \ + --dtype ${PREC_RAW} \ + --qformat ${PREC_QUANT} \ + --kv_cache_dtype ${PREC_QUANT} \ + --output_dir tmp/tllm_checkpoints/${ENGINE} \ + --batch_size 1 \ + --calib_size 128 \ + --tp_size ${TP} + +``` + +## Build TensorRT engine +```bash +# Enable fp8 context fmha to get further acceleration by setting `--use_fp8_context_fmha enable` +# Use --workers to enable parallel build +trtllm-build --checkpoint_dir ./tmp/tllm_checkpoints/${ENGINE} \ + --output_dir ./tmp/trt_engines/${ENGINE} \ + --gpt_attention_plugin ${PREC_RAW} \ + --gemm_plugin ${PREC_RAW} \ + --workers ${TP} +``` + +## Run Engine +Test your engine with the [run.py](../../../run.py) script: + +```bash +mpirun -n ${TP} --allow-run-as-root python ../../../run.py --engine_dir ./tmp/trt_engines/${ENGINE} --tokenizer_dir tmp/hf_checkpoints/${HF_MODEL} --max_output_len 20 --input_text "The future of AI is" +``` + +For more usage examples see [`examples/models/core/llama/README.md`](../llama/README.md) diff --git a/examples/models/core/mixtral/README.md b/examples/models/core/mixtral/README.md new file mode 100644 index 000000000000..079753f88c92 --- /dev/null +++ b/examples/models/core/mixtral/README.md @@ -0,0 +1,219 @@ +# Mixtral + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document shows how to build and run a Mixtral model in TensorRT LLM on both single GPU, single node multi-GPU and +multi-node multi-GPU. Mixtral 8x22B is also supported and can be replace Mixtral 8x7B below as long as GPU memory is +sufficient. + +## Overview + +The TensorRT LLM Mixtral implementation is based on the LLaMA model, with Mixture of Experts enabled. The implementation can +be found in [tensorrt_llm/models/llama/model.py](../../../../tensorrt_llm/models/llama/model.py). +See the LLaMA example [`examples/models/core/llama`](../llama) for details. + +### Build TensorRT engine(s) + +#### Download Mixtral 8x7b weights +Get the weights by downloading from HF https://huggingface.co/mistralai/Mixtral-8x7B-v0.1. +See also https://huggingface.co/docs/transformers/main/en/model_doc/mixtral + +```bash +git lfs install +git clone https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 +``` + +#### Download Mixtral 8x22b weights +Get the weights by downloading from HF https://huggingface.co/mistralai/Mixtral-8x22B-v0.1. +See also https://huggingface.co/docs/transformers/main/en/model_doc/mixtral + +```bash +git lfs install +git clone https://huggingface.co/mistralai/Mixtral-8x22B-v0.1 +``` + +We use the LLaMA `convert_checkpoint.py` script to convert and build the model. TensorRT LLM LLaMA builds TensorRT engine(s) from HF checkpoint provided by `--model_dir`. +If no checkpoint directory is specified, TensorRT LLM will build engine(s) with dummy weights. + +`trtllm-build` uses one GPU by default, but if you have already more GPUs available at build time, +you may enable parallel builds to make the engine building process faster by adding the `--workers` argument. + +Here are some examples: + +```bash +# Build Mixtral8x7B with pipeline parallelism +python ../llama/convert_checkpoint.py --model_dir ./Mixtral-8x7B-v0.1 \ + --output_dir ./tllm_checkpoint_mixtral_2gpu \ + --dtype float16 \ + --pp_size 2 +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_2gpu \ + --output_dir ./trt_engines/mixtral/pp2 \ + --gemm_plugin float16 + +``` + +```bash +# Build Mixtral8x7B with tensor parallelism +python ../llama/convert_checkpoint.py --model_dir ./Mixtral-8x7B-v0.1 \ + --output_dir ./tllm_checkpoint_mixtral_2gpu \ + --dtype float16 \ + --tp_size 2 \ + --moe_tp_size 2 +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_2gpu \ + --output_dir ./trt_engines/mixtral/tp2 \ + --gemm_plugin float16 + + +# Build Mixtral8x22B with tensor parallelism and expert parallelism +python ../llama/convert_checkpoint.py --model_dir ./Mixtral-8x22B-v0.1 \ + --output_dir ./tllm_checkpoint_mixtral_8gpu \ + --dtype float16 \ + --tp_size 8 \ + --moe_tp_size 2 \ + --moe_ep_size 4 +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_8gpu \ + --output_dir ./trt_engines/mixtral/tp2ep4 \ + --gemm_plugin float16 +``` + +Then, you can test your engine with the [run.py](../../../run.py) script: + +```bash +mpirun -n 2 python3 ../../../run.py --engine_dir ./trt_engines/mixtral/tp2 --tokenizer_dir ./Mixtral-8x7B-v0.1 --max_output_len 8 --input_text "I love french quiche" +``` + +For more examples see [`examples/models/core/llama/README.md`](../llama/README.md) + +### Parallelism Modes + +Mixture of Experts supports 3 parallelism modes, these are Expert Parallelism (EP), Tensor Parallelism (TP), and the hybrid of the two (TP+EP). + +In TP mode (default) expert weight matrices are sliced evenly between all GPUs, so that all GPUs work together to calculate the result for each expert. + +In EP mode each GPU is assigned a subset of the expert weights matrices, so each GPU works independently to calculate the result for its assigned experts. This may cause load balancing issues where some GPUs have more work than others, thus increasing latency. + +In TP+EP mode, both strategies are used simultaneously. This means each GPU handles a portion of the expert weights matrices (as in EP mode) and these weights are further sliced across multiple GPUs (as in TP mode). This hybrid approach aims to balance the workload more evenly across GPUs, enhancing efficiency and reducing the likelihood of bottlenecks associated with EP mode alone. + +You can enable Expert Parallel or hybrid parallel by setting `--moe_tp_size` and `--moe_ep_size` when calling `convert_coneckpoint.py`. If only `--moe_tp_size` is provided, TRT-LLM will use Tensor Parallel for the MoE model; if only `--moe_ep_size` is provided, TRT-LLM will use Expert Parallel; if both are provided, the hybrid parallel will be used. + +Be sure that the product of `moe_tp_size` and `moe_ep_size` should equal to `tp_size`, since the total number of MoE parallelism across all GPUs must match the total number of parallelism in other parts of the model. + +```bash +# Build Mixtral8x7B with Expert Parallelism +python ../llama/convert_checkpoint.py --model_dir ./Mixtral-8x7B-v0.1 \ + --output_dir ./tllm_checkpoint_mixtral_2gpu \ + --dtype float16 \ + --tp_size 2 \ + --moe_ep_size 2 +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_2gpu \ + --output_dir ./trt_engines/mixtral/ep2 \ + --gemm_plugin float16 + +# Build Mixtral8x7B with Expert Parallelism and Tensor Parallelism +python ../llama/convert_checkpoint.py --model_dir ./Mixtral-8x7B-v0.1 \ + --output_dir ./tllm_checkpoint_mixtral_4gpu \ + --dtype float16 \ + --tp_size 4 \ + --moe_tp_size 2 \ + --moe_ep_size 2 +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_4gpu \ + --output_dir ./trt_engines/mixtral/tp2ep2 \ + --gemm_plugin float16 +``` + +### Normalization Modes + +MOE Supports different normalization modes which influence how the scales are calculated for the final weighted sum in +of the different top-k values. + +- 0 (NONE) corresponds to: `scales = topk(softmax(routing values))` +- 1 (RENORM) corresponds to: `scales = softmax(topk(routing values))` +- 2 (SPARSE_MIXER) corresponds to: `scales = sparsemixer(routing values)` + +Mixtral uses `RENORM` mode, this is set as the default. To use a different mode use the `--moe_normalization_mode` flag. +See [tensorrt_llm/layers/moe.py](../../../../tensorrt_llm/layers/moe.py#L56) for available values + + +## Quantization + +### Weight-only Quantization + +Mixtral supports weight only quantization + +```bash +# Build Mixtral8x7B with weight only +python ../llama/convert_checkpoint.py --model_dir ./Mixtral-8x7B-v0.1 \ + --output_dir ./tllm_checkpoint_mixtral_2gpu \ + --dtype float16 \ + --tp_size 2 \ + --use_weight_only \ + --weight_only_precision int8 +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_2gpu \ + --output_dir ./trt_engines/mixtral/tp2 \ + --gemm_plugin float16 +``` + +### FP8 Post-Training Quantization + +Mixtral supports FP8 quantization, using Modelopt. See [`examples/models/core/llama/README.md`](../llama/README.md#fp8-post-training-quantization) for full details on installing Modelopt + +```bash +# Quantize HF Mixtral into FP8 and export trtllm checkpoint +python ../../../quantization/quantize.py --model_dir ./Mixtral-8x7B-v0.1 \ + --dtype float16 \ + --qformat fp8 \ + --kv_cache_dtype fp8 \ + --output_dir ./tllm_checkpoint_mixtral_2gpu \ + --calib_size 512 \ + --tp_size 2 + +# Build trtllm engines from the trtllm checkpoint +# Enable fp8 context fmha to get further acceleration by setting `--use_fp8_context_fmha enable` +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_2gpu \ + --output_dir ./engine_outputs \ + --workers 2 +``` + +### AWQ Quantization + +Mixtral supports AWQ quantization using [AutoAWQ](https://github.com/casper-hansen/AutoAWQ). + +```bash +# Convert AutoAWQ HF checkpoints into TRT-LLM checkpoint +python ../llama/convert_checkpoint.py --model_dir ./tmp/mixtral-8x7b-v0.1-AWQ/ \ + --output_dir ./tllm_checkpoint_mixtral_awq_1gpu + +# Build trtllm engines from the trtllm checkpoint +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_awq_1gpu \ + --output_dir ./engine_outputs +``` + +You may found `quant_algo = W4A16_GPTQ` in the configuration file of the converted checkpoints, and that's because AutoAWQ is using exactly the same components as GPTQ. + +### NVFP4 Post-Training Quantization + +Mixtral supports NVFP4 quantization. + +```bash +# Quantize HF Mixtral into FP8 and export trtllm checkpoint +python ../../../quantization/quantize.py --model_dir ./Mixtral-8x7B-v0.1 \ + --dtype float16 \ + --qformat nvfp4 \ + --kv_cache_dtype fp8 \ + --output_dir ./tllm_checkpoint_mixtral_nvfp4_1gpu \ + --calib_size 512 \ + --tp_size 1 + +# Build trtllm engines from the trtllm checkpoint +# Enable fp8 context fmha to get further acceleration by setting `--use_fp8_context_fmha enable` +trtllm-build --checkpoint_dir ./tllm_checkpoint_mixtral_nvfp4_1gpu \ + --output_dir ./engine_outputs +``` + +## OOTB + +Mixtral supports OOTB operation without the plugin, however this comes at a significant performance cost. Users should prefer using the plugin path whenever possible diff --git a/examples/models/core/mixtral/requirements.txt b/examples/models/core/mixtral/requirements.txt new file mode 100644 index 000000000000..c1b98e5d3f7a --- /dev/null +++ b/examples/models/core/mixtral/requirements.txt @@ -0,0 +1,4 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +transformers==4.56.0 +accelerate==0.25.0 diff --git a/examples/models/core/multimodal/README.md b/examples/models/core/multimodal/README.md index 924561126cfb..159e26d4c7d8 100644 --- a/examples/models/core/multimodal/README.md +++ b/examples/models/core/multimodal/README.md @@ -1,17 +1,1169 @@ # Multi-Modal -The engine-build multimodal workflow that used to live here -(`build_multimodal_engine.py` / `run.py` / `eval.py` on top of -`trtllm-build`) was removed together with the legacy TensorRT backend. +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. -Multimodal models are supported on the PyTorch backend. See: +This document shows how to run multimodal pipelines with TensorRT-LLM, e.g. from image+text input modalities to text output. -- [Supported models](https://nvidia.github.io/TensorRT-LLM/models/supported-models.html) -- [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) and the - [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) -- Multimodal serving examples under `examples/llm-api/` and - `examples/serve/` +Multimodal models' LLM part has an additional parameter `--max_multimodal_len` compared to LLM-only build commands. Under the hood, `max_multimodal_len` and `max_prompt_embedding_table_size` are effectively the same concept, i.e., prepended/concatenated embeddings (either multimodal feature embeddings or prompt tuning embeddings) to the LLM input embeddings. The multimodal features from the visual encoder of shape `[batch_size, num_visual_features, visual_hidden_dim]` is flattened as `[batch_size * num_visual_features, visual_hidden_dim]` and passed like a prompt embedding table. + +We first describe three runtime modes for running multimodal models and how to run each model on a single GPU. We then provide general guidelines on using tensor parallelism for the LLM part of the pipeline. + +- [Runtime Mode](#runtime-modes) +- [BLIP2](#blip2) +- [CogVLM](#cogvlm) +- [Deplot](#deplot) +- [Fuyu](#fuyu) +- [Gemma3](#gemma3) +- [InternLM-XComposer2](#internlm-xcomposer2) +- [InternVL2](#internvl2) +- [Kosmos-2](#kosmos-2) +- [LLaVA, LLaVa-NeXT, LLaVA-OneVision and VILA](#llava-llava-next-llava-onevision-and-vila) +- [MLLaMA](#mllama) +- [NeVA](#neva) +- [Nougat](#nougat) +- [Phi-3-vision](#phi-3-vision) +- [Phi-4-multimodal](#phi-4-multimodal) +- [Qwen2-VL](#qwen2-vl) +- [Qwen-Image-Bench Evaluator](#qwen-image-bench-evaluator) +- [Video NeVA](#video-neva) +- [Dataset Evaluation](#dataset-evaluation) +- [Enabling Tensor Parallelism for multi-GPU](#enabling-tensor-parallelism-for-multi-gpu) +- [Enabling Embedding Table Offloading](#enabling-embedding-table-offloading) + +## Runtime Modes +TensorRT LLM supports three runtime modes for running multimodal models. +- `cpp_llm_only` (default): vision engine runs in python runtime, LLM in pybind C++ runtime +- `python`: everything runs in python runtime +- `cpp`: everything runs in C++ runtime + +This can be specified by the `--session RUNTIME_MODE` argument in `run.py` (see instructions of each model below). +Not all models supports end-to-end `cpp` mode, the checked ones below are supported. See footnotes for reasons models are unsupported +- [ ] BLIP-2-T5 [^1] +- [x] BLIP-2-OPT +- [x] CogVLM +- [ ] Deplot [^1] +- [ ] Pix2Struct [^1] +- [x] Fuyu +- [x] InternVL2-2b +- [x] Kosmos-2 +- [x] LLaVA +- [ ] LLaVA-NeXT / OneVision [^2] +- [x] VILA [^3] +- [ ] Mllama [^1] +- [x] NeVA +- [ ] Nougat [^1] +- [ ] Phi-3-Vision [^2] +- [ ] Phi-4-multimodal +- [ ] Qwen2-VL [^4] +- [x] Video-NeVA + + +[^1]: Model uses cross attention to feed visiual features to LLM decoder, which is not supported +[^2]: Model requires post processing its encoder output features, which is not supported +[^3]: Currently C++ runtime only supports single image per request (VILA mode 2) +[^4]: Vision encoder requires additional inputs not supported by the C++ runtime + +## BLIP2 + +This BLIP section covers both BLIP2-OPT and BLIP2-T5, with minor changes needed when switching the LLM backbone. + +1. Download Huggingface weights and convert original checkpoint to TRT-LLM checkpoint format + following example in `examples/models/contrib/opt/README.md` and `examples/models/core/enc_dec/README.md`. + + ```bash + export MODEL_NAME="blip2-opt-2.7b" # options: blip2-opt-6.7b, blip2-flan-t5-xl, blip2-flan-t5-xxl + git clone https://huggingface.co/Salesforce/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` + + For BLIP2-OPT family, + ```bash + python ../../contrib/opt/convert_checkpoint.py --model_type blip2 \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --dtype float16 + ``` + + For BLIP2-T5 family, + ```bash + python ../enc_dec/convert_checkpoint.py --model_type blip2 \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/bfloat16 \ + --tp_size 1 \ + --pp_size 1 \ + --dtype bfloat16 + ``` + +2. Build TRT-LLM engine from TRT-LLM checkpoint + + For BLIP2-OPT family, + ```bash + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gemm_plugin float16 \ + --max_beam_width 1 \ + --max_batch_size 8 \ + --max_seq_len 1024 \ + --max_input_len 924 \ + --max_multimodal_len 256 # 8 (max_batch_size) * 32 (num_visual_features) + ``` + + For BLIP2-T5 family, + ```bash + trtllm-build --checkpoint_dir tmp/trt_models/${MODEL_NAME}/bfloat16/encoder \ + --output_dir tmp/trt_engines/${MODEL_NAME}/bfloat16/llm/encoder \ + --paged_kv_cache disable \ + --moe_plugin disable \ + --gemm_plugin bfloat16 \ + --bert_attention_plugin bfloat16 \ + --gpt_attention_plugin bfloat16 \ + --remove_input_padding enable \ + --context_fmha disable \ + --max_beam_width 1 \ + --max_batch_size 8 \ + --max_input_len 924 \ + --max_multimodal_len 256 # 8 (max_batch_size) * 32 (num_visual_features) + + trtllm-build --checkpoint_dir tmp/trt_models/${MODEL_NAME}/bfloat16/decoder \ + --output_dir tmp/trt_engines/${MODEL_NAME}/bfloat16/llm/decoder \ + --paged_kv_cache disable \ + --moe_plugin disable \ + --gemm_plugin bfloat16 \ + --bert_attention_plugin bfloat16 \ + --gpt_attention_plugin bfloat16 \ + --remove_input_padding enable \ + --context_fmha disable \ + --max_beam_width 1 \ + --max_batch_size 8 \ + --max_seq_len 1024 \ + --max_encoder_input_len 924 \ + --max_input_len 1 # Same command for decoder but don't set --max_multimodal_len + ``` + + **NOTE**: `max_multimodal_len = max_batch_size * num_visual_features`, so if you change max_batch_size, max multimodal length **MUST** be changed accordingly. + +3. Build TensorRT engines for vision encoders + + ```bash + python build_multimodal_engine.py --model_type blip2 --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/bfloat16/vision --max_batch_size 8 + ``` + + The built engines are located in `tmp/trt_engines/${MODEL_NAME}/bfloat16/vision` for BLIP2-T5, similarly for BLIP-OPT. + + To run the BLIP2 pipeline with batch size > 1, change `--max_batch_size` argument to `build_multimodal_engine.py` accordingly. + +4. Assemble everything into BLIP2 pipeline + + For BLIP2-OPT family, + ```bash + python run.py \ + --max_new_tokens 30 \ + --input_text "Question: which city is this? Answer:" \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu + ``` + + For BLIP2-T5 family, + ```bash + python run.py \ + --max_new_tokens 30 \ + --input_text "Question: which city is this? Answer:" \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/bfloat16 + ``` + +5. (Optional) INT8/INT4 weight-only quantization for OPT can be enabled using commands as follows (take `INT4` as an example, while `INT8` is the default precision for weight-only quantization): + ```bash + python ../../contrib/opt/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --dtype float16 \ + --output_dir tmp/trt_models/${MODEL_NAME}/int4_weightonly/1-gpu \ + --use_weight_only \ + --weight_only_precision int4 + + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/int4_weightonly/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/int4_weightonly/1-gpu/llm \ + --gemm_plugin float16 \ + --max_beam_width 1 \ + --max_batch_size 8 \ + --max_multimodal_len 256 \ + --max_input_len 924 \ + --max_seq_len 1024 + ``` + + The built OPT engines lie in `tmp/trt_engines/${MODEL_NAME}/int4_weightonly/1-gpu/llm`. + You should use this directory without the `llm` part as `--engine_dir` argument to `run.py` + + **NOTE:** INT8/INT4 option is not supported for BLIP2-T5, because quantization support has not been + added for encoder-decoder models yet. + +## CogVLM + +Currently, CogVLM only support bfloat16 precision. + +1. Download Huggingface weights + + ```bash + export MODEL_NAME="cogvlm-chat-hf" + git clone https://huggingface.co/THUDM/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + export TOKENIZER_NAME="vicuna-7b-v1.5" + git clone https://huggingface.co/lmsys/${TOKENIZER_NAME} tmp/hf_models/${TOKENIZER_NAME} + ``` + + Because currently onnx doesn't support `xops.memory_efficient_attention`, we need to modify some source code of the huggingface CogVLM. + ``` + cd tmp/hf_models/${MODEL_NAME} + sed -i '4s/.*//;40s/.*/ out = self.attention(q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2)).transpose(1, 2).contiguous()/;41s/.*//;42s/.*//' visual.py # It will replace memory_efficient_attention with some basic ops + ``` + +2. Convert Huggingface weights into TRT-LLM checkpoints and build TRT engines using scripts in `examples/cogvlm` + + CogVLM uses a Vit encoder as LLM encoder and a modified Llama as decoder. + + ```bash + python ../../contrib/cogvlm/convert_checkpoint.py --model_dir tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_models/${MODEL_NAME} --dtype bfloat16 --use_prompt_tuning + + trtllm-build --checkpoint_dir tmp/trt_models/${MODEL_NAME} \ + --output_dir tmp/trt_engines/${MODEL_NAME}/bf16/1-gpu/llm \ + --gemm_plugin bfloat16 \ + --gpt_attention_plugin bfloat16 \ + --remove_input_padding enable \ + --max_batch_size 48 \ + --max_input_len 2048 \ + --max_seq_len 3076 \ + --paged_kv_cache enable \ + --bert_attention_plugin disable \ + --moe_plugin disable \ + --max_multimodal_len 61440 # 48 (max_batch_size) * 1280 (max_num_visual_features) + ``` + +3. Generate TensorRT engines for visual components and combine everything into final pipeline. + + ```bash + python build_multimodal_engine.py --model_type cogvlm --model_path tmp/hf_models/${MODEL_NAME} --max_batch_size 48 --output_dir tmp/trt_engines/${MODEL_NAME}/bf16/1-gpu/vision + + python run.py \ + --max_new_tokens 1000 \ + --input_text " [INST] please describe this image in detail [/INST] " \ + --hf_model_dir tmp/hf_models/${TOKENIZER_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/bf16/1-gpu \ + --batch_size 1 \ + --top_p 0.4 \ + --top_k 1 \ + --temperature 0.2 \ + --repetition_penalty 1.2 \ + --enable_context_fmha_fp32_acc + + CogVLM uses model_runner_cpp for its LLM decoder by default. To switch to model_runner, set `--session python` in the command mentioned above. + ``` + +## Deplot + +1. Download Huggingface weights and convert original checkpoint to TRT-LLM checkpoint format + following example in `examples/models/core/enc_dec/README.md`. + + ```bash + export MODEL_NAME="deplot" + git clone https://huggingface.co/google/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + + python ../enc_dec/convert_checkpoint.py --model_type pix2struct \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/float16 \ + --tp_size 1 \ + --pp_size 1 \ + --dtype float16 + ``` + +2. Build TRT-LLM engine from TRT-LLM checkpoint + + ```bash + trtllm-build --checkpoint_dir tmp/trt_models/${MODEL_NAME}/float16/decoder \ + --output_dir tmp/trt_engines/${MODEL_NAME}/1-gpu/float16/llm/decoder \ + --paged_kv_cache disable \ + --moe_plugin disable \ + --gemm_plugin float16 \ + --bert_attention_plugin float16 \ + --gpt_attention_plugin float16 \ + --remove_input_padding enable \ + --context_fmha disable \ + --max_beam_width 1 \ + --max_batch_size 8 \ + --max_seq_len 2558 \ + --max_encoder_input_len 2048 \ + --max_input_len 1 + ``` + + The built deplot engines are located in `tmp/trt_engines/${MODEL_NAME}/1-gpu/float16`. + +3. Build TensorRT engines for visual components + + ```bash + python build_multimodal_engine.py --model_type pix2struct --model_path tmp/hf_models/${MODEL_NAME} --max_batch_size 8 --output_dir tmp/trt_engines/${MODEL_NAME}/1-gpu/float16/vision + ``` + + The built visual engines are located in `tmp/trt_engines/${MODEL_NAME}/1-gpu/float16/vision`. + + To run the deplot pipeline with batch size > 1, change `--max_batch_size` argument to `build_multimodal_engine.py` accordingly. + +4. Assemble everything into deplot pipeline + + ```bash + python run.py \ + --max_new_tokens 100 \ + --input_text "" \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/1-gpu/float16 + ``` + +## Fuyu + +1. Download Huggingface weights + + ```bash + export MODEL_NAME="fuyu-8b" + git clone https://huggingface.co/adept/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` + +2. Convert Huggingface weights into TRT-LLM checkpoints and build TRT engines using scripts in `examples/models/core/gpt`. + The LLM portion of Fuyu uses a Persimmon model + ```bash + python ../gpt/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --dtype float16 \ + --gpt_variant persimmon + + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gemm_plugin float16 \ + --use_fused_mlp=enable \ + --max_batch_size 1 \ + --max_input_len 2048 \ + --max_seq_len 2560 \ + --max_multimodal_len 2048 + ``` + +3. Generate TensorRT engines for visual components and combine everything into final pipeline. + + ```bash + python build_multimodal_engine.py --model_type fuyu --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision + + python run.py \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu + ``` + +## Gemma3 + +**NOTE: We only support Gemma3 VLMs in Pytorch workflow.** + +Gemma3VL decoder requires a custom attention mask while processing images. During the context phase: +- Text tokens attend to other tokens in a causal fashion (standard autoregressive behavior) +- Image tokens attend to other tokens in a causal fashion AND attend to other tokens from the same image in a bidirectional manner + +**Reference:** [Gemma3 Model Documentation](https://huggingface.co/docs/transformers/en/model_doc/gemma3) + +We support this custom mask with FlashInfer attention backend. + +### Requirements + +To ensure expected behavior with Gemma3VL, the following configurations are **required**: +- **Attention Backend**: Use the FlashInfer attention backend +- **Chunked Prefill**: Must be disabled +- **KV Cache Reuse**: Must be disabled + +### Quick Start + +#### 1. Download Model Weights + +```bash +export MODEL_NAME="gemma-3-27b-it" +git clone https://huggingface.co/google/${MODEL_NAME} +``` + +#### 2. Interactive Testing + +Use the `quickstart_multimodal.py` script for quick testing: + +```bash +python3 examples/llm-api/quickstart_multimodal.py \ + --model_dir ${MODEL_NAME}/ \ + --modality image \ + --image_format pil \ + --attention_backend FLASHINFER \ + --disable_kv_cache_reuse +``` + +#### 3. Model Serving + +Serve the model using `trtllm-serve` with the required llmapi arguments mentioned in a yaml file: + +```bash +# Create the configuration file +cat > extra-llm-api-options.yaml << 'EOF' +cuda_graph_config: null +attn_backend: "FLASHINFER" +enable_chunked_prefill: false +kv_cache_config: + enable_block_reuse: false +EOF + +# Serve the model +trtllm-serve ${MODEL_NAME}/ \ + --backend pytorch \ + --tp_size 1 \ + --port 8000 \ + --max_batch_size 4 \ + --config extra-llm-api-options.yaml +``` + +### Supported Model Variants + +Currently supported Gemma3 variants: 4B, 12B, 27B + + +## InternLM-XComposer2 + +**NOTE: We only support InternLM-XComposer-VL-7b for now** + +Firstly, please install transformers with 4.45.2 +```bash + pip install -r requirements-internlm-xcomposer2.txt +``` + +1. Convert Huggingface weights to TRT-LLM checkpoint format using `examples/models/contrib/internlm/README.md`. + +2. Use `trtllm-build` command to build TRT-LLM engine for OPT. + +3. The full list of commands is as follows: + + ```bash + export MODEL_NAME=internlm-xcomposer2-vl-7b + git lfs clone https://huggingface.co/internlm/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + + python ../internlm2/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --dtype float16 \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu + + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gemm_plugin float16 \ + --lora_plugin float16 \ + --lora_dir . \ + --max_lora_rank 256 \ + --max_input_len 1536 \ + --max_batch_size 48 \ + --max_multimodal_len 58800 # 58800 = 1225(visual token/img) * 48 (max_batch_size), as each image corresponds to 1225 visual tokens in the ViT here + + python build_multimodal_engine.py \ + --model_type internlm-xcomposer2 \ + --model_path tmp/hf_models/${MODEL_NAME} \ + --output_dir trt_engines/${MODEL_NAME}/fp16/1-gpu/vision \ + --max_batch_size 48 + + python run.py \ + --max_new_tokens 200 \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir trt_engines/${MODEL_NAME}/fp16/1-gpu \ + --batch_size 1 + ``` + +## InternVL2 + +[InternVL Family](https://github.com/OpenGVLab/InternVL): Closing the Gap to Commercial Multimodal Models with Open-Source Suites —— A Pioneering Open-Source Alternative to GPT-4o. Here we show how to deploy InternVL2‑1B/InternVL2‑2B/InternVL2‑4B/InternVL2‑8B/InternVL2‑26B in TensorRT-LLM. + +Firstly, please install transformers with 4.37.2 +```bash + pip install transformers==4.37.2 +``` + +1. Download Huggingface weights + - For InternVL2-1B + ```bash + export MODEL_NAME="InternVL2-1B" + git clone https://huggingface.co/OpenGVLab/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + export LLM_MODEL_NAME="qwen" + ``` + + - For InternVL2-2B/InternVL2‑8B/InternVL2‑26B + ```bash + export MODEL_NAME="InternVL2-2B" # or InternVL2‑8B, InternVL2‑26B + git clone https://huggingface.co/OpenGVLab/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + export LLM_MODEL_NAME="internlm2" + ``` + + - For InternVL2-4B + ```bash + export MODEL_NAME="InternVL2-4B" + git clone https://huggingface.co/OpenGVLab/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + export LLM_MODEL_NAME="phi" + ``` + +2. Convert Huggingface weights into TRT-LLM checkpoints + ```bash + python ../${LLM_MODEL_NAME}/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --dtype float16 + ``` + +3. Build TRT engines + ```bash + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gemm_plugin auto \ + --max_batch_size 1 \ + --max_input_len 4096 \ + --max_seq_len 4608 \ + --max_multimodal_len 3328 + ``` + +4. Generate TensorRT engines for visual components and combine everything into final pipeline. + ```bash + python build_multimodal_engine.py --model_type internvl --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision + python run.py \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/ \ + --image_path tmp/hf_models/${MODEL_NAME}/examples/image1.jpg + ``` + +5. (Optional) FP8 and INT8 SmoothQuant quantization is supported for the InternVL2-4B variant (LLM model only). + + ```bash + # FP8 quantization + python ../../../quantization/quantize.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp8/1-gpu \ + --dtype bfloat16 \ + --qformat fp8 \ + --kv_cache_dtype fp8 + + # INT8 SmoothQuant quantization + python ../../../quantization/quantize.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/int8/1-gpu \ + --dtype bfloat16 \ + --qformat int8_sq + ``` + + Then follow the same `trtllm-build`, `build_multimodal_engine.py` and `run.py` steps as before. + + +## Kosmos-2 + +1. Download Huggingface weights + + ```bash + export MODEL_NAME="kosmos-2" + git clone https://huggingface.co/microsoft/kosmos-2-patch14-224 tmp/hf_models/${MODEL_NAME} + ``` + +2. Convert Huggingface weights into TRT-LLM checkpoints and build TRT engines using scripts in `examples/models/core/gpt`. + ```bash + python ../gpt/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --dtype float16 \ + --gpt_variant ${MODEL_NAME} + + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gpt_attention_plugin float16 \ + --gemm_plugin float16 \ + --max_batch_size 1 \ + --max_input_len 512 \ + --max_seq_len 1024 \ + --max_multimodal_len 64 # 1 (max_batch_size) * 64 (num_visual_features) + ``` + +3. Generate TensorRT engines for visual components and combine everything into final pipeline. + + ```bash + python build_multimodal_engine.py --model_type kosmos-2 --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision + + python run.py \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu + ``` + +## LLaVA, LLaVa-NeXT, LLaVA-OneVision and VILA + +[LLaVA](https://github.com/haotian-liu/LLaVA) and [VILA](https://github.com/Efficient-Large-Model/VILA) are both visual language models (VLM) that can be deployed in TensorRT LLM with many quantization options. [LLaVA-NeXT](https://huggingface.co/collections/llava-hf/llava-next-65f75c4afac77fd37dbbe6cf) is an extension of LLaVA. TRT-LLM currently supports [Mistral-7b](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf) and [ Nous-Hermes-2-Yi-34B](https://huggingface.co/llava-hf/llava-v1.6-34b-hf) variant of LLaVA-NeXT. [LLaVA-OneVision](https://huggingface.co/collections/llava-hf/llava-onevision-66bb1e9ce8856e210a7ed1fe) is another extension of LLaVA. + +1. Download Huggingface model weights. These models have both visual and LLM components + unlike BLIP2 example which downloads only LLM components from Huggingface. + + For LLaVA, + + ```bash + export MODEL_NAME="llava-1.5-7b-hf" # also llava-1.5-13b-hf + git clone https://huggingface.co/llava-hf/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` + For LLaVA-NeXT, + + ```bash + export MODEL_NAME="llava-v1.6-mistral-7b-hf" #for 34b variant "llava-v1.6-34b-hf" + git clone https://huggingface.co/llava-hf/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` + + For LLaVA-OneVision, + + ```bash + export MODEL_NAME="llava-onevision-qwen2-7b-ov-hf" # also llava-onevision-qwen2-0.5b-ov-hf, llava-onevision-qwen2-72b-ov-hf, etc + git clone https://huggingface.co/llava-hf/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` + + For VILA, we need a few more steps until it is added to HF model zoo + + ```bash + # install the following dependency + pip install -r requirements-vila.txt + + # clone original VILA repo + export VILA_PATH="tmp/hf_models/VILA" + git clone https://github.com/Efficient-Large-Model/VILA.git ${VILA_PATH} + + # download VILA checkpoints + export MODEL_NAME="vila1.5-3b" # NOTE: name must contain vila or VILA! it's used to identify whether we need to register the non-HF VILA codebase in HF Auto class + git clone https://huggingface.co/Efficient-Large-Model/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` + +2. Generate TRT-LLM engine for LLaMA following example in `examples/models/core/llama/README.md` and `examples/models/core/qwen/README.md` + + ```bash + python ../llama/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --dtype float16 + + # for LLaVA + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gemm_plugin float16 \ + --use_fused_mlp=enable \ + --max_batch_size 1 \ + --max_input_len 2048 \ + --max_seq_len 2560 \ + --max_multimodal_len 576 # 1 (max_batch_size) * 576 (num_visual_features) + + # for LLaVA-NeXT + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gpt_attention_plugin float16 \ + --gemm_plugin float16 \ + --use_fused_mlp=enable \ + --max_batch_size 1 \ + --max_input_len 4096 \ + --max_seq_len 5120 \ + --max_num_tokens 4096 \ + --max_multimodal_len 4096 # 1 (max_batch_size) * 4096 (max_input_len) + + # for VILA + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gemm_plugin float16 \ + --use_fused_mlp=enable \ + --max_batch_size 1 \ + --max_input_len 2048 \ + --max_seq_len 2560 \ + --max_multimodal_len 196 # 1 (max_batch_size) * 196 (num_visual_features) + + # for LLaVA-OneVision + python ../qwen/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --dtype float16 + + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gemm_plugin float16 \ + --use_fused_mlp=enable \ + --max_batch_size 1 \ + --max_input_len 7228 \ + --max_seq_len 7328 \ + --max_multimodal_len 7128 # max_batch_size * num_visual_features(depends on the image size or the specified video num frame) + ``` + +3. Build TensorRT engines for visual components + + ```bash + python build_multimodal_engine.py --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision --model_type llava # for LLaVA + + python build_multimodal_engine.py --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision --model_type llava_next --max_batch_size 5 # 1 (max_batch_size) * 5 (because LLAVA-NeXT visual encoder can have at most 5 patches) # for LLaVA-NeXT + + python build_multimodal_engine.py --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision --model_type llava_onevision --max_batch_size 32 # max_batch_size * patch for image or frame for video # for LLaVA-OneVision + + python build_multimodal_engine.py --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision --model_type vila --vila_path ${VILA_PATH} # for VILA + ``` + + ```bash + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu \ + --input_text "\n Which city is this?" # for LLaVA and for LLaVA-NeXT + + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu \ + --image_path=https://github.com/Efficient-Large-Model/VILA/raw/main/demo_images/av.png,https://storage.googleapis.com/sfr-vision-language-research/LAVIS/assets/merlion.png \ + --input_text "\n Please elaborate what you see in the images?","\n Which city is this?" \ + --batch_size=2 # for LLaVA + ``` + + Note that Llava can support N pairs inference batching, `--batch_size=N` should be used. There should be N images listed under `--image_path` and N text prompts listed under `--input_text`. Don't forget to set the `--max_batch_size` and `--max_multimodal_len` during engine building. + + For LLaVA-OneVision, you can use either image or video as inputs. + ```bash + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu \ + --input_text "What is shown in this image?" \ + --image_path image.png + + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu \ + --input_text "Why is this video funny?" \ + --video_path video.mp4 + --video_num_frames 8 # sample uniformly 8 frames from the video, up to 32 frames + ``` + + For VILA, you can use either local file or web url as input images. + Suppose you have a local image `av.png` downloaded from `https://github.com/Efficient-Large-Model/VILA/blob/main/demo_trt_llm/av.png` and the url of `merlion.png` + ```bash + wget -O av.png https://raw.githubusercontent.com/Efficient-Large-Model/VILA/main/demo_images/av.png + + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu \ + --image_path=av.png,https://storage.googleapis.com/sfr-vision-language-research/LAVIS/assets/merlion.png \ + --input_text="\n\n Please elaborate what you see in the images?" \ + --batch_size=1 # for VILA mode 1 + + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu \ + --image_path=av.png,https://storage.googleapis.com/sfr-vision-language-research/LAVIS/assets/merlion.png \ + --input_text="\n Please elaborate what you see in the images?","\n Which city is this?" \ + --batch_size=2 \ + --check_accuracy # for VILA mode 2 + ``` + + Note that VILA can support different modes in terms of batching: + - Mode 1: if you want to query N images as a whole using a prompt, `--batch_size=1` should be used (which is the default value). Example is given above. + - Mode 2: if you want to query N pairs, `--batch_size=N` should be used. There should be N images listed under `--image_path` and N text prompts listed under `--input_text`. Don't forget to set the `--max_batch_size` and `--max_multimodal_len` during engine building. + + Note: use `--run_profiling` for performance measurement, use `--check_accuracy` for accuracy check. + +4. (Optional) Different quantization methods supported in LLaMA and Qwen can be applied to LLaVA/VILA/LLaVA-OneVision as well, such as INT4/INT8 weight-only, SmoothQuant, and INT4 Activation-Aware Quantization (AWQ). Detailed instructions can be found in LLaMA [README](../llama/README.md) and Qwen [README](../qwen/README.md). + + For example, + + ```bash + # INT4 weight only + python ../llama/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --dtype float16 \ + --output_dir tmp/trt_models/${MODEL_NAME}/int4_weightonly/1-gpu \ + --use_weight_only \ + --weight_only_precision int4 + + # INT4 AWQ + python ../../../quantization/quantize.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/int4_awq/1-gpu \ + --dtype float16 \ + --qformat int4_awq \ + --calib_size 32 + ``` + + Then follow the same `trtllm-build` and `run.py` steps as before. NOTE: for `trtllm-build` command, do not use `--use_fused_mlp=enable` in these quantization modes. + +## MLLaMA + +This section shows how to build and run a LLaMA-3.2 Vision model in TensorRT-LLM. We use [Llama-3.2-11B-Vision/](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision) as an example. + +For LLaMA-3.2 text model, please refer to the [examples/models/core/llama/README.md](../llama/README.md) because it shares the model architecture of llama. + +### Support data types + * BF16 + * Tensor Parallel + * INT8 & INT4 Weight-Only + * FP8 + +### Build and run vision model + +* build engine of vision encoder model + +```bash +python examples/models/core/multimodal/build_multimodal_engine.py --model_type mllama \ + --model_path Llama-3.2-11B-Vision/ \ + --output_dir /tmp/mllama/trt_engines/vision/ +``` + +* build engine of decoder model + +```bash +python examples/models/core/mllama/convert_checkpoint.py --model_dir Llama-3.2-11B-Vision/ \ + --output_dir /tmp/mllama/trt_ckpts \ + --dtype bfloat16 + +trtllm-build --checkpoint_dir /tmp/mllama/trt_ckpts \ + --output_dir /tmp/mllama/trt_engines/llm/ \ + --max_num_tokens 4096 \ + --max_seq_len 2048 \ + --workers 1 \ + --gemm_plugin auto \ + --max_batch_size 4 \ + --max_encoder_input_len 4100 \ + --input_timing_cache model.cache +``` + +Note that for instruct Vision model, please set the `max_encoder_input_len` as `6404`. + +* Run test on multimodal/run.py with C++ runtime (LLM part only) + +```bash +python3 examples/models/core/multimodal/run.py --engine_dir /tmp/mllama/trt_engines/ \ + --hf_model_dir Llama-3.2-11B-Vision/ \ + --image_path https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg \ + --input_text "<|image|><|begin_of_text|>If I had to write a haiku for this one" \ + --max_new_tokens 50 \ + --batch_size 2 + +Use model_runner_cpp by default. To switch to model_runner, set `--session python` in the command mentioned above. + +python3 examples/models/core/multimodal/eval.py \ + --engine_dir /tmp/mllama/trt_engines/ \ + --hf_model_dir Llama-3.2-11B-Vision/ \ + --test_trtllm \ + --accuracy_threshold 65 \ + --eval_task lmms-lab/ai2d +``` + +### Run MLLaMA decoder part by FP8 + +```bash +# install modelopt 0.21.0 +pip install nvidia-modelopt[torch]~=0.21.0 + +python ./examples/quantization/quantize.py --model_dir Llama-3.2-11B-Vision/ \ + --dtype bfloat16 \ + --qformat fp8 \ + --output_dir /tmp/llama-3.2-11B-Vision/fp8/ \ + --kv_cache_dtype fp8 \ + --calib_size 512 \ + --calib_dataset scienceqa + +trtllm-build --checkpoint_dir /tmp/llama-3.2-11B-Vision/fp8/ \ + --output_dir /tmp/trt_engines/llama-3.2-11B-Vision/fp8/llm \ + --max_num_tokens 4096 \ + --max_seq_len 2048 \ + --workers 1 \ + --gemm_plugin auto \ + --max_batch_size 4 \ + --max_encoder_input_len 4100 \ + --input_timing_cache model.cache \ + --use_paged_context_fmha enable \ + --use_fp8_context_fmha enable + +# copy visiual engine directory `/tmp/mllama/trt_engines/vision/` to fp8 engine directory `/tmp/trt_engines/llama-3.2-11B-Vision/fp8/vision` + +python3 examples/models/core/multimodal/run.py --engine_dir /tmp/trt_engines/llama-3.2-11B-Vision/fp8/ \ + --hf_model_dir Llama-3.2-11B-Vision/ \ + --image_path https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg \ + --input_text "<|image|><|begin_of_text|>If I had to write a haiku for this one" \ + --max_new_tokens 50 \ + --batch_size 2 + +python3 examples/models/core/multimodal/eval.py --engine_dir /tmp/trt_engines/llama-3.2-11B-Vision/fp8/ \ + --hf_model_dir Llama-3.2-11B-Vision/ \ + --test_trtllm \ + --accuracy_threshold 65 \ + --eval_task lmms-lab/ai2d +``` + +Note that for instruct Vision model, please set the `max_encoder_input_len` as `6404`. + +## NeVA + +[NeVA](https://docs.nvidia.com/nemo-framework/user-guide/24.12/nemotoolkit/multimodal/mllm/neva.html) is a groundbreaking addition to the NeMo Multimodal ecosystem. This model seamlessly integrates large language-centric models with a vision encoder, that can be deployed in TensorRT-LLM. + +1. Generate TRT-LLM engine for NVGPT following example in `examples/models/core/gpt/README.md`. To adhere to the NVGPT conventions of the conversion script, some layer keys have to be remapped using `--nemo_rename_key`. + + ```bash + export MODEL_NAME="neva" + python ../gpt/convert_checkpoint.py \ + --nemo_ckpt_path ./${MODEL_NAME}.nemo \ + --dtype bfloat16 \ + --output_dir tmp/trt_models/${MODEL_NAME} \ + --nemo_rename_key model:model.language_model \ + attention.linear_qkv.layer_norm_bias:input_layernorm.bias \ + attention.linear_qkv.layer_norm_weight:input_layernorm.weight \ + mlp.linear_fc1.layer_norm_bias:post_attention_layernorm.bias \ + mlp.linear_fc1.layer_norm_weight:post_attention_layernorm.weight \ + linear_qkv:query_key_value \ + linear_fc1:dense_h_to_4h \ + linear_fc2:dense_4h_to_h \ + linear_proj:dense \ + decoder:encoder + + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME} \ + --output_dir tmp/trt_engines/${MODEL_NAME}/bf16/1-gpu/llm \ + --gpt_attention_plugin bfloat16 \ + --gemm_plugin bfloat16 \ + --max_batch_size 1 \ + --max_input_len 2048 \ + --max_seq_len 2560 \ + --max_multimodal_len 729 # 1 (max_batch_size) * 729 (num_visual_features) + ``` + +2. Build TensorRT engines for visual components + + ```bash + python build_multimodal_engine.py --model_path ./${MODEL_NAME}.nemo --model_type neva --output_dir tmp/trt_engines/${MODEL_NAME}/bf16/1-gpu/vision + ``` + + ```bash + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir tmp/trt_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/bf16/1-gpu \ + --input_text "Question: which city is this? Answer:" + ``` + + Note: use `--run_profiling` for performance measurement, use `--check_accuracy` for accuracy check. + +## Nougat + +1. Download Huggingface weights + + ```bash + export MODEL_NAME="nougat-base" # also nougat-small + git clone https://huggingface.co/facebook/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` + +2. Convert Huggingface weights into TRT-LLM checkpoints and build TRT engines using scripts in `examples/models/core/enc_dec` + + Nougat uses mBART architecture but replaces the LLM encoder with a Swin Transformer encoder. + To achieve this, we add an extra `--nougat` flag (over mBART example) to + `convert_checkpoint.py` in `examples/models/core/enc_dec` and `trtllm-build`. + + ```bash + python ../enc_dec/convert_checkpoint.py --model_type bart \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/bfloat16 \ + --tp_size 1 \ + --pp_size 1 \ + --dtype bfloat16 \ + --nougat + + trtllm-build --checkpoint_dir tmp/trt_models/${MODEL_NAME}/bfloat16/decoder \ + --output_dir tmp/trt_engines/${MODEL_NAME}/1-gpu/bfloat16/llm/decoder \ + --paged_kv_cache disable \ + --moe_plugin disable \ + --gemm_plugin bfloat16 \ + --bert_attention_plugin bfloat16 \ + --gpt_attention_plugin bfloat16 \ + --remove_input_padding enable \ + --max_beam_width 1 \ + --max_batch_size 1 \ + --max_seq_len 101 \ + --max_input_len 1 \ + --max_encoder_input_len 588 # 1 (max_batch_size) * 588 (num_visual_features) + ``` + +3. Generate TensorRT engines for visual components and combine everything into final pipeline. + + ```bash + python build_multimodal_engine.py --model_type nougat --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/1-gpu/bfloat16/vision + + python run.py \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/1-gpu/bfloat16 + ``` + + Note: Nougat models usually do not need a text prompt. + + +## Phi-3-vision + +1. Download Huggingface weights + + ```bash + export MODEL_NAME="Phi-3-vision-128k-instruct" # or Phi-3.5-vision-instruct + git clone https://huggingface.co/microsoft/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` + +2. Convert Huggingface weights into TRT-LLM checkpoints and build TRT engines using scripts in `examples/models/core/phi`. + ```bash + python ../phi/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --dtype float16 + + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gpt_attention_plugin float16 \ + --gemm_plugin float16 \ + --max_batch_size 1 \ + --max_input_len 4096 \ + --max_seq_len 4608 \ + --max_multimodal_len 4096 + ``` + +3. Generate TensorRT engines for visual components and combine everything into final pipeline. + + ```bash + python build_multimodal_engine.py --model_type phi-3-vision --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision + + python run.py \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --kv_cache_free_gpu_memory_fraction 0.7 \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/ \ + --image_path=https://storage.googleapis.com/sfr-vision-language-research/LAVIS/assets/merlion.png + ``` +## Phi-4-multimodal +Navigate to the folder `TensorRT-LLM/examples/models/core/multimodal` + +1. Download Huggingface weights + + ```bash + export MODEL_NAME="Phi-4-multimodal-instruct" + export HF_DIR="tmp/hf_models/${MODEL_NAME}" + export CKPT_DIR="tmp/trt_models/${MODEL_NAME}/fp16/1-gpu" + export ENGINE_DIR="tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu" + git clone https://huggingface.co/microsoft/${MODEL_NAME} ${HF_DIR} + + ``` + +2. Convert Huggingface weights into TRT-LLM checkpoints and build TRT engines using scripts in `examples/models/core/phi`. + ```bash + python ../phi/convert_checkpoint.py \ + --model_dir ${HF_DIR} \ + --output_dir ${CKPT_DIR} \ + --dtype float16 + + trtllm-build \ + --checkpoint_dir ${CKPT_DIR} \ + --output_dir ${ENGINE_DIR} \ + --gpt_attention_plugin float16 \ + --gemm_plugin float16 \ + --max_batch_size 1 \ + --max_input_len 4096 \ + --max_seq_len 4608 \ + --max_multimodal_len 4096 + ``` + +3. Generate TensorRT engines for visual components and combine everything into final pipeline. +*Note: the encoders are not the TRT engines but are pure Pytorch ones* + + ```bash + python build_multimodal_engine.py --model_type phi-4-multimodal --model_path ${HF_DIR} --output_dir ${ENGINE_DIR} + + python run.py \ + --hf_model_dir ${HF_DIR} \ + --kv_cache_free_gpu_memory_fraction 0.7 \ + --engine_dir ${ENGINE_DIR} \ + --image_path=https://storage.googleapis.com/sfr-vision-language-research/LAVIS/assets/merlion.png + --audio_path=${HF_DIR}/examples/what_is_shown_in_this_image.wav + ``` +## Qwen2-VL +[Qwen2-VL Family](https://github.com/QwenLM/Qwen2-VL): is the latest version of the vision language models in the Qwen model families. Here we show how to deploy Qwen2-VL 2B and 7B in TensorRT-LLM. + +Firstly, please install transformers and qwen-vl-utils +```bash +pip install -r requirements-qwen2vl.txt +``` +### Support data types + * FP16 + * FP8 + +### Build and run vision model +* Download Huggingface weights + ```bash + export MODEL_NAME="Qwen2-VL-7B-Instruct" # or Qwen2-VL-2B-Instruct + git clone https://huggingface.co/Qwen/${MODEL_NAME} tmp/hf_models/${MODEL_NAME} + ``` +* Build engine of decoder model + + ```bash + python3 ../qwen/convert_checkpoint.py \ + --model_dir=tmp/hf_models/${MODEL_NAME} \ + --output_dir=tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --dtype float16 + + trtllm-build --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/llm \ + --gemm_plugin=float16 \ + --gpt_attention_plugin=float16 \ + --max_batch_size=4 \ + --max_input_len=2048 \ + --max_seq_len=3072 \ + --max_multimodal_len=1296 #(max_batch_size) * 324 (num_visual_features), this's for image_shape=[504,504] + ``` + +* Build engine of vision encoder model + ```bash + python build_multimodal_engine.py --model_type qwen2_vl --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision + ``` + +* Run test on multimodal/run.py with C++ runtime (LLM part only) + ```bash + python3 run.py \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/ + ``` +### Run Qwen2-VL decoder part by FP8 +* Build engine + ```bash + python ./examples/quantization/quantize.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --dtype float16 \ + --qformat fp8 \ + --kv_cache_dtype fp8 \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp8/1-gpu \ + --calib_size 512 + + trtllm-build --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp8/1-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp8/1-gpu/llm \ + --max_input_len=2048 \ + --max_seq_len 3072 \ + --gemm_plugin auto \ + --max_batch_size 4 \ + --max_multimodal_len=1296 + + # copy visiual engine directory `tmp/trt_engines/${MODEL_NAME}/fp16/1-gpu/vision/` to fp8 engine directory `tmp/trt_engines/${MODEL_NAME}/fp8/1-gpu/vision` + ``` +* Run test on multimodal/run.py with C++ runtime (LLM part only) + ```bash + python3 run.py \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp8/1-gpu/ + ``` ## Qwen-Image-Bench Evaluator @@ -31,3 +1183,153 @@ python examples/models/core/multimodal/qwen_image_bench_eval.py \ The script evaluates all five Qwen-Image-Bench level-1 dimensions by default: Quality, Aesthetics, Alignment, Real-world Fidelity, and Creative Generation. + +## Video NeVA + +[Video NeVA](https://github.com/NVIDIA/NeMo/blob/main/docs/source/multimodal/mllm/video_neva.rst) is a groundbreaking addition to the NeMo Multimodal ecosystem that could work with video modality. This model seamlessly integrates large language-centric models with a vision encoder, that can be deployed in TensorRT-LLM. + +1. Generate TRT-LLM engine for Nemotron model following example in `examples/models/core/nemotron/README.md`. To adhere to the NVGPT conventions of the conversion script. This will be used as our base LM for inference. + + ```bash + pip install decord # used for loading video + + python3 ../../../quantization/quantize.py \ + --nemo_ckpt_path /path/to/nemotron/model.nemo \ + --dtype bfloat16 \ + --batch_size 64 \ + --qformat full_prec \ + --output_dir nemotron-3/trt_ckpt/bf16/1-gpu + + + trtllm-build \ + --checkpoint_dir nemotron-3/trt_ckpt/bf16/1-gpu \ + --output_dir tmp/trt_engines/nemotron-3/bf16/1-gpu/llm \ + --gpt_attention_plugin bfloat16 \ + --gemm_plugin bfloat16 \ + --max_batch_size 1 \ + --max_input_len 4096 \ + --max_seq_len 4352 \ + --max_multimodal_len 3072 # 1 (max_batch_size) * (12 num_frames) * (256 image_token_len) + ``` + +2. Build TensorRT engines for visual components + + ```bash + python build_multimodal_engine.py --model_path /path/to/video/neva/projector.nemo --model_type video-neva --output_dir tmp/trt_engines/nemotron-3/visual_encoder --output_dir tmp/trt_engines/nemotron-3/bf16/1-gpu/vision + ``` + + ```bash + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir nemotron-3/trt_ckpt/bf16/1-gpu \ + --engine_dir tmp/trt_engines/nemotron-3/bf16/1-gpu \ + --input_text "Question: what is in the video? Answer:" \ + --video_path /path/to/your/local/video/file + ``` + + Note: use `--run_profiling` for performance measurement, use `--check_accuracy` for accuracy check. + +## Dataset Evaluation + +This section explains how to evaluate datasets using our provided script, including supported models and configurations. + +### Evaluation Command +To run an evaluation, use the following command: + +```bash +python ./examples/models/core/multimodal/eval.py \ + --model_type \ + --engine_dir \ + --hf_model_dir \ + --dataset_dir \ + --test_trtllm (or --test_hf, or both) \ + --accuracy_threshold \ + --eval_task \ + --max_ite 20 \ + --visual_engine_name +``` + +### Parameters +- `--model_type`: Specify the model type to evaluate. +- `--engine_dir`: Path to the model engines directory. +- `--hf_model_dir`: Path to the Hugging Face model directory. +- `--dataset_dir`: Path to the dataset directory. If not specified, will load the dataset from HF with the `--eval_task` as dataset tag. +- `--test_trtllm` or `--test_hf`: Specify which evaluation framework to use. Both can be used simultaneously. +- `--accuracy_threshold`: Set the accuracy threshold for evaluation. +- `--eval_task`: Specify the evaluation task. Supported tasks: `['lmms-lab/ai2d', 'lmms-lab/VQAv2', 'lmms-lab/MME']`. Default to `'lmms-lab/VQAv2'`. +- `--max_ite`: Maximum number of iterations, default to 20. +- `--visual_engine_name`: Name of the visual engine. + +### Supported Evaluation Tasks +The following evaluation tasks are supported: +- `lmms-lab/ai2d` +- `lmms-lab/VQAv2` +- `lmms-lab/MME` + +### Supported Model Types +The script supports the following model types: +- `blip2` +- `fuyu` +- `kosmos-2` +- `llava` +- `llava_next` +- `llava_onevision` +- `phi-3-vision` +- `qwen2_vl` +- `mllama` +- `vila` +- `cogvlm` +- `neva` +- `internvl` + +**Note:** The models `vila`, `cogvlm`, `neva`, and `internvl` do not support the `--test_hf` evaluation framework. + +## Enabling Tensor Parallelism for multi-GPU + +The LLM part of the pipeline can be run on multiple GPUs using tensor parallelism. +The visual encoder will be replicated on each GPU and operate in a data parallel fashion. + +To enable tensor parallelism, both weight conversion step (from Huggingface to FT format) +and engine building step should use additional arguments. Finally `run.py` should be prefixed +with `mpirun -n NUM_GPUS --allow-run-as-root`. + +The full set of commands to enable 2-way tensor parallelism for LLaVA is: + + ```bash + export MODEL_NAME="llava-1.5-7b-hf" + + python ../llama/convert_checkpoint.py \ + --model_dir tmp/hf_models/${MODEL_NAME} \ + --output_dir tmp/trt_models/${MODEL_NAME}/fp16/2-gpu \ + --dtype float16 --tp_size 2 + + trtllm-build \ + --checkpoint_dir tmp/trt_models/${MODEL_NAME}/fp16/2-gpu \ + --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/2-gpu/llm \ + --gemm_plugin float16 \ + --max_batch_size 1 \ + --max_input_len 2048 \ + --max_seq_len 2560 \ + --max_multimodal_len 576 + + python build_multimodal_engine.py --model_type llava --model_path tmp/hf_models/${MODEL_NAME} --output_dir tmp/trt_engines/${MODEL_NAME}/fp16/2-gpu/vision + + mpirun -n 2 --allow-run-as-root \ + python run.py \ + --max_new_tokens 30 \ + --hf_model_dir tmp/hf_models/${MODEL_NAME} \ + --engine_dir tmp/trt_engines/${MODEL_NAME}/fp16/2-gpu \ + ``` +## Enabling Embedding Table Offloading + +Embedding Table Offloading is a memory optimization technique that helps manage large embedding tables more efficiently. It offloads the embedding table to CPU memory and uses a chunked prefetching mechanism during processing. This approach is only available when operating in context chunk mode. + +To enable this feature, use the `--mm_embedding_offloading` argument: +```bash +python run.py \ + --enable_chunked_context \ + --mm_embedding_offloading true \ + --hf_model_dir ${HF_MODEL_PATH} \ + --engine_dir ${ENGINE_PATH} +``` +When not explicitly specified, this feature automatically enables if you're using a multimodal model along with context chunking enabled. diff --git a/examples/scaffolding/mcp/e2b/README.md b/examples/models/core/multimodal/__init__.py similarity index 100% rename from examples/scaffolding/mcp/e2b/README.md rename to examples/models/core/multimodal/__init__.py diff --git a/examples/models/core/multimodal/build_multimodal_engine.py b/examples/models/core/multimodal/build_multimodal_engine.py new file mode 100644 index 000000000000..59e8bb4ffa25 --- /dev/null +++ b/examples/models/core/multimodal/build_multimodal_engine.py @@ -0,0 +1,12 @@ +import argparse + +from tensorrt_llm.tools.multimodal_builder import (MultimodalEngineBuilder, + add_multimodal_arguments) + +if __name__ == '__main__': + parser = argparse.ArgumentParser() + parser = add_multimodal_arguments(parser) + args = parser.parse_args() + + builder = MultimodalEngineBuilder(args) + builder.build() diff --git a/examples/models/core/multimodal/eval.py b/examples/models/core/multimodal/eval.py new file mode 100644 index 000000000000..01804d290dee --- /dev/null +++ b/examples/models/core/multimodal/eval.py @@ -0,0 +1,317 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import os + +import aiohttp +import datasets +import torch +from transformers import AutoProcessor +from utils import add_common_args + +import tensorrt_llm +import tensorrt_llm.profiler as profiler +from tensorrt_llm import logger +from tensorrt_llm.runtime import MultimodalModelRunner + +SUPPORTED_MODEL_TYPES = { + 'blip2': 'Blip2ForConditionalGeneration', + 'fuyu': 'FuyuForCausalLM', + 'kosmos-2': 'Kosmos2ForConditionalGeneration', + 'llava': 'LlavaForConditionalGeneration', + 'llava_next': 'LlavaNextForConditionalGeneration', + 'llava_onevision': 'LlavaOnevisionForConditionalGeneration', + 'phi-3-vision': 'AutoModelForCausalLM', + 'qwen2_vl': 'Qwen2VLForConditionalGeneration', # not tested for TRT-LLM yet + 'mllama': 'MllamaForConditionalGeneration', + 'vila': None, + 'cogvlm': None, # not tested for TRT-LLM yet + 'neva': None, # not tested for TRT-LLM yet + 'internvl': None, +} +EVAL_TASKS = ['lmms-lab/ai2d', 'lmms-lab/VQAv2', 'lmms-lab/MME'] + + +def parse_arguments(args=None): + parser = argparse.ArgumentParser() + parser = add_common_args(parser) + parser.add_argument('--test_trtllm', + action='store_true', + default=None, + help="Evaluate the TensorRT-LLM.") + parser.add_argument('--test_hf', + action='store_true', + default=None, + help="Evaluate the Huggingface.") + parser.add_argument('--max_ite', type=int, default=20) + parser.add_argument('--eval_task', + type=str, + choices=EVAL_TASKS, + default='lmms-lab/VQAv2') + parser.add_argument('--model_type', + type=str, + default=None, + choices=SUPPORTED_MODEL_TYPES.keys()) + parser.add_argument( + '--accuracy_threshold', + type=float, + default=None, + help= + 'used to check the accuracy of test_trtllm. Should be between 0 and 100.' + ) + parser.add_argument( + '--dataset_dir', + type=str, + default=None, + help="The local directory of the dataset for evaluation; " + "will download the dataset from huggingface hub if not specified.") + parser.add_argument( + '--dataset_cache_dir', + type=str, + default=None, + help="The local cache directory for dataset; " + "will use `~/.cache/huggingface/datasets` if not specified.") + return parser.parse_args(args=args) + + +def load_dataset(args) -> datasets.Dataset: + split_name = 'validation' if 'VQAv2' in args.eval_task else 'test' + + if args.dataset_dir is not None and os.path.exists( + os.path.join(args.dataset_dir, "dataset_info.json")): + logger.info(f"load dataset by load_from_disk from {args.dataset_dir}") + dataset = datasets.load_from_disk(args.dataset_dir) + + else: + logger.info( + f"load dataset by load_dataset from {args.dataset_dir or args.eval_task}" + ) + dataset = datasets.load_dataset( + args.dataset_dir or args.eval_task, + cache_dir=args.dataset_cache_dir, + split=split_name, + storage_options={ + 'client_kwargs': { + 'timeout': aiohttp.ClientTimeout(total=3600) + } + }, + trust_remote_code=True, + ) + return dataset + + +def load_hf_model(args): + if SUPPORTED_MODEL_TYPES[args.model_type] is None: + raise ValueError(f"Unsupported HF model_type: {args.model_type}") + profiler.start('load HF model') + model_class = getattr(__import__('transformers'), + SUPPORTED_MODEL_TYPES[args.model_type]) + hf_model = model_class.from_pretrained(args.hf_model_dir, + dtype=torch.float16, + device_map="cuda:0", + trust_remote_code=True) + profiler.stop('load HF model') + + logger.info( + f'Load HF model takes: {profiler.elapsed_time_in_sec("load HF model")} sec' + ) + return hf_model + + +def load_trtllm_model(args): + profiler.start('load TensorRT LLM model') + trtllm_model = MultimodalModelRunner(args) + profiler.stop('load TensorRT LLM model') + logger.info( + f'Load TensorRT LLM model takes: {profiler.elapsed_time_in_sec("load TensorRT LLM model")} sec' + ) + return trtllm_model + + +def prepare_prompts(task, data, model_type, processor) -> str: + prompts = None + question = data['question'] + if question[-1] != '?': + question += '?' + + if task == 'lmms-lab/ai2d': + for j, option in enumerate(data['options']): + question += f" ({j}) {option}" + + if model_type in ['blip2', 'neva']: + prompts = f"Question: {question} Answer: " + elif model_type == 'fuyu': + prompts = f"Answer the following {task} question based on the image: {question}" + elif model_type == 'kosmos-2': + prompts = f" Question: {question} Answer: " + elif model_type == 'cogvlm': + prompts = f"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {question} ASSISTANT:" + elif model_type in ['llava', 'llava_next', 'llava_onevision']: + conversation = [ + { + "role": + "user", + "content": [ + { + "type": "image" + }, + { + "type": "text", + "text": question + }, + ], + }, + ] + prompts = processor.apply_chat_template(conversation, + add_generation_prompt=True) + elif model_type in ['vila', 'internvl']: + prompts = f"\n{question}" + elif model_type == 'phi-3-vision': + messages = [ + { + "role": "user", + "content": f"<|image_1|>\n{question}" + }, + ] + prompts = processor.tokenizer.apply_chat_template( + messages, tokenize=False, add_generation_prompt=True) + elif model_type == 'qwen2_vl': + conversation = [{ + "role": + "user", + "content": [{ + "type": "image", + }, { + "type": "text", + "text": question + }] + }] + prompts = processor.apply_chat_template(conversation, + add_generation_prompt=True) + elif model_type == 'mllama': + prompts = processor.apply_chat_template(images=data['image'], + text=question + "; answer: ") + else: + raise ValueError(f"Unsupported model_type: {model_type}") + + return prompts + + +def eval(output, task, data) -> bool: + output = output.strip().lower() + if task == 'lmms-lab/VQAv2': + return any(answer['answer'] in output for answer in data['answers']) + else: + return data['answer'].lower() in output + + +os.environ["TOKENIZERS_PARALLELISM"] = "false" +args = parse_arguments() +if args.model_type not in SUPPORTED_MODEL_TYPES: + raise ValueError(f"Unsupported model_type: {args.model_type}") + +logger.set_level(args.log_level) + +runtime_rank = tensorrt_llm.mpi_rank() +dataset = load_dataset(args) +hf_model = load_hf_model(args) if args.test_hf else None +trtllm_model = load_trtllm_model(args) if args.test_trtllm else None +if SUPPORTED_MODEL_TYPES[args.model_type] is None: + hf_processor = None +else: + hf_processor = AutoProcessor.from_pretrained(args.hf_model_dir, + trust_remote_code=True) +hf_correct = trtllm_correct = 0 + +if args.model_type == 'mllama': + from tensorrt_llm.runtime.processor_wrapper import MllamaProcessorWrapper + hf_processor = MllamaProcessorWrapper(hf_processor, logger) + +torch.random.manual_seed(0) +profiler.start('evaluation') +if args.test_trtllm or args.test_hf: + for i in range(args.max_ite): + logger.debug(f"Ite: {i:3d}") + data = dataset[i] + if i > len(dataset): + break + prompts = prepare_prompts(args.eval_task, data, args.model_type, + hf_processor) + image = data['image'] + + if args.test_hf: + assert hf_model is not None, f"Unsupported HF model_type: {args.model_type}" + profiler.start('hf') + inputs = hf_processor( + images=image, + text=prompts, + return_tensors="pt", + ).to(hf_model.device, + torch.float16) # add torch.float16 for llava-onevision + input_length = inputs.input_ids.shape[-1] + hf_output = hf_model.generate(**inputs, + max_new_tokens=args.max_new_tokens) + hf_result = (hf_processor.batch_decode( + hf_output, skip_special_tokens=True)[0] if args.model_type in [ + 'blip2' + ] else hf_processor.decode(hf_output[0][input_length:], + skip_special_tokens=True)) + hf_correct += eval(hf_result, args.eval_task, data) + profiler.stop('hf') + + if args.test_trtllm: + profiler.start('tensorrt_llm') + _, output_text = trtllm_model.run( + input_text=prompts, + input_image=image, + input_audio=None, + max_new_tokens=args.max_new_tokens) + if runtime_rank == 0: + trtllm_result = output_text[0][0] + trtllm_correct += eval(trtllm_result, args.eval_task, data) + profiler.stop('tensorrt_llm') + + if runtime_rank == 0: + if args.eval_task == 'lmms-lab/VQAv2': + answer = data['answers'] + else: + answer = data['answer'] + logger.debug(f"prompts: {prompts}") + logger.debug(f"reference answer: {answer}") + if args.test_hf: + logger.debug(f"HF's answer: {hf_result}") + if args.test_trtllm: + logger.debug(f"TRT-LLM's answer: {trtllm_result}") + + if runtime_rank == 0: + logger.info(f"total iterations: {args.max_ite}") + if args.test_hf: + logger.info( + f"HF's accuracy: {100 * hf_correct / args.max_ite:4.2f}%") + if args.test_trtllm: + logger.info( + f"TRT-LLM's accuracy: {100 * trtllm_correct / args.max_ite:4.2f}%" + ) + # check if the accuracy is above the threshold + if args.accuracy_threshold is not None and args.test_trtllm: + assert trtllm_correct / args.max_ite >= args.accuracy_threshold / 100, \ + f"TRT-LLM's accuracy is below the threshold: {args.accuracy_threshold}%." +else: + logger.info("Neither enable test_trtllm nor enable test_hf") + +profiler.stop('evaluation') +logger.info( + f'Evaluation takes: {profiler.elapsed_time_in_sec("evaluation")} sec') diff --git a/examples/models/core/multimodal/requirements-eclair.txt b/examples/models/core/multimodal/requirements-eclair.txt new file mode 100644 index 000000000000..281c8ae93a91 --- /dev/null +++ b/examples/models/core/multimodal/requirements-eclair.txt @@ -0,0 +1 @@ +timm diff --git a/examples/models/core/multimodal/requirements-internlm-xcomposer2.txt b/examples/models/core/multimodal/requirements-internlm-xcomposer2.txt new file mode 100644 index 000000000000..5d27a97ebb68 --- /dev/null +++ b/examples/models/core/multimodal/requirements-internlm-xcomposer2.txt @@ -0,0 +1 @@ +transformers==4.56.0 diff --git a/examples/models/core/multimodal/requirements-llava_onevision.txt b/examples/models/core/multimodal/requirements-llava_onevision.txt new file mode 100644 index 000000000000..126cbda08e15 --- /dev/null +++ b/examples/models/core/multimodal/requirements-llava_onevision.txt @@ -0,0 +1,4 @@ +git+https://github.com/LLaVA-VL/LLaVA-NeXT.git +transformers>=4.56.0 +einops +av diff --git a/examples/models/core/multimodal/requirements-qwen2vl.txt b/examples/models/core/multimodal/requirements-qwen2vl.txt new file mode 100644 index 000000000000..50f14d1d8095 --- /dev/null +++ b/examples/models/core/multimodal/requirements-qwen2vl.txt @@ -0,0 +1,2 @@ +accelerate +qwen-vl-utils==0.0.8 # 0.0.9 has bug https://github.com/QwenLM/Qwen2-VL/pull/673, rollback until a newer version is released diff --git a/examples/models/core/multimodal/requirements-vila.txt b/examples/models/core/multimodal/requirements-vila.txt new file mode 100644 index 000000000000..00775aa0cdfa --- /dev/null +++ b/examples/models/core/multimodal/requirements-vila.txt @@ -0,0 +1,2 @@ +git+https://github.com/bfshi/scaling_on_scales.git +transformers==4.56.0 diff --git a/examples/models/core/multimodal/run.py b/examples/models/core/multimodal/run.py new file mode 100644 index 000000000000..fb13554848e7 --- /dev/null +++ b/examples/models/core/multimodal/run.py @@ -0,0 +1,128 @@ +import argparse +import os + +from utils import add_common_args, compute_str_match_rate + +import tensorrt_llm +import tensorrt_llm.profiler as profiler +from tensorrt_llm import logger +from tensorrt_llm.runtime import MultimodalModelRunner + + +def print_result(model, input_text, output_text, args): + logger.info("---------------------------------------------------------") + if model.model_type != 'nougat': + logger.info(f"\n[Q] {input_text}") + for i in range(len(output_text)): + logger.info(f"\n[A]: {output_text[i]}") + + if args.num_beams == 1: + output_ids = model.tokenizer(output_text[0][0], + add_special_tokens=False)['input_ids'] + logger.info(f"Generated {len(output_ids)} tokens") + + if args.check_accuracy: + if model.model_type != 'nougat': + if model.model_type == "vila": + for i in range(len(args.image_path.split(args.path_sep))): + if i % 2 == 0: + assert output_text[i][0].lower( + ) == "the image captures a bustling city intersection teeming with life. from the perspective of a car's dashboard camera, we see" + else: + assert output_text[i][0].lower( + ) == "the image captures the iconic merlion statue in singapore, a renowned worldwide landmark. the merlion, a mythical" + elif model.model_type == "llava": + for i in range(len(args.image_path.split(args.path_sep))): + assert output_text[i][0].lower() == 'singapore' + elif model.model_type == 'fuyu': + assert output_text[0][0].lower() == '4' + elif model.model_type == "pix2struct": + assert "characteristic | cat food, day | cat food, wet | cat treats" in output_text[ + 0][0].lower() + elif model.model_type in [ + 'blip2', 'neva', 'phi-3-vision', 'llava_next', + 'phi-4-multimodal', 'pixtral' + ]: + assert 'singapore' in output_text[0][0].lower() + elif model.model_type == 'video-neva': + assert 'robot' in output_text[0][0].lower() + elif model.model_type == 'kosmos-2': + assert 'snowman' in output_text[0][0].lower() + elif model.model_type == "mllama": + if "If I had to write a haiku for this one" in input_text: + ref_1 = ", it would be:.\\nPeter Rabbit is a rabbit.\\nHe lives in a cozy little house.\\nHe's a very good rabbit.\\" + ref_2 = "Here is a haiku for the image:\n\n" + + elif "Answer:" in input_text: + ref_1 = "2,173. A 1 2 3 4 5 6 Date Income 2005-12-17" + ref_2 = "Answer: 2,173. 1 2 3 4 5 6 Date Income 2005-12-17" + + elif "The key to life is" in input_text: + ref_1 = "to find your passion and pursue it with all your heart. For me, that passion is photography. I love capturing the beauty of the world around me" + ref_2 = "not to be found in the external world," + output = output_text[0][0] + match_rate = max(compute_str_match_rate(ref_1, output), + compute_str_match_rate(ref_2, output)) + logger.info(f"match rate: {match_rate}") + assert match_rate >= 50, \ + f"expected results: '{ref_1}' or '{ref_2}', generated results: '{output}'" + + elif model.model_type == 'llava_onevision': + if args.video_path is None: + assert 'singapore' in output_text[0][0].lower() + else: + assert 'the video is funny because the child\'s actions are' in output_text[ + 0][0].lower() + elif model.model_type == "qwen2_vl": + assert 'dog' in output_text[0][0].lower() + else: + assert output_text[0][0].lower() == 'singapore' + + if args.run_profiling: + msec_per_batch = lambda name: 1000 * profiler.elapsed_time_in_sec( + name) / args.profiling_iterations + logger.info('Latencies per batch (msec)') + logger.info('e2e generation: %.1f' % (msec_per_batch('Generate'))) + logger.info(' ' * 2 + 'Preprocessing: %.1f' % + (msec_per_batch('Preprocess'))) + logger.info(' ' * 4 + 'Vision encoder: %.1f' % + (msec_per_batch('Vision encoder'))) + if profiler.elapsed_time_in_sec('Feature transform') is not None: + logger.info(' ' * 4 + 'Feature transform: %.1f' % + (msec_per_batch('Feature transform'))) + logger.info(' ' * 2 + 'LLM generate: %.1f' % (msec_per_batch('LLM'))) + logger.info(' ' * 2 + 'Tokenizer decode: %.1f' % + (msec_per_batch('Tokenizer decode'))) + + logger.info("---------------------------------------------------------") + + +if __name__ == '__main__': + os.environ["TOKENIZERS_PARALLELISM"] = "false" + parser = argparse.ArgumentParser() + parser = add_common_args(parser) + args = parser.parse_args() + logger.set_level(args.log_level) + + model = MultimodalModelRunner(args) + visual_data = model.load_test_data(args.image_path, args.video_path) + audio_data = model.load_test_audio(args.audio_path) + + if args.run_profiling: + num_warmup_iters = 3 # Multiple iterations to load both vision and LLM engines into memory + for _ in range(num_warmup_iters): + input_text, output_text = model.run(args.input_text, visual_data, + audio_data, args.max_new_tokens) + profiler.reset() + + num_iters = args.profiling_iterations if args.run_profiling else 1 + + for _ in range(num_iters): + input_text, output_text = model.run(args.input_text, visual_data, + audio_data, args.max_new_tokens) + + runtime_rank = tensorrt_llm.mpi_rank() + if runtime_rank == 0: + print_result(model, input_text, output_text, args) + +# TODO: raise error if VILA mode 1 with C++ runtime diff --git a/examples/models/core/multimodal/utils.py b/examples/models/core/multimodal/utils.py new file mode 100644 index 000000000000..6d13fb600f40 --- /dev/null +++ b/examples/models/core/multimodal/utils.py @@ -0,0 +1,163 @@ +def add_common_args(parser): + parser.add_argument('--max_new_tokens', type=int, default=128) + parser.add_argument('--batch_size', type=int, default=1) + parser.add_argument('--log_level', type=str, default='info') + parser.add_argument('--engine_dir', + type=str, + default=None, + help='Directory containing visual and LLM TRT engines') + parser.add_argument('--visual_engine_name', + type=str, + default='model.engine', + help='Name of visual TRT engine') + parser.add_argument('--audio_engine_name', + type=str, + default='model.engine', + help='Name of audio TRT engine') + parser.add_argument('--hf_model_dir', + type=str, + default=None, + help="Directory containing tokenizer") + parser.add_argument('--input_text', + type=str, + nargs='+', + default=None, + help='Text prompt to LLM') + parser.add_argument('--num_beams', + type=int, + help="Use beam search if num_beams >1", + default=1) + parser.add_argument('--top_k', type=int, default=1) + parser.add_argument('--top_p', type=float, default=0.0) + parser.add_argument('--temperature', type=float, default=1.0) + parser.add_argument('--repetition_penalty', type=float, default=1.0) + parser.add_argument('--run_profiling', + action='store_true', + help='Profile runtime over several iterations') + parser.add_argument('--profiling_iterations', + type=int, + help="Number of iterations to run profiling", + default=20) + parser.add_argument('--check_accuracy', + action='store_true', + help='Check correctness of text output') + parser.add_argument( + '--video_path', + type=str, + default=None, + help= + 'Path to your local video file, using \'llava-onevision-accuracy\' to check the Llava-OneVision model accuracy' + ) + parser.add_argument( + '--video_num_frames', + type=int, + help= + "The number of frames sampled from the video in the Llava-OneVision model.", + default=None) + parser.add_argument("--image_path", + type=str, + nargs='+', + default=None, + help='List of input image paths, separated by symbol') + parser.add_argument("--audio_path", + type=str, + default=None, + help='input audio path') + parser.add_argument("--path_sep", + type=str, + default=",", + help='Path separator symbol') + parser.add_argument("--prompt_sep", + type=str, + default=",", + help="Prompt separator symbol") + parser.add_argument('--enable_context_fmha_fp32_acc', + action='store_true', + default=None, + help="Enable FMHA runner FP32 accumulation.") + parser.add_argument( + '--enable_chunked_context', + action='store_true', + help='Enables chunked context (only available with cpp session).', + ) + parser.add_argument( + '--mm_embedding_offloading', + type=lambda s: s.lower() == "true", + default=None, + help= + 'Enable position table offloading. When not specified, defaults to True if using a multimodal model with chunked context.' + ) + parser.add_argument( + '--session', + default='cpp_llm_only', + type=str, + choices=['python', 'cpp_llm_only', 'cpp'], + help= + 'Rumtime used to run the models. \n`cpp_llm_only`: vision engine run in python runtime, but LLM in pybind cpp runtime\n`python`: everything runs in python runtime\n`cpp`: everything runs in C++ runtime' + ) + parser.add_argument( + '--kv_cache_free_gpu_memory_fraction', + default=0.7, + type=float, + help='Specify the free gpu memory fraction.', + ) + parser.add_argument( + '--cross_kv_cache_fraction', + default=0.5, + type=float, + help= + 'Specify the kv cache fraction reserved for cross attention. Only applicable for encoder-decoder models. By default 0.5 for self and 0.5 for cross.', + ) + parser.add_argument( + '--multi_block_mode', + type=lambda s: s.lower() in + ("yes", "true", "t", "1" + ), # custom boolean function to convert input string to boolean + default=True, + help= + "Distribute the work across multiple CUDA thread-blocks on the GPU for masked MHA kernel." + ) + parser.add_argument( + '--lora_task_uids', + type=str, + default=None, + nargs="+", + help="The list of LoRA task uids; use -1 to disable the LoRA module") + parser.add_argument('--debug_mode', + default=False, + action='store_true', + help="Whether or not to turn on the debug mode") + parser.add_argument( + '--trust_remote_code', + action='store_true', + default=False, + help='Allow loading models with custom remote code from HuggingFace Hub. ' + 'Only enable this for models from trusted sources.') + return parser + + +def levenshtein_distance(s1, s2): + if len(s1) < len(s2): + return levenshtein_distance(s2, s1) + + if len(s2) == 0: + return len(s1) + + previous_row = range(len(s2) + 1) + for i, c1 in enumerate(s1): + current_row = [i + 1] + for j, c2 in enumerate(s2): + insertions = previous_row[j + 1] + 1 + deletions = current_row[j] + 1 + substitutions = previous_row[j] + (c1 != c2) + current_row.append(min(insertions, deletions, substitutions)) + previous_row = current_row + + return previous_row[-1] + + +def compute_str_match_rate(s1, s2): + distance = levenshtein_distance(s1, s2) + max_length = max(len(s1), len(s2)) + match_rate = (1 - distance / max_length) * 100 + return match_rate diff --git a/examples/models/core/nemotron/README_nemotron-3.md b/examples/models/core/nemotron/README_nemotron-3.md new file mode 100644 index 000000000000..0816eb995fd8 --- /dev/null +++ b/examples/models/core/nemotron/README_nemotron-3.md @@ -0,0 +1,214 @@ +# Nemotron-3 + +This document demonstrates how to build the Nemotron models using TensorRT LLM and run on a single GPU or multiple GPUs. + +- [Nemotron](#nemotron) + - [Overview](#overview) + - [Support Matrix](#support-matrix) + - [Usage](#usage) + - [Download weights from HuggingFace Transformers](#download-weights-from-huggingface-transformers) + - [Build TensorRT engine(s)](#build-tensorrt-engines) + - [FP8 Quantization](#fp8-quantization) + - [INT4 AWQ Quantization](#int4-awq-quantization) + - [Run Inference](#run-inference) + +## Overview + +The TensorRT LLM Nemotron implementation is based on the GPT model, which can be found in [`tensorrt_llm/models/gpt/model.py`](../../../../tensorrt_llm/models/gpt/model.py). The TensorRT LLM Nemotron example is located in [`examples/models/core/nemotron`](./). + +In addition, there are two shared files in the parent folder [`examples`](../../../) for inference and evaluation: + +* [`run.py`](../../../run.py) to run the inference on an input text; +* [`summarize.py`](../../../summarize.py) to summarize the articles in the [cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +## Support Matrix + * FP16/BF16 + * FP8 + * INT4 AWQ + * Tensor Parallel + * Pipeline Parallel + * Inflight Batching + * PAGED_KV_CACHE + * STRONGLY TYPED + * checkpoint type: Nemo, Huggingface (HF) + +## Nemo checkpoint - Usage + +### Download weights from HuggingFace Transformers + + +Install the dependencies and setup `git-lfs`. + +```bash +# Install dependencies +pip install -r requirements.txt + +# Setup git-lfs +git lfs install +``` + +Download one or more Nemotron models that you would like to build to TensorRT LLM engines. You can download from the [HuggingFace](https://huggingface.co) hub: + +```bash +# Download nemotron-3-8b-base-4k +git clone https://huggingface.co/nvidia/nemotron-3-8b-base-4k + +# Download nemotron-3-8b-chat-4k-sft +git clone https://huggingface.co/nvidia/nemotron-3-8b-chat-4k-sft + +# Download nemotron-3-8b-chat-4k-rlhf +git clone https://huggingface.co/nvidia/nemotron-3-8b-chat-4k-rlhf +``` + +### Build TensorRT engine(s) +The [`examples/quantization/quantize.py`](../../../quantization/quantize.py) script can quantize the Nemotron models and export to TensorRT LLM checkpoints. You may optionally skip the quantization step by specifying `--qformat full_prec` and thus export float16 or bfloat16 TensorRT LLM checkpoints. + +The `trtllm-build` command builds TensorRT LLM engines from TensorRT LLM checkpoints. The number of engine files is same to the number of GPUs used to run inference. Normally, `trtllm-build` uses one GPU by default, but if you have already more GPUs available at build time, you may enable parallel builds to make the engine building process faster by adding the `--workers` argument. + +Here are some examples: + +```bash +# single gpu, dtype bfloat16 +python3 ../../../quantization/quantize.py \ + --nemo_ckpt_path nemotron-3-8b-base-4k/Nemotron-3-8B-Base-4k.nemo \ + --dtype bfloat16 \ + --batch_size 64 \ + --qformat full_prec \ + --output_dir nemotron-3-8b/trt_ckpt/bf16/1-gpu + +trtllm-build --checkpoint_dir nemotron-3-8b/trt_ckpt/bf16/1-gpu \ + --gpt_attention_plugin bfloat16 \ + --gemm_plugin bfloat16 \ + --output_dir nemotron-3-8b/trt_engines/bf16/1-gpu +``` + +```bash +# 2-way tensor parallelism +python3 ../../../quantization/quantize.py \ + --nemo_ckpt_path nemotron-3-8b-base-4k/Nemotron-3-8B-Base-4k.nemo \ + --dtype bfloat16 \ + --batch_size 64 \ + --qformat full_prec \ + --tp_size 2 \ + --output_dir nemotron-3-8b/trt_ckpt/bf16/tp2 + +trtllm-build --checkpoint_dir nemotron-3-8b/trt_ckpt/bf16/tp2 \ + --gpt_attention_plugin bfloat16 \ + --gemm_plugin bfloat16 \ + --workers 2 \ + --output_dir nemotron-3-8b/trt_engines/bf16/tp2 +``` + +```bash +# 2-way tensor parallelism for both calibration and inference +mpirun -np 2 \ + python3 ../../../quantization/quantize.py \ + --nemo_ckpt_path nemotron-3-8b-base-4k/Nemotron-3-8B-Base-4k.nemo \ + --dtype bfloat16 \ + --batch_size 64 \ + --qformat full_prec \ + --calib_tp_size 2 \ + --tp_size 2 \ + --output_dir nemotron-3-8b/trt_ckpt/bf16/tp2 + +trtllm-build --checkpoint_dir nemotron-3-8b/trt_ckpt/bf16/tp2 \ + --gpt_attention_plugin bfloat16 \ + --gemm_plugin bfloat16 \ + --workers 2 \ + --output_dir nemotron-3-8b/trt_engines/bf16/tp2 +``` + +#### FP8 Quantization + +Quantize the Nemotron models to FP8 by specifying `--qformat fp8` to `quantize.py`. + +```bash +# single gpu, fp8 quantization +python3 ../../../quantization/quantize.py \ + --nemo_ckpt_path nemotron-3-8b-base-4k/Nemotron-3-8B-Base-4k.nemo \ + --dtype bfloat16 \ + --batch_size 64 \ + --qformat fp8 \ + --output_dir nemotron-3-8b/trt_ckpt/fp8/1-gpu + +trtllm-build --checkpoint_dir nemotron-3-8b/trt_ckpt/fp8/1-gpu \ + --gpt_attention_plugin bfloat16 \ + --output_dir nemotron-3-8b/trt_engines/fp8/1-gpu +``` + +#### INT4 AWQ Quantization + +Quantize the Nemotron models using INT4 AWQ by specifying `--qformat int4_awq` to `quantize.py`. + +```bash +# single gpu, int4 awq quantization +python3 ../../../quantization/quantize.py \ + --nemo_ckpt_path nemotron-3-8b-base-4k/Nemotron-3-8B-Base-4k.nemo \ + --dtype bfloat16 \ + --batch_size 64 \ + --qformat int4_awq \ + --output_dir nemotron-3-8b/trt_ckpt/int4_awq/1-gpu + +trtllm-build --checkpoint_dir nemotron-3-8b/trt_ckpt/int4_awq/1-gpu \ + --gpt_attention_plugin bfloat16 \ + --output_dir nemotron-3-8b/trt_engines/int4_awq/1-gpu +``` + +### Run Inference + +The `summarize.py` script can run the built engines to summarize the articles from the +[cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) dataset. + +```bash +# single gpu +python3 ../../../summarize.py --test_trt_llm \ + --no_add_special_tokens \ + --engine_dir nemotron-3-8b/trt_engines/bf16/1-gpu \ + --vocab_file nemotron-3-8b/trt_ckpt/bf16/1-gpu/tokenizer.model + +# multiple gpus +mpirun -np 2 \ + python3 ../../../summarize.py --test_trt_llm \ + --no_add_special_tokens \ + --engine_dir nemotron-3-8b/trt_engines/bf16/tp2 \ + --vocab_file nemotron-3-8b/trt_ckpt/bf16/tp2/tokenizer.model +``` + +If the engines are run successfully, you will see output like: +``` +...... +[04/23/2024-09:55:54] [TRT-LLM] [I] TensorRT LLM (total latency: 14.926485538482666 sec) +[04/23/2024-09:55:54] [TRT-LLM] [I] TensorRT LLM (total output tokens: 2000) +[04/23/2024-09:55:54] [TRT-LLM] [I] TensorRT LLM (tokens per second: 133.99001357980129) +[04/23/2024-09:55:54] [TRT-LLM] [I] TensorRT LLM beam 0 result +[04/23/2024-09:55:54] [TRT-LLM] [I] rouge1 : 19.48743720965424 +[04/23/2024-09:55:54] [TRT-LLM] [I] rouge2 : 6.272381295466071 +[04/23/2024-09:55:54] [TRT-LLM] [I] rougeL : 15.011005943152721 +[04/23/2024-09:55:54] [TRT-LLM] [I] rougeLsum : 17.76145734406502 +``` + +## HF checkpoint - Usage +Support for Nemotron models was added with transformers 4.44.0 release. + +```bash +# install transformers library +pip install transformers>=4.44.0 +# Download hf minitron model +git clone https://huggingface.co/nvidia/Minitron-4B-Base + +# Convert to TensorRT LLM checkpoint +python3 ../gpt/convert_checkpoint.py --model_dir Minitron-4B-Base \ + --dtype bfloat16 \ + --output_dir minitron/trt_ckpt/bf16/1-gpu + +# Build TensorRT LLM engines +trtllm-build --checkpoint_dir minitron/trt_ckpt/bf16/1-gpu \ + --gemm_plugin auto \ + --output_dir minitron/trt_engines/bf16/1-gpu + +# Run inference +python3 ../../../run.py --engine_dir minitron/trt_engines/bf16/1-gpu \ + --tokenizer_dir Minitron-4B-Base \ + --input_text "def print_hello_world():" \ + --max_output_len 20 +``` diff --git a/examples/models/core/nemotron/requirements.txt b/examples/models/core/nemotron/requirements.txt new file mode 100644 index 000000000000..f8d1f8e1abf4 --- /dev/null +++ b/examples/models/core/nemotron/requirements.txt @@ -0,0 +1,7 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +nemo-toolkit[all]==2.0.0rc1 +megatron-core @ git+https://github.com/NVIDIA/Megatron-LM@core_r0.8.0 +datasets==3.1.0 +evaluate +rouge_score diff --git a/examples/models/core/qwen2audio/README.md b/examples/models/core/qwen2audio/README.md new file mode 100644 index 000000000000..92115345077e --- /dev/null +++ b/examples/models/core/qwen2audio/README.md @@ -0,0 +1,80 @@ +# Guide to Qwen2-Audio deployment pipeline + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +1. Download the Qwen2-Audio model. + ```bash + git lfs install + export MODEL_PATH="tmp/Qwen2-Audio-7B-Instruct" + git clone https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct $MODEL_PATH + ``` +2. Generate the TensorRT engine of audio encoder. + ```bash + export ENGINE_DIR="./trt_engines/qwen2audio/fp16" + python3 ../multimodal/build_multimodal_engine.py --model_type qwen2_audio --model_path $MODEL_PATH --max_batch_size 32 --output_dir ${ENGINE_DIR}/audio + ``` + + The TensorRT engine will be generated under `${ENGINE_DIR}/audio`. + +3. Build Qwen2 LLM TensorRT engine. +- Convert checkpoint + 1. Install packages + ```bash + pip install -r requirements.txt + ``` + 2. Convert + 2.1 FP16 checkpoint + ```bash + python3 ../qwen/convert_checkpoint.py --model_dir=$MODEL_PATH \ + --dtype=float16 \ + --output_dir=./tllm_checkpoint_1gpu_fp16 + ``` + 2.2 (Optional) INT8 Weight Only checkpoint + ```bash + python3 ../qwen/convert_checkpoint.py --model_dir=$MODEL_PATH \ + --dtype=float16 \ + --use_weight_only \ + --weight_only_precision=int8 \ + --output_dir=./tllm_checkpoint_1gpu_fp16_wo8 + ``` + +- Build TensorRT LLM engine + + NOTE: `max_prompt_embedding_table_size = query_token_num * max_batch_size`, therefore, if you change `max_batch_size`, `--max_prompt_embedding_table_size` must be reset accordingly. + ```bash + trtllm-build --checkpoint_dir=./tllm_checkpoint_1gpu_fp16 \ + --gemm_plugin=float16 --gpt_attention_plugin=float16 \ + --max_batch_size=1 --max_prompt_embedding_table_size=4096 \ + --output_dir=${ENGINE_DIR}/llm + ``` + The built Qwen engines are located in `${ENGINE_DIR}/llm`. + + You can replace the `--checkpoint_dir` with INT8 Weight Only checkpoint to build INT8 Weight Only engine as well. + For more information about Qwen, refer to the README.md in [`example/models/core/qwen`](../qwen). + +4. Assemble everything into the Qwen2-Audio pipeline. + + 4.1 Run with FP16 LLM engine + ```bash + python3 run.py \ + --tokenizer_dir=$MODEL_PATH \ + --engine_dir=${ENGINE_DIR}/llm \ + --audio_engine_path=${ENGINE_DIR}/audio/model.engine \ + --audio_url='./audio/glass-breaking-151256.mp3' + ``` + 4.2 (Optional) For multiple rounds of dialogue, you can run: + ```bash + python3 run_chat.py \ + --tokenizer_dir=$MODEL_PATH \ + --engine_dir=${ENGINE_DIR}/llm \ + --audio_engine_path=${ENGINE_DIR}/audio/model.engine \ + --max_new_tokens=256 + ``` + + Note: + - This example supports reusing the KV Cache for audio segments by assigning unique audio IDs. + - To further optimize performance, users can also cache the audio features (encoder output) to bypass the audio encoder if the original audio data remains unchanged. diff --git a/examples/models/core/qwen2audio/audio/glass-breaking-151256.mp3 b/examples/models/core/qwen2audio/audio/glass-breaking-151256.mp3 new file mode 100644 index 000000000000..150e1080f2e5 Binary files /dev/null and b/examples/models/core/qwen2audio/audio/glass-breaking-151256.mp3 differ diff --git a/examples/models/core/qwen2audio/requirements.txt b/examples/models/core/qwen2audio/requirements.txt new file mode 100644 index 000000000000..1d6d844e4b64 --- /dev/null +++ b/examples/models/core/qwen2audio/requirements.txt @@ -0,0 +1,10 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.dev0 +datasets==3.1.0 +evaluate +rouge_score +transformers>=4.45.0 +transformers-stream-generator +sentencepiece>=0.1.99 +tiktoken +einops diff --git a/examples/models/core/qwen2audio/run.py b/examples/models/core/qwen2audio/run.py new file mode 100644 index 000000000000..a0b0a68fb1c9 --- /dev/null +++ b/examples/models/core/qwen2audio/run.py @@ -0,0 +1,640 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import argparse +import json +import os +from io import BytesIO +from urllib.request import urlopen + +import librosa +import tensorrt as trt +import torch +from transformers import AutoConfig, AutoProcessor, AutoTokenizer +from utils import add_common_args + +import tensorrt_llm +import tensorrt_llm.profiler as profiler +from tensorrt_llm import logger +from tensorrt_llm.llmapi.kv_cache_type import KVCacheType +from tensorrt_llm.quantization import QuantMode +from tensorrt_llm.runtime import (PYTHON_BINDINGS, ModelConfig, ModelRunner, + SamplingConfig, Session, TensorInfo) + +if PYTHON_BINDINGS: + from tensorrt_llm.runtime import ModelRunnerCpp + + +def get_engine_name(rank): + return "rank{}.engine".format(rank) + + +def trt_dtype_to_torch(dtype): + if dtype == trt.float16: + return torch.float16 + elif dtype == trt.float32: + return torch.float32 + elif dtype == trt.int32: + return torch.int32 + else: + raise TypeError("%s is not supported" % dtype) + + +class QWenInfer(object): + + def __init__(self, + audio_engine_path, + tokenizer_dir, + engine_dir, + log_level, + output_csv, + output_npy, + num_beams, + gpu_id=0): + self.audio_engine_path = audio_engine_path + self.tokenizer_dir = tokenizer_dir + self.engine_dir = engine_dir + self.log_level = log_level + self.max_seq_len = 0 + self.runner = None + self.hf_audio_tower = None + self.tokenizer = None + self.config = None + self.sampling_config = None + self.output_csv = output_csv + self.output_npy = output_npy + self.num_beams = num_beams + self.model_config = None + self.gpu_device = torch.device("cuda", gpu_id) + + def get_model(self): + # --load the tokenizer and engines # + tokenizer = AutoTokenizer.from_pretrained( + self.tokenizer_dir, + legacy=False, + trust_remote_code=True, + ) + processor = AutoProcessor.from_pretrained(self.tokenizer_dir, + trust_remote_code=True) + config_path = os.path.join(self.engine_dir, "config.json") + with open(config_path, "r") as f: + config = json.load(f) + self.max_seq_len = config["build_config"]["max_seq_len"] + assert self.max_seq_len > 0, "max_seq_len must be positive" + + gen_config_path = os.path.join(self.tokenizer_dir, + "generation_config.json") + with open(gen_config_path, "r") as f: + gen_config = json.load(f) + top_k = gen_config["top_k"] + top_p = gen_config["top_p"] + eos_token_id = tokenizer.pad_token_id + pad_token_id = tokenizer.pad_token_id + + use_gpt_attention_plugin = config["build_config"]["plugin_config"][ + "gpt_attention_plugin"] + remove_input_padding = config["build_config"]["plugin_config"][ + "remove_input_padding"] + dtype = config["pretrained_config"]["dtype"] + tp_size = config["pretrained_config"]["mapping"]["tp_size"] + pp_size = config["pretrained_config"]["mapping"]["pp_size"] + world_size = tp_size * pp_size + assert ( + world_size == tensorrt_llm.mpi_world_size() + ), f"Engine world size ({world_size}) != Runtime world size ({tensorrt_llm.mpi_world_size()})" + num_heads = config["pretrained_config"][ + "num_attention_heads"] // world_size + max_batch_size = config["build_config"]["max_batch_size"] + hidden_size = config["pretrained_config"]["hidden_size"] // world_size + vocab_size = config["pretrained_config"]["vocab_size"] + num_layers = config["pretrained_config"]["num_hidden_layers"] + num_kv_heads = config["pretrained_config"].get("num_key_value_heads", + num_heads) + if "kv_cache_type" in config["build_config"]: + kv_cache_type = KVCacheType(config["build_config"]["kv_cache_type"]) + else: + kv_cache_type = KVCacheType.CONTINUOUS + + tokens_per_block = config["build_config"]["plugin_config"][ + "tokens_per_block"] + max_prompt_embedding_table_size = config["build_config"].get( + "max_prompt_embedding_table_size", 0) + quant_mode = QuantMode.from_quant_algo( + config["pretrained_config"]["quantization"]["quant_algo"], + config["pretrained_config"]["quantization"]["kv_cache_quant_algo"], + ) + if config["pretrained_config"].get("multi_query_mode", False): + tensorrt_llm.logger.warning( + "`multi_query_mode` config is deprecated. Please rebuild the engine." + ) + num_kv_heads = 1 + + runtime_rank = tensorrt_llm.mpi_rank() + runtime_mapping = tensorrt_llm.Mapping(world_size=world_size, + rank=runtime_rank, + tp_size=tp_size, + pp_size=pp_size) + + model_config = ModelConfig( + max_batch_size=max_batch_size, + num_heads=num_heads, + num_kv_heads=num_kv_heads, + hidden_size=hidden_size, + vocab_size=vocab_size, + num_layers=num_layers, + gpt_attention_plugin=use_gpt_attention_plugin, + kv_cache_type=kv_cache_type, + tokens_per_block=tokens_per_block, + remove_input_padding=remove_input_padding, + dtype=dtype, + quant_mode=quant_mode, + max_prompt_embedding_table_size=max_prompt_embedding_table_size, + max_beam_width=self.num_beams, + ) + sampling_config = SamplingConfig( + end_id=eos_token_id, + pad_id=pad_token_id, + num_beams=self.num_beams, + top_k=top_k, + top_p=top_p, + temperature=1.0, + ) + + engine_name = get_engine_name(runtime_rank) + serialize_path = os.path.join(self.engine_dir, engine_name) + print(f"Loading engine from {serialize_path}") + return ( + model_config, + sampling_config, + runtime_mapping, + runtime_rank, + serialize_path, + tokenizer, + processor, + eos_token_id, + pad_token_id, + ) + + def qwen_model_init(self, args): + logger.info(f"Loading audio engine from {self.audio_engine_path}") + with open(self.audio_engine_path, "rb") as f: + engine_buffer = f.read() + logger.info(f"Creating session from engine {self.audio_engine_path}") + self.session_audio = Session.from_serialized_engine(engine_buffer) + + self.config, _ = AutoConfig.from_pretrained( + self.tokenizer_dir, + return_unused_kwargs=True, + trust_remote_code=True, + ) + + ( + model_config, + sampling_config, + runtime_mapping, + runtime_rank, + serialize_path, + tokenizer, + processor, + eos_token_id, + pad_token_id, + ) = self.get_model() + runner_cls = ModelRunner if args.use_py_session else ModelRunnerCpp + runner_kwargs = dict( + engine_dir=args.engine_dir, + lora_dir=args.lora_dir, + rank=runtime_rank, + debug_mode=args.debug_mode, + lora_ckpt_source=args.lora_ckpt_source, + gpu_weights_percent=args.gpu_weights_percent, + max_output_len=args.max_new_tokens, + ) + if not args.use_py_session: + runner_kwargs.update( + is_enc_dec=False, + max_batch_size=model_config.max_batch_size, + max_input_len=self.max_seq_len - args.max_new_tokens, + max_beam_width=model_config.max_beam_width, + max_attention_window_size=args.max_attention_window_size, + sink_token_length=args.sink_token_length, + max_tokens_in_paged_kv_cache=args.max_tokens_in_paged_kv_cache, + kv_cache_enable_block_reuse=args.kv_cache_enable_block_reuse, + kv_cache_free_gpu_memory_fraction=args. + kv_cache_free_gpu_memory_fraction, + cross_kv_cache_fraction=None, + enable_chunked_context=args.enable_chunked_context, + multi_block_mode=args.multi_block_mode, + cuda_graph_mode=args.cuda_graph_mode, + device_ids=[args.gpu_id]) + runner_kwargs.update( + enable_context_fmha_fp32_acc=args.enable_context_fmha_fp32_acc) + self.runner = runner_cls.from_dir(**runner_kwargs) + self.tokenizer = tokenizer + self.processor = processor + self.sampling_config = sampling_config + self.model_config = model_config + + def ptuning_setup(self, prompt_table, dtype, hidden_size, tasks, input_ids): + if prompt_table is not None: + task_vocab_size = torch.tensor([prompt_table.shape[0]], + dtype=torch.int32, + device=self.gpu_device) + prompt_table = prompt_table.to( + dtype=tensorrt_llm._utils.str_dtype_to_torch(dtype), + device=self.gpu_device) + else: + prompt_table = torch.empty([1, hidden_size], device=self.gpu_device) + task_vocab_size = torch.zeros([1], device=self.gpu_device) + + if tasks is not None: + tasks = torch.tensor([int(t) for t in tasks.split(",")], + dtype=torch.int32, + device=self.gpu_device) + assert (tasks.shape[0] == input_ids.shape[0] + ), "Number of supplied tasks must match input batch size" + else: + tasks = torch.zeros([input_ids.size(0)], + dtype=torch.int32, + device=self.gpu_device) + + return [prompt_table, tasks, task_vocab_size] + + def build_user_input(self, audio=None, text=None): + assert isinstance(audio, str) or isinstance( + text, str), "audio or text must be provided as user input" + content = [] + if audio: + content.append({'type': 'audio', 'audio_url': audio}) + if text: + content.append({'type': 'text', 'text': text}) + user_input = {'role': 'user', 'content': content} + return user_input + + def get_raw_audios(self, audio_url): + audios = [] + for url in audio_url: + if os.path.isfile(url): + audio_data, _ = librosa.load( + url, sr=self.processor.feature_extractor.sampling_rate) + else: + audio_data, _ = librosa.load( + BytesIO(urlopen(url).read()), + sr=self.processor.feature_extractor.sampling_rate) + audios.append(audio_data) + return audios + + def audio_tower(self, audios, mask, stream, run_time=1): + audios = audios.to(self.gpu_device) + mask = mask.to(self.gpu_device) + audio_inputs = {"input": audios.float(), "mask": mask} + audio_output_info = self.session_audio.infer_shapes([ + TensorInfo("input", trt.DataType.FLOAT, audios.shape), + TensorInfo("mask", trt.DataType.HALF, mask.shape) + ]) + audio_outputs = { + t.name: + torch.empty(tuple(t.shape), + dtype=trt_dtype_to_torch(t.dtype), + device=self.gpu_device) + for t in audio_output_info + } + profiler.start("Audio") + for _ in range(run_time): + ok = self.session_audio.run(audio_inputs, audio_outputs, + stream.cuda_stream) + stream.synchronize() + audio_time = profiler.stop("Audio") / run_time + logger.info(f"TensorRT LLM Audio latency: {audio_time:3f} sec ") + + assert ok, "Runtime execution failed for audio session" + + audio_features = audio_outputs["output"] + + return audio_features + + def generate_for_qwen_audio( + self, + input_tokens, + args, + prompt_table=None, + extra_ids=None, + run_time=1, + ): + input_ids = torch.as_tensor(input_tokens, + device=self.gpu_device, + dtype=torch.int32) + input_lengths = torch.tensor([input_ids.size(1)], + device=self.gpu_device, + dtype=torch.int32) + max_input_length = torch.max(input_lengths).item() + max_new_tokens = min(args.max_new_tokens, + self.max_seq_len - max_input_length) + + prompt_table = prompt_table.unsqueeze(0) + profiler.start("QWen") + for _ in range(run_time): + outputs = self.runner.generate( + batch_input_ids=input_ids, + max_new_tokens=max_new_tokens, + max_attention_window_size=args.max_attention_window_size, + sink_token_length=args.sink_token_length, + end_id=self.sampling_config.end_id, + pad_id=self.sampling_config.pad_id, + temperature=args.temperature, + top_k=args.top_k, + top_p=args.top_p, + num_beams=args.num_beams, + num_return_sequences=args.num_return_sequences, + length_penalty=args.length_penalty, + early_stopping=args.early_stopping, + repetition_penalty=args.repetition_penalty, + presence_penalty=args.presence_penalty, + frequency_penalty=args.frequency_penalty, + stop_words_list=[[[151643], [151645]]], + bad_words_list=self.sampling_config.bad_words_list, + random_seed=args.random_seed, + lora_uids=args.lora_task_uids, + prompt_table=prompt_table, + prompt_tasks="0", + output_sequence_lengths=True, + no_repeat_ngram_size=args.no_repeat_ngram_size, + return_dict=True, + return_all_generated_tokens=False, + input_token_extra_ids=extra_ids) + output_ids = outputs['output_ids'] + torch.cuda.synchronize() + Qwen_time = profiler.stop("QWen") / run_time + + return output_ids, Qwen_time + + def get_feat_extract_output_lengths(self, input_lengths: torch.LongTensor): + """ + Computes the output length of the convolutional layers and the output length of the audio encoder + """ + input_lengths = (input_lengths - 1) // 2 + 1 + output_lengths = (input_lengths - 2) // 2 + 1 + return input_lengths, output_lengths + + def qwen_infer(self, + input_text, + audios, + audio_ids, + args, + stream, + history=None, + past_audio_features=None, + run_time=1): + assert input_text, "input_text must be provided" + assert torch.cuda.is_available(), "no gpu available" + # preprocess on CPU maybe faster + device = torch.device("cpu") + if isinstance(history, list): + history.append(input_text) + full_text = self.processor.apply_chat_template( + history, add_generation_prompt=True, tokenize=False) + else: + full_text = input_text + inputs = self.processor( + text=full_text, + audios=audios, + return_tensors="pt", + padding=True, + sampling_rate=self.processor.feature_extractor.sampling_rate) + inputs = inputs.to(device) + input_ids = inputs.input_ids + + if hasattr(inputs, + 'input_features') and inputs.input_features is not None: + # audio tower + batch_size, _, max_mel_seq_len = inputs.input_features.shape + feature_attention_mask = inputs.feature_attention_mask + + audio_feat_lengths, num_audio_tokens = self.get_feat_extract_output_lengths( + feature_attention_mask.sum(-1)) + + max_seq_len = (max_mel_seq_len - 2) // 2 + 1 + # Create a sequence tensor of shape (batch_size, max_seq_len) + seq_range = (torch.arange(0, + max_seq_len, + dtype=audio_feat_lengths.dtype, + device=device).unsqueeze(0).expand( + batch_size, max_seq_len)) + lengths_expand = audio_feat_lengths.unsqueeze(1).expand( + batch_size, max_seq_len) + # Create mask + padding_mask = seq_range >= lengths_expand + + audio_attention_mask_ = padding_mask.view( + batch_size, 1, 1, max_seq_len).expand(batch_size, 1, + max_seq_len, max_seq_len) + audio_attention_mask = audio_attention_mask_.to(dtype=torch.float16, + device=device) + audio_attention_mask[audio_attention_mask_] = float("-inf") + + audio_features = self.audio_tower(inputs.input_features, + audio_attention_mask, stream, + run_time) + + # merge audio features and input ids + num_audios, max_audio_tokens, embed_dim = audio_features.shape + audio_features_mask = torch.arange( + max_audio_tokens, device=device).expand( + num_audios, + max_audio_tokens) < num_audio_tokens.unsqueeze(1) + masked_audio_features = audio_features[audio_features_mask].view( + -1, embed_dim) + batch_size, _ = input_ids.shape + + # 1. Create a mask to know where special audio tokens are + special_audio_token_mask = input_ids == self.config.audio_token_index + special_audio_token_num = special_audio_token_mask.sum().item() + if past_audio_features is not None: + assert isinstance(past_audio_features, + list), f'past_audio_features should be a list' + assert ( + special_audio_token_num == len(past_audio_features) + + num_audios + ), f'special_audio_token_num {special_audio_token_num} should be equal to len(past_audio_features) + num_audios ({len(past_audio_features)} + {num_audios})' + # split to get current audio features + cur_audio_features = torch.split(masked_audio_features, + num_audio_tokens.tolist()) + if len(past_audio_features) > 0: + # concat past and current audio features + masked_audio_features = torch.cat( + (torch.cat(past_audio_features).to( + masked_audio_features.device), + masked_audio_features)) + # get past audio tokens number + past_num_audio_tokens = torch.tensor([ + past_feat.size(0) for past_feat in past_audio_features + ]) + # concat past and current audio tokens number + num_audio_tokens = torch.cat( + (past_num_audio_tokens.to(num_audio_tokens.device), + num_audio_tokens)) + # extend past audio features, cache them in CPU memory + past_audio_features.extend( + [cur_feat.cpu() for cur_feat in cur_audio_features]) + + batch_indices, non_audio_indices = torch.where( + input_ids != self.config.audio_token_index) + + # 2. Fill the final input ids based on the mask. + batch_indices, audio_indices = torch.where( + input_ids == self.config.audio_token_index) + + vocab_size = self.config.vocab_size + fake_prompt_id = torch.arange(vocab_size, + vocab_size + num_audio_tokens.sum(), + device=device) + + input_ids[batch_indices, audio_indices] = fake_prompt_id + input_lengths = torch.tensor(input_ids.size(1), + dtype=torch.int32, + device=self.gpu_device) + dtype = self.model_config.dtype + prompt_table, tasks, task_vocab_size = self.ptuning_setup( + masked_audio_features, dtype, embed_dim, None, input_ids) + + # build extra ids + assert isinstance(audio_ids, list), "audio_ids must be a list" + assert ( + len(audio_ids) == num_audio_tokens.size(0) + ), f"audio_ids length doesn't match with num_audio_tokens ({len(audio_ids)} != {num_audio_tokens.size(0)})" + for i in audio_ids: + assert isinstance( + i, int + ) and i > 0, "audio_id should be an integer greater than 0" + extra_ids = torch.zeros_like(input_ids, + dtype=torch.int64, + device=device) + seq_extra_ids = torch.cat([ + torch.full((n, ), audio_ids[i], dtype=torch.int64) + for i, n in enumerate(num_audio_tokens) + ]).to(device) + extra_ids[batch_indices, audio_indices] = seq_extra_ids + extra_ids = extra_ids.tolist() + else: + input_ids = input_ids.to(dtype=torch.int32, device=self.gpu_device) + input_lengths = torch.tensor(input_ids.size(1), + dtype=torch.int32, + device=self.gpu_device) + dtype = self.model_config.dtype + prompt_table, tasks, task_vocab_size = self.ptuning_setup( + None, dtype, self.model_config.hidden_size, None, input_ids) + extra_ids = torch.zeros_like(input_ids, dtype=torch.int64).tolist() + + # print(f"extra_ids: {extra_ids}") + output_ids, Qwen_time = self.generate_for_qwen_audio( + input_ids, args, prompt_table, extra_ids, run_time) + + runtime_rank = tensorrt_llm.mpi_rank() + input_lengths = torch.tensor([input_ids.size(1)], + device=self.gpu_device, + dtype=torch.int32) + effective_output_token = 0 + if runtime_rank == 0: + if self.output_csv is None and self.output_npy is None: + for b in range(input_lengths.size(0)): + inputs = input_ids[b] + if self.num_beams <= 1: + outputs = output_ids[b][0, len(inputs):].tolist() + try: + effective_output_token = (effective_output_token + + outputs.index(151643)) + except: + effective_output_token = 1 + output_text = self.tokenizer.decode( + outputs, skip_special_tokens=True) + print(f'Output: "{output_text}"') + else: + for beam in range(self.num_beams): + outputs = output_ids[b][beam, len(inputs):].tolist() + output_text = self.tokenizer.decode( + outputs, skip_special_tokens=True) + print(f'Output(beam: {beam}): "{output_text}"') + logger.info(f"Input length={input_lengths[b]}") + logger.info(f"Output length={output_ids.shape}") + logger.info(f"TensorRT LLM QWen time: {Qwen_time:3f} sec ") + if isinstance(history, list): + history.append({'role': 'assistant', 'content': output_text}) + return output_text, past_audio_features + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument("--max_new_tokens", type=int, default=10) + parser.add_argument( + "--audio_engine_path", + type=str, + default="plan/audio_encoder/audio_encoder_fp16.plan", + ) + parser.add_argument( + "--input_text", + type=str, + default= + "<|audio_bos|><|AUDIO|><|audio_eos|>Generate the caption in English:") + parser.add_argument( + "--audio_url", + nargs="+", + type=str, + default=["./audio/glass-breaking-151256.mp3"], + ) + parser.add_argument( + "--input_tokens", + dest="input_file", + type=str, + help= + "CSV or Numpy file containing tokenized input. Alternative to text input.", + default=None, + ) + parser.add_argument( + "--output_csv", + type=str, + help="CSV file where the tokenized output is stored.", + default=None, + ) + parser.add_argument( + "--output_npy", + type=str, + help="Numpy file where the tokenized output is stored.", + default=None, + ) + parser.add_argument( + "--gpu_id", + type=int, + help= + "Specify GPU device index for running. Should be the index seen by torch, not original index", + default=0, + ) + parser = add_common_args(parser) + + return parser.parse_args() + + +if __name__ == "__main__": + args = parse_arguments() + tensorrt_llm.logger.set_level(args.log_level) + + # use cudaSetDevice before loading audio engine + torch.cuda.set_device(args.gpu_id) + qinfer = QWenInfer(args.audio_engine_path, args.tokenizer_dir, + args.engine_dir, args.log_level, args.output_csv, + args.output_npy, args.num_beams, args.gpu_id) + qinfer.qwen_model_init(args) + + audios = qinfer.get_raw_audios(args.audio_url) + gpu_device = torch.device("cuda", args.gpu_id) + stream = torch.cuda.current_stream(device=gpu_device) + qinfer.qwen_infer(args.input_text, audios, [1], args, stream, None, None, 1) diff --git a/examples/models/core/qwen2audio/run_chat.py b/examples/models/core/qwen2audio/run_chat.py new file mode 100644 index 000000000000..00d58cdc862b --- /dev/null +++ b/examples/models/core/qwen2audio/run_chat.py @@ -0,0 +1,81 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# isort: off +import torch +from run import QWenInfer, parse_arguments + +import tensorrt_llm +# isort: on + +if __name__ == '__main__': + args = parse_arguments() + stream = torch.cuda.current_stream() + tensorrt_llm.logger.set_level(args.log_level) + qinfer = QWenInfer( + args.audio_engine_path, + args.tokenizer_dir, + args.engine_dir, + args.log_level, + args.output_csv, + args.output_npy, + args.num_beams, + ) + qinfer.qwen_model_init(args) + + run_i = 0 + history = [] + audios = None + global_audio_id = 1 + audio_ids = [] + + while True: + input_text = None + try: + input_text = input( + "Text (type 'q' to quit, or 'audio_url:[url]' to input audio): " + ) + except: + continue + + if input_text == "clear history": + history = [] + audios = None + continue + + if input_text.lower() == 'q': + break + print('\n') + + if input_text.startswith('audio_url:'): + audio_url = input_text[len('audio_url:'):].strip() + if isinstance(audios, list): + audios.extend(qinfer.get_raw_audios([audio_url])) + else: + audios = qinfer.get_raw_audios([audio_url]) + user_input = qinfer.build_user_input(audio=audio_url) + audio_ids.append(global_audio_id) + global_audio_id += 1 + else: + user_input = qinfer.build_user_input(text=input_text) + + qinfer.qwen_infer( + user_input, + audios, + audio_ids, + args, + stream, + history, + ) diff --git a/examples/models/core/qwen2audio/utils.py b/examples/models/core/qwen2audio/utils.py new file mode 100644 index 000000000000..3252beebbf7d --- /dev/null +++ b/examples/models/core/qwen2audio/utils.py @@ -0,0 +1,130 @@ +from argparse import BooleanOptionalAction + + +def add_common_args(parser): + # sampling arguments + parser.add_argument('--num_beams', + type=int, + help="Use beam search if num_beams > 1", + default=1) + parser.add_argument('--num_return_sequences', + type=int, + help="Number of sequences to generate for each input.", + default=None) + parser.add_argument('--temperature', type=float, default=1.0) + parser.add_argument('--top_k', type=int, default=1) + parser.add_argument('--top_p', type=float, default=0.0) + parser.add_argument('--length_penalty', type=float, default=1.0) + parser.add_argument('--repetition_penalty', type=float, default=1.0) + parser.add_argument('--presence_penalty', type=float, default=0.0) + parser.add_argument('--frequency_penalty', type=float, default=0.0) + parser.add_argument('--random_seed', type=int, default=0) + parser.add_argument('--early_stopping', + type=int, + help='Use early stopping if num_beams > 1, ' + '1 for early-stopping, 0 for non-early-stopping' + 'other values for stopping by length', + default=1) + parser.add_argument('--no_repeat_ngram_size', type=int, default=None) + + # common runtime arguments + parser.add_argument('--sink_token_length', + type=int, + default=None, + help='The sink token length.') + parser.add_argument( + '--max_attention_window_size', + type=int, + default=None, + nargs="+", + help= + 'The attention window size that controls the sliding window attention kv cache behavior' + ) + parser.add_argument( + '--multi_block_mode', + type=lambda s: s.lower() in + ("yes", "true", "t", "1" + ), # custom boolean function to convert input string to boolean + default=True, + help= + "Distribute the work across multiple CUDA thread-blocks on the GPU for masked MHA kernel." + ) + parser.add_argument('--enable_context_fmha_fp32_acc', + action='store_true', + help="Enable FMHA runner FP32 accumulation.") + parser.add_argument('--cuda_graph_mode', + action='store_true', + help="Enable cuda graphs in the inference.") + parser.add_argument( + '--log_level', + type=str, + choices=['verbose', 'info', 'warning', 'error', 'internal_error'], + default='info') + parser.add_argument('--use_py_session', + default=False, + action='store_true', + help="Whether or not to use Python runtime session") + parser.add_argument('--debug_mode', + default=False, + action='store_true', + help="Whether or not to turn on the debug mode") + parser.add_argument('--lora_dir', + type=str, + default=None, + nargs="+", + help="The directory of LoRA weights") + parser.add_argument('--lora_ckpt_source', + type=str, + default="hf", + choices=["hf", "nemo"], + help="The source of lora checkpoint.") + parser.add_argument( + '--lora_task_uids', + type=str, + default=None, + nargs="+", + help="The list of LoRA task uids; use -1 to disable the LoRA module") + + # model arguments + parser.add_argument('--engine_dir', type=str, default='engine_outputs') + parser.add_argument('--hf_model_dir', '--model_dir', type=str, default=None) + parser.add_argument( + '--tokenizer_dir', + default=None, + help='tokenizer path; defaults to hf_model_dir if left unspecified') + + # memory argument + parser.add_argument( + '--gpu_weights_percent', + default=1, + type=float, + help= + 'Specify the percentage of weights that reside on GPU instead of CPU and streaming load during runtime.', + ) + parser.add_argument( + '--max_tokens_in_paged_kv_cache', + default=None, + type=int, + help= + 'Specify the maximum number of tokens in a kv cache page (only available with cpp session).', + ) + parser.add_argument( + '--kv_cache_enable_block_reuse', + default=True, + action=BooleanOptionalAction, + help= + 'Enables block reuse in kv cache (only available with cpp session).', + ) + parser.add_argument( + '--kv_cache_free_gpu_memory_fraction', + default=0.9, + type=float, + help='Specify the free gpu memory fraction.', + ) + parser.add_argument( + '--enable_chunked_context', + action='store_true', + help='Enables chunked context (only available with cpp session).', + ) + + return parser diff --git a/examples/models/core/qwenvl/README.md b/examples/models/core/qwenvl/README.md new file mode 100644 index 000000000000..27fc5d1dfce0 --- /dev/null +++ b/examples/models/core/qwenvl/README.md @@ -0,0 +1,114 @@ +# Guide to Qwen-VL deployment pipeline + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +1. Download the Qwen vision-language model (Qwen-VL). + ```bash + git lfs install + git clone https://huggingface.co/Qwen/Qwen-VL-Chat + ``` +2. Generate the Vision Transformer (ViT) ONNX model and the TensorRT engine. +- If you don't have ONNX file, run: + ```bash + python3 vit_onnx_trt.py --pretrained_model_path ./Qwen-VL-Chat + ``` + The ONNX and TensorRT engine will be generated under `./onnx/visual_encoder` and `./plan/visual_encoder` respectively. + +- If you already have an ONNX file under `./onnx/visual_encoder` and want to build a TensorRT engine with it, run: + ```bash + python3 vit_onnx_trt.py --pretrained_model_path ./Qwen-VL-Chat --only_trt + ``` + This command saves the test image tensor to `image.pt` for later pipeline inference. + +3. Build Qwen TensorRT engine. +- Convert checkpoint + 1. Install packages + ```bash + pip install -r requirements.txt + ``` + 2. Convert + ```bash + python3 ./examples/models/core/qwen/convert_checkpoint.py --model_dir=./Qwen-VL-Chat \ + --output_dir=./tllm_checkpoint_1gpu \ + --dtype float16 + ``` + +- Build TensorRT LLM engine + + NOTE: `max_prompt_embedding_table_size = query_token_num * max_batch_size`, therefore, if you change `max_batch_size`, `--max_prompt_embedding_table_size` must be reset accordingly. + ```bash + trtllm-build --checkpoint_dir=./tllm_checkpoint_1gpu \ + --gemm_plugin=float16 --gpt_attention_plugin=float16 \ + --max_input_len=2048 --max_seq_len=3072 \ + --max_batch_size=8 --max_prompt_embedding_table_size=2048 \ + --remove_input_padding=enable \ + --output_dir=./trt_engines/Qwen-VL-7B-Chat + ``` + The built Qwen engines are located in `./trt_engines/Qwen-VL-7B-Chat`. + For more information about Qwen, refer to the README.md in [`example/qwen`](../qwen). + +4. Assemble everything into the Qwen-VL pipeline. + + 4.1 Run with INT4 GPTQ weight-only quantization engine + ```bash + python3 run.py \ + --tokenizer_dir=./Qwen-VL-Chat \ + --qwen_engine_dir=./trt_engines/Qwen-VL-7B-Chat \ + --vit_engine_path=./plan/visual_encoder/visual_encoder_fp16.plan \ + --images_path='{"image": "./pics/demo.jpeg"}' + ``` + 4.2 (Optional) For multiple rounds of dialogue, you can run: + ```bash + python3 run_chat.py \ + --tokenizer_dir=./Qwen-VL-Chat \ + --qwen_engine_dir=./trt_engines/Qwen-VL-7B-Chat \ + --vit_engine_path=./plan/visual_encoder/visual_encoder_fp16.plan \ + --images_path='{"image": "./pics/demo.jpeg"}' + ``` + 4.3 (Optional) To show the bounding box result in the demo picture, install OpenCV, ZMQ, and request: + ```bash + pip install opencv-python==4.5.5.64 + pip install opencv-python-headless==4.5.5.64 + pip install zmq + pip install request + ``` + +   4.3.1 If the current program is executed on a remote machine, run the following command on a local machine: + + ```bash + python3 show_pic.py --ip=127.0.0.1 --port=8006 + ``` + +   Replace the `ip` and `port` values, where `ip` is your remote machine IP address. + +   Run the following command on the remote machine: + + ```bash + python3 run_chat.py \ + --tokenizer_dir=./Qwen-VL-Chat \ + --qwen_engine_dir=./trt_engines/Qwen-VL-7B-Chat \ + --vit_engine_path=./plan/visual_encoder/visual_encoder_fp16.plan \ + --display \ + --port=8006 + ``` + +   Replace the `port` value. + +   4.3.2 If the current program is executed on the local machine, run the following command: + + ```bash + python3 run_chat.py \ + --tokenizer_dir=./Qwen-VL-Chat \ + --qwen_engine_dir=./trt_engines/Qwen-VL-7B-Chat \ + --vit_engine_path=./plan/visual_encoder/visual_encoder_fp16.plan \ + --display \ + --local_machine + ``` + +   The question "Print the bounding box of the girl" is displayed. You should see the following image: + + ![image](./pics/1.png) diff --git a/examples/models/core/qwenvl/pics/1.png b/examples/models/core/qwenvl/pics/1.png new file mode 100644 index 000000000000..09e872ac90da Binary files /dev/null and b/examples/models/core/qwenvl/pics/1.png differ diff --git a/examples/models/core/qwenvl/pics/demo.jpeg b/examples/models/core/qwenvl/pics/demo.jpeg new file mode 100644 index 000000000000..9fdc04005062 Binary files /dev/null and b/examples/models/core/qwenvl/pics/demo.jpeg differ diff --git a/examples/models/core/qwenvl/requirements.txt b/examples/models/core/qwenvl/requirements.txt new file mode 100644 index 000000000000..9257ddb9545a --- /dev/null +++ b/examples/models/core/qwenvl/requirements.txt @@ -0,0 +1,11 @@ +-c ../../../constraints.txt +tensorrt_llm>=0.0.0.dev0 +datasets==3.1.0 +evaluate +rouge_score +transformers-stream-generator +sentencepiece>=0.1.99 +tiktoken +einops +matplotlib +torchvision diff --git a/examples/models/core/qwenvl/run.py b/examples/models/core/qwenvl/run.py new file mode 100644 index 000000000000..c996b20fd643 --- /dev/null +++ b/examples/models/core/qwenvl/run.py @@ -0,0 +1,549 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import argparse +import json +import os +from typing import List, Tuple + +import tensorrt as trt +import torch +from transformers import AutoConfig, AutoTokenizer +from vit_onnx_trt import Preprocss + +import tensorrt_llm +import tensorrt_llm.profiler as profiler +from tensorrt_llm import logger +from tensorrt_llm._deprecation import emit_engine_arch_deprecation +from tensorrt_llm.llmapi.kv_cache_type import KVCacheType +from tensorrt_llm.quantization import QuantMode +from tensorrt_llm.runtime import (ModelConfig, SamplingConfig, Session, + TensorInfo) + + +def get_engine_name(rank): + return "rank{}.engine".format(rank) + + +def trt_dtype_to_torch(dtype): + if dtype == trt.float16: + return torch.float16 + elif dtype == trt.float32: + return torch.float32 + elif dtype == trt.int32: + return torch.int32 + else: + raise TypeError("%s is not supported" % dtype) + + +class QWenInfer(object): + + def __init__( + self, + tokenizer_dir, + qwen_engine_dir, + log_level, + output_csv, + output_npy, + num_beams, + ): + self.tokenizer_dir = tokenizer_dir + self.qwen_engine_dir = qwen_engine_dir + self.log_level = log_level + self.global_max_input_len = 2048 + self.decoder = None + self.tokenizer = None + self.config = None + self.sampling_config = None + self.output_csv = output_csv + self.output_npy = output_npy + self.num_beams = num_beams + self.model_config = None + + def get_model(self): + # --load the tokenizer and engine # + tokenizer = AutoTokenizer.from_pretrained( + self.tokenizer_dir, + legacy=False, + trust_remote_code=True, + ) + config_path = os.path.join(self.qwen_engine_dir, "config.json") + with open(config_path, "r") as f: + config = json.load(f) + gen_config_path = os.path.join(self.tokenizer_dir, + "generation_config.json") + with open(gen_config_path, "r") as f: + gen_config = json.load(f) + top_k = gen_config["top_k"] + top_p = gen_config["top_p"] + chat_format = gen_config["chat_format"] + if chat_format == "raw": + eos_token_id = gen_config["eos_token_id"] + pad_token_id = gen_config["pad_token_id"] + elif chat_format == "chatml": + pad_token_id = eos_token_id = tokenizer.im_end_id + else: + raise Exception("unknown chat format ", chat_format) + + use_gpt_attention_plugin = config["build_config"]["plugin_config"][ + "gpt_attention_plugin"] + gemm_allreduce_plugin = config["build_config"]["plugin_config"][ + "gemm_allreduce_plugin"] + + remove_input_padding = config["build_config"]["plugin_config"][ + "remove_input_padding"] + dtype = config["pretrained_config"]["dtype"] + tp_size = config["pretrained_config"]["mapping"]["tp_size"] + pp_size = config["pretrained_config"]["mapping"]["pp_size"] + world_size = tp_size * pp_size + assert ( + world_size == tensorrt_llm.mpi_world_size() + ), f"Engine world size ({world_size}) != Runtime world size ({tensorrt_llm.mpi_world_size()})" + num_heads = config["pretrained_config"][ + "num_attention_heads"] // world_size + max_batch_size = config["build_config"]["max_batch_size"] + hidden_size = config["pretrained_config"]["hidden_size"] // world_size + vocab_size = config["pretrained_config"]["vocab_size"] + num_layers = config["pretrained_config"]["num_hidden_layers"] + num_kv_heads = config["pretrained_config"].get("num_key_value_heads", + num_heads) + if "kv_cache_type" in config["build_config"]: + kv_cache_type = KVCacheType(config["build_config"]["kv_cache_type"]) + else: + kv_cache_type = KVCacheType.CONTINUOUS + + tokens_per_block = config["build_config"]["plugin_config"][ + "tokens_per_block"] + max_prompt_embedding_table_size = config["build_config"].get( + "max_prompt_embedding_table_size", 0) + quant_mode = QuantMode.from_quant_algo( + config["pretrained_config"]["quantization"]["quant_algo"], + config["pretrained_config"]["quantization"]["kv_cache_quant_algo"], + ) + if config["pretrained_config"].get("multi_query_mode", False): + tensorrt_llm.logger.warning( + "`multi_query_mode` config is deprecated. Please rebuild the engine." + ) + num_kv_heads = 1 + + runtime_rank = tensorrt_llm.mpi_rank() + runtime_mapping = tensorrt_llm.Mapping(world_size=world_size, + rank=runtime_rank, + tp_size=tp_size, + pp_size=pp_size) + torch.cuda.set_device(runtime_rank % runtime_mapping.gpus_per_node) + + model_config = ModelConfig( + max_batch_size=max_batch_size, + num_heads=num_heads, + num_kv_heads=num_kv_heads, + hidden_size=hidden_size, + vocab_size=vocab_size, + num_layers=num_layers, + gpt_attention_plugin=use_gpt_attention_plugin, + gemm_allreduce_plugin=gemm_allreduce_plugin, + kv_cache_type=kv_cache_type, + tokens_per_block=tokens_per_block, + remove_input_padding=remove_input_padding, + dtype=dtype, + quant_mode=quant_mode, + max_prompt_embedding_table_size=max_prompt_embedding_table_size, + max_beam_width=self.num_beams, + ) + sampling_config = SamplingConfig( + end_id=eos_token_id, + pad_id=pad_token_id, + num_beams=self.num_beams, + top_k=top_k, + top_p=top_p, + temperature=1.0, + ) + + engine_name = get_engine_name(runtime_rank) + serialize_path = os.path.join(self.qwen_engine_dir, engine_name) + print(f"Loading engine from {serialize_path}") + return ( + model_config, + sampling_config, + runtime_mapping, + runtime_rank, + serialize_path, + tokenizer, + eos_token_id, + pad_token_id, + ) + + def qwen_model_init(self): + ( + model_config, + sampling_config, + runtime_mapping, + runtime_rank, + serialize_path, + tokenizer, + eos_token_id, + pad_token_id, + ) = self.get_model() + with open(serialize_path, "rb") as f: + engine_buffer = f.read() + self.decoder = tensorrt_llm.runtime.GenerationSession( + model_config, + engine_buffer, + runtime_mapping, + ) + self.tokenizer = tokenizer + self.sampling_config = sampling_config + self.model_config = model_config + self.config, _ = AutoConfig.from_pretrained( + self.tokenizer_dir, + return_unused_kwargs=True, + trust_remote_code=True, + ) + + def ptuning_setup(self, prompt_table, dtype, hidden_size, tasks, input_ids): + if prompt_table is not None: + task_vocab_size = torch.tensor([prompt_table.shape[1]], + dtype=torch.int32, + device="cuda") + prompt_table = prompt_table.view( + (prompt_table.shape[0] * prompt_table.shape[1], + prompt_table.shape[2])) + prompt_table = prompt_table.cuda().to( + dtype=tensorrt_llm._utils.str_dtype_to_torch(dtype)) + else: + prompt_table = torch.empty([1, hidden_size]).cuda() + task_vocab_size = torch.zeros([1]).cuda() + + if tasks is not None: + tasks = torch.tensor([int(t) for t in tasks.split(",")], + dtype=torch.int32, + device="cuda") + assert (tasks.shape[0] == input_ids.shape[0] + ), "Number of supplied tasks must match input batch size" + else: + tasks = torch.zeros([input_ids.size(0)], dtype=torch.int32).cuda() + + return [prompt_table, tasks, task_vocab_size] + + def make_context( + self, + query: str, + history: List[Tuple[str, str]] = None, + system: str = "You are a helpful assistant.", + max_window_size: int = 6144, + ): + if history is None: + history = [] + + im_start, im_end = "<|im_start|>", "<|im_end|>" + im_start_tokens = [self.tokenizer.im_start_id] # 151644 + im_end_tokens = [self.tokenizer.im_end_id] # [151645] + nl_tokens = self.tokenizer.encode("\n") + + def _tokenize_str(role, content): + return f"{role}\n{content}", self.tokenizer.encode( + role, allowed_special=set(self.tokenizer.IMAGE_ST) + ) + nl_tokens + self.tokenizer.encode( + content, allowed_special=set(self.tokenizer.IMAGE_ST)) + + system_text, system_tokens_part = _tokenize_str("system", system) + system_tokens = im_start_tokens + system_tokens_part + im_end_tokens + + raw_text = "" + context_tokens = [] + + for turn_query, turn_response in reversed(history): + query_text, query_tokens_part = _tokenize_str("user", turn_query) + query_tokens = im_start_tokens + query_tokens_part + im_end_tokens + if turn_response is not None: + response_text, response_tokens_part = _tokenize_str( + "assistant", turn_response) + response_tokens = im_start_tokens + response_tokens_part + im_end_tokens + + next_context_tokens = (nl_tokens + query_tokens + nl_tokens + + response_tokens) + prev_chat = f"\n{im_start}{query_text}{im_end}\n{im_start}{response_text}{im_end}" + else: + next_context_tokens = nl_tokens + query_tokens + nl_tokens + prev_chat = f"\n{im_start}{query_text}{im_end}\n" + + current_context_size = (len(system_tokens) + + len(next_context_tokens) + + len(context_tokens)) + if current_context_size < max_window_size: + context_tokens = next_context_tokens + context_tokens + raw_text = prev_chat + raw_text + else: + break + + context_tokens = system_tokens + context_tokens + raw_text = f"{im_start}{system_text}{im_end}" + raw_text + context_tokens += (nl_tokens + im_start_tokens + + _tokenize_str("user", query)[1] + im_end_tokens + + nl_tokens + im_start_tokens + + self.tokenizer.encode("assistant") + nl_tokens) + raw_text += f"\n{im_start}user\n{query}{im_end}\n{im_start}assistant\n" + + return raw_text, context_tokens + + def generate_for_qwenvl( + self, + input_tokens, + max_new_tokens: int, + prompt_table=None, + tasks=None, + task_vocab_size=None, + num_beams=1, + ): + input_ids = None + input_lengths = None + input_ids = torch.as_tensor(input_tokens, + device="cuda", + dtype=torch.int32) + input_lengths = torch.tensor([input_ids.size(1)], + device="cuda", + dtype=torch.int32) + max_input_length = torch.max(input_lengths).item() + max_new_tokens = min(max_new_tokens, + self.global_max_input_len - max_input_length) + + profiler.start("QWen") + run_time = 10 + for _ in range(run_time): + self.decoder.setup( + batch_size=input_lengths.size(0), + max_context_length=max_input_length, + max_new_tokens=max_new_tokens, + beam_width=num_beams, + ) + output_ids = self.decoder.decode( + input_ids, + input_lengths, + self.sampling_config, + prompt_table, + tasks, + task_vocab_size, + ) + torch.cuda.synchronize() + profiler.stop("QWen") + Qwen_time = profiler.elapsed_time_in_sec("QWen") / run_time + + return output_ids, Qwen_time + + def qwen_infer( + self, + input_vit, + images_path, + input_text, + max_new_tokens, + num_beams=1, + history=None, + ): + if images_path is None: + content_list = [] + else: + content_list = images_path + if history is None: + history = [] + content_list.append({"text": input_text}) + query = self.tokenizer.from_list_format(content_list) + raw_text, context_tokens = self.make_context(query, history=history) + # context_tokens = self.tokenizer.encode(query) + input_ids = torch.tensor([context_tokens]).to("cuda") + bos_pos = torch.where(input_ids == self.config.visual["image_start_id"]) + eos_pos = torch.where( + input_ids == self.config.visual["image_start_id"] + 1) + assert (bos_pos[0] == eos_pos[0]).all() + img_pos = torch.stack((bos_pos[0], bos_pos[1], eos_pos[1]), dim=1) + vocab_size = self.config.vocab_size + fake_prompt_id = torch.arange( + vocab_size, + vocab_size + input_vit.shape[0] * input_vit.shape[1], + device="cuda", + ) + fake_prompt_id = fake_prompt_id.reshape(input_vit.shape[0], + input_vit.shape[1]) + for idx, (i, a, b) in enumerate(img_pos): + input_ids[i][a + 1:b] = fake_prompt_id[idx] + input_ids = input_ids.contiguous().to(torch.int32).cuda() + input_lengths = torch.tensor(input_ids.size(1), + dtype=torch.int32).cuda() + dtype = self.model_config.dtype + prompt_table, tasks, task_vocab_size = self.ptuning_setup( + input_vit, dtype, self.config.hidden_size, None, input_ids) + + output_ids, Qwen_time = self.generate_for_qwenvl( + input_ids, max_new_tokens, prompt_table, tasks, task_vocab_size, + num_beams) + + runtime_rank = tensorrt_llm.mpi_rank() + input_lengths = torch.tensor([input_ids.size(1)], + device="cuda", + dtype=torch.int32) + effective_output_token = 0 + if runtime_rank == 0: + if self.output_csv is None and self.output_npy is None: + for b in range(input_lengths.size(0)): + inputs = input_ids[b] + if content_list is not None: + print(f'Input: "{content_list}"') + print("\n") + if self.num_beams <= 1: + outputs = output_ids[b][0, len(inputs):].tolist() + try: + effective_output_token = (effective_output_token + + outputs.index(151643)) + except: + effective_output_token = 1 + output_text = self.tokenizer.decode( + outputs, skip_special_tokens=True) + print(f'Output: "{output_text}"') + print("\n") + else: + for beam in range(self.num_beams): + outputs = output_ids[b][beam, len(inputs):].tolist() + output_text = self.tokenizer.decode( + outputs, skip_special_tokens=True) + print(f'Output(beam: {beam}): "{output_text}"') + logger.info(f"Input length={input_lengths[b]}") + logger.info(f"Output length={output_ids.shape}") + logger.info(f"TensorRT LLM QWen time: {Qwen_time:3f} sec ") + history.append((query, output_text)) + return output_text + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument("--max_new_tokens", type=int, default=200) + parser.add_argument("--log_level", type=str, default="info") + parser.add_argument( + "--vit_engine_path", + type=str, + default="plan/visual_encoder/visual_encoder_fp16.plan", + ) + parser.add_argument( + "--qwen_engine_dir", + type=str, + default="qwen_outputs", + ) + parser.add_argument( + "--tokenizer_dir", + type=str, + default=".", + help="Directory containing the tokenizer.model.", + ) + parser.add_argument("--input_text", + type=str, + default="Describe the picture") + parser.add_argument( + "--images_path", + nargs="+", + type=json.loads, + default=[{ + "image": "./pics/demo.jpeg" + }], + ) + parser.add_argument( + "--input_tokens", + dest="input_file", + type=str, + help= + "CSV or Numpy file containing tokenized input. Alternative to text input.", + default=None, + ) + parser.add_argument( + "--output_csv", + type=str, + help="CSV file where the tokenized output is stored.", + default=None, + ) + parser.add_argument( + "--output_npy", + type=str, + help="Numpy file where the tokenized output is stored.", + default=None, + ) + parser.add_argument("--num_beams", + type=int, + help="Use beam search if num_beams >1", + default=1) + parser.add_argument("--display", default=False, action='store_true') + parser.add_argument('--port', type=str, default='8006') + parser.add_argument("--local_machine", default=False, action='store_true') + + return parser.parse_args() + + +def vit_process(image_path, vit_engine_path, stream): + img_processor = Preprocss(448) + logger.info(f"Loading engine from {vit_engine_path}") + with open(vit_engine_path, "rb") as f: + engine_buffer = f.read() + logger.info(f"Creating session from engine {vit_engine_path}") + session_vit = Session.from_serialized_engine(engine_buffer) + device = torch.device("cuda") if torch.cuda.is_available() else "cpu" + image_path_list = [] + for item in image_path: + image_path_list.append(next(iter(item.values()))) + images = img_processor.encode(image_path_list).to(device) + batch_size = images.size(0) + images = images.expand(batch_size, -1, -1, -1).contiguous() + visual_inputs = {"input": images.float()} + visual_output_info = session_vit.infer_shapes( + [TensorInfo("input", trt.DataType.FLOAT, images.shape)]) + visual_outputs = { + t.name: + torch.empty(tuple(t.shape), + dtype=trt_dtype_to_torch(t.dtype), + device="cuda") + for t in visual_output_info + } + profiler.start("ViT") + + run_time = 10 + for _ in range(run_time): + ok = session_vit.run(visual_inputs, visual_outputs, stream) + profiler.stop("ViT") + Vit_time = profiler.elapsed_time_in_sec("ViT") / run_time + logger.info(f"TensorRT LLM ViT latency: {Vit_time:3f} sec ") + + assert ok, "Runtime execution failed for vit session" + + image_embeds = visual_outputs["output"] + return image_embeds + + +if __name__ == "__main__": + emit_engine_arch_deprecation("run.py") + args = parse_arguments() + stream = torch.cuda.current_stream().cuda_stream + tensorrt_llm.logger.set_level(args.log_level) + image_embeds = vit_process(args.images_path, args.vit_engine_path, stream) + qinfer = QWenInfer( + args.tokenizer_dir, + args.qwen_engine_dir, + args.log_level, + args.output_csv, + args.output_npy, + args.num_beams, + ) + qinfer.qwen_model_init() + qinfer.qwen_infer( + image_embeds, + args.images_path, + args.input_text, + args.max_new_tokens, + args.num_beams, + history=[], + ) diff --git a/examples/models/core/qwenvl/run_chat.py b/examples/models/core/qwenvl/run_chat.py new file mode 100644 index 000000000000..1f1ba6fb6faf --- /dev/null +++ b/examples/models/core/qwenvl/run_chat.py @@ -0,0 +1,128 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import re + +# isort: off +import torch +from run import QWenInfer, parse_arguments, vit_process +# isort: on + + +def make_display(port=8006): + import cv2 + import zmq + context = zmq.Context() + socket = context.socket(zmq.REP) + socket.bind(f"tcp://*:{port}") + + def func(image): + data = cv2.imencode(".jpg", image)[1].tobytes() + socket.recv() + socket.send(data) + + return func + + +def show_pic(image_path, port): + import cv2 + image = cv2.imread(image_path) + display_obj = make_display(port) + display_obj(image) + + +def show_pic_local(image_path): + import cv2 + import matplotlib.pyplot as plt + image = cv2.imread(image_path) + image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) + + plt.imshow(image_rgb) + plt.pause(0.1) + + +def cooridinate_extract_show(input, history, tokenizer, local_machine, port): + pattern = r"\((\d+),(\d+)\)" + coordinates = re.findall(pattern, input) + result = "Box({},{})".format(coordinates[0][0], + coordinates[0][1]) + result += ",({},{})".format(coordinates[1][0], coordinates[1][1]) + + image = tokenizer.draw_bbox_on_latest_picture(result, history) + if image: + image.save('1.png') + if local_machine: + show_pic_local('1.png') + else: + show_pic('1.png', port) + else: + print("======No bounding boxes are detected!") + + +def exist_cooridinate(input): + pattern = r"\((\d+),(\d+)\)" + match = re.search(pattern, input) + if match: + return True + else: + return False + + +if __name__ == '__main__': + args = parse_arguments() + stream = torch.cuda.current_stream().cuda_stream + image_embeds = vit_process(args.images_path, args.vit_engine_path, stream) + qinfer = QWenInfer(args.tokenizer_dir, args.qwen_engine_dir, args.log_level, + args.output_csv, args.output_npy, args.num_beams) + qinfer.qwen_model_init() + + run_i = 0 + history = [] + if args.display: + if args.local_machine: + show_pic_local("./pics/demo.jpeg") + else: + show_pic("./pics/demo.jpeg", args.port) + + while True: + input_text = None + try: + input_text = input("Text (or 'q' to quit): ") + except: + continue + + if input_text == "clear history": + history = [] + continue + + if input_text.lower() == 'q': + break + print('\n') + + content_list = args.images_path + content_list.append({'text': input_text}) + + if run_i == 0: + query = qinfer.tokenizer.from_list_format(content_list) + else: + query = input_text + + run_i = run_i + 1 + output_text = qinfer.qwen_infer(image_embeds, None, query, + args.max_new_tokens, args.num_beams, + history) + if args.display: + if exist_cooridinate(output_text): + cooridinate_extract_show(output_text, history, qinfer.tokenizer, + args.local_machine, args.port) diff --git a/examples/models/core/qwenvl/show_pic.py b/examples/models/core/qwenvl/show_pic.py new file mode 100644 index 000000000000..f390ec860f58 --- /dev/null +++ b/examples/models/core/qwenvl/show_pic.py @@ -0,0 +1,34 @@ +import argparse + +import cv2 +import numpy as np +import zmq + +context = zmq.Context() +socket = context.socket(zmq.REQ) + + +def parse_arguments(): + parser = argparse.ArgumentParser() + parser.add_argument('--ip', type=str, default='127.0.0.1') + parser.add_argument('--port', type=str, default='8006') + return parser.parse_args() + + +args = parse_arguments() +ip_addr = "tcp://" + args.ip + ":" + args.port +socket.connect(ip_addr) + +while True: + socket.send(b"a") + message = socket.recv() + if len(message) == 1 and message == b'x': + break + image = np.frombuffer(message, dtype=np.uint8) + image = cv2.imdecode(image, 1) + image = cv2.resize(image, dsize=(512, 384)) + cv2.imshow("image", image) + key = cv2.waitKey(1) & 0xFF + + if key == ord('q'): + break diff --git a/examples/models/core/qwenvl/vit_onnx_trt.py b/examples/models/core/qwenvl/vit_onnx_trt.py new file mode 100644 index 000000000000..ba21fc93ef0a --- /dev/null +++ b/examples/models/core/qwenvl/vit_onnx_trt.py @@ -0,0 +1,196 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import argparse +import os +import time +from typing import List + +import requests +import tensorrt as trt +import torch +from PIL import Image +from torchvision import transforms +from torchvision.transforms import InterpolationMode +from transformers import AutoModelForCausalLM + +from tensorrt_llm._utils import release_gc, str_dtype_to_torch + + +class Preprocss: + + def __init__(self, image_size: int): + mean = (0.48145466, 0.4578275, 0.40821073) + std = (0.26862954, 0.26130258, 0.27577711) + self.image_transform = transforms.Compose([ + transforms.Resize((image_size, image_size), + interpolation=InterpolationMode.BICUBIC), + transforms.ToTensor(), + transforms.Normalize(mean=mean, std=std), + ]) + + def encode(self, image_paths: List[str]): + images = [] + for image_path in image_paths: + if image_path.startswith("http://") or image_path.startswith( + "https://"): + image = Image.open(requests.get(image_path, stream=True).raw) + else: + image = Image.open(image_path) + image = image.convert("RGB") + images.append(self.image_transform(image)) + images = torch.stack(images, dim=0) + return images + + +class ONNX_TRT: + + def __init__(self, image_size): + self.image_size = image_size + + def export_onnx(self, onnx_file_path, pretrained_model_path, image_url): + print("Start converting ONNX model!") + image_pre_obj = Preprocss(self.image_size) + torch_dtype = str_dtype_to_torch("float16") + model = AutoModelForCausalLM.from_pretrained( + pretrained_model_path, + device_map="cuda", + dtype=torch_dtype, + fp16=True, + trust_remote_code=True, + ).eval() + device = torch.device("cuda") if torch.cuda.is_available() else "cpu" + image = image_pre_obj.encode(image_url).to(device) + if not os.path.exists("image.pt"): + torch.save(image, "image.pt") + + model_visual = model.transformer.visual + model_visual.eval() + del model # To save GPU memory + + torch.onnx.export( + model_visual, + image.to("cuda"), + onnx_file_path, + opset_version=17, + input_names=["input"], + output_names=["output"], + dynamic_axes={"input": { + 0: "batch" + }}, + # Required for pytorch>=2.9.0 as dynamo becomes the default and introduces bugs as it does not support opset_version=17 natively + dynamo=False) + release_gc() # Further release memory + print( + f"Export to ONNX file successfully! The ONNX file stays in {onnx_file_path}" + ) + + def generate_trt_engine(self, + onnxFile, + planFile, + minBS=1, + optBS=2, + maxBS=4): + print("Start converting TRT engine!") + logger = trt.Logger(trt.Logger.VERBOSE) + builder = trt.Builder(logger) + network = builder.create_network( + 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)) + profile = builder.create_optimization_profile() + config = builder.create_builder_config() + config.set_flag(trt.BuilderFlag.FP16) + parser = trt.OnnxParser(network, logger) + + with open(onnxFile, "rb") as model: + if not parser.parse(model.read(), "/".join(onnxFile.split("/"))): + print("Failed parsing %s" % onnxFile) + for error in range(parser.num_errors): + print(parser.get_error(error)) + print("Succeeded parsing %s" % onnxFile) + + nBS = -1 + nMinBS = minBS + nOptBS = optBS + nMaxBS = maxBS + inputT = network.get_input(0) + inputT.shape = [nBS, 3, self.image_size, self.image_size] + profile.set_shape( + inputT.name, + [nMinBS, 3, self.image_size, self.image_size], + [nOptBS, 3, self.image_size, self.image_size], + [nMaxBS, 3, self.image_size, self.image_size], + ) + + config.add_optimization_profile(profile) + + t0 = time.time() + engineString = builder.build_serialized_network(network, config) + t1 = time.time() + if engineString is None: + print("Failed building %s" % planFile) + else: + print("Succeeded building %s in %d s" % (planFile, t1 - t0)) + with open(planFile, "wb") as f: + f.write(engineString) + + +def parse_arguments(): + parser = argparse.ArgumentParser() + # onnx/visual_encoder + parser.add_argument("--onnxFile", + type=str, + default="visual_encoder/visual_encoder.onnx", + help="") + parser.add_argument("--pretrained_model_path", + type=str, + default="Qwen-VL-Chat", + help="") + parser.add_argument( + "--planFile", + type=str, + default="plan/visual_encoder/visual_encoder_fp16.plan", + help="", + ) + parser.add_argument( + "--only_trt", + action="store_true", + help="Run only convert the onnx to TRT engine.", + ) + parser.add_argument("--minBS", type=int, default=1) + parser.add_argument("--optBS", type=int, default=1) + parser.add_argument("--maxBS", type=int, default=4) + parser.add_argument("--image_url", nargs="+", default=["./pics/demo.jpeg"]) + args = parser.parse_args() + return args + + +if __name__ == "__main__": + args = parse_arguments() + onnx_file_dir = os.path.dirname(args.onnxFile) + if not onnx_file_dir == "" and not os.path.exists(onnx_file_dir): + os.makedirs(onnx_file_dir) + plan_file_dir = os.path.dirname(args.planFile) + if not os.path.exists(plan_file_dir): + os.makedirs(plan_file_dir) + + onnx_trt_obj = ONNX_TRT(448) # or ONNX_TRT(config.visual['image_size']) + + if args.only_trt: + onnx_trt_obj.generate_trt_engine(args.onnxFile, args.planFile, + args.minBS, args.optBS, args.maxBS) + else: + onnx_trt_obj.export_onnx(args.onnxFile, args.pretrained_model_path, + args.image_url) + onnx_trt_obj.generate_trt_engine(args.onnxFile, args.planFile, + args.minBS, args.optBS, args.maxBS) diff --git a/examples/ngram/README.md b/examples/ngram/README.md index 4b5a00ee43ac..1e4dc8792b08 100644 --- a/examples/ngram/README.md +++ b/examples/ngram/README.md @@ -1,32 +1,39 @@ # NGram Speculative Decoding -This document shows how to run a model with NGram speculative decoding -(supported as `ASSISTED_GENERATION` in transformers and vLLM, source: -[GitHub](https://github.com/apoorvumang/prompt-lookup-decoding/tree/main)) -in TensorRT LLM. +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +This document shows how to build and run a model using NGram speculative decoding (supported as `ASSISTED_GENERATION` in transformers and vLLM, source: [GitHub](https://github.com/apoorvumang/prompt-lookup-decoding/tree/main)) in TensorRT LLM on single GPU, or single node multiple GPU. ## Overview -NGram builds a pattern pool from the prompt and previously generated tokens -and proposes draft tokens by matching the tail of the current sequence -against that pool. It has 2 hyperparameters that control the process of -generation: - -- `max_draft_len`: the maximum number of tokens provided as draft tokens in - one iteration, which is usually from 4 to 10 in common usage (default - value: 4). Empirically, the larger the value is, the higher acceptance rate - but higher overhead is expected at the same time, so the right balance - based on the models and application scenarios needs to be found. -- `max_matching_ngram_size`: the maximum number of tokens extracted from the - tail of the input prompt or generated output as a pattern, which is used to - search corresponding draft tokens (default value: 2). Empirically, the - larger the value is, the more precise context can be matched from the - existed sequence, indicating higher acceptance rate, but the higher - probability of miss-match and higher overhead appear, which fall back to - normal generation (one token per iteration). +We provide two styles of workflow to run NGram (named V1 and V2 respectively) now. V1 is in TRT workflow and similar to the Draft-Target-Model workflow, running in orchestrator mode and calling `runner.generate()` multiple times to get outputs, which is more flexible for customizing but slightly more overhead. V2 is in pytorch workflow and similar to the Look-Ahead workflow, running in leader mode and calling `runner.generate()` only one time to get outputs, which provides higher performance but fixed process. -## Support Matrix +The NGram has 3 additional hyperparameters that you need to specify to control the process of generation: +- `max_draft_len`: the maximum number of tokens provided as draft tokens in one iteration, which is usually from 4 to 10 in common usage (default value: 4). Empirically, the larger the value is, the higher acceptance rate but higher overhead is expected at the same time, so the right balance based on the models and application scenarios needs to be found. +- `max_matching_ngram_size`: the maximum number of tokens extracted from the tail of the input prompt or generated output as a pattern, which is used to search corresponding draft tokens (default value: 2). Empirically, the larger the value is, the more precise context can be matched from the existed sequence, indicating higher acceptance rate, but the higher probability of miss-match and higher overhead appear, which fall back to normal generation (one token per iteration). +- `device_list`: the index list of device(s) to run the model in V1 workflow. The length of it must be the same as the TP size of the draft model engine. For instances, `device_list=[0]` means using tp_size=1 and GPU 0 for the model, `device_list=[4,5,6,7]` means using tp=4 and GPU from 4 to 7 for the model. This parameter is neddless in V2 workflow. + ++ For example, the process of getting draft tokens using `max_draft_len=2` and `max_matching_ngram_size=4` with a sentence `prefix=[..., t1, t2, t3, t4]` is like below: + +```Python +pattern = prefix[:-2] # pattern=[t3, t4] (length=2) +if pattern in pool and len(pool[pattern]) == 4: # assuming it is {(t3, t4): (t5, t6, t7, t8)} + return pool[pattern] # draft token = [t5, t6, t7, t8] +elif pattern in pool and len(pool[pattern]) == <4: # assuming it is {(t3, t4): (t9, t10, t11)} + return pool[pattern] # draft token = [t9, t10, t11] +pattern = prefix[:-1] # Try shorter pattern if no candidate of length=2 exists, pattern=[t4] (length=1) +if pattern in pool and len(pool[pattern]) == 4: # The same process as above + return pool[pattern] +elif pattern in pool and len(pool[pattern]) == <4: + return pool[pattern] +return None # No any candidate exists +``` +## Support Matrix * GPU Compute Capability >= 8.0 (Ampere or newer) * FP16 / BF16 / FP8 * Paged KV Cache @@ -34,6 +41,59 @@ generation: ## Usage +### V1 workflow + ++ We use an open-source `llama-v2-13B` models in this example. ++ `--use_paged_context_fmha=enable` must be specified since we need KVcache reuse in this approach. ++ `--speculative_decoding_mode=draft_tokens_external` must be specified. ++ `--max_draft_len` must be specified as the length maximum of the draft tokens. ++ `--ngram_config` is corresponding configuration of NGram, we can see its usage in [util.py](../util.py). + + As an example, `[10,2,[0]]` means `max_draft_len=10`, `max_matching_ngram_size=2`, and device of target model is `GPU0`. ++ `--kv_cache_enable_block_reuse` must be specified for this approach. ++ Only CPP session is supported, so `--use_py_session` must not be specified. ++ `--num_beams` can not be specified as larger than 1 since beam search is not supported in this approach yet. + +```bash +# Build engine +python3 examples/models/core/llama/convert_checkpoint.py \ + --model_dir \ + --output_dir ./ckpt-target \ + --dtype float16 + +trtllm-build \ + --checkpoint_dir ./ckpt-target \ + --output_dir ./target-engine \ + --gemm_plugin float16 \ + --use_paged_context_fmha enable \ + --speculative_decoding_mode draft_tokens_external \ + --max_draft_len 10 \ + --max_batch_size 4 \ + --max_input_len 3200 \ + --max_seq_len 4800 + +# Run decoding +python3 examples/run.py \ + --tokenizer_dir \ + --engine_dir ./target-engine \ + --ngram_config "[10,2,[0]]" \ + --max_output_len 256 \ + --kv_cache_enable_block_reuse \ + --input_text "How does Draft-Sampling work?" + +# Run summarization tasks +python examples/summarize.py \ + --test_hf \ + --test_trt_llm \ + --check_accuracy \ + --hf_model_dir \ + --engine_dir ./target-engine \ + --batch_size 1 \ + --ngram_config "[10,2,[0]]" \ + --kv_cache_enable_block_reuse +``` + +### V2 workflow + ```bash python3 examples/llm-api/quickstart_advanced.py \ --spec_decode_max_draft_len 4 \ @@ -41,8 +101,3 @@ python3 examples/llm-api/quickstart_advanced.py \ --disable_overlap_scheduler \ --disable_kv_cache_reuse ``` - -With the LLM API, configure NGram through `NGramDecodingConfig` -(`speculative_config`). See the -[speculative decoding documentation](https://nvidia.github.io/TensorRT-LLM/features/speculative-decoding.html) -for details. diff --git a/examples/ngram/run_dtm_ngram.py b/examples/ngram/run_dtm_ngram.py new file mode 100644 index 000000000000..d0cd8687ef86 --- /dev/null +++ b/examples/ngram/run_dtm_ngram.py @@ -0,0 +1,382 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import ast + +import numpy as np +import torch +from ordered_set import OrderedSet + +from tensorrt_llm.logger import logger +from tensorrt_llm.runtime import ModelRunnerCpp + + +class NgramPool: # Ngrams pool for Ngram + + def __init__( + self, + input_batch_size: int, + max_draft_len: int, + max_matching_ngram_size: int, + end_id: int, + max_seq_len: list[int], + is_keep_all: bool = True, + is_use_oldest: bool = True, + ): + self.input_batch_size = input_batch_size + self.max_draft_len = max_draft_len + self.max_matching_ngram_size = max_matching_ngram_size + self.end_id = end_id + self.max_seq_len = max_seq_len + self.is_keep_all = is_keep_all + self.is_use_oldest = is_use_oldest + self.pool = [{} for _ in range(input_batch_size)] + self.start_index = [0 for _ in range(input_batch_size)] + + assert self.max_draft_len > 0, f"max_draft_len must be greater than 0, but got {self.max_draft_len}" + assert self.max_matching_ngram_size > 0, f"max_matching_ngram_size must be greater than 0, but got {self.max_matching_ngram_size}" + + def print_pool(self): + """ + For debug + """ + logger.info(f"Batch size = {self.input_batch_size}") + for i, map in enumerate(self.pool): + logger.info(f"Slot {i}, size = {len(map)}") + for key, values in map.items(): + logger.info(f" {key}->{values}") + + def get_draft_tokens(self, prefix: list[torch.Tensor], + batch_slot: list[int]): + """ + Get draft tokens from a batch of requests + modified from `transformers/generation/candidate_generator.py` + """ + batch_size = len(prefix) + prefix_len = [len(prefix[bi]) for bi in range(batch_size)] + draft_tokens = [] # `logits` is useless yet + for bi in range(batch_size): + gbi = batch_slot[bi] # Global index in the input batch + chosen_ids = [self.end_id] + # Skip search if prefix is length of `max_length - 1` + if prefix_len[bi] >= self.max_seq_len[gbi] - 1: + draft_tokens.append(chosen_ids) + continue + + # Update pool + sequence = prefix[bi][self.start_index[gbi]:].tolist() + for size in range( + min(self.max_matching_ngram_size, prefix_len[bi] - 1), 0, + -1): + # Find each possible key-value combination, and use tuple for hash + for l in range(len(sequence) - size): + r = min(l + size + self.max_draft_len, len(sequence)) + key = tuple(sequence[l:l + size]) + value = tuple(sequence[l + size:r]) + if key not in self.pool[gbi] or not self.is_keep_all or \ + len(self.pool[gbi][key][0]) < self.max_draft_len: + # Update the value if + # 1. the key does not exist + # 2. we only keep the newest one value for each key (MRU) + # 3. the length of the value saved before is less than `max_draft_len` + self.pool[gbi][key] = OrderedSet((value, )) + elif value not in self.pool[gbi][key]: + # Extend the value if the key is already existed but count of values is not enough + self.pool[gbi][key].add(value) + + # Find match + for size in range( + min(self.max_matching_ngram_size, prefix_len[bi] - 1), 0, + -1): + pattern = tuple(prefix[bi][-size:].tolist()) + if pattern not in self.pool[gbi]: + continue + if self.is_use_oldest: + # Always choose the oldest match, aligned with HF + chosen_ids = self.pool[gbi][pattern][0] + else: + # Always choose the newest match + chosen_ids = self.pool[gbi][pattern][-1] + break + draft_tokens.append(chosen_ids) + self.start_index[gbi] = max( + 0, prefix_len[bi] - + (self.max_draft_len + self.max_matching_ngram_size - 1)) + + return draft_tokens, None + + +def run_dtm_ngram(batch_input_ids, + args, + runtime_rank, + end_id, + pad_id, + stop_words_list, + bad_words_list, + vocab_size, + *, + target_runner=None): + # `dtm` for Draft-Target-Model, `ngram` for NGram + is_dtm = (args.draft_target_model_config is not None) + is_ngram = (args.ngram_config is not None) + assert is_dtm ^ is_ngram, "`--draft_target_model_config` and `--ngram_config` can not be specified at the same time." + if is_dtm: + assert args.draft_engine_dir is not None, "`--draft_engine_dir` must be specified in Draft-Target-Model." + draft_len, draft_device_list, target_device_list, use_logits = ast.literal_eval( + args.draft_target_model_config) + logger.info(f"Using Draft-Target-Model speculative decoding") + logger.info(f"draft_len: {draft_len}") + logger.info(f"Device(s) for draft model: {draft_device_list}") + logger.info(f"Device(s) for target model: {target_device_list}") + logger.info(f"Use logits to accept tokens: {use_logits}") + if is_ngram: + logger.info(f"Using NGram speculative decoding V1 workflow") + max_draft_len, max_matching_ngram_size, target_device_list = ast.literal_eval( + args.ngram_config) + logger.info(f"max_draft_len: {max_draft_len}") + logger.info(f"max_matching_ngram_size: {max_matching_ngram_size}") + logger.info(f"Device(s) for the model: {target_device_list}") + use_logits = False # `logits` is useless in this approach yet + + # Variables keeping constant during decoding + input_batch_size = len(batch_input_ids) # Note as `BS` + beam_width = args.num_beams # Note as `BW` + is_compute_acceptance_ratio = logger.level == 'verbose' # Only for verbose + input_len = [len(p) for p in batch_input_ids] + max_seq_len = [i + args.max_output_len for i in input_len] + # Variables changing during decoding + n_iteration = 0 + prefix = batch_input_ids # Input for each iteration + batch_slot = list(range(input_batch_size)) # Index of requests + if is_compute_acceptance_ratio: + n_draft_token = [0 for _ in range(input_batch_size)] + n_accept_token = [0 for _ in range(input_batch_size)] + + if is_ngram: + ngram_pool = NgramPool(input_batch_size, max_draft_len, + max_matching_ngram_size, end_id, max_seq_len) + + # Repack the output like the output of function `generate` + outputs = {} + outputs["output_ids"] = torch.full( + [input_batch_size, beam_width, + max(max_seq_len)], + end_id, + dtype=torch.int32) + for bi in range(input_batch_size): + outputs["output_ids"][bi, :, :input_len[bi]] = batch_input_ids[bi] + outputs["sequence_lengths"] = torch.full([input_batch_size, beam_width], + 0, + dtype=torch.int32) + outputs["context_logits"] = None + outputs["generation_logits"] = torch.full( + [input_batch_size, beam_width, + max(max_seq_len), vocab_size], + 0, + dtype=torch.float16) + outputs['cum_log_probs'] = None + outputs['log_probs'] = None + + # Model runner + common_runner_kwargs = dict( + lora_dir=args.lora_dir, + rank=runtime_rank, + debug_mode=args.debug_mode, + lora_ckpt_source=args.lora_ckpt_source, + gpu_weights_percent=args.gpu_weights_percent, + max_output_len=args.max_output_len, + is_enc_dec=False, + max_batch_size=input_batch_size, + max_input_len=max(input_len) + args.max_output_len, + max_beam_width=beam_width, + max_attention_window_size=args.max_attention_window_size, + sink_token_length=args.sink_token_length, + max_tokens_in_paged_kv_cache=args.max_tokens_in_paged_kv_cache, + kv_cache_enable_block_reuse=args.kv_cache_enable_block_reuse, + kv_cache_free_gpu_memory_fraction=args. + kv_cache_free_gpu_memory_fraction, + cross_kv_cache_fraction=None, + enable_chunked_context=args.enable_chunked_context, + multi_block_mode=args.multi_block_mode, + cuda_graph_mode=args.cuda_graph_mode, + enable_context_fmha_fp32_acc=args.enable_context_fmha_fp32_acc, + is_orchestrator_mode=True, + ) + + if is_dtm: + draft_runner_kwargs = common_runner_kwargs.copy() + draft_runner_kwargs.update(engine_dir=args.draft_engine_dir, + device_ids=draft_device_list) + draft_runner = ModelRunnerCpp.from_dir(**draft_runner_kwargs) + + if target_runner is None: # Skip this constructor if we have prepared the runner before + target_runner_kwargs = common_runner_kwargs.copy() + target_runner_kwargs.update(engine_dir=args.engine_dir, + device_ids=target_device_list) + target_runner = ModelRunnerCpp.from_dir(**target_runner_kwargs) + + if is_dtm and use_logits: + assert draft_runner.gather_generation_logits and target_runner.gather_generation_logits, "`--gather_generation_logits` must be specified while building draft/target models for using logits to accept" + + common_generaion_kwargs = dict( + max_attention_window_size=args.max_attention_window_size, + sink_token_length=args.sink_token_length, + end_id=end_id, + pad_id=pad_id, + temperature=args.temperature, + top_k=args.top_k, + top_p=args.top_p, + num_beams=beam_width, + num_return_sequences=args.num_return_sequences, + length_penalty=args.length_penalty, + early_stopping=args.early_stopping, + beam_width_array=None, + repetition_penalty=args.repetition_penalty, + presence_penalty=args.presence_penalty, + frequency_penalty=args.frequency_penalty, + min_p=args.min_p, + stop_words_list=stop_words_list, + bad_words_list=bad_words_list, + random_seed=args.random_seed, + lora_uids=args.lora_task_uids, + prompt_table=args.prompt_table_path, + prompt_tasks=args.prompt_tasks, + streaming=False, + output_sequence_lengths=True, + no_repeat_ngram_size=args.no_repeat_ngram_size, + return_dict=True, + return_all_generated_tokens=args.return_all_generated_tokens, + ) + + while True: + n_iteration += 1 + # Dynamic batch_size, decreases if some requests finish + batch_size = len(prefix) + prefix_len = [len(prefix[i]) for i in range(batch_size)] + # Get draft tokens + # `d_*` means variables from draft + # `d_seq_len` includes input part, but `d_len` doesn't + if is_dtm: + draft_generation_kwargs = common_generaion_kwargs.copy() + draft_generation_kwargs.update( + batch_input_ids=prefix, + max_new_tokens=draft_len, + streaming=False, + output_sequence_lengths=True, + return_dict=True, + ) + draft = draft_runner.generate(**draft_generation_kwargs) + torch.cuda.synchronize() + + # draft["output_ids"].shape -> [BS, BW, maxSL] + # draft["sequence_lengths"].shape -> [BS, BW] + # draft["generation_logits"].shape -> [BS, BW, draft_len, vocab_size] + d_ids = [[end_id]] * batch_size + d_logits = [None] * batch_size if use_logits else None + d_seq_len = draft["sequence_lengths"][:, 0].tolist() + d_len = [d_seq_len[bi] - prefix_len[bi] for bi in range(batch_size)] + for bi in range(batch_size): + l, r = prefix_len[bi], d_seq_len[bi] + if l >= r: # No useful draft tokens + continue + d_ids[bi] = draft["output_ids"][bi, 0, l:r].tolist() + if use_logits: + d_logits[bi] = draft["generation_logits"][bi, 0, + -d_len[bi]:, :] + if is_ngram: + d_ids, d_logits = ngram_pool.get_draft_tokens(prefix, batch_slot) + d_len = [len(i) for i in d_ids] + + # Run target model + # `t_*` means variables from target model + # `t_seq_len` and `t_seq_ids` include input part, but `t_len` or `t_ids` don't + target_generation_kwargs = common_generaion_kwargs.copy() + target_generation_kwargs.update(batch_input_ids=prefix, + draft_tokens_list=d_ids, + draft_logits_list=d_logits) + if is_dtm: + max_new_tokens = draft_len + 1 + if is_ngram: + max_new_tokens = max_draft_len + 1 + target_generation_kwargs.update(max_new_tokens=max_new_tokens) + target = target_runner.generate(**target_generation_kwargs) + torch.cuda.synchronize() + + t_ids = [None] * batch_size + t_seq_ids = [None] * batch_size + t_seq_len = target["sequence_lengths"][:, 0].tolist() + t_len = [t_seq_len[bi] - prefix_len[bi] for bi in range(batch_size)] + + # Update output and tokens for next iteration + for bi in range(batch_size): + gbi = batch_slot[bi] # Global index in the input batch + l = prefix_len[bi] + r = min(t_seq_len[bi], max_seq_len[gbi]) + t_ids[bi] = target["output_ids"][bi, 0, l:r].tolist() + t_seq_ids[bi] = target["output_ids"][bi, 0, :r] + outputs["output_ids"][gbi, 0, l:r] = torch.IntTensor(t_ids[bi]) + outputs["sequence_lengths"][gbi, 0] = r + if use_logits: + outputs["generation_logits"][gbi, 0, (l - input_len[bi]):(r - input_len[bi])] = \ + target["generation_logits"][bi][0,:(r-l)].detach().cpu() + if is_compute_acceptance_ratio: + n_draft_token[gbi] += d_len[bi] + length = min(d_len[bi], t_len[bi], + max_seq_len[gbi] - prefix_len[bi]) + res = [d_ids[bi][i] == t_ids[bi][i] for i in range(length)] + n_accept_token[gbi] += \ + ((~torch.BoolTensor(res)).cumsum(axis=-1) < 1).sum() + + # Yield output if using streaming + if args.streaming and not n_iteration % args.streaming_interval: + yield outputs + + # Evaluate stop criteria and prepare inputs for next iteration + prefix_next = [] + batch_slot_next = [] + for bi in range(batch_size): + gbi = batch_slot[bi] # Global index in the input batch + # Stop due to output length + if len(t_seq_ids[bi]) >= max_seq_len[gbi]: + continue # No need to update for the stopped requests + # Stop due to the same output. Normally target should return 1 more token. + # if (d_ids is not None and np.array_equal(d_ids[bi], t_ids[bi])): + # continue + # Stop due to no change (hit early stopping) + if np.array_equal(t_seq_ids[bi].cpu().numpy(), + prefix[bi].cpu().numpy()): + continue + # Stop due to end words + if end_id in t_seq_ids[bi][prefix_len[bi]:]: + continue + # TODO: Check bad words and stop words criteria + prefix_next.append(t_seq_ids[bi]) + batch_slot_next.append(gbi) + prefix = prefix_next + batch_slot = batch_slot_next + if len(prefix) == 0: # Leave while loop if no request remained + break + + if is_compute_acceptance_ratio: + logger.debug(f"Count of iteration(s): {n_iteration}") + logger.debug(f"Acceptance ratio:") + for i, (a, d) in enumerate(zip(n_accept_token, n_draft_token)): + logger.debug(f"Request {i}: {a / d * 100 :6.2f}%") + + # Return runner in No-Streaming mode + if args.streaming: + yield outputs + else: + yield outputs, target_runner diff --git a/examples/openai_triton/README.md b/examples/openai_triton/README.md new file mode 100644 index 000000000000..b5f39d105974 --- /dev/null +++ b/examples/openai_triton/README.md @@ -0,0 +1,7 @@ +# Integration for OpenAI Triton + +The typical approach to integrate a kernel into TensorRT LLM is to create TensorRT plugins. +Specially for integrating OpenAI Triton kernels, there are two methods: + +1. Creating TensorRT plugin manually, you can refer to [manual plugin example](./manual_plugin/) for details, +2. Generate the TensorRT plugins automatically, please refer to [automatic plugin example](./plugin_autogen/) for details. diff --git a/examples/openai_triton/manual_plugin/CMakeLists.txt b/examples/openai_triton/manual_plugin/CMakeLists.txt new file mode 100644 index 000000000000..bec14231511e --- /dev/null +++ b/examples/openai_triton/manual_plugin/CMakeLists.txt @@ -0,0 +1,113 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & +# AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. +cmake_minimum_required(VERSION 3.1) + +# Enable C++ +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED TRUE) + +# Define project name +set(TARGET_NAME trt_llm_custom_plugins) +project(${TARGET_NAME}) + +set(CMAKE_VERBOSE_MAKEFILE 1) + +# Compile options +set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall -pthread ") +set(CMAKE_C_FLAGS_DEBUG "-g -O0") +set(CMAKE_C_FLAGS_RELEASE "-O2") +set(CMAKE_CXX_FLAGS "${CMAKE_C_FLAGS} -lstdc++") +set(CMAKE_CXX_FLAGS_DEBUG ${CMAKE_C_FLAGS_DEBUG}) +set(CMAKE_CXX_FLAGS_RELEASE ${CMAKE_C_FLAGS_RELEASE}) + +set(CMAKE_BUILD_TYPE release) + +find_package(CUDA REQUIRED) +message(STATUS "CUDA library status:") +message(STATUS " config: ${CUDA_DIR}") +message(STATUS " version: ${CUDA_VERSION}") +message(STATUS " libraries: ${CUDA_LIBRARIES}") +message(STATUS " include path: ${CUDA_INCLUDE_DIRS}") + +if(NOT DEFINED TRT_INCLUDE_DIR) + set(TRT_INCLUDE_DIR "/usr/local/tensorrt/include") + if(NOT EXISTS ${TRT_INCLUDE_DIR}) + # In case of TensorRT installed from a deb package. + set(TRT_INCLUDE_DIR "/usr/include/x86_64-linux-gnu") + endif() +endif() +message(STATUS "tensorrt include path: ${TRT_INCLUDE_DIR}") +if(DEFINED TRT_LLM_INCLUDE_DIR) + message( + STATUS "openai_triton/manual_plugin example has been self-contained " + "and TRT_LLM_INCLUDE_DIR is now unnecessary to specify the path of " + "C++ runtime source files.") +endif() + +if(NOT DEFINED TRT_LIB_DIR) + set(TRT_LIB_DIR "/usr/local/tensorrt/lib") + if(NOT EXISTS ${TRT_INCLUDE_DIR}) + # In case of TensorRT installed from a deb package. + set(TRT_LIB_DIR "/lib/${CMAKE_SYSTEM_PROCESSOR}-linux-gnu") + endif() +endif() +find_library( + TRT_LIB_PATH nvinfer + HINTS ${TRT_LIB_DIR} + NO_DEFAULT_PATH) +find_library(TRT_LIB_PATH nvinfer REQUIRED) +message(STATUS "TRT_LIB_DIR: ${TRT_LIB_DIR}") +message(STATUS "Found nvinfer library: ${TRT_LIB_PATH}") + +if(NOT DEFINED TRT_LLM_LIB_DIR) + # Find at tensorrt_llm/libs. + execute_process( + COMMAND ${CMAKE_COMMAND} -E env "PYTHONPATH=${PYTHONPATH}" "python" "-c" + "import tensorrt_llm; print(f'{tensorrt_llm.__path__[0]}/libs')" + OUTPUT_VARIABLE TRT_LLM_LIB_DIR + OUTPUT_STRIP_TRAILING_WHITESPACE) + # Find /tensorrt_llm/libs. + list(APPEND TRT_LLM_LIB_DIR "../../../tensorrt_llm/libs") +endif() +find_library(TRT_LLM_LIB_PATH nvinfer_plugin_tensorrt_llm + HINTS ${TRT_LLM_LIB_DIR} NO_DEFAULT_PATH) +find_library(TRT_LLM_LIB_PATH nvinfer_plugin_tensorrt_llm REQUIRED) +message(STATUS "Found nvinfer_plugin_tensorrt_llm library: ${TRT_LLM_LIB_PATH}") + +find_library(TRT_LLM_COMMON_LIB_PATH th_common HINTS ${TRT_LLM_LIB_DIR} + NO_DEFAULT_PATH) +find_library(TRT_LLM_COMMON_LIB_PATH th_common REQUIRED) +message(STATUS "Found th_common library: ${TRT_LLM_COMMON_LIB_PATH}") + +# Declare the target library. +add_library( + ${TARGET_NAME} SHARED + tritonPlugins.cpp + TritonFlashAttentionPlugin.cpp + aot/fmha_kernel_fp16.c + aot/fmha_kernel_fp32.c + aot/fp16/fmha_kernel_d64_fp16.fbf0f274_0d1d2d3d4d5d6789.c + aot/fp32/fmha_kernel_d64_fp32.f30323ef_0d1d2d3d4d5d6789.c) + +target_link_libraries( + ${TARGET_NAME} PUBLIC cuda ${CUDA_LIBRARIES} ${TRT_LLM_LIB_PATH} + ${TRT_LLM_COMMON_LIB_PATH} ${TRT_LIB_PATH}) + +if(NOT MSVC) + set_property(TARGET ${TARGET_NAME} PROPERTY LINK_FLAGS "-Wl,--no-undefined") +endif() + +target_include_directories(${TARGET_NAME} PUBLIC /usr/local/cuda/include) +target_include_directories(${TARGET_NAME} PUBLIC ${TRT_INCLUDE_DIR}) diff --git a/examples/openai_triton/manual_plugin/README.md b/examples/openai_triton/manual_plugin/README.md new file mode 100644 index 000000000000..5c8b5d481d52 --- /dev/null +++ b/examples/openai_triton/manual_plugin/README.md @@ -0,0 +1,177 @@ +# OpenAI Triton Plugin in TensorRT-LLM + +This document describes how to build and run a custom plugin leveraging [OpenAI Triton](https://github.com/openai/triton) in TensorRT-LLM. +The workflow can be summarized as follows. + 1. Implement a kernel using Triton in Python. + 2. Compile that kernel using Triton AoT (Ahead-of-Time) compilation tool to generate C files. + 3. Implement a custom TensorRT LLM plugin to execute the compiled kernel. + 4. Build the TensorRT engine. + 5. It is ready to be executed by TensorRT. + +In this example, we show how to create a TensorRT LLM plugin to wrap a [Fused Attention]((fmha_triton.py)) kernel implemented in OpenAI Triton. +As a prerequisite, it is necessary to have the TensorRT LLM C++ runtime library. +The instructions to build that library can be found [here](../../README.md#build-from-source). + +## 1. Triton AoT Preparation + +OpenAI Triton offers an Ahead-of-Time (AoT) compilation tool to generate C files that wrap compiled GPU kernel. +To use the AoT feature, you need a Triton version posterior to the [d0c35b3](https://github.com/openai/triton/commit/d0c35b3b7d6badf0c0d56a821dddab7ace73b4de) commit +and this example has been tested on the [b43c28f](https://github.com/openai/triton/tree/b43c28fdd7a2f95b2e87180cba5d984732120d5c) commit. +```bash +git clone https://github.com/openai/triton +cd triton/python/ +git checkout d4644d6cb3ae674e1f15932cac1f28104795744f +pip install cmake && pip install . +cd - +``` + +For AoT compilation, it is necessary to provide a kernel signature and specify the values of `tl.constexpr` parameters in a comma-separated format. +Details can be found in the [compile.py](https://github.com/openai/triton/blob/main/python/triton/tools/compile.py) file in the Triton project. + +Here are examples of kernel AOT compilations for the [Fused Attention](fmha_triton.py) kernel. +```bash +# Kernel for data type=float16, BLOCK_M=128, BLOCK_DMODEL=64, BLOCK_N=128 +export TRITON_ROOT=$(pip show triton | grep Location | cut -d' ' -f2) +rm -rf aot +mkdir -p aot/fp16 +python ${TRITON_ROOT}/triton/tools/compile.py \ + fmha_triton.py \ + -n fused_attention_kernel \ + -o aot/fp16/fmha_kernel_d64_fp16 \ + --out-name fmha_d64_fp16 \ + -w 4 \ + -ns 2 \ + -s "*fp16:16, *fp32:16, *fp32:16, *fp16:16, *fp16:16, *fp16:16, fp32, i32, i32, i32, 128, 64, 128" \ + -g "(seq_len + 127) / 128, batch_size * num_heads, 1" +# Kernel for data type=float32, BLOCK_M=64, BLOCK_DMODEL=64, BLOCK_N=64 +mkdir -p aot/fp32 +python ${TRITON_ROOT}/triton/tools/compile.py \ + fmha_triton.py \ + -n fused_attention_kernel \ + -o aot/fp32/fmha_kernel_d64_fp32 \ + --out-name fmha_d64_fp32 \ + -w 4 \ + -ns 2 \ + -s "*fp32:16, *fp32:16, *fp32:16, *fp32:16, *fp32:16, *fp32:16, fp32, i32, i32, i32, 64, 64, 64" \ + -g "(seq_len + 63) / 64, batch_size * num_heads, 1" + +# Link generated headers and create dispatchers. +python ${TRITON_ROOT}/triton/tools/link.py aot/fp16/*.h -o aot/fmha_kernel_fp16 +python ${TRITON_ROOT}/triton/tools/link.py aot/fp32/*.h -o aot/fmha_kernel_fp32 +``` +The tool will generate .c and .h files to launch the GPU kernel. +Note that it is necessary to specify the kernel name using the --out-name option, it allows to define dispatcher names for the different data types. +The above invocations will generate `aot/fmha_kernel_{fp16|fp32}.{c|h}` files that contain three functions: + - the `load_fmha_d64_{fp16|fp32}` function to load the code of the GPU kernel, + - the `fmha_d64_{fp16|fp32}` function to launch the kernel, + - the `unload_fmha_d64_{fp16|fp32}` function to unload the GPU kernel. + +If GPU resources are limited, it is recommended to adjust the number of stages or warps accordingly. For example, on the V100, the aforementioned arguments might fail due to insufficient shared memory of the GPU. This can be mitigated by reducing the number of stages by one, using `-ns 1`. + + +## 2. Implement a Custom TensorRT Plugin + +This section describes how to implement a custom plugin for TensorRT LLM to execute the Triton kernel created in the previous section. +We provide an example of plugin implementation. + - TritonFlashAttentionPlugin([.cpp](TritonFlashAttentionPlugin.cpp), [.h](TritonFlashAttentionPlugin.h)): TensorRT plugin. + - [plugin.py](plugin.py): Python wrapper. + +`TritonFlashAttentionPlugin` is a TensorRT plugin that integrates a Triton kernel generated with the AoT compiler. +The `initialize` and `terminate` functions show how to initialize and terminate the TensorRT plugin. +The `enqueue` member function shows how to call the generated Triton kernel on the GPU. +Note that the name of the Triton kernel depends on the function's signature, meaning that different types or specialization leads a different kernel name. +Thus, if you change an option during AoT compilation like `-s `, you also have to update file names in CMakeLists.txt in order to match the names generated by the AoT compiler. + +To build a shared library for the custom Triton plugin, run: +```bash +mkdir -p build && cd build +cmake .. && make +cd .. +``` +As mentioned in the previous section, it is necessary to have the TensorRT LLM C++ runtime library. +If you want to specify the library paths, run: +```bash +cmake -DTRT_LIB_DIR= -DTRT_INCLUDE_DIR= -DTRT_LLM_LIB_DIR= .. +``` +If the build is successful, you should be able to find a shared library for the custom plugin at `build/libtrt_llm_custom_plugins.so`. + +A Python wrapper of the Fused Multihead Attention (FMHA) operator and the corresponding TensorRT LLM layer are implemented in [plugin.py](plugin.py). +It is similar to other TensorRT LLM operators and layers implemented in [functional.py](../../tensorrt_llm/functional.py) and [layers](../../tensorrt_llm/layers), respectively. +That FMHA operator uses the custom plugin that wraps the functions generated from the Triton kernel. + +## 3. Build and Run the TensorRT Engine + +We are now ready to build and run the TensorRT engine that uses the Triton kernel. +Here are the two commands to build and run the engine: +```bash +python build.py --num_heads 32 --head_size 64 --max_batch_size 8 --max_seq_len 512 --dtype float16 +python run.py --num_heads 32 --head_size 64 --batch_size 8 --seq_len 512 --log_level verbose --benchmark +``` + +## 4. Known Issues + +### 1. A generated dispatcher might not execute a kernel without raising an error due to a missing branch. + +The kernel dispatcher written by `link.py` has a missing branch, which can result in returning without executing a kernel. +For instance, in our example, the generated dispatcher looks like this: +```c++ +CUresult fmha_d64_fp16(CUstream stream, unsigned int gX, unsigned int gY, unsigned int gZ, CUdeviceptr Out, CUdeviceptr L, CUdeviceptr M, CUdeviceptr Q, CUdeviceptr K, CUdeviceptr V, float sm_scale, int32_t seq_len){ + if ((Out % 16 == 0) && (L % 16 == 0) && (M % 16 == 0) && (Q % 16 == 0) && (K % 16 == 0) && (V % 16 == 0)) + return fmha_d64_fp16_0eb6b090_0d1d2d3d4d5d67(stream, gX, gY, gZ, Out, L, M, Q, K, V, sm_scale, seq_len); +} +``` +It is recommended to manually update the generated functions by `link.py` to return a proper error for proper error handling. + + +### 2. The shared memory required by a generated kernel may exceed the hardware limitation. + +The AoT compiler does not verify the limitations of shared memory size during compilation time, which could potentially lead to the out-of-resource errors during runtime. +It would be helpful to verify if the requirement of the dynamic shared memory size in a generated kernel exceeds the hardware limitation. +You can find the number at the line of `cuLaunchKernel` call in the generated `.c` file. +For instance, the shared memory size is 114690 bytes in our example. +```c++ +CUresult fmha_d64_fp16_0eb6b090_0d1d2d3d4d5d67(CUstream stream, unsigned int gX, unsigned int gY, unsigned int gZ, CUdeviceptr Out, CUdeviceptr L, CUdeviceptr M, CUdeviceptr Q, CUdeviceptr K, CUdeviceptr V, float sm_scale, int32_t seq_len) { + if (fmha_d64_fp16_0eb6b090_0d1d2d3d4d5d67_func == NULL) + load_fmha_d64_fp16_0eb6b090_0d1d2d3d4d5d67(); + void *args[8] = { &Out, &L, &M, &Q, &K, &V, &sm_scale, &seq_len }; + // TODO: shared memory + if(gX * gY * gZ > 0) + return cuLaunchKernel(fmha_d64_fp16_0eb6b090_0d1d2d3d4d5d67_func, gX, gY, gZ, 4 * 32, 1, 1, 114690, stream, args, NULL); +} +``` +It may be resolved by reduing the block size. + + +### 3. AttributeError: module 'triton' has no attribute 'jit' + +This problem may arise if Triton is installed in editable mode. To resolve this issue, please install Triton using the non-editable mode. Refer https://github.com/openai/triton/issues/1693. + +### 4. Unload the same module more than once while building the engine +When the plugin is used more than once within a model, the function cuModuleUnload() will be invoked multiple times during the engine building stage. Related code is generated by Openai Triton and can be found in the folder `examples/openai_triton/manual_plugin/aot/`. One example is: + +```c++ +void unload_fmha_d64_fp32_f30323ef_0d1d2d3d4d5d6789(void) { + CUDA_CHECK(cuModuleUnload(fmha_d64_fp32_f30323ef_0d1d2d3d4d5d6789_mod)); +} +``` + +As the generated code didn't check the value of the module object, this function might unload the same module multiple times, which will cause an error as follows: + +``` +Triton Error [CUDA]: invalid resource handle\n/opt/rapids/src/cudf/cpp/build/_deps/arrow-src/cpp/src/arrow/filesystem/s3fs.cc:2904:  arrow::fs::FinalizeS3 was not called even though S3 was initialized.  This could lead to a segmentation fault at exit +``` + +The error message is ambiguous. If we use compute-sanitizer to help debug, we can get the following information: +``` +========= Program hit CUDA_ERROR_INVALID_HANDLE (error 400) due to "invalid resource handle" on CUDA API call to cuModuleUnload. +``` + +So we need to modify the above generated code as follows to avoid the above error. +```c++ +void unload_fmha_d64_fp32_f30323ef_0d1d2d3d4d5d6789(void) { + if(fmha_d64_fp32_f30323ef_0d1d2d3d4d5d6789_mod){ + CUDA_CHECK(cuModuleUnload(fmha_d64_fp32_f30323ef_0d1d2d3d4d5d6789_mod)); + } + fmha_d64_fp32_f30323ef_0d1d2d3d4d5d6789_mod=NULL; +} +``` diff --git a/examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.cpp b/examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.cpp new file mode 100644 index 000000000000..198d8be1ca16 --- /dev/null +++ b/examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.cpp @@ -0,0 +1,386 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "TritonFlashAttentionPlugin.h" + +// Import a generated header to use generated triton kernels. +extern "C" +{ +#include "aot/fmha_kernel_fp16.h" +#include "aot/fmha_kernel_fp32.h" +} + +#include +#include +#include +#include + +using namespace nvinfer1; +using openai_triton::plugin::TritonFlashAttentionPluginCreator; +using openai_triton::plugin::TritonFlashAttentionPlugin; + +static char const* TRITON_FLASH_ATTENTION_PLUGIN_VERSION{"1"}; +static char const* TRITON_FLASH_ATTENTION_PLUGIN_NAME{"TritonFlashAttention"}; +PluginFieldCollection TritonFlashAttentionPluginCreator::mFC{}; +std::vector TritonFlashAttentionPluginCreator::mPluginAttributes; + +namespace openai_triton::plugin +{ + +// Write values into buffer +template +void writeArg(char*& buffer, T const& val) +{ + std::memcpy(buffer, &val, sizeof(T)); + buffer += sizeof(T); +} + +// Read values from buffer +template +void readArg(char const*& buffer, T& val) +{ + std::memcpy(&val, buffer, sizeof(T)); + buffer += sizeof(T); +} + +std::uintptr_t constexpr kCudaMemAlign = 128; + +int8_t* nextWorkspacePtr(int8_t* ptr, uintptr_t previousWorkspaceSize) +{ + uintptr_t addr = (uintptr_t) ptr; + addr += previousWorkspaceSize; + if (addr % kCudaMemAlign) + { + addr += kCudaMemAlign - addr % kCudaMemAlign; + } + return (int8_t*) addr; +} + +TritonFlashAttentionPlugin::TritonFlashAttentionPlugin( + int numHeads, int headSize, float softmaxScale, nvinfer1::DataType type) + : mNumHeads(numHeads) + , mHeadSize(headSize) + , mSoftmaxScale(softmaxScale) + , mType(type) +{ +} + +// Parameterized constructor +TritonFlashAttentionPlugin::TritonFlashAttentionPlugin(void const* data, size_t length) +{ + char const *d = reinterpret_cast(data), *a = d; + readArg(d, mNumHeads); + readArg(d, mHeadSize); + readArg(d, mSoftmaxScale); + readArg(d, mType); + TLLM_CHECK(d == a + length); +} + +// IPluginV2DynamicExt Methods +nvinfer1::IPluginV2DynamicExt* TritonFlashAttentionPlugin::clone() const noexcept +{ + auto* plugin = new TritonFlashAttentionPlugin(*this); + plugin->setPluginNamespace(mNamespace.c_str()); + return plugin; +} + +nvinfer1::DimsExprs TritonFlashAttentionPlugin::getOutputDimensions( + int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, nvinfer1::IExprBuilder& exprBuilder) noexcept +{ + // Output shape. + // output tensor [batchSize, seqLen, mNumHeads, head_size] + assert(outputIndex == 0); + return inputs[outputIndex]; +} + +bool TritonFlashAttentionPlugin::supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept +{ + // In this example, inputs: Q, K, V, outputs: Out + assert(nbInputs + nbOutputs == 4); + assert(0 <= pos && pos < nbInputs + nbOutputs); + + bool is_valid = false; + if (0 <= pos && pos < 3) // Q, K, V + { + is_valid = inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + } + else if (pos == nbInputs) // Out + { + is_valid = inOut[pos].type == mType && inOut[pos].format == TensorFormat::kLINEAR; + } + return is_valid; +} + +void TritonFlashAttentionPlugin::configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept +{ +} + +size_t TritonFlashAttentionPlugin::getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept +{ + // Set workspace size if needed. In this example, we need for L and m buffers. + auto const Q = inputs[0]; + int const batchSize = Q.dims.d[0]; + int const seqLen = Q.dims.d[2]; + int const numBuffers = 2; + size_t workspaces[numBuffers]; + workspaces[0] = sizeof(float) * batchSize * mNumHeads * seqLen; + workspaces[1] = sizeof(float) * batchSize * mNumHeads * seqLen; + + size_t total = 0; + for (int i = 0; i < numBuffers; i++) + { + total += workspaces[i]; + if (workspaces[i] % kCudaMemAlign) + { + total += kCudaMemAlign - (workspaces[i] % kCudaMemAlign); + } + } + return total; +} + +template +int TritonFlashAttentionPlugin::enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) +{ + assert(inputDesc[0].dims.d[1] == mNumHeads && inputDesc[0].dims.d[3] == mHeadSize); + assert(inputDesc[1].dims.d[1] == mNumHeads && inputDesc[1].dims.d[3] == mHeadSize); + assert(inputDesc[2].dims.d[1] == mNumHeads && inputDesc[2].dims.d[3] == mHeadSize); + + int batchSize = inputDesc[0].dims.d[0]; + int seqLen = inputDesc[0].dims.d[2]; + + T* Out = reinterpret_cast(outputs[0]); + + const size_t bufSize = sizeof(float) * batchSize * mNumHeads * seqLen; + float* L = reinterpret_cast(workspace); + float* M = reinterpret_cast(nextWorkspacePtr(reinterpret_cast(L), bufSize)); + + T const* Q = reinterpret_cast(inputs[0]); + T const* K = reinterpret_cast(inputs[1]); + T const* V = reinterpret_cast(inputs[2]); + + // Launch a cuda kernel generated by Triton AoT. + int res = 0; + if (std::is_same::value) + { + res = fmha_d64_fp32_default(stream, reinterpret_cast(Out), reinterpret_cast(L), + reinterpret_cast(M), reinterpret_cast(Q), reinterpret_cast(K), + reinterpret_cast(V), mSoftmaxScale, batchSize, mNumHeads, seqLen); + } + else + { + res = fmha_d64_fp16_default(stream, reinterpret_cast(Out), reinterpret_cast(L), + reinterpret_cast(M), reinterpret_cast(Q), reinterpret_cast(K), + reinterpret_cast(V), mSoftmaxScale, batchSize, mNumHeads, seqLen); + } + return res; +} + +int TritonFlashAttentionPlugin::enqueue(nvinfer1::PluginTensorDesc const* inputDesc, + nvinfer1::PluginTensorDesc const* outputDesc, void const* const* inputs, void* const* outputs, void* workspace, + cudaStream_t stream) noexcept +{ + int res = 1; + if (mType == DataType::kHALF) + { + res = enqueueImpl(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + else if (mType == DataType::kFLOAT) + { + res = enqueueImpl(inputDesc, outputDesc, inputs, outputs, workspace, stream); + } + sync_check_cuda_error(); + return res; +} + +// IPluginV2Ext Methods +nvinfer1::DataType TritonFlashAttentionPlugin::getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept +{ + assert(index == 0); + return inputTypes[0]; +} + +// IPluginV2 Methods + +char const* TritonFlashAttentionPlugin::getPluginType() const noexcept +{ + return TRITON_FLASH_ATTENTION_PLUGIN_NAME; +} + +char const* TritonFlashAttentionPlugin::getPluginVersion() const noexcept +{ + return TRITON_FLASH_ATTENTION_PLUGIN_VERSION; +} + +int TritonFlashAttentionPlugin::getNbOutputs() const noexcept +{ + return 1; +} + +int TritonFlashAttentionPlugin::initialize() noexcept +{ + // Load kernels generated by Triton AoT. + load_fmha_d64_fp32(); + load_fmha_d64_fp16(); + return 0; +} + +void TritonFlashAttentionPlugin::terminate() noexcept +{ + // Unload kernels generated by Triton AoT. + unload_fmha_d64_fp32(); + unload_fmha_d64_fp16(); +} + +size_t TritonFlashAttentionPlugin::getSerializationSize() const noexcept +{ + return sizeof(mNumHeads) + sizeof(mHeadSize) + sizeof(mSoftmaxScale) + sizeof(mType); +} + +void TritonFlashAttentionPlugin::serialize(void* buffer) const noexcept +{ + char *d = static_cast(buffer), *a = d; + writeArg(d, mNumHeads); + writeArg(d, mHeadSize); + writeArg(d, mSoftmaxScale); + writeArg(d, mType); + TLLM_CHECK(d == a + getSerializationSize()); +} + +void TritonFlashAttentionPlugin::destroy() noexcept +{ + // This gets called when the network containing plugin is destroyed + delete this; +} + +void TritonFlashAttentionPlugin::setPluginNamespace(char const* libNamespace) noexcept +{ + mNamespace = libNamespace; +} + +char const* TritonFlashAttentionPlugin::getPluginNamespace() const noexcept +{ + return mNamespace.c_str(); +} + +/////////////// + +TritonFlashAttentionPluginCreator::TritonFlashAttentionPluginCreator() +{ + // Fill PluginFieldCollection with PluginField arguments metadata + mPluginAttributes.clear(); + mPluginAttributes.emplace_back(PluginField("num_heads", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("head_size", nullptr, PluginFieldType::kINT32)); + mPluginAttributes.emplace_back(PluginField("softmax_scale", nullptr, PluginFieldType::kFLOAT32)); + mPluginAttributes.emplace_back(PluginField("type_id", nullptr, PluginFieldType::kINT32)); + mFC.nbFields = mPluginAttributes.size(); + mFC.fields = mPluginAttributes.data(); +} + +char const* TritonFlashAttentionPluginCreator::getPluginName() const noexcept +{ + return TRITON_FLASH_ATTENTION_PLUGIN_NAME; +} + +char const* TritonFlashAttentionPluginCreator::getPluginVersion() const noexcept +{ + return TRITON_FLASH_ATTENTION_PLUGIN_VERSION; +} + +PluginFieldCollection const* TritonFlashAttentionPluginCreator::getFieldNames() noexcept +{ + return &mFC; +} + +IPluginV2* TritonFlashAttentionPluginCreator::createPlugin(char const* name, PluginFieldCollection const* fc) noexcept +{ + PluginField const* fields = fc->fields; + int numHeads = 0; + int headSize = 0; + float softmaxScale = 1.0f; + nvinfer1::DataType type; + // Read configurations from each fields + for (int i = 0; i < fc->nbFields; ++i) + { + char const* attrName = fields[i].name; + if (!strcmp(attrName, "num_heads")) + { + assert(fields[i].type == PluginFieldType::kINT32); + numHeads = static_cast(*(static_cast(fields[i].data))); + } + else if (!strcmp(attrName, "head_size")) + { + assert(fields[i].type == PluginFieldType::kINT32); + headSize = static_cast(*(static_cast(fields[i].data))); + } + else if (!strcmp(attrName, "softmax_scale")) + { + assert(fields[i].type == PluginFieldType::kFLOAT32); + softmaxScale = static_cast(*(static_cast(fields[i].data))); + } + else if (!strcmp(attrName, "type_id")) + { + assert(fields[i].type == PluginFieldType::kINT32); + type = static_cast(*(static_cast(fields[i].data))); + } + } + try + { + auto* obj = new TritonFlashAttentionPlugin(numHeads, headSize, softmaxScale, type); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + std::cerr << "Caught exception: " << e.what() << std::endl; + } + return nullptr; +} + +IPluginV2* TritonFlashAttentionPluginCreator::deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept +{ + // This object will be deleted when the network is destroyed, which will + // call TritonFlashAttentionPlugin::destroy() + try + { + auto* obj = new TritonFlashAttentionPlugin(serialData, serialLength); + obj->setPluginNamespace(mNamespace.c_str()); + return obj; + } + catch (std::exception const& e) + { + std::cerr << "Caught exception: " << e.what() << std::endl; + } + return nullptr; +} + +void TritonFlashAttentionPluginCreator::setPluginNamespace(char const* libNamespace) noexcept +{ + mNamespace = libNamespace; +} + +char const* TritonFlashAttentionPluginCreator::getPluginNamespace() const noexcept +{ + return mNamespace.c_str(); +} + +} // namespace openai_triton::plugin diff --git a/examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.h b/examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.h new file mode 100644 index 000000000000..cd95eb48ec2a --- /dev/null +++ b/examples/openai_triton/manual_plugin/TritonFlashAttentionPlugin.h @@ -0,0 +1,113 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#pragma once + +#include + +#include +#include +#include +#include + +#include +#include + +namespace openai_triton::plugin +{ + +class TritonFlashAttentionPlugin : public nvinfer1::IPluginV2DynamicExt +{ +public: + TritonFlashAttentionPlugin(int numHeads, int headSize, float softmaxScale, nvinfer1::DataType type); + + TritonFlashAttentionPlugin(void const* data, size_t length); + + ~TritonFlashAttentionPlugin() override = default; + + // IPluginV2DynamicExt Methods + nvinfer1::IPluginV2DynamicExt* clone() const noexcept override; + nvinfer1::DimsExprs getOutputDimensions(int outputIndex, nvinfer1::DimsExprs const* inputs, int nbInputs, + nvinfer1::IExprBuilder& exprBuilder) noexcept override; + bool supportsFormatCombination( + int pos, nvinfer1::PluginTensorDesc const* inOut, int nbInputs, int nbOutputs) noexcept override; + void configurePlugin(nvinfer1::DynamicPluginTensorDesc const* in, int nbInputs, + nvinfer1::DynamicPluginTensorDesc const* out, int nbOutputs) noexcept override; + size_t getWorkspaceSize(nvinfer1::PluginTensorDesc const* inputs, int nbInputs, + nvinfer1::PluginTensorDesc const* outputs, int nbOutputs) const noexcept override; + int enqueue(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream) noexcept override; + + template + int enqueueImpl(nvinfer1::PluginTensorDesc const* inputDesc, nvinfer1::PluginTensorDesc const* outputDesc, + void const* const* inputs, void* const* outputs, void* workspace, cudaStream_t stream); + + // IPluginV2Ext Methods + nvinfer1::DataType getOutputDataType( + int index, nvinfer1::DataType const* inputTypes, int nbInputs) const noexcept override; + + // IPluginV2 Methods + char const* getPluginType() const noexcept override; + char const* getPluginVersion() const noexcept override; + int getNbOutputs() const noexcept override; + int initialize() noexcept override; + void terminate() noexcept override; + size_t getSerializationSize() const noexcept override; + void serialize(void* buffer) const noexcept override; + void destroy() noexcept override; + void setPluginNamespace(char const* pluginNamespace) noexcept override; + char const* getPluginNamespace() const noexcept override; + +private: + const std::string mLayerName; + std::string mNamespace; + + int mNumHeads; + int mHeadSize; + float mSoftmaxScale; + nvinfer1::DataType mType; + + CUmodule mModule; + CUfunction mKernel; +}; + +class TritonFlashAttentionPluginCreator : public nvinfer1::IPluginCreator +{ +public: + TritonFlashAttentionPluginCreator(); + + char const* getPluginName() const noexcept override; + + char const* getPluginVersion() const noexcept override; + + nvinfer1::PluginFieldCollection const* getFieldNames() noexcept override; + + nvinfer1::IPluginV2* createPlugin(char const* name, nvinfer1::PluginFieldCollection const* fc) noexcept override; + + nvinfer1::IPluginV2* deserializePlugin( + char const* name, void const* serialData, size_t serialLength) noexcept override; + + void setPluginNamespace(char const* pluginNamespace) noexcept override; + + char const* getPluginNamespace() const noexcept override; + +private: + static nvinfer1::PluginFieldCollection mFC; + static std::vector mPluginAttributes; + std::string mNamespace; +}; + +} // namespace openai_triton::plugin diff --git a/examples/openai_triton/manual_plugin/build.py b/examples/openai_triton/manual_plugin/build.py new file mode 100644 index 000000000000..12b3ca883e7c --- /dev/null +++ b/examples/openai_triton/manual_plugin/build.py @@ -0,0 +1,137 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import argparse +import math +import time +from pathlib import Path + +import tensorrt as trt +from plugin import LAYER_NAME, FmhaLayer, get_engine_name + +import tensorrt_llm +from tensorrt_llm.builder import Builder, BuilderConfig +from tensorrt_llm.logger import logger +from tensorrt_llm.network import net_guard + + +def build_engine(builder: Builder, builder_config: BuilderConfig, + engine_name: str, args: argparse.Namespace) -> trt.IHostMemory: + ''' + + @brief: Build a TensorRT engine. + @param args: The cmd line arguments. + @return: The built or refitted engine. + ''' + + # Initialize Module + softmax_scale = 1.0 / math.sqrt(args.head_size) + layer = FmhaLayer(args.num_heads, args.head_size, softmax_scale, args.dtype) + + # Module -> Network + network = builder.create_network() + network.trt_network.name = engine_name + network.plugin_config.to_legacy_setting() + with net_guard(network): + # Prepare + inputs = layer.prepare_inputs(args.max_batch_size, args.max_seq_len) + # Forward + logger.debug(f'model inputs: {inputs}') + out = layer(*inputs) + out.trt_tensor.name = 'out' + + # Network -> Engine + engine = builder.build_engine(network, builder_config) + config_path = Path(args.output_dir) / 'config.json' + builder.save_config(builder_config, str(config_path)) + return engine + + +def build(args): + tensorrt_llm.logger.set_level(args.log_level) + output_dir = Path(args.output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + + builder = Builder() + cache = None + builder_config = builder.create_builder_config( + name=LAYER_NAME, + precision=args.dtype, + timing_cache=args.timing_cache if cache is None else cache, + profiling_verbosity=args.profiling_verbosity) + + engine_name = get_engine_name(args.head_size, args.dtype) + engine = build_engine(builder, builder_config, engine_name, args) + assert engine is not None + + engine_path = output_dir / engine_name + logger.info(f'Serializing engine to {str(engine_path)}...') + tik = time.time() + with engine_path.open('wb') as f: + f.write(engine) + tok = time.time() + t = time.strftime('%H:%M:%S', time.gmtime(tok - tik)) + logger.info(f'Engine serialized. Total time: {t}') + + ok = builder.save_timing_cache(builder_config, + Path(args.output_dir) / "model.cache") + assert ok, "Failed to save timing cache." + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + formatter_class=argparse.ArgumentDefaultsHelpFormatter) + parser.add_argument('--max_batch_size', type=int, default=4) + parser.add_argument('--max_seq_len', type=int, default=256) + parser.add_argument('--num_heads', type=int, default=8) + parser.add_argument('--head_size', type=int, default=64) + parser.add_argument('--dtype', + type=str, + default='float16', + choices=['float16', 'float32']) + parser.add_argument( + '--timing_cache', + type=str, + default='model.cache', + help='The path of to read timing cache from, will be ignored ' + 'if the file does not exist') + parser.add_argument( + '--profiling_verbosity', + type=str, + default='layer_names_only', + choices=['layer_names_only', 'detailed', 'none'], + help= + 'The profiling verbosity for the generated TRT engine. Set to detailed can inspect tactic choices and kernel parameters.' + ) + parser.add_argument('--log_level', type=str, default='info') + parser.add_argument( + '--output_dir', + type=str, + default='outputs', + help='The path to save the serialized engine files, timing cache ' + 'file and model configs') + args = parser.parse_args() + + logger.set_level(args.log_level) + logger.info('Parameters'.center(40, '=')) + for k, v in vars(args).items(): + logger.info(f' - {k.ljust(15, ".")}: {v}') + logger.info(''.center(40, '=')) + + tik = time.time() + logger.info('Build TensorRT engine.') + build(args) + tok = time.time() + t = time.strftime('%H:%M:%S', time.gmtime(tok - tik)) + logger.info(f'Total time of building TRT engine: {t}') diff --git a/examples/openai_triton/manual_plugin/fmha_triton.py b/examples/openai_triton/manual_plugin/fmha_triton.py new file mode 100644 index 000000000000..3e47dff263f0 --- /dev/null +++ b/examples/openai_triton/manual_plugin/fmha_triton.py @@ -0,0 +1,135 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Fused attention from triton tutorial. +Modified from the original implementation + - https://github.com/openai/triton/blob/main/python/tutorials/06-fused-attention.py +=============== + +This is a Triton implementation of the Flash Attention algorithm +(see: Dao et al., https://arxiv.org/pdf/2205.14135v2.pdf; Rabe and Staats https://arxiv.org/pdf/2112.05682v2.pdf) +""" + +import torch +import triton +import triton.language as tl + + +# yapf: disable +@triton.jit +def fused_attention_kernel( + Out, L, M, # outputs + Q, K, V, + sm_scale, + batch_size, num_heads, seq_len, + BLOCK_M: tl.constexpr, BLOCK_DMODEL: tl.constexpr, + BLOCK_N: tl.constexpr, +): + start_m = tl.program_id(0) + off_hz = tl.program_id(1) + stride_h = BLOCK_DMODEL * seq_len + + # initialize offsets + offs_m = start_m * BLOCK_M + tl.arange(0, BLOCK_M) + offs_n = tl.arange(0, BLOCK_N) + offs_d = tl.arange(0, BLOCK_DMODEL) + off_q = off_hz * stride_h + offs_m[:, None] * BLOCK_DMODEL + offs_d[None, :] + off_k = off_hz * stride_h + offs_n[None, :] * BLOCK_DMODEL + offs_d[:, None] + off_v = off_hz * stride_h + offs_n[:, None] * BLOCK_DMODEL + offs_d[None, :] + # Initialize pointers to Q, K, V + q_ptrs = Q + off_q + k_ptrs = K + off_k + v_ptrs = V + off_v + # initialize pointer to m and l + m_prev = tl.zeros([BLOCK_M], dtype=tl.float32) - float("inf") + l_prev = tl.zeros([BLOCK_M], dtype=tl.float32) + acc = tl.zeros([BLOCK_M, BLOCK_DMODEL], dtype=tl.float32) + # load q: it will stay in SRAM throughout + q = tl.load(q_ptrs) + # loop over k, v and update accumulator + for start_n in range(0, (start_m + 1) * BLOCK_M, BLOCK_N): + # -- compute qk ---- + k = tl.load(k_ptrs) + qk = tl.zeros([BLOCK_M, BLOCK_N], dtype=tl.float32) + qk += tl.dot(q, k) + qk *= sm_scale + qk = tl.where(offs_m[:, None] >= (start_n + offs_n[None, :]), qk, float("-inf")) + # compute new m + m_curr = tl.maximum(tl.max(qk, 1), m_prev) + # correct old l + l_prev *= tl.exp(m_prev - m_curr) + # attention weights + p = tl.exp(qk - m_curr[:, None]) + l_curr = tl.sum(p, 1) + l_prev + # rescale operands of matmuls + l_rcp = 1. / l_curr + p *= l_rcp[:, None] + acc *= (l_prev * l_rcp)[:, None] + # update acc + p = p.to(Q.dtype.element_ty) + v = tl.load(v_ptrs) + acc += tl.dot(p, v) + # update m_i and l_i + l_prev = l_curr + m_prev = m_curr + # update pointers + k_ptrs += BLOCK_N * BLOCK_DMODEL + v_ptrs += BLOCK_N * BLOCK_DMODEL + # rematerialize offsets to save registers + start_m = tl.program_id(0) + offs_m = start_m * BLOCK_M + tl.arange(0, BLOCK_M) + # write back l and m + l_ptrs = L + off_hz * seq_len + offs_m + m_ptrs = M + off_hz * seq_len + offs_m + tl.store(l_ptrs, l_prev) + tl.store(m_ptrs, m_prev) + # initialize pointers to output + offs_n = tl.arange(0, BLOCK_DMODEL) + off_o = off_hz * stride_h + offs_m[:, None] * BLOCK_DMODEL + offs_n[None, :] + out_ptrs = Out + off_o + tl.store(out_ptrs, acc) + + +def fused_attention(q, k, v, sm_scale, o_buf=None, l_buf=None, m_buf=None): + BLOCK = 128 if q.dtype == torch.float16 else 64 + # shape constraints + Lq, Lk, Lv = q.shape[-1], k.shape[-1], v.shape[-1] + assert Lq == Lk and Lk == Lv + assert Lk in {16, 32, 64, 128} + o = torch.empty_like(q) if o_buf is None else o_buf + grid = (triton.cdiv(q.shape[2], BLOCK), q.shape[0] * q.shape[1], 1) + shape = (q.shape[0] * q.shape[1], q.shape[2]) + L = torch.empty(shape, device=q.device, dtype=torch.float32) if l_buf is None else l_buf + m = torch.empty(shape, device=q.device, dtype=torch.float32) if m_buf is None else m_buf + + num_warps = 4 if Lk <= 64 else 8 + # Adjust num_stages for limited resource cases. + num_stages = 2 if torch.cuda.get_device_capability() >= (8, 0) else 1 + + fused_attention_kernel[grid]( + o, L, m, + q, k, v, + sm_scale, + q.shape[0], q.shape[1], q.shape[2], + # tl.constexpr + BLOCK_M=BLOCK, + BLOCK_N=BLOCK, + BLOCK_DMODEL=Lk, + num_warps=num_warps, + num_stages=num_stages, + ) + + return o +# yapf: enable diff --git a/examples/openai_triton/manual_plugin/plugin.py b/examples/openai_triton/manual_plugin/plugin.py new file mode 100644 index 000000000000..7009caaeb680 --- /dev/null +++ b/examples/openai_triton/manual_plugin/plugin.py @@ -0,0 +1,133 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import ctypes +from collections import OrderedDict +from pathlib import Path +from typing import List + +import numpy as np +import tensorrt as trt + +from tensorrt_llm._common import default_trtnet +from tensorrt_llm._utils import str_dtype_to_trt +from tensorrt_llm.functional import Tensor, _create_tensor +from tensorrt_llm.module import Module + +TRT_LLM_PLUGIN_NAMESPACE = 'tensorrt_llm' +LAYER_NAME = 'TritonFlashAttentionLayer' +FMHA_KERNEL_BLOCK_SIZE = 128 + + +def _load_triton_plugin_lib(): + triton_plugin_dir = Path(__file__).parent.absolute() + plugin_lib = triton_plugin_dir / 'build/libtrt_llm_custom_plugins.so' + handle = ctypes.CDLL(plugin_lib, mode=ctypes.RTLD_GLOBAL) + if handle is None: + raise ImportError('TensorRT LLM Triton Plugin is unavailable') + handle.initOpenAiTritonPlugins.argtypes = [ctypes.c_void_p, ctypes.c_char_p] + handle.initOpenAiTritonPlugins.restype = ctypes.c_bool + assert handle.initOpenAiTritonPlugins( + None, TRT_LLM_PLUGIN_NAMESPACE.encode('utf-8')) + + +_load_triton_plugin_lib() + + +def flash_attention_op(num_heads: int, head_size: int, softmax_scale: float, + inputs: List[trt.ITensor]) -> Tensor: + # Create a plugin instance. + plugin_creator = trt.get_plugin_registry().get_plugin_creator( + 'TritonFlashAttention', '1', TRT_LLM_PLUGIN_NAMESPACE) + assert plugin_creator is not None + + pfc = trt.PluginFieldCollection([ + trt.PluginField("num_heads", np.array([num_heads], np.int32), + trt.PluginFieldType.INT32), + trt.PluginField("head_size", np.array([head_size], np.int32), + trt.PluginFieldType.INT32), + trt.PluginField("softmax_scale", np.array([softmax_scale], np.float32), + trt.PluginFieldType.FLOAT32), + trt.PluginField("type_id", np.array([int(inputs[0].dtype)], np.int32), + trt.PluginFieldType.INT32) + ]) + plugin = plugin_creator.create_plugin("flash_attention", pfc) + layer = default_trtnet().add_plugin_v2(inputs, plugin) + return _create_tensor(layer.get_output(0), layer) + + +class FmhaLayer(Module): + + def __init__(self, num_heads: int, head_size: int, softmax_scale: float, + dtype: str): + super().__init__() + self.num_heads = num_heads + self.head_size = head_size + self.softmax_scale = softmax_scale + self.dtype = str_dtype_to_trt(dtype) + + def forward(self, Q: Tensor, K: Tensor, V: Tensor): + inputs = [Q, K, V] + out = flash_attention_op(num_heads=self.num_heads, + head_size=self.head_size, + softmax_scale=self.softmax_scale, + inputs=[p.trt_tensor for p in inputs]) + out.mark_output('out', self.dtype) + return out + + def prepare_inputs(self, max_batch_size: int, max_len: int) -> List[Tensor]: + ''' + + @brief: Prepare inputs Tensors for the model, the given sizes are used to + determine the ranges of the dimensions of when using TRT dynamic shapes. + + @return: a list contains values which can be fed into the self.forward() + ''' + + bs_range = [1, (max_batch_size + 1) // 2, max_batch_size] + max_len_range = [1, (max_len + 1) // 2, max_len] + + dynamic_shape = [-1, self.num_heads, -1, self.head_size] + Q = Tensor(name='Q', + dtype=self.dtype, + shape=dynamic_shape, + dim_range=OrderedDict([ + ('batch_size', [bs_range]), + ('num_heads', [self.num_heads]), + ('seq_len', [max_len_range]), + ('head_size', [self.head_size]), + ])) + K = Tensor(name='K', + dtype=self.dtype, + shape=dynamic_shape, + dim_range=OrderedDict([ + ('batch_size', [bs_range]), + ('num_heads', [self.num_heads]), + ('seq_len', [max_len_range]), + ('head_size', [self.head_size]), + ])) + V = Tensor(name='V', + dtype=self.dtype, + shape=dynamic_shape, + dim_range=OrderedDict([ + ('batch_size', [bs_range]), + ('num_heads', [self.num_heads]), + ('seq_len', [max_len_range]), + ('head_size', [self.head_size]), + ])) + return [Q, K, V] + + +def get_engine_name(head_size, dtype): + return f'{LAYER_NAME}_{FMHA_KERNEL_BLOCK_SIZE}_d{head_size}_{dtype}.engine' diff --git a/examples/openai_triton/manual_plugin/run.py b/examples/openai_triton/manual_plugin/run.py new file mode 100644 index 000000000000..ec7cf4dd5600 --- /dev/null +++ b/examples/openai_triton/manual_plugin/run.py @@ -0,0 +1,170 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import argparse +import json +import math +from pathlib import Path + +import torch +from fmha_triton import fused_attention +from plugin import get_engine_name + +from tensorrt_llm import profiler +from tensorrt_llm._deprecation import emit_engine_arch_deprecation +from tensorrt_llm._utils import (str_dtype_to_torch, str_dtype_to_trt, + trt_dtype_to_torch) +from tensorrt_llm.logger import logger +from tensorrt_llm.runtime.session import Session, TensorInfo + + +def run(engine_dir, + batch_size, + seq_len, + num_heads, + head_size, + do_benchmark=False): + # Load trt engine. + engine_dir = Path(engine_dir) + config_path = engine_dir / 'config.json' + with config_path.open('r') as f: + config = json.load(f) + dtype = config['builder_config']['precision'] + serialize_path = engine_dir / get_engine_name(head_size, dtype) + + with open(serialize_path, 'rb') as f: + session = Session.from_serialized_engine(f.read()) + + # Prepare input tensors. + torch_dtype = str_dtype_to_torch(dtype) if isinstance(dtype, str) else dtype + shape = (batch_size, num_heads, seq_len, head_size) + q = torch.normal(mean=0.1, + std=0.2, + size=shape, + dtype=torch_dtype, + device='cuda') + k = torch.normal(mean=0.4, + std=0.2, + size=shape, + dtype=torch_dtype, + device='cuda') + v = torch.normal(mean=0.3, + std=0.2, + size=shape, + dtype=torch_dtype, + device='cuda') + inputs = {'Q': q, 'K': k, 'V': v} + + # Prepare output tensors. + output_info = session.infer_shapes([ + TensorInfo(name, str_dtype_to_trt(dtype), tensor.shape) + for name, tensor in inputs.items() + ]) + logger.debug(f'output info {output_info}') + outputs = { + t.name: + torch.empty(tuple(t.shape), + dtype=trt_dtype_to_torch(t.dtype), + device='cuda') + for t in output_info + } + + # Execute model inference + stream = torch.cuda.Stream() + ok = session.run(inputs=inputs, outputs=outputs, stream=stream.cuda_stream) + assert ok, 'Engine execution failed' + + # Sanity check + stream.synchronize() + sm_scale = 1.0 / math.sqrt(head_size) + ref = fused_attention(q, k, v, sm_scale) + out = outputs["out"] + logger.debug( + f'Out: vals: {out.view(1, -1)} abs_sum: {out.float().abs().sum()}') + logger.debug( + f'Ref: vals: {ref.view(1, -1)} abs_sum: {ref.float().abs().sum()}') + torch.testing.assert_close(out, ref) + + if do_benchmark: + n_repeats = 10 + + # For fair comparison, pre-allocate buffers as trt plugin does. + shape = (q.shape[0] * q.shape[1], q.shape[2]) + L = torch.empty(shape, device=q.device, dtype=torch.float32) + m = torch.empty(shape, device=q.device, dtype=torch.float32) + o = torch.empty_like(q) + + # Triton warm-up + fused_attention(q, k, v, sm_scale, l_buf=L, m_buf=m, o_buf=o) + stream.synchronize() + for _ in range(n_repeats): + profiler.start('Triton') + fused_attention(q, k, v, sm_scale, l_buf=L, m_buf=m, o_buf=o) + stream.synchronize() + profiler.stop('Triton') + + # TRT warm-up + stream.synchronize() + ok = session.run(inputs=inputs, + outputs=outputs, + stream=stream.cuda_stream) + stream.synchronize() + for _ in range(n_repeats): + profiler.start('TRT Plugin') + ok = session.run(inputs=inputs, + outputs=outputs, + stream=stream.cuda_stream) + stream.synchronize() + profiler.stop('TRT Plugin') + assert ok + profiler.summary() + + +if __name__ == '__main__': + emit_engine_arch_deprecation("run.py") + parser = argparse.ArgumentParser( + formatter_class=argparse.ArgumentDefaultsHelpFormatter) + parser.add_argument('--batch_size', type=int, default=4) + parser.add_argument('--seq_len', type=int, default=128) + parser.add_argument('--num_heads', type=int, default=8) + parser.add_argument('--head_size', type=int, default=64) + parser.add_argument('--log_level', type=str, default='info') + parser.add_argument( + '--engine_dir', + type=Path, + default='outputs', + help='The directory where serialized engine files locate.') + parser.add_argument( + '--benchmark', + action='store_true', + help='Do performance benchmark compared to triton baseline.') + args = parser.parse_args() + + logger.set_level(args.log_level) + logger.info('Parameters'.center(40, '=')) + for k, v in vars(args).items(): + logger.info(f' - {k.ljust(15, ".")}: {v}') + logger.info(''.center(40, '=')) + + assert args.engine_dir.exists(), \ + f"Engine file {str(args.engine_dir)} doesn't exists." + + logger.info('Inference using the built TensorRT engine.') + run(args.engine_dir, + args.batch_size, + args.seq_len, + args.num_heads, + args.head_size, + do_benchmark=args.benchmark) + logger.info('Done.') diff --git a/examples/openai_triton/manual_plugin/tritonPlugins.cpp b/examples/openai_triton/manual_plugin/tritonPlugins.cpp new file mode 100644 index 000000000000..27b1ece08448 --- /dev/null +++ b/examples/openai_triton/manual_plugin/tritonPlugins.cpp @@ -0,0 +1,133 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "NvInferRuntime.h" +#include "TritonFlashAttentionPlugin.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace +{ + +// This singleton ensures that each plugin is only registered once for a given +// namespace and type, and attempts of duplicate registration are ignored. +class TritonPluginCreatorRegistry +{ +public: + static TritonPluginCreatorRegistry& getInstance() + { + static TritonPluginCreatorRegistry instance; + return instance; + } + + template + void addPluginCreator(void* logger, char const* libNamespace) + { + // Make accesses to the plugin creator registry thread safe + std::lock_guard lock(mRegistryLock); + + std::string errorMsg; + std::string verboseMsg; + + std::unique_ptr pluginCreator{new CreatorType{}}; + pluginCreator->setPluginNamespace(libNamespace); + + nvinfer1::ILogger* trtLogger = static_cast(logger); + std::string pluginType = std::string{pluginCreator->getPluginNamespace()} + + "::" + std::string{pluginCreator->getPluginName()} + " version " + + std::string{pluginCreator->getPluginVersion()}; + + if (mRegistryList.find(pluginType) == mRegistryList.end()) + { + bool status = getPluginRegistry()->registerCreator(*pluginCreator, libNamespace); + if (status) + { + mRegistry.push(std::move(pluginCreator)); + mRegistryList.insert(pluginType); + verboseMsg = "Registered plugin creator - " + pluginType; + } + else + { + errorMsg = "Could not register plugin creator - " + pluginType; + } + } + else + { + verboseMsg = "Plugin creator already registered - " + pluginType; + } + + if (trtLogger) + { + if (!errorMsg.empty()) + { + trtLogger->log(nvinfer1::ILogger::Severity::kERROR, errorMsg.c_str()); + } + + if (!verboseMsg.empty()) + { + trtLogger->log(nvinfer1::ILogger::Severity::kVERBOSE, verboseMsg.c_str()); + } + } + } + + ~TritonPluginCreatorRegistry() + { + std::lock_guard lock(mRegistryLock); + + // Release pluginCreators in LIFO order of registration. + while (!mRegistry.empty()) + { + mRegistry.pop(); + } + mRegistryList.clear(); + } + +private: + TritonPluginCreatorRegistry() {} + + std::mutex mRegistryLock; + std::stack> mRegistry; + std::unordered_set mRegistryList; + +public: + TritonPluginCreatorRegistry(TritonPluginCreatorRegistry const&) = delete; + void operator=(TritonPluginCreatorRegistry const&) = delete; +}; + +template +void initializeTritonPlugin(void* logger, char const* libNamespace) +{ + TritonPluginCreatorRegistry::getInstance().addPluginCreator(logger, libNamespace); +} + +} // namespace + +// New Plugin APIs + +extern "C" +{ + bool initOpenAiTritonPlugins(void* logger, char const* libNamespace) + { + initializeTritonPlugin(logger, libNamespace); + return true; + } +} // extern "C" diff --git a/examples/openai_triton/plugin_autogen/README.md b/examples/openai_triton/plugin_autogen/README.md new file mode 100644 index 000000000000..0c046330fa0b --- /dev/null +++ b/examples/openai_triton/plugin_autogen/README.md @@ -0,0 +1,97 @@ +# Integrating Triton Kernel with TensorRT Plugin Generator + +In the previous [OpenAI Triton Plugin in TensorRT-LLM](../../openai_triton/README.md) tutorial, it is demonstrated how to integrate a Triton kernel by manually writing a TensorRT plugin in C++ as well as a Python wrapper. In the latest TensorRT-LLM, we now have an end-to-end tool called PluginGen that simplifies this process. All you need to do is providing a plugin configuration. + +In this example, we will introduce the usage of the PluginGen tool and demonstrate the integration of the [Fused Attention](../openai_triton/fmha_triton.py) kernel. + + +To use the feature, you need a Triton version posterior to the [d0c35b3](https://github.com/openai/triton/commit/d0c35b3b7d6badf0c0d56a821dddab7ace73b4de) commit +and this example has been tested on the [d4644d6](https://github.com/openai/triton/tree/d4644d6cb3ae674e1f15932cac1f28104795744f) commit. + +## Introduction to the PluginGen Toolkit + +The PluginGen script can be found at `tensorrt_llm/tools/triton_integration/plugin_gen.py`. Its usage is as follows: + +```sh +usage: plugin_gen.py [-h] --workspace WORKSPACE --kernel_config KERNEL_CONFIG [--tensorrt_llm_include_path TENSORRT_LLM_INCLUDE_PATH] +``` + +There are three command-line arguments: + +1. `workspace`: This is the root directory to hold the temporary generation files. PluginGen should not alter anything outside of the workspace, +2. `kernel_config`: This is a Python file that holds a variable called `KERNELS` of type `List[KernelMetaData]`. PluginGen can process one or more kernels at a time, +3. `tensorrt_llm_include_path`: This is the path to the TensorRT LLM include directory. It is used to include the TensorRT LLM header files in the generated plugin. + +You can refer to [./kernel_config.py](./kernel_config.py) for an example of `KernelMetaData` for the Fused Attention kernel. It contains several fields: + +1. `ios` (short for "input and outputs"): This holds all the metadata of the inputs and outputs of the Triton kernel, including the data type, shape, and the name of the tensor. There are several kinds of arguments: + - `InputArg`: A common variable for this kernel. + - `OutputArg`: An output of the kernel. + - `ParamArg`: A special input that is a constant; it will be mapped to a PluginField in the generated plugin. + - `DimSizeArg`: A special input that is an expression of the input tensors' shape size; it requires an inference rule to compute the value. +2. `shape_infer_rules`: This field contains two types of rules: + a) Rules for deducing the shape of the output tensors from the input tensors. The syntax is like `input0[dim_names], input1[dim_names] -> output0[dim_names]`. + b) Rules for inferring `DimSizeArg`. The syntax is like `input0[dim_names]: some_dim_expression -> arg_name`. + +The user should provide the kernel configurations as well as the Triton kernel script, and the PluginGen toolkit will handle the following steps: + +1. Trigger the Triton AOT tool to obtain the necessary C files. +2. Generate the C++ code for a TensorRT plugin. +3. Generate the CMAKE code for compiling all the C/C++ files. +4. Perform the compilation and generate `libtriton_plugins.so`. +5. Generate a `functional.py` containing a Python wrapper for this plugin. + +After the generation, you should have `libtriton_plugins.so` and `functional.py` in the workspace. You can use them to integrate the Triton kernel by simply using the corresponding Python methods in the generated `functional.py` during the model-building stage, just like other layers located in the TensorRT LLM built-in `functional.py`. + +## End-to-End Example for FHMA Kernel Integration + +In this section, we will demonstrate the integration of the Fused Attention kernel. The steps are as follows: + +### Pre-Stage: Install Triton with a Specific Version + +In case the Triton AOT tool's update breaks compatibility, we recommend installing a specific version of Triton. The commit we tested is [d4644d6](https://github.com/openai/triton/tree/d4644d6cb3ae674e1f15932cac1f28104795744f). + +Install Triton with the following commands: + +```sh +git clone https://github.com/openai/triton +cd triton/python/ +pip install cmake && pip install . +cd - +``` + +### Step 1: Prepare the Configuration for FHMA + +To instruct the PluginGen toolkit on how to generate the plugin, please provide a Python file containing the metadata of the kernels. You can refer to [./kernel_config.py](./kernel_config.py) for an example of preparing `KernelMetaData` for the Fused Attention kernel. + +### Step 2: Run the PluginGen Tool and Generate the Plugin + +```sh +python3 {GIT_ROOT_DIR}/tensorrt_llm/tools/plugin_gen/plugin_gen.py --workspace ./tmp --kernel_config ./kernel_config.py +``` + +PluginGen will generate all the necessary files within the `./tmp` directory. The final output will be located in the `./tmp/output` directory, where you should ideally find two files: + +``` +-rw-r--r-- 1 1001 1001 2163 Sep 21 17:13 functional.py +-rwxr-xr-x 1 1001 1001 3748464 Sep 21 17:13 libtriton_plugins.so +``` + +### Post-Stage: Use the Plugin + +To use the plugin in a TensorRT LLM model, please refer to the generated `output/functional.py`. It should contain Python wrappers for all the plugins. To use the plugins, first import `functional.py` and then use the corresponding Python methods to build the model. + +For an example of using the Fused Attention plugin in a model, please refer to [build_engine.py](./build_engine.py) for building the TensorRT engine and [run_engine.py](./run_engine.py) for running the engine in the runtime. + +To run the example, you can use the following commands: + +```sh +# copy the triton script to the current directory +cp ../manual_plugin/fmha_triton.py . + +# build the TensorRT engine +python3 build_engine.py + +# run the engine +python3 run_engine.py +``` diff --git a/examples/openai_triton/plugin_autogen/build_engine.py b/examples/openai_triton/plugin_autogen/build_engine.py new file mode 100644 index 000000000000..23b829f0b7e4 --- /dev/null +++ b/examples/openai_triton/plugin_autogen/build_engine.py @@ -0,0 +1,206 @@ +import argparse +import math +# include plugins +# yapf: disable +import os +import sys +import time +from pathlib import Path +from typing import List, OrderedDict + +import tensorrt as trt + +# from plugin import LAYER_NAME, FmhaLayer, get_engine_name +import tensorrt_llm +from tensorrt_llm import Module, str_dtype_to_trt +from tensorrt_llm.builder import Builder, BuilderConfig +from tensorrt_llm.functional import Tensor +from tensorrt_llm.logger import logger +from tensorrt_llm.network import net_guard + +sys.path.append(os.environ.get('PLUGIN_GEN_WORKSPACE', './tmp')) +from functional import fused_attention_kernel # isort:skip +# yapf: enable + + +def get_engine_name(head_size: int, dtype: str) -> str: + return f'fmha_{head_size}_{dtype}.engine' + + +class FmhaLayer(Module): + + def __init__(self, num_heads: int, head_size: int, softmax_scale: float): + super().__init__() + self.num_heads = num_heads + self.head_size = head_size + self.softmax_scale = softmax_scale + self.dtype = str_dtype_to_trt('float16') + + def forward(self, Q: Tensor, K: Tensor, V: Tensor): + inputs = [Q, K, V] + Out, L, M = fused_attention_kernel(self.softmax_scale, self.num_heads, + *[p.trt_tensor for p in inputs]) + Out.mark_output('out', self.dtype) + L.mark_output('L', self.dtype) + M.mark_output('M', self.dtype) + return Out, L, M + + def prepare_inputs(self, max_batch_size: int, max_len: int) -> List[Tensor]: + ''' + + @brief: Prepare inputs Tensors for the model, the given sizes are used to + determine the ranges of the dimensions of when using TRT dynamic shapes. + + @return: a list contains values which can be fed into the self.forward() + ''' + + bs_range = [1, (max_batch_size + 1) // 2, max_batch_size] + max_len_range = [1, (max_len + 1) // 2, max_len] + + dynamic_shape = [-1, self.num_heads, -1, self.head_size] + Q = Tensor(name='Q', + dtype=trt.float16, + shape=dynamic_shape, + dim_range=OrderedDict([ + ('batch_size', [bs_range]), + ('num_heads', [self.num_heads]), + ('seq_len', [max_len_range]), + ('head_size', [self.head_size]), + ])) + K = Tensor(name='K', + dtype=trt.float16, + shape=dynamic_shape, + dim_range=OrderedDict([ + ('batch_size', [bs_range]), + ('num_heads', [self.num_heads]), + ('seq_len', [max_len_range]), + ('head_size', [self.head_size]), + ])) + V = Tensor(name='V', + dtype=trt.float16, + shape=dynamic_shape, + dim_range=OrderedDict([ + ('batch_size', [bs_range]), + ('num_heads', [self.num_heads]), + ('seq_len', [max_len_range]), + ('head_size', [self.head_size]), + ])) + return [Q, K, V] + + +def build_engine(builder: Builder, builder_config: BuilderConfig, + engine_name: str, args: argparse.Namespace) -> trt.IHostMemory: + ''' + @brief: Build a TensorRT engine. + @param args: The cmd line arguments. + @return: The built or refitted engine. + ''' + + # Initialize Module + softmax_scale = 1.0 / math.sqrt(args.head_size) + layer = FmhaLayer(args.num_heads, args.head_size, softmax_scale) + + # Module -> Network + network = builder.create_network() + network.trt_network.name = engine_name + network.plugin_config.to_legacy_setting() + with net_guard(network): + # Prepare + inputs = layer.prepare_inputs(args.max_batch_size, args.max_seq_len) + # Forward + logger.debug(f'model inputs: {inputs}') + layer(*inputs) + + print('dot:') + print(network.to_dot()) + + layer = network.get_layer_by_name(next( + network.get_layers()).name).as_layer() + print('layer', layer.plugin.plugin_type) + print('layer', layer.plugin.plugin_version) + print('layer', layer.plugin.plugin_namespace) + + # Network -> Engine + engine = builder.build_engine(network, builder_config) + config_path = Path(args.output_dir) / 'config.json' + builder.save_config(builder_config, str(config_path)) + return engine + + +def build(args): + tensorrt_llm.logger.set_level(args.log_level) + output_dir = Path(args.output_dir) + output_dir.mkdir(parents=True, exist_ok=True) + + builder = Builder() + cache = None + builder_config = builder.create_builder_config( + name='fmha_triton', + precision=args.dtype, + timing_cache=args.timing_cache if cache is None else cache, + profiling_verbosity=args.profiling_verbosity) + + engine_name = get_engine_name(args.head_size, args.dtype) + engine = build_engine(builder, builder_config, engine_name, args) + assert engine is not None + + engine_path = output_dir / engine_name + logger.info(f'Serializing engine to {str(engine_path)}...') + tik = time.time() + with engine_path.open('wb') as f: + f.write(engine) + tok = time.time() + t = time.strftime('%H:%M:%S', time.gmtime(tok - tik)) + logger.info(f'Engine serialized. Total time: {t}') + + ok = builder.save_timing_cache(builder_config, + Path(args.output_dir) / "model.cache") + assert ok, "Failed to save timing cache." + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + formatter_class=argparse.ArgumentDefaultsHelpFormatter) + parser.add_argument('--max_batch_size', type=int, default=4) + parser.add_argument('--max_seq_len', type=int, default=256) + parser.add_argument('--num_heads', type=int, default=8) + parser.add_argument('--head_size', type=int, default=64) + parser.add_argument('--dtype', + type=str, + default='float16', + choices=['float16', 'float32']) + parser.add_argument( + '--timing_cache', + type=str, + default='model.cache', + help='The path of to read timing cache from, will be ignored ' + 'if the file does not exist') + parser.add_argument( + '--profiling_verbosity', + type=str, + default='layer_names_only', + choices=['layer_names_only', 'detailed', 'none'], + help= + 'The profiling verbosity for the generated TRT engine. Set to detailed can inspect tactic choices and kernel parameters.' + ) + parser.add_argument('--log_level', type=str, default='info') + parser.add_argument( + '--output_dir', + type=str, + default='outputs', + help='The path to save the serialized engine files, timing cache ' + 'file and model configs') + args = parser.parse_args() + + logger.set_level(args.log_level) + logger.info('Parameters'.center(40, '=')) + for k, v in vars(args).items(): + logger.info(f' - {k.ljust(15, ".")}: {v}') + logger.info(''.center(40, '=')) + + tik = time.time() + logger.info('Build TensorRT engine.') + build(args) + tok = time.time() + t = time.strftime('%H:%M:%S', time.gmtime(tok - tik)) + logger.info(f'Total time of building TRT engine: {t}') diff --git a/examples/openai_triton/plugin_autogen/kernel_config.py b/examples/openai_triton/plugin_autogen/kernel_config.py new file mode 100644 index 000000000000..93d64c187d61 --- /dev/null +++ b/examples/openai_triton/plugin_autogen/kernel_config.py @@ -0,0 +1,55 @@ +import os + +import torch + +from tensorrt_llm.tools.plugin_gen.core import * + +openai_triton_example_root = os.path.join( + os.path.dirname(os.path.abspath(__file__)), "..", "manual_plugin") + + +def get_fmha_kernel_meta_data(): + block_size = 128 + num_stages = 2 if torch.cuda.get_device_capability() >= (8, 0) else 1 + + return KernelMetaData( + kernel_name='fused_attention_kernel', + ios=[ + # outputs + OutputArg('Out', Type('tensor[fp16]'), hints=['16', '16']), + OutputArg('L', Type('tensor[fp32]'), hints=['16', '16']), + OutputArg('M', Type('tensor[fp32]'), hints=['16', '16']), + # inputs + InputArg('Q', Type('tensor[fp16]'), hints=['16', '16']), + InputArg('K', Type('tensor[fp16]'), hints=['16', '16']), + InputArg('V', Type('tensor[fp16]'), hints=['16', '16']), + ParamArg('sm_scale', Type('fp32')), + DimSizeArg('batch_size'), + ParamArg('num_heads', Type('i32')), + DimSizeArg('seq_len', hints=['', '16']), + # constexprs + Constexpr(block_size), + Constexpr(64), + Constexpr(block_size), + ], + shape_infer_rules=[ + # The following rules helps to deduce the shapes of the output tensors + "Q[*] -> Out[*]", + "Q[m,n,k,*] -> L[m,n,k]", + "Q[m,n,k,*] -> M[m,n,k]", + + # The following rules helps to deduce both DimSizeArgs: batch_size and seq_len + "Q[m,n,k,*] : m -> batch_size", + "Q[m,n,k,*] : k -> seq_len", + ], + version=0, + kernel_file=f'{openai_triton_example_root}/fmha_triton.py', + num_warps=4, + num_stages=num_stages, + grid_dims=(f"(seq_len + {block_size-1}) / {block_size}", + "batch_size * num_heads", "1")) + + +KERNELS = [ + get_fmha_kernel_meta_data(), +] diff --git a/examples/openai_triton/plugin_autogen/run_engine.py b/examples/openai_triton/plugin_autogen/run_engine.py new file mode 100644 index 000000000000..9a438d1d8d87 --- /dev/null +++ b/examples/openai_triton/plugin_autogen/run_engine.py @@ -0,0 +1,170 @@ +import argparse +import json +import math +# include plugins +# yapf: disable +import sys +from pathlib import Path + +import torch +from fmha_triton import fused_attention + +from tensorrt_llm import profiler +from tensorrt_llm._utils import (str_dtype_to_torch, str_dtype_to_trt, + trt_dtype_to_torch) +from tensorrt_llm.logger import logger +from tensorrt_llm.runtime.session import Session, TensorInfo + +# from tensorrt_llm.plugin import get_engine_name + + +sys.path.append('./tmp') +from functional import fused_attention_kernel # isort:skip +# yapf: enable + + +def get_engine_name(head_size, dtype): + return f'fmha_{head_size}_{dtype}.engine' + + +def run(engine_dir, + batch_size, + seq_len, + num_heads, + head_size, + do_benchmark=False): + # Load trt engine. + engine_dir = Path(engine_dir) + config_path = engine_dir / 'config.json' + with config_path.open('r') as f: + config = json.load(f) + dtype = config['builder_config']['precision'] + serialize_path = engine_dir / get_engine_name(head_size, dtype) + + with open(serialize_path, 'rb') as f: + session = Session.from_serialized_engine(f.read()) + + # Prepare input tensors. + torch_dtype = str_dtype_to_torch(dtype) if isinstance(dtype, str) else dtype + shape = (batch_size, num_heads, seq_len, head_size) + q = torch.normal(mean=0.1, + std=0.2, + size=shape, + dtype=torch_dtype, + device='cuda') + k = torch.normal(mean=0.4, + std=0.2, + size=shape, + dtype=torch_dtype, + device='cuda') + v = torch.normal(mean=0.3, + std=0.2, + size=shape, + dtype=torch_dtype, + device='cuda') + batch_size = q.shape[0] + seq_len = q.shape[2] + + inputs = {'Q': q, 'K': k, 'V': v} + + # Prepare output tensors. + output_info = session.infer_shapes([ + TensorInfo(name, str_dtype_to_trt(dtype), tensor.shape) + for name, tensor in inputs.items() + ]) + logger.debug(f'output info {output_info}') + outputs = { + t.name: + torch.empty(tuple(t.shape), + dtype=trt_dtype_to_torch(t.dtype), + device='cuda') + for t in output_info + } + + # Execute model inference + stream = torch.cuda.current_stream() + ok = session.run(inputs=inputs, outputs=outputs, stream=stream.cuda_stream) + assert ok, 'Engine execution failed' + + # Sanity check + stream.synchronize() + sm_scale = 1.0 / math.sqrt(head_size) + ref = fused_attention(q, k, v, sm_scale) + out = outputs["out"] + logger.debug( + f'Out: vals: {out.view(1, -1)} abs_sum: {out.float().abs().sum()}') + logger.debug( + f'Ref: vals: {ref.view(1, -1)} abs_sum: {ref.float().abs().sum()}') + torch.testing.assert_close(out, ref) + + if do_benchmark: + n_repeats = 10 + + # For fair comparison, pre-allocate buffers as trt plugin does. + shape = (q.shape[0] * q.shape[1], q.shape[2]) + L = torch.empty(shape, device=q.device, dtype=torch.float32) + m = torch.empty(shape, device=q.device, dtype=torch.float32) + o = torch.empty_like(q) + + # Triton warm-up + fused_attention(q, k, v, sm_scale, l_buf=L, m_buf=m, o_buf=o) + stream.synchronize() + for _ in range(n_repeats): + profiler.start('Triton') + fused_attention(q, k, v, sm_scale, l_buf=L, m_buf=m, o_buf=o) + stream.synchronize() + profiler.stop('Triton') + + # TRT warm-up + stream.synchronize() + ok = session.run(inputs=inputs, + outputs=outputs, + stream=stream.cuda_stream) + stream.synchronize() + for _ in range(n_repeats): + profiler.start('TRT Plugin') + ok = session.run(inputs=inputs, + outputs=outputs, + stream=stream.cuda_stream) + stream.synchronize() + profiler.stop('TRT Plugin') + assert ok + profiler.summary() + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + formatter_class=argparse.ArgumentDefaultsHelpFormatter) + parser.add_argument('--batch_size', type=int, default=4) + parser.add_argument('--seq_len', type=int, default=128) + parser.add_argument('--num_heads', type=int, default=8) + parser.add_argument('--head_size', type=int, default=64) + parser.add_argument('--log_level', type=str, default='info') + parser.add_argument( + '--engine_dir', + type=Path, + default='outputs', + help='The directory where serialized engine files locate.') + parser.add_argument( + '--benchmark', + action='store_true', + help='Do performance benchmark compared to triton baseline.') + args = parser.parse_args() + + logger.set_level(args.log_level) + logger.info('Parameters'.center(40, '=')) + for k, v in vars(args).items(): + logger.info(f' - {k.ljust(15, ".")}: {v}') + logger.info(''.center(40, '=')) + + assert args.engine_dir.exists(), \ + f"Engine file {str(args.engine_dir)} doesn't exists." + + logger.info('Inference using the built TensorRT engine.') + run(args.engine_dir, + args.batch_size, + args.seq_len, + args.num_heads, + args.head_size, + do_benchmark=args.benchmark) + logger.info('Done.') diff --git a/examples/python_plugin/README.md b/examples/python_plugin/README.md new file mode 100644 index 000000000000..8079d381109a --- /dev/null +++ b/examples/python_plugin/README.md @@ -0,0 +1,120 @@ +# TensorRT LLM Python Plugin + +TensorRT LLM provides a Python plugin interface to integrate TensorRT LLM with pure Python. + ++ `openai_triton_plugin`: plugin package ++ `build_lookup.py`: Build a TensorRT engine with TensorRT LLM Python plugin ++ `run_lookup.py`: Run the engine and compare the result with PyTorch + +## Plugin Definition + +The following code shows how to create a look-up plugin. +We only need to do a few things to define a TensorRT LLM plugin. + +1. Inherit the `PluginBase`. +2. Register the plugin class to TensorRT LLM by using `@trtllm_plugin("your_plugin_name")`. +3. Define an `__init__` function and initialize the base class. +4. Define a shape and dtype inference function. +5. Define the compute flow. + +```python +@trtllm_plugin("TritonLookUp") +class LookUpPlugin(PluginBase): + + def __init__(self, use_torch_tensor, fp32_output): + super().__init__() + self.use_torch_tensor = use_torch_tensor + self.fp32_output = fp32_output + + def shape_dtype_inference(self, inputs: Sequence[SymTensor]) -> SymTensor: + shape = inputs[1].shape + shape[0] = inputs[0].shape[0] + inputs[1].shape[0] - inputs[1].shape[0] + return SymTensor( + inputs[1].dtype if not self.fp32_output else torch.float32, shape) + + def forward(self, inputs: Sequence[TensorWrapper], + outputs: Sequence[TensorWrapper]): + assert len(inputs) == 2 + assert inputs[0].dtype in [torch.int32 or torch.int64] + assert inputs[1].dtype in [torch.float32, torch.float16, torch.bfloat16] + assert (self.fp32_output and outputs[0].dtype + == torch.float32) or outputs[0].dtype == inputs[1].dtype + + x = inputs[0] + y = inputs[1] + z = outputs[0] + if self.use_torch_tensor: + x = convert_to_torch_tensor(x) + y = convert_to_torch_tensor(y) + z = convert_to_torch_tensor(z) + MAX_BLOCK_NUM = 65536 + MAX_BLOCK_SIZE = 512 + grid = lambda meta: (min(MAX_BLOCK_NUM, x.shape[0]) * min( + MAX_BLOCK_SIZE, y.shape[1]), ) + lookup_kernel[grid](x, y, z, y.shape[0], y.shape[1], x.shape[0]) + +``` + +## Adding a TensorRT LLM Plugin to a Network + +You only need an instance of the plugin object and then call it with `tensorrt_llm.Tensor` as input arguments. + +```python +builder = tensorrt_llm.Builder() +network = builder.create_network() +with tensorrt_llm.net_guard(network): + x = Tensor(name='x', + shape=index_shape, + dtype=tensorrt_llm.str_dtype_to_trt('int32')) + y = Tensor(name='y', + shape=(vocab_size, n_embed), + dtype=torch_dtype_to_trt(dtype)) + + def lookup(x, y): + lookup_plugin = LookUpPlugin(False) + return lookup_plugin(x, y) + + output = lookup(x, y) + output.mark_output('output', torch_dtype_to_str(dtype)) +``` + +## Plugin Code Structure + +Because TensorRT LLM performs plugin registration when importing the custom TensorRT LLM plugin, there are some code structure conventions to register the plugin at runtime. + +```text +plugin_lib +├──__init__.py +├──lookup_plugin.py +└──lookup_kernel.py +``` + +The `__init__.py` file imports all the plugins in the plugin package. +With this convention, users only need to import the plugin package to register the plugins and do not need to manually import them. + +```python +# __init__.py +from .lookup_plugin import LookUpPlugin + +__all__ = ["LookUpPlugin"] +``` + +## Deserialize an Engine with TensorRT LLM Plugin + +During deserialization, TensorRT needs to find the user-defined plugin. Thus, we need to import the plugin once to register them. If the plugin follows the code structure convention, users only need to import that package to register all the custom plugins. + +```python +from tensorrt_llm.runtime.session import Session, TensorInfo + +import openai_triton_plugin # isort: skip + +if __name__ == "__main__": + + def run_engine(dtype): + output_dir = Path('tmp') / torch_dtype_to_str(dtype) + + engine_path = output_dir / "lookup.engine" + + with engine_path.open('rb') as f: + session = Session.from_serialized_engine(f.read()) +``` diff --git a/examples/python_plugin/build_lookup.py b/examples/python_plugin/build_lookup.py new file mode 100644 index 000000000000..48cccd58882e --- /dev/null +++ b/examples/python_plugin/build_lookup.py @@ -0,0 +1,61 @@ +from pathlib import Path + +import torch +from plugin_lib import LookUpPlugin + +import tensorrt_llm +from tensorrt_llm import Tensor +from tensorrt_llm._utils import torch_dtype_to_str, torch_dtype_to_trt + +if __name__ == "__main__": + + # meta data + batch_size = 10 + vocab_size = 1000 + n_embed = 1024 + + # test data + ## input index + index_shape = (batch_size, ) + index_data = torch.randint(0, vocab_size, index_shape, + dtype=torch.int32).cuda() + + def test(dtype): + builder = tensorrt_llm.Builder() + builder.strongly_typed = True + network = builder.create_network() + with tensorrt_llm.net_guard(network): + x = Tensor( + name="x", + shape=index_shape, + dtype=tensorrt_llm.str_dtype_to_trt("int32"), + ) + y = Tensor(name="y", + shape=(vocab_size, n_embed), + dtype=torch_dtype_to_trt(dtype)) + + def lookup(x, y): + lookup_plugin = LookUpPlugin(False, True) + return lookup_plugin(x, y) + + output = lookup(x, y) + + output.mark_output("output", torch_dtype_to_str(torch.float32)) + + builder_config = builder.create_builder_config("float32") + engine = builder.build_engine(network, builder_config) + assert engine is not None + + output_dir = Path("tmp") / torch_dtype_to_str(dtype) + output_dir.mkdir(parents=True, exist_ok=True) + + engine_path = output_dir / "lookup.engine" + config_path = output_dir / "config.json" + + with engine_path.open("wb") as f: + f.write(engine) + builder.save_config(builder_config, str(config_path)) + + test(torch.bfloat16) + test(torch.float16) + test(torch.float32) diff --git a/examples/python_plugin/plugin_lib/__init__.py b/examples/python_plugin/plugin_lib/__init__.py new file mode 100644 index 000000000000..f27e0ded3a1c --- /dev/null +++ b/examples/python_plugin/plugin_lib/__init__.py @@ -0,0 +1,3 @@ +from .lookup_plugin import LookUpPlugin + +__all__ = ["LookUpPlugin"] diff --git a/examples/python_plugin/plugin_lib/lookup_kernel.py b/examples/python_plugin/plugin_lib/lookup_kernel.py new file mode 100644 index 000000000000..25cf66704b96 --- /dev/null +++ b/examples/python_plugin/plugin_lib/lookup_kernel.py @@ -0,0 +1,17 @@ +import triton +import triton.language as tl + + +@triton.jit +def lookup_kernel(X, Y, Z, vocab_size, hidden_size, token_num): + pid = tl.program_id(axis=0) + while pid < token_num * hidden_size: + row_idx = pid // hidden_size + col_idx = pid % hidden_size + word_idx = tl.load(X + row_idx) + embedding = tl.load( + Y + word_idx * hidden_size + col_idx, + mask=word_idx < vocab_size, + ) + tl.store(Z + pid, embedding) + pid += tl.num_programs(0) diff --git a/examples/python_plugin/plugin_lib/lookup_plugin.py b/examples/python_plugin/plugin_lib/lookup_plugin.py new file mode 100644 index 000000000000..612ee7a4db1e --- /dev/null +++ b/examples/python_plugin/plugin_lib/lookup_plugin.py @@ -0,0 +1,60 @@ +# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Sequence + +import torch + +from tensorrt_llm import PluginBase +from tensorrt_llm._utils import TensorWrapper, convert_to_torch_tensor +from tensorrt_llm.python_plugin import SymTensor, trtllm_plugin + +from .lookup_kernel import lookup_kernel + + +@trtllm_plugin("TritonLookUp") +class LookUpPlugin(PluginBase): + + def __init__(self, use_torch_tensor, fp32_output): + super().__init__() + self.use_torch_tensor = use_torch_tensor + self.fp32_output = fp32_output + + def shape_dtype_inference(self, inputs: Sequence[SymTensor]) -> SymTensor: + shape = inputs[1].shape + shape[0] = inputs[0].shape[0] + inputs[1].shape[0] - inputs[1].shape[0] + return SymTensor( + inputs[1].dtype if not self.fp32_output else torch.float32, shape) + + def forward(self, inputs: Sequence[TensorWrapper], + outputs: Sequence[TensorWrapper]): + assert len(inputs) == 2 + assert inputs[0].dtype in [torch.int32 or torch.int64] + assert inputs[1].dtype in [torch.float32, torch.float16, torch.bfloat16] + assert (self.fp32_output and outputs[0].dtype + == torch.float32) or outputs[0].dtype == inputs[1].dtype + + x = inputs[0] + y = inputs[1] + z = outputs[0] + if self.use_torch_tensor: + x = convert_to_torch_tensor(x) + y = convert_to_torch_tensor(y) + z = convert_to_torch_tensor(z) + MAX_BLOCK_NUM = 65536 + MAX_BLOCK_SIZE = 512 + grid = lambda meta: (min(MAX_BLOCK_NUM, x.shape[0]) * min( + MAX_BLOCK_SIZE, y.shape[1]), ) + lookup_kernel[grid](x, y, z, y.shape[0], y.shape[1], x.shape[0]) diff --git a/examples/python_plugin/run_lookup.py b/examples/python_plugin/run_lookup.py new file mode 100644 index 000000000000..055e86008c07 --- /dev/null +++ b/examples/python_plugin/run_lookup.py @@ -0,0 +1,65 @@ +from pathlib import Path + +import torch + +from tensorrt_llm import logger +from tensorrt_llm._utils import (torch_dtype_to_str, torch_dtype_to_trt, + trt_dtype_to_torch) +from tensorrt_llm.runtime.session import Session, TensorInfo + +import plugin_lib # isort: skip + +if __name__ == "__main__": + + def run_engine(dtype): + output_dir = Path('tmp') / torch_dtype_to_str(dtype) + + engine_path = output_dir / "lookup.engine" + + with engine_path.open('rb') as f: + session = Session.from_serialized_engine(f.read()) + + # meta data + batch_size = 10 + vocab_size = 1000 + n_embed = 1024 + + # test data + ## input index + index_shape = (batch_size, ) + index_data = torch.randint(0, + vocab_size, + index_shape, + dtype=torch.int32).cuda() + weight_data = torch.rand(vocab_size, n_embed, dtype=dtype).cuda() + + inputs = {"x": index_data, "y": weight_data} + + output_info = session.infer_shapes([ + TensorInfo(name, torch_dtype_to_trt(tensor.dtype), tensor.shape) + for name, tensor in inputs.items() + ]) + logger.debug(f'output info {output_info}') + outputs = { + t.name: + torch.empty(tuple(t.shape), + dtype=trt_dtype_to_torch(t.dtype), + device='cuda') + for t in output_info + } + + stream = torch.cuda.Stream() + ok = session.run(inputs=inputs, + outputs=outputs, + stream=stream.cuda_stream) + assert ok, 'Engine execution failed' + + embedding = torch.nn.Embedding.from_pretrained(weight_data) + torch_out = embedding(index_data).to(torch.float32) + trt_out = outputs['output'] + + torch.testing.assert_close(trt_out, torch_out) + + run_engine(torch.bfloat16) + run_engine(torch.float16) + run_engine(torch.float32) diff --git a/examples/quantization/README.md b/examples/quantization/README.md index 23bfa5d40c8e..b3b2e35b20ff 100644 --- a/examples/quantization/README.md +++ b/examples/quantization/README.md @@ -1,19 +1,249 @@ -# Model Quantization +# TensorRT LLM Quantization Toolkit Installation Guide -To run quantized models with TensorRT LLM: +## Introduction -- Use a pre-quantized Hugging Face checkpoint (for example the FP8/NVFP4 - checkpoints published on the [NVIDIA Hugging Face hub](https://huggingface.co/nvidia)). - Quantization settings are detected automatically when the model loads. -- To quantize your own model, use the - [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) - Hugging Face export flow (`examples/llm_ptq` in that repository). +This document introduces: -See the [quantization feature documentation](https://nvidia.github.io/TensorRT-LLM/features/quantization.html) -for supported formats per GPU architecture. +- The steps to install the TensorRT LLM quantization toolkit. +- The Python APIs to quantize the models. -## Mixed-precision MoE checkpoints +The detailed LLM quantization recipe is distributed to the README.md of the corresponding model examples. -[`quantize_mixed_precision_moe.py`](quantize_mixed_precision_moe.py) builds a -mixed-precision MoE checkpoint from separately quantized checkpoints; see the -script's argparse help for usage. +## Installation + +The NVIDIA Model Optimizer quantization toolkit is installed automatically as a dependency of TensorRT-LLM. + +```bash +# Install the additional requirements +cd examples/quantization +pip install -r requirements.txt +``` + +## Usage + +```bash +# FP8 quantization. +python quantize.py --model_dir $MODEL_PATH --qformat fp8 --kv_cache_dtype fp8 --output_dir $OUTPUT_PATH + +# INT4_AWQ tp4 quantization. +python quantize.py --model_dir $MODEL_PATH --qformat int4_awq --awq_block_size 64 --tp_size 4 --output_dir $OUTPUT_PATH + +# INT8 SQ with INT8 kv cache. +python quantize.py --model_dir $MODEL_PATH --qformat int8_sq --kv_cache_dtype int8 --output_dir $OUTPUT_PATH + +# Auto quantization(e.g. fp8 + int4_awq + w4a8_awq) using average weights bits 5 +python quantize.py --model_dir $MODEL_PATH --autoq_format fp8,int4_awq,w4a8_awq --output_dir $OUTPUT_PATH --auto_quantize_bits 5 --tp_size 2 + +# FP8 quantization for NeMo model. +python quantize.py --nemo_ckpt_path nemotron-3-8b-base-4k/Nemotron-3-8B-Base-4k.nemo \ + --dtype bfloat16 \ + --batch_size 64 \ + --qformat fp8 \ + --output_dir nemotron-3-8b/trt_ckpt/fp8/1-gpu + +# FP8 quantization for Medusa model. +python quantize.py --model_dir $MODEL_PATH\ + --dtype float16 \ + --qformat fp8 \ + --kv_cache_dtype fp8 \ + --output_dir $OUTPUT_PATH \ + --calib_size 512 \ + --tp_size 1 \ + --medusa_model_dir /path/to/medusa_head/ \ + --num_medusa_heads 4 +``` +Checkpoint saved in `output_dir` can be directly passed to `trtllm-build`. + +### Quantization Arguments: + +- model_dir: Hugging Face model path. +- qformat: Specify the quantization algorithm applied to the checkpoint. + - nvfp4: Weights are quantized to NVFP4 block-wise with size 16. Activation global scale are calibrated. + - fp8: Weights are quantized to FP8 tensor wise. Activation ranges are calibrated tensor wise. + - fp8_pc_pt: Weights are quantized to FP8 per-channel. Activation ranges are calibrated and quantized per-token. + - int8_sq: Weights are smoothed and quantized to INT8 channel wise. Activation ranges are calibrated tensor wise. + - int4_awq: Weights are re-scaled and block-wise quantized to INT4. Block size is specified by `awq_block_size`. + - w4a8_awq: Weights are re-scaled and block-wise quantized to INT4. Block size is specified by `awq_block_size`. Activation ranges are calibrated tensor wise. + - int8_wo: Actually nothing is applied to weights. Weights are quantized to INT8 channel wise when TRTLLM building the engine. + - int4_wo: Same as int8_wo but in INT4. + - full_prec: No quantization. +- autoq_format: Specific quantization algorithms are searched in auto quantization. The algorithm must in ['fp8', 'int4_awq', 'w4a8_awq', 'int8_sq'] and you can use ',' to separate more than one quantization algorithms, such as `--autoq_format fp8,int4_awq,w4a8_awq`. Please attention that using int8_sq and fp8 together is not supported. +- auto_quantize_bits: Effective bits constraint for auto quantization. If not set, regular quantization without auto quantization search is applied. Note: it must be set within correct range otherwise it will be set by lowest value if possible. For example, the weights of LLMs have 16 bits defaultly and it results in a weight compression rate of 40% if we set `auto_quantize_bits` to 9.6 (9.6 / 16 = 0.6), which means the average bits of the weights are 9.6 but not 16. However, which format to choose is determined by solving an optimization problem, so you need to generate the according checkpoint manually if you want to customize your checkpoint formats. The format of mixed precision checkpoint is described in detail below. +- output_dir: Path to save the quantized checkpoint. +- dtype: Specify data type of model when loading from Hugging Face. +- kv_cache_dtype: Specify kv cache data type. + - int8: Use int8 kv cache. + - fp8: Use FP8 kv cache. + - None (default): Use kv cache as model dtype. +- batch_size: Batch size for calibration. Default is 1. +- calib_size: Number of samples. Default is 512. +- calib_max_seq_length: Max sequence length of calibration samples. Default is 512. +- tp_size: Checkpoint is tensor paralleled by tp_size. Default is 1. +- pp_size: Checkpoint is pipeline paralleled by pp_size. Default is 1. +- awq_block_size: AWQ algorithm specific parameter. Indicate the block size when quantizing weights. 64 and 128 are supported by TRTLLM. +- quantize_lm_head: Enable quantization of lm_head layer. This is only supported for FP8 quantization. Default is false. + +#### NeMo model specific arguments: + +- nemo_ckpt_path: NeMo checkpoint path. +- calib_tp_size: TP size for NeMo checkpoint calibration. +- calib_pp_size: PP size for NeMo checkpoint calibration. + +#### Medusa specific arguments: + +- medusa_model_dir: Model path of medusa. +- quant_medusa_head: Whether to quantize the weights of medusa heads. +- num_medusa_heads: Number of medusa heads. +- num_medusa_layers: Number of medusa layers. +- max_draft_len: Max length of draft. +- medusa_hidden_act: Activation function of medusa. + +### Building Arguments: + +There are several arguments for the building stage which relate to quantization. +- use_fp8_context_fmha: This is Hopper-only feature. Use FP8 Gemm to calculate the attention operation. + +```python +qkv scale = 1.0 +FP_O = quantize(softmax(FP8_Q * FP8_K), scale=1.0) * FP8_V +FP_O * output_scale = FP8_O +``` + +### Checkpoint Conversion Arguments (not supported by all models) + +- FP8 + - use_fp8_rowwise: Enable FP8 per-token per-channel quantization for linear layer. (FP8 from `quantize.py` is per-tensor). +- INT8 + - smoothquant: Enable INT8 quantization for linear layer. Set the α parameter (see https://arxiv.org/pdf/2211.10438.pdf) to Smoothquant the model, and output int8 weights. A good first try is 0.5. Must be in [0, 1]. + - per_channel: Using per-channel quantization for weight when `smoothquant` is enabled. + - per_token: Using per-token quantization for activation when `smoothquant` is enabled. +- Weight-Only + - use_weight_only: Weights are quantized to INT4 or INT8 channel wise. + - weight_only_precision: Indicate `int4` or `int8` when `use_weight_only` is enabled. Or `int4_gptq` when `quant_ckpt_path` is provided which means checkpoint is for GPTQ. + - quant_ckpt_path: Path of a GPTQ quantized model checkpoint in `.safetensors` format. + - group_size: Group size used in GPTQ quantization. + - per_group: Should be enabled when load from GPTQ. +- KV Cache + - int8_kv_cache: By default, we use dtype for KV cache. int8_kv_cache chooses int8 quantization for KV cache. + - fp8_kv_cache: By default, we use dtype for KV cache. fp8_kv_cache chooses fp8 quantization for KV cache. + +### Format of Mixed Precision Checkpoints + +ModelOpt can produce a mixed precision TensorRT LLM checkpoint. After producing the quantized checkpoint, you can build engine directly by `trtllm-build` command: +```bash +trtllm-build --checkpoint_dir --output_dir $OUTPUT_PATH +``` +If you have some special needs about the model weights, such as int4 for MLP and int8 for the rest, you need to generate the checkpoint and config files by yourself. + +The `trtllm-build` command consumes the same format of weights, which is presented in [TensorRT LLM checkpoint formats](https://nvidia.github.io/TensorRT-LLM/architecture/checkpoint.html), but has different quantization method for every linear. Therefore, each layer, such as layer30.mlp.fc, layer30.attention.dense, and so on, keeps the same model weights according to the quantization formats in TensorRT LLM checkpoint. What's more, the `quantization` field in `config.json` will be like this: +``` + "quantization": { + "quant_algo": "MIXED_PRECISION", + "kv_cache_quant_algo": "FP8" // The quant_algo of KV cache may change + }, +``` +There will be another file about per-layer quantization information named `quant_cfg.json` in the same directory, the format of it is like: +``` +{ + "quant_algo": "MIXED_PRECISION", + "kv_cache_quant_algo": "FP8", + "quantized_layers": { // one more filed presents per-layer's information + "transformer.layers.0.attention.qkv": { + "quant_algo": "FP8" // specific algorithm for each linear + }, + "transformer.layers.0.attention.dense": { + "quant_algo": "FP8" + }, + "transformer.layers.0.mlp.fc": { + "quant_algo": "W4A16_AWQ", + "group_size": 128, + "has_zero_point": false, + "pre_quant_scale": true + }, + "transformer.layers.0.mlp.proj": { + "quant_algo": "W8A8_SQ_PER_CHANNEL" + }, + ... + "transformer.layers.31.mlp.proj": { + "quant_algo": "FP8" + } + } +} +``` + +TensorRT LLM will automatically read `quant_cfg.json` after recogniziong the `MIXED_PRECISION` quantization method in `config.json`. All the specific algorithm keeps the same as what in `quantization` field before. If some layers are not listed, they'll be treated as no quantization. + +## APIs + +[`quantize.py`](./quantize.py) uses the quantization toolkit to calibrate the PyTorch models and export TensorRT LLM checkpoints. Each TensorRT LLM checkpoint contains a config file (in .json format) and one or several rank weight files (in .safetensors format). It will produce one another quantization config for per-layer's information when setting auto quantization. The checkpoints can be directly used by `trtllm-build` command to build TensorRT LLM engines. See this [`doc`](../../docs/source/architecture/checkpoint.md) for more details on the TensorRT LLM checkpoint format. + +> *This quantization step may take a long time to finish and requires large GPU memory. Please use a server grade GPU if a GPU out-of-memory error occurs* + +> *If the model is trained with multi-GPU with tensor parallelism, the PTQ calibration process requires the same amount of GPUs as the training time too.* + + +### PTQ (Post Training Quantization) + +PTQ can be achieved with simple calibration on a small set of training or evaluation data (typically 128-512 samples) after converting a regular PyTorch model to a quantized model. + +```python +import torch +from torch.utils.data import DataLoader +from transformers import AutoModelForCausalLM +import modelopt.torch.quantization as mtq +import modelopt.torch.utils.dataset_utils as dataset_utils + +model = AutoModelForCausalLM.from_pretrained(...) + +# Select the quantization config, for example, FP8 +config = mtq.FP8_DEFAULT_CFG + +# Prepare the calibration set and define a forward loop +calib_dataloader = DataLoader(...) +calibrate_loop = dataset_utils.create_forward_loop( + calib_dataloader, dataloader=calib_dataloader +) + +# PTQ with in-place replacement to quantized modules +with torch.no_grad(): + mtq.quantize(model, config, forward_loop=calibrate_loop) + +# or PTQ with auto quantization +with torch.no_grad(): + model, search_history = mtq.auto_quantize( + model, + data_loader=calib_dataloader, + loss_func=lambda output, batch: output.loss, + constraints={"effective_bits": auto_quantize_bits}, # The average bits of quantized weights + forward_step=lambda model, batch: model(**batch), + quantization_formats=[quant_algo1, quant_algo2,...] + [None], + num_score_steps=min( + num_calib_steps=len(calib_dataloader), + len(calib_dataloader), 128 // batch_size + ), # Limit the number of score steps to avoid long calibration time + verbose=True, + ) +``` + +### Export Quantized Model + +After the model is quantized, it can be exported to a TensorRT LLM checkpoint, which includes + +- One json file recording the model structure and metadata, and +- One or several rank weight files storing quantized model weights and scaling factors. + +The export API is + +```python +from modelopt.torch.export import export_tensorrt_llm_checkpoint + +with torch.inference_mode(): + export_tensorrt_llm_checkpoint( + model, # The quantized model. + decoder_type, # The type of the model as str, e.g gptj, llama or gptnext. + dtype, # The exported weights data type as torch.dtype. + export_dir, # The directory where the exported files will be stored. + inference_tensor_parallel=tp_size, # The tensor parallelism size for inference. + inference_pipeline_parallel=pp_size, # The pipeline parallelism size for inference. + ) +``` diff --git a/examples/quantization/quantize.py b/examples/quantization/quantize.py new file mode 100644 index 000000000000..29c2fc5ca179 --- /dev/null +++ b/examples/quantization/quantize.py @@ -0,0 +1,208 @@ +import argparse + +import torch.multiprocessing as mp + +from tensorrt_llm.quantization import (quantize_and_export, + quantize_nemo_and_export) + +if __name__ == "__main__": + mp.set_start_method("spawn", force=True) + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model_dir", + help="Specify where the HuggingFace model is", + default=None) + parser.add_argument('--nemo_ckpt_path', + help="Specify where the NeMo checkpoint is", + default=None) + parser.add_argument( + '--decoder_type', + type=str, + default='gptnext', + choices=['gptnext', 'llama'], + help="Decoder type; effective for NeMo checkpoint only.") + parser.add_argument( + '--device', + help= + "The device to run calibration; effective for HuggingFace model only.", + default='cuda', + choices=['cuda', 'cpu']) + parser.add_argument( + "--device_map", + help="How to map the model on the devices", + default="auto", + choices=["auto", "sequential", "cpu", "gpu"], + ) + parser.add_argument( + '--calib_dataset', + type=str, + default='cnn_dailymail', + help= + "The huggingface dataset name or the local directory of the dataset for calibration." + ) + parser.add_argument( + '--calib_tp_size', + type=int, + default=1, + help= + "Tensor parallel size for calibration; effective for NeMo checkpoint only." + ) + parser.add_argument( + '--calib_pp_size', + type=int, + default=1, + help= + "Pipeline parallel size for calibration; effective for NeMo checkpoint only." + ) + + parser.add_argument( + '--dtype', + type=str, + default='auto', + choices=['auto', 'float16', 'bfloat16', 'float32'], + help= + "The data type for the model weights and activations of the non-quantized part, e.g., embedding and lm_head. " + "If 'auto', the data type is automatically inferred from the source model; " + "however, if the source dtype is float32, it is converted to float16.") + parser.add_argument( + "--qformat", + help="Quantization format.", + default="full_prec", + choices=[ + "nvfp4", + "fp8", + "fp8_pc_pt", + "int8_sq", + "int4_awq", + "w4a8_awq", + "int8_wo", + "int4_wo", + "full_prec", + ], + ) + parser.add_argument( + "--seed", + help="Seed the generate random numbers, the value will be used to call" + "random.seed(value) and numpy.random.seed(value)", + type=int, + default=1234) + parser.add_argument("--tokenizer_max_seq_length", + help="Max sequence length to init the tokenizers", + type=int, + default=2048) + + parser.add_argument("--batch_size", + help="Batch size for calibration.", + type=int, + default=1) + parser.add_argument("--calib_size", + help="Number of samples for calibration.", + type=int, + default=512) + parser.add_argument("--calib_max_seq_length", + help="Max sequence length for calibration", + type=int, + default=512) + parser.add_argument("--output_dir", default="exported_model") + parser.add_argument("--tp_size", type=int, default=1) + parser.add_argument("--pp_size", type=int, default=1) + parser.add_argument("--cp_size", type=int, default=1) + parser.add_argument("--awq_block_size", type=int, default=128) + parser.add_argument("--kv_cache_dtype", + help="KV Cache dtype.", + default=None, + choices=["int8", "fp8", None]) + parser.add_argument("--quantize_lm_head", + action='store_true', + default=False) + # Medusa + parser.add_argument('--num_medusa_heads', type=int, default=4) + parser.add_argument('--num_medusa_layers', type=int, default=1) + parser.add_argument('--max_draft_len', type=int, default=63) + parser.add_argument('--medusa_hidden_act', type=str, default="silu") + parser.add_argument('--medusa_model_dir', type=str, default=None) + parser.add_argument('--quant_medusa_head', + default=False, + action='store_true', + help="whether to quantize the weights of medusa heads") + + # auto quantization + parser.add_argument( + '--autoq_format', + default=None, + type=str, + help= + "Specific quantization algorithms will be searched in auto quantization." + "The algorithm must in ['fp8', 'int4_awq', 'w4a8_awq', 'int8_sq']." + "You can use ',' to separate more than one quantization algorithms(e.g. --autoq_format fp8,int4_awq,w4a8_awq)." + "Notice: fp8 and int8_sq can't be used at the same time.") + parser.add_argument( + '--auto_quantize_bits', + type=float, + default=None, + help="Effective bits constraint for auto quantization. If not set, " + "regular quantization without auto quantization search will be applied." + "You can't set it lower than the num_bits of most aggressive quantization format." + "For example, if 'int4_awq' is in autoq_format, it can't be lower than 4.0." + ) + + args = parser.parse_args() + + # auto_quantize_bits check + if args.autoq_format: + lower_bound, upper_bound = 4 if '4' in args.autoq_format else 8, 16 + if args.auto_quantize_bits is None or args.auto_quantize_bits < lower_bound or args.auto_quantize_bits > upper_bound: + print( + f"invalid auto_quantize_bits value, will be set to {lower_bound}" + ) + args.auto_quantize_bits = lower_bound + + if args.model_dir is not None: + quantize_and_export( + model_dir=args.model_dir, + device=args.device, + calib_dataset=args.calib_dataset, + dtype=args.dtype, + qformat=args.qformat + if args.auto_quantize_bits is None else args.autoq_format, + kv_cache_dtype=args.kv_cache_dtype, + calib_size=args.calib_size, + batch_size=args.batch_size, + calib_max_seq_length=args.calib_max_seq_length, + awq_block_size=args.awq_block_size, + output_dir=args.output_dir, + tp_size=args.tp_size, + pp_size=args.pp_size, + cp_size=args.cp_size, + seed=args.seed, + tokenizer_max_seq_length=args.tokenizer_max_seq_length, + num_medusa_heads=args.num_medusa_heads, + num_medusa_layers=args.num_medusa_layers, + max_draft_len=args.max_draft_len, + medusa_hidden_act=args.medusa_hidden_act, + medusa_model_dir=args.medusa_model_dir, + quant_medusa_head=args.quant_medusa_head, + auto_quantize_bits=args.auto_quantize_bits, + device_map=args.device_map, + quantize_lm_head=args.quantize_lm_head) + elif args.nemo_ckpt_path is not None: + quantize_nemo_and_export(nemo_ckpt_path=args.nemo_ckpt_path, + decoder_type=args.decoder_type, + calib_dataset=args.calib_dataset, + calib_tp_size=args.calib_tp_size, + calib_pp_size=args.calib_pp_size, + dtype=args.dtype, + qformat=args.qformat, + kv_cache_dtype=args.kv_cache_dtype, + calib_size=args.calib_size, + batch_size=args.batch_size, + calib_max_seq_length=args.calib_max_seq_length, + awq_block_size=args.awq_block_size, + output_dir=args.output_dir, + tp_size=args.tp_size, + pp_size=args.pp_size, + cp_size=args.cp_size, + seed=args.seed) + else: + raise ValueError( + "One of source checkpoint (model_dir, nemo_ckpt_path) must be specified" + ) diff --git a/examples/run.py b/examples/run.py new file mode 100755 index 000000000000..7ce36bbe9848 --- /dev/null +++ b/examples/run.py @@ -0,0 +1,710 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import ast +import csv +import os +from pathlib import Path +from typing import List, Optional + +import numpy as np +import torch +from utils import (DEFAULT_HF_MODEL_DIRS, DEFAULT_PROMPT_TEMPLATES, + add_common_args, get_beam_width_array, load_tokenizer, + prepare_enc_dec_inputs, read_model_name, + supports_inflight_batching, throttle_generator) + +import tensorrt_llm +import tensorrt_llm.profiler +from tensorrt_llm.logger import logger +from tensorrt_llm.runtime import PYTHON_BINDINGS, ModelRunner + +if PYTHON_BINDINGS: + from tensorrt_llm.runtime import ModelRunnerCpp + +from ngram.run_dtm_ngram import run_dtm_ngram + + +def parse_arguments(args=None): + # see `add_common_args` for extended list of arguments + parser = argparse.ArgumentParser() + parser.add_argument('--max_input_length', type=int, default=923) + parser.add_argument('--max_output_len', type=int, required=True) + parser.add_argument( + '--draft_engine_dir', + type=str, + default=None, + help='Path to engine of draft model in Draft-Target-Model mode.') + parser.add_argument( + '--input_text', + type=str, + nargs='+', + default=["Born in north-east France, Soyer trained as a"]) + parser.add_argument( + '--input_file', + type=str, + help= + 'CSV or Numpy file containing tokenized input. Alternative to text input.', + default=None) + parser.add_argument('--multimodal_input_file', + type=str, + help='Path to multimodal input file.') + parser.add_argument( + '--input_token_extra_ids', + type=int, + nargs='+', + help= + 'Input token extra ids for using p-tuning and KV Cache reuse together (only available with cpp session).', + default=None) + parser.add_argument( + '--input_token_extra_ids_file', + type=str, + help= + 'CSV or Numpy file containing input token extra ids file. Alternative to text input (only available with cpp session).', + default=None) + parser.add_argument('--output_csv', + type=str, + help='CSV file where the tokenized output is stored.', + default=None) + parser.add_argument('--output_npy', + type=str, + help='Numpy file where the tokenized output is stored.', + default=None) + parser.add_argument('--output_generation_logits', + default=False, + action='store_true', + help="Enable gathering generation logits.") + parser.add_argument( + '--output_logits_npy', + type=str, + help= + 'Numpy file where the generation logits are stored. Use only when num_beams==1', + default=None) + parser.add_argument('--output_log_probs_npy', + type=str, + help='Numpy file where the log_probs are stored', + default=None) + parser.add_argument('--output_cum_log_probs_npy', + type=str, + help='Numpy file where the cum_log_probs are stored', + default=None) + parser.add_argument( + '--run_profiling', + default=False, + action='store_true', + help="Run several 10 iterations to profile the inference latencies.") + parser.add_argument( + '--fail_fast_on_attention_window_too_large', + action='store_true', + default=False, + help= + 'Exit with runtime error when attention window is too large to fit even a single sequence in the KV cache.' + ) + + parser = add_common_args(parser) + + return parser.parse_args(args=args) + + +def parse_input(tokenizer, + input_text=None, + prompt_template=None, + input_file=None, + add_special_tokens=True, + max_input_length=923, + pad_id=None, + num_prepend_vtokens=[], + model_name=None, + model_version=None): + if pad_id is None: + pad_id = tokenizer.pad_token_id + + batch_input_ids = [] + if input_file is None: + if 'whisper' in model_name.lower(): + batch_input_ids.append(tokenizer.prefix_tokens) + else: + for curr_text in input_text: + if prompt_template is not None: + curr_text = prompt_template.format(input_text=curr_text) + input_ids = tokenizer.encode( + curr_text, + add_special_tokens=add_special_tokens, + truncation=True, + max_length=max_input_length) + batch_input_ids.append(input_ids) + else: + if input_file.endswith('.csv'): + with open(input_file, 'r') as csv_file: + csv_reader = csv.reader(csv_file, delimiter=',') + for line in csv_reader: + input_ids = np.array(line, dtype='int32') + batch_input_ids.append(input_ids[-max_input_length:]) + elif input_file.endswith('.npy'): + inputs = np.load(input_file) + for row in inputs: + input_ids = row[row != pad_id] + batch_input_ids.append(input_ids[-max_input_length:]) + + elif input_file.endswith('.txt'): + with open(input_file, 'r', encoding='utf-8', + errors='replace') as txt_file: + input_text = txt_file.readlines() + batch_input_ids = tokenizer( + input_text, + add_special_tokens=add_special_tokens, + truncation=True, + max_length=max_input_length)["input_ids"] + else: + print('Input file format not supported.') + raise SystemExit + + if num_prepend_vtokens: + assert len(num_prepend_vtokens) == len(batch_input_ids) + base_vocab_size = tokenizer.vocab_size + for i, length in enumerate(num_prepend_vtokens): + batch_input_ids[i] = list( + range(base_vocab_size, + base_vocab_size + length)) + batch_input_ids[i] + + if input_file is None and 'GLM' in model_name and model_version == 'glm': + for ids in batch_input_ids: + ids.append(tokenizer.sop_token_id) + + batch_input_ids = [ + torch.tensor(x, dtype=torch.int32) for x in batch_input_ids + ] + + logger.debug(f"Input token ids (batch_size = {len(batch_input_ids)}):") + for i, input_ids in enumerate(batch_input_ids): + logger.debug(f"Request {i}: {input_ids.tolist()}") + + return batch_input_ids + + +def parse_input_token_extra_ids(prompt_table_path, kv_cache_enable_block_reuse, + input_token_extra_ids, + input_token_extra_ids_file, max_input_length): + batch_extra_ids = None + if prompt_table_path and kv_cache_enable_block_reuse: + assert input_token_extra_ids or input_token_extra_ids_file, \ + "Input token extra ids must be provided when p-tuning and KV Cache reuse are both enabled" + batch_extra_ids = [] + if input_token_extra_ids_file: + if input_token_extra_ids_file.endswith('.csv'): + with open(input_token_extra_ids_file, 'r') as csv_file: + csv_reader = csv.reader(csv_file, delimiter=',') + for line in csv_reader: + extra_ids = [int(num) for num in line] + batch_extra_ids.append(extra_ids[-max_input_length:]) + elif input_token_extra_ids_file.endswith('.npy'): + inputs = np.load(input_token_extra_ids_file) + for extra_ids in inputs: + batch_extra_ids.append(extra_ids[-max_input_length:]) + else: + print('Input file format not supported.') + raise SystemExit + else: + batch_extra_ids.append(input_token_extra_ids) + return batch_extra_ids + + +def print_output(tokenizer, + output_ids: torch.Tensor, + input_lengths: List[int], + sequence_lengths: torch.Tensor, + output_csv: Optional[str] = None, + output_npy: Optional[str] = None, + context_logits: Optional[torch.Tensor] = None, + generation_logits: Optional[torch.Tensor] = None, + cum_log_probs: Optional[torch.Tensor] = None, + log_probs: Optional[torch.Tensor] = None, + output_logits_npy: Optional[str] = None, + output_cum_log_probs_npy: Optional[str] = None, + output_log_probs_npy: Optional[str] = None): + num_output_sents, num_beams, _ = output_ids.size() + batch_size = len(input_lengths) + num_return_sequences = num_output_sents // batch_size + + if output_csv is None and output_npy is None and tokenizer is not None: + for i in range(batch_size * num_return_sequences): + batch_idx = i // num_return_sequences + seq_idx = i % num_return_sequences + inputs = output_ids[i][0][:input_lengths[batch_idx]].tolist() + input_text = tokenizer.decode(inputs) + if seq_idx == 0: + print(f'Input [Text {batch_idx}]: \"{input_text}\"') + + for beam in range(num_beams): + output_begin = input_lengths[batch_idx] + output_end = sequence_lengths[i][beam] + outputs = output_ids[i][beam][output_begin:output_end].tolist() + output_text = tokenizer.decode(outputs) + index_str = (f'Text {batch_idx} Seq {seq_idx} Beam {beam}' + if num_return_sequences > 1 else + f'Text {batch_idx} Beam {beam}') + print(f'Output [{index_str}]: \"{output_text}\"') + logger.debug(str(outputs)) + + output_ids = output_ids.reshape((-1, output_ids.size(2))) + + if output_csv is not None: + output_file = Path(output_csv) + output_file.parent.mkdir(exist_ok=True, parents=True) + outputs = output_ids.tolist() + with open(output_file, 'w') as csv_file: + writer = csv.writer(csv_file, delimiter=',') + writer.writerows(outputs) + + if output_npy is not None: + output_file = Path(output_npy) + output_file.parent.mkdir(exist_ok=True, parents=True) + outputs = np.array(output_ids.cpu().contiguous(), dtype='int32') + np.save(output_file, outputs) + + # Save context logits + if context_logits is not None and output_logits_npy is not None: + context_logits = torch.cat(context_logits, axis=0) + vocab_size_padded = context_logits.shape[-1] + context_logits = context_logits.reshape([1, -1, vocab_size_padded]) + + output_context_logits_npy = output_logits_npy.split( + '.npy')[0] + "_context" + output_context_logits_file = Path(output_context_logits_npy) + context_outputs = np.array( + context_logits.squeeze(0).cpu().contiguous(), + dtype='float32') # [promptLengthSum, vocabSize] + np.save(output_context_logits_file, context_outputs) + + # Save generation logits + if generation_logits is not None and output_logits_npy is not None and num_beams == 1: + output_generation_logits_npy = output_logits_npy.split( + '.npy')[0] + "_generation" + output_generation_logits_file = Path(output_generation_logits_npy) + generation_outputs = np.array(generation_logits.cpu().contiguous(), + dtype='float32') + np.save(output_generation_logits_file, generation_outputs) + + # Save cum log probs + if cum_log_probs is not None and output_cum_log_probs_npy is not None: + cum_log_probs_file = Path(output_cum_log_probs_npy) + cum_log_probs_outputs = np.array(cum_log_probs.cpu().contiguous(), + dtype='float32') + np.save(cum_log_probs_file, cum_log_probs_outputs) + + # Save cum log probs + if log_probs is not None and output_log_probs_npy is not None: + log_probs_file = Path(output_log_probs_npy) + log_probs_outputs = np.array(log_probs.cpu().contiguous(), + dtype='float32') + np.save(log_probs_file, log_probs_outputs) + + +def main(args): + runtime_rank = tensorrt_llm.mpi_rank() + logger.set_level(args.log_level) + + # different handling if encoder-decoder models + is_enc_dec = {'encoder', 'decoder'}.issubset({ + name + for name in os.listdir(args.engine_dir) + if os.path.isdir(os.path.join(args.engine_dir, name)) + }) + if is_enc_dec: + logger.warning( + "This path is an encoder-decoder model. Using different handling.") + assert not args.use_py_session, "Encoder-decoder models don't have a unified python runtime, please use its own examples/models/core/enc_dec/run.py instead." + + model_name, model_version = read_model_name( + args.engine_dir if not is_enc_dec else os.path. + join(args.engine_dir, 'encoder')) + + if args.tokenizer_dir is None and model_name in DEFAULT_HF_MODEL_DIRS: + logger.warning( + "tokenizer_dir is not specified. Try to infer from model_name, but this may be incorrect." + ) + args.tokenizer_dir = DEFAULT_HF_MODEL_DIRS[model_name] + + tokenizer, pad_id, end_id = load_tokenizer( + tokenizer_dir=args.tokenizer_dir, + vocab_file=args.vocab_file, + model_name=model_name, + model_version=model_version, + tokenizer_type=args.tokenizer_type, + ) + + if args.end_id: + end_id = args.end_id + + prompt_template = None + if args.use_prompt_template and model_name in DEFAULT_PROMPT_TEMPLATES: + prompt_template = DEFAULT_PROMPT_TEMPLATES[model_name] + + batch_input_ids = parse_input(tokenizer=tokenizer, + input_text=args.input_text, + prompt_template=prompt_template, + input_file=args.input_file, + add_special_tokens=args.add_special_tokens, + max_input_length=args.max_input_length, + pad_id=pad_id, + num_prepend_vtokens=args.num_prepend_vtokens, + model_name=model_name, + model_version=model_version) + + stop_words_list = None + if args.stop_words: + stop_words_list = tensorrt_llm.runtime.decode_words_list( + args.stop_words, tokenizer) + if model_version == 'glm4': # add default stop token ids for GLM-4 + glm4_stop_ids = [[151329], [151336], [151338]] + if stop_words_list is None: + stop_words_list = [glm4_stop_ids] * len(batch_input_ids) + else: + for req_stop_words_list in stop_words_list: + req_stop_words_list.extend(glm4_stop_ids) + + bad_words_list = None + if args.bad_words: + bad_words_list = tensorrt_llm.runtime.decode_words_list( + args.bad_words, tokenizer) + + if is_enc_dec: + encoder_input_ids, encoder_input_features, encoder_output_lengths, decoder_input_ids = prepare_enc_dec_inputs( + batch_input_ids, model_name, args.engine_dir, + args.multimodal_input_file) + + input_token_extra_ids = parse_input_token_extra_ids( + args.prompt_table_path, args.kv_cache_enable_block_reuse, + args.input_token_extra_ids, args.input_token_extra_ids_file, + args.max_input_length) + + input_lengths = [x.size(0) for x in decoder_input_ids + ] if is_enc_dec else [x.size(0) for x in batch_input_ids] + + encoder_input_lengths = [ + x.size(0) for x in (encoder_input_features or encoder_input_ids) + ] if is_enc_dec else None + + if args.beam_width_array is not None: + logger.info("Enable Variable-Beam-Width-Search (VBWS)") + assert not args.use_py_session, "`--use_py_session` is not supported in VBWS." + args.beam_width_array, args.num_beams = get_beam_width_array( + args.beam_width_array) + + if not args.use_py_session and not supports_inflight_batching( + os.path.join(args.engine_dir, "decoder") if is_enc_dec else args. + engine_dir): + logger.warning( + "The given engine does not support in-flight batching, fallback to python session" + ) + args.use_py_session = True + + if not PYTHON_BINDINGS and not args.use_py_session: + logger.warning( + "Python bindings of C++ session is unavailable, fallback to Python session." + ) + args.use_py_session = True + if args.debug_mode and not args.use_py_session: + logger.warning( + "Debug mode is not supported in C++ session for now, fallback to Python session." + ) + args.use_py_session = True + if args.return_all_generated_tokens and args.use_py_session: + raise ValueError( + "Returning all the generated tokens at each step is not supported in the Python session, use C++ session instead." + ) + if (not args.return_all_generated_tokens) and args.streaming and ( + args.num_beams > 1): + logger.warning( + "Setting return_all_generated_tokens to True since streaming AND beam search are done simultaneously. " + "Returning the full beams at each streaming step is needed because beam search + streaming can change previous outputs. " + "WARNING: using this option may increase network usage significantly (quadratically w.r.t output length)." + ) + args.return_all_generated_tokens = True + + logger.info(f"Using {'Python' if args.use_py_session else 'C++'} session") + + if args.draft_target_model_config is not None or args.ngram_config is not None: + # Speculative-Decoding of Draft-Target-Model (DTM) and NGram + # If the parameters of `runner_kwargs` and `runner.generate()` in the "else" branch change, the same change should be done for `examples/ngram/run_dtm_ngram.py` + assert args.kv_cache_enable_block_reuse, "`--kv_cache_enable_block_reuse` must be specified in speculative decoding." + assert not args.use_py_session, "`--use_py_session` is not supported in Speculative decoding." + assert not is_enc_dec, "Encoder-Decoder model is not supported in Speculative decoding." + assert args.num_beams == 1, "`--num_beams>1` is not supported in Speculative decoding." + + outputs = run_dtm_ngram(batch_input_ids, args, runtime_rank, end_id, + pad_id, stop_words_list, bad_words_list, + len(tokenizer)) + if not args.streaming: # Unpack runner from the return value in No-Streaming mode + outputs, runner = list(outputs)[0] + + else: # Normal run + runner_cls = ModelRunner if args.use_py_session else ModelRunnerCpp + runner_kwargs = dict( + engine_dir=args.engine_dir, + lora_dir=args.lora_dir, + rank=runtime_rank, + debug_mode=args.debug_mode, + lora_ckpt_source=args.lora_ckpt_source, + gpu_weights_percent=args.gpu_weights_percent, + max_output_len=args.max_output_len, + enable_context_fmha_fp32_acc=args.enable_context_fmha_fp32_acc, + fail_fast_on_attention_window_too_large=args. + fail_fast_on_attention_window_too_large, + ) + if args.medusa_choices is not None: + args.medusa_choices = ast.literal_eval(args.medusa_choices) + assert args.temperature == 1.0, "Medusa should use temperature == 1.0" + assert args.num_beams == 1, "Medusa should use num_beams == 1" + runner_kwargs.update(medusa_choices=args.medusa_choices) + if args.eagle_choices is not None or args.eagle_posterior_threshold is not None or args.eagle_use_dynamic_tree: + assert args.num_beams == 1, "Eagle should use num_beams == 1" + assert not args.use_py_session, "Eagle does not support py session" + if args.eagle_choices is not None and not args.eagle_use_dynamic_tree: + args.eagle_choices = ast.literal_eval(args.eagle_choices) + runner_kwargs.update(eagle_choices=args.eagle_choices) + if args.eagle_posterior_threshold is not None: + runner_kwargs.update( + eagle_posterior_threshold=args.eagle_posterior_threshold) + if args.eagle_use_dynamic_tree: + runner_kwargs.update( + eagle_use_dynamic_tree=args.eagle_use_dynamic_tree) + assert args.eagle_dynamic_tree_max_top_k is not None and args.eagle_dynamic_tree_max_top_k > 0 + runner_kwargs.update(eagle_dynamic_tree_max_top_k=args. + eagle_dynamic_tree_max_top_k) + if args.lookahead_config is not None: + args.lookahead_config = ast.literal_eval(args.lookahead_config) + assert len( + args.lookahead_config + ) == 3, "Lookahead needs [max_window_size, max_ngram_size, max_verification_set_size]" + runner_kwargs.update(lookahead_config=args.lookahead_config) + if not args.use_py_session: + runner_kwargs.update( + is_enc_dec=is_enc_dec, + max_batch_size=len(batch_input_ids), + max_input_len=max( + encoder_input_lengths if is_enc_dec else input_lengths), + max_beam_width=args.num_beams, + max_attention_window_size=args.max_attention_window_size, + sink_token_length=args.sink_token_length, + max_tokens_in_paged_kv_cache=args.max_tokens_in_paged_kv_cache, + kv_cache_enable_block_reuse=args.kv_cache_enable_block_reuse, + kv_cache_free_gpu_memory_fraction=args. + kv_cache_free_gpu_memory_fraction, + cross_kv_cache_fraction=args.cross_kv_cache_fraction + if is_enc_dec else None, + enable_chunked_context=args.enable_chunked_context, + multi_block_mode=args.multi_block_mode, + cuda_graph_mode=args.cuda_graph_mode, + gather_generation_logits=args.output_generation_logits, + use_variable_beam_width_search=(args.beam_width_array + is not None), + ) + runner = runner_cls.from_dir(**runner_kwargs) + + with torch.no_grad(): + outputs = runner.generate( + batch_input_ids=decoder_input_ids + if is_enc_dec else batch_input_ids, + encoder_input_ids=encoder_input_ids if is_enc_dec else None, + encoder_input_features=encoder_input_features + if is_enc_dec else None, + encoder_output_lengths=encoder_output_lengths + if is_enc_dec else None, + max_new_tokens=args.max_output_len, + max_attention_window_size=args.max_attention_window_size, + sink_token_length=args.sink_token_length, + end_id=end_id, + pad_id=pad_id, + temperature=args.temperature, + top_k=args.top_k, + top_p=args.top_p, + num_beams=args.num_beams, + num_return_sequences=args.num_return_sequences, + length_penalty=args.length_penalty, + early_stopping=args.early_stopping, + beam_width_array=args.beam_width_array, + repetition_penalty=args.repetition_penalty, + presence_penalty=args.presence_penalty, + frequency_penalty=args.frequency_penalty, + prompt_ignore_length=args.prompt_ignore_length, + min_p=args.min_p, + stop_words_list=stop_words_list, + bad_words_list=bad_words_list, + output_cum_log_probs=(args.output_cum_log_probs_npy != None), + output_log_probs=(args.output_log_probs_npy != None), + random_seed=args.random_seed, + lora_uids=args.lora_task_uids, + prompt_table=args.prompt_table_path, + prompt_tasks=args.prompt_tasks, + streaming=args.streaming, + output_sequence_lengths=True, + output_generation_logits=args.output_generation_logits, + no_repeat_ngram_size=args.no_repeat_ngram_size, + return_dict=True, + medusa_choices=args.medusa_choices, + eagle_choices=args.eagle_choices, + return_all_generated_tokens=args.return_all_generated_tokens, + input_token_extra_ids=input_token_extra_ids, + fail_fast_on_attention_window_too_large=args. + fail_fast_on_attention_window_too_large, + language_adapter_uids=args.language_task_uids) + torch.cuda.synchronize() + + # Receive output, print to screen or save to file + if args.streaming: + for curr_outputs in throttle_generator(outputs, + args.streaming_interval): + if runtime_rank == 0: + output_ids = curr_outputs['output_ids'] + sequence_lengths = curr_outputs['sequence_lengths'] + cum_log_probs = None + log_probs = None + if args.output_cum_log_probs_npy is not None: + cum_log_probs = curr_outputs['cum_log_probs'] + if args.output_log_probs_npy is not None: + log_probs = curr_outputs['log_probs'] + print_output( + tokenizer, + output_ids, + input_lengths, + sequence_lengths, + output_csv=args.output_csv, + output_npy=args.output_npy, + cum_log_probs=cum_log_probs, + log_probs=log_probs, + output_cum_log_probs_npy=args.output_cum_log_probs_npy, + output_log_probs_npy=args.output_log_probs_npy) + else: + if runtime_rank == 0: + output_ids = outputs['output_ids'] + sequence_lengths = outputs['sequence_lengths'] + context_logits = None + generation_logits = None + cum_log_probs = None + log_probs = None + if runner.gather_context_logits: + context_logits = outputs['context_logits'] + if runner.gather_generation_logits or args.output_generation_logits: + generation_logits = outputs['generation_logits'] + if args.output_cum_log_probs_npy is not None: + cum_log_probs = outputs['cum_log_probs'] + if args.output_log_probs_npy is not None: + log_probs = outputs['log_probs'] + print_output(tokenizer, + output_ids, + input_lengths, + sequence_lengths, + output_csv=args.output_csv, + output_npy=args.output_npy, + context_logits=context_logits, + generation_logits=generation_logits, + output_logits_npy=args.output_logits_npy, + cum_log_probs=cum_log_probs, + log_probs=log_probs, + output_cum_log_probs_npy=args.output_cum_log_probs_npy, + output_log_probs_npy=args.output_log_probs_npy) + + # Profiling + if args.run_profiling: + ite = 10 + # warmup + for _ in range(ite): + with torch.no_grad(): + outputs = runner.generate( + batch_input_ids, + max_new_tokens=args.max_output_len, + max_attention_window_size=args.max_attention_window_size, + end_id=end_id, + pad_id=pad_id, + temperature=args.temperature, + top_k=args.top_k, + top_p=args.top_p, + num_beams=args.num_beams, + length_penalty=args.length_penalty, + early_stopping=args.early_stopping, + beam_width_array=args.beam_width_array, + repetition_penalty=args.repetition_penalty, + presence_penalty=args.presence_penalty, + frequency_penalty=args.frequency_penalty, + prompt_ignore_length=args.prompt_ignore_length, + min_p=args.min_p, + stop_words_list=stop_words_list, + bad_words_list=bad_words_list, + output_cum_log_probs=(args.output_cum_log_probs_npy + is not None), + output_log_probs=(args.output_log_probs_npy is not None), + random_seed=args.random_seed, + lora_uids=args.lora_task_uids, + lookahead_config=args.lookahead_config, + prompt_table=args.prompt_table_path, + prompt_tasks=args.prompt_tasks, + streaming=args.streaming, + output_sequence_lengths=True, + return_dict=True, + return_all_generated_tokens=args. + return_all_generated_tokens, + input_token_extra_ids=input_token_extra_ids) + torch.cuda.synchronize() + + tensorrt_llm.profiler.start("tmp") + for _ in range(ite): + with torch.no_grad(): + outputs = runner.generate( + batch_input_ids, + max_new_tokens=args.max_output_len, + max_attention_window_size=args.max_attention_window_size, + end_id=end_id, + pad_id=pad_id, + temperature=args.temperature, + top_k=args.top_k, + top_p=args.top_p, + num_beams=args.num_beams, + length_penalty=args.length_penalty, + early_stopping=args.early_stopping, + beam_width_array=args.beam_width_array, + repetition_penalty=args.repetition_penalty, + presence_penalty=args.presence_penalty, + frequency_penalty=args.frequency_penalty, + prompt_ignore_length=args.prompt_ignore_length, + stop_words_list=stop_words_list, + bad_words_list=bad_words_list, + output_cum_log_probs=(args.output_cum_log_probs_npy + != None), + output_log_probs=(args.output_log_probs_npy != None), + random_seed=args.random_seed, + lora_uids=args.lora_task_uids, + prompt_table=args.prompt_table_path, + prompt_tasks=args.prompt_tasks, + streaming=args.streaming, + output_sequence_lengths=True, + return_dict=True, + return_all_generated_tokens=args. + return_all_generated_tokens, + input_token_extra_ids=input_token_extra_ids, + fail_fast_on_attention_window_too_large=args. + fail_fast_on_attention_window_too_large) + torch.cuda.synchronize() + tensorrt_llm.profiler.stop("tmp") + + print( + f"batch_size: {len(batch_input_ids)}, avg latency of {ite} iterations: : {tensorrt_llm.profiler.elapsed_time_in_sec('tmp') / ite} sec" + ) + + +if __name__ == '__main__': + args = parse_arguments() + main(args) diff --git a/examples/sample_weight_stripping/README.md b/examples/sample_weight_stripping/README.md new file mode 100644 index 000000000000..cb3c04404902 --- /dev/null +++ b/examples/sample_weight_stripping/README.md @@ -0,0 +1,275 @@ +# Sample Weight-Stripping + +> [!WARNING] +> The `convert_checkpoint.py` / `trtllm-build` / `run.py` workflow described +> below is **legacy** and will not receive new features. New projects should use +> [`trtllm-serve`](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html) +> or the [LLM Python API](https://nvidia.github.io/TensorRT-LLM/llm-api/index.html) instead. + +## Table Of Contents + +- [Overview](#overview) + * [Build Weights Stripped Engine](#build-weights-stripped-engine) + * [Engine Refitter](#engine-refitter) +- [Prerequisites](#prerequisites) +- [Weight-Stripping Workflow Example](#weight-stripping-workflow-example) + * [GPT-J](#gpt-j) + * [Llama-7b INT4](#llama-7b-int4) + * [Llama-7b FP16 + WoQ INT8](#llama-7b-fp16-woq-int8) + * [Llama2-70b FP8 with TP=2](#llama2-70b-fp8-with-tp2) +- [Engine Plan File Size Results](#engine-plan-file-size-results) +- [Prototype](#prototype) + * [Checkpoint Pruner](#checkpoint-pruner) + * [Pruning a TensorRT LLM Checkpoint](#pruning-a-tensorrt-llm-checkpoint) + +## Overview + +This workflow introduces a new script `trtllm-refit`. `trtllm-refit` allows you to refit the generated engine with weights from any TensorRT LLM checkpoint matching the same architecture, so long as you build the engine as refittable or stripped. + +### Build Weights Stripped Engine +TensorRT can generate refittable engines with the same performance as the non-refittable ones when TensorRT builder optimize under the assumption that the engine will be refitted with weights identical to those provide at build time. Those refittable weights can be stripped to reduce the engine plan file size, with the option to subsequently supply them via the refit interface. + +New option `--strip_plan` is introduced in `trtllm-build` + +```bash +trtllm-build --strip_plan --checkpoint_dir ${CHECKPOINT_DIR} --output_dir ${ENGINE_DIR} ... +``` + +### Engine Refitter +The refitter allows you to refit an engine with weights in a TensorRT LLM checkpoint. It does this by doing a textual match between engine and checkpoint weight names. In order for the refitter to work, the engine must be built with refitting enabled. This can be accomplished by passing `--strip_plan` to `trtllm-build`. + +After building a stripped engine via `trtllm-build`, run + +```bash +trtllm-refit --checkpoint_dir ${CHECKPOINT_DIR} --engine_dir ${ENGINE_DIR} +``` + + +## Prerequisites + +Install [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/blob/main/README.md) either through [pip](https://github.com/NVIDIA/TensorRT-LLM/blob/main/README.md#installation) or [from the source](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/installation/build-from-source.md). + +## Weight-Stripping Workflow Example + +### GPT-J + +1. Download the weights. +```bash +# 1. Weights & config +git clone https://huggingface.co/EleutherAI/gpt-j-6b +pushd gpt-j-6b && \ + rm -f pytorch_model.bin && \ + wget https://huggingface.co/EleutherAI/gpt-j-6b/resolve/main/pytorch_model.bin && \ +popd + +# 2. Vocab and merge table +wget https://huggingface.co/EleutherAI/gpt-j-6b/resolve/main/vocab.json +wget https://huggingface.co/EleutherAI/gpt-j-6b/resolve/main/merges.txt +``` + +2. Convert the Hugging Face checkpoint into TensorRT LLM format. +Run below command lines in [`examples/models/contrib/gpt`](../gptj) directory. +```bash +# Build a float16 checkpoint using HF weights. +python convert_checkpoint.py --model_dir ./gpt-j-6b \ + --dtype float16 \ + --output_dir ./trt_ckpt/gptj_fp16_tp1/ + +# Build an int8 weight-only checkpoint using HF weights. +python convert_checkpoint.py --model_dir ./gpt-j-6b \ + --dtype float16 \ + --use_weight_only \ + --weight_only_precision int8 \ + --output_dir ./trt_ckpt/gptj_int8_tp1/ + +``` + +3. Build the weights stripped engine. +```bash +# Build with --strip_plan. Requires TRT>=10.0.0 +trtllm-build --checkpoint_dir ./trt_ckpt/gptj_fp16_tp1/ \ + --output_dir ./trt_engines/gptj_fp16_tp1/ \ + --gemm_plugin float16 \ + --max_batch_size=32 \ + --max_input_len=1919 \ + --max_seq_len=2047 \ + --strip_plan +``` + +4. Refit the engine. The refit engine lives at `${ENGINE_DIR}.refit`. +```bash +# --checkpoint_dir points to the path of the weights you want refit, in this case the original weights. +trtllm-refit --checkpoint_dir ./trt_ckpt/gptj_fp16_tp1/ --engine_dir ./trt_engines/gptj_fp16_tp1/ --output_dir ./trt_engines/gptj_fp16_tp1.refit/ +``` + +5. Verify the engine. +```bash +# Run the summarization task. +python3 ../summarize.py --engine_dir ./trt_engines/gptj_fp16_tp1.refit \ + --hf_model_dir ./gpt-j-6b \ + --batch_size 1 \ + --test_trt_llm \ + --tensorrt_llm_rouge1_threshold 14 \ + --data_type fp16 \ + --check_accuracy +``` + +### Llama-7b INT4 + +1. Download the llama-7b-hf checkpoint and saved in /llm-models/llama-models/llama-7b-hf/. + +2. Calibrate the checkpoint and convert into TensorRT LLM format. +Run below command lines in [`examples/models/core/llama`](../models/core/llama) directory. +```bash +# Calibrate INT4 using AMMO. +python ../quantization/quantize.py --model_dir /llm-models/llama-models/llama-7b-hf/ \ + --dtype float16 \ + --qformat int4_awq \ + --awq_block_size 128 \ + --output_dir ./quantized_int4-awq \ + --calib_size 32 +``` + +3. Build the weights stripped engine. +```bash +# Build with --strip_plan. Requires TRT>=10.0.0 +trtllm-build --checkpoint_dir ./quantized_int4-awq \ + --strip_plan \ + --gemm_plugin float16 \ + --output_dir trt_int4_AWQ +``` + +4. Refit the engine. +```bash +trtllm-refit --checkpoint_dir ./quantized_int4-awq \ + --engine_dir trt_int4_AWQ \ + --output_dir trt_int4_AWQ_full_from_wtless +``` + +5. Verify the engine. +```bash +python3 ../summarize.py --engine_dir trt_int4_AWQ_full_from_wtless \ + --hf_model_dir /llm-models/llama-models/llama-7b-hf/ \ + --batch_size 1 \ + --test_trt_llm \ + --check_accuracy +``` + +### Llama-7b FP16 + WoQ INT8 + +1. Download the llama-7b-hf checkpoint and saved in /llm-models/llama-models/llama-7b-hf/. + +2. Convert the checkpoint into TensorRT LLM format. +Run below command lines in [`examples/models/core/llama`](../models/core/llama) directory. +```bash +python3 convert_checkpoint.py --model_dir /llm-models/llama-models/llama-7b-hf/ \ + --output_dir ./llama-7b-hf-fp16-woq \ + --dtype float16 \ + --use_weight_only \ + --weight_only_precision int8 +``` + +3. Build the weights stripped engine. +```bash +# Build with --strip_plan. Requires TRT>=10.0.0 +trtllm-build --checkpoint_dir ./llama-7b-hf-fp16-woq \ + --output_dir ./engines/llama-7b-hf-fp16-woq-1gpu-wtless \ + --strip_plan \ + --gemm_plugin float16 +``` + +4. Refit the engine. +```bash +trtllm-refit --checkpoint_dir ./llama-7b-hf-fp16-woq \ + --engine_dir ./engines/llama-7b-hf-fp16-woq-1gpu-wtless \ + --output_dir ./engines/llama-7b-hf-fp16-woq-1gpu-wtless-to-full +``` + +5. Verify the engine. +```bash +python3 ../summarize.py --engine_dir ./engines/llama-7b-hf-fp16-woq-1gpu-wtless-to-full \ + --hf_model_dir /llm-models/llama-models/llama-7b-hf/ \ + --batch_size 1 \ + --test_trt_llm \ + --check_accuracy +``` + + +### Llama2-70b FP8 with TP=2 + +1. Download the llama-v2-70b-hf checkpoint and saved in /llm-models/llama-models-v2/llama-v2-70b-hf/. + +2. Calibrate the checkpoint and convert into TensorRT LLM format. +Run below command lines in [`examples/models/core/llama`](../models/core/llama) directory. +```bash +# Calibrate FP8 using AMMO. +python ../quantization/quantize.py --model_dir /llm-models/llama-models-v2/llama-v2-70b-hf/ \ + --dtype float16 \ + --qformat fp8 \ + --kv_cache_dtype fp8 \ + --output_dir ./llama2-70b-hf-fp8-tp2 \ + --calib_size 512 \ + --tp_size 2 +``` + +3. Build the weights stripped engine. +```bash +trtllm-build --checkpoint_dir ./llama2-70b-hf-fp8-tp2 \ + --output_dir engines/llama2-70b-hf-fp8-tp2 \ + --gemm_plugin float16 \ + --workers 2 +``` + +4. Refit the engine. +```bash +trtllm-refit --checkpoint_dir ./llama2-70b-hf-fp8-tp2 \ + --engine_dir engines/llama2-70b-hf-fp8-tp2 \ + --output_dir engines/llama2-70b-hf-fp8-tp2.refit +``` + +5. Verify the engine. +```bash +python3 ../summarize.py --engine_dir engines/llama2-70b-hf-fp8-tp2.refit \ + --hf_model_dir /llm-models/llama-models-v2/llama-v2-70b-hf/ \ + --batch_size 1 \ + --test_trt_llm \ + --check_accuracy +``` + + +## Engine Plan File Size Results + +| **Model** | **Full Engine Plan Size** | **Weight-Stripped Engine Plan Size** | +|:---------:|:----------:|:----:| +|llama-7b INT4 | 3.7GB | 5.3MB | +|llama-7b FP16 + WoQ INT8 | 6.54GB | 28.69MB | +|llama2-70b FP8 + TP=2 | 64.78GB | 60.61MB | + +## Prototype +### Checkpoint Pruner +The checkpoint pruner allows you to strip `Conv` and `Gemm` weights out of a TensorRT LLM [checkpoint](https://nvidia.github.io/TensorRT-LLM/0.21.0/architecture/checkpoint.html). Since these make up the vast majority of weights, the pruner will decrease the size of your checkpoint up to 99%. + +When building an engine with a pruned checkpoint, TensorRT LLM fills in the missing weights with random ones. These weights should later be [refit](#engine-refitter) with the original weights to preserve the intended behavior. + +Building an engine from a pruned checkpoint will also allow the engine to be [refit](#engine-refitter). + +#### Pruning a TensorRT LLM Checkpoint + +1. Install [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/blob/main/README.md) either through [pip](https://github.com/NVIDIA/TensorRT-LLM/blob/main/README.md#installation) or [from the source](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/installation/build-from-source.md). +2. Download a model of your choice and convert it to a TensorRT LLM checkpoint ([llama instructions](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/models/core/llama/README.md#usage)). +3. (Optional) Run the `trtllm-prune` command. +```bash +# Prunes the TRT-LLM checkpoint at ${CHECKPOINT_DIR}, and stores it in the directory ${CHECKPOINT_DIR}.pruned +trtllm-prune --checkpoint_dir ${CHECKPOINT_DIR} +``` + +The pruned checkpoint lives at `${CHECKPOINT_DIR}.pruned` by default, however, this can be overridden by issuing the `--out_dir` flag. + +4. Build the stripped engine. + +```bash +# From pruned checkpoint. +trtllm-build --checkpoint_dir ${CHECKPOINT_DIR}.pruned \ + --output_dir ${ENGINE_OUT_DIR} \ + ${EXTRA_ARGS} +``` diff --git a/examples/scaffolding/benchmarks/__main__.py b/examples/scaffolding/benchmarks/__main__.py index d5e5941967b1..6fc89f696244 100644 --- a/examples/scaffolding/benchmarks/__main__.py +++ b/examples/scaffolding/benchmarks/__main__.py @@ -14,7 +14,6 @@ """ import argparse -import os import sys from tensorrt_llm.scaffolding import TaskMetricsCollector @@ -22,7 +21,6 @@ from .agent_benchmark import async_agent_benchmark, async_burst_agent_benchmark from .benchmark_utils import run_benchmark_in_thread from .chat_benchmark import async_chat_benchmark -from .coder_benchmark import async_coder_benchmark from .multiround_chat_benchmark import async_multiround_chat_benchmark @@ -131,68 +129,6 @@ def parse_arguments(): help="[Agent only] Enable query collector for debugging", ) - # Coder agent parameters - parser.add_argument( - "--enable_coder", - action="store_true", - help="Enable Coder agent benchmark (uses Apiary sandboxes via ApiaryMCPWorker)", - ) - parser.add_argument( - "--coder_concurrency", - type=int, - default=32, - help="Concurrency for Coder agent benchmark (default: 32)", - ) - parser.add_argument( - "--coder_prompt_num", - type=int, - default=8, - help="Number of prompts for Coder benchmark (default: 8)", - ) - parser.add_argument( - "--coder_max_iterations", - type=int, - default=50, - help="Max tool-calling iterations per Coder request (default: 50)", - ) - parser.add_argument( - "--coder_max_connections", - type=int, - default=200, - help="Max concurrent Apiary sandbox connections for Coder (default: 200)", - ) - parser.add_argument( - "--mcp_url", - type=str, - default="http://0.0.0.0:8083/sse", - help="Coder Apiary MCP server URL (default: http://0.0.0.0:8083/sse)", - ) - parser.add_argument( - "--coder_image", - type=str, - default="ubuntu:22.04", - help="Docker image name used for Coder benchmark sandboxes (default: ubuntu:22.04)", - ) - parser.add_argument( - "--apiary_url", - type=str, - default=os.getenv("APIARY_URL", "http://127.0.0.1:8080"), - help="[Coder only] Apiary daemon URL used to register the sandbox image " - "(default: $APIARY_URL or http://127.0.0.1:8080)", - ) - parser.add_argument( - "--apiary_token", - type=str, - default=os.getenv("APIARY_API_TOKEN"), - help="[Coder only] Bearer token for the Apiary daemon (default: $APIARY_API_TOKEN)", - ) - parser.add_argument( - "--coder_rate", - type=float, - default=1.0, - help="[Rate mode] Poisson arrival rate (req/s) for Coder benchmark. Default: 1.0.", - ) - # Burst agent parameters parser.add_argument( "--enable_burst_agent", @@ -520,7 +456,6 @@ def parse_arguments(): # Benchmark registry: (async_func, display_name, flag_name) BENCHMARK_REGISTRY = [ (async_agent_benchmark, "Agent-Benchmark", "enable_normal_agent"), - (async_coder_benchmark, "Coder-Benchmark", "enable_coder"), (async_burst_agent_benchmark, "Burst-Agent-Benchmark", "enable_burst_agent"), (async_chat_benchmark, "Chat-Benchmark", "enable_chat"), (async_multiround_chat_benchmark, "Multiround-Chat-Benchmark", "enable_multiround_chat"), @@ -562,8 +497,8 @@ def main(): if not enabled_benchmarks: print( - "No benchmark enabled. Use --enable_normal_agent, --enable_coder, " - "--enable_burst_agent, --enable_chat, or --enable_multiround_chat" + "No benchmark enabled. Use --enable_normal_agent, --enable_burst_agent, " + "--enable_chat, or --enable_multiround_chat" ) sys.exit(1) diff --git a/examples/scaffolding/benchmarks/coder_benchmark.py b/examples/scaffolding/benchmarks/coder_benchmark.py deleted file mode 100644 index 5650c5d0d93a..000000000000 --- a/examples/scaffolding/benchmarks/coder_benchmark.py +++ /dev/null @@ -1,203 +0,0 @@ -"""Coder agent benchmark for scaffolding benchmarks. - -Runs the Coder agent against Apiary sandboxes with configurable concurrency. -Each concurrent request gets its own isolated sandbox via ApiaryMCPWorker. -""" - -import sys - -from apiary_client import AsyncApiary -from openai import AsyncOpenAI - -from tensorrt_llm.scaffolding import ApiaryMCPWorker, QueryCollector, TRTOpenaiWorker -from tensorrt_llm.scaffolding.benchmark import ScaffoldingBenchRequest, async_scaffolding_benchmark -from tensorrt_llm.scaffolding.contrib.Coder import create_coder_scaffolding_llm -from tensorrt_llm.scaffolding.load_generation_strategy import ( - ConcurrentStrategy, - PoissonRateStrategy, - UniformWarmupStrategy, -) - -from .benchmark_utils import print_benchmark_results, print_lock, shutdown_llm - -DEFAULT_CODER_PROMPTS = [ - "Add comprehensive error handling to all public functions in the project.", - "Implement a thread-safe LRU cache with configurable capacity.", - "Write a CLI tool that converts CSV files to JSON with streaming support.", - "Create a retry decorator with exponential backoff and jitter.", - "Implement a simple key-value store with TTL-based expiration.", - "Add input validation and type checking to the API endpoint handlers.", - "Write a log aggregation utility that merges and deduplicates log entries.", - "Create a configuration loader that supports YAML, JSON, and environment variables.", -] - - -def load_coder_prompts(num_prompts: int) -> list[str]: - """Load prompts for the Coder benchmark from built-in coding tasks.""" - prompts = DEFAULT_CODER_PROMPTS.copy() - if len(prompts) < num_prompts: - original = prompts.copy() - repeat = (num_prompts + len(original) - 1) // len(original) - prompts = [] - for i in range(repeat): - for p in original: - tag = f"[{i}]." if i > 0 else "" - prompts.append(f"{tag}{p}") - return prompts[:num_prompts] - - -async def register_coder_image(args) -> None: - """Register the Coder benchmark image with the Apiary daemon. - - The image used by every benchmark request must be registered before the - MCP server creates sandboxes against it (otherwise Apiary returns 404). - Failures are surfaced as a hard error so the benchmark doesn't silently - produce 502s for every iteration. - """ - image = getattr(args, "coder_image", "ubuntu:22.04") - apiary_url = getattr(args, "apiary_url", "http://127.0.0.1:8080") - apiary_token = getattr(args, "apiary_token", None) - - apiary = AsyncApiary( - apiary_url=apiary_url, - apiary_token=apiary_token, - images=[image], - ) - try: - if not await apiary.health_check(retries=10, interval=1.0): - raise RuntimeError( - f"Apiary daemon at {apiary_url} is not reachable. " - "Start it with `apiary init && apiary daemon --bind ...`." - ) - status = await apiary.load() - if status is not None and image in status.failed: - reason = next( - ( - entry.get("reason") - for entry in status.failed_images - if entry.get("name") == image - ), - "unknown", - ) - raise RuntimeError(f"Failed to register image {image!r} with Apiary: {reason}") - finally: - await apiary.close() - - -async def create_coder_resources(args): - """Create isolated resources for a Coder benchmark run. - - Returns: - Tuple of (llm, mcp_worker, generation_worker) for cleanup. - """ - client = AsyncOpenAI(api_key=args.openai_api_key, base_url=args.base_url) - generation_worker = TRTOpenaiWorker( - client, args.model, getattr(args, "kv_cache_hint_agent", False) - ) - - mcp_url = getattr(args, "mcp_url", "http://0.0.0.0:8083/sse") - max_conns = getattr(args, "coder_max_connections", 200) - mcp_worker = ApiaryMCPWorker(mcp_url, max_connections=max_conns) - - llm = create_coder_scaffolding_llm( - generation_worker, - mcp_worker, - max_tokens=getattr(args, "max_tokens_agent", 65536), - max_iterations=getattr(args, "coder_max_iterations", 50), - max_parallel_requests=getattr(args, "max_parallel_requests", 1024), - enable_statistics=getattr(args, "enable_statistics", False), - ) - - return llm, mcp_worker, generation_worker - - -async def cleanup_coder_resources(llm, mcp_worker): - """Cleanup Coder benchmark resources.""" - await mcp_worker.async_shutdown() - await shutdown_llm(llm) - - -async def run_coder_benchmark_core( - llm, prompts, concurrency, benchmark_name, args, use_poisson_arrival=True -): - """Core Coder benchmark logic. - - Args: - llm: The ScaffoldingLlm instance. - prompts: List of prompts to benchmark. - concurrency: Number of concurrent requests. - benchmark_name: Name for the benchmark (used in output). - args: Command line arguments. - - Returns: - Tuple of (results, requests_start_time, requests_execution_time, total_time). - """ - task_collection_types = {} - requests = [ - ScaffoldingBenchRequest( - prompt=prompt, - scope_params={"image": getattr(args, "coder_image", "ubuntu:22.04")}, - ) - for prompt in prompts - ] - - if use_poisson_arrival and getattr(args, "load_mode", "concurrent") == "rate": - strategy = PoissonRateStrategy( - rate=getattr(args, "coder_rate", 1.0), - random_seed=getattr(args, "rate_seed", 42), - ) - elif getattr(args, "warmup_window", None) is not None: - strategy = UniformWarmupStrategy( - num_requests=len(requests), - warmup_window=args.warmup_window, - max_concurrency=concurrency, - ) - else: - strategy = ConcurrentStrategy(concurrency=concurrency) - print(f" Strategy: {strategy}") - - ( - results, - requests_start_time, - requests_execution_time, - total_time, - ) = await async_scaffolding_benchmark(llm, task_collection_types, requests, strategy=strategy) - - print_benchmark_results( - benchmark_name, - results, - requests_start_time, - requests_execution_time, - total_time, - ) - - if getattr(args, "enable_query_collector", False): - QueryCollector.get_global_info() - with print_lock: - print(f"Query info dumped to query_result.json! ({benchmark_name})") - - return results, requests_start_time, requests_execution_time, total_time - - -async def async_coder_benchmark(args): - """Run the Coder agent benchmark. - - Returns: - Tuple of (results, requests_start_time, requests_execution_time, total_time). - """ - concurrency = getattr(args, "coder_concurrency", 32) - num_prompts = getattr(args, "coder_prompt_num", 8) - - await register_coder_image(args) - - llm, mcp_worker, _ = await create_coder_resources(args) - prompts = load_coder_prompts(num_prompts) - - with print_lock: - print(f"\n[Coder] Starting benchmark with {num_prompts} prompts, concurrency={concurrency}") - sys.stdout.flush() - - try: - return await run_coder_benchmark_core(llm, prompts, concurrency, "Coder", args) - finally: - await cleanup_coder_resources(llm, mcp_worker) diff --git a/examples/scaffolding/contrib/Coder/README.md b/examples/scaffolding/contrib/Coder/README.md deleted file mode 100644 index 9593e7114fa1..000000000000 --- a/examples/scaffolding/contrib/Coder/README.md +++ /dev/null @@ -1,260 +0,0 @@ -# Scaffolding Coder Agent - -Agentic coding system built on the TensorRT-LLM Scaffolding framework. The Coder agent uses an LLM for reasoning and planning, and executes filesystem and shell operations inside isolated Apiary sandboxes through a dedicated MCP server. - -## Architecture - -```text -LLM server <-> ScaffoldingLlm <-> Coder / SWEBenchCoder - | - v - ApiaryMCPWorker - | - v - examples/.../coder_mcp.py - | - v - apiary_client.ApiarySessionMux - | - v - Apiary daemon / sessions -``` - -Three services are involved: - -- Apiary daemon: manages sandbox sessions, image registry (populated at runtime via HTTP), and command execution -- `coder_mcp.py`: exposes the Coder tool surface over MCP SSE -- LLM server: OpenAI-compatible endpoint used by Scaffolding - -## Tooling Model - -The Coder agent expects these MCP tools: - -- `read_file` -- `list_dir` -- `grep_files` -- `exec` -- `shell` -- `update_plan` -- `think` -- `complete_task` - -File edits go through `shell` (e.g. `sed -i`, `cat <<'EOF' > path` heredocs, `tee`). - -`ApiaryMCPWorker` opens one SSE connection per Scaffolding execution scope, so parallel branches naturally get isolated sandboxes. - -## Prerequisites - -```bash -# Apiary Python bindings (shared with coder_mcp, runners, and SWE-bench helpers) -pip install /path/to/apiary/bindings/python -``` - -For HuggingFace SWE-bench datasets, install the `swebench` extra: - -```bash -pip install '/path/to/apiary/bindings/python[swebench]' -``` - -## Start Apiary - - -The recommended deployment method is the Apiary container shipped under `/apiary/docker-compose.yml`. - -### Container (recommended) - -```bash -cd /path/to/apiary - -# Build and start the container; 8080 is exposed on the host by default. -docker compose up -d -``` - -The container's entrypoint runs `apiary init && apiary daemon --bind 0.0.0.0:8080`, leaving you with an empty pool ready to accept image registrations from clients. - -Useful environment variables (see the Apiary README for the full list): - -| Variable | Default | Purpose | -|---|---|---| -| `APIARY_PORT` | `8080` | Host port to publish | -| `APIARY_BIND` | `0.0.0.0:8080` | Bind address inside the container | -| `APIARY_API_TOKEN` | (empty) | Bearer token for API auth (empty disables auth) | -| `APIARY_MAX_SANDBOXES` | `40` | Pool concurrency cap | -| `APIARY_LAYERS_DIR` | `/var/lib/apiary/layers` | Layer cache (named volume) | -| `APIARY_OVERLAY_DIR` | `/var/lib/apiary/overlays` | Overlay scratch (named volume) | - -Verify from the host: - -```bash -curl -s http://172.17.0.1:8080/healthz # {"status":"ok"} -curl -s http://172.17.0.1:8080/api/v1/status # pool counters + registered_images -``` - -### Native install - -Only needed when Docker is unavailable. Requires Linux 5.11+, cgroups v2 with delegation, and the `uidmap` package. - -```bash -cd /path/to/apiary -cargo build --release - -apiary init --max-sandboxes 40 -apiary daemon --bind 0.0.0.0:8080 -``` - -The Coder runners (`run_coder.py`, `run_swebench.py`, the benchmark) then register the images they need via `POST /api/v1/images` before they dispatch any work. - -## Start the MCP Server - -```bash -python examples/scaffolding/mcp/coder/coder_mcp.py \ - --apiary-url http://172.17.0.1:8080 \ - --default-image ubuntu:22.04 \ - --port 8083 -``` - -Key flags: - -- `--apiary-url` — Apiary daemon URL -- `--apiary-token` — bearer token for daemon auth -- `--mcp-token` — bearer token required on the SSE endpoint -- `--default-image` — fallback Docker image for sandbox sessions when an SSE client omits the `image` query parameter (must already be registered with the daemon) -- `--working-dir` — default sandbox working directory -- `--idle-timeout` — idle session reap timeout in seconds - -Per-request image selection works through the SSE `image` query parameter. `ApiaryMCPWorker.set_scope_params(..., image=...)` is how the Scaffolding runners select the correct sandbox image for each request. The image is expected to be registered with the daemon already; the runners listed below do that for you. - -## Run a Single Coder Task - -```bash -python examples/scaffolding/contrib/Coder/run_coder.py \ - --base_url http://localhost:8000/v1 \ - --model Qwen3/Qwen3-30B-A3B \ - --apiary_url http://172.17.0.1:8080 \ - --mcp_url http://127.0.0.1:8083/sse \ - --image ubuntu:22.04 \ - --prompt "Implement a thread-safe LRU cache in Python" \ - --max_iterations 50 \ - --enable_tracing -``` - -The runner registers `--image` with the Apiary daemon on startup (via `AsyncApiary`) and only dispatches the request once the image is loaded. - -Important flags: - -- `--image`: Docker image used for the request's sandbox (auto-registered) -- `--apiary_url` / `--apiary_token`: How to reach the Apiary daemon for image registration (defaults to `$APIARY_URL` / `$APIARY_API_TOKEN`) -- `--mcp_url`: `coder_mcp.py` SSE endpoint -- `--max_mcp_connections`: Max concurrent SSE / sandbox connections - -## Run SWE-bench - -The runner resolves the SWE-bench image set from the dataset, registers all unique images with the Apiary daemon (with per-image progress logging), and only then starts dispatching agent requests. - -```bash -python examples/scaffolding/contrib/Coder/run_swebench.py \ - --dataset lite \ - --split dev \ - --apiary_url http://172.17.0.1:8080 \ - --base_url http://localhost:8000/v1 \ - --model Qwen3/Qwen3-30B-A3B \ - --mcp_url http://0.0.0.0:8083/sse \ - --max_parallel_requests 16 -``` - -Useful flags: - -- `--apiary_url` / `--apiary_token`: Apiary daemon target -- `--apiary_load_timeout`: Bound the wait for image registration (default: no timeout — large SWE-bench splits can take a while on a cold cache) - -If the daemon is unreachable the runner aborts immediately with a clear error before any LLM work is dispatched. Per-image failures are logged and surfaced as warnings; only those instances are affected — the rest of the batch still runs against the images that did load. - -If you want to pre-load the image set out of band (for example to share a warmed-up daemon across multiple runs), use the `apiary-load-swebench` CLI shipped with `apiary-client[swebench]`: - -```bash -apiary-load-swebench --apiary-url http://172.17.0.1:8080 --dataset lite -``` - -The runner is idempotent: already-loaded images are reported as `alreadypresent` and skip the load pipeline. - -Outputs: - -- `swebench_output/---